2024-04-26T01:19:50.422 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T01:19:50.422 DEBUG:teuthology.run:Teuthology command: teuthology --name leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} --archive /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244 --verbose --owner scheduled_leonidus@teuthology -- /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/orig.config.yaml 2024-04-26T01:19:50.484 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T01:19:50.573 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7674244' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: afa1933b0fdbb6c99c947d1eda34d661d23cd327 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3448 sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f sleep_before_teardown: 0 subset: 200/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph suite_sha1: afa1933b0fdbb6c99c947d1eda34d661d23cd327 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBBNzsTXM9Wpr+WED+Yxz5T4VXMwHSAByAiUm8DupeamFZuCPKxNWYGvMMwZXxDMtu4ojQGVqHkls7nekgy6MeA= smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdzfaroBoqBqlk5NNToTIGJEX96lvgz/J2jXAz5KcOK3XxIWy0dvKJBN6qekKg7QOcuQlQmD4OxdR16cS+7ruY= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCww6rPjgGqw4giVR74eCtSZPt/X0gpwYQNij325FnYYCFpBIDgC0tlZQjfWzHiH3jPbnvvffyPhsPDj+jmPVlc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_01:02:28 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T01:19:50.573 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa; will attempt to use it 2024-04-26T01:19:50.574 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks 2024-04-26T01:19:50.574 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T01:19:50.575 INFO:teuthology.task.internal:Checking packages... 2024-04-26T01:19:50.599 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd69f13d827952bb07bb9b7973bca14f97d764d6f' 2024-04-26T01:19:50.599 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T01:19:50.599 INFO:teuthology.packaging:ref: None 2024-04-26T01:19:50.599 INFO:teuthology.packaging:tag: None 2024-04-26T01:19:50.599 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-26T01:19:50.599 INFO:teuthology.packaging:sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:19:50.600 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-26T01:19:50.775 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.gd69f13d8 2024-04-26T01:19:50.776 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T01:19:50.783 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T01:19:50.783 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T01:19:50.805 INFO:teuthology.task.internal:Saving configuration 2024-04-26T01:19:50.832 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T01:19:50.838 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T01:19:50.864 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:14:39.186321', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBBNzsTXM9Wpr+WED+Yxz5T4VXMwHSAByAiUm8DupeamFZuCPKxNWYGvMMwZXxDMtu4ojQGVqHkls7nekgy6MeA='} 2024-04-26T01:19:50.887 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:14:39.190078', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdzfaroBoqBqlk5NNToTIGJEX96lvgz/J2jXAz5KcOK3XxIWy0dvKJBN6qekKg7QOcuQlQmD4OxdR16cS+7ruY='} 2024-04-26T01:19:50.910 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:14:39.188288', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCww6rPjgGqw4giVR74eCtSZPt/X0gpwYQNij325FnYYCFpBIDgC0tlZQjfWzHiH3jPbnvvffyPhsPDj+jmPVlc='} 2024-04-26T01:19:50.910 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T01:19:50.917 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-26T01:19:50.917 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-26T01:19:50.917 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-26T01:19:50.917 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T01:19:51.022 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcb9f2620d0>, signals=[15]) 2024-04-26T01:19:51.022 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T01:19:51.030 INFO:teuthology.task.internal:Opening connections... 2024-04-26T01:19:51.030 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-04-26T01:19:51.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:19:51.115 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-04-26T01:19:51.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:19:51.201 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-04-26T01:19:51.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:19:51.293 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T01:19:51.352 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-04-26T01:19:51.379 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-04-26T01:19:51.379 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-04-26T01:19:51.447 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2024-04-26T01:19:51.448 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2024-04-26T01:19:51.448 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2024-04-26T01:19:51.448 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2024-04-26T01:19:51.448 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2024-04-26T01:19:51.448 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2024-04-26T01:19:51.448 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T01:19:51.448 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2024-04-26T01:19:51.448 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2024-04-26T01:19:51.448 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T01:19:51.449 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2024-04-26T01:19:51.449 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T01:19:51.449 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T01:19:51.449 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T01:19:51.449 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-04-26T01:19:51.475 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-04-26T01:19:51.494 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-04-26T01:19:51.495 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-04-26T01:19:51.557 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-04-26T01:19:51.557 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-04-26T01:19:51.557 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-04-26T01:19:51.557 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-04-26T01:19:51.557 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-04-26T01:19:51.557 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-04-26T01:19:51.557 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T01:19:51.557 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-04-26T01:19:51.557 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-04-26T01:19:51.558 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T01:19:51.558 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-04-26T01:19:51.558 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T01:19:51.558 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T01:19:51.558 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T01:19:51.558 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-04-26T01:19:51.590 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-04-26T01:19:51.608 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-04-26T01:19:51.609 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-04-26T01:19:51.666 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2024-04-26T01:19:51.666 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="9" 2024-04-26T01:19:51.666 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2024-04-26T01:19:51.666 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2024-04-26T01:19:51.666 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="9" 2024-04-26T01:19:51.667 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el9" 2024-04-26T01:19:51.667 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T01:19:51.667 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-04-26T01:19:51.667 INFO:teuthology.orchestra.run.smithi188.stdout:LOGO="fedora-logo-icon" 2024-04-26T01:19:51.667 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T01:19:51.667 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2024-04-26T01:19:51.667 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T01:19:51.667 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T01:19:51.667 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T01:19:51.667 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-04-26T01:19:51.698 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T01:19:51.795 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T01:19:52.050 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T01:19:52.050 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:19:52.053 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:19:52.057 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:19:52.076 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T01:19:52.086 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T01:19:52.086 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:19:52.111 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:19:52.116 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:19:52.216 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T01:19:52.298 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T01:19:52.317 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-04-26T01:19:52.317 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-26T01:19:52.317 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-26T01:19:52.318 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:19:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:19:52.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-26T01:19:52.511 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T01:19:52.511 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-26T01:19:52.512 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-04-26T01:19:52.529 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-437.el9.x86_64 2024-04-26T01:19:52.530 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T01:19:52.530 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-26T01:19:52.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum upgrade -y linux-firmware 2024-04-26T01:19:54.907 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-26T01:19:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-04-26T01:20:01.042 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.2 MB 00:00 2024-04-26T01:20:02.908 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-04-26T01:20:04.570 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-26T01:20:09.826 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-26T01:20:10.016 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:01 ago on Fri 26 Apr 2024 01:20:09 AM UTC. 2024-04-26T01:20:11.898 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-04-26T01:20:11.899 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-04-26T01:20:11.899 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-04-26T01:20:11.993 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T01:20:11.994 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T01:20:12.233 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-26T01:20:12.233 DEBUG:teuthology.orchestra.run.smithi006:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-26T01:20:12.239 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi006.front.sepia.ceph.com to complete... 2024-04-26T01:20:28.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26 01:20:28 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-26T01:20:28.862 INFO:teuthology.orchestra.run.smithi006.stderr:FINISHED --2024-04-26 01:20:28-- 2024-04-26T01:20:28.862 INFO:teuthology.orchestra.run.smithi006.stderr:Total wall clock time: 17s 2024-04-26T01:20:28.862 INFO:teuthology.orchestra.run.smithi006.stderr:Downloaded: 1 files, 341M in 16s (20.9 MB/s) 2024-04-26T01:20:28.865 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T01:20:28.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-26T01:20:31.056 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying... ######################################## 2024-04-26T01:20:31.176 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing... ######################################## 2024-04-26T01:20:31.291 INFO:teuthology.orchestra.run.smithi006.stdout:Updating / installing... 2024-04-26T01:20:39.570 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-26T01:21:19.684 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-26T01:21:19.685 DEBUG:teuthology.orchestra.run.smithi006:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-26T01:21:19.772 INFO:teuthology.orchestra.run.smithi006.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-26T01:21:19.772 INFO:teuthology.orchestra.run.smithi006.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-26T01:21:19.772 INFO:teuthology.orchestra.run.smithi006.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-26T01:21:19.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-26T01:21:19.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-26T01:21:19.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-26T01:21:19.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-26T01:21:19.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-26T01:21:19.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-26T01:21:19.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-26T01:21:19.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-26T01:21:19.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-26T01:21:19.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-26T01:21:19.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-26T01:21:19.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-26T01:21:19.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-26T01:21:19.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-26T01:21:19.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-26T01:21:19.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-26T01:21:19.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-26T01:21:19.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-26T01:21:19.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-26T01:21:19.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-26T01:21:19.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-26T01:21:19.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-26T01:21:19.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-26T01:21:19.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-26T01:21:19.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-26T01:21:19.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-26T01:21:19.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-26T01:21:19.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-26T01:21:19.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-26T01:21:19.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-26T01:21:19.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-26T01:21:19.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-26T01:21:19.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-26T01:21:19.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-26T01:21:19.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-26T01:21:19.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-26T01:21:19.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-26T01:21:19.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-26T01:21:19.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-26T01:21:19.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-26T01:21:19.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-26T01:21:19.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-26T01:21:19.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-26T01:21:19.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-26T01:21:19.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-26T01:21:19.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-26T01:21:19.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-26T01:21:19.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-26T01:21:19.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-26T01:21:19.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-26T01:21:19.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-26T01:21:19.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-26T01:21:19.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-26T01:21:19.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-26T01:21:19.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-26T01:21:19.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-26T01:21:19.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-26T01:21:19.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-26T01:21:19.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-26T01:21:19.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-26T01:21:19.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-26T01:21:19.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-26T01:21:19.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-26T01:21:19.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-26T01:21:19.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-26T01:21:19.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-26T01:21:19.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-26T01:21:19.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-26T01:21:19.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-26T01:21:19.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-26T01:21:19.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-26T01:21:19.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-26T01:21:19.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-26T01:21:19.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-26T01:21:19.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-26T01:21:19.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-26T01:21:19.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-26T01:21:19.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-26T01:21:19.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-26T01:21:19.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-26T01:21:19.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-26T01:21:19.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-26T01:21:19.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-26T01:21:19.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-26T01:21:19.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-26T01:21:19.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-26T01:21:19.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-26T01:21:19.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-26T01:21:19.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-26T01:21:19.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-26T01:21:19.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-26T01:21:19.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-26T01:21:19.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-26T01:21:19.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-26T01:21:19.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-26T01:21:19.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-26T01:21:19.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-26T01:21:19.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-26T01:21:19.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-26T01:21:19.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-26T01:21:19.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-26T01:21:19.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-26T01:21:19.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-26T01:21:19.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-26T01:21:19.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-26T01:21:19.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-26T01:21:19.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-26T01:21:19.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-26T01:21:19.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-26T01:21:19.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-26T01:21:19.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-26T01:21:19.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-26T01:21:19.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-26T01:21:19.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-26T01:21:19.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-26T01:21:19.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-26T01:21:19.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-26T01:21:19.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-26T01:21:19.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-26T01:21:19.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-26T01:21:19.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-26T01:21:19.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-26T01:21:19.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-26T01:21:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-26T01:21:19.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-26T01:21:19.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-26T01:21:19.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-26T01:21:19.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-26T01:21:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-26T01:21:19.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-26T01:21:19.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-26T01:21:19.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-26T01:21:19.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-26T01:21:19.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-26T01:21:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-26T01:21:19.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-26T01:21:19.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-26T01:21:19.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-26T01:21:19.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-26T01:21:19.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-26T01:21:19.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-26T01:21:19.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-26T01:21:19.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-26T01:21:19.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-26T01:21:19.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-26T01:21:19.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-26T01:21:19.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-26T01:21:19.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-26T01:21:19.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-26T01:21:19.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-26T01:21:19.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-26T01:21:19.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-26T01:21:19.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-26T01:21:19.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-26T01:21:19.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-26T01:21:19.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-26T01:21:19.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-26T01:21:19.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-26T01:21:19.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-26T01:21:19.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-26T01:21:19.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-26T01:21:19.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-26T01:21:19.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-26T01:21:19.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-26T01:21:19.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-26T01:21:19.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-26T01:21:19.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-26T01:21:19.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-26T01:21:19.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-26T01:21:19.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-26T01:21:19.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-26T01:21:19.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-26T01:21:19.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-26T01:21:19.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-26T01:21:19.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-26T01:21:19.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-26T01:21:19.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-26T01:21:19.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-26T01:21:19.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-26T01:21:19.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-26T01:21:19.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-26T01:21:19.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-26T01:21:19.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-26T01:21:19.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-26T01:21:19.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-26T01:21:19.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-26T01:21:19.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-26T01:21:19.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-26T01:21:19.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-26T01:21:19.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-26T01:21:19.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-26T01:21:19.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-26T01:21:19.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-26T01:21:19.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-26T01:21:19.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-26T01:21:19.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-26T01:21:19.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-26T01:21:19.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-26T01:21:19.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-26T01:21:19.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-26T01:21:19.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-26T01:21:19.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-26T01:21:19.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-26T01:21:19.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-26T01:21:19.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-26T01:21:19.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-26T01:21:19.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-26T01:21:19.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-26T01:21:19.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-26T01:21:19.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-26T01:21:19.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-26T01:21:19.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-26T01:21:19.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-26T01:21:19.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-26T01:21:19.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-26T01:21:19.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-26T01:21:19.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-26T01:21:19.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-26T01:21:19.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-26T01:21:19.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-26T01:21:19.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-26T01:21:19.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-26T01:21:19.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-26T01:21:19.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-26T01:21:19.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-26T01:21:19.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-26T01:21:19.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-26T01:21:19.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-26T01:21:19.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-26T01:21:19.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-26T01:21:19.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-26T01:21:19.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-26T01:21:19.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-26T01:21:19.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-26T01:21:19.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-26T01:21:19.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-26T01:21:19.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-26T01:21:19.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-26T01:21:19.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-26T01:21:19.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-26T01:21:19.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-26T01:21:19.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-26T01:21:19.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-26T01:21:19.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-26T01:21:19.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-26T01:21:19.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-26T01:21:19.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-26T01:21:19.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-26T01:21:19.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-26T01:21:19.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-26T01:21:19.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-26T01:21:19.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-26T01:21:19.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-26T01:21:19.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-26T01:21:19.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-26T01:21:19.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-26T01:21:19.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-26T01:21:19.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-26T01:21:19.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-26T01:21:19.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-26T01:21:19.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-26T01:21:19.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-26T01:21:19.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-26T01:21:19.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-26T01:21:19.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-26T01:21:19.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-26T01:21:19.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-26T01:21:19.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-26T01:21:19.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-26T01:21:19.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-26T01:21:19.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-26T01:21:19.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-26T01:21:19.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-26T01:21:19.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-26T01:21:19.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-26T01:21:19.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-26T01:21:19.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-26T01:21:19.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-26T01:21:19.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-26T01:21:19.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-26T01:21:19.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-26T01:21:19.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-26T01:21:19.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-26T01:21:19.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-26T01:21:19.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-26T01:21:19.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-26T01:21:19.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-26T01:21:19.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-26T01:21:19.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-26T01:21:19.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-26T01:21:19.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-26T01:21:19.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-26T01:21:19.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-26T01:21:19.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-26T01:21:19.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-26T01:21:19.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-26T01:21:19.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-26T01:21:19.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-26T01:21:19.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-26T01:21:19.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-26T01:21:19.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-26T01:21:19.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-26T01:21:19.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-26T01:21:19.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-26T01:21:19.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-26T01:21:19.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-26T01:21:19.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-26T01:21:19.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-26T01:21:19.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-26T01:21:19.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-26T01:21:19.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-26T01:21:19.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-26T01:21:19.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-26T01:21:19.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-26T01:21:19.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-26T01:21:19.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-26T01:21:19.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-26T01:21:19.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-26T01:21:19.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-26T01:21:19.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-26T01:21:19.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-26T01:21:19.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-26T01:21:19.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-26T01:21:19.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-26T01:21:19.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-26T01:21:19.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-26T01:21:19.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-26T01:21:19.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-26T01:21:19.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-26T01:21:19.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-26T01:21:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-26T01:21:19.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-26T01:21:19.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-26T01:21:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-26T01:21:19.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-26T01:21:19.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-26T01:21:19.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-26T01:21:19.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-26T01:21:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-26T01:21:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-26T01:21:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-26T01:21:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-26T01:21:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-26T01:21:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-26T01:21:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-26T01:21:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-26T01:21:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-26T01:21:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-26T01:21:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-26T01:21:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-26T01:21:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-26T01:21:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-26T01:21:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-26T01:21:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-26T01:21:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-26T01:21:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-26T01:21:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-26T01:21:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-26T01:21:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-26T01:21:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-26T01:21:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-26T01:21:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-26T01:21:19.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-26T01:21:19.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-26T01:21:19.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-26T01:21:19.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-26T01:21:19.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-26T01:21:19.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-26T01:21:19.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-26T01:21:19.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-26T01:21:19.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-26T01:21:19.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-26T01:21:19.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-26T01:21:19.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-26T01:21:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-26T01:21:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-26T01:21:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-26T01:21:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-26T01:21:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-26T01:21:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-26T01:21:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-26T01:21:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-26T01:21:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-26T01:21:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-26T01:21:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-26T01:21:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-26T01:21:19.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-26T01:21:19.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-26T01:21:19.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-26T01:21:19.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-26T01:21:19.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-26T01:21:19.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-26T01:21:19.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-26T01:21:19.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-26T01:21:19.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-26T01:21:19.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-26T01:21:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-26T01:21:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-26T01:21:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-26T01:21:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-26T01:21:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-26T01:21:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-26T01:21:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-26T01:21:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-26T01:21:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-26T01:21:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-26T01:21:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-26T01:21:19.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-26T01:21:19.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-26T01:21:19.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-26T01:21:19.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-26T01:21:19.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-26T01:21:19.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-26T01:21:19.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-26T01:21:19.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-26T01:21:19.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-26T01:21:19.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-26T01:21:19.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-26T01:21:19.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-26T01:21:19.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-26T01:21:19.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-26T01:21:19.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-26T01:21:19.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-26T01:21:19.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-26T01:21:19.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-26T01:21:19.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-26T01:21:19.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-26T01:21:19.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-26T01:21:19.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-26T01:21:19.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-26T01:21:19.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-26T01:21:19.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-26T01:21:19.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-26T01:21:19.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-26T01:21:19.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-26T01:21:19.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-26T01:21:19.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-26T01:21:19.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-26T01:21:19.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-26T01:21:19.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-26T01:21:19.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-26T01:21:19.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-26T01:21:19.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-26T01:21:19.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-26T01:21:19.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-26T01:21:19.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-26T01:21:19.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-26T01:21:19.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-26T01:21:19.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-26T01:21:19.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-26T01:21:19.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-26T01:21:19.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-26T01:21:19.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-26T01:21:19.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-26T01:21:19.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-26T01:21:19.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-26T01:21:19.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-26T01:21:19.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-26T01:21:19.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-26T01:21:19.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-26T01:21:19.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-26T01:21:19.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-26T01:21:19.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-26T01:21:19.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-26T01:21:19.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-26T01:21:19.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-26T01:21:19.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-26T01:21:19.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-26T01:21:19.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-26T01:21:19.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-26T01:21:19.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-26T01:21:19.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-26T01:21:19.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-26T01:21:19.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-26T01:21:19.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-26T01:21:19.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-26T01:21:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-26T01:21:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-26T01:21:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-26T01:21:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-26T01:21:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-26T01:21:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-26T01:21:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-26T01:21:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-26T01:21:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-26T01:21:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-26T01:21:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-26T01:21:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-26T01:21:19.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-26T01:21:19.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-26T01:21:19.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-26T01:21:19.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-26T01:21:19.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-26T01:21:19.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-26T01:21:19.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-26T01:21:19.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-26T01:21:19.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-26T01:21:19.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-26T01:21:19.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-26T01:21:19.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-26T01:21:19.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-26T01:21:19.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-26T01:21:19.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-26T01:21:19.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-26T01:21:19.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-26T01:21:19.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-26T01:21:19.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-26T01:21:19.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-26T01:21:19.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-26T01:21:19.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-26T01:21:19.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-26T01:21:19.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-26T01:21:19.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-26T01:21:19.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-26T01:21:19.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-26T01:21:19.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-26T01:21:19.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-26T01:21:19.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-26T01:21:19.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-26T01:21:19.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-26T01:21:19.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-26T01:21:19.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-26T01:21:19.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-26T01:21:19.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-26T01:21:19.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-26T01:21:19.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-26T01:21:19.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-26T01:21:19.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-26T01:21:19.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-26T01:21:19.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-26T01:21:19.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-26T01:21:19.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-26T01:21:19.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-26T01:21:19.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-26T01:21:19.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-26T01:21:19.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-26T01:21:19.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-26T01:21:19.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-26T01:21:19.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-26T01:21:19.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-26T01:21:19.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-26T01:21:19.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-26T01:21:19.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-26T01:21:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-26T01:21:19.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-26T01:21:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-26T01:21:19.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-26T01:21:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-26T01:21:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-26T01:21:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-26T01:21:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-26T01:21:19.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-26T01:21:19.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-26T01:21:19.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-26T01:21:19.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-26T01:21:19.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-26T01:21:19.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-26T01:21:19.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-26T01:21:19.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-26T01:21:19.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-26T01:21:19.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-26T01:21:19.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-26T01:21:19.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-26T01:21:19.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-26T01:21:19.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-26T01:21:19.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-26T01:21:19.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-26T01:21:19.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-26T01:21:19.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-26T01:21:19.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-26T01:21:19.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-26T01:21:19.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-26T01:21:19.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-26T01:21:19.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-26T01:21:19.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-26T01:21:19.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-26T01:21:19.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-26T01:21:19.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-26T01:21:19.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-26T01:21:19.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-26T01:21:19.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-26T01:21:19.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-26T01:21:19.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-26T01:21:19.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-26T01:21:19.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-26T01:21:19.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-26T01:21:19.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-26T01:21:19.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-26T01:21:19.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-26T01:21:19.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-26T01:21:19.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-26T01:21:19.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-26T01:21:19.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-26T01:21:19.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-26T01:21:19.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-26T01:21:19.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-26T01:21:19.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-26T01:21:19.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-26T01:21:19.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-26T01:21:19.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-26T01:21:19.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-26T01:21:19.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-26T01:21:19.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-26T01:21:19.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-26T01:21:19.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-26T01:21:19.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-26T01:21:19.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-26T01:21:19.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-26T01:21:19.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-26T01:21:19.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-26T01:21:19.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-26T01:21:19.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-26T01:21:19.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-26T01:21:19.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-26T01:21:19.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-26T01:21:19.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-26T01:21:19.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-26T01:21:19.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-26T01:21:19.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-26T01:21:19.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-26T01:21:19.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-26T01:21:19.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-26T01:21:19.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-26T01:21:19.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-26T01:21:19.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-26T01:21:19.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-26T01:21:19.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-26T01:21:19.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-26T01:21:19.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-26T01:21:19.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-26T01:21:19.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-26T01:21:19.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-26T01:21:19.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-26T01:21:19.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-26T01:21:19.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-26T01:21:19.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-26T01:21:19.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-26T01:21:19.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-26T01:21:19.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-26T01:21:19.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-26T01:21:19.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-26T01:21:19.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-26T01:21:19.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-26T01:21:19.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-26T01:21:19.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-26T01:21:19.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-26T01:21:19.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-26T01:21:19.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-26T01:21:19.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-26T01:21:19.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-26T01:21:19.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-26T01:21:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-26T01:21:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-26T01:21:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-26T01:21:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-26T01:21:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-26T01:21:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-26T01:21:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-26T01:21:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-26T01:21:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-26T01:21:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-26T01:21:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-26T01:21:19.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-26T01:21:19.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-26T01:21:19.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-26T01:21:19.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-26T01:21:19.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-26T01:21:19.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-26T01:21:19.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-26T01:21:19.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-26T01:21:19.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-26T01:21:19.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-26T01:21:19.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-26T01:21:19.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-26T01:21:19.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-26T01:21:19.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-26T01:21:19.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-26T01:21:19.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-26T01:21:19.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-26T01:21:19.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-26T01:21:19.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-26T01:21:19.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-26T01:21:19.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-26T01:21:19.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-26T01:21:19.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-26T01:21:19.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-26T01:21:19.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-26T01:21:19.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-26T01:21:19.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-26T01:21:19.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-26T01:21:19.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-26T01:21:19.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-26T01:21:19.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-26T01:21:19.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-26T01:21:19.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-26T01:21:19.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-26T01:21:19.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-26T01:21:19.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-26T01:21:19.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-26T01:21:19.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-26T01:21:19.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-26T01:21:19.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-26T01:21:19.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-26T01:21:19.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-26T01:21:19.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-26T01:21:19.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-26T01:21:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-26T01:21:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-26T01:21:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-26T01:21:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-26T01:21:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-26T01:21:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-26T01:21:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-26T01:21:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-26T01:21:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-26T01:21:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-26T01:21:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-26T01:21:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-26T01:21:19.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-26T01:21:19.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-26T01:21:19.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-26T01:21:19.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-26T01:21:19.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-26T01:21:19.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-26T01:21:19.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-26T01:21:19.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-26T01:21:19.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-26T01:21:19.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-26T01:21:19.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-26T01:21:19.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-26T01:21:19.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-26T01:21:19.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-26T01:21:19.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-26T01:21:19.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-26T01:21:19.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-26T01:21:19.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-26T01:21:19.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-26T01:21:19.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-26T01:21:19.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-26T01:21:19.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-26T01:21:19.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-26T01:21:19.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-26T01:21:19.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-26T01:21:19.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-26T01:21:19.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-26T01:21:19.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-26T01:21:19.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-26T01:21:19.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-26T01:21:19.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-26T01:21:19.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-26T01:21:19.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-26T01:21:19.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-26T01:21:19.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-26T01:21:19.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-26T01:21:19.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-26T01:21:19.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-26T01:21:19.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-26T01:21:19.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-26T01:21:19.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-26T01:21:19.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-26T01:21:19.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-26T01:21:19.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-26T01:21:19.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-26T01:21:19.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-26T01:21:19.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-26T01:21:19.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-26T01:21:19.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-26T01:21:19.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-26T01:21:19.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-26T01:21:19.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-26T01:21:19.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-26T01:21:19.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-26T01:21:19.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-26T01:21:19.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-26T01:21:19.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-26T01:21:19.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-26T01:21:19.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-26T01:21:19.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-26T01:21:19.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-26T01:21:19.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-26T01:21:19.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-26T01:21:19.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-26T01:21:19.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-26T01:21:19.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-26T01:21:19.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-26T01:21:19.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-26T01:21:19.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-26T01:21:19.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-26T01:21:19.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-26T01:21:19.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-26T01:21:19.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-26T01:21:19.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-26T01:21:19.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-26T01:21:19.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-26T01:21:19.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-26T01:21:19.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-26T01:21:19.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-26T01:21:19.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-26T01:21:19.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-26T01:21:19.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-26T01:21:19.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-26T01:21:19.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-26T01:21:19.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-26T01:21:19.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-26T01:21:19.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-26T01:21:19.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-26T01:21:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-26T01:21:19.889 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-26T01:21:19.890 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-26T01:21:19.891 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-26T01:21:19.892 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-26T01:21:19.893 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-26T01:21:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-26T01:21:19.895 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-26T01:21:19.895 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-26T01:21:19.895 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-26T01:21:19.895 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-26T01:21:19.895 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-26T01:21:19.895 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-26T01:21:19.895 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-26T01:21:19.895 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-26T01:21:19.895 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-26T01:21:19.895 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-26T01:21:19.895 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-26T01:21:19.895 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-26T01:21:19.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-26T01:21:19.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-26T01:21:19.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-26T01:21:19.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-26T01:21:19.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-26T01:21:19.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-26T01:21:19.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-26T01:21:19.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-26T01:21:19.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-26T01:21:19.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-26T01:21:19.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-26T01:21:19.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-26T01:21:19.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-26T01:21:19.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-26T01:21:19.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-26T01:21:19.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-26T01:21:19.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-26T01:21:19.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-26T01:21:19.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-26T01:21:19.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-26T01:21:19.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-26T01:21:19.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-26T01:21:19.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-26T01:21:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-26T01:21:19.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-26T01:21:19.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-26T01:21:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-26T01:21:19.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-26T01:21:19.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-26T01:21:19.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-26T01:21:19.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-26T01:21:19.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-26T01:21:19.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-26T01:21:19.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-26T01:21:19.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-26T01:21:19.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-26T01:21:19.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-26T01:21:19.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-26T01:21:19.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-26T01:21:19.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-26T01:21:19.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-26T01:21:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-26T01:21:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-26T01:21:19.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-26T01:21:19.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-26T01:21:19.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-26T01:21:19.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-26T01:21:19.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-26T01:21:19.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-26T01:21:19.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-26T01:21:19.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-26T01:21:19.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-26T01:21:19.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-26T01:21:19.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-26T01:21:19.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-26T01:21:19.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-26T01:21:19.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-26T01:21:19.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-26T01:21:19.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-26T01:21:19.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-26T01:21:19.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-26T01:21:19.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-26T01:21:19.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-26T01:21:19.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-26T01:21:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-26T01:21:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-26T01:21:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-26T01:21:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-26T01:21:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-26T01:21:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-26T01:21:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-26T01:21:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-26T01:21:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-26T01:21:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-26T01:21:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-26T01:21:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-26T01:21:19.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-26T01:21:19.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-26T01:21:19.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-26T01:21:19.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-26T01:21:19.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-26T01:21:19.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-26T01:21:19.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-26T01:21:19.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-26T01:21:19.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-26T01:21:19.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-26T01:21:19.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-26T01:21:19.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-26T01:21:19.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-26T01:21:19.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-26T01:21:19.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-26T01:21:19.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-26T01:21:19.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-26T01:21:19.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-26T01:21:19.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-26T01:21:19.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-26T01:21:19.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-26T01:21:19.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-26T01:21:19.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-26T01:21:19.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-26T01:21:19.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-26T01:21:19.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-26T01:21:19.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-26T01:21:19.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-26T01:21:19.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-26T01:21:19.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-26T01:21:19.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-26T01:21:19.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-26T01:21:19.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-26T01:21:19.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-26T01:21:19.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-26T01:21:19.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-26T01:21:19.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-26T01:21:19.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-26T01:21:19.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-26T01:21:19.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-26T01:21:19.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-26T01:21:19.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-26T01:21:19.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T01:21:19.932 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-26T01:21:19.932 DEBUG:teuthology.orchestra.run.smithi006:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-26T01:21:19.957 INFO:teuthology.orchestra.run.smithi006.stdout:postinstall scriptlet (using /bin/sh): 2024-04-26T01:21:19.958 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T01:21:19.958 INFO:teuthology.orchestra.run.smithi006.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T01:21:19.958 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-04-26T01:21:19.958 INFO:teuthology.orchestra.run.smithi006.stdout:for file in vmlinuz System.map config; do 2024-04-26T01:21:19.958 INFO:teuthology.orchestra.run.smithi006.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-26T01:21:19.958 INFO:teuthology.orchestra.run.smithi006.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-26T01:21:19.958 INFO:teuthology.orchestra.run.smithi006.stdout: fi 2024-04-26T01:21:19.958 INFO:teuthology.orchestra.run.smithi006.stdout:done 2024-04-26T01:21:19.958 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-26T01:21:19.958 INFO:teuthology.orchestra.run.smithi006.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-26T01:21:19.958 INFO:teuthology.orchestra.run.smithi006.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T01:21:19.959 INFO:teuthology.orchestra.run.smithi006.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T01:21:19.959 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-04-26T01:21:19.959 INFO:teuthology.orchestra.run.smithi006.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-26T01:21:19.959 INFO:teuthology.orchestra.run.smithi006.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-26T01:21:19.959 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-04-26T01:21:19.959 INFO:teuthology.orchestra.run.smithi006.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-26T01:21:19.959 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T01:21:19.959 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-26T01:21:19.959 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-04-26T01:21:19.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2024-04-26T01:21:20.035 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 68.el9 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8264254 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-26T01:21:20.036 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2024-04-26T01:21:20.037 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2024-04-26T01:21:20.037 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-26T01:21:20.037 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2024-04-26T01:21:20.037 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2024-04-26T01:21:20.037 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:21:20.037 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-26T01:21:20.037 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-26T01:21:20.037 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-26T01:21:20.037 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2024-04-26T01:21:20.037 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:21:20.037 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2024-04-26T01:21:20.038 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-26T01:21:20.038 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 6.9.0-rc3-g806ec36b2112 2024-04-26T01:21:20.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-26T01:21:20.720 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-04-26T01:21:22.270 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-26T01:21:22.315 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-04-26T01:21:22.317 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-04-26T01:21:22.334 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.zicDRamF9o 2024-04-26T01:21:22.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zicDRamF9o 2024-04-26T01:21:22.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.zicDRamF9o 2024-04-26T01:21:22.538 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.zicDRamF9o is 6KB 2024-04-26T01:21:22.587 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.zicDRamF9o 2024-04-26T01:21:22.602 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2024-04-26T01:21:22.666 INFO:teuthology.orchestra.run.smithi006.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-26T01:21:22.666 INFO:teuthology.orchestra.run.smithi006.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-26T01:21:22.666 INFO:teuthology.orchestra.run.smithi006.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-26T01:21:22.666 INFO:teuthology.orchestra.run.smithi006.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-26T01:21:22.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-26T01:21:22.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-04-26T01:21:52.837 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-26T01:21:52.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-26T01:21:52.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:22:11.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-04-26T01:22:20.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-26T01:22:20.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:23:20.362 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T01:23:32.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-26T01:23:32.365 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:23:32.771 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-04-26T01:23:33.398 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-26T01:23:33.399 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-26T01:23:33.399 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-04-26T01:23:33.453 INFO:teuthology.orchestra.run.smithi006.stdout:6.9.0-rc3-g806ec36b2112 2024-04-26T01:23:33.453 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-26T01:23:33.453 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T01:23:33.453 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-26T01:23:34.455 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:34.455 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T01:23:34.462 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T01:23:34.462 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:23:34.469 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:23:34.472 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:23:34.521 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T01:23:34.527 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T01:23:34.538 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T01:23:34.538 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:23:34.543 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:23:34.546 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:23:34.654 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T01:23:34.661 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T01:23:34.661 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:23:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:34.678 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:23:34.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:34.695 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:23:34.702 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:23:34.721 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:23:34.753 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:23:34.776 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:23:34.781 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:23:34.793 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:23:34.829 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:23:34.888 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:23:34.893 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T01:23:34.900 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T01:23:34.900 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:23:34.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:23:34.907 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:23:34.989 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T01:23:34.998 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T01:23:34.998 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:23:35.003 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:23:35.006 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:23:35.054 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:23:35.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:23:35.233 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:23:35.321 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:23:35.408 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:23:35.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:23:35.492 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:23:35.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:23:35.589 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:23:35.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:23:35.716 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-26T01:23:35.716 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:23:35.784 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:23:35.828 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:23:35.876 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:23:35.966 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:23:36.002 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-26T01:23:36.002 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:23:36.067 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-04-26T01:23:36.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-04-26T01:23:36.077 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-04-26T01:23:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T01:23:36.142 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T01:23:36.182 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T01:23:36.651 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T01:23:36.658 INFO:teuthology.task.internal:Starting timer... 2024-04-26T01:23:36.658 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T01:23:36.667 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T01:23:36.674 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-26T01:23:36.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2024-04-26T01:23:36.795 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2024-04-26T01:23:36.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-04-26T01:23:36.920 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-04-26T01:23:36.922 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2024-04-26T01:23:36.983 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2024-04-26T01:23:36.985 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T01:23:36.985 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2024-04-26T01:23:37.048 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2024-04-26T01:23:37.048 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-04-26T01:23:37.076 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-04-26T01:23:37.076 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2024-04-26T01:23:37.098 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2024-04-26T01:23:37.098 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2024-04-26T01:23:37.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T01:23:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:37.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T01:23:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:37.203 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T01:23:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:37.229 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T01:23:37.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2024-04-26T01:23:37.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-04-26T01:23:37.333 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2024-04-26T01:23:37.358 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T01:23:37.368 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T01:23:37.385 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T01:23:37.386 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi029.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T01:28:49.655 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-04-26T01:28:49.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-26T01:28:49.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:28:49.777 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-04-26T01:28:49.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-26T01:28:49.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-26T01:28:49.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:28:50.002 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-04-26T01:28:50.087 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-26T01:28:50.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-26T01:28:50.088 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:28:50.159 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-04-26T01:28:50.240 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-26T01:28:50.241 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T01:28:50.254 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T01:28:50.255 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:28:50.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:28:50.263 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:28:50.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:28:50.265 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:28:50.265 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:28:50.295 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T01:28:50.308 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T01:28:50.310 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T01:28:50.322 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T01:28:50.335 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-04-26T01:28:50.346 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2024-04-26T01:28:50.346 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-04-26T01:28:50.356 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T01:28:50.356 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2024-04-26T01:28:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T01:28:50.369 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T01:28:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T01:28:50.382 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T01:28:50.400 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T01:28:50.438 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-04-26T01:28:50.439 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-04-26T01:28:50.441 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T01:28:50.441 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-26T01:28:50.442 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:50.442 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:50.442 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:50.442 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:50.443 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T01:28:50.443 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-04-26T01:28:50.443 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:50.443 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:50.443 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:50.443 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:50.479 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2024-04-26T01:28:50.514 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2024-04-26T01:28:50.560 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T01:28:50.606 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T01:28:50.756 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-04-26T01:28:50.765 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T01:28:50.765 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-04-26T01:28:50.765 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:50.765 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:50.766 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:50.766 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:50.767 INFO:teuthology.run_tasks:Running task install... 2024-04-26T01:28:50.776 DEBUG:teuthology.task.install:project ceph 2024-04-26T01:28:50.776 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'}} 2024-04-26T01:28:50.776 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'} 2024-04-26T01:28:50.777 INFO:teuthology.task.install:Using flavor: default 2024-04-26T01:28:50.782 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T01:28:50.782 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T01:28:50.783 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-26T01:28:50.783 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:28:50.785 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-26T01:28:50.785 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:28:50.786 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-26T01:28:50.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:28:50.937 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-26T01:28:50.937 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-26T01:28:50.941 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-26T01:28:50.941 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-26T01:28:51.006 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-26T01:28:51.007 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-26T01:28:51.213 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T01:28:51.213 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-26T01:28:51.214 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T01:28:51.242 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T01:28:51.242 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T01:28:51.253 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T01:28:51.253 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-26T01:28:51.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T01:28:51.257 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T01:28:51.257 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:28:51.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T01:28:51.283 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T01:28:51.283 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T01:28:51.308 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T01:28:51.352 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T01:28:51.352 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T01:28:51.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T01:28:51.383 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T01:28:51.432 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T01:28:51.447 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2024-04-26T01:28:51.449 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2024-04-26T01:28:51.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T01:28:51.498 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-04-26T01:28:51.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-04-26T01:28:51.604 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T01:28:51.675 INFO:teuthology.orchestra.run.smithi188.stdout:48 files removed 2024-04-26T01:28:51.689 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2024-04-26T01:28:51.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2024-04-26T01:28:51.702 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T01:28:51.739 INFO:teuthology.orchestra.run.smithi029.stdout:48 files removed 2024-04-26T01:28:51.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T01:28:52.029 INFO:teuthology.orchestra.run.smithi006.stdout:48 files removed 2024-04-26T01:28:52.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T01:28:52.636 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 123 kB/s | 83 kB 00:00 2024-04-26T01:28:52.683 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 130 kB/s | 83 kB 00:00 2024-04-26T01:28:53.257 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 113 kB/s | 83 kB 00:00 2024-04-26T01:28:53.262 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 20 kB/s | 12 kB 00:00 2024-04-26T01:28:53.267 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 22 kB/s | 12 kB 00:00 2024-04-26T01:28:53.711 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-04-26T01:28:53.762 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 4.1 kB/s | 1.9 kB 00:00 2024-04-26T01:28:53.958 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 19 kB/s | 12 kB 00:00 2024-04-26T01:28:54.645 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-04-26T01:28:54.889 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 7.0 MB/s | 8.0 MB 00:01 2024-04-26T01:28:55.307 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.0 MB 00:01 2024-04-26T01:28:55.561 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 9.7 MB/s | 8.0 MB 00:00 2024-04-26T01:28:57.210 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-26T01:28:57.283 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-26T01:28:57.780 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-04-26T01:29:01.476 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-26T01:29:01.555 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-26T01:29:02.531 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.2 MB 00:00 2024-04-26T01:29:03.054 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-04-26T01:29:03.340 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-04-26T01:29:04.814 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-26T01:29:05.417 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-04-26T01:29:07.386 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-26T01:29:10.203 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-26T01:29:12.464 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:29:12.464 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:29:12.476 INFO:teuthology.orchestra.run.smithi029.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T01:29:12.476 INFO:teuthology.orchestra.run.smithi029.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T01:29:12.477 INFO:teuthology.orchestra.run.smithi029.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T01:29:12.478 INFO:teuthology.orchestra.run.smithi029.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T01:29:12.480 INFO:teuthology.orchestra.run.smithi029.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T01:29:12.482 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T01:29:12.483 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T01:29:12.547 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-04-26T01:29:12.572 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-04-26T01:29:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-04-26T01:29:12.572 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-04-26T01:29:12.572 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-04-26T01:29:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T01:29:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-26T01:29:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:29:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-26T01:29:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-26T01:29:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-26T01:29:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-26T01:29:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-26T01:29:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-26T01:29:12.574 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-26T01:29:12.574 INFO:teuthology.orchestra.run.smithi029.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T01:29:12.574 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T01:29:12.574 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T01:29:12.574 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T01:29:12.574 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T01:29:12.574 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T01:29:12.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-26T01:29:12.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-26T01:29:12.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-26T01:29:12.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-26T01:29:12.574 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-26T01:29:12.575 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-26T01:29:12.576 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T01:29:12.577 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-26T01:29:12.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T01:29:12.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T01:29:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T01:29:12.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T01:29:12.583 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T01:29:12.583 INFO:teuthology.orchestra.run.smithi029.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T01:29:12.583 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T01:29:12.583 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T01:29:12.583 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-04-26T01:29:12.583 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T01:29:12.583 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:29:12.583 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-04-26T01:29:12.583 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-04-26T01:29:12.583 INFO:teuthology.orchestra.run.smithi029.stdout:Install 148 Packages 2024-04-26T01:29:12.583 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 4 Packages 2024-04-26T01:29:12.583 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:29:12.584 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 222 M 2024-04-26T01:29:12.584 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-04-26T01:29:13.569 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-26T01:29:13.889 INFO:teuthology.orchestra.run.smithi029.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 28 kB/s | 6.6 kB 00:00 2024-04-26T01:29:14.165 INFO:teuthology.orchestra.run.smithi029.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 3.2 MB/s | 890 kB 00:00 2024-04-26T01:29:14.233 INFO:teuthology.orchestra.run.smithi029.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 2.1 MB/s | 145 kB 00:00 2024-04-26T01:29:14.776 INFO:teuthology.orchestra.run.smithi029.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 4.4 MB/s | 2.4 MB 00:00 2024-04-26T01:29:15.101 INFO:teuthology.orchestra.run.smithi029.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 4.6 MB/s | 1.5 MB 00:00 2024-04-26T01:29:16.047 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:29:16.048 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:29:16.070 INFO:teuthology.orchestra.run.smithi006.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T01:29:16.071 INFO:teuthology.orchestra.run.smithi006.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T01:29:16.072 INFO:teuthology.orchestra.run.smithi006.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T01:29:16.073 INFO:teuthology.orchestra.run.smithi006.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T01:29:16.076 INFO:teuthology.orchestra.run.smithi006.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T01:29:16.079 INFO:teuthology.orchestra.run.smithi006.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T01:29:16.081 INFO:teuthology.orchestra.run.smithi006.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T01:29:16.118 INFO:teuthology.orchestra.run.smithi029.stdout:(6/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 2.2 MB/s | 5.4 MB 00:02 2024-04-26T01:29:16.157 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-04-26T01:29:16.171 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-04-26T01:29:16.171 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-04-26T01:29:16.171 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-04-26T01:29:16.171 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-04-26T01:29:16.171 INFO:teuthology.orchestra.run.smithi006.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-26T01:29:16.172 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T01:29:16.173 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T01:29:16.174 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T01:29:16.175 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T01:29:16.176 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T01:29:16.177 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T01:29:16.178 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T01:29:16.180 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T01:29:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-04-26T01:29:16.182 INFO:teuthology.orchestra.run.smithi006.stdout:Install 148 Packages 2024-04-26T01:29:16.183 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 4 Packages 2024-04-26T01:29:16.183 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:29:16.183 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 222 M 2024-04-26T01:29:16.183 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-04-26T01:29:16.285 INFO:teuthology.orchestra.run.smithi029.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 4.0 MB/s | 4.7 MB 00:01 2024-04-26T01:29:17.338 INFO:teuthology.orchestra.run.smithi006.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 13 kB/s | 6.6 kB 00:00 2024-04-26T01:29:18.871 INFO:teuthology.orchestra.run.smithi006.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 581 kB/s | 890 kB 00:01 2024-04-26T01:29:19.196 INFO:teuthology.orchestra.run.smithi006.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 446 kB/s | 145 kB 00:00 2024-04-26T01:29:19.361 INFO:teuthology.orchestra.run.smithi029.stdout:(8/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 3.4 MB/s | 11 MB 00:03 2024-04-26T01:29:19.452 INFO:teuthology.orchestra.run.smithi029.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 275 kB/s | 25 kB 00:00 2024-04-26T01:29:20.246 INFO:teuthology.orchestra.run.smithi006.stdout:(4/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 1.6 MB/s | 5.4 MB 00:03 2024-04-26T01:29:21.038 INFO:teuthology.orchestra.run.smithi006.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 1.9 MB/s | 1.5 MB 00:00 2024-04-26T01:29:22.863 INFO:teuthology.orchestra.run.smithi006.stdout:(6/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 659 kB/s | 2.4 MB 00:03 2024-04-26T01:29:22.952 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:19 2024-04-26T01:29:23.128 INFO:teuthology.orchestra.run.smithi029.stdout:(10/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 2.4 MB/s | 17 MB 00:07 2024-04-26T01:29:23.214 INFO:teuthology.orchestra.run.smithi006.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 2.2 MB/s | 4.7 MB 00:02 2024-04-26T01:29:23.270 INFO:teuthology.orchestra.run.smithi029.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 1.5 MB/s | 225 kB 00:00 2024-04-26T01:29:23.362 INFO:teuthology.orchestra.run.smithi029.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 295 kB/s | 27 kB 00:00 2024-04-26T01:29:23.438 INFO:teuthology.orchestra.run.smithi029.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 421 kB/s | 32 kB 00:00 2024-04-26T01:29:23.763 INFO:teuthology.orchestra.run.smithi029.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 2.2 MB/s | 747 kB 00:00 2024-04-26T01:29:24.106 INFO:teuthology.orchestra.run.smithi029.stdout:(15/152): ceph-common-19.0.0-3156.gd69f13d8.el7 2.0 MB/s | 21 MB 00:10 2024-04-26T01:29:24.131 INFO:teuthology.orchestra.run.smithi029.stdout:(16/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 444 kB/s | 163 kB 00:00 2024-04-26T01:29:24.214 INFO:teuthology.orchestra.run.smithi006.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 2.9 MB/s | 21 MB 00:07 2024-04-26T01:29:24.248 INFO:teuthology.orchestra.run.smithi029.stdout:(17/152): librados-devel-19.0.0-3156.gd69f13d8. 891 kB/s | 126 kB 00:00 2024-04-26T01:29:24.298 INFO:teuthology.orchestra.run.smithi006.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 305 kB/s | 25 kB 00:00 2024-04-26T01:29:24.400 INFO:teuthology.orchestra.run.smithi029.stdout:(18/152): libradosstriper1-19.0.0-3156.gd69f13d 1.8 MB/s | 500 kB 00:00 2024-04-26T01:29:24.484 INFO:teuthology.orchestra.run.smithi029.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 540 kB/s | 45 kB 00:00 2024-04-26T01:29:24.585 INFO:teuthology.orchestra.run.smithi029.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 1.3 MB/s | 135 kB 00:00 2024-04-26T01:29:24.685 INFO:teuthology.orchestra.run.smithi029.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.6 MB/s | 161 kB 00:00 2024-04-26T01:29:24.869 INFO:teuthology.orchestra.run.smithi029.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 1.7 MB/s | 320 kB 00:00 2024-04-26T01:29:25.045 INFO:teuthology.orchestra.run.smithi029.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 1.7 MB/s | 298 kB 00:00 2024-04-26T01:29:25.154 INFO:teuthology.orchestra.run.smithi029.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 916 kB/s | 99 kB 00:00 2024-04-26T01:29:25.254 INFO:teuthology.orchestra.run.smithi029.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 852 kB/s | 85 kB 00:00 2024-04-26T01:29:26.696 INFO:teuthology.orchestra.run.smithi029.stdout:(26/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 2.1 MB/s | 3.1 MB 00:01 2024-04-26T01:29:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:(27/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 2.1 MB/s | 5.4 MB 00:02 2024-04-26T01:29:26.839 INFO:teuthology.orchestra.run.smithi029.stdout:(28/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 1.2 MB/s | 170 kB 00:00 2024-04-26T01:29:26.898 INFO:teuthology.orchestra.run.smithi029.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3156.g 408 kB/s | 34 kB 00:00 2024-04-26T01:29:26.932 INFO:teuthology.orchestra.run.smithi029.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.5 MB/s | 142 kB 00:00 2024-04-26T01:29:27.937 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-26T01:29:28.164 INFO:teuthology.orchestra.run.smithi006.stdout:(10/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 2.1 MB/s | 11 MB 00:04 2024-04-26T01:29:28.314 INFO:teuthology.orchestra.run.smithi006.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 1.5 MB/s | 225 kB 00:00 2024-04-26T01:29:28.407 INFO:teuthology.orchestra.run.smithi006.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 295 kB/s | 27 kB 00:00 2024-04-26T01:29:28.490 INFO:teuthology.orchestra.run.smithi006.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 384 kB/s | 32 kB 00:00 2024-04-26T01:29:28.534 INFO:teuthology.orchestra.run.smithi029.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 2.2 MB/s | 3.5 MB 00:01 2024-04-26T01:29:28.666 INFO:teuthology.orchestra.run.smithi029.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.8 MB/s | 249 kB 00:00 2024-04-26T01:29:28.752 INFO:teuthology.orchestra.run.smithi029.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 580 kB/s | 49 kB 00:00 2024-04-26T01:29:28.828 INFO:teuthology.orchestra.run.smithi029.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 219 kB/s | 16 kB 00:00 2024-04-26T01:29:28.849 INFO:teuthology.orchestra.run.smithi006.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 2.0 MB/s | 747 kB 00:00 2024-04-26T01:29:28.953 INFO:teuthology.orchestra.run.smithi029.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 2.1 MB/s | 273 kB 00:00 2024-04-26T01:29:28.991 INFO:teuthology.orchestra.run.smithi006.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.1 MB/s | 163 kB 00:00 2024-04-26T01:29:29.075 INFO:teuthology.orchestra.run.smithi006.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.5 MB/s | 126 kB 00:00 2024-04-26T01:29:29.195 INFO:teuthology.orchestra.run.smithi029.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 3.1 MB/s | 762 kB 00:00 2024-04-26T01:29:29.254 INFO:teuthology.orchestra.run.smithi029.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 311 kB/s | 18 kB 00:00 2024-04-26T01:29:29.283 INFO:teuthology.orchestra.run.smithi006.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 2.4 MB/s | 500 kB 00:00 2024-04-26T01:29:29.446 INFO:teuthology.orchestra.run.smithi029.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 419 kB/s | 80 kB 00:00 2024-04-26T01:29:29.597 INFO:teuthology.orchestra.run.smithi029.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 5.2 MB/s | 803 kB 00:00 2024-04-26T01:29:29.706 INFO:teuthology.orchestra.run.smithi029.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-04-26T01:29:29.823 INFO:teuthology.orchestra.run.smithi029.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-04-26T01:29:29.940 INFO:teuthology.orchestra.run.smithi029.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 10 MB/s | 1.2 MB 00:00 2024-04-26T01:29:30.016 INFO:teuthology.orchestra.run.smithi029.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-26T01:29:30.063 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:29:30.064 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:29:30.075 INFO:teuthology.orchestra.run.smithi188.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T01:29:30.076 INFO:teuthology.orchestra.run.smithi188.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T01:29:30.077 INFO:teuthology.orchestra.run.smithi188.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T01:29:30.077 INFO:teuthology.orchestra.run.smithi188.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T01:29:30.079 INFO:teuthology.orchestra.run.smithi188.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T01:29:30.081 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T01:29:30.082 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T01:29:30.133 INFO:teuthology.orchestra.run.smithi029.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-04-26T01:29:30.142 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-26T01:29:30.157 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-04-26T01:29:30.158 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-26T01:29:30.158 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-04-26T01:29:30.158 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-26T01:29:30.158 INFO:teuthology.orchestra.run.smithi188.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T01:29:30.158 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-26T01:29:30.158 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:29:30.158 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-26T01:29:30.158 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-26T01:29:30.158 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-26T01:29:30.158 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-26T01:29:30.158 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-26T01:29:30.158 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-26T01:29:30.159 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:29:30.160 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-26T01:29:30.161 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-26T01:29:30.162 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T01:29:30.163 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-26T01:29:30.164 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T01:29:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T01:29:30.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T01:29:30.167 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T01:29:30.168 INFO:teuthology.orchestra.run.smithi188.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout:Installing weak dependencies: 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout:Install 148 Packages 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 4 Packages 2024-04-26T01:29:30.169 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:29:30.170 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 222 M 2024-04-26T01:29:30.170 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-26T01:29:30.275 INFO:teuthology.orchestra.run.smithi029.stdout:(45/152): ceph-mgr-diskprediction-local-19.0.0- 2.2 MB/s | 7.4 MB 00:03 2024-04-26T01:29:30.367 INFO:teuthology.orchestra.run.smithi029.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 533 kB/s | 125 kB 00:00 2024-04-26T01:29:30.394 INFO:teuthology.orchestra.run.smithi029.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-04-26T01:29:30.595 INFO:teuthology.orchestra.run.smithi029.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 4.6 MB/s | 943 kB 00:00 2024-04-26T01:29:30.654 INFO:teuthology.orchestra.run.smithi029.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 689 kB 00:00 2024-04-26T01:29:30.679 INFO:teuthology.orchestra.run.smithi029.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.3 MB/s | 107 kB 00:00 2024-04-26T01:29:30.705 INFO:teuthology.orchestra.run.smithi029.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 425 kB/s | 21 kB 00:00 2024-04-26T01:29:30.739 INFO:teuthology.orchestra.run.smithi029.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-26T01:29:30.764 INFO:teuthology.orchestra.run.smithi029.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 560 kB/s | 33 kB 00:00 2024-04-26T01:29:30.807 INFO:teuthology.orchestra.run.smithi029.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 451 kB/s | 18 kB 00:00 2024-04-26T01:29:30.882 INFO:teuthology.orchestra.run.smithi029.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-04-26T01:29:30.908 INFO:teuthology.orchestra.run.smithi029.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 113 kB/s | 11 kB 00:00 2024-04-26T01:29:30.934 INFO:teuthology.orchestra.run.smithi029.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-26T01:29:30.959 INFO:teuthology.orchestra.run.smithi029.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 896 kB/s | 45 kB 00:00 2024-04-26T01:29:31.001 INFO:teuthology.orchestra.run.smithi029.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-04-26T01:29:31.035 INFO:teuthology.orchestra.run.smithi029.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 582 kB 00:00 2024-04-26T01:29:31.061 INFO:teuthology.orchestra.run.smithi029.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 438 kB/s | 25 kB 00:00 2024-04-26T01:29:31.088 INFO:teuthology.orchestra.run.smithi029.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 907 kB/s | 47 kB 00:00 2024-04-26T01:29:31.122 INFO:teuthology.orchestra.run.smithi029.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 292 kB 00:00 2024-04-26T01:29:31.155 INFO:teuthology.orchestra.run.smithi029.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-04-26T01:29:31.157 INFO:teuthology.orchestra.run.smithi188.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 19 kB/s | 6.6 kB 00:00 2024-04-26T01:29:31.181 INFO:teuthology.orchestra.run.smithi029.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 601 kB/s | 35 kB 00:00 2024-04-26T01:29:31.266 INFO:teuthology.orchestra.run.smithi006.stdout:(18/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 2.7 MB/s | 5.4 MB 00:01 2024-04-26T01:29:31.342 INFO:teuthology.orchestra.run.smithi006.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 610 kB/s | 45 kB 00:00 2024-04-26T01:29:31.381 INFO:teuthology.orchestra.run.smithi029.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-04-26T01:29:31.392 INFO:teuthology.orchestra.run.smithi006.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 2.7 MB/s | 135 kB 00:00 2024-04-26T01:29:31.474 INFO:teuthology.orchestra.run.smithi029.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-04-26T01:29:31.524 INFO:teuthology.orchestra.run.smithi029.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-26T01:29:31.525 INFO:teuthology.orchestra.run.smithi006.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.2 MB/s | 161 kB 00:00 2024-04-26T01:29:31.566 INFO:teuthology.orchestra.run.smithi029.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 9.9 MB/s | 911 kB 00:00 2024-04-26T01:29:31.600 INFO:teuthology.orchestra.run.smithi029.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.8 MB/s | 133 kB 00:00 2024-04-26T01:29:31.718 INFO:teuthology.orchestra.run.smithi029.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 8.7 MB/s | 1.0 MB 00:00 2024-04-26T01:29:31.726 INFO:teuthology.orchestra.run.smithi006.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 1.6 MB/s | 320 kB 00:00 2024-04-26T01:29:31.774 INFO:teuthology.orchestra.run.smithi188.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.4 MB/s | 890 kB 00:00 2024-04-26T01:29:31.835 INFO:teuthology.orchestra.run.smithi029.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-04-26T01:29:31.851 INFO:teuthology.orchestra.run.smithi006.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 2.3 MB/s | 298 kB 00:00 2024-04-26T01:29:31.860 INFO:teuthology.orchestra.run.smithi029.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 627 kB/s | 87 kB 00:00 2024-04-26T01:29:31.883 INFO:teuthology.orchestra.run.smithi188.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 1.3 MB/s | 145 kB 00:00 2024-04-26T01:29:31.935 INFO:teuthology.orchestra.run.smithi006.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.2 MB/s | 99 kB 00:00 2024-04-26T01:29:32.010 INFO:teuthology.orchestra.run.smithi006.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.1 MB/s | 85 kB 00:00 2024-04-26T01:29:32.089 INFO:teuthology.orchestra.run.smithi188.stdout:(4/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 4.2 MB/s | 5.4 MB 00:01 2024-04-26T01:29:32.350 INFO:teuthology.orchestra.run.smithi188.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 5.7 MB/s | 1.5 MB 00:00 2024-04-26T01:29:32.661 INFO:teuthology.orchestra.run.smithi029.stdout:(74/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 3.7 MB/s | 49 MB 00:13 2024-04-26T01:29:32.693 INFO:teuthology.orchestra.run.smithi006.stdout:(26/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 1.7 MB/s | 17 MB 00:09 2024-04-26T01:29:32.695 INFO:teuthology.orchestra.run.smithi029.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 293 kB/s | 244 kB 00:00 2024-04-26T01:29:32.800 INFO:teuthology.orchestra.run.smithi188.stdout:(6/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 2.6 MB/s | 2.4 MB 00:00 2024-04-26T01:29:32.820 INFO:teuthology.orchestra.run.smithi029.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 6.1 MB/s | 6.0 MB 00:00 2024-04-26T01:29:32.856 INFO:teuthology.orchestra.run.smithi029.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.3 MB/s | 248 kB 00:00 2024-04-26T01:29:32.881 INFO:teuthology.orchestra.run.smithi029.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 140 kB/s | 26 kB 00:00 2024-04-26T01:29:32.902 INFO:teuthology.orchestra.run.smithi006.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 822 kB/s | 170 kB 00:00 2024-04-26T01:29:32.907 INFO:teuthology.orchestra.run.smithi029.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 225 kB/s | 19 kB 00:00 2024-04-26T01:29:32.932 INFO:teuthology.orchestra.run.smithi029.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 688 kB/s | 35 kB 00:00 2024-04-26T01:29:32.959 INFO:teuthology.orchestra.run.smithi188.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 7.7 MB/s | 4.7 MB 00:00 2024-04-26T01:29:32.966 INFO:teuthology.orchestra.run.smithi029.stdout:(81/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-26T01:29:32.977 INFO:teuthology.orchestra.run.smithi006.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 463 kB/s | 34 kB 00:00 2024-04-26T01:29:33.000 INFO:teuthology.orchestra.run.smithi029.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-04-26T01:29:33.034 INFO:teuthology.orchestra.run.smithi029.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-04-26T01:29:33.143 INFO:teuthology.orchestra.run.smithi029.stdout:(84/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 5.1 MB 00:00 2024-04-26T01:29:33.168 INFO:teuthology.orchestra.run.smithi029.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 471 kB/s | 77 kB 00:00 2024-04-26T01:29:33.193 INFO:teuthology.orchestra.run.smithi029.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 270 kB/s | 42 kB 00:00 2024-04-26T01:29:33.219 INFO:teuthology.orchestra.run.smithi029.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-26T01:29:33.253 INFO:teuthology.orchestra.run.smithi029.stdout:(88/152): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 269 kB 00:00 2024-04-26T01:29:33.278 INFO:teuthology.orchestra.run.smithi029.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.3 MB/s | 279 kB 00:00 2024-04-26T01:29:33.303 INFO:teuthology.orchestra.run.smithi029.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 614 kB/s | 51 kB 00:00 2024-04-26T01:29:33.329 INFO:teuthology.orchestra.run.smithi029.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-26T01:29:33.344 INFO:teuthology.orchestra.run.smithi006.stdout:(29/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 2.3 MB/s | 3.1 MB 00:01 2024-04-26T01:29:33.354 INFO:teuthology.orchestra.run.smithi029.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 820 kB/s | 42 kB 00:00 2024-04-26T01:29:33.389 INFO:teuthology.orchestra.run.smithi029.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 774 kB/s | 44 kB 00:00 2024-04-26T01:29:33.424 INFO:teuthology.orchestra.run.smithi029.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.3 MB/s | 221 kB 00:00 2024-04-26T01:29:33.466 INFO:teuthology.orchestra.run.smithi029.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 485 kB/s | 36 kB 00:00 2024-04-26T01:29:33.508 INFO:teuthology.orchestra.run.smithi029.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 586 kB/s | 47 kB 00:00 2024-04-26T01:29:33.576 INFO:teuthology.orchestra.run.smithi029.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.8 MB/s | 305 kB 00:00 2024-04-26T01:29:33.839 INFO:teuthology.orchestra.run.smithi006.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 166 kB/s | 142 kB 00:00 2024-04-26T01:29:33.851 INFO:teuthology.orchestra.run.smithi029.stdout:(98/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-04-26T01:29:33.869 INFO:teuthology.orchestra.run.smithi188.stdout:(8/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 12 MB/s | 11 MB 00:00 2024-04-26T01:29:33.885 INFO:teuthology.orchestra.run.smithi029.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 155 kB/s | 58 kB 00:00 2024-04-26T01:29:33.935 INFO:teuthology.orchestra.run.smithi188.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 376 kB/s | 25 kB 00:00 2024-04-26T01:29:34.294 INFO:teuthology.orchestra.run.smithi029.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 131 kB/s | 94 kB 00:00 2024-04-26T01:29:34.328 INFO:teuthology.orchestra.run.smithi029.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 102 kB/s | 45 kB 00:00 2024-04-26T01:29:34.353 INFO:teuthology.orchestra.run.smithi029.stdout:(102/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 137 kB/s | 64 kB 00:00 2024-04-26T01:29:34.379 INFO:teuthology.orchestra.run.smithi029.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 264 kB/s | 22 kB 00:00 2024-04-26T01:29:34.421 INFO:teuthology.orchestra.run.smithi029.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 9.1 MB/s | 863 kB 00:00 2024-04-26T01:29:34.588 INFO:teuthology.orchestra.run.smithi029.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 116 kB/s | 19 kB 00:00 2024-04-26T01:29:34.739 INFO:teuthology.orchestra.run.smithi029.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 864 kB/s | 309 kB 00:00 2024-04-26T01:29:34.789 INFO:teuthology.orchestra.run.smithi029.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-04-26T01:29:34.823 INFO:teuthology.orchestra.run.smithi029.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 310 kB/s | 26 kB 00:00 2024-04-26T01:29:34.865 INFO:teuthology.orchestra.run.smithi029.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 642 kB/s | 49 kB 00:00 2024-04-26T01:29:34.901 INFO:teuthology.orchestra.run.smithi029.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 872 kB/s | 67 kB 00:00 2024-04-26T01:29:34.943 INFO:teuthology.orchestra.run.smithi029.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.9 MB/s | 151 kB 00:00 2024-04-26T01:29:35.085 INFO:teuthology.orchestra.run.smithi029.stdout:(112/152): libarrow-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 4.4 MB 00:00 2024-04-26T01:29:35.119 INFO:teuthology.orchestra.run.smithi029.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 3.0 MB/s | 548 kB 00:00 2024-04-26T01:29:35.153 INFO:teuthology.orchestra.run.smithi029.stdout:(114/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-04-26T01:29:35.178 INFO:teuthology.orchestra.run.smithi029.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 468 kB/s | 43 kB 00:00 2024-04-26T01:29:35.204 INFO:teuthology.orchestra.run.smithi029.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 381 kB/s | 32 kB 00:00 2024-04-26T01:29:35.229 INFO:teuthology.orchestra.run.smithi029.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 196 kB/s | 15 kB 00:00 2024-04-26T01:29:35.255 INFO:teuthology.orchestra.run.smithi029.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-26T01:29:35.278 INFO:teuthology.orchestra.run.smithi006.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 1.8 MB/s | 3.5 MB 00:01 2024-04-26T01:29:35.288 INFO:teuthology.orchestra.run.smithi029.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.1 MB/s | 358 kB 00:00 2024-04-26T01:29:35.314 INFO:teuthology.orchestra.run.smithi029.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-04-26T01:29:35.340 INFO:teuthology.orchestra.run.smithi029.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 417 kB/s | 35 kB 00:00 2024-04-26T01:29:35.365 INFO:teuthology.orchestra.run.smithi029.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.6 MB/s | 206 kB 00:00 2024-04-26T01:29:35.416 INFO:teuthology.orchestra.run.smithi029.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-26T01:29:35.475 INFO:teuthology.orchestra.run.smithi029.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-26T01:29:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.2 MB/s | 249 kB 00:00 2024-04-26T01:29:35.502 INFO:teuthology.orchestra.run.smithi029.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 206 kB/s | 18 kB 00:00 2024-04-26T01:29:35.544 INFO:teuthology.orchestra.run.smithi029.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 336 kB/s | 23 kB 00:00 2024-04-26T01:29:35.569 INFO:teuthology.orchestra.run.smithi029.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 288 kB/s | 19 kB 00:00 2024-04-26T01:29:35.579 INFO:teuthology.orchestra.run.smithi006.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 543 kB/s | 49 kB 00:00 2024-04-26T01:29:35.628 INFO:teuthology.orchestra.run.smithi029.stdout:(128/152): python3-grpcio-1.46.7-10.el9.x86_64. 6.5 MB/s | 2.0 MB 00:00 2024-04-26T01:29:35.654 INFO:teuthology.orchestra.run.smithi029.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 179 kB/s | 20 kB 00:00 2024-04-26T01:29:35.662 INFO:teuthology.orchestra.run.smithi006.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 199 kB/s | 16 kB 00:00 2024-04-26T01:29:35.704 INFO:teuthology.orchestra.run.smithi029.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-04-26T01:29:35.730 INFO:teuthology.orchestra.run.smithi029.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 458 kB/s | 46 kB 00:00 2024-04-26T01:29:35.755 INFO:teuthology.orchestra.run.smithi029.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 779 kB/s | 79 kB 00:00 2024-04-26T01:29:35.789 INFO:teuthology.orchestra.run.smithi029.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 682 kB/s | 58 kB 00:00 2024-04-26T01:29:35.815 INFO:teuthology.orchestra.run.smithi029.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.1 MB/s | 272 kB 00:00 2024-04-26T01:29:35.840 INFO:teuthology.orchestra.run.smithi029.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 194 kB/s | 16 kB 00:00 2024-04-26T01:29:35.846 INFO:teuthology.orchestra.run.smithi006.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 1.5 MB/s | 273 kB 00:00 2024-04-26T01:29:35.866 INFO:teuthology.orchestra.run.smithi029.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-26T01:29:35.891 INFO:teuthology.orchestra.run.smithi029.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 370 kB/s | 28 kB 00:00 2024-04-26T01:29:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 403 kB/s | 31 kB 00:00 2024-04-26T01:29:35.942 INFO:teuthology.orchestra.run.smithi029.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-26T01:29:35.968 INFO:teuthology.orchestra.run.smithi029.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 776 kB/s | 59 kB 00:00 2024-04-26T01:29:35.995 INFO:teuthology.orchestra.run.smithi029.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 459 kB/s | 36 kB 00:00 2024-04-26T01:29:36.020 INFO:teuthology.orchestra.run.smithi029.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 656 kB/s | 51 kB 00:00 2024-04-26T01:29:36.054 INFO:teuthology.orchestra.run.smithi029.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.6 MB/s | 230 kB 00:00 2024-04-26T01:29:36.080 INFO:teuthology.orchestra.run.smithi029.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-04-26T01:29:36.114 INFO:teuthology.orchestra.run.smithi029.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-04-26T01:29:36.139 INFO:teuthology.orchestra.run.smithi029.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 236 kB/s | 20 kB 00:00 2024-04-26T01:29:36.173 INFO:teuthology.orchestra.run.smithi029.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-26T01:29:36.246 INFO:teuthology.orchestra.run.smithi006.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 1.9 MB/s | 762 kB 00:00 2024-04-26T01:29:36.305 INFO:teuthology.orchestra.run.smithi006.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 318 kB/s | 18 kB 00:00 2024-04-26T01:29:36.351 INFO:teuthology.orchestra.run.smithi188.stdout:(10/152): ceph-common-19.0.0-3156.gd69f13d8.el7 3.8 MB/s | 21 MB 00:05 2024-04-26T01:29:36.423 INFO:teuthology.orchestra.run.smithi029.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 5.1 MB/s | 1.6 MB 00:00 2024-04-26T01:29:36.434 INFO:teuthology.orchestra.run.smithi188.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.6 MB/s | 225 kB 00:00 2024-04-26T01:29:36.518 INFO:teuthology.orchestra.run.smithi188.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 325 kB/s | 27 kB 00:00 2024-04-26T01:29:36.564 INFO:teuthology.orchestra.run.smithi006.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 312 kB/s | 80 kB 00:00 2024-04-26T01:29:36.591 INFO:teuthology.orchestra.run.smithi029.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 8.1 MB/s | 1.3 MB 00:00 2024-04-26T01:29:36.601 INFO:teuthology.orchestra.run.smithi188.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 380 kB/s | 32 kB 00:00 2024-04-26T01:29:36.827 INFO:teuthology.orchestra.run.smithi188.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 3.2 MB/s | 747 kB 00:00 2024-04-26T01:29:36.833 INFO:teuthology.orchestra.run.smithi029.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 9.9 MB/s | 2.4 MB 00:00 2024-04-26T01:29:36.906 INFO:teuthology.orchestra.run.smithi006.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-04-26T01:29:36.910 INFO:teuthology.orchestra.run.smithi188.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.9 MB/s | 163 kB 00:00 2024-04-26T01:29:37.048 INFO:teuthology.orchestra.run.smithi006.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-04-26T01:29:37.092 INFO:teuthology.orchestra.run.smithi029.stdout:(151/152): librados2-19.0.0-3156.gd69f13d8.el7. 3.6 MB/s | 3.4 MB 00:00 2024-04-26T01:29:37.102 INFO:teuthology.orchestra.run.smithi188.stdout:(16/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 3.9 MB/s | 17 MB 00:04 2024-04-26T01:29:37.128 INFO:teuthology.orchestra.run.smithi188.stdout:(17/152): librados-devel-19.0.0-3156.gd69f13d8. 583 kB/s | 126 kB 00:00 2024-04-26T01:29:37.156 INFO:teuthology.orchestra.run.smithi006.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-04-26T01:29:37.403 INFO:teuthology.orchestra.run.smithi188.stdout:(18/152): libradosstriper1-19.0.0-3156.gd69f13d 1.6 MB/s | 500 kB 00:00 2024-04-26T01:29:37.423 INFO:teuthology.orchestra.run.smithi006.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 4.6 MB/s | 1.2 MB 00:00 2024-04-26T01:29:37.433 INFO:teuthology.orchestra.run.smithi029.stdout:(152/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 2.5 MB/s | 3.1 MB 00:01 2024-04-26T01:29:37.439 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-04-26T01:29:37.439 INFO:teuthology.orchestra.run.smithi029.stdout:Total 8.9 MB/s | 222 MB 00:24 2024-04-26T01:29:37.490 INFO:teuthology.orchestra.run.smithi006.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-26T01:29:37.511 INFO:teuthology.orchestra.run.smithi188.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 417 kB/s | 45 kB 00:00 2024-04-26T01:29:37.566 INFO:teuthology.orchestra.run.smithi006.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-04-26T01:29:37.641 INFO:teuthology.orchestra.run.smithi006.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-26T01:29:37.728 INFO:teuthology.orchestra.run.smithi188.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 622 kB/s | 135 kB 00:00 2024-04-26T01:29:37.775 INFO:teuthology.orchestra.run.smithi006.stdout:(46/152): ceph-mgr-diskprediction-local-19.0.0- 1.9 MB/s | 7.4 MB 00:03 2024-04-26T01:29:37.808 INFO:teuthology.orchestra.run.smithi006.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.3 MB/s | 215 kB 00:00 2024-04-26T01:29:37.933 INFO:teuthology.orchestra.run.smithi006.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 689 kB 00:00 2024-04-26T01:29:37.970 INFO:teuthology.orchestra.run.smithi188.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 668 kB/s | 161 kB 00:00 2024-04-26T01:29:38.084 INFO:teuthology.orchestra.run.smithi006.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 943 kB 00:00 2024-04-26T01:29:38.109 INFO:teuthology.orchestra.run.smithi006.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 622 kB/s | 107 kB 00:00 2024-04-26T01:29:38.154 INFO:teuthology.orchestra.run.smithi188.stdout:(22/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 5.2 MB/s | 5.4 MB 00:01 2024-04-26T01:29:38.178 INFO:teuthology.orchestra.run.smithi006.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 239 kB/s | 21 kB 00:00 2024-04-26T01:29:38.210 INFO:teuthology.orchestra.run.smithi006.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 3.1 MB/s | 313 kB 00:00 2024-04-26T01:29:38.235 INFO:teuthology.orchestra.run.smithi006.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 592 kB/s | 33 kB 00:00 2024-04-26T01:29:38.237 INFO:teuthology.orchestra.run.smithi188.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 3.5 MB/s | 298 kB 00:00 2024-04-26T01:29:38.263 INFO:teuthology.orchestra.run.smithi188.stdout:(24/152): python3-rados-19.0.0-3156.gd69f13d8.e 1.1 MB/s | 320 kB 00:00 2024-04-26T01:29:38.313 INFO:teuthology.orchestra.run.smithi188.stdout:(25/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.3 MB/s | 99 kB 00:00 2024-04-26T01:29:38.355 INFO:teuthology.orchestra.run.smithi188.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 926 kB/s | 85 kB 00:00 2024-04-26T01:29:38.427 INFO:teuthology.orchestra.run.smithi006.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 96 kB/s | 18 kB 00:00 2024-04-26T01:29:38.511 INFO:teuthology.orchestra.run.smithi006.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 10 MB/s | 3.0 MB 00:00 2024-04-26T01:29:38.527 INFO:teuthology.orchestra.run.smithi006.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 119 kB/s | 11 kB 00:00 2024-04-26T01:29:38.560 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-04-26T01:29:38.561 INFO:teuthology.orchestra.run.smithi006.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 160 kB 00:00 2024-04-26T01:29:38.578 INFO:teuthology.orchestra.run.smithi006.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 940 kB/s | 45 kB 00:00 2024-04-26T01:29:38.637 INFO:teuthology.orchestra.run.smithi006.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 8.8 MB/s | 662 kB 00:00 2024-04-26T01:29:38.678 INFO:teuthology.orchestra.run.smithi006.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 5.8 MB/s | 582 kB 00:00 2024-04-26T01:29:38.704 INFO:teuthology.orchestra.run.smithi006.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 418 kB/s | 25 kB 00:00 2024-04-26T01:29:38.721 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-04-26T01:29:38.722 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-04-26T01:29:38.729 INFO:teuthology.orchestra.run.smithi006.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 969 kB/s | 47 kB 00:00 2024-04-26T01:29:38.763 INFO:teuthology.orchestra.run.smithi006.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.0 MB/s | 292 kB 00:00 2024-04-26T01:29:38.796 INFO:teuthology.orchestra.run.smithi006.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 188 kB 00:00 2024-04-26T01:29:38.822 INFO:teuthology.orchestra.run.smithi006.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 632 kB/s | 35 kB 00:00 2024-04-26T01:29:38.847 INFO:teuthology.orchestra.run.smithi188.stdout:(27/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 10 MB/s | 49 MB 00:04 2024-04-26T01:29:38.872 INFO:teuthology.orchestra.run.smithi188.stdout:(28/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 330 kB/s | 170 kB 00:00 2024-04-26T01:29:38.898 INFO:teuthology.orchestra.run.smithi188.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3156.g 685 kB/s | 34 kB 00:00 2024-04-26T01:29:38.957 INFO:teuthology.orchestra.run.smithi188.stdout:(30/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 4.8 MB/s | 3.1 MB 00:00 2024-04-26T01:29:38.982 INFO:teuthology.orchestra.run.smithi188.stdout:(31/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.3 MB/s | 142 kB 00:00 2024-04-26T01:29:38.997 INFO:teuthology.orchestra.run.smithi006.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-04-26T01:29:39.131 INFO:teuthology.orchestra.run.smithi006.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-26T01:29:39.157 INFO:teuthology.orchestra.run.smithi188.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.4 MB/s | 249 kB 00:00 2024-04-26T01:29:39.181 INFO:teuthology.orchestra.run.smithi006.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-04-26T01:29:39.241 INFO:teuthology.orchestra.run.smithi188.stdout:(33/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 10 MB/s | 3.5 MB 00:00 2024-04-26T01:29:39.253 INFO:teuthology.orchestra.run.smithi006.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 7.4 MB/s | 911 kB 00:00 2024-04-26T01:29:39.266 INFO:teuthology.orchestra.run.smithi188.stdout:(34/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 454 kB/s | 49 kB 00:00 2024-04-26T01:29:39.281 INFO:teuthology.orchestra.run.smithi006.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-04-26T01:29:39.291 INFO:teuthology.orchestra.run.smithi188.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3156.gd 328 kB/s | 16 kB 00:00 2024-04-26T01:29:39.375 INFO:teuthology.orchestra.run.smithi188.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 8.9 MB/s | 762 kB 00:00 2024-04-26T01:29:39.408 INFO:teuthology.orchestra.run.smithi188.stdout:(37/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 1.9 MB/s | 273 kB 00:00 2024-04-26T01:29:39.434 INFO:teuthology.orchestra.run.smithi188.stdout:(38/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 312 kB/s | 18 kB 00:00 2024-04-26T01:29:39.449 INFO:teuthology.orchestra.run.smithi006.stdout:(71/152): postgresql-server-13.14-1.el9.x86_64. 30 MB/s | 5.8 MB 00:00 2024-04-26T01:29:39.499 INFO:teuthology.orchestra.run.smithi006.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.1 MB/s | 1.0 MB 00:00 2024-04-26T01:29:39.524 INFO:teuthology.orchestra.run.smithi006.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-26T01:29:39.575 INFO:teuthology.orchestra.run.smithi188.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 482 kB/s | 80 kB 00:00 2024-04-26T01:29:39.634 INFO:teuthology.orchestra.run.smithi188.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-04-26T01:29:39.659 INFO:teuthology.orchestra.run.smithi188.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-26T01:29:39.683 INFO:teuthology.orchestra.run.smithi006.stdout:(74/152): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 6.0 MB 00:00 2024-04-26T01:29:39.684 INFO:teuthology.orchestra.run.smithi188.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-26T01:29:39.716 INFO:teuthology.orchestra.run.smithi006.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-04-26T01:29:39.726 INFO:teuthology.orchestra.run.smithi188.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-26T01:29:39.750 INFO:teuthology.orchestra.run.smithi006.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.8 MB/s | 248 kB 00:00 2024-04-26T01:29:39.752 INFO:teuthology.orchestra.run.smithi188.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-26T01:29:39.775 INFO:teuthology.orchestra.run.smithi006.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 464 kB/s | 26 kB 00:00 2024-04-26T01:29:39.777 INFO:teuthology.orchestra.run.smithi188.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-26T01:29:39.801 INFO:teuthology.orchestra.run.smithi006.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 389 kB/s | 19 kB 00:00 2024-04-26T01:29:39.812 INFO:teuthology.orchestra.run.smithi188.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-26T01:29:39.834 INFO:teuthology.orchestra.run.smithi006.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-26T01:29:39.836 INFO:teuthology.orchestra.run.smithi188.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-26T01:29:39.859 INFO:teuthology.orchestra.run.smithi006.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 628 kB/s | 35 kB 00:00 2024-04-26T01:29:39.870 INFO:teuthology.orchestra.run.smithi188.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 12 MB/s | 689 kB 00:00 2024-04-26T01:29:39.903 INFO:teuthology.orchestra.run.smithi188.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-04-26T01:29:39.929 INFO:teuthology.orchestra.run.smithi188.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-26T01:29:40.012 INFO:teuthology.orchestra.run.smithi188.stdout:(51/152): ceph-mgr-diskprediction-local-19.0.0- 7.0 MB/s | 7.4 MB 00:01 2024-04-26T01:29:40.037 INFO:teuthology.orchestra.run.smithi188.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 161 kB/s | 21 kB 00:00 2024-04-26T01:29:40.043 INFO:teuthology.orchestra.run.smithi006.stdout:(81/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-04-26T01:29:40.062 INFO:teuthology.orchestra.run.smithi188.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-04-26T01:29:40.077 INFO:teuthology.orchestra.run.smithi006.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 250 kB 00:00 2024-04-26T01:29:40.088 INFO:teuthology.orchestra.run.smithi188.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 440 kB/s | 33 kB 00:00 2024-04-26T01:29:40.110 INFO:teuthology.orchestra.run.smithi006.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 222 kB 00:00 2024-04-26T01:29:40.152 INFO:teuthology.orchestra.run.smithi006.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 77 kB 00:00 2024-04-26T01:29:40.186 INFO:teuthology.orchestra.run.smithi006.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 567 kB/s | 42 kB 00:00 2024-04-26T01:29:40.220 INFO:teuthology.orchestra.run.smithi006.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 269 kB 00:00 2024-04-26T01:29:40.221 INFO:teuthology.orchestra.run.smithi188.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 115 kB/s | 18 kB 00:00 2024-04-26T01:29:40.245 INFO:teuthology.orchestra.run.smithi006.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.8 MB/s | 159 kB 00:00 2024-04-26T01:29:40.279 INFO:teuthology.orchestra.run.smithi006.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.8 MB/s | 279 kB 00:00 2024-04-26T01:29:40.295 INFO:teuthology.orchestra.run.smithi006.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 51 kB 00:00 2024-04-26T01:29:40.297 INFO:teuthology.orchestra.run.smithi188.stdout:(56/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-04-26T01:29:40.322 INFO:teuthology.orchestra.run.smithi188.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 48 kB/s | 11 kB 00:00 2024-04-26T01:29:40.339 INFO:teuthology.orchestra.run.smithi006.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 913 kB/s | 54 kB 00:00 2024-04-26T01:29:40.348 INFO:teuthology.orchestra.run.smithi188.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-04-26T01:29:40.390 INFO:teuthology.orchestra.run.smithi188.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 681 kB/s | 45 kB 00:00 2024-04-26T01:29:40.423 INFO:teuthology.orchestra.run.smithi188.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-04-26T01:29:40.507 INFO:teuthology.orchestra.run.smithi188.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 3.6 MB/s | 582 kB 00:00 2024-04-26T01:29:40.532 INFO:teuthology.orchestra.run.smithi188.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 181 kB/s | 25 kB 00:00 2024-04-26T01:29:40.557 INFO:teuthology.orchestra.run.smithi188.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 355 kB/s | 47 kB 00:00 2024-04-26T01:29:40.582 INFO:teuthology.orchestra.run.smithi188.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-26T01:29:40.596 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-04-26T01:29:40.597 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-04-26T01:29:40.607 INFO:teuthology.orchestra.run.smithi188.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-26T01:29:40.633 INFO:teuthology.orchestra.run.smithi188.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 472 kB/s | 35 kB 00:00 2024-04-26T01:29:40.833 INFO:teuthology.orchestra.run.smithi188.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 19 MB/s | 4.7 MB 00:00 2024-04-26T01:29:40.871 INFO:teuthology.orchestra.run.smithi006.stdout:(91/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 3.0 MB/s | 49 MB 00:16 2024-04-26T01:29:41.059 INFO:teuthology.orchestra.run.smithi188.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 3.7 MB/s | 1.6 MB 00:00 2024-04-26T01:29:41.125 INFO:teuthology.orchestra.run.smithi188.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 9.0 MB/s | 4.4 MB 00:00 2024-04-26T01:29:41.150 INFO:teuthology.orchestra.run.smithi188.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 2.8 MB/s | 911 kB 00:00 2024-04-26T01:29:41.180 INFO:teuthology.orchestra.run.smithi006.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 51 kB/s | 42 kB 00:00 2024-04-26T01:29:41.184 INFO:teuthology.orchestra.run.smithi188.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 1.1 MB/s | 133 kB 00:00 2024-04-26T01:29:41.334 INFO:teuthology.orchestra.run.smithi188.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-04-26T01:29:41.397 INFO:teuthology.orchestra.run.smithi006.stdout:(93/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-04-26T01:29:41.409 INFO:teuthology.orchestra.run.smithi188.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-04-26T01:29:41.435 INFO:teuthology.orchestra.run.smithi188.stdout:(74/152): python3-attrs-20.3.0-7.el9.noarch.rpm 351 kB/s | 87 kB 00:00 2024-04-26T01:29:41.438 INFO:teuthology.orchestra.run.smithi006.stdout:(94/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 80 kB/s | 44 kB 00:00 2024-04-26T01:29:41.473 INFO:teuthology.orchestra.run.smithi006.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 765 kB/s | 221 kB 00:00 2024-04-26T01:29:41.497 INFO:teuthology.orchestra.run.smithi006.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 647 kB/s | 36 kB 00:00 2024-04-26T01:29:41.523 INFO:teuthology.orchestra.run.smithi006.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 590 kB/s | 47 kB 00:00 2024-04-26T01:29:41.556 INFO:teuthology.orchestra.run.smithi006.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 305 kB 00:00 2024-04-26T01:29:41.560 INFO:teuthology.orchestra.run.smithi188.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 6.0 MB 00:00 2024-04-26T01:29:41.582 INFO:teuthology.orchestra.run.smithi006.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-04-26T01:29:41.585 INFO:teuthology.orchestra.run.smithi188.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-04-26T01:29:41.607 INFO:teuthology.orchestra.run.smithi006.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.1 MB/s | 94 kB 00:00 2024-04-26T01:29:41.618 INFO:teuthology.orchestra.run.smithi188.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.3 MB/s | 248 kB 00:00 2024-04-26T01:29:41.633 INFO:teuthology.orchestra.run.smithi006.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 866 kB/s | 64 kB 00:00 2024-04-26T01:29:41.635 INFO:teuthology.orchestra.run.smithi188.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 345 kB/s | 26 kB 00:00 2024-04-26T01:29:41.652 INFO:teuthology.orchestra.run.smithi188.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 574 kB/s | 19 kB 00:00 2024-04-26T01:29:41.658 INFO:teuthology.orchestra.run.smithi006.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 928 kB/s | 45 kB 00:00 2024-04-26T01:29:41.677 INFO:teuthology.orchestra.run.smithi188.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-26T01:29:41.683 INFO:teuthology.orchestra.run.smithi006.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 304 kB/s | 22 kB 00:00 2024-04-26T01:29:41.694 INFO:teuthology.orchestra.run.smithi188.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 596 kB/s | 35 kB 00:00 2024-04-26T01:29:41.725 INFO:teuthology.orchestra.run.smithi006.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 9.3 MB/s | 863 kB 00:00 2024-04-26T01:29:41.778 INFO:teuthology.orchestra.run.smithi188.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-04-26T01:29:41.862 INFO:teuthology.orchestra.run.smithi188.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-04-26T01:29:41.890 INFO:teuthology.orchestra.run.smithi188.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 222 kB 00:00 2024-04-26T01:29:41.912 INFO:teuthology.orchestra.run.smithi188.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 589 kB/s | 77 kB 00:00 2024-04-26T01:29:41.938 INFO:teuthology.orchestra.run.smithi188.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 564 kB/s | 42 kB 00:00 2024-04-26T01:29:41.963 INFO:teuthology.orchestra.run.smithi188.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-26T01:29:41.988 INFO:teuthology.orchestra.run.smithi188.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-26T01:29:42.014 INFO:teuthology.orchestra.run.smithi188.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-04-26T01:29:42.039 INFO:teuthology.orchestra.run.smithi188.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-26T01:29:42.042 INFO:teuthology.orchestra.run.smithi006.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 62 kB/s | 19 kB 00:00 2024-04-26T01:29:42.064 INFO:teuthology.orchestra.run.smithi188.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 717 kB/s | 54 kB 00:00 2024-04-26T01:29:42.113 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://opencolo.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 50.117.120.66) 2024-04-26T01:29:42.410 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://opencolo.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 50.117.120.66) 2024-04-26T01:29:42.431 INFO:teuthology.orchestra.run.smithi188.stdout:(92/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-04-26T01:29:42.432 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://forksystems.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 23.147.64.7) 2024-04-26T01:29:42.456 INFO:teuthology.orchestra.run.smithi188.stdout:(93/152): python3-toml-0.10.2-6.el9.noarch.rpm 107 kB/s | 42 kB 00:00 2024-04-26T01:29:42.481 INFO:teuthology.orchestra.run.smithi188.stdout:(94/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 107 kB/s | 44 kB 00:00 2024-04-26T01:29:42.507 INFO:teuthology.orchestra.run.smithi188.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-04-26T01:29:42.532 INFO:teuthology.orchestra.run.smithi188.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 721 kB/s | 36 kB 00:00 2024-04-26T01:29:42.545 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://forksystems.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 23.147.64.7) 2024-04-26T01:29:42.557 INFO:teuthology.orchestra.run.smithi188.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 633 kB/s | 47 kB 00:00 2024-04-26T01:29:42.582 INFO:teuthology.orchestra.run.smithi188.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 305 kB 00:00 2024-04-26T01:29:42.607 INFO:teuthology.orchestra.run.smithi188.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-26T01:29:42.633 INFO:teuthology.orchestra.run.smithi188.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-26T01:29:42.658 INFO:teuthology.orchestra.run.smithi188.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 848 kB/s | 64 kB 00:00 2024-04-26T01:29:42.667 INFO:teuthology.orchestra.run.smithi006.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 562 kB/s | 548 kB 00:00 2024-04-26T01:29:42.683 INFO:teuthology.orchestra.run.smithi188.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 891 kB/s | 45 kB 00:00 2024-04-26T01:29:42.700 INFO:teuthology.orchestra.run.smithi006.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 306 kB/s | 309 kB 00:01 2024-04-26T01:29:42.708 INFO:teuthology.orchestra.run.smithi188.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-04-26T01:29:42.740 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for http://opencolo.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 50.117.120.66) 2024-04-26T01:29:42.742 INFO:teuthology.orchestra.run.smithi188.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 10 MB/s | 863 kB 00:00 2024-04-26T01:29:42.801 INFO:teuthology.orchestra.run.smithi006.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 490 kB/s | 49 kB 00:00 2024-04-26T01:29:42.909 INFO:teuthology.orchestra.run.smithi006.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 626 kB/s | 67 kB 00:00 2024-04-26T01:29:43.043 INFO:teuthology.orchestra.run.smithi006.stdout:(110/152): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2024-04-26T01:29:43.045 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for https://opencolo.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 50.117.120.66) 2024-04-26T01:29:43.050 INFO:teuthology.orchestra.run.smithi188.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 63 kB/s | 19 kB 00:00 2024-04-26T01:29:43.085 INFO:teuthology.orchestra.run.smithi006.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 865 kB/s | 151 kB 00:00 2024-04-26T01:29:43.086 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for http://forksystems.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 23.147.64.7) 2024-04-26T01:29:43.096 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://southfront.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 147.160.170.17) 2024-04-26T01:29:43.098 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for https://forksystems.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 23.147.64.7) 2024-04-26T01:29:43.114 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://opencolo.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 50.117.120.66) 2024-04-26T01:29:43.144 INFO:teuthology.orchestra.run.smithi006.stdout:(112/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 55 kB/s | 26 kB 00:00 2024-04-26T01:29:43.168 INFO:teuthology.orchestra.run.smithi188.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 675 kB/s | 309 kB 00:00 2024-04-26T01:29:43.209 INFO:teuthology.orchestra.run.smithi188.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-26T01:29:43.219 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://southfront.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 147.160.170.17) 2024-04-26T01:29:43.221 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for https://southfront.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 147.160.170.17) 2024-04-26T01:29:43.293 INFO:teuthology.orchestra.run.smithi188.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2024-04-26T01:29:43.293 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for http://southfront.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 147.160.170.17) 2024-04-26T01:29:43.295 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://opencolo.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 50.117.120.66) 2024-04-26T01:29:43.336 INFO:teuthology.orchestra.run.smithi006.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-04-26T01:29:43.337 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://forksystems.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 23.147.64.7) 2024-04-26T01:29:43.355 INFO:teuthology.orchestra.run.smithi006.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://forksystems.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 23.147.64.7) 2024-04-26T01:29:43.410 INFO:teuthology.orchestra.run.smithi188.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 576 kB/s | 67 kB 00:00 2024-04-26T01:29:43.453 INFO:teuthology.orchestra.run.smithi006.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 277 kB/s | 32 kB 00:00 2024-04-26T01:29:43.502 INFO:teuthology.orchestra.run.smithi188.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-04-26T01:29:43.502 INFO:teuthology.orchestra.run.smithi006.stdout:(115/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 838 kB 00:00 2024-04-26T01:29:43.561 INFO:teuthology.orchestra.run.smithi188.stdout:(111/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 66 kB/s | 26 kB 00:00 2024-04-26T01:29:43.620 INFO:teuthology.orchestra.run.smithi006.stdout:(116/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 91 kB/s | 43 kB 00:00 2024-04-26T01:29:43.624 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://southfront.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 147.160.170.17) 2024-04-26T01:29:43.719 INFO:teuthology.orchestra.run.smithi188.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-04-26T01:29:43.928 INFO:teuthology.orchestra.run.smithi188.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 208 kB/s | 43 kB 00:00 2024-04-26T01:29:43.937 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://southfront.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 147.160.170.17) 2024-04-26T01:29:44.003 INFO:teuthology.orchestra.run.smithi188.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 429 kB/s | 32 kB 00:00 2024-04-26T01:29:44.062 INFO:teuthology.orchestra.run.smithi006.stdout:(117/152): python3-cherrypy-18.6.1-2.el9.noarch 813 kB/s | 358 kB 00:00 2024-04-26T01:29:44.128 INFO:teuthology.orchestra.run.smithi188.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 120 kB/s | 15 kB 00:00 2024-04-26T01:29:44.220 INFO:teuthology.orchestra.run.smithi006.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.9 MB/s | 303 kB 00:00 2024-04-26T01:29:44.279 INFO:teuthology.orchestra.run.smithi188.stdout:(116/152): libarrow-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 4.4 MB 00:01 2024-04-26T01:29:44.329 INFO:teuthology.orchestra.run.smithi006.stdout:(119/152): python3-certifi-2021.10.8-2.el9.noar 17 kB/s | 15 kB 00:00 2024-04-26T01:29:44.354 INFO:teuthology.orchestra.run.smithi006.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 265 kB/s | 35 kB 00:00 2024-04-26T01:29:44.413 INFO:teuthology.orchestra.run.smithi006.stdout:(121/152): python3-cheroot-8.6.0-4.el9.noarch.r 189 kB/s | 172 kB 00:00 2024-04-26T01:29:44.471 INFO:teuthology.orchestra.run.smithi006.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.4 MB/s | 206 kB 00:00 2024-04-26T01:29:44.604 INFO:teuthology.orchestra.run.smithi188.stdout:(117/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 761 kB/s | 838 kB 00:01 2024-04-26T01:29:44.662 INFO:teuthology.orchestra.run.smithi188.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 322 kB/s | 172 kB 00:00 2024-04-26T01:29:44.688 INFO:teuthology.orchestra.run.smithi188.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 877 kB/s | 358 kB 00:00 2024-04-26T01:29:44.788 INFO:teuthology.orchestra.run.smithi188.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 281 kB/s | 35 kB 00:00 2024-04-26T01:29:44.814 INFO:teuthology.orchestra.run.smithi188.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.6 MB/s | 206 kB 00:00 2024-04-26T01:29:44.839 INFO:teuthology.orchestra.run.smithi188.stdout:(122/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.3 MB/s | 303 kB 00:00 2024-04-26T01:29:44.863 INFO:teuthology.orchestra.run.smithi006.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 4.0 MB/s | 2.0 MB 00:00 2024-04-26T01:29:44.897 INFO:teuthology.orchestra.run.smithi006.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 25 kB/s | 11 kB 00:00 2024-04-26T01:29:44.897 INFO:teuthology.orchestra.run.smithi188.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-04-26T01:29:44.930 INFO:teuthology.orchestra.run.smithi006.stdout:(125/152): python3-grpcio-tools-1.46.7-10.el9.x 279 kB/s | 144 kB 00:00 2024-04-26T01:29:44.948 INFO:teuthology.orchestra.run.smithi188.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-26T01:29:44.956 INFO:teuthology.orchestra.run.smithi006.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-04-26T01:29:45.007 INFO:teuthology.orchestra.run.smithi006.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 212 kB/s | 23 kB 00:00 2024-04-26T01:29:45.014 INFO:teuthology.orchestra.run.smithi188.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-04-26T01:29:45.032 INFO:teuthology.orchestra.run.smithi188.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 132 kB/s | 18 kB 00:00 2024-04-26T01:29:45.048 INFO:teuthology.orchestra.run.smithi188.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-04-26T01:29:45.099 INFO:teuthology.orchestra.run.smithi188.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-04-26T01:29:45.116 INFO:teuthology.orchestra.run.smithi188.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 294 kB/s | 20 kB 00:00 2024-04-26T01:29:45.249 INFO:teuthology.orchestra.run.smithi188.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-04-26T01:29:45.281 INFO:teuthology.orchestra.run.smithi006.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 60 kB/s | 20 kB 00:00 2024-04-26T01:29:45.307 INFO:teuthology.orchestra.run.smithi006.stdout:(129/152): python3-jaraco-functools-3.5.0-2.el9 52 kB/s | 19 kB 00:00 2024-04-26T01:29:45.383 INFO:teuthology.orchestra.run.smithi188.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 163 kB/s | 46 kB 00:00 2024-04-26T01:29:45.408 INFO:teuthology.orchestra.run.smithi188.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 270 kB/s | 79 kB 00:00 2024-04-26T01:29:45.433 INFO:teuthology.orchestra.run.smithi188.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 314 kB/s | 58 kB 00:00 2024-04-26T01:29:45.465 INFO:teuthology.orchestra.run.smithi006.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 501 kB/s | 79 kB 00:00 2024-04-26T01:29:45.675 INFO:teuthology.orchestra.run.smithi188.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 932 kB/s | 272 kB 00:00 2024-04-26T01:29:45.700 INFO:teuthology.orchestra.run.smithi188.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 56 kB/s | 16 kB 00:00 2024-04-26T01:29:45.715 INFO:teuthology.orchestra.run.smithi006.stdout:(131/152): python3-natsort-7.1.1-5.el9.noarch.r 231 kB/s | 58 kB 00:00 2024-04-26T01:29:45.726 INFO:teuthology.orchestra.run.smithi188.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 308 kB/s | 90 kB 00:00 2024-04-26T01:29:45.759 INFO:teuthology.orchestra.run.smithi188.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-04-26T01:29:45.799 INFO:teuthology.orchestra.run.smithi006.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 1.3 MB/s | 1.0 MB 00:00 2024-04-26T01:29:45.824 INFO:teuthology.orchestra.run.smithi006.stdout:(133/152): python3-logutils-0.3.5-21.el9.noarch 86 kB/s | 46 kB 00:00 2024-04-26T01:29:45.826 INFO:teuthology.orchestra.run.smithi188.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 245 kB/s | 31 kB 00:00 2024-04-26T01:29:45.851 INFO:teuthology.orchestra.run.smithi188.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-04-26T01:29:45.877 INFO:teuthology.orchestra.run.smithi188.stdout:(140/152): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-04-26T01:29:45.966 INFO:teuthology.orchestra.run.smithi006.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-04-26T01:29:45.977 INFO:teuthology.orchestra.run.smithi188.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 407 kB/s | 51 kB 00:00 2024-04-26T01:29:45.991 INFO:teuthology.orchestra.run.smithi006.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 86 kB/s | 16 kB 00:00 2024-04-26T01:29:46.004 INFO:teuthology.orchestra.run.smithi188.stdout:(142/152): python3-tempora-5.0.0-2.el9.noarch.r 203 kB/s | 36 kB 00:00 2024-04-26T01:29:46.027 INFO:teuthology.orchestra.run.smithi188.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-04-26T01:29:46.033 INFO:teuthology.orchestra.run.smithi006.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 432 kB/s | 90 kB 00:00 2024-04-26T01:29:46.053 INFO:teuthology.orchestra.run.smithi188.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-26T01:29:46.184 INFO:teuthology.orchestra.run.smithi006.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 161 kB/s | 31 kB 00:00 2024-04-26T01:29:46.209 INFO:teuthology.orchestra.run.smithi006.stdout:(138/152): python3-pyperclip-1.8.2-2.el9.noarch 117 kB/s | 28 kB 00:00 2024-04-26T01:29:46.243 INFO:teuthology.orchestra.run.smithi006.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 905 kB/s | 188 kB 00:00 2024-04-26T01:29:46.245 INFO:teuthology.orchestra.run.smithi188.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 92 kB/s | 20 kB 00:00 2024-04-26T01:29:46.270 INFO:teuthology.orchestra.run.smithi188.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 1.6 MB/s | 427 kB 00:00 2024-04-26T01:29:46.295 INFO:teuthology.orchestra.run.smithi188.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 791 kB/s | 192 kB 00:00 2024-04-26T01:29:46.310 INFO:teuthology.orchestra.run.smithi006.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 474 kB/s | 59 kB 00:00 2024-04-26T01:29:46.343 INFO:teuthology.orchestra.run.smithi006.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 269 kB/s | 36 kB 00:00 2024-04-26T01:29:46.537 INFO:teuthology.orchestra.run.smithi188.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 5.4 MB/s | 1.6 MB 00:00 2024-04-26T01:29:46.588 INFO:teuthology.orchestra.run.smithi188.stdout:(149/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 11 MB/s | 3.1 MB 00:00 2024-04-26T01:29:46.618 INFO:teuthology.orchestra.run.smithi006.stdout:(142/152): python3-websocket-client-1.2.3-2.el9 327 kB/s | 90 kB 00:00 2024-04-26T01:29:46.629 INFO:teuthology.orchestra.run.smithi188.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 15 MB/s | 1.3 MB 00:00 2024-04-26T01:29:46.644 INFO:teuthology.orchestra.run.smithi006.stdout:(143/152): python3-typing-extensions-4.4.0-2.el 127 kB/s | 51 kB 00:00 2024-04-26T01:29:46.655 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-04-26T01:29:46.713 INFO:teuthology.orchestra.run.smithi188.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 19 MB/s | 2.4 MB 00:00 2024-04-26T01:29:46.769 INFO:teuthology.orchestra.run.smithi006.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 505 kB/s | 230 kB 00:00 2024-04-26T01:29:46.794 INFO:teuthology.orchestra.run.smithi006.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 133 kB/s | 20 kB 00:00 2024-04-26T01:29:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T01:29:46.953 INFO:teuthology.orchestra.run.smithi006.stdout:(146/152): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-26T01:29:47.006 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T01:29:47.069 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T01:29:47.128 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T01:29:47.162 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T01:29:47.207 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T01:29:47.338 INFO:teuthology.orchestra.run.smithi188.stdout:(152/152): librados2-19.0.0-3156.gd69f13d8.el7. 3.3 MB/s | 3.4 MB 00:01 2024-04-26T01:29:47.347 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-26T01:29:47.347 INFO:teuthology.orchestra.run.smithi188.stdout:Total 13 MB/s | 222 MB 00:17 2024-04-26T01:29:47.420 INFO:teuthology.orchestra.run.smithi006.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 295 kB/s | 192 kB 00:00 2024-04-26T01:29:47.495 INFO:teuthology.orchestra.run.smithi006.stdout:(148/152): python3-werkzeug-2.0.3-3.el9.1.noarc 488 kB/s | 427 kB 00:00 2024-04-26T01:29:47.587 INFO:teuthology.orchestra.run.smithi006.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 15 MB/s | 1.3 MB 00:00 2024-04-26T01:29:47.616 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T01:29:47.704 INFO:teuthology.orchestra.run.smithi006.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 21 MB/s | 2.4 MB 00:00 2024-04-26T01:29:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:29:47.908 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:29:47.935 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:29:48.028 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:29:48.066 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:29:48.154 INFO:teuthology.orchestra.run.smithi006.stdout:(151/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 4.3 MB/s | 3.1 MB 00:00 2024-04-26T01:29:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T01:29:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T01:29:48.303 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:29:48.396 INFO:teuthology.orchestra.run.smithi006.stdout:(152/152): librados2-19.0.0-3156.gd69f13d8.el7. 2.4 MB/s | 3.4 MB 00:01 2024-04-26T01:29:48.406 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-04-26T01:29:48.407 INFO:teuthology.orchestra.run.smithi006.stdout:Total 6.9 MB/s | 222 MB 00:32 2024-04-26T01:29:48.409 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:29:48.423 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-26T01:29:48.438 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:29:48.501 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:29:48.525 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:29:48.550 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:29:48.568 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-26T01:29:48.568 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-26T01:29:48.761 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:29:48.802 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:29:48.914 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T01:29:49.020 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-26T01:29:49.063 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T01:29:49.099 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T01:29:49.180 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T01:29:49.224 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T01:29:49.277 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T01:29:49.310 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T01:29:49.348 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T01:29:49.430 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T01:29:49.490 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T01:29:49.528 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T01:29:49.573 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T01:29:49.651 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T01:29:49.692 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-26T01:29:49.736 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-26T01:29:49.783 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-26T01:29:49.857 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T01:29:49.908 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T01:29:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T01:29:50.133 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T01:29:50.189 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T01:29:50.356 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-26T01:29:50.356 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-26T01:29:50.752 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T01:29:50.848 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T01:29:50.883 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T01:29:50.927 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T01:29:50.975 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-26T01:29:51.027 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T01:29:51.225 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T01:29:51.290 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T01:29:51.305 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-04-26T01:29:51.347 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T01:29:51.432 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T01:29:51.469 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T01:29:51.506 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T01:29:51.554 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T01:29:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-04-26T01:29:51.573 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-04-26T01:29:51.606 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T01:29:51.659 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T01:29:51.711 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T01:29:51.859 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T01:29:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T01:29:52.217 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T01:29:52.267 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T01:29:52.369 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T01:29:52.402 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T01:29:52.443 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T01:29:52.477 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T01:29:52.511 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-26T01:29:52.590 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T01:29:52.732 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T01:29:52.775 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T01:29:52.805 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T01:29:52.848 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T01:29:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T01:29:52.920 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T01:29:52.958 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T01:29:52.993 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T01:29:53.027 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T01:29:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T01:29:53.114 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T01:29:53.205 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T01:29:53.255 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T01:29:53.303 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T01:29:53.351 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T01:29:53.462 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T01:29:53.526 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T01:29:54.311 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T01:29:54.338 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-04-26T01:29:54.338 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-04-26T01:29:54.369 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T01:29:54.403 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T01:29:54.518 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T01:29:54.572 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T01:29:54.673 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T01:29:54.717 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T01:29:54.902 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T01:29:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T01:29:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T01:29:55.723 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T01:29:55.852 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T01:29:56.254 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-26T01:29:56.341 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T01:29:56.463 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T01:29:56.575 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T01:29:56.609 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T01:29:56.651 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T01:29:56.695 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T01:29:57.079 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T01:29:57.135 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:29:57.168 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T01:29:57.257 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:29:57.277 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T01:29:57.287 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:29:57.314 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T01:29:57.348 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T01:29:57.377 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:29:57.408 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:29:57.561 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T01:29:57.613 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T01:29:57.641 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:29:57.677 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T01:29:57.730 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:29:57.753 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:29:57.776 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:29:57.830 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:29:57.830 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:29:57.851 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:29:57.869 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-26T01:29:57.871 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:29:57.914 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T01:29:58.059 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:29:58.093 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:29:58.199 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T01:29:58.296 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-26T01:29:58.325 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T01:29:58.360 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T01:29:58.420 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-26T01:29:58.424 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T01:29:58.459 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T01:29:58.460 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:29:58.502 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T01:29:58.532 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:29:58.535 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T01:29:58.563 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T01:29:58.564 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-26T01:29:58.631 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T01:29:58.686 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T01:29:58.724 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T01:29:58.759 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T01:29:58.843 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T01:29:58.879 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-26T01:29:58.922 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-26T01:29:58.951 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-26T01:29:59.020 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T01:29:59.065 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T01:29:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T01:29:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T01:29:59.335 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T01:29:59.860 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T01:29:59.956 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T01:29:59.984 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T01:30:00.027 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T01:30:00.065 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-26T01:30:00.107 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T01:30:00.260 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T01:30:00.344 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T01:30:00.407 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T01:30:00.479 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T01:30:00.511 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T01:30:00.547 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T01:30:00.594 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T01:30:00.630 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T01:30:00.673 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T01:30:00.725 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T01:30:00.776 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-04-26T01:30:00.892 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T01:30:00.938 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T01:30:00.959 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T01:30:00.971 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T01:30:00.985 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:30:01.001 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T01:30:01.026 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:30:01.096 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T01:30:01.097 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T01:30:01.135 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T01:30:01.175 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T01:30:01.201 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T01:30:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-26T01:30:01.235 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T01:30:01.300 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T01:30:01.305 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T01:30:01.339 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T01:30:01.427 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T01:30:01.458 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T01:30:01.467 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T01:30:01.486 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T01:30:01.520 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T01:30:01.559 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T01:30:01.591 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T01:30:01.595 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:30:01.621 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T01:30:01.642 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:30:01.655 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T01:30:01.688 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T01:30:01.726 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:30:01.726 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T01:30:01.726 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T01:30:01.726 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:01.730 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T01:30:01.761 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:30:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T01:30:01.934 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T01:30:01.977 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T01:30:02.028 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T01:30:02.044 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:30:02.060 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T01:30:02.107 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T01:30:02.205 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T01:30:02.246 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:30:02.253 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T01:30:02.289 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:30:02.428 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:30:02.472 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:30:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T01:30:02.901 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T01:30:02.926 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:30:02.972 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T01:30:03.059 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T01:30:03.092 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T01:30:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:30:03.136 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:30:03.192 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T01:30:03.251 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T01:30:03.258 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:30:03.287 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:30:03.338 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T01:30:03.368 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:30:03.377 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T01:30:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T01:30:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T01:30:03.644 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T01:30:03.689 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:30:03.747 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:30:03.950 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T01:30:04.085 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-26T01:30:04.121 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T01:30:04.170 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T01:30:04.264 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T01:30:04.286 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T01:30:04.325 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T01:30:04.389 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T01:30:04.406 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T01:30:04.440 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T01:30:04.478 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T01:30:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T01:30:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T01:30:04.704 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T01:30:04.760 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T01:30:04.883 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T01:30:04.917 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-26T01:30:04.971 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-26T01:30:05.007 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-26T01:30:05.134 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T01:30:05.194 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T01:30:05.307 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T01:30:05.524 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T01:30:05.598 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T01:30:05.627 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T01:30:05.721 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T01:30:05.749 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T01:30:05.782 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T01:30:06.086 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T01:30:06.129 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:30:06.202 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:30:06.226 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-26T01:30:06.262 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T01:30:06.335 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T01:30:06.441 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T01:30:06.480 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T01:30:06.525 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T01:30:06.565 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-26T01:30:06.635 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T01:30:06.739 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-26T01:30:06.766 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:30:06.837 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:30:06.859 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-26T01:30:06.914 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T01:30:06.975 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T01:30:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T01:30:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T01:30:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T01:30:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T01:30:07.335 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T01:30:07.389 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T01:30:07.453 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T01:30:07.519 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T01:30:07.743 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T01:30:07.803 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T01:30:07.841 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T01:30:07.902 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T01:30:08.045 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T01:30:08.080 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T01:30:08.131 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T01:30:08.163 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T01:30:08.188 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-26T01:30:08.381 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T01:30:08.574 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T01:30:08.640 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T01:30:08.691 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T01:30:08.747 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T01:30:08.821 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T01:30:08.875 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T01:30:08.916 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T01:30:08.970 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T01:30:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T01:30:09.105 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T01:30:09.137 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:30:09.159 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T01:30:09.181 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:30:09.311 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T01:30:09.385 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T01:30:09.425 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T01:30:09.497 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T01:30:09.657 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T01:30:09.717 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:30:09.721 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T01:30:09.767 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:30:09.859 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:30:09.859 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T01:30:09.860 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T01:30:09.860 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:09.893 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:30:10.749 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T01:30:10.833 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T01:30:10.870 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T01:30:11.006 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T01:30:11.075 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T01:30:11.188 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T01:30:11.230 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T01:30:11.469 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T01:30:11.510 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T01:30:11.621 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T01:30:12.506 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T01:30:12.654 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T01:30:14.339 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T01:30:14.433 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T01:30:14.473 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T01:30:14.518 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T01:30:14.911 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T01:30:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:30:15.078 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:30:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-26T01:30:15.189 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T01:30:15.793 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-26T01:30:15.823 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:30:15.975 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:30:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-26T01:30:17.017 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:30:17.018 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-04-26T01:30:17.018 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-04-26T01:30:17.018 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-04-26T01:30:17.018 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-04-26T01:30:17.018 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-04-26T01:30:17.018 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2024-04-26T01:30:17.018 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-04-26T01:30:17.018 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:17.198 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:30:17.214 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:30:17.214 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:17.214 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T01:30:17.214 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:18.747 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:30:18.774 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:30:18.774 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:18.774 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T01:30:18.774 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:30:18.774 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:30:18.774 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:30:18.980 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:30:19.015 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:30:19.039 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:30:19.040 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:19.040 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T01:30:19.040 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:30:19.040 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:30:19.040 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:19.508 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:30:19.532 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:30:19.532 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:19.532 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T01:30:19.532 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:30:19.532 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:30:19.532 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:19.641 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T01:30:19.667 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:30:19.699 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T01:30:19.717 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:30:19.765 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T01:30:19.899 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:30:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T01:30:19.899 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T01:30:19.899 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:30:19.943 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:30:19.971 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T01:30:20.033 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T01:30:20.113 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T01:30:20.215 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:30:20.359 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:30:20.404 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T01:30:20.437 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T01:30:20.473 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T01:30:20.940 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:30:20.976 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:30:21.018 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-26T01:30:21.091 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-26T01:30:21.127 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:30:21.590 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:30:21.591 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2024-04-26T01:30:21.591 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2024-04-26T01:30:21.591 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2024-04-26T01:30:21.591 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2024-04-26T01:30:21.591 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2024-04-26T01:30:21.591 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /root 2024-04-26T01:30:21.591 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2024-04-26T01:30:21.591 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:21.764 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:30:21.779 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:30:21.779 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:21.779 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T01:30:21.779 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:22.227 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:30:22.272 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:30:23.260 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:30:23.285 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:30:23.285 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:23.285 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T01:30:23.285 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:30:23.285 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:30:23.285 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:23.288 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:30:23.315 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:30:23.424 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:30:23.502 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:30:23.525 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:30:23.525 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:23.525 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T01:30:23.525 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:30:23.525 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:30:23.525 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:23.595 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-26T01:30:23.629 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:30:23.653 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:30:23.653 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:23.653 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T01:30:23.654 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:30:23.654 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:30:23.654 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:23.705 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:30:23.716 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:30:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-26T01:30:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-26T01:30:23.961 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:30:23.983 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:30:23.984 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:23.984 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T01:30:23.984 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:30:23.984 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:30:23.984 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:24.091 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T01:30:24.148 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T01:30:24.204 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T01:30:24.389 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T01:30:24.437 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T01:30:24.514 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T01:30:24.604 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:30:24.735 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:30:24.779 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T01:30:24.811 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T01:30:24.838 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T01:30:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T01:30:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:30:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:30:24.908 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:24.908 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T01:30:24.908 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:30:24.908 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:30:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:24.953 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:30:24.976 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:30:24.976 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:24.976 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T01:30:24.976 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:25.030 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:30:25.054 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:30:25.054 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:25.054 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T01:30:25.054 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:30:25.054 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:30:25.055 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:25.157 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:30:25.187 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:30:25.219 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-26T01:30:25.281 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-26T01:30:25.319 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:30:25.374 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:30:25.397 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:30:25.397 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:25.397 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T01:30:25.397 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:30:25.397 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:30:25.397 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:26.355 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:30:26.387 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:30:27.348 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:30:27.371 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:30:27.473 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:30:27.634 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-26T01:30:27.656 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:30:27.679 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:30:27.679 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:27.680 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T01:30:27.680 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:30:27.680 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:30:27.680 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:27.723 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:30:27.734 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:30:27.833 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-26T01:30:27.881 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-26T01:30:28.817 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T01:30:28.851 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:30:28.874 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:30:28.874 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:28.874 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T01:30:28.874 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:30:28.874 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:30:28.875 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:28.919 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:30:28.940 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:30:28.941 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:28.941 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T01:30:28.941 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:28.986 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:30:29.008 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:30:29.009 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:29.009 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T01:30:29.009 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:30:29.009 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:30:29.009 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:29.311 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:30:29.334 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:30:29.334 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:29.334 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T01:30:29.334 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:30:29.334 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:30:29.334 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:30.275 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-26T01:30:30.325 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-26T01:30:30.367 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-26T01:30:30.399 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T01:30:30.426 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T01:30:30.634 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-26T01:30:30.698 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T01:30:30.760 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-26T01:30:31.414 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:30:31.465 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:30:31.487 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:30:32.410 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-26T01:30:32.519 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T01:30:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T01:30:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T01:30:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T01:30:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:30:32.814 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:30:32.855 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:30:32.855 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:30:32.954 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:30:33.043 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:30:33.478 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:30:33.478 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2024-04-26T01:30:33.479 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2024-04-26T01:30:33.479 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2024-04-26T01:30:33.479 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2024-04-26T01:30:33.479 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2024-04-26T01:30:33.479 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2024-04-26T01:30:33.479 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2024-04-26T01:30:33.479 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:30:33.732 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:30:33.780 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:30:33.780 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:33.780 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T01:30:33.780 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:30:33.907 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-26T01:30:33.948 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-26T01:30:33.988 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-26T01:30:34.021 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T01:30:34.048 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T01:30:34.238 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-26T01:30:34.294 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T01:30:34.324 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-26T01:30:34.906 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:30:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:30:34.966 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:30:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:30:35.558 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:30:35.558 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:35.558 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T01:30:35.558 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:30:35.559 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:30:35.559 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:30:35.804 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-26T01:30:35.819 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:30:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:30:35.889 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:35.889 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T01:30:35.889 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:30:35.890 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:30:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:30:35.894 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T01:30:35.935 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T01:30:36.018 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T01:30:36.049 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T01:30:36.049 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:30:36.133 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:30:36.174 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:30:36.174 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:30:36.257 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:30:36.322 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:30:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:30:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:30:36.476 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:36.476 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T01:30:36.476 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:30:36.476 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:30:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:30:36.613 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T01:30:36.677 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T01:30:36.752 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T01:30:36.997 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T01:30:37.050 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T01:30:37.140 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T01:30:37.265 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:30:37.446 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:30:37.511 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T01:30:37.557 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T01:30:37.589 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T01:30:38.267 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:30:38.333 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:30:38.398 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-26T01:30:38.518 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-26T01:30:38.561 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:30:39.912 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:30:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:30:41.131 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:30:41.163 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:30:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:30:41.504 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-26T01:30:41.532 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:30:41.607 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:30:41.608 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:41.608 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T01:30:41.608 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:30:41.608 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:30:41.608 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:30:41.666 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:30:41.705 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:30:41.821 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-26T01:30:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-26T01:30:42.122 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:30:42.123 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-26T01:30:42.124 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-26T01:30:42.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-26T01:30:42.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-26T01:30:42.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-26T01:30:42.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-26T01:30:42.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-26T01:30:42.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-26T01:30:42.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-26T01:30:42.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-26T01:30:42.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-26T01:30:42.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-26T01:30:42.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-26T01:30:42.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T01:30:42.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T01:30:42.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T01:30:42.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T01:30:42.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-26T01:30:42.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T01:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T01:30:42.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-26T01:30:42.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T01:30:42.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T01:30:42.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:30:42.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T01:30:42.935 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T01:30:42.969 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:30:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:30:43.048 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:43.048 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T01:30:43.048 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:30:43.048 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:30:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:30:43.094 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:30:43.166 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:30:43.166 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:43.166 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T01:30:43.166 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:30:43.236 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:30:43.328 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:30:43.328 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:43.328 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T01:30:43.328 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:30:43.328 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:30:43.328 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:30:43.700 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:30:43.700 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-26T01:30:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-26T01:30:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-26T01:30:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-26T01:30:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-26T01:30:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-26T01:30:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-26T01:30:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:30:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:30:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:30:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-26T01:30:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-26T01:30:43.701 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:30:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-26T01:30:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-26T01:30:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-26T01:30:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-26T01:30:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-26T01:30:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-26T01:30:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-26T01:30:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-26T01:30:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-26T01:30:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-26T01:30:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-26T01:30:43.702 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-26T01:30:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-26T01:30:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-26T01:30:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-26T01:30:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-26T01:30:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-26T01:30:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-26T01:30:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-26T01:30:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-26T01:30:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-26T01:30:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-26T01:30:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-26T01:30:43.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T01:30:43.704 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T01:30:43.705 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T01:30:43.705 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T01:30:43.705 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T01:30:43.705 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T01:30:43.705 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T01:30:43.705 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T01:30:43.705 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T01:30:43.705 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T01:30:43.705 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T01:30:43.705 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T01:30:43.705 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T01:30:43.705 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T01:30:43.706 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T01:30:43.706 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T01:30:43.706 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T01:30:43.706 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T01:30:43.706 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T01:30:43.706 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T01:30:43.706 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T01:30:43.706 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T01:30:43.706 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T01:30:43.706 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T01:30:43.706 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T01:30:43.706 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T01:30:43.707 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T01:30:43.708 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T01:30:43.708 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T01:30:43.708 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T01:30:43.708 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T01:30:43.708 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T01:30:43.708 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T01:30:43.708 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T01:30:43.708 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T01:30:43.708 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T01:30:43.708 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T01:30:43.708 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T01:30:43.708 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T01:30:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T01:30:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T01:30:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T01:30:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T01:30:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T01:30:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T01:30:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T01:30:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T01:30:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-26T01:30:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-26T01:30:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T01:30:43.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T01:30:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T01:30:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-26T01:30:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T01:30:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T01:30:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T01:30:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T01:30:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:30:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:30:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T01:30:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T01:30:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T01:30:43.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T01:30:43.711 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T01:30:43.711 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T01:30:43.711 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T01:30:43.711 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T01:30:43.711 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T01:30:43.711 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T01:30:43.711 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T01:30:43.711 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T01:30:43.711 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T01:30:43.711 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T01:30:43.711 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T01:30:43.711 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T01:30:43.712 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T01:30:43.713 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-26T01:30:43.713 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T01:30:43.713 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-26T01:30:43.713 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T01:30:43.713 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T01:30:43.713 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:30:43.713 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T01:30:43.729 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:30:43.803 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:30:43.803 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:30:43.803 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T01:30:43.803 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:30:43.804 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:30:43.804 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:30:44.233 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:30:44.233 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.235 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.236 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.236 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:44.236 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:44.236 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.236 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.236 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:44.236 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T01:30:44.236 INFO:teuthology.orchestra.run.smithi188.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T01:30:44.236 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T01:30:44.236 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T01:30:44.237 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T01:30:44.237 INFO:teuthology.orchestra.run.smithi188.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T01:30:44.237 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T01:30:44.237 INFO:teuthology.orchestra.run.smithi188.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T01:30:44.237 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T01:30:44.237 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-26T01:30:44.237 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-26T01:30:44.237 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.237 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T01:30:44.238 INFO:teuthology.orchestra.run.smithi188.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T01:30:44.239 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T01:30:44.240 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T01:30:44.240 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T01:30:44.240 INFO:teuthology.orchestra.run.smithi188.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T01:30:44.240 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T01:30:44.240 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T01:30:44.240 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T01:30:44.240 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.240 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.240 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.240 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T01:30:44.241 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T01:30:44.242 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.243 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T01:30:44.244 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T01:30:44.244 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T01:30:44.244 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.244 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T01:30:44.244 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T01:30:44.244 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T01:30:44.244 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T01:30:44.244 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T01:30:44.244 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T01:30:44.244 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T01:30:44.244 INFO:teuthology.orchestra.run.smithi188.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T01:30:44.244 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T01:30:44.245 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T01:30:44.246 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T01:30:44.246 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:30:44.246 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-26T01:30:44.583 DEBUG:teuthology.parallel:result is None 2024-04-26T01:30:45.977 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:30:45.977 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.978 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.979 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T01:30:45.980 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T01:30:45.981 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T01:30:45.982 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T01:30:45.983 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T01:30:45.984 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T01:30:45.985 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T01:30:45.986 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T01:30:45.987 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T01:30:45.988 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.988 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.988 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:45.988 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T01:30:45.988 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T01:30:45.988 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T01:30:45.988 INFO:teuthology.orchestra.run.smithi029.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T01:30:45.988 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T01:30:45.988 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T01:30:45.988 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T01:30:45.989 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-26T01:30:45.989 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-04-26T01:30:46.497 DEBUG:teuthology.parallel:result is None 2024-04-26T01:30:49.102 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-26T01:30:49.155 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-26T01:30:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-26T01:30:49.316 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T01:30:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T01:30:49.729 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-26T01:30:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T01:30:49.818 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-26T01:30:50.762 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:30:50.807 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:30:50.869 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:30:52.306 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-26T01:30:52.415 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T01:30:52.518 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T01:30:52.711 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T01:30:52.754 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T01:30:52.754 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:30:52.878 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:30:52.920 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:30:52.920 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:30:53.069 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:30:53.202 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:30:53.509 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:30:53.509 INFO:teuthology.orchestra.run.smithi006.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-26T01:30:53.509 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:30:59.358 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-26T01:30:59.358 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-26T01:30:59.358 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-26T01:30:59.358 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-26T01:30:59.358 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-26T01:30:59.361 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-26T01:30:59.361 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-26T01:30:59.361 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:30:59.361 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:30:59.361 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-26T01:30:59.362 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-26T01:30:59.363 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-26T01:30:59.375 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-26T01:30:59.376 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-26T01:30:59.376 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-26T01:30:59.376 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-26T01:30:59.376 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-26T01:30:59.376 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-26T01:30:59.376 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-26T01:30:59.376 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-26T01:30:59.376 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-26T01:30:59.376 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-26T01:30:59.376 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-26T01:30:59.376 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T01:30:59.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T01:30:59.378 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T01:30:59.378 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T01:30:59.378 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T01:30:59.378 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T01:30:59.378 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T01:30:59.378 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T01:30:59.378 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T01:30:59.378 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T01:30:59.378 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T01:30:59.378 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T01:30:59.378 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T01:30:59.378 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T01:30:59.379 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T01:30:59.380 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T01:30:59.380 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T01:30:59.380 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T01:30:59.380 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T01:30:59.380 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T01:30:59.380 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T01:30:59.380 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T01:30:59.380 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T01:30:59.380 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T01:30:59.380 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T01:30:59.380 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T01:30:59.381 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T01:30:59.382 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T01:30:59.382 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T01:30:59.382 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T01:30:59.382 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T01:30:59.382 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T01:30:59.382 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T01:30:59.382 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T01:30:59.382 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T01:30:59.382 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-26T01:30:59.382 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-26T01:30:59.382 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T01:30:59.382 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T01:30:59.383 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T01:30:59.384 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T01:30:59.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T01:30:59.386 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:30:59.386 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T01:31:01.678 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:31:01.678 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:31:01.678 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-04-26T01:31:01.678 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:31:01.679 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.680 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.681 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:31:01.681 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:31:01.681 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.681 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.681 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:31:01.681 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T01:31:01.681 INFO:teuthology.orchestra.run.smithi006.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T01:31:01.681 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T01:31:01.681 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T01:31:01.682 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T01:31:01.683 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T01:31:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T01:31:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T01:31:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T01:31:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T01:31:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:31:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T01:31:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T01:31:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T01:31:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T01:31:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T01:31:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T01:31:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T01:31:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:31:01.691 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-04-26T01:31:02.331 DEBUG:teuthology.parallel:result is None 2024-04-26T01:31:02.331 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:31:02.565 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T01:31:02.621 INFO:teuthology.orchestra.run.smithi006.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-26T01:31:02.622 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-26T01:31:02.622 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-26T01:31:02.624 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:31:02.768 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T01:31:02.796 INFO:teuthology.orchestra.run.smithi029.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-26T01:31:02.796 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-26T01:31:02.796 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-26T01:31:02.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:31:02.945 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T01:31:02.971 INFO:teuthology.orchestra.run.smithi188.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-26T01:31:02.971 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-26T01:31:02.971 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-26T01:31:02.973 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T01:31:02.973 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:31:02.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:31:03.051 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-26T01:31:03.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:31:03.083 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-26T01:31:03.083 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:31:03.114 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T01:31:03.114 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:31:03.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:31:03.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:31:03.293 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-26T01:31:03.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:31:03.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:31:03.393 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-26T01:31:03.393 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:31:03.422 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:31:03.487 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T01:31:03.487 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:31:03.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:31:03.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:31:03.676 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-26T01:31:03.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:31:03.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:31:03.775 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-26T01:31:03.775 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:31:03.799 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:31:03.863 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T01:31:03.863 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:31:03.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:31:03.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:31:04.044 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-26T01:31:04.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:31:04.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:31:04.143 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-26T01:31:04.143 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:31:04.167 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:31:04.230 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T01:31:04.336 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-26T01:31:04.336 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:31:04.337 INFO:tasks.cephadm:Cluster fsid is a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:31:04.337 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T01:31:04.337 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.6', 'mon.b': '172.21.15.29', 'mon.c': '172.21.15.188'} 2024-04-26T01:31:04.337 INFO:tasks.cephadm:First mon is mon.a on smithi006 2024-04-26T01:31:04.337 INFO:tasks.cephadm:First mgr is x 2024-04-26T01:31:04.337 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T01:31:04.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo hostname $(hostname -s) 2024-04-26T01:31:04.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-04-26T01:31:04.438 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname $(hostname -s) 2024-04-26T01:31:04.461 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T01:31:04.461 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:31:04.677 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78795/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-gd69f13d8', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.gd69f13d8'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 12:00:06.590653', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-26T01:31:04.821 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 d69f13d827952bb07bb9b7973bca14f97d764d6f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:31:04.822 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm 2024-04-26T01:31:04.823 DEBUG:teuthology.orchestra.run.smithi006:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T01:31:05.343 INFO:teuthology.orchestra.run.smithi006.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 26 01:31 /home/ubuntu/cephtest/cephadm 2024-04-26T01:31:05.343 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T01:31:06.065 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 26 01:31 /home/ubuntu/cephtest/cephadm 2024-04-26T01:31:06.065 DEBUG:teuthology.orchestra.run.smithi188:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T01:31:06.750 INFO:teuthology.orchestra.run.smithi188.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 26 01:31 /home/ubuntu/cephtest/cephadm 2024-04-26T01:31:06.751 DEBUG:teuthology.orchestra.run.smithi006:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T01:31:06.798 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T01:31:06.818 DEBUG:teuthology.orchestra.run.smithi188:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T01:31:06.853 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f on all hosts... 2024-04-26T01:31:06.854 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-26T01:31:06.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-26T01:31:06.861 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-26T01:31:08.397 INFO:teuthology.orchestra.run.smithi188.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:31:09.146 INFO:teuthology.orchestra.run.smithi006.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:31:09.677 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:31:37.272 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-26T01:31:37.272 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-26T01:31:37.272 INFO:teuthology.orchestra.run.smithi188.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-26T01:31:37.273 INFO:teuthology.orchestra.run.smithi188.stdout: "repo_digests": [ 2024-04-26T01:31:37.273 INFO:teuthology.orchestra.run.smithi188.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-26T01:31:37.273 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-26T01:31:37.273 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-26T01:32:05.864 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-26T01:32:05.865 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-26T01:32:05.865 INFO:teuthology.orchestra.run.smithi029.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-26T01:32:05.865 INFO:teuthology.orchestra.run.smithi029.stdout: "repo_digests": [ 2024-04-26T01:32:05.865 INFO:teuthology.orchestra.run.smithi029.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-26T01:32:05.865 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-04-26T01:32:05.865 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-26T01:32:15.920 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:32:15.921 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-26T01:32:15.921 INFO:teuthology.orchestra.run.smithi006.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-26T01:32:15.921 INFO:teuthology.orchestra.run.smithi006.stdout: "repo_digests": [ 2024-04-26T01:32:15.921 INFO:teuthology.orchestra.run.smithi006.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-26T01:32:15.921 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-26T01:32:15.921 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:32:15.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph 2024-04-26T01:32:16.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-04-26T01:32:16.115 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph 2024-04-26T01:32:16.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /etc/ceph 2024-04-26T01:32:16.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-04-26T01:32:16.272 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /etc/ceph 2024-04-26T01:32:16.300 INFO:tasks.cephadm:Writing seed config... 2024-04-26T01:32:16.301 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-26T01:32:16.301 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T01:32:16.301 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T01:32:16.301 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-26T01:32:16.302 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-26T01:32:16.303 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-26T01:32:16.303 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-26T01:32:16.303 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-26T01:32:16.303 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T01:32:16.303 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T01:32:16.303 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T01:32:16.303 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T01:32:16.303 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T01:32:16.303 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T01:32:16.303 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-26T01:32:16.303 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T01:32:16.304 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-26T01:32:16.305 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T01:32:16.305 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-26T01:32:16.306 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:32:16.306 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T01:32:16.366 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a5b72bee-036c-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-26T01:32:16.367 DEBUG:teuthology.orchestra.run.smithi006:mon.a> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T01:32:16.415 DEBUG:teuthology.orchestra.run.smithi006:mgr.x> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T01:32:16.461 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T01:32:16.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f -v bootstrap --fsid a5b72bee-036c-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.6 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T01:32:16.732 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-04-26T01:32:16.732 INFO:teuthology.orchestra.run.smithi006.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f', '-v', 'bootstrap', '--fsid', 'a5b72bee-036c-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.6', '--skip-admin-label'] 2024-04-26T01:32:16.733 INFO:teuthology.orchestra.run.smithi006.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T01:32:16.733 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying podman|docker is present... 2024-04-26T01:32:16.794 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T01:32:16.795 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying lvm2 is present... 2024-04-26T01:32:16.795 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying time synchronization is in place... 2024-04-26T01:32:16.813 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T01:32:16.813 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T01:32:16.834 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T01:32:16.834 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-04-26T01:32:16.852 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout enabled 2024-04-26T01:32:16.871 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout active 2024-04-26T01:32:16.872 INFO:teuthology.orchestra.run.smithi006.stdout:Unit chronyd.service is enabled and running 2024-04-26T01:32:16.872 INFO:teuthology.orchestra.run.smithi006.stdout:Repeating the final host check... 2024-04-26T01:32:16.930 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T01:32:16.930 INFO:teuthology.orchestra.run.smithi006.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T01:32:16.930 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl is present 2024-04-26T01:32:16.931 INFO:teuthology.orchestra.run.smithi006.stdout:lvcreate is present 2024-04-26T01:32:16.944 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T01:32:16.944 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T01:32:16.962 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T01:32:16.962 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-04-26T01:32:16.987 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout enabled 2024-04-26T01:32:17.008 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout active 2024-04-26T01:32:17.009 INFO:teuthology.orchestra.run.smithi006.stdout:Unit chronyd.service is enabled and running 2024-04-26T01:32:17.009 INFO:teuthology.orchestra.run.smithi006.stdout:Host looks OK 2024-04-26T01:32:17.009 INFO:teuthology.orchestra.run.smithi006.stdout:Cluster fsid: a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:17.009 INFO:teuthology.orchestra.run.smithi006.stdout:Acquiring lock 140301182025680 on /run/cephadm/a5b72bee-036c-11ef-bc93-c7b262605968.lock 2024-04-26T01:32:17.010 INFO:teuthology.orchestra.run.smithi006.stdout:Lock 140301182025680 acquired on /run/cephadm/a5b72bee-036c-11ef-bc93-c7b262605968.lock 2024-04-26T01:32:17.011 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying IP 172.21.15.6 port 3300 ... 2024-04-26T01:32:17.012 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying IP 172.21.15.6 port 6789 ... 2024-04-26T01:32:17.012 INFO:teuthology.orchestra.run.smithi006.stdout:Base mon IP(s) is [172.21.15.6:3300, 172.21.15.6:6789], mon addrv is [v2:172.21.15.6:3300,v1:172.21.15.6:6789] 2024-04-26T01:32:17.027 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.6 metric 100 2024-04-26T01:32:17.027 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.6 metric 100 2024-04-26T01:32:17.040 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T01:32:17.040 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-26T01:32:17.040 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-26T01:32:17.051 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T01:32:17.051 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T01:32:17.051 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T01:32:17.051 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-26T01:32:17.052 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-04-26T01:32:17.052 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T01:32:17.054 INFO:teuthology.orchestra.run.smithi006.stdout:Mon IP `172.21.15.6` is in CIDR network `172.21.0.0/20` 2024-04-26T01:32:17.054 INFO:teuthology.orchestra.run.smithi006.stdout:Mon IP `172.21.15.6` is in CIDR network `172.21.0.0/20` 2024-04-26T01:32:17.055 INFO:teuthology.orchestra.run.smithi006.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T01:32:17.057 INFO:teuthology.orchestra.run.smithi006.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T01:32:17.060 INFO:teuthology.orchestra.run.smithi006.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:32:17.817 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-26T01:32:17.817 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:32:17.817 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T01:32:17.817 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Copying blob sha256:7f03586bcf9c6a2b4d690c118c4aa5b771f9c6f252d75a20fb283dd07093a738 2024-04-26T01:32:17.818 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-26T01:32:17.818 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Copying config sha256:da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-26T01:32:17.818 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T01:32:19.134 INFO:teuthology.orchestra.run.smithi006.stdout:ceph: stdout ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-26T01:32:19.135 INFO:teuthology.orchestra.run.smithi006.stdout:Ceph version: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-26T01:32:19.135 INFO:teuthology.orchestra.run.smithi006.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T01:32:20.434 INFO:teuthology.orchestra.run.smithi006.stdout:stat: stdout 167 167 2024-04-26T01:32:20.434 INFO:teuthology.orchestra.run.smithi006.stdout:Creating initial keys... 2024-04-26T01:32:21.770 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph-authtool: stdout AQAkBCtm2sBXOBAAIa/RwzOBKyxkIFXCHnCzDw== 2024-04-26T01:32:23.197 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph-authtool: stdout AQAmBCtmTl8FERAAOpR1/movhNffwZn13Yzc/w== 2024-04-26T01:32:24.312 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph-authtool: stdout AQAnBCtmeJLDKRAAEEdx3idMn5J6bXsfhk8R/A== 2024-04-26T01:32:24.314 INFO:teuthology.orchestra.run.smithi006.stdout:Creating initial monmap... 2024-04-26T01:32:25.449 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T01:32:25.449 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T01:32:25.450 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:25.450 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T01:32:25.450 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool for a [v2:172.21.15.6:3300,v1:172.21.15.6:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T01:32:25.450 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-04-26T01:32:25.450 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: set fsid to a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:25.450 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T01:32:25.450 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:32:25.451 INFO:teuthology.orchestra.run.smithi006.stdout:Creating mon... 2024-04-26T01:32:26.605 INFO:teuthology.orchestra.run.smithi006.stdout:create mon.a on 2024-04-26T01:32:26.981 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T01:32:27.230 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T01:32:27.537 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a5b72bee-036c-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-a5b72bee-036c-11ef-bc93-c7b262605968.target. 2024-04-26T01:32:27.537 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a5b72bee-036c-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-a5b72bee-036c-11ef-bc93-c7b262605968.target. 2024-04-26T01:32:27.968 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.a 2024-04-26T01:32:27.968 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T01:32:28.271 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a5b72bee-036c-11ef-bc93-c7b262605968.target.wants/ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-a5b72bee-036c-11ef-bc93-c7b262605968@.service. 2024-04-26T01:32:28.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:28 smithi006 podman[28036]: 2024-04-26 01:32:28.700099991 +0000 UTC m=+0.136334500 container create 7934c93ceb1e05ace13dfc740f0434033cbc9c7ffc6f6ed95259d195ceba9ccd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T01:32:28.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:28 smithi006 podman[28036]: 2024-04-26 01:32:28.981129667 +0000 UTC m=+0.417364183 container init 7934c93ceb1e05ace13dfc740f0434033cbc9c7ffc6f6ed95259d195ceba9ccd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T01:32:29.111 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T01:32:29.111 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-04-26T01:32:29.127 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T01:32:29.127 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-04-26T01:32:29.127 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-04-26T01:32:29.127 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T01:32:29.127 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mon to start... 2024-04-26T01:32:29.127 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mon... 2024-04-26T01:32:29.276 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:28 smithi006 podman[28036]: 2024-04-26 01:32:28.993126268 +0000 UTC m=+0.429360783 container start 7934c93ceb1e05ace13dfc740f0434033cbc9c7ffc6f6ed95259d195ceba9ccd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-26T01:32:29.276 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:29 smithi006 ceph-mon[28055]: mkfs a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:29.276 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:29 smithi006 ceph-mon[28055]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T01:32:29.276 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:29 smithi006 bash[28036]: 7934c93ceb1e05ace13dfc740f0434033cbc9c7ffc6f6ed95259d195ceba9ccd 2024-04-26T01:32:29.276 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:29 smithi006 systemd[1]: Started Ceph mon.a for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:32:30.165 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:30 smithi006 ceph-mon[28055]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T01:32:30.165 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:30 smithi006 ceph-mon[28055]: monmap epoch 1 2024-04-26T01:32:30.165 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:30 smithi006 ceph-mon[28055]: fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:30.165 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:30 smithi006 ceph-mon[28055]: last_changed 2024-04-26T01:32:24.866051+0000 2024-04-26T01:32:30.165 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:30 smithi006 ceph-mon[28055]: created 2024-04-26T01:32:24.866051+0000 2024-04-26T01:32:30.165 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:30 smithi006 ceph-mon[28055]: min_mon_release 19 (squid) 2024-04-26T01:32:30.165 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:30 smithi006 ceph-mon[28055]: election_strategy: 1 2024-04-26T01:32:30.165 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:30 smithi006 ceph-mon[28055]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-26T01:32:30.165 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:30 smithi006 ceph-mon[28055]: fsmap 2024-04-26T01:32:30.165 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:30 smithi006 ceph-mon[28055]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T01:32:30.166 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:30 smithi006 ceph-mon[28055]: mgrmap e1: no daemons active 2024-04-26T01:32:30.166 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:30 smithi006 ceph-mon[28055]: from='client.? 172.21.15.6:0/4189986799' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T01:32:30.577 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T01:32:30.578 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout id: a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:30.578 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T01:32:30.578 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:32:30.578 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout services: 2024-04-26T01:32:30.578 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.771274s) 2024-04-26T01:32:30.579 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T01:32:30.579 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T01:32:30.579 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:32:30.579 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout data: 2024-04-26T01:32:30.580 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T01:32:30.580 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T01:32:30.580 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T01:32:30.580 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T01:32:30.580 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:32:30.580 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.745+0000 7f2a53800700 1 Processor -- start 2024-04-26T01:32:30.582 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.745+0000 7f2a53800700 1 -- start start 2024-04-26T01:32:30.582 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.746+0000 7f2a53800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 0x7f2a4c10b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:30.582 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.746+0000 7f2a53800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c063d60 con 0x7f2a4c108fb0 2024-04-26T01:32:30.582 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.747+0000 7f2a50e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 0x7f2a4c10b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:30.582 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.747+0000 7f2a50e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 0x7f2a4c10b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52244/0 (socket says 172.21.15.6:52244) 2024-04-26T01:32:30.582 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.747+0000 7f2a50e00700 1 -- 172.21.15.6:0/960145570 learned_addr learned my addr 172.21.15.6:0/960145570 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:30.582 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.747+0000 7f2a50e00700 1 -- 172.21.15.6:0/960145570 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4c10b8e0 con 0x7f2a4c108fb0 2024-04-26T01:32:30.582 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.748+0000 7f2a50e00700 1 --2- 172.21.15.6:0/960145570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 0x7f2a4c10b3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a400098a0 tx=0x7f2a40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a4c1f952e323cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:30.582 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.748+0000 7f2a4b400700 1 -- 172.21.15.6:0/960145570 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a40004160 con 0x7f2a4c108fb0 2024-04-26T01:32:30.582 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.748+0000 7f2a4b400700 1 -- 172.21.15.6:0/960145570 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a40004300 con 0x7f2a4c108fb0 2024-04-26T01:32:30.582 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.748+0000 7f2a4b400700 1 -- 172.21.15.6:0/960145570 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a400036a0 con 0x7f2a4c108fb0 2024-04-26T01:32:30.582 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.749+0000 7f2a53800700 1 -- 172.21.15.6:0/960145570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 msgr2=0x7f2a4c10b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:30.583 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.749+0000 7f2a53800700 1 --2- 172.21.15.6:0/960145570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 0x7f2a4c10b3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a400098a0 tx=0x7f2a40009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:30.583 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.749+0000 7f2a53800700 1 -- 172.21.15.6:0/960145570 shutdown_connections 2024-04-26T01:32:30.583 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.749+0000 7f2a53800700 1 --2- 172.21.15.6:0/960145570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 0x7f2a4c10b3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:30.585 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.749+0000 7f2a53800700 1 -- 172.21.15.6:0/960145570 >> 172.21.15.6:0/960145570 conn(0x7f2a4c102a40 msgr2=0x7f2a4c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:30.585 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.749+0000 7f2a53800700 1 -- 172.21.15.6:0/960145570 shutdown_connections 2024-04-26T01:32:30.586 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.749+0000 7f2a53800700 1 -- 172.21.15.6:0/960145570 wait complete. 2024-04-26T01:32:30.586 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.751+0000 7f2a53800700 1 Processor -- start 2024-04-26T01:32:30.586 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.751+0000 7f2a53800700 1 -- start start 2024-04-26T01:32:30.586 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.751+0000 7f2a53800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 0x7f2a4c19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:30.586 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.751+0000 7f2a53800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c10bdb0 con 0x7f2a4c108fb0 2024-04-26T01:32:30.586 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.751+0000 7f2a50e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 0x7f2a4c19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:30.586 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.751+0000 7f2a50e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 0x7f2a4c19de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52246/0 (socket says 172.21.15.6:52246) 2024-04-26T01:32:30.586 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.751+0000 7f2a50e00700 1 -- 172.21.15.6:0/4189986799 learned_addr learned my addr 172.21.15.6:0/4189986799 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:30.586 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.752+0000 7f2a50e00700 1 -- 172.21.15.6:0/4189986799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4c19e390 con 0x7f2a4c108fb0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.752+0000 7f2a50e00700 1 --2- 172.21.15.6:0/4189986799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 0x7f2a4c19de50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2a40000c00 tx=0x7f2a40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.752+0000 7f2a49600700 1 -- 172.21.15.6:0/4189986799 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a40003db0 con 0x7f2a4c108fb0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.752+0000 7f2a49600700 1 -- 172.21.15.6:0/4189986799 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a40011630 con 0x7f2a4c108fb0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.752+0000 7f2a53800700 1 -- 172.21.15.6:0/4189986799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a4c19e620 con 0x7f2a4c108fb0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.752+0000 7f2a49600700 1 -- 172.21.15.6:0/4189986799 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a40011950 con 0x7f2a4c108fb0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.753+0000 7f2a53800700 1 -- 172.21.15.6:0/4189986799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a4c1a12c0 con 0x7f2a4c108fb0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.753+0000 7f2a49600700 1 -- 172.21.15.6:0/4189986799 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2a40011af0 con 0x7f2a4c108fb0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.753+0000 7f2a49600700 1 -- 172.21.15.6:0/4189986799 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2a4001c450 con 0x7f2a4c108fb0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.754+0000 7f2a53800700 1 -- 172.21.15.6:0/4189986799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a4c0640a0 con 0x7f2a4c108fb0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.757+0000 7f2a49600700 1 -- 172.21.15.6:0/4189986799 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2a40032020 con 0x7f2a4c108fb0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.839+0000 7f2a53800700 1 -- 172.21.15.6:0/4189986799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f2a4c02e240 con 0x7f2a4c108fb0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.840+0000 7f2a49600700 1 -- 172.21.15.6:0/4189986799 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f2a40004260 con 0x7f2a4c108fb0 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.841+0000 7f2a53800700 1 -- 172.21.15.6:0/4189986799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 msgr2=0x7f2a4c19de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.841+0000 7f2a53800700 1 --2- 172.21.15.6:0/4189986799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 0x7f2a4c19de50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2a40000c00 tx=0x7f2a40004750 comp rx=0 tx=0).stop 2024-04-26T01:32:30.588 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.842+0000 7f2a53800700 1 -- 172.21.15.6:0/4189986799 shutdown_connections 2024-04-26T01:32:30.588 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.842+0000 7f2a53800700 1 --2- 172.21.15.6:0/4189986799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a4c108fb0 0x7f2a4c19de50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:30.589 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.842+0000 7f2a53800700 1 -- 172.21.15.6:0/4189986799 >> 172.21.15.6:0/4189986799 conn(0x7f2a4c102a40 msgr2=0x7f2a4c1038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:30.589 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.842+0000 7f2a53800700 1 -- 172.21.15.6:0/4189986799 shutdown_connections 2024-04-26T01:32:30.589 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.842+0000 7f2a53800700 1 -- 172.21.15.6:0/4189986799 wait complete. 2024-04-26T01:32:30.589 INFO:teuthology.orchestra.run.smithi006.stdout:mon is available 2024-04-26T01:32:30.589 INFO:teuthology.orchestra.run.smithi006.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T01:32:31.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:31 smithi006 ceph-mon[28055]: from='client.? 172.21.15.6:0/141749351' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T01:32:31.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:31 smithi006 ceph-mon[28055]: from='client.? 172.21.15.6:0/141749351' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T01:32:31.956 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:32:31.956 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T01:32:31.956 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout fsid = a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:31.956 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T01:32:31.957 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.6:3300,v1:172.21.15.6:6789] 2024-04-26T01:32:31.957 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T01:32:31.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.246+0000 7f0606400700 1 Processor -- start 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.247+0000 7f0606400700 1 -- start start 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.247+0000 7f0606400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 0x7f060010b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.247+0000 7f0606400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f060007be00 con 0x7f0600109030 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.247+0000 7f05ff400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 0x7f060010b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.248+0000 7f05ff400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 0x7f060010b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52258/0 (socket says 172.21.15.6:52258) 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.248+0000 7f05ff400700 1 -- 172.21.15.6:0/1274458835 learned_addr learned my addr 172.21.15.6:0/1274458835 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.248+0000 7f05ff400700 1 -- 172.21.15.6:0/1274458835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060010b960 con 0x7f0600109030 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.248+0000 7f05ff400700 1 --2- 172.21.15.6:0/1274458835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 0x7f060010b420 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f05f00098a0 tx=0x7f05f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f90b956eeaddca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.249+0000 7f05fe000700 1 -- 172.21.15.6:0/1274458835 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05f0004160 con 0x7f0600109030 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.249+0000 7f05fe000700 1 -- 172.21.15.6:0/1274458835 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f0004300 con 0x7f0600109030 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.249+0000 7f05fe000700 1 -- 172.21.15.6:0/1274458835 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05f00036a0 con 0x7f0600109030 2024-04-26T01:32:31.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.249+0000 7f0606400700 1 -- 172.21.15.6:0/1274458835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 msgr2=0x7f060010b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.249+0000 7f0606400700 1 --2- 172.21.15.6:0/1274458835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 0x7f060010b420 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f05f00098a0 tx=0x7f05f0009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.249+0000 7f0606400700 1 -- 172.21.15.6:0/1274458835 shutdown_connections 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.249+0000 7f0606400700 1 --2- 172.21.15.6:0/1274458835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 0x7f060010b420 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.249+0000 7f0606400700 1 -- 172.21.15.6:0/1274458835 >> 172.21.15.6:0/1274458835 conn(0x7f06001029a0 msgr2=0x7f0600104de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.249+0000 7f0606400700 1 -- 172.21.15.6:0/1274458835 shutdown_connections 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.249+0000 7f0606400700 1 -- 172.21.15.6:0/1274458835 wait complete. 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.250+0000 7f0606400700 1 Processor -- start 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.251+0000 7f0606400700 1 -- start start 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.251+0000 7f0606400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 0x7f06001088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.251+0000 7f0606400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f060010bd70 con 0x7f0600109030 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.252+0000 7f05ff400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 0x7f06001088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.252+0000 7f05ff400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 0x7f06001088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52266/0 (socket says 172.21.15.6:52266) 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.252+0000 7f05ff400700 1 -- 172.21.15.6:0/141749351 learned_addr learned my addr 172.21.15.6:0/141749351 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.252+0000 7f05ff400700 1 -- 172.21.15.6:0/141749351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0600108df0 con 0x7f0600109030 2024-04-26T01:32:31.967 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.252+0000 7f05ff400700 1 --2- 172.21.15.6:0/141749351 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 0x7f06001088b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f05f0000c00 tx=0x7f05f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:31.967 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.253+0000 7f05f7e00700 1 -- 172.21.15.6:0/141749351 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05f0003db0 con 0x7f0600109030 2024-04-26T01:32:31.967 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.253+0000 7f05f7e00700 1 -- 172.21.15.6:0/141749351 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f0011630 con 0x7f0600109030 2024-04-26T01:32:31.967 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.253+0000 7f05f7e00700 1 -- 172.21.15.6:0/141749351 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05f0011950 con 0x7f0600109030 2024-04-26T01:32:31.967 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.253+0000 7f0606400700 1 -- 172.21.15.6:0/141749351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0600106fe0 con 0x7f0600109030 2024-04-26T01:32:31.967 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.253+0000 7f0606400700 1 -- 172.21.15.6:0/141749351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0600107520 con 0x7f0600109030 2024-04-26T01:32:31.967 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.253+0000 7f05f7e00700 1 -- 172.21.15.6:0/141749351 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f05f001a460 con 0x7f0600109030 2024-04-26T01:32:31.967 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.253+0000 7f05f7e00700 1 -- 172.21.15.6:0/141749351 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f05f001b780 con 0x7f0600109030 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.254+0000 7f0606400700 1 -- 172.21.15.6:0/141749351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06000640a0 con 0x7f0600109030 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.257+0000 7f05f7e00700 1 -- 172.21.15.6:0/141749351 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f05f001ba20 con 0x7f0600109030 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.335+0000 7f0606400700 1 -- 172.21.15.6:0/141749351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f0600195260 con 0x7f0600109030 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.340+0000 7f05f7e00700 1 -- 172.21.15.6:0/141749351 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+490 (secure 0 0 0) 0x7f05f001bc00 con 0x7f0600109030 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.341+0000 7f05f7e00700 1 -- 172.21.15.6:0/141749351 <== mon.0 v2:172.21.15.6:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f05f001bde0 con 0x7f0600109030 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.342+0000 7f0606400700 1 -- 172.21.15.6:0/141749351 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 msgr2=0x7f06001088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.342+0000 7f0606400700 1 --2- 172.21.15.6:0/141749351 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 0x7f06001088b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f05f0000c00 tx=0x7f05f0004750 comp rx=0 tx=0).stop 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.342+0000 7f0606400700 1 -- 172.21.15.6:0/141749351 shutdown_connections 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.342+0000 7f0606400700 1 --2- 172.21.15.6:0/141749351 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0600109030 0x7f06001088b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.342+0000 7f0606400700 1 -- 172.21.15.6:0/141749351 >> 172.21.15.6:0/141749351 conn(0x7f06001029a0 msgr2=0x7f0600104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.343+0000 7f0606400700 1 -- 172.21.15.6:0/141749351 shutdown_connections 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.343+0000 7f0606400700 1 -- 172.21.15.6:0/141749351 wait complete. 2024-04-26T01:32:31.968 INFO:teuthology.orchestra.run.smithi006.stdout:Generating new minimal ceph.conf... 2024-04-26T01:32:33.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:32 smithi006 ceph-mon[28055]: from='client.? 172.21.15.6:0/1625385378' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:32:41.256 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.692+0000 7efcd9200700 1 Processor -- start 2024-04-26T01:32:41.257 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.692+0000 7efcd9200700 1 -- start start 2024-04-26T01:32:41.257 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.692+0000 7efcd9200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 0x7efcd410ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:41.257 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.692+0000 7efcd9200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efcd410b150 con 0x7efcd410a760 2024-04-26T01:32:41.257 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.693+0000 7efcd1e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 0x7efcd410ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:41.257 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.693+0000 7efcd1e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 0x7efcd410ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52270/0 (socket says 172.21.15.6:52270) 2024-04-26T01:32:41.257 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.693+0000 7efcd1e00700 1 -- 172.21.15.6:0/3640484269 learned_addr learned my addr 172.21.15.6:0/3640484269 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:41.257 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.693+0000 7efcd1e00700 1 -- 172.21.15.6:0/3640484269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd410b910 con 0x7efcd410a760 2024-04-26T01:32:41.257 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.694+0000 7efcd1e00700 1 --2- 172.21.15.6:0/3640484269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 0x7efcd410ab80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7efcc40098a0 tx=0x7efcc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f3c23a9edde3db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:41.258 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.694+0000 7efcd0a00700 1 -- 172.21.15.6:0/3640484269 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc4004160 con 0x7efcd410a760 2024-04-26T01:32:41.258 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.694+0000 7efcd0a00700 1 -- 172.21.15.6:0/3640484269 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efcc4004300 con 0x7efcd410a760 2024-04-26T01:32:41.258 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.694+0000 7efcd0a00700 1 -- 172.21.15.6:0/3640484269 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc4011af0 con 0x7efcd410a760 2024-04-26T01:32:41.258 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.695+0000 7efcd9200700 1 -- 172.21.15.6:0/3640484269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 msgr2=0x7efcd410ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:41.258 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.695+0000 7efcd9200700 1 --2- 172.21.15.6:0/3640484269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 0x7efcd410ab80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7efcc40098a0 tx=0x7efcc4009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:41.258 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.695+0000 7efcd9200700 1 -- 172.21.15.6:0/3640484269 shutdown_connections 2024-04-26T01:32:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.695+0000 7efcd9200700 1 --2- 172.21.15.6:0/3640484269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 0x7efcd410ab80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.695+0000 7efcd9200700 1 -- 172.21.15.6:0/3640484269 >> 172.21.15.6:0/3640484269 conn(0x7efcd407dc20 msgr2=0x7efcd41084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.695+0000 7efcd9200700 1 -- 172.21.15.6:0/3640484269 shutdown_connections 2024-04-26T01:32:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.695+0000 7efcd9200700 1 -- 172.21.15.6:0/3640484269 wait complete. 2024-04-26T01:32:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.696+0000 7efcd9200700 1 Processor -- start 2024-04-26T01:32:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.696+0000 7efcd9200700 1 -- start start 2024-04-26T01:32:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.696+0000 7efcd9200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 0x7efcd419e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.696+0000 7efcd9200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efcd410ecc0 con 0x7efcd410a760 2024-04-26T01:32:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.697+0000 7efcd1e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 0x7efcd419e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.697+0000 7efcd1e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 0x7efcd419e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52274/0 (socket says 172.21.15.6:52274) 2024-04-26T01:32:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.697+0000 7efcd1e00700 1 -- 172.21.15.6:0/1625385378 learned_addr learned my addr 172.21.15.6:0/1625385378 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.697+0000 7efcd1e00700 1 -- 172.21.15.6:0/1625385378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd419ed30 con 0x7efcd410a760 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.697+0000 7efcd1e00700 1 --2- 172.21.15.6:0/1625385378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 0x7efcd419e7f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efcc40117b0 tx=0x7efcc4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.698+0000 7efccaa00700 1 -- 172.21.15.6:0/1625385378 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc4003af0 con 0x7efcd410a760 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.698+0000 7efccaa00700 1 -- 172.21.15.6:0/1625385378 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efcc4020070 con 0x7efcd410a760 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.698+0000 7efccaa00700 1 -- 172.21.15.6:0/1625385378 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc4019630 con 0x7efcd410a760 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.698+0000 7efcd9200700 1 -- 172.21.15.6:0/1625385378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcd419efc0 con 0x7efcd410a760 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.698+0000 7efcd9200700 1 -- 172.21.15.6:0/1625385378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcd419fae0 con 0x7efcd410a760 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.699+0000 7efccaa00700 1 -- 172.21.15.6:0/1625385378 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efcc40198a0 con 0x7efcd410a760 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.699+0000 7efccaa00700 1 -- 172.21.15.6:0/1625385378 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efcc4025dd0 con 0x7efcd410a760 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.700+0000 7efcd9200700 1 -- 172.21.15.6:0/1625385378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcd40640a0 con 0x7efcd410a760 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.703+0000 7efccaa00700 1 -- 172.21.15.6:0/1625385378 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efcc40042a0 con 0x7efcd410a760 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.780+0000 7efcd9200700 1 -- 172.21.15.6:0/1625385378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7efcd419fe40 con 0x7efcd410a760 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.781+0000 7efccaa00700 1 -- 172.21.15.6:0/1625385378 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+173 (secure 0 0 0) 0x7efcc4019d20 con 0x7efcd410a760 2024-04-26T01:32:41.260 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.783+0000 7efcd9200700 1 -- 172.21.15.6:0/1625385378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 msgr2=0x7efcd419e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:41.261 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.783+0000 7efcd9200700 1 --2- 172.21.15.6:0/1625385378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 0x7efcd419e7f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efcc40117b0 tx=0x7efcc4011b70 comp rx=0 tx=0).stop 2024-04-26T01:32:41.261 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.783+0000 7efcd9200700 1 -- 172.21.15.6:0/1625385378 shutdown_connections 2024-04-26T01:32:41.261 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.783+0000 7efcd9200700 1 --2- 172.21.15.6:0/1625385378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcd410a760 0x7efcd419e7f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:41.261 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.783+0000 7efcd9200700 1 -- 172.21.15.6:0/1625385378 >> 172.21.15.6:0/1625385378 conn(0x7efcd407dc20 msgr2=0x7efcd4107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:41.261 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.783+0000 7efcd9200700 1 -- 172.21.15.6:0/1625385378 shutdown_connections 2024-04-26T01:32:41.261 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:32.783+0000 7efcd9200700 1 -- 172.21.15.6:0/1625385378 wait complete. 2024-04-26T01:32:41.261 INFO:teuthology.orchestra.run.smithi006.stdout:Restarting the monitor... 2024-04-26T01:32:41.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:41 smithi006 systemd[1]: Stopping Ceph mon.a for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:32:43.018 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:42 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a[28051]: 2024-04-26T01:32:42.652+0000 7fe233000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:32:43.018 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:42 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a[28051]: 2024-04-26T01:32:42.652+0000 7fe233000700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T01:32:43.018 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:42 smithi006 podman[28298]: 2024-04-26 01:32:42.725234666 +0000 UTC m=+1.065236067 container died 7934c93ceb1e05ace13dfc740f0434033cbc9c7ffc6f6ed95259d195ceba9ccd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-04-26T01:32:43.313 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:43 smithi006 podman[28298]: 2024-04-26 01:32:43.217818703 +0000 UTC m=+1.557820155 container cleanup 7934c93ceb1e05ace13dfc740f0434033cbc9c7ffc6f6ed95259d195ceba9ccd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:32:43.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:43 smithi006 bash[28298]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a 2024-04-26T01:32:43.813 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:43 smithi006 podman[28311]: 2024-04-26 01:32:43.418748872 +0000 UTC m=+0.677069332 container remove 7934c93ceb1e05ace13dfc740f0434033cbc9c7ffc6f6ed95259d195ceba9ccd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-26T01:32:44.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:43 smithi006 systemd[1]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T01:32:44.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:43 smithi006 systemd[1]: Stopped Ceph mon.a for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:32:44.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:43 smithi006 systemd[1]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.a.service: Consumed 2.009s CPU time. 2024-04-26T01:32:44.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:43 smithi006 systemd[1]: Starting Ceph mon.a for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:32:44.701 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 podman[28399]: 2024-04-26 01:32:44.24391723 +0000 UTC m=+0.033761062 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:32:44.701 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 podman[28399]: 2024-04-26 01:32:44.352557636 +0000 UTC m=+0.142401535 container create 3d4cd5294d32443d4cb0bd7e3122659a74439ad64ce72f1163da9b03b96c02a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T01:32:44.809 INFO:teuthology.orchestra.run.smithi006.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T01:32:44.961 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 podman[28399]: 2024-04-26 01:32:44.706475165 +0000 UTC m=+0.496319071 container init 3d4cd5294d32443d4cb0bd7e3122659a74439ad64ce72f1163da9b03b96c02a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-26T01:32:44.961 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 podman[28399]: 2024-04-26 01:32:44.720536217 +0000 UTC m=+0.510380071 container start 3d4cd5294d32443d4cb0bd7e3122659a74439ad64ce72f1163da9b03b96c02a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=) 2024-04-26T01:32:44.961 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T01:32:44.961 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-26T01:32:44.961 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: pidfile_write: ignore empty --pid-file 2024-04-26T01:32:44.961 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: load: jerasure load: lrc 2024-04-26T01:32:44.961 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: RocksDB version: 7.9.2 2024-04-26T01:32:44.961 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Git sha 0 2024-04-26T01:32:44.961 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-26T01:32:44.961 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: DB SUMMARY 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: DB Session ID: P5EQ71NQMHUC3A2093X4 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: CURRENT file: CURRENT 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105412 ; 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.error_if_exists: 0 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.create_if_missing: 0 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.paranoid_checks: 1 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.env: 0x55cb54dffcc0 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T01:32:44.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.info_log: 0x55cb74127c40 2024-04-26T01:32:44.963 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T01:32:44.963 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.statistics: (nil) 2024-04-26T01:32:44.963 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.use_fsync: 0 2024-04-26T01:32:44.963 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_log_file_size: 0 2024-04-26T01:32:44.963 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T01:32:44.963 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.allow_fallocate: 1 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.use_direct_reads: 0 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.db_log_dir: 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.wal_dir: 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T01:32:44.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.write_buffer_manager: 0x55cb740bb0e0 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T01:32:44.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.unordered_write: 0 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.row_cache: None 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.wal_filter: None 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.two_write_queues: 0 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.wal_compression: 0 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.atomic_flush: 0 2024-04-26T01:32:44.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.log_readahead_size: 0 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_background_jobs: 2 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_background_compactions: -1 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_subcompactions: 1 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T01:32:44.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_open_files: -1 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_background_flushes: -1 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Compression algorithms supported: 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T01:32:44.968 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: kZSTD supported: 0 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: kXpressCompression supported: 0 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: kZlibCompression supported: 1 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: kSnappyCompression supported: 1 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: kLZ4Compression supported: 1 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: kBZip2Compression supported: 0 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.merge_operator: 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_filter: None 2024-04-26T01:32:44.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cb74127e00) 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout: cache_index_and_filter_blocks: 1 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout: pin_top_level_index_and_filter: 1 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout: index_type: 0 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout: data_block_index_type: 0 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout: index_shortening: 1 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout: checksum: 4 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout: no_block_cache: 0 2024-04-26T01:32:44.970 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_cache: 0x55cb740ed1f0 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_cache_name: BinnedLRUCache 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_cache_options: 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: capacity : 536870912 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: num_shard_bits : 4 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: strict_capacity_limit : 0 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: high_pri_pool_ratio: 0.000 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_cache_compressed: (nil) 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: persistent_cache: (nil) 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_size: 4096 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_size_deviation: 10 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_restart_interval: 16 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: index_block_restart_interval: 1 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: metadata_block_size: 4096 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: partition_filters: 0 2024-04-26T01:32:44.971 INFO:journalctl@ceph.mon.a.smithi006.stdout: use_delta_encoding: 1 2024-04-26T01:32:44.972 INFO:journalctl@ceph.mon.a.smithi006.stdout: filter_policy: bloomfilter 2024-04-26T01:32:44.972 INFO:journalctl@ceph.mon.a.smithi006.stdout: whole_key_filtering: 1 2024-04-26T01:32:44.972 INFO:journalctl@ceph.mon.a.smithi006.stdout: verify_compression: 0 2024-04-26T01:32:44.972 INFO:journalctl@ceph.mon.a.smithi006.stdout: read_amp_bytes_per_bit: 0 2024-04-26T01:32:44.972 INFO:journalctl@ceph.mon.a.smithi006.stdout: format_version: 5 2024-04-26T01:32:44.972 INFO:journalctl@ceph.mon.a.smithi006.stdout: enable_index_compression: 1 2024-04-26T01:32:44.972 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_align: 0 2024-04-26T01:32:44.972 INFO:journalctl@ceph.mon.a.smithi006.stdout: max_auto_readahead_size: 262144 2024-04-26T01:32:44.972 INFO:journalctl@ceph.mon.a.smithi006.stdout: prepopulate_block_cache: 0 2024-04-26T01:32:44.972 INFO:journalctl@ceph.mon.a.smithi006.stdout: initial_auto_readahead_size: 8192 2024-04-26T01:32:44.972 INFO:journalctl@ceph.mon.a.smithi006.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T01:32:44.973 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T01:32:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T01:32:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compression: NoCompression 2024-04-26T01:32:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T01:32:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T01:32:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T01:32:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.num_levels: 7 2024-04-26T01:32:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T01:32:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T01:32:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T01:32:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T01:32:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T01:32:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T01:32:44.975 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T01:32:44.975 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:32:44.975 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T01:32:44.975 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T01:32:44.975 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:32:44.975 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:32:44.975 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T01:32:44.975 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T01:32:44.975 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T01:32:44.975 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T01:32:44.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T01:32:44.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.inplace_update_support: 0 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T01:32:44.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.bloom_locality: 0 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.max_successive_merges: 0 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.ttl: 2592000 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.enable_blob_files: false 2024-04-26T01:32:44.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.min_blob_size: 0 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ce249ad8-4c3c-490c-952f-f60ad47f0898 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714095164777486, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T01:32:44.980 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714095164779733, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100524, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98409, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14264, "raw_average_key_size": 49, "raw_value_size": 90663, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714095164, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ce249ad8-4c3c-490c-952f-f60ad47f0898", "db_session_id": "P5EQ71NQMHUC3A2093X4", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714095164779915, "job": 1, "event": "recovery_finished"} 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cb74202000 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: DB pointer 0x55cb741ec000 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout: ** DB Stats ** 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:32:44.981 INFO:journalctl@ceph.mon.a.smithi006.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:32:44.982 INFO:journalctl@ceph.mon.a.smithi006.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T01:32:44.982 INFO:journalctl@ceph.mon.a.smithi006.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:32:44.982 INFO:journalctl@ceph.mon.a.smithi006.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:32:44.982 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-04-26T01:32:44.982 INFO:journalctl@ceph.mon.a.smithi006.stdout: ** Compaction Stats [default] ** 2024-04-26T01:32:44.982 INFO:journalctl@ceph.mon.a.smithi006.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:32:44.982 INFO:journalctl@ceph.mon.a.smithi006.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T01:32:44.982 INFO:journalctl@ceph.mon.a.smithi006.stdout: L0 2/0 100.03 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 60.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-26T01:32:44.982 INFO:journalctl@ceph.mon.a.smithi006.stdout: Sum 2/0 100.03 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 60.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-26T01:32:44.983 INFO:journalctl@ceph.mon.a.smithi006.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 60.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-26T01:32:44.983 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-04-26T01:32:44.983 INFO:journalctl@ceph.mon.a.smithi006.stdout: ** Compaction Stats [default] ** 2024-04-26T01:32:44.983 INFO:journalctl@ceph.mon.a.smithi006.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:32:44.985 INFO:journalctl@ceph.mon.a.smithi006.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T01:32:44.985 INFO:journalctl@ceph.mon.a.smithi006.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 60.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-26T01:32:44.985 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-04-26T01:32:44.985 INFO:journalctl@ceph.mon.a.smithi006.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T01:32:44.985 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-04-26T01:32:44.985 INFO:journalctl@ceph.mon.a.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:32:44.985 INFO:journalctl@ceph.mon.a.smithi006.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T01:32:44.985 INFO:journalctl@ceph.mon.a.smithi006.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T01:32:44.986 INFO:journalctl@ceph.mon.a.smithi006.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T01:32:44.986 INFO:journalctl@ceph.mon.a.smithi006.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T01:32:44.986 INFO:journalctl@ceph.mon.a.smithi006.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T01:32:44.986 INFO:journalctl@ceph.mon.a.smithi006.stdout: Cumulative compaction: 0.00 GB write, 14.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:32:44.986 INFO:journalctl@ceph.mon.a.smithi006.stdout: Interval compaction: 0.00 GB write, 14.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:32:44.986 INFO:journalctl@ceph.mon.a.smithi006.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T01:32:44.986 INFO:journalctl@ceph.mon.a.smithi006.stdout: Block cache BinnedLRUCache@0x55cb740ed1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 2.3e-05 secs_since: 0 2024-04-26T01:32:44.986 INFO:journalctl@ceph.mon.a.smithi006.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-26T01:32:44.986 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-04-26T01:32:44.986 INFO:journalctl@ceph.mon.a.smithi006.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T01:32:44.986 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: starting mon.a rank 0 at public addrs [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] at bind addrs [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:44.986 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: mon.a@-1(???) e1 preinit fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: mon.a@-1(???).mds e1 new map 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: mon.a@-1(???).mds e1 print_map 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout: e1 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout: legacy client fscid: -1 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout: No filesystems configured 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 bash[28399]: 3d4cd5294d32443d4cb0bd7e3122659a74439ad64ce72f1163da9b03b96c02a0 2024-04-26T01:32:44.987 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 systemd[1]: Started Ceph mon.a for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:32:44.988 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T01:32:44.988 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: monmap epoch 1 2024-04-26T01:32:44.988 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:44.988 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: last_changed 2024-04-26T01:32:24.866051+0000 2024-04-26T01:32:44.988 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: created 2024-04-26T01:32:24.866051+0000 2024-04-26T01:32:44.988 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: min_mon_release 19 (squid) 2024-04-26T01:32:44.988 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: election_strategy: 1 2024-04-26T01:32:44.988 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-26T01:32:44.988 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: fsmap 2024-04-26T01:32:44.988 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T01:32:44.988 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:44 smithi006 ceph-mon[28419]: mgrmap e1: no daemons active 2024-04-26T01:32:46.283 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.472+0000 7f859f400700 1 Processor -- start 2024-04-26T01:32:46.288 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.472+0000 7f859f400700 1 -- start start 2024-04-26T01:32:46.288 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.472+0000 7f859f400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 0x7f859807b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:46.288 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.472+0000 7f859f400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f859807b810 con 0x7f859807cde0 2024-04-26T01:32:46.288 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.473+0000 7f859ca00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 0x7f859807b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:46.288 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.473+0000 7f859ca00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 0x7f859807b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54068/0 (socket says 172.21.15.6:54068) 2024-04-26T01:32:46.289 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.473+0000 7f859ca00700 1 -- 172.21.15.6:0/3245597079 learned_addr learned my addr 172.21.15.6:0/3245597079 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:46.289 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.474+0000 7f859ca00700 1 -- 172.21.15.6:0/3245597079 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f859807b990 con 0x7f859807cde0 2024-04-26T01:32:46.289 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.474+0000 7f859ca00700 1 --2- 172.21.15.6:0/3245597079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 0x7f859807b240 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f858c0098a0 tx=0x7f858c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112675a823534dce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:46.289 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.475+0000 7f8597400700 1 -- 172.21.15.6:0/3245597079 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f858c004160 con 0x7f859807cde0 2024-04-26T01:32:46.289 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.475+0000 7f8597400700 1 -- 172.21.15.6:0/3245597079 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f858c004300 con 0x7f859807cde0 2024-04-26T01:32:46.289 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.475+0000 7f8597400700 1 -- 172.21.15.6:0/3245597079 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f858c011af0 con 0x7f859807cde0 2024-04-26T01:32:46.289 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.476+0000 7f859f400700 1 -- 172.21.15.6:0/3245597079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 msgr2=0x7f859807b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:46.289 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.476+0000 7f859f400700 1 --2- 172.21.15.6:0/3245597079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 0x7f859807b240 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f858c0098a0 tx=0x7f858c009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:46.289 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.476+0000 7f859f400700 1 -- 172.21.15.6:0/3245597079 shutdown_connections 2024-04-26T01:32:46.289 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.476+0000 7f859f400700 1 --2- 172.21.15.6:0/3245597079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 0x7f859807b240 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:46.289 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.476+0000 7f859f400700 1 -- 172.21.15.6:0/3245597079 >> 172.21.15.6:0/3245597079 conn(0x7f8598103b50 msgr2=0x7f8598105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.476+0000 7f859f400700 1 -- 172.21.15.6:0/3245597079 shutdown_connections 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.476+0000 7f859f400700 1 -- 172.21.15.6:0/3245597079 wait complete. 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.477+0000 7f859f400700 1 Processor -- start 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.477+0000 7f859f400700 1 -- start start 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.477+0000 7f859f400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 0x7f85981a2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.477+0000 7f859f400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f859810a6e0 con 0x7f859807cde0 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.478+0000 7f859ca00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 0x7f85981a2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.478+0000 7f859ca00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 0x7f85981a2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54072/0 (socket says 172.21.15.6:54072) 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.478+0000 7f859ca00700 1 -- 172.21.15.6:0/2178792255 learned_addr learned my addr 172.21.15.6:0/2178792255 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.478+0000 7f859ca00700 1 -- 172.21.15.6:0/2178792255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85981a2fd0 con 0x7f859807cde0 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.479+0000 7f859ca00700 1 --2- 172.21.15.6:0/2178792255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 0x7f85981a2a90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f858c000c00 tx=0x7f858c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.479+0000 7f8595600700 1 -- 172.21.15.6:0/2178792255 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f858c003b20 con 0x7f859807cde0 2024-04-26T01:32:46.290 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.479+0000 7f8595600700 1 -- 172.21.15.6:0/2178792255 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f858c01a8c0 con 0x7f859807cde0 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.479+0000 7f859f400700 1 -- 172.21.15.6:0/2178792255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85981a3260 con 0x7f859807cde0 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.479+0000 7f8595600700 1 -- 172.21.15.6:0/2178792255 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f858c024940 con 0x7f859807cde0 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.479+0000 7f859f400700 1 -- 172.21.15.6:0/2178792255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85981a5f50 con 0x7f859807cde0 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.480+0000 7f8595600700 1 -- 172.21.15.6:0/2178792255 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f858c022020 con 0x7f859807cde0 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.480+0000 7f8595600700 1 -- 172.21.15.6:0/2178792255 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f858c02dde0 con 0x7f859807cde0 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.480+0000 7f859f400700 1 -- 172.21.15.6:0/2178792255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85980640a0 con 0x7f859807cde0 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.484+0000 7f8595600700 1 -- 172.21.15.6:0/2178792255 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f858c0042a0 con 0x7f859807cde0 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.562+0000 7f859f400700 1 -- 172.21.15.6:0/2178792255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f859819bff0 con 0x7f859807cde0 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.566+0000 7f8595600700 1 -- 172.21.15.6:0/2178792255 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f858c01aa70 con 0x7f859807cde0 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.567+0000 7f859f400700 1 -- 172.21.15.6:0/2178792255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 msgr2=0x7f85981a2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.567+0000 7f859f400700 1 --2- 172.21.15.6:0/2178792255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 0x7f85981a2a90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f858c000c00 tx=0x7f858c011b70 comp rx=0 tx=0).stop 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.568+0000 7f859f400700 1 -- 172.21.15.6:0/2178792255 shutdown_connections 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.568+0000 7f859f400700 1 --2- 172.21.15.6:0/2178792255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f859807cde0 0x7f85981a2a90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.568+0000 7f859f400700 1 -- 172.21.15.6:0/2178792255 >> 172.21.15.6:0/2178792255 conn(0x7f8598103b50 msgr2=0x7f8598105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:46.292 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.568+0000 7f859f400700 1 -- 172.21.15.6:0/2178792255 shutdown_connections 2024-04-26T01:32:46.292 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.568+0000 7f859f400700 1 -- 172.21.15.6:0/2178792255 wait complete. 2024-04-26T01:32:46.292 INFO:teuthology.orchestra.run.smithi006.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T01:32:46.313 INFO:teuthology.orchestra.run.smithi006.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T01:32:46.314 INFO:teuthology.orchestra.run.smithi006.stdout:Creating mgr... 2024-04-26T01:32:46.315 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T01:32:46.315 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T01:32:46.733 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.x 2024-04-26T01:32:46.733 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-26T01:32:46.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:46 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2178792255' entity='client.admin' 2024-04-26T01:32:47.020 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a5b72bee-036c-11ef-bc93-c7b262605968.target.wants/ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-a5b72bee-036c-11ef-bc93-c7b262605968@.service. 2024-04-26T01:32:47.314 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:47 smithi006 systemd[1]: Starting Ceph mgr.x for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:32:47.734 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:47 smithi006 podman[28642]: 2024-04-26 01:32:47.352534817 +0000 UTC m=+0.036514956 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:32:47.734 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:47 smithi006 podman[28642]: 2024-04-26 01:32:47.73871188 +0000 UTC m=+0.422692037 container create 879fdfd52bdac43e4d44cbd18863b628c21d8a988bb9b32ab81c1ca2d15dc47c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-26T01:32:48.043 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:47 smithi006 podman[28642]: 2024-04-26 01:32:47.967560024 +0000 UTC m=+0.651540204 container init 879fdfd52bdac43e4d44cbd18863b628c21d8a988bb9b32ab81c1ca2d15dc47c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-26T01:32:48.044 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:47 smithi006 podman[28642]: 2024-04-26 01:32:47.9779227 +0000 UTC m=+0.661902859 container start 879fdfd52bdac43e4d44cbd18863b628c21d8a988bb9b32ab81c1ca2d15dc47c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-26T01:32:48.074 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T01:32:48.074 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-04-26T01:32:48.089 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T01:32:48.089 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-04-26T01:32:48.089 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-04-26T01:32:48.089 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T01:32:48.108 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T01:32:48.108 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-04-26T01:32:48.124 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T01:32:48.124 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-04-26T01:32:48.124 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-04-26T01:32:48.124 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T01:32:48.125 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr to start... 2024-04-26T01:32:48.125 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr... 2024-04-26T01:32:48.313 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:48 smithi006 bash[28642]: 879fdfd52bdac43e4d44cbd18863b628c21d8a988bb9b32ab81c1ca2d15dc47c 2024-04-26T01:32:48.314 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:48 smithi006 systemd[1]: Started Ceph mgr.x for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:32:48.314 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:48 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:48.231+0000 7f34f103e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:32:48.614 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:48 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:48.501+0000 7f34f103e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:32:48.864 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:48 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:48.645+0000 7f34f103e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:32:49.208 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:48 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/4234114664' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T01:32:49.209 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:49 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:49.100+0000 7f34f103e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:32:49.516 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:49 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:49.221+0000 7f34f103e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:32:49.529 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:32:49.540 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:32:49.540 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:32:49.540 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T01:32:49.540 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T01:32:49.540 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T01:32:49.540 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T01:32:49.540 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:49.540 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:49.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T01:32:49.542 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T01:32:49.543 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T01:32:29.072125+0000", 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.776+0000 7fdb7be00700 1 Processor -- start 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.777+0000 7fdb7be00700 1 -- start start 2024-04-26T01:32:49.544 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.777+0000 7fdb7be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 0x7fdb7c108770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:49.545 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.777+0000 7fdb7be00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c108d40 con 0x7fdb7c108350 2024-04-26T01:32:49.545 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.777+0000 7fdb7aa00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 0x7fdb7c108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:49.545 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.778+0000 7fdb7aa00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 0x7fdb7c108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57244/0 (socket says 172.21.15.6:57244) 2024-04-26T01:32:49.545 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.778+0000 7fdb7aa00700 1 -- 172.21.15.6:0/2069906355 learned_addr learned my addr 172.21.15.6:0/2069906355 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:49.545 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.778+0000 7fdb7aa00700 1 -- 172.21.15.6:0/2069906355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c109550 con 0x7fdb7c108350 2024-04-26T01:32:49.545 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.778+0000 7fdb7aa00700 1 --2- 172.21.15.6:0/2069906355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 0x7fdb7c108770 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0098a0 tx=0x7fdb6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1fac62d4624509 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:49.545 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.779+0000 7fdb79600700 1 -- 172.21.15.6:0/2069906355 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb6c004160 con 0x7fdb7c108350 2024-04-26T01:32:49.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.779+0000 7fdb79600700 1 -- 172.21.15.6:0/2069906355 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdb6c004300 con 0x7fdb7c108350 2024-04-26T01:32:49.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.779+0000 7fdb79600700 1 -- 172.21.15.6:0/2069906355 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb6c011af0 con 0x7fdb7c108350 2024-04-26T01:32:49.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.779+0000 7fdb7be00700 1 -- 172.21.15.6:0/2069906355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 msgr2=0x7fdb7c108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:49.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.779+0000 7fdb7be00700 1 --2- 172.21.15.6:0/2069906355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 0x7fdb7c108770 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0098a0 tx=0x7fdb6c009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:49.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.780+0000 7fdb7be00700 1 -- 172.21.15.6:0/2069906355 shutdown_connections 2024-04-26T01:32:49.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.780+0000 7fdb7be00700 1 --2- 172.21.15.6:0/2069906355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 0x7fdb7c108770 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:49.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.780+0000 7fdb7be00700 1 -- 172.21.15.6:0/2069906355 >> 172.21.15.6:0/2069906355 conn(0x7fdb7c103990 msgr2=0x7fdb7c105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:49.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.780+0000 7fdb7be00700 1 -- 172.21.15.6:0/2069906355 shutdown_connections 2024-04-26T01:32:49.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.780+0000 7fdb7be00700 1 -- 172.21.15.6:0/2069906355 wait complete. 2024-04-26T01:32:49.547 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.781+0000 7fdb7be00700 1 Processor -- start 2024-04-26T01:32:49.547 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.781+0000 7fdb7be00700 1 -- start start 2024-04-26T01:32:49.547 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.782+0000 7fdb7be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 0x7fdb7c19e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:49.547 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.782+0000 7fdb7be00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c10c900 con 0x7fdb7c108350 2024-04-26T01:32:49.547 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.782+0000 7fdb7aa00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 0x7fdb7c19e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:49.547 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.782+0000 7fdb7aa00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 0x7fdb7c19e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57258/0 (socket says 172.21.15.6:57258) 2024-04-26T01:32:49.547 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.782+0000 7fdb7aa00700 1 -- 172.21.15.6:0/4234114664 learned_addr learned my addr 172.21.15.6:0/4234114664 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:49.547 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.782+0000 7fdb7aa00700 1 -- 172.21.15.6:0/4234114664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c19ea30 con 0x7fdb7c108350 2024-04-26T01:32:49.547 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.783+0000 7fdb7aa00700 1 --2- 172.21.15.6:0/4234114664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 0x7fdb7c19e4f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c003f40 tx=0x7fdb6c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.783+0000 7fdb73400700 1 -- 172.21.15.6:0/4234114664 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb6c0038b0 con 0x7fdb7c108350 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.783+0000 7fdb73400700 1 -- 172.21.15.6:0/4234114664 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdb6c003a50 con 0x7fdb7c108350 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.783+0000 7fdb73400700 1 -- 172.21.15.6:0/4234114664 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb6c01a5c0 con 0x7fdb7c108350 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.783+0000 7fdb7be00700 1 -- 172.21.15.6:0/4234114664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb7c19ecc0 con 0x7fdb7c108350 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.783+0000 7fdb7be00700 1 -- 172.21.15.6:0/4234114664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb7c1a19b0 con 0x7fdb7c108350 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.783+0000 7fdb73400700 1 -- 172.21.15.6:0/4234114664 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdb6c01ac00 con 0x7fdb7c108350 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.783+0000 7fdb73400700 1 -- 172.21.15.6:0/4234114664 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdb6c019700 con 0x7fdb7c108350 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.784+0000 7fdb7be00700 1 -- 172.21.15.6:0/4234114664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb7c069ba0 con 0x7fdb7c108350 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.789+0000 7fdb73400700 1 -- 172.21.15.6:0/4234114664 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fdb6c0199a0 con 0x7fdb7c108350 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.869+0000 7fdb7be00700 1 -- 172.21.15.6:0/4234114664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fdb7c1a1b80 con 0x7fdb7c108350 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.869+0000 7fdb73400700 1 -- 172.21.15.6:0/4234114664 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fdb6c019b80 con 0x7fdb7c108350 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.871+0000 7fdb7be00700 1 -- 172.21.15.6:0/4234114664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 msgr2=0x7fdb7c19e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:49.548 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.871+0000 7fdb7be00700 1 --2- 172.21.15.6:0/4234114664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 0x7fdb7c19e4f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c003f40 tx=0x7fdb6c011890 comp rx=0 tx=0).stop 2024-04-26T01:32:49.549 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.871+0000 7fdb7be00700 1 -- 172.21.15.6:0/4234114664 shutdown_connections 2024-04-26T01:32:49.549 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.871+0000 7fdb7be00700 1 --2- 172.21.15.6:0/4234114664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb7c108350 0x7fdb7c19e4f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:49.549 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.871+0000 7fdb7be00700 1 -- 172.21.15.6:0/4234114664 >> 172.21.15.6:0/4234114664 conn(0x7fdb7c103990 msgr2=0x7fdb7c104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:49.549 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.871+0000 7fdb7be00700 1 -- 172.21.15.6:0/4234114664 shutdown_connections 2024-04-26T01:32:49.549 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.872+0000 7fdb7be00700 1 -- 172.21.15.6:0/4234114664 wait complete. 2024-04-26T01:32:49.549 INFO:teuthology.orchestra.run.smithi006.stdout:mgr not available, waiting (1/15)... 2024-04-26T01:32:51.298 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:50 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:50.843+0000 7f34f103e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:32:51.298 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:51 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:51.074+0000 7f34f103e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:32:51.298 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:51 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:51.303+0000 7f34f103e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:32:51.970 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:51 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:51.695+0000 7f34f103e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:32:51.971 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:51 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:51.828+0000 7f34f103e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:32:52.220 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:52 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:52.089+0000 7f34f103e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:32:52.524 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:52 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/4006158457' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T01:32:52.524 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:52 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:52.356+0000 7f34f103e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:32:52.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:32:52.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:32:52.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:32:52.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T01:32:52.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:52.842 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T01:32:52.842 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T01:32:52.842 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:52.842 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T01:32:52.842 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:52.842 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T01:32:52.842 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T01:32:52.842 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T01:32:52.842 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T01:32:52.842 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T01:32:52.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T01:32:52.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T01:32:52.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:52.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T01:32:52.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T01:32:52.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T01:32:52.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T01:32:52.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T01:32:52.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T01:32:52.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T01:32:52.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T01:32:29.072125+0000", 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.132+0000 7fba21000700 1 Processor -- start 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.133+0000 7fba21000700 1 -- start start 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.133+0000 7fba21000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 0x7fba1c10b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.133+0000 7fba21000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fba1c063ce0 con 0x7fba1c109030 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.134+0000 7fba19e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 0x7fba1c10b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:52.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.134+0000 7fba19e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 0x7fba1c10b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57264/0 (socket says 172.21.15.6:57264) 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.134+0000 7fba19e00700 1 -- 172.21.15.6:0/546594160 learned_addr learned my addr 172.21.15.6:0/546594160 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.134+0000 7fba19e00700 1 -- 172.21.15.6:0/546594160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1c10b960 con 0x7fba1c109030 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.135+0000 7fba19e00700 1 --2- 172.21.15.6:0/546594160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 0x7fba1c10b420 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba0c0098a0 tx=0x7fba0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551377348da1b08a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.135+0000 7fba18a00700 1 -- 172.21.15.6:0/546594160 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba0c004160 con 0x7fba1c109030 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.135+0000 7fba18a00700 1 -- 172.21.15.6:0/546594160 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fba0c004300 con 0x7fba1c109030 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.135+0000 7fba18a00700 1 -- 172.21.15.6:0/546594160 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba0c011af0 con 0x7fba1c109030 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.136+0000 7fba21000700 1 -- 172.21.15.6:0/546594160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 msgr2=0x7fba1c10b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.136+0000 7fba21000700 1 --2- 172.21.15.6:0/546594160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 0x7fba1c10b420 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba0c0098a0 tx=0x7fba0c009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.136+0000 7fba21000700 1 -- 172.21.15.6:0/546594160 shutdown_connections 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.136+0000 7fba21000700 1 --2- 172.21.15.6:0/546594160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 0x7fba1c10b420 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.136+0000 7fba21000700 1 -- 172.21.15.6:0/546594160 >> 172.21.15.6:0/546594160 conn(0x7fba1c102a40 msgr2=0x7fba1c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.136+0000 7fba21000700 1 -- 172.21.15.6:0/546594160 shutdown_connections 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.136+0000 7fba21000700 1 -- 172.21.15.6:0/546594160 wait complete. 2024-04-26T01:32:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.137+0000 7fba21000700 1 Processor -- start 2024-04-26T01:32:52.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.137+0000 7fba21000700 1 -- start start 2024-04-26T01:32:52.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.137+0000 7fba21000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 0x7fba1c19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:52.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.137+0000 7fba21000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fba1c10c8c0 con 0x7fba1c109030 2024-04-26T01:32:52.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.138+0000 7fba19e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 0x7fba1c19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:52.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.138+0000 7fba19e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 0x7fba1c19e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57266/0 (socket says 172.21.15.6:57266) 2024-04-26T01:32:52.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.138+0000 7fba19e00700 1 -- 172.21.15.6:0/4006158457 learned_addr learned my addr 172.21.15.6:0/4006158457 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:52.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.138+0000 7fba19e00700 1 -- 172.21.15.6:0/4006158457 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1c19ec00 con 0x7fba1c109030 2024-04-26T01:32:52.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.138+0000 7fba19e00700 1 --2- 172.21.15.6:0/4006158457 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 0x7fba1c19e6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fba0c003f40 tx=0x7fba0c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:52.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.139+0000 7fba12a00700 1 -- 172.21.15.6:0/4006158457 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba0c0038b0 con 0x7fba1c109030 2024-04-26T01:32:52.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.139+0000 7fba12a00700 1 -- 172.21.15.6:0/4006158457 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fba0c003a50 con 0x7fba1c109030 2024-04-26T01:32:52.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.139+0000 7fba21000700 1 -- 172.21.15.6:0/4006158457 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba1c19ee90 con 0x7fba1c109030 2024-04-26T01:32:52.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.139+0000 7fba12a00700 1 -- 172.21.15.6:0/4006158457 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba0c01a5c0 con 0x7fba1c109030 2024-04-26T01:32:52.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.139+0000 7fba21000700 1 -- 172.21.15.6:0/4006158457 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba1c1a1b80 con 0x7fba1c109030 2024-04-26T01:32:52.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.139+0000 7fba12a00700 1 -- 172.21.15.6:0/4006158457 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fba0c01ac00 con 0x7fba1c109030 2024-04-26T01:32:52.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.139+0000 7fba12a00700 1 -- 172.21.15.6:0/4006158457 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fba0c019700 con 0x7fba1c109030 2024-04-26T01:32:52.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.140+0000 7fba21000700 1 -- 172.21.15.6:0/4006158457 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba1c0640a0 con 0x7fba1c109030 2024-04-26T01:32:52.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.143+0000 7fba12a00700 1 -- 172.21.15.6:0/4006158457 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fba0c019900 con 0x7fba1c109030 2024-04-26T01:32:52.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.222+0000 7fba21000700 1 -- 172.21.15.6:0/4006158457 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fba1c1a1d50 con 0x7fba1c109030 2024-04-26T01:32:52.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.223+0000 7fba12a00700 1 -- 172.21.15.6:0/4006158457 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fba0c019ae0 con 0x7fba1c109030 2024-04-26T01:32:52.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.224+0000 7fba21000700 1 -- 172.21.15.6:0/4006158457 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 msgr2=0x7fba1c19e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:52.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.224+0000 7fba21000700 1 --2- 172.21.15.6:0/4006158457 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 0x7fba1c19e6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fba0c003f40 tx=0x7fba0c011890 comp rx=0 tx=0).stop 2024-04-26T01:32:52.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.224+0000 7fba21000700 1 -- 172.21.15.6:0/4006158457 shutdown_connections 2024-04-26T01:32:52.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.224+0000 7fba21000700 1 --2- 172.21.15.6:0/4006158457 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba1c109030 0x7fba1c19e6c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:52.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.224+0000 7fba21000700 1 -- 172.21.15.6:0/4006158457 >> 172.21.15.6:0/4006158457 conn(0x7fba1c102a40 msgr2=0x7fba1c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:52.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.224+0000 7fba21000700 1 -- 172.21.15.6:0/4006158457 shutdown_connections 2024-04-26T01:32:52.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.224+0000 7fba21000700 1 -- 172.21.15.6:0/4006158457 wait complete. 2024-04-26T01:32:52.856 INFO:teuthology.orchestra.run.smithi006.stdout:mgr not available, waiting (2/15)... 2024-04-26T01:32:53.563 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:53.165+0000 7f34f103e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:32:54.063 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:53.729+0000 7f34f103e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:32:54.063 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:53.858+0000 7f34f103e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:32:54.563 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:54 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:54.127+0000 7f34f103e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:32:55.048 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:54 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:54.789+0000 7f34f103e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:32:55.048 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:54 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:54.929+0000 7f34f103e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:32:55.313 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:55.270+0000 7f34f103e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:32:55.593 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:55.454+0000 7f34f103e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:32:55.593 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:55.582+0000 7f34f103e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:32:55.905 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:55 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1241042070' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T01:32:55.905 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:55.739+0000 7f34f103e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:32:56.212 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:56 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:56.118+0000 7f34f103e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:32:56.243 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:32:56.243 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:32:56.243 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T01:32:56.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T01:32:56.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:56.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T01:32:29.072125+0000", 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T01:32:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:32:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.502+0000 7fedfee00700 1 Processor -- start 2024-04-26T01:32:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.502+0000 7fedfee00700 1 -- start start 2024-04-26T01:32:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.503+0000 7fedfee00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 0x7fedf810ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.503+0000 7fedfee00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fedf810b160 con 0x7fedf810a770 2024-04-26T01:32:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.503+0000 7fedf7e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 0x7fedf810ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.503+0000 7fedf7e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 0x7fedf810ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57282/0 (socket says 172.21.15.6:57282) 2024-04-26T01:32:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.503+0000 7fedf7e00700 1 -- 172.21.15.6:0/1472310262 learned_addr learned my addr 172.21.15.6:0/1472310262 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.504+0000 7fedf7e00700 1 -- 172.21.15.6:0/1472310262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf810b920 con 0x7fedf810a770 2024-04-26T01:32:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.504+0000 7fedf7e00700 1 --2- 172.21.15.6:0/1472310262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 0x7fedf810ab90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fede80098a0 tx=0x7fede8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d76ef8ea0f82eaf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.504+0000 7fedf6a00700 1 -- 172.21.15.6:0/1472310262 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fede8004160 con 0x7fedf810a770 2024-04-26T01:32:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.505+0000 7fedf6a00700 1 -- 172.21.15.6:0/1472310262 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fede8004300 con 0x7fedf810a770 2024-04-26T01:32:56.249 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.505+0000 7fedf6a00700 1 -- 172.21.15.6:0/1472310262 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fede8011af0 con 0x7fedf810a770 2024-04-26T01:32:56.249 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.505+0000 7fedfee00700 1 -- 172.21.15.6:0/1472310262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 msgr2=0x7fedf810ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:56.249 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.505+0000 7fedfee00700 1 --2- 172.21.15.6:0/1472310262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 0x7fedf810ab90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fede80098a0 tx=0x7fede8009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:56.249 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.505+0000 7fedfee00700 1 -- 172.21.15.6:0/1472310262 shutdown_connections 2024-04-26T01:32:56.249 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.505+0000 7fedfee00700 1 --2- 172.21.15.6:0/1472310262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 0x7fedf810ab90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:56.249 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.505+0000 7fedfee00700 1 -- 172.21.15.6:0/1472310262 >> 172.21.15.6:0/1472310262 conn(0x7fedf807db60 msgr2=0x7fedf81084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:56.249 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.505+0000 7fedfee00700 1 -- 172.21.15.6:0/1472310262 shutdown_connections 2024-04-26T01:32:56.249 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.506+0000 7fedfee00700 1 -- 172.21.15.6:0/1472310262 wait complete. 2024-04-26T01:32:56.249 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.507+0000 7fedfee00700 1 Processor -- start 2024-04-26T01:32:56.249 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.507+0000 7fedfee00700 1 -- start start 2024-04-26T01:32:56.249 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.507+0000 7fedfee00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 0x7fedf819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.507+0000 7fedfee00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fedf810ecd0 con 0x7fedf810a770 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.508+0000 7fedf7e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 0x7fedf819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.508+0000 7fedf7e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 0x7fedf819e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57298/0 (socket says 172.21.15.6:57298) 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.508+0000 7fedf7e00700 1 -- 172.21.15.6:0/1241042070 learned_addr learned my addr 172.21.15.6:0/1241042070 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.508+0000 7fedf7e00700 1 -- 172.21.15.6:0/1241042070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf819ebd0 con 0x7fedf810a770 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.508+0000 7fedf7e00700 1 --2- 172.21.15.6:0/1241042070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 0x7fedf819e690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fede80117b0 tx=0x7fede8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.509+0000 7fedf4c00700 1 -- 172.21.15.6:0/1241042070 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fede8003af0 con 0x7fedf810a770 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.509+0000 7fedf4c00700 1 -- 172.21.15.6:0/1241042070 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fede8020070 con 0x7fedf810a770 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.509+0000 7fedfee00700 1 -- 172.21.15.6:0/1241042070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf819ee60 con 0x7fedf810a770 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.509+0000 7fedf4c00700 1 -- 172.21.15.6:0/1241042070 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fede8019630 con 0x7fedf810a770 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.509+0000 7fedfee00700 1 -- 172.21.15.6:0/1241042070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf81a1b50 con 0x7fedf810a770 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.509+0000 7fedf4c00700 1 -- 172.21.15.6:0/1241042070 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fede8019c30 con 0x7fedf810a770 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.509+0000 7fedf4c00700 1 -- 172.21.15.6:0/1241042070 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fede800b720 con 0x7fedf810a770 2024-04-26T01:32:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.510+0000 7fedfee00700 1 -- 172.21.15.6:0/1241042070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedf80640a0 con 0x7fedf810a770 2024-04-26T01:32:56.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.513+0000 7fedf4c00700 1 -- 172.21.15.6:0/1241042070 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fede80042c0 con 0x7fedf810a770 2024-04-26T01:32:56.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.592+0000 7fedfee00700 1 -- 172.21.15.6:0/1241042070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fedf81a20f0 con 0x7fedf810a770 2024-04-26T01:32:56.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.593+0000 7fedf4c00700 1 -- 172.21.15.6:0/1241042070 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7fede801a9c0 con 0x7fedf810a770 2024-04-26T01:32:56.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.594+0000 7fedfee00700 1 -- 172.21.15.6:0/1241042070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 msgr2=0x7fedf819e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:56.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.594+0000 7fedfee00700 1 --2- 172.21.15.6:0/1241042070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 0x7fedf819e690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fede80117b0 tx=0x7fede8011b70 comp rx=0 tx=0).stop 2024-04-26T01:32:56.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.594+0000 7fedfee00700 1 -- 172.21.15.6:0/1241042070 shutdown_connections 2024-04-26T01:32:56.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.594+0000 7fedfee00700 1 --2- 172.21.15.6:0/1241042070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf810a770 0x7fedf819e690 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:56.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.594+0000 7fedfee00700 1 -- 172.21.15.6:0/1241042070 >> 172.21.15.6:0/1241042070 conn(0x7fedf807db60 msgr2=0x7fedf8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:56.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.595+0000 7fedfee00700 1 -- 172.21.15.6:0/1241042070 shutdown_connections 2024-04-26T01:32:56.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.595+0000 7fedfee00700 1 -- 172.21.15.6:0/1241042070 wait complete. 2024-04-26T01:32:56.251 INFO:teuthology.orchestra.run.smithi006.stdout:mgr not available, waiting (3/15)... 2024-04-26T01:32:57.208 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:56 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:56.902+0000 7f34f103e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:32:57.563 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:32:57.213+0000 7f34f103e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:32:57.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: Activating manager daemon x 2024-04-26T01:32:57.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: mgrmap e2: x(active, starting, since 0.0111899s) 2024-04-26T01:32:57.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:32:57.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:32:57.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:32:57.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:32:57.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:32:57.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:32:57.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:32:57.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:32:57.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: Manager daemon x is now available 2024-04-26T01:32:57.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:32:57.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:32:57.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:57 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' 2024-04-26T01:32:58.536 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:58 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' 2024-04-26T01:32:58.536 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:58 smithi006 ceph-mon[28419]: from='mgr.14100 172.21.15.6:0/2354670900' entity='mgr.x' 2024-04-26T01:32:58.536 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:58 smithi006 ceph-mon[28419]: mgrmap e3: x(active, since 1.01936s) 2024-04-26T01:32:59.563 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:32:59 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2599969469' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T01:32:59.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:32:59.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:32:59.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:32:59.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T01:32:59.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T01:32:59.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T01:32:59.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:59.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T01:32:59.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T01:32:59.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T01:32:59.862 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T01:32:59.862 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T01:32:59.862 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T01:32:59.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T01:32:29.072125+0000", 2024-04-26T01:32:59.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.863+0000 7f1513a00700 1 Processor -- start 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.863+0000 7f1513a00700 1 -- start start 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.864+0000 7f1513a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 0x7f150c108960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.864+0000 7f1513a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f150c108f30 con 0x7f150c108540 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.864+0000 7f1511000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 0x7f150c108960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.864+0000 7f1511000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 0x7f150c108960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42596/0 (socket says 172.21.15.6:42596) 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.864+0000 7f1511000700 1 -- 172.21.15.6:0/1286985528 learned_addr learned my addr 172.21.15.6:0/1286985528 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.865+0000 7f1511000700 1 -- 172.21.15.6:0/1286985528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150c1096f0 con 0x7f150c108540 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.865+0000 7f1511000700 1 --2- 172.21.15.6:0/1286985528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 0x7f150c108960 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0098a0 tx=0x7f14fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93304f6d449e60ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.865+0000 7f150be00700 1 -- 172.21.15.6:0/1286985528 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14fc004160 con 0x7f150c108540 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.865+0000 7f150be00700 1 -- 172.21.15.6:0/1286985528 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f14fc004300 con 0x7f150c108540 2024-04-26T01:32:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.865+0000 7f150be00700 1 -- 172.21.15.6:0/1286985528 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14fc011af0 con 0x7f150c108540 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.866+0000 7f1513a00700 1 -- 172.21.15.6:0/1286985528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 msgr2=0x7f150c108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.866+0000 7f1513a00700 1 --2- 172.21.15.6:0/1286985528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 0x7f150c108960 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0098a0 tx=0x7f14fc009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.866+0000 7f1513a00700 1 -- 172.21.15.6:0/1286985528 shutdown_connections 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.866+0000 7f1513a00700 1 --2- 172.21.15.6:0/1286985528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 0x7f150c108960 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.866+0000 7f1513a00700 1 -- 172.21.15.6:0/1286985528 >> 172.21.15.6:0/1286985528 conn(0x7f150c103ae0 msgr2=0x7f150c105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.866+0000 7f1513a00700 1 -- 172.21.15.6:0/1286985528 shutdown_connections 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.866+0000 7f1513a00700 1 -- 172.21.15.6:0/1286985528 wait complete. 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.867+0000 7f1513a00700 1 Processor -- start 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.867+0000 7f1513a00700 1 -- start start 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.868+0000 7f1513a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 0x7f150c19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.868+0000 7f1513a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f150c10caa0 con 0x7f150c108540 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.868+0000 7f1511000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 0x7f150c19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.868+0000 7f1511000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 0x7f150c19e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42604/0 (socket says 172.21.15.6:42604) 2024-04-26T01:32:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.868+0000 7f1511000700 1 -- 172.21.15.6:0/2599969469 learned_addr learned my addr 172.21.15.6:0/2599969469 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.869+0000 7f1511000700 1 -- 172.21.15.6:0/2599969469 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150c19ec50 con 0x7f150c108540 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.869+0000 7f1511000700 1 --2- 172.21.15.6:0/2599969469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 0x7f150c19e710 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0117b0 tx=0x7f14fc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.869+0000 7f150a000700 1 -- 172.21.15.6:0/2599969469 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14fc003af0 con 0x7f150c108540 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.869+0000 7f150a000700 1 -- 172.21.15.6:0/2599969469 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f14fc020070 con 0x7f150c108540 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.869+0000 7f1513a00700 1 -- 172.21.15.6:0/2599969469 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f150c19eee0 con 0x7f150c108540 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.869+0000 7f150a000700 1 -- 172.21.15.6:0/2599969469 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14fc019630 con 0x7f150c108540 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.869+0000 7f1513a00700 1 -- 172.21.15.6:0/2599969469 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f150c1a1bd0 con 0x7f150c108540 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.870+0000 7f150a000700 1 -- 172.21.15.6:0/2599969469 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 3) ==== 46441+0+0 (secure 0 0 0) 0x7f14fc01a9c0 con 0x7f150c108540 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.870+0000 7f1513a00700 1 -- 172.21.15.6:0/2599969469 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f150c0640a0 con 0x7f150c108540 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.871+0000 7f150a000700 1 --2- 172.21.15.6:0/2599969469 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f14ec039490 0x7f14ec03b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.871+0000 7f150a000700 1 -- 172.21.15.6:0/2599969469 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f14fc04d8d0 con 0x7f150c108540 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.871+0000 7f1503e00700 1 --2- 172.21.15.6:0/2599969469 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f14ec039490 0x7f14ec03b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.872+0000 7f1503e00700 1 --2- 172.21.15.6:0/2599969469 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f14ec039490 0x7f14ec03b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f14f4006fd0 tx=0x7f14f4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:58.877+0000 7f150a000700 1 -- 172.21.15.6:0/2599969469 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f14fc00e260 con 0x7f150c108540 2024-04-26T01:32:59.867 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:59.199+0000 7f1513a00700 1 -- 172.21.15.6:0/2599969469 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f150c1a1da0 con 0x7f150c108540 2024-04-26T01:32:59.868 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:59.200+0000 7f150a000700 1 -- 172.21.15.6:0/2599969469 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f150c1a1da0 con 0x7f150c108540 2024-04-26T01:32:59.868 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:59.203+0000 7f1513a00700 1 -- 172.21.15.6:0/2599969469 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f14ec039490 msgr2=0x7f14ec03b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:59.868 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:59.203+0000 7f1513a00700 1 --2- 172.21.15.6:0/2599969469 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f14ec039490 0x7f14ec03b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f14f4006fd0 tx=0x7f14f4006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:59.868 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:59.203+0000 7f1513a00700 1 -- 172.21.15.6:0/2599969469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 msgr2=0x7f150c19e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:59.868 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:59.203+0000 7f1513a00700 1 --2- 172.21.15.6:0/2599969469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 0x7f150c19e710 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0117b0 tx=0x7f14fc011b70 comp rx=0 tx=0).stop 2024-04-26T01:32:59.868 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:59.203+0000 7f1513a00700 1 -- 172.21.15.6:0/2599969469 shutdown_connections 2024-04-26T01:32:59.868 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:59.203+0000 7f1513a00700 1 --2- 172.21.15.6:0/2599969469 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f14ec039490 0x7f14ec03b950 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:59.868 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:59.203+0000 7f1513a00700 1 --2- 172.21.15.6:0/2599969469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f150c108540 0x7f150c19e710 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:59.868 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:59.203+0000 7f1513a00700 1 -- 172.21.15.6:0/2599969469 >> 172.21.15.6:0/2599969469 conn(0x7f150c103ae0 msgr2=0x7f150c104630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:59.868 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:59.204+0000 7f1513a00700 1 -- 172.21.15.6:0/2599969469 shutdown_connections 2024-04-26T01:32:59.868 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:59.204+0000 7f1513a00700 1 -- 172.21.15.6:0/2599969469 wait complete. 2024-04-26T01:32:59.868 INFO:teuthology.orchestra.run.smithi006.stdout:mgr is available 2024-04-26T01:33:00.458 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:00 smithi006 ceph-mon[28419]: mgrmap e4: x(active, since 2s) 2024-04-26T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:01 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/787842491' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T01:33:01.392 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:33:01.392 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T01:33:01.392 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout fsid = a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:01.392 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T01:33:01.392 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T01:33:01.392 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T01:33:01.392 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T01:33:01.392 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T01:33:01.392 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:33:01.392 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T01:33:01.392 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T01:33:01.393 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-04-26T01:33:01.393 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T01:33:01.393 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-26T01:33:01.393 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T01:33:01.395 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T01:33:01.395 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T01:33:01.395 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T01:33:01.395 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T01:33:01.395 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.463+0000 7f57f6800700 1 Processor -- start 2024-04-26T01:33:01.395 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.463+0000 7f57f6800700 1 -- start start 2024-04-26T01:33:01.395 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.464+0000 7f57f6800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 0x7f57f0110640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:01.396 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.464+0000 7f57f6800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0110c10 con 0x7f57f0110220 2024-04-26T01:33:01.396 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.464+0000 7f57ef400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 0x7f57f0110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:01.396 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.464+0000 7f57ef400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 0x7f57f0110640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42610/0 (socket says 172.21.15.6:42610) 2024-04-26T01:33:01.396 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.464+0000 7f57ef400700 1 -- 172.21.15.6:0/602804759 learned_addr learned my addr 172.21.15.6:0/602804759 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:01.396 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.465+0000 7f57ef400700 1 -- 172.21.15.6:0/602804759 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f010ece0 con 0x7f57f0110220 2024-04-26T01:33:01.396 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.465+0000 7f57ef400700 1 --2- 172.21.15.6:0/602804759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 0x7f57f0110640 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f57e00098a0 tx=0x7f57e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6061794fbd426a96 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:01.396 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.466+0000 7f57ee000700 1 -- 172.21.15.6:0/602804759 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57e0004160 con 0x7f57f0110220 2024-04-26T01:33:01.396 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.466+0000 7f57ee000700 1 -- 172.21.15.6:0/602804759 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f57e0004300 con 0x7f57f0110220 2024-04-26T01:33:01.396 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.466+0000 7f57ee000700 1 -- 172.21.15.6:0/602804759 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57e0011af0 con 0x7f57f0110220 2024-04-26T01:33:01.396 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.467+0000 7f57f6800700 1 -- 172.21.15.6:0/602804759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 msgr2=0x7f57f0110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:01.396 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.467+0000 7f57f6800700 1 --2- 172.21.15.6:0/602804759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 0x7f57f0110640 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f57e00098a0 tx=0x7f57e0009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:01.396 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.467+0000 7f57f6800700 1 -- 172.21.15.6:0/602804759 shutdown_connections 2024-04-26T01:33:01.397 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.467+0000 7f57f6800700 1 --2- 172.21.15.6:0/602804759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 0x7f57f0110640 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:01.397 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.467+0000 7f57f6800700 1 -- 172.21.15.6:0/602804759 >> 172.21.15.6:0/602804759 conn(0x7f57f007de20 msgr2=0x7f57f0080240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:01.401 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.467+0000 7f57f6800700 1 -- 172.21.15.6:0/602804759 shutdown_connections 2024-04-26T01:33:01.401 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.467+0000 7f57f6800700 1 -- 172.21.15.6:0/602804759 wait complete. 2024-04-26T01:33:01.401 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.468+0000 7f57f6800700 1 Processor -- start 2024-04-26T01:33:01.401 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.468+0000 7f57f6800700 1 -- start start 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.469+0000 7f57f6800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 0x7f57f01a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.469+0000 7f57f6800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0113140 con 0x7f57f0110220 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.469+0000 7f57ef400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 0x7f57f01a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.469+0000 7f57ef400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 0x7f57f01a2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42620/0 (socket says 172.21.15.6:42620) 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.469+0000 7f57ef400700 1 -- 172.21.15.6:0/787842491 learned_addr learned my addr 172.21.15.6:0/787842491 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.469+0000 7f57ef400700 1 -- 172.21.15.6:0/787842491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f01a3160 con 0x7f57f0110220 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.470+0000 7f57ef400700 1 --2- 172.21.15.6:0/787842491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 0x7f57f01a2c20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f57e00117b0 tx=0x7f57e0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.470+0000 7f57e7e00700 1 -- 172.21.15.6:0/787842491 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57e0003af0 con 0x7f57f0110220 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.470+0000 7f57e7e00700 1 -- 172.21.15.6:0/787842491 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f57e0020070 con 0x7f57f0110220 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.470+0000 7f57e7e00700 1 -- 172.21.15.6:0/787842491 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57e0019630 con 0x7f57f0110220 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.470+0000 7f57f6800700 1 -- 172.21.15.6:0/787842491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f01a33f0 con 0x7f57f0110220 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.470+0000 7f57f6800700 1 -- 172.21.15.6:0/787842491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f01a60e0 con 0x7f57f0110220 2024-04-26T01:33:01.402 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.471+0000 7f57e7e00700 1 -- 172.21.15.6:0/787842491 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f57e001a9c0 con 0x7f57f0110220 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.471+0000 7f57f6800700 1 -- 172.21.15.6:0/787842491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57f00640a0 con 0x7f57f0110220 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.472+0000 7f57e7e00700 1 --2- 172.21.15.6:0/787842491 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f57d40395b0 0x7f57d403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.472+0000 7f57e7e00700 1 -- 172.21.15.6:0/787842491 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f57e004dae0 con 0x7f57f0110220 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.472+0000 7f57eea00700 1 --2- 172.21.15.6:0/787842491 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f57d40395b0 0x7f57d403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.473+0000 7f57eea00700 1 --2- 172.21.15.6:0/787842491 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f57d40395b0 0x7f57d403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f57dc006fd0 tx=0x7f57dc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.477+0000 7f57e7e00700 1 -- 172.21.15.6:0/787842491 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f57e000e260 con 0x7f57f0110220 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.707+0000 7f57f6800700 1 -- 172.21.15.6:0/787842491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f57f01a7780 con 0x7f57f0110220 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.708+0000 7f57e7e00700 1 -- 172.21.15.6:0/787842491 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+436 (secure 0 0 0) 0x7f57e004d500 con 0x7f57f0110220 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.712+0000 7f57f6800700 1 -- 172.21.15.6:0/787842491 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f57d40395b0 msgr2=0x7f57d403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.712+0000 7f57f6800700 1 --2- 172.21.15.6:0/787842491 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f57d40395b0 0x7f57d403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f57dc006fd0 tx=0x7f57dc006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.712+0000 7f57f6800700 1 -- 172.21.15.6:0/787842491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 msgr2=0x7f57f01a2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.712+0000 7f57f6800700 1 --2- 172.21.15.6:0/787842491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 0x7f57f01a2c20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f57e00117b0 tx=0x7f57e0011b70 comp rx=0 tx=0).stop 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.712+0000 7f57f6800700 1 -- 172.21.15.6:0/787842491 shutdown_connections 2024-04-26T01:33:01.403 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.712+0000 7f57f6800700 1 --2- 172.21.15.6:0/787842491 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f57d40395b0 0x7f57d403ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:01.404 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.712+0000 7f57f6800700 1 --2- 172.21.15.6:0/787842491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57f0110220 0x7f57f01a2c20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:01.404 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.712+0000 7f57f6800700 1 -- 172.21.15.6:0/787842491 >> 172.21.15.6:0/787842491 conn(0x7f57f007de20 msgr2=0x7f57f007f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:01.404 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.712+0000 7f57f6800700 1 -- 172.21.15.6:0/787842491 shutdown_connections 2024-04-26T01:33:01.404 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:00.712+0000 7f57f6800700 1 -- 172.21.15.6:0/787842491 wait complete. 2024-04-26T01:33:01.404 INFO:teuthology.orchestra.run.smithi006.stdout:Enabling cephadm module... 2024-04-26T01:33:02.685 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:02 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1790667076' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T01:33:02.686 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:02 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: ignoring --setuser ceph since I am not root 2024-04-26T01:33:02.686 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:02 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: ignoring --setgroup ceph since I am not root 2024-04-26T01:33:02.686 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:02 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:02.690+0000 7f610be40200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:33:03.063 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:02 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:02.938+0000 7f610be40200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:33:03.078 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.024+0000 7f5a2dc00700 1 Processor -- start 2024-04-26T01:33:03.084 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.024+0000 7f5a2dc00700 1 -- start start 2024-04-26T01:33:03.084 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.025+0000 7f5a2dc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 0x7f5a28108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:03.084 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.025+0000 7f5a2dc00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5a28108f20 con 0x7f5a28108530 2024-04-26T01:33:03.084 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.025+0000 7f5a26a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 0x7f5a28108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:03.084 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.025+0000 7f5a26a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 0x7f5a28108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42628/0 (socket says 172.21.15.6:42628) 2024-04-26T01:33:03.084 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.025+0000 7f5a26a00700 1 -- 172.21.15.6:0/3832211276 learned_addr learned my addr 172.21.15.6:0/3832211276 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:03.084 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.026+0000 7f5a26a00700 1 -- 172.21.15.6:0/3832211276 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a28109730 con 0x7f5a28108530 2024-04-26T01:33:03.084 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.026+0000 7f5a26a00700 1 --2- 172.21.15.6:0/3832211276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 0x7f5a28108950 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5a180098a0 tx=0x7f5a18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3aba4e16532686 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:03.084 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.026+0000 7f5a25600700 1 -- 172.21.15.6:0/3832211276 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a18004160 con 0x7f5a28108530 2024-04-26T01:33:03.084 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.026+0000 7f5a25600700 1 -- 172.21.15.6:0/3832211276 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5a18004300 con 0x7f5a28108530 2024-04-26T01:33:03.084 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.026+0000 7f5a25600700 1 -- 172.21.15.6:0/3832211276 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a18011af0 con 0x7f5a28108530 2024-04-26T01:33:03.084 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.027+0000 7f5a2dc00700 1 -- 172.21.15.6:0/3832211276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 msgr2=0x7f5a28108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.027+0000 7f5a2dc00700 1 --2- 172.21.15.6:0/3832211276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 0x7f5a28108950 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5a180098a0 tx=0x7f5a18009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.027+0000 7f5a2dc00700 1 -- 172.21.15.6:0/3832211276 shutdown_connections 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.027+0000 7f5a2dc00700 1 --2- 172.21.15.6:0/3832211276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 0x7f5a28108950 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.027+0000 7f5a2dc00700 1 -- 172.21.15.6:0/3832211276 >> 172.21.15.6:0/3832211276 conn(0x7f5a28103b50 msgr2=0x7f5a28105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.027+0000 7f5a2dc00700 1 -- 172.21.15.6:0/3832211276 shutdown_connections 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.027+0000 7f5a2dc00700 1 -- 172.21.15.6:0/3832211276 wait complete. 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.028+0000 7f5a2dc00700 1 Processor -- start 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.028+0000 7f5a2dc00700 1 -- start start 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.029+0000 7f5a2dc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 0x7f5a2807ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.029+0000 7f5a2dc00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2810cae0 con 0x7f5a28108530 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.029+0000 7f5a26a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 0x7f5a2807ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.029+0000 7f5a26a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 0x7f5a2807ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42638/0 (socket says 172.21.15.6:42638) 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.029+0000 7f5a26a00700 1 -- 172.21.15.6:0/1790667076 learned_addr learned my addr 172.21.15.6:0/1790667076 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.029+0000 7f5a26a00700 1 -- 172.21.15.6:0/1790667076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a2807cfb0 con 0x7f5a28108530 2024-04-26T01:33:03.085 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.030+0000 7f5a26a00700 1 --2- 172.21.15.6:0/1790667076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 0x7f5a2807ca70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a18000c00 tx=0x7f5a18011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.030+0000 7f5a1f400700 1 -- 172.21.15.6:0/1790667076 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a18003b20 con 0x7f5a28108530 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.030+0000 7f5a1f400700 1 -- 172.21.15.6:0/1790667076 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5a1801a8c0 con 0x7f5a28108530 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.030+0000 7f5a1f400700 1 -- 172.21.15.6:0/1790667076 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a18024940 con 0x7f5a28108530 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.030+0000 7f5a2dc00700 1 -- 172.21.15.6:0/1790667076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a2807b1a0 con 0x7f5a28108530 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.030+0000 7f5a2dc00700 1 -- 172.21.15.6:0/1790667076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a2807b6e0 con 0x7f5a28108530 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.031+0000 7f5a1f400700 1 -- 172.21.15.6:0/1790667076 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f5a18022020 con 0x7f5a28108530 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.031+0000 7f5a2dc00700 1 -- 172.21.15.6:0/1790667076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a280640a0 con 0x7f5a28108530 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.032+0000 7f5a1f400700 1 --2- 172.21.15.6:0/1790667076 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f5a0c0395b0 0x7f5a0c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.032+0000 7f5a1f400700 1 -- 172.21.15.6:0/1790667076 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5a1804d490 con 0x7f5a28108530 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.032+0000 7f5a26000700 1 --2- 172.21.15.6:0/1790667076 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f5a0c0395b0 0x7f5a0c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.033+0000 7f5a26000700 1 --2- 172.21.15.6:0/1790667076 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f5a0c0395b0 0x7f5a0c03ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a14006fd0 tx=0x7f5a14006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.037+0000 7f5a1f400700 1 -- 172.21.15.6:0/1790667076 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5a1804d1f0 con 0x7f5a28108530 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.334+0000 7f5a2dc00700 1 -- 172.21.15.6:0/1790667076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f5a2807b960 con 0x7f5a28108530 2024-04-26T01:33:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.388+0000 7f5a1f400700 1 -- 172.21.15.6:0/1790667076 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f5a1802ad60 con 0x7f5a28108530 2024-04-26T01:33:03.087 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.392+0000 7f5a1f400700 1 -- 172.21.15.6:0/1790667076 <== mon.0 v2:172.21.15.6:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f5a1801aa70 con 0x7f5a28108530 2024-04-26T01:33:03.087 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.393+0000 7f5a2dc00700 1 -- 172.21.15.6:0/1790667076 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f5a0c0395b0 msgr2=0x7f5a0c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:03.087 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.393+0000 7f5a2dc00700 1 --2- 172.21.15.6:0/1790667076 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f5a0c0395b0 0x7f5a0c03ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a14006fd0 tx=0x7f5a14006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:03.087 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.393+0000 7f5a2dc00700 1 -- 172.21.15.6:0/1790667076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 msgr2=0x7f5a2807ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:03.087 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.393+0000 7f5a2dc00700 1 --2- 172.21.15.6:0/1790667076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 0x7f5a2807ca70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a18000c00 tx=0x7f5a18011b70 comp rx=0 tx=0).stop 2024-04-26T01:33:03.087 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.394+0000 7f5a2dc00700 1 -- 172.21.15.6:0/1790667076 shutdown_connections 2024-04-26T01:33:03.087 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.394+0000 7f5a2dc00700 1 --2- 172.21.15.6:0/1790667076 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f5a0c0395b0 0x7f5a0c03ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:03.087 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.394+0000 7f5a2dc00700 1 --2- 172.21.15.6:0/1790667076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a28108530 0x7f5a2807ca70 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a18000c00 tx=0x7f5a18011b70 comp rx=0 tx=0).stop 2024-04-26T01:33:03.087 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.394+0000 7f5a2dc00700 1 -- 172.21.15.6:0/1790667076 >> 172.21.15.6:0/1790667076 conn(0x7f5a28103b50 msgr2=0x7f5a281045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:03.087 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.394+0000 7f5a2dc00700 1 -- 172.21.15.6:0/1790667076 shutdown_connections 2024-04-26T01:33:03.087 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:02.394+0000 7f5a2dc00700 1 -- 172.21.15.6:0/1790667076 wait complete. 2024-04-26T01:33:03.375 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:03 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:03.073+0000 7f610be40200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:33:03.658 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:03 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1790667076' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T01:33:03.658 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:03 smithi006 ceph-mon[28419]: mgrmap e5: x(active, since 5s) 2024-04-26T01:33:03.658 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:03 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:03.498+0000 7f610be40200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:33:03.658 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:03 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:03.616+0000 7f610be40200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:33:04.705 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:04 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/395959892' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T01:33:04.718 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:33:04.719 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T01:33:04.719 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T01:33:04.719 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T01:33:04.719 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T01:33:04.719 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:33:04.719 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.722+0000 7fea65a00700 1 Processor -- start 2024-04-26T01:33:04.719 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.722+0000 7fea65a00700 1 -- start start 2024-04-26T01:33:04.719 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.723+0000 7fea65a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 0x7fea60107260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:04.719 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.723+0000 7fea65a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fea60063cc0 con 0x7fea60106e40 2024-04-26T01:33:04.719 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.723+0000 7fea5ea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 0x7fea60107260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:04.719 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.723+0000 7fea5ea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 0x7fea60107260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42666/0 (socket says 172.21.15.6:42666) 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.723+0000 7fea5ea00700 1 -- 172.21.15.6:0/113294686 learned_addr learned my addr 172.21.15.6:0/113294686 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.724+0000 7fea5ea00700 1 -- 172.21.15.6:0/113294686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea601077a0 con 0x7fea60106e40 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.724+0000 7fea5ea00700 1 --2- 172.21.15.6:0/113294686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 0x7fea60107260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fea50009e80 tx=0x7fea5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66fad55d9fb23767 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.724+0000 7fea5d600700 1 -- 172.21.15.6:0/113294686 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea50017070 con 0x7fea60106e40 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.724+0000 7fea5d600700 1 -- 172.21.15.6:0/113294686 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fea5000b8c0 con 0x7fea60106e40 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.724+0000 7fea5d600700 1 -- 172.21.15.6:0/113294686 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea50004490 con 0x7fea60106e40 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.725+0000 7fea65a00700 1 -- 172.21.15.6:0/113294686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 msgr2=0x7fea60107260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.725+0000 7fea65a00700 1 --2- 172.21.15.6:0/113294686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 0x7fea60107260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fea50009e80 tx=0x7fea5000b330 comp rx=0 tx=0).stop 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.725+0000 7fea65a00700 1 -- 172.21.15.6:0/113294686 shutdown_connections 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.725+0000 7fea65a00700 1 --2- 172.21.15.6:0/113294686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 0x7fea60107260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.725+0000 7fea65a00700 1 -- 172.21.15.6:0/113294686 >> 172.21.15.6:0/113294686 conn(0x7fea60102a40 msgr2=0x7fea60104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.725+0000 7fea65a00700 1 -- 172.21.15.6:0/113294686 shutdown_connections 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.725+0000 7fea65a00700 1 -- 172.21.15.6:0/113294686 wait complete. 2024-04-26T01:33:04.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.726+0000 7fea65a00700 1 Processor -- start 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.726+0000 7fea65a00700 1 -- start start 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.727+0000 7fea65a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 0x7fea6019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.727+0000 7fea65a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fea601087d0 con 0x7fea60106e40 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.727+0000 7fea5ea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 0x7fea6019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.727+0000 7fea5ea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 0x7fea6019e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42678/0 (socket says 172.21.15.6:42678) 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.727+0000 7fea5ea00700 1 -- 172.21.15.6:0/395959892 learned_addr learned my addr 172.21.15.6:0/395959892 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.727+0000 7fea5ea00700 1 -- 172.21.15.6:0/395959892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6019eb50 con 0x7fea60106e40 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.728+0000 7fea5ea00700 1 --2- 172.21.15.6:0/395959892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 0x7fea6019e610 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fea5000b7b0 tx=0x7fea5000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.728+0000 7fea57400700 1 -- 172.21.15.6:0/395959892 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea50020020 con 0x7fea60106e40 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.728+0000 7fea57400700 1 -- 172.21.15.6:0/395959892 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fea50003d10 con 0x7fea60106e40 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.728+0000 7fea57400700 1 -- 172.21.15.6:0/395959892 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea50017040 con 0x7fea60106e40 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.728+0000 7fea65a00700 1 -- 172.21.15.6:0/395959892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea6019ede0 con 0x7fea60106e40 2024-04-26T01:33:04.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.728+0000 7fea65a00700 1 -- 172.21.15.6:0/395959892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea601a1ad0 con 0x7fea60106e40 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.729+0000 7fea65a00700 1 -- 172.21.15.6:0/395959892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea600640a0 con 0x7fea60106e40 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.730+0000 7fea57400700 1 -- 172.21.15.6:0/395959892 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fea50023020 con 0x7fea60106e40 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.730+0000 7fea57400700 1 --2- 172.21.15.6:0/395959892 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7fea440395b0 0x7fea4403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.730+0000 7fea57400700 1 -- 172.21.15.6:0/395959892 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fea5000f050 con 0x7fea60106e40 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.730+0000 7fea5e000700 1 -- 172.21.15.6:0/395959892 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7fea440395b0 msgr2=0x7fea4403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/801052348 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.730+0000 7fea5e000700 1 --2- 172.21.15.6:0/395959892 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7fea440395b0 0x7fea4403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.735+0000 7fea57400700 1 -- 172.21.15.6:0/395959892 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea5002a020 con 0x7fea60106e40 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.931+0000 7fea5e000700 1 -- 172.21.15.6:0/395959892 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7fea440395b0 msgr2=0x7fea4403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/801052348 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:03.931+0000 7fea5e000700 1 --2- 172.21.15.6:0/395959892 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7fea440395b0 0x7fea4403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:04.023+0000 7fea65a00700 1 -- 172.21.15.6:0/395959892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fea6002e250 con 0x7fea60106e40 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:04.024+0000 7fea57400700 1 -- 172.21.15.6:0/395959892 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fea5002a200 con 0x7fea60106e40 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:04.026+0000 7fea65a00700 1 -- 172.21.15.6:0/395959892 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7fea440395b0 msgr2=0x7fea4403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:33:04.722 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:04.026+0000 7fea65a00700 1 --2- 172.21.15.6:0/395959892 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7fea440395b0 0x7fea4403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:04.723 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:04.026+0000 7fea65a00700 1 -- 172.21.15.6:0/395959892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 msgr2=0x7fea6019e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:04.723 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:04.026+0000 7fea65a00700 1 --2- 172.21.15.6:0/395959892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 0x7fea6019e610 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fea5000b7b0 tx=0x7fea5000be90 comp rx=0 tx=0).stop 2024-04-26T01:33:04.723 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:04.027+0000 7fea65a00700 1 -- 172.21.15.6:0/395959892 shutdown_connections 2024-04-26T01:33:04.723 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:04.027+0000 7fea65a00700 1 --2- 172.21.15.6:0/395959892 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7fea440395b0 0x7fea4403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:04.723 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:04.027+0000 7fea65a00700 1 --2- 172.21.15.6:0/395959892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea60106e40 0x7fea6019e610 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:04.723 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:04.027+0000 7fea65a00700 1 -- 172.21.15.6:0/395959892 >> 172.21.15.6:0/395959892 conn(0x7fea60102a40 msgr2=0x7fea601952b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:04.723 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:04.027+0000 7fea65a00700 1 -- 172.21.15.6:0/395959892 shutdown_connections 2024-04-26T01:33:04.723 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:04.027+0000 7fea65a00700 1 -- 172.21.15.6:0/395959892 wait complete. 2024-04-26T01:33:04.723 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for the mgr to restart... 2024-04-26T01:33:04.723 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr epoch 5... 2024-04-26T01:33:05.301 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:05 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:05.219+0000 7f610be40200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:33:05.563 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:05 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:05.454+0000 7f610be40200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:33:06.059 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:05 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:05.671+0000 7f610be40200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:33:06.313 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:06 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:06.064+0000 7f610be40200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:33:06.313 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:06 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:06.188+0000 7f610be40200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:33:06.700 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:06 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:06.444+0000 7f610be40200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:33:07.063 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:06 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:06.705+0000 7f610be40200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:33:07.813 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:07 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:07.508+0000 7f610be40200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:33:08.480 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:08 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:08.078+0000 7f610be40200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:33:08.480 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:08 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:08.209+0000 7f610be40200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:33:08.480 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:08 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:08.485+0000 7f610be40200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:33:09.563 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:09 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:09.156+0000 7f610be40200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:33:09.564 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:09 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:09.282+0000 7f610be40200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:33:09.919 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:09 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:09.610+0000 7f610be40200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:33:09.919 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:09 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:09.793+0000 7f610be40200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:33:09.919 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:09 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:09.923+0000 7f610be40200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:33:10.313 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:10 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:10.077+0000 7f610be40200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:33:10.813 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:10 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:10.458+0000 7f610be40200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:33:11.518 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:11.239+0000 7f610be40200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:33:11.813 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:11.531+0000 7f610be40200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:33:11.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-mon[28419]: Active manager daemon x restarted 2024-04-26T01:33:11.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-mon[28419]: Activating manager daemon x 2024-04-26T01:33:11.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-mon[28419]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T01:33:11.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-mon[28419]: mgrmap e6: x(active, starting, since 0.0109565s) 2024-04-26T01:33:11.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:33:11.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:33:11.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:33:11.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:33:11.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:33:11.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-mon[28419]: Manager daemon x is now available 2024-04-26T01:33:11.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:11 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:12.813 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:12 smithi006 ceph-mon[28419]: Found migration_current of "None". Setting to last migration. 2024-04-26T01:33:12.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:12 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:12.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:12 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:12.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:12 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:12.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:12 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:33:12.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:12 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:33:12.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:12 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:12.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:12 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:12.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:12 smithi006 ceph-mon[28419]: mgrmap e7: x(active, since 1.01893s) 2024-04-26T01:33:13.309 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:33:13.309 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T01:33:13.309 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T01:33:13.309 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:33:13.309 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.306+0000 7f1817200700 1 Processor -- start 2024-04-26T01:33:13.309 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.306+0000 7f1817200700 1 -- start start 2024-04-26T01:33:13.309 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.307+0000 7f1817200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 0x7f181007c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:13.309 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.307+0000 7f1817200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f181007c8e0 con 0x7f181007de60 2024-04-26T01:33:13.309 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.307+0000 7f180fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 0x7f181007c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:13.309 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.307+0000 7f180fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 0x7f181007c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42684/0 (socket says 172.21.15.6:42684) 2024-04-26T01:33:13.309 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.307+0000 7f180fe00700 1 -- 172.21.15.6:0/462407094 learned_addr learned my addr 172.21.15.6:0/462407094 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:13.309 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.308+0000 7f180fe00700 1 -- 172.21.15.6:0/462407094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181007ca60 con 0x7f181007de60 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.308+0000 7f180fe00700 1 --2- 172.21.15.6:0/462407094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 0x7f181007c310 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f18000098a0 tx=0x7f1800009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44092d225cfb361 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.309+0000 7f180ea00700 1 -- 172.21.15.6:0/462407094 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1800004160 con 0x7f181007de60 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.309+0000 7f180ea00700 1 -- 172.21.15.6:0/462407094 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1800004300 con 0x7f181007de60 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.309+0000 7f180ea00700 1 -- 172.21.15.6:0/462407094 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1800011af0 con 0x7f181007de60 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.309+0000 7f1817200700 1 -- 172.21.15.6:0/462407094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 msgr2=0x7f181007c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.309+0000 7f1817200700 1 --2- 172.21.15.6:0/462407094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 0x7f181007c310 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f18000098a0 tx=0x7f1800009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.310+0000 7f1817200700 1 -- 172.21.15.6:0/462407094 shutdown_connections 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.310+0000 7f1817200700 1 --2- 172.21.15.6:0/462407094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 0x7f181007c310 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.310+0000 7f1817200700 1 -- 172.21.15.6:0/462407094 >> 172.21.15.6:0/462407094 conn(0x7f1810103b40 msgr2=0x7f1810105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.310+0000 7f1817200700 1 -- 172.21.15.6:0/462407094 shutdown_connections 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.310+0000 7f1817200700 1 -- 172.21.15.6:0/462407094 wait complete. 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.311+0000 7f1817200700 1 Processor -- start 2024-04-26T01:33:13.310 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.311+0000 7f1817200700 1 -- start start 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.311+0000 7f1817200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 0x7f181019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.311+0000 7f1817200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f181010eab0 con 0x7f181007de60 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.311+0000 7f180fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 0x7f181019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.311+0000 7f180fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 0x7f181019e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42694/0 (socket says 172.21.15.6:42694) 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.311+0000 7f180fe00700 1 -- 172.21.15.6:0/911112522 learned_addr learned my addr 172.21.15.6:0/911112522 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.312+0000 7f180fe00700 1 -- 172.21.15.6:0/911112522 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181019ec00 con 0x7f181007de60 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.312+0000 7f180fe00700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 0x7f181019e6c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1800011ac0 tx=0x7f1800011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.312+0000 7f180cc00700 1 -- 172.21.15.6:0/911112522 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1800003970 con 0x7f181007de60 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.312+0000 7f180cc00700 1 -- 172.21.15.6:0/911112522 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1800003b10 con 0x7f181007de60 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.312+0000 7f1817200700 1 -- 172.21.15.6:0/911112522 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f181019ee90 con 0x7f181007de60 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.313+0000 7f180cc00700 1 -- 172.21.15.6:0/911112522 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1800024610 con 0x7f181007de60 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.313+0000 7f1817200700 1 -- 172.21.15.6:0/911112522 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18101a1b80 con 0x7f181007de60 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.314+0000 7f180cc00700 1 -- 172.21.15.6:0/911112522 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f180001a430 con 0x7f181007de60 2024-04-26T01:33:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.314+0000 7f180cc00700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 0x7f17f403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:13.312 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.314+0000 7f180cc00700 1 -- 172.21.15.6:0/911112522 --> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f17f403c1d0 con 0x7f17f4039600 2024-04-26T01:33:13.312 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.315+0000 7f180cc00700 1 -- 172.21.15.6:0/911112522 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f180004d8c0 con 0x7f181007de60 2024-04-26T01:33:13.312 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.315+0000 7f180f400700 1 -- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 msgr2=0x7f17f403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/801052348 2024-04-26T01:33:13.312 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.315+0000 7f180f400700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 0x7f17f403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:33:13.312 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.516+0000 7f180f400700 1 -- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 msgr2=0x7f17f403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/801052348 2024-04-26T01:33:13.312 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.516+0000 7f180f400700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 0x7f17f403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:33:13.312 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.917+0000 7f180f400700 1 -- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 msgr2=0x7f17f403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/801052348 2024-04-26T01:33:13.312 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:05.917+0000 7f180f400700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 0x7f17f403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T01:33:13.312 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:06.718+0000 7f180f400700 1 -- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 msgr2=0x7f17f403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/801052348 2024-04-26T01:33:13.312 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:06.718+0000 7f180f400700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 0x7f17f403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T01:33:13.312 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:08.320+0000 7f180f400700 1 -- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 msgr2=0x7f17f403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/801052348 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:08.320+0000 7f180f400700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 0x7f17f403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:11.522+0000 7f180f400700 1 -- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 msgr2=0x7f17f403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/801052348 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:11.522+0000 7f180f400700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 0x7f17f403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:11.540+0000 7f180cc00700 1 -- 172.21.15.6:0/911112522 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f1800024840 con 0x7f181007de60 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:11.541+0000 7f180cc00700 1 -- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 msgr2=0x7f17f403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:11.541+0000 7f180cc00700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 0x7f17f403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.546+0000 7f180cc00700 1 -- 172.21.15.6:0/911112522 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f180004f3d0 con 0x7f181007de60 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.547+0000 7f180cc00700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f17f403cef0 0x7f17f403f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.547+0000 7f180cc00700 1 -- 172.21.15.6:0/911112522 --> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f180002ee50 con 0x7f17f403cef0 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.548+0000 7f180f400700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f17f403cef0 0x7f17f403f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.550+0000 7f180f400700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f17f403cef0 0x7f17f403f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f17fc003a10 tx=0x7f17fc0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.551+0000 7f180cc00700 1 -- 172.21.15.6:0/911112522 <== mgr.14120 v2:172.21.15.6:6800/635066163 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f180002ee50 con 0x7f17f403cef0 2024-04-26T01:33:13.313 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.560+0000 7f1817200700 1 -- 172.21.15.6:0/911112522 --> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f1810069ba0 con 0x7f17f403cef0 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.561+0000 7f180cc00700 1 -- 172.21.15.6:0/911112522 <== mgr.14120 v2:172.21.15.6:6800/635066163 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f1810069ba0 con 0x7f17f403cef0 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.561+0000 7f1817200700 1 -- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f17f403cef0 msgr2=0x7f17f403f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.561+0000 7f1817200700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f17f403cef0 0x7f17f403f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f17fc003a10 tx=0x7f17fc0092b0 comp rx=0 tx=0).stop 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.561+0000 7f1817200700 1 -- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 msgr2=0x7f181019e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.561+0000 7f1817200700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 0x7f181019e6c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1800011ac0 tx=0x7f1800011ba0 comp rx=0 tx=0).stop 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.561+0000 7f1817200700 1 -- 172.21.15.6:0/911112522 shutdown_connections 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.561+0000 7f1817200700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f17f403cef0 0x7f17f403f2e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.561+0000 7f1817200700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f181007de60 0x7f181019e6c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.561+0000 7f1817200700 1 --2- 172.21.15.6:0/911112522 >> [v2:172.21.15.6:6800/801052348,v1:172.21.15.6:6801/801052348] conn(0x7f17f4039600 0x7f17f403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.561+0000 7f1817200700 1 -- 172.21.15.6:0/911112522 >> 172.21.15.6:0/911112522 conn(0x7f1810103b40 msgr2=0x7f1810104730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.562+0000 7f1817200700 1 -- 172.21.15.6:0/911112522 shutdown_connections 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:12.562+0000 7f1817200700 1 -- 172.21.15.6:0/911112522 wait complete. 2024-04-26T01:33:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:mgr epoch 5 is available 2024-04-26T01:33:13.315 INFO:teuthology.orchestra.run.smithi006.stdout:Setting orchestrator backend to cephadm... 2024-04-26T01:33:13.893 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:13 smithi006 ceph-mon[28419]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T01:33:13.893 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:13 smithi006 ceph-mon[28419]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T01:33:13.893 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:13 smithi006 ceph-mon[28419]: [26/Apr/2024:01:33:13] ENGINE Bus STARTING 2024-04-26T01:33:13.893 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:13 smithi006 ceph-mon[28419]: [26/Apr/2024:01:33:13] ENGINE Serving on https://172.21.15.6:7150 2024-04-26T01:33:13.893 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:13 smithi006 ceph-mon[28419]: [26/Apr/2024:01:33:13] ENGINE Serving on http://172.21.15.6:8765 2024-04-26T01:33:13.893 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:13 smithi006 ceph-mon[28419]: [26/Apr/2024:01:33:13] ENGINE Bus STARTED 2024-04-26T01:33:13.893 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:13 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:14.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:14 smithi006 ceph-mon[28419]: mgrmap e8: x(active, since 2s) 2024-04-26T01:33:14.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:14 smithi006 ceph-mon[28419]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:14.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:14 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:14.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:14 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:15.053 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.897+0000 7f1098000700 1 Processor -- start 2024-04-26T01:33:15.054 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.897+0000 7f1098000700 1 -- start start 2024-04-26T01:33:15.054 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.897+0000 7f1098000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 0x7f1090108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:15.054 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.897+0000 7f1098000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1090108f20 con 0x7f1090108530 2024-04-26T01:33:15.055 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.898+0000 7f1095600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 0x7f1090108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:15.055 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.898+0000 7f1095600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 0x7f1090108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46510/0 (socket says 172.21.15.6:46510) 2024-04-26T01:33:15.056 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.898+0000 7f1095600700 1 -- 172.21.15.6:0/2339706318 learned_addr learned my addr 172.21.15.6:0/2339706318 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:15.056 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.899+0000 7f1095600700 1 -- 172.21.15.6:0/2339706318 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1090109730 con 0x7f1090108530 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.899+0000 7f1095600700 1 --2- 172.21.15.6:0/2339706318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 0x7f1090108950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1084009e80 tx=0x7f108400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9145bd40ae0316b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.899+0000 7f108fe00700 1 -- 172.21.15.6:0/2339706318 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1084017070 con 0x7f1090108530 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.899+0000 7f108fe00700 1 -- 172.21.15.6:0/2339706318 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f108400b8c0 con 0x7f1090108530 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.899+0000 7f108fe00700 1 -- 172.21.15.6:0/2339706318 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1084004490 con 0x7f1090108530 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.900+0000 7f1098000700 1 -- 172.21.15.6:0/2339706318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 msgr2=0x7f1090108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.900+0000 7f1098000700 1 --2- 172.21.15.6:0/2339706318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 0x7f1090108950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1084009e80 tx=0x7f108400b330 comp rx=0 tx=0).stop 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.901+0000 7f1098000700 1 -- 172.21.15.6:0/2339706318 shutdown_connections 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.901+0000 7f1098000700 1 --2- 172.21.15.6:0/2339706318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 0x7f1090108950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.901+0000 7f1098000700 1 -- 172.21.15.6:0/2339706318 >> 172.21.15.6:0/2339706318 conn(0x7f1090103b50 msgr2=0x7f1090105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.901+0000 7f1098000700 1 -- 172.21.15.6:0/2339706318 shutdown_connections 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.901+0000 7f1098000700 1 -- 172.21.15.6:0/2339706318 wait complete. 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.902+0000 7f1098000700 1 Processor -- start 2024-04-26T01:33:15.057 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.902+0000 7f1098000700 1 -- start start 2024-04-26T01:33:15.061 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.902+0000 7f1098000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 0x7f109019c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:15.061 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.902+0000 7f1098000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f109010cae0 con 0x7f1090108530 2024-04-26T01:33:15.061 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.903+0000 7f1095600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 0x7f109019c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:15.061 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.903+0000 7f1095600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 0x7f109019c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46514/0 (socket says 172.21.15.6:46514) 2024-04-26T01:33:15.061 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.903+0000 7f1095600700 1 -- 172.21.15.6:0/1975824635 learned_addr learned my addr 172.21.15.6:0/1975824635 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:15.061 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.903+0000 7f1095600700 1 -- 172.21.15.6:0/1975824635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109019c9a0 con 0x7f1090108530 2024-04-26T01:33:15.061 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.903+0000 7f1095600700 1 --2- 172.21.15.6:0/1975824635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 0x7f109019c460 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1084004a60 tx=0x7f108400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.904+0000 7f108e000700 1 -- 172.21.15.6:0/1975824635 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1084020020 con 0x7f1090108530 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.904+0000 7f108e000700 1 -- 172.21.15.6:0/1975824635 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1084003c60 con 0x7f1090108530 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.904+0000 7f108e000700 1 -- 172.21.15.6:0/1975824635 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1084017040 con 0x7f1090108530 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.904+0000 7f1098000700 1 -- 172.21.15.6:0/1975824635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f109019cc30 con 0x7f1090108530 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.904+0000 7f1098000700 1 -- 172.21.15.6:0/1975824635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f109019f920 con 0x7f1090108530 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.905+0000 7f1098000700 1 -- 172.21.15.6:0/1975824635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10900640a0 con 0x7f1090108530 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.905+0000 7f108e000700 1 -- 172.21.15.6:0/1975824635 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1084023020 con 0x7f1090108530 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.906+0000 7f108e000700 1 --2- 172.21.15.6:0/1975824635 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f1078039600 0x7f107803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.906+0000 7f108e000700 1 -- 172.21.15.6:0/1975824635 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f108400f050 con 0x7f1090108530 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.906+0000 7f1094c00700 1 --2- 172.21.15.6:0/1975824635 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f1078039600 0x7f107803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.907+0000 7f1094c00700 1 --2- 172.21.15.6:0/1975824635 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f1078039600 0x7f107803bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f107c006fd0 tx=0x7f107c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:13.912+0000 7f108e000700 1 -- 172.21.15.6:0/1975824635 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f108402a020 con 0x7f1090108530 2024-04-26T01:33:15.062 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:14.160+0000 7f1098000700 1 -- 172.21.15.6:0/1975824635 --> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f1090104d30 con 0x7f1078039600 2024-04-26T01:33:15.063 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:14.171+0000 7f108e000700 1 -- 172.21.15.6:0/1975824635 <== mgr.14120 v2:172.21.15.6:6800/635066163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1090104d30 con 0x7f1078039600 2024-04-26T01:33:15.063 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:14.175+0000 7f1098000700 1 -- 172.21.15.6:0/1975824635 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f1078039600 msgr2=0x7f107803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:15.063 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:14.175+0000 7f1098000700 1 --2- 172.21.15.6:0/1975824635 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f1078039600 0x7f107803bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f107c006fd0 tx=0x7f107c006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:15.063 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:14.175+0000 7f1098000700 1 -- 172.21.15.6:0/1975824635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 msgr2=0x7f109019c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:15.063 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:14.175+0000 7f1098000700 1 --2- 172.21.15.6:0/1975824635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 0x7f109019c460 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1084004a60 tx=0x7f108400bd50 comp rx=0 tx=0).stop 2024-04-26T01:33:15.063 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:14.175+0000 7f1098000700 1 -- 172.21.15.6:0/1975824635 shutdown_connections 2024-04-26T01:33:15.063 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:14.175+0000 7f1098000700 1 --2- 172.21.15.6:0/1975824635 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f1078039600 0x7f107803bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:15.063 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:14.175+0000 7f1098000700 1 --2- 172.21.15.6:0/1975824635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1090108530 0x7f109019c460 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:15.063 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:14.175+0000 7f1098000700 1 -- 172.21.15.6:0/1975824635 >> 172.21.15.6:0/1975824635 conn(0x7f1090103b50 msgr2=0x7f1090104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:15.064 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:14.175+0000 7f1098000700 1 -- 172.21.15.6:0/1975824635 shutdown_connections 2024-04-26T01:33:15.064 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:14.176+0000 7f1098000700 1 -- 172.21.15.6:0/1975824635 wait complete. 2024-04-26T01:33:16.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T01:33:16.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.677+0000 7f2a2fe00700 1 Processor -- start 2024-04-26T01:33:16.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.677+0000 7f2a2fe00700 1 -- start start 2024-04-26T01:33:16.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.677+0000 7f2a2fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 0x7f2a2810ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:16.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.677+0000 7f2a2fe00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2810b150 con 0x7f2a2810a760 2024-04-26T01:33:16.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.678+0000 7f2a2d400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 0x7f2a2810ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:16.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.678+0000 7f2a2d400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 0x7f2a2810ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46522/0 (socket says 172.21.15.6:46522) 2024-04-26T01:33:16.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.678+0000 7f2a2d400700 1 -- 172.21.15.6:0/3414269506 learned_addr learned my addr 172.21.15.6:0/3414269506 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:16.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.679+0000 7f2a2d400700 1 -- 172.21.15.6:0/3414269506 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2810b910 con 0x7f2a2810a760 2024-04-26T01:33:16.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.679+0000 7f2a2d400700 1 --2- 172.21.15.6:0/3414269506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 0x7f2a2810ab80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009e80 tx=0x7f2a1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595b585b929fbf34 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:16.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.679+0000 7f2a27e00700 1 -- 172.21.15.6:0/3414269506 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a18017070 con 0x7f2a2810a760 2024-04-26T01:33:16.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.679+0000 7f2a27e00700 1 -- 172.21.15.6:0/3414269506 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2a1800b8c0 con 0x7f2a2810a760 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.679+0000 7f2a27e00700 1 -- 172.21.15.6:0/3414269506 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a18004490 con 0x7f2a2810a760 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.680+0000 7f2a2fe00700 1 -- 172.21.15.6:0/3414269506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 msgr2=0x7f2a2810ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.680+0000 7f2a2fe00700 1 --2- 172.21.15.6:0/3414269506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 0x7f2a2810ab80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009e80 tx=0x7f2a1800b330 comp rx=0 tx=0).stop 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.680+0000 7f2a2fe00700 1 -- 172.21.15.6:0/3414269506 shutdown_connections 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.680+0000 7f2a2fe00700 1 --2- 172.21.15.6:0/3414269506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 0x7f2a2810ab80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.680+0000 7f2a2fe00700 1 -- 172.21.15.6:0/3414269506 >> 172.21.15.6:0/3414269506 conn(0x7f2a2807dc20 msgr2=0x7f2a281084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.680+0000 7f2a2fe00700 1 -- 172.21.15.6:0/3414269506 shutdown_connections 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.680+0000 7f2a2fe00700 1 -- 172.21.15.6:0/3414269506 wait complete. 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.682+0000 7f2a2fe00700 1 Processor -- start 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.682+0000 7f2a2fe00700 1 -- start start 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.682+0000 7f2a2fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 0x7f2a2819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.683+0000 7f2a2fe00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2810ecc0 con 0x7f2a2810a760 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.683+0000 7f2a2d400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 0x7f2a2819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:16.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.683+0000 7f2a2d400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 0x7f2a2819e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46534/0 (socket says 172.21.15.6:46534) 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.683+0000 7f2a2d400700 1 -- 172.21.15.6:0/1879028793 learned_addr learned my addr 172.21.15.6:0/1879028793 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.683+0000 7f2a2d400700 1 -- 172.21.15.6:0/1879028793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2819ecc0 con 0x7f2a2810a760 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.684+0000 7f2a2d400700 1 --2- 172.21.15.6:0/1879028793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 0x7f2a2819e780 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2a18004a60 tx=0x7f2a1800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.684+0000 7f2a26000700 1 -- 172.21.15.6:0/1879028793 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a18020020 con 0x7f2a2810a760 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.684+0000 7f2a26000700 1 -- 172.21.15.6:0/1879028793 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2a18012430 con 0x7f2a2810a760 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.684+0000 7f2a2fe00700 1 -- 172.21.15.6:0/1879028793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a2819ef50 con 0x7f2a2810a760 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.684+0000 7f2a26000700 1 -- 172.21.15.6:0/1879028793 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a18017040 con 0x7f2a2810a760 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.684+0000 7f2a2fe00700 1 -- 172.21.15.6:0/1879028793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a2819fa70 con 0x7f2a2810a760 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.685+0000 7f2a2fe00700 1 -- 172.21.15.6:0/1879028793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a28069ba0 con 0x7f2a2810a760 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.686+0000 7f2a26000700 1 -- 172.21.15.6:0/1879028793 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f2a18023020 con 0x7f2a2810a760 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.686+0000 7f2a26000700 1 --2- 172.21.15.6:0/1879028793 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f2a0c0395b0 0x7f2a0c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.686+0000 7f2a26000700 1 -- 172.21.15.6:0/1879028793 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2a1800f050 con 0x7f2a2810a760 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.686+0000 7f2a2ca00700 1 --2- 172.21.15.6:0/1879028793 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f2a0c0395b0 0x7f2a0c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:16.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.687+0000 7f2a2ca00700 1 --2- 172.21.15.6:0/1879028793 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f2a0c0395b0 0x7f2a0c03ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2a14006fd0 tx=0x7f2a14006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.692+0000 7f2a26000700 1 -- 172.21.15.6:0/1879028793 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2a1802a020 con 0x7f2a2810a760 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.923+0000 7f2a2fe00700 1 -- 172.21.15.6:0/1879028793 --> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f2a28108340 con 0x7f2a0c0395b0 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.925+0000 7f2a26000700 1 -- 172.21.15.6:0/1879028793 <== mgr.14120 v2:172.21.15.6:6800/635066163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f2a28108340 con 0x7f2a0c0395b0 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.928+0000 7f2a2fe00700 1 -- 172.21.15.6:0/1879028793 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f2a0c0395b0 msgr2=0x7f2a0c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.928+0000 7f2a2fe00700 1 --2- 172.21.15.6:0/1879028793 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f2a0c0395b0 0x7f2a0c03ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2a14006fd0 tx=0x7f2a14006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.928+0000 7f2a2fe00700 1 -- 172.21.15.6:0/1879028793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 msgr2=0x7f2a2819e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.928+0000 7f2a2fe00700 1 --2- 172.21.15.6:0/1879028793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 0x7f2a2819e780 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2a18004a60 tx=0x7f2a1800bd50 comp rx=0 tx=0).stop 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.928+0000 7f2a2fe00700 1 -- 172.21.15.6:0/1879028793 shutdown_connections 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.928+0000 7f2a2fe00700 1 --2- 172.21.15.6:0/1879028793 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f2a0c0395b0 0x7f2a0c03ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.928+0000 7f2a2fe00700 1 --2- 172.21.15.6:0/1879028793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a2810a760 0x7f2a2819e780 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.928+0000 7f2a2fe00700 1 -- 172.21.15.6:0/1879028793 >> 172.21.15.6:0/1879028793 conn(0x7f2a2807dc20 msgr2=0x7f2a28107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.928+0000 7f2a2fe00700 1 -- 172.21.15.6:0/1879028793 shutdown_connections 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:15.929+0000 7f2a2fe00700 1 -- 172.21.15.6:0/1879028793 wait complete. 2024-04-26T01:33:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:Generating ssh key... 2024-04-26T01:33:16.922 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:16 smithi006 ceph-mon[28419]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:17.881 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-mon[28419]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:17.882 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-mon[28419]: Generating ssh key... 2024-04-26T01:33:18.225 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: Generating public/private rsa key pair. 2024-04-26T01:33:18.225 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: Your identification has been saved in /tmp/tmp3d7sm48v/key. 2024-04-26T01:33:18.225 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: Your public key has been saved in /tmp/tmp3d7sm48v/key.pub. 2024-04-26T01:33:18.225 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: The key fingerprint is: 2024-04-26T01:33:18.225 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: SHA256:MuVTusHS+awGemc4Vnnx60wMQSzyVXC5oOEflF2Zvh8 ceph-a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:18.225 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: The key's randomart image is: 2024-04-26T01:33:18.225 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: +---[RSA 3072]----+ 2024-04-26T01:33:18.225 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: | .o=oo.o | 2024-04-26T01:33:18.225 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: | . o.*.o o | 2024-04-26T01:33:18.225 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: | +.*o. o | 2024-04-26T01:33:18.225 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: | =++o.. . | 2024-04-26T01:33:18.226 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: | + So.+ . | 2024-04-26T01:33:18.226 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: | .+o*oo. .E | 2024-04-26T01:33:18.226 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: | . +..o o. ..| 2024-04-26T01:33:18.226 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: | . = +. o. .| 2024-04-26T01:33:18.226 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: | o =. .o | 2024-04-26T01:33:18.226 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: +----[SHA256]-----+ 2024-04-26T01:33:18.549 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.240+0000 7f05f8c00700 1 Processor -- start 2024-04-26T01:33:18.550 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.241+0000 7f05f8c00700 1 -- start start 2024-04-26T01:33:18.550 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.241+0000 7f05f8c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 0x7f05f410b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:18.550 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.241+0000 7f05f8c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f05f4063d60 con 0x7f05f4108fb0 2024-04-26T01:33:18.551 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.241+0000 7f05f1e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 0x7f05f410b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:18.551 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.241+0000 7f05f1e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 0x7f05f410b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46546/0 (socket says 172.21.15.6:46546) 2024-04-26T01:33:18.551 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.241+0000 7f05f1e00700 1 -- 172.21.15.6:0/3524077078 learned_addr learned my addr 172.21.15.6:0/3524077078 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:18.551 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.242+0000 7f05f1e00700 1 -- 172.21.15.6:0/3524077078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f410b8e0 con 0x7f05f4108fb0 2024-04-26T01:33:18.552 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.242+0000 7f05f1e00700 1 --2- 172.21.15.6:0/3524077078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 0x7f05f410b3a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f05e4009e80 tx=0x7f05e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4103d329c05f21c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:18.552 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.243+0000 7f05f1400700 1 -- 172.21.15.6:0/3524077078 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05e4017070 con 0x7f05f4108fb0 2024-04-26T01:33:18.553 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.243+0000 7f05f1400700 1 -- 172.21.15.6:0/3524077078 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f05e400b8c0 con 0x7f05f4108fb0 2024-04-26T01:33:18.553 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.243+0000 7f05f1400700 1 -- 172.21.15.6:0/3524077078 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05e4004490 con 0x7f05f4108fb0 2024-04-26T01:33:18.554 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.243+0000 7f05f8c00700 1 -- 172.21.15.6:0/3524077078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 msgr2=0x7f05f410b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:18.554 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.243+0000 7f05f8c00700 1 --2- 172.21.15.6:0/3524077078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 0x7f05f410b3a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f05e4009e80 tx=0x7f05e400b330 comp rx=0 tx=0).stop 2024-04-26T01:33:18.554 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.243+0000 7f05f8c00700 1 -- 172.21.15.6:0/3524077078 shutdown_connections 2024-04-26T01:33:18.554 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.243+0000 7f05f8c00700 1 --2- 172.21.15.6:0/3524077078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 0x7f05f410b3a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:18.554 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.243+0000 7f05f8c00700 1 -- 172.21.15.6:0/3524077078 >> 172.21.15.6:0/3524077078 conn(0x7f05f4102a40 msgr2=0x7f05f4104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:18.554 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.244+0000 7f05f8c00700 1 -- 172.21.15.6:0/3524077078 shutdown_connections 2024-04-26T01:33:18.554 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.244+0000 7f05f8c00700 1 -- 172.21.15.6:0/3524077078 wait complete. 2024-04-26T01:33:18.554 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.244+0000 7f05f8c00700 1 Processor -- start 2024-04-26T01:33:18.554 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.245+0000 7f05f8c00700 1 -- start start 2024-04-26T01:33:18.558 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.245+0000 7f05f8c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 0x7f05f4108830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:18.558 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.245+0000 7f05f8c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f05f410c8b0 con 0x7f05f4108fb0 2024-04-26T01:33:18.558 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.245+0000 7f05f1e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 0x7f05f4108830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:18.558 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.245+0000 7f05f1e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 0x7f05f4108830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46554/0 (socket says 172.21.15.6:46554) 2024-04-26T01:33:18.558 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.245+0000 7f05f1e00700 1 -- 172.21.15.6:0/3496621808 learned_addr learned my addr 172.21.15.6:0/3496621808 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:18.558 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.246+0000 7f05f1e00700 1 -- 172.21.15.6:0/3496621808 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f4108d70 con 0x7f05f4108fb0 2024-04-26T01:33:18.558 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.246+0000 7f05f1e00700 1 --2- 172.21.15.6:0/3496621808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 0x7f05f4108830 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f05e400b7b0 tx=0x7f05e400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:18.558 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.246+0000 7f05eb400700 1 -- 172.21.15.6:0/3496621808 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05e4020020 con 0x7f05f4108fb0 2024-04-26T01:33:18.558 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.246+0000 7f05eb400700 1 -- 172.21.15.6:0/3496621808 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f05e4012430 con 0x7f05f4108fb0 2024-04-26T01:33:18.558 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.246+0000 7f05eb400700 1 -- 172.21.15.6:0/3496621808 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05e4017040 con 0x7f05f4108fb0 2024-04-26T01:33:18.558 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.246+0000 7f05f8c00700 1 -- 172.21.15.6:0/3496621808 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05f4106f60 con 0x7f05f4108fb0 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.246+0000 7f05f8c00700 1 -- 172.21.15.6:0/3496621808 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05f41074a0 con 0x7f05f4108fb0 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.247+0000 7f05eb400700 1 -- 172.21.15.6:0/3496621808 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f05e4023020 con 0x7f05f4108fb0 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.247+0000 7f05f8c00700 1 -- 172.21.15.6:0/3496621808 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05f40640a0 con 0x7f05f4108fb0 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.248+0000 7f05eb400700 1 --2- 172.21.15.6:0/3496621808 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f05d80395b0 0x7f05d803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.248+0000 7f05eb400700 1 -- 172.21.15.6:0/3496621808 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f05e400f050 con 0x7f05f4108fb0 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.248+0000 7f05e9400700 1 --2- 172.21.15.6:0/3496621808 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f05d80395b0 0x7f05d803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.249+0000 7f05e9400700 1 --2- 172.21.15.6:0/3496621808 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f05d80395b0 0x7f05d803ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f05dc006fd0 tx=0x7f05dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.253+0000 7f05eb400700 1 -- 172.21.15.6:0/3496621808 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f05e402a020 con 0x7f05f4108fb0 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.482+0000 7f05f8c00700 1 -- 172.21.15.6:0/3496621808 --> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f05f4107d00 con 0x7f05d80395b0 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.905+0000 7f05eb400700 1 -- 172.21.15.6:0/3496621808 <== mgr.14120 v2:172.21.15.6:6800/635066163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f05f4107d00 con 0x7f05d80395b0 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.908+0000 7f05f8c00700 1 -- 172.21.15.6:0/3496621808 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f05d80395b0 msgr2=0x7f05d803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.908+0000 7f05f8c00700 1 --2- 172.21.15.6:0/3496621808 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f05d80395b0 0x7f05d803ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f05dc006fd0 tx=0x7f05dc006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.908+0000 7f05f8c00700 1 -- 172.21.15.6:0/3496621808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 msgr2=0x7f05f4108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:18.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.908+0000 7f05f8c00700 1 --2- 172.21.15.6:0/3496621808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 0x7f05f4108830 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f05e400b7b0 tx=0x7f05e400be20 comp rx=0 tx=0).stop 2024-04-26T01:33:18.560 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.909+0000 7f05f8c00700 1 -- 172.21.15.6:0/3496621808 shutdown_connections 2024-04-26T01:33:18.560 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.909+0000 7f05f8c00700 1 --2- 172.21.15.6:0/3496621808 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f05d80395b0 0x7f05d803ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:18.560 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.909+0000 7f05f8c00700 1 --2- 172.21.15.6:0/3496621808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05f4108fb0 0x7f05f4108830 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:18.560 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.909+0000 7f05f8c00700 1 -- 172.21.15.6:0/3496621808 >> 172.21.15.6:0/3496621808 conn(0x7f05f4102a40 msgr2=0x7f05f4103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:18.560 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.909+0000 7f05f8c00700 1 -- 172.21.15.6:0/3496621808 shutdown_connections 2024-04-26T01:33:18.560 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:17.909+0000 7f05f8c00700 1 -- 172.21.15.6:0/3496621808 wait complete. 2024-04-26T01:33:19.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:18 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:19.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:18 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:20.217 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:19 smithi006 ceph-mon[28419]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:20.240 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9/IoH5V/4QN52cB2pRQb7vo4jzMp2nlrc481f8N4iab7SfNWfUay7jRS0EXUjv+08XdAjLoF6PgmE1Bd9QVPVfM/7BYrC1nleXw0JLy/MK9vs2WMR32v4hVBcVH9cCwft+nXpgelFsaRCgEMU0O3hE6+WYUxPH6frcNc6sQDPGa5dDCq1qC8Mc1RFt2/wNNIz7ocTONVBiDDMla+F/E6xn376PNOp77vfALWHaTFhob3N8sVn/F4DXPAbdLonscPDru4xM01Mxc8r/oGlVlk/zow6XnFNM6J37UyTqs/WEwsmJezVCeIj3T/eOof7vnnZgW451ABzf0AUO+ilaZ0s7M8mncr76KFQG2zashvJ+L7HjOGCH/0WpcJop+xptAIxyT1L0D96K4a+Zb7vXHjo/HUSrR0IBo2g7EaA0uRCVRpavnFvuJGv6D5Jk92f2KtLV8fgD33YEQms1gPcA5GZVitYdyFKzG2zTWLINlJoIVZjm/Cru0kXTHlJLozv6XM= ceph-a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:20.241 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.247+0000 7f96a3600700 1 Processor -- start 2024-04-26T01:33:20.241 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.247+0000 7f96a3600700 1 -- start start 2024-04-26T01:33:20.241 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.247+0000 7f96a3600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 0x7f969c10ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:20.241 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.247+0000 7f96a3600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f969c10b150 con 0x7f969c10a760 2024-04-26T01:33:20.241 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.248+0000 7f96a0c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 0x7f969c10ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.248+0000 7f96a0c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 0x7f969c10ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45528/0 (socket says 172.21.15.6:45528) 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.248+0000 7f96a0c00700 1 -- 172.21.15.6:0/3888844617 learned_addr learned my addr 172.21.15.6:0/3888844617 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.248+0000 7f96a0c00700 1 -- 172.21.15.6:0/3888844617 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969c10b910 con 0x7f969c10a760 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.249+0000 7f96a0c00700 1 --2- 172.21.15.6:0/3888844617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 0x7f969c10ab80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f968c0098a0 tx=0x7f968c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42dfdbf7e246873c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.249+0000 7f969be00700 1 -- 172.21.15.6:0/3888844617 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f968c004160 con 0x7f969c10a760 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.249+0000 7f969be00700 1 -- 172.21.15.6:0/3888844617 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f968c004300 con 0x7f969c10a760 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.249+0000 7f969be00700 1 -- 172.21.15.6:0/3888844617 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f968c011af0 con 0x7f969c10a760 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.250+0000 7f96a3600700 1 -- 172.21.15.6:0/3888844617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 msgr2=0x7f969c10ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.250+0000 7f96a3600700 1 --2- 172.21.15.6:0/3888844617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 0x7f969c10ab80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f968c0098a0 tx=0x7f968c009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.250+0000 7f96a3600700 1 -- 172.21.15.6:0/3888844617 shutdown_connections 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.250+0000 7f96a3600700 1 --2- 172.21.15.6:0/3888844617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 0x7f969c10ab80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.250+0000 7f96a3600700 1 -- 172.21.15.6:0/3888844617 >> 172.21.15.6:0/3888844617 conn(0x7f969c07dc20 msgr2=0x7f969c1084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.250+0000 7f96a3600700 1 -- 172.21.15.6:0/3888844617 shutdown_connections 2024-04-26T01:33:20.242 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.250+0000 7f96a3600700 1 -- 172.21.15.6:0/3888844617 wait complete. 2024-04-26T01:33:20.243 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.251+0000 7f96a3600700 1 Processor -- start 2024-04-26T01:33:20.243 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.251+0000 7f96a3600700 1 -- start start 2024-04-26T01:33:20.243 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.252+0000 7f96a3600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 0x7f969c19e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:20.243 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.252+0000 7f96a3600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f969c10ecc0 con 0x7f969c10a760 2024-04-26T01:33:20.243 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.252+0000 7f96a0c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 0x7f969c19e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:20.243 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.252+0000 7f96a0c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 0x7f969c19e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45536/0 (socket says 172.21.15.6:45536) 2024-04-26T01:33:20.243 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.252+0000 7f96a0c00700 1 -- 172.21.15.6:0/419238160 learned_addr learned my addr 172.21.15.6:0/419238160 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:20.243 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.252+0000 7f96a0c00700 1 -- 172.21.15.6:0/419238160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969c19ed30 con 0x7f969c10a760 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.253+0000 7f96a0c00700 1 --2- 172.21.15.6:0/419238160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 0x7f969c19e7f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f968c0117b0 tx=0x7f968c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.253+0000 7f969a000700 1 -- 172.21.15.6:0/419238160 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f968c003af0 con 0x7f969c10a760 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.253+0000 7f969a000700 1 -- 172.21.15.6:0/419238160 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f968c020070 con 0x7f969c10a760 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.253+0000 7f96a3600700 1 -- 172.21.15.6:0/419238160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f969c19efc0 con 0x7f969c10a760 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.253+0000 7f969a000700 1 -- 172.21.15.6:0/419238160 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f968c019630 con 0x7f969c10a760 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.253+0000 7f96a3600700 1 -- 172.21.15.6:0/419238160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f969c19fae0 con 0x7f969c10a760 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.254+0000 7f96a3600700 1 -- 172.21.15.6:0/419238160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f969c0640a0 con 0x7f969c10a760 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.255+0000 7f969a000700 1 -- 172.21.15.6:0/419238160 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f968c01a9c0 con 0x7f969c10a760 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.255+0000 7f969a000700 1 --2- 172.21.15.6:0/419238160 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f9680039600 0x7f968003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.255+0000 7f969a000700 1 -- 172.21.15.6:0/419238160 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f968c04db70 con 0x7f969c10a760 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.255+0000 7f9693e00700 1 --2- 172.21.15.6:0/419238160 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f9680039600 0x7f968003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.256+0000 7f9693e00700 1 --2- 172.21.15.6:0/419238160 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f9680039600 0x7f968003bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9684006fd0 tx=0x7f9684006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:20.244 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.260+0000 7f969a000700 1 -- 172.21.15.6:0/419238160 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f968c0042a0 con 0x7f969c10a760 2024-04-26T01:33:20.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.496+0000 7f96a3600700 1 -- 172.21.15.6:0/419238160 --> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f969c108390 con 0x7f9680039600 2024-04-26T01:33:20.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.498+0000 7f969a000700 1 -- 172.21.15.6:0/419238160 <== mgr.14120 v2:172.21.15.6:6800/635066163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f969c108390 con 0x7f9680039600 2024-04-26T01:33:20.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.500+0000 7f96a3600700 1 -- 172.21.15.6:0/419238160 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f9680039600 msgr2=0x7f968003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:20.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.500+0000 7f96a3600700 1 --2- 172.21.15.6:0/419238160 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f9680039600 0x7f968003bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9684006fd0 tx=0x7f9684006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:20.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.500+0000 7f96a3600700 1 -- 172.21.15.6:0/419238160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 msgr2=0x7f969c19e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:20.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.500+0000 7f96a3600700 1 --2- 172.21.15.6:0/419238160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 0x7f969c19e7f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f968c0117b0 tx=0x7f968c011b70 comp rx=0 tx=0).stop 2024-04-26T01:33:20.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.501+0000 7f96a3600700 1 -- 172.21.15.6:0/419238160 shutdown_connections 2024-04-26T01:33:20.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.501+0000 7f96a3600700 1 --2- 172.21.15.6:0/419238160 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f9680039600 0x7f968003bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:20.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.501+0000 7f96a3600700 1 --2- 172.21.15.6:0/419238160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f969c10a760 0x7f969c19e7f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:20.245 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.501+0000 7f96a3600700 1 -- 172.21.15.6:0/419238160 >> 172.21.15.6:0/419238160 conn(0x7f969c07dc20 msgr2=0x7f969c107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:20.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.501+0000 7f96a3600700 1 -- 172.21.15.6:0/419238160 shutdown_connections 2024-04-26T01:33:20.246 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:19.501+0000 7f96a3600700 1 -- 172.21.15.6:0/419238160 wait complete. 2024-04-26T01:33:20.246 INFO:teuthology.orchestra.run.smithi006.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T01:33:20.246 INFO:teuthology.orchestra.run.smithi006.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T01:33:20.246 INFO:teuthology.orchestra.run.smithi006.stdout:Adding host smithi006... 2024-04-26T01:33:21.738 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:21 smithi006 ceph-mon[28419]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi006", "addr": "172.21.15.6", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:22.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:22 smithi006 ceph-mon[28419]: Deploying cephadm binary to smithi006 2024-04-26T01:33:27.954 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Added host 'smithi006' with addr '172.21.15.6' 2024-04-26T01:33:27.955 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.815+0000 7f99d3400700 1 Processor -- start 2024-04-26T01:33:27.955 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.816+0000 7f99d3400700 1 -- start start 2024-04-26T01:33:27.955 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.816+0000 7f99d3400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 0x7f99cc10aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:27.955 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.816+0000 7f99d3400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc10b170 con 0x7f99cc10a780 2024-04-26T01:33:27.955 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.817+0000 7f99d0a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 0x7f99cc10aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:27.956 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.817+0000 7f99d0a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 0x7f99cc10aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45552/0 (socket says 172.21.15.6:45552) 2024-04-26T01:33:27.956 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.817+0000 7f99d0a00700 1 -- 172.21.15.6:0/2642030981 learned_addr learned my addr 172.21.15.6:0/2642030981 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:27.956 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.817+0000 7f99d0a00700 1 -- 172.21.15.6:0/2642030981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc10b930 con 0x7f99cc10a780 2024-04-26T01:33:27.956 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.818+0000 7f99d0a00700 1 --2- 172.21.15.6:0/2642030981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 0x7f99cc10aba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f99b40098a0 tx=0x7f99b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bc4ef850dbc97b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:27.956 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.818+0000 7f99cbe00700 1 -- 172.21.15.6:0/2642030981 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99b4004160 con 0x7f99cc10a780 2024-04-26T01:33:27.956 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.818+0000 7f99cbe00700 1 -- 172.21.15.6:0/2642030981 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f99b4004300 con 0x7f99cc10a780 2024-04-26T01:33:27.959 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.818+0000 7f99cbe00700 1 -- 172.21.15.6:0/2642030981 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99b4011af0 con 0x7f99cc10a780 2024-04-26T01:33:27.959 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.819+0000 7f99d3400700 1 -- 172.21.15.6:0/2642030981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 msgr2=0x7f99cc10aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:27.959 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.819+0000 7f99d3400700 1 --2- 172.21.15.6:0/2642030981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 0x7f99cc10aba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f99b40098a0 tx=0x7f99b4009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:27.959 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.819+0000 7f99d3400700 1 -- 172.21.15.6:0/2642030981 shutdown_connections 2024-04-26T01:33:27.959 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.819+0000 7f99d3400700 1 --2- 172.21.15.6:0/2642030981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 0x7f99cc10aba0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:27.959 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.819+0000 7f99d3400700 1 -- 172.21.15.6:0/2642030981 >> 172.21.15.6:0/2642030981 conn(0x7f99cc105d40 msgr2=0x7f99cc108160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:27.959 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.819+0000 7f99d3400700 1 -- 172.21.15.6:0/2642030981 shutdown_connections 2024-04-26T01:33:27.959 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.819+0000 7f99d3400700 1 -- 172.21.15.6:0/2642030981 wait complete. 2024-04-26T01:33:27.959 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.820+0000 7f99d3400700 1 Processor -- start 2024-04-26T01:33:27.959 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.820+0000 7f99d3400700 1 -- start start 2024-04-26T01:33:27.960 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.821+0000 7f99d3400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 0x7f99cc19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:27.960 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.821+0000 7f99d3400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc10ece0 con 0x7f99cc10a780 2024-04-26T01:33:27.960 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.821+0000 7f99d0a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 0x7f99cc19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:27.960 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.821+0000 7f99d0a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 0x7f99cc19e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45568/0 (socket says 172.21.15.6:45568) 2024-04-26T01:33:27.960 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.821+0000 7f99d0a00700 1 -- 172.21.15.6:0/1866813258 learned_addr learned my addr 172.21.15.6:0/1866813258 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:27.960 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.821+0000 7f99d0a00700 1 -- 172.21.15.6:0/1866813258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc19ebc0 con 0x7f99cc10a780 2024-04-26T01:33:27.960 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.822+0000 7f99d0a00700 1 --2- 172.21.15.6:0/1866813258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 0x7f99cc19e680 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f99b4000c00 tx=0x7f99b4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:27.960 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.822+0000 7f99ca000700 1 -- 172.21.15.6:0/1866813258 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99b4003b20 con 0x7f99cc10a780 2024-04-26T01:33:27.960 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.822+0000 7f99ca000700 1 -- 172.21.15.6:0/1866813258 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f99b401a8c0 con 0x7f99cc10a780 2024-04-26T01:33:27.960 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.822+0000 7f99ca000700 1 -- 172.21.15.6:0/1866813258 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99b4024940 con 0x7f99cc10a780 2024-04-26T01:33:27.963 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.822+0000 7f99d3400700 1 -- 172.21.15.6:0/1866813258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99cc19ee50 con 0x7f99cc10a780 2024-04-26T01:33:27.963 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.822+0000 7f99d3400700 1 -- 172.21.15.6:0/1866813258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99cc1a1b40 con 0x7f99cc10a780 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.823+0000 7f99d3400700 1 -- 172.21.15.6:0/1866813258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99cc0640a0 con 0x7f99cc10a780 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.823+0000 7f99ca000700 1 -- 172.21.15.6:0/1866813258 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f99b4022020 con 0x7f99cc10a780 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.824+0000 7f99ca000700 1 --2- 172.21.15.6:0/1866813258 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f99ac039660 0x7f99ac03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.824+0000 7f99ca000700 1 -- 172.21.15.6:0/1866813258 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f99b404d8a0 con 0x7f99cc10a780 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.825+0000 7f99c3e00700 1 --2- 172.21.15.6:0/1866813258 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f99ac039660 0x7f99ac03bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.825+0000 7f99c3e00700 1 --2- 172.21.15.6:0/1866813258 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f99ac039660 0x7f99ac03bb20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f99bc006fd0 tx=0x7f99bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:20.830+0000 7f99ca000700 1 -- 172.21.15.6:0/1866813258 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99b400be50 con 0x7f99cc10a780 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:21.064+0000 7f99d3400700 1 -- 172.21.15.6:0/1866813258 --> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi006", "addr": "172.21.15.6", "target": ["mon-mgr", ""]}) -- 0x7f99cc106ef0 con 0x7f99ac039660 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:27.233+0000 7f99ca000700 1 -- 172.21.15.6:0/1866813258 <== mgr.14120 v2:172.21.15.6:6800/635066163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f99cc106ef0 con 0x7f99ac039660 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:27.237+0000 7f99d3400700 1 -- 172.21.15.6:0/1866813258 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f99ac039660 msgr2=0x7f99ac03bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:27.237+0000 7f99d3400700 1 --2- 172.21.15.6:0/1866813258 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f99ac039660 0x7f99ac03bb20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f99bc006fd0 tx=0x7f99bc006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:27.237+0000 7f99d3400700 1 -- 172.21.15.6:0/1866813258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 msgr2=0x7f99cc19e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:27.964 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:27.237+0000 7f99d3400700 1 --2- 172.21.15.6:0/1866813258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 0x7f99cc19e680 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f99b4000c00 tx=0x7f99b4011b70 comp rx=0 tx=0).stop 2024-04-26T01:33:27.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:27.237+0000 7f99d3400700 1 -- 172.21.15.6:0/1866813258 shutdown_connections 2024-04-26T01:33:27.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:27.237+0000 7f99d3400700 1 --2- 172.21.15.6:0/1866813258 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f99ac039660 0x7f99ac03bb20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:27.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:27.237+0000 7f99d3400700 1 --2- 172.21.15.6:0/1866813258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99cc10a780 0x7f99cc19e680 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:27.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:27.237+0000 7f99d3400700 1 -- 172.21.15.6:0/1866813258 >> 172.21.15.6:0/1866813258 conn(0x7f99cc105d40 msgr2=0x7f99cc1067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:27.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:27.237+0000 7f99d3400700 1 -- 172.21.15.6:0/1866813258 shutdown_connections 2024-04-26T01:33:27.965 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:27.238+0000 7f99d3400700 1 -- 172.21.15.6:0/1866813258 wait complete. 2024-04-26T01:33:27.965 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying unmanaged mon service... 2024-04-26T01:33:28.563 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:28 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:28.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:28 smithi006 ceph-mon[28419]: Added host smithi006 2024-04-26T01:33:28.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:28 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:29.845 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T01:33:29.846 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.618+0000 7f2b09200700 1 Processor -- start 2024-04-26T01:33:29.846 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.618+0000 7f2b09200700 1 -- start start 2024-04-26T01:33:29.846 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.619+0000 7f2b09200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 0x7f2b04108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:29.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.619+0000 7f2b09200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2b04108f20 con 0x7f2b04108530 2024-04-26T01:33:29.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.619+0000 7f2b01e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 0x7f2b04108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:29.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.619+0000 7f2b01e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 0x7f2b04108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48772/0 (socket says 172.21.15.6:48772) 2024-04-26T01:33:29.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.619+0000 7f2b01e00700 1 -- 172.21.15.6:0/3083585622 learned_addr learned my addr 172.21.15.6:0/3083585622 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:29.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.620+0000 7f2b01e00700 1 -- 172.21.15.6:0/3083585622 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b04109730 con 0x7f2b04108530 2024-04-26T01:33:29.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.620+0000 7f2b01e00700 1 --2- 172.21.15.6:0/3083585622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 0x7f2b04108950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2af40098a0 tx=0x7f2af4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477f1a7cb54cad6c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:29.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.620+0000 7f2b01400700 1 -- 172.21.15.6:0/3083585622 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2af4004160 con 0x7f2b04108530 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.620+0000 7f2b01400700 1 -- 172.21.15.6:0/3083585622 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2af4004300 con 0x7f2b04108530 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.620+0000 7f2b01400700 1 -- 172.21.15.6:0/3083585622 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2af4011af0 con 0x7f2b04108530 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.621+0000 7f2b09200700 1 -- 172.21.15.6:0/3083585622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 msgr2=0x7f2b04108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.621+0000 7f2b09200700 1 --2- 172.21.15.6:0/3083585622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 0x7f2b04108950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2af40098a0 tx=0x7f2af4009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.621+0000 7f2b09200700 1 -- 172.21.15.6:0/3083585622 shutdown_connections 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.621+0000 7f2b09200700 1 --2- 172.21.15.6:0/3083585622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 0x7f2b04108950 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.621+0000 7f2b09200700 1 -- 172.21.15.6:0/3083585622 >> 172.21.15.6:0/3083585622 conn(0x7f2b04103b50 msgr2=0x7f2b04105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.621+0000 7f2b09200700 1 -- 172.21.15.6:0/3083585622 shutdown_connections 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.621+0000 7f2b09200700 1 -- 172.21.15.6:0/3083585622 wait complete. 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.623+0000 7f2b09200700 1 Processor -- start 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.623+0000 7f2b09200700 1 -- start start 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.623+0000 7f2b09200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 0x7f2b0419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:29.852 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.623+0000 7f2b09200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0410cae0 con 0x7f2b04108530 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.624+0000 7f2b01e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 0x7f2b0419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.624+0000 7f2b01e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 0x7f2b0419e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48784/0 (socket says 172.21.15.6:48784) 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.624+0000 7f2b01e00700 1 -- 172.21.15.6:0/344598053 learned_addr learned my addr 172.21.15.6:0/344598053 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.624+0000 7f2b01e00700 1 -- 172.21.15.6:0/344598053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0419ec10 con 0x7f2b04108530 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.624+0000 7f2b01e00700 1 --2- 172.21.15.6:0/344598053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 0x7f2b0419e6d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2af4003f40 tx=0x7f2af4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.625+0000 7f2afaa00700 1 -- 172.21.15.6:0/344598053 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2af40038b0 con 0x7f2b04108530 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.625+0000 7f2afaa00700 1 -- 172.21.15.6:0/344598053 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2af4003a50 con 0x7f2b04108530 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.625+0000 7f2afaa00700 1 -- 172.21.15.6:0/344598053 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2af401a5c0 con 0x7f2b04108530 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.625+0000 7f2b09200700 1 -- 172.21.15.6:0/344598053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b0419eea0 con 0x7f2b04108530 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.625+0000 7f2b09200700 1 -- 172.21.15.6:0/344598053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b041a1b90 con 0x7f2b04108530 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.626+0000 7f2b09200700 1 -- 172.21.15.6:0/344598053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b040640a0 con 0x7f2b04108530 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.626+0000 7f2afaa00700 1 -- 172.21.15.6:0/344598053 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f2af4024430 con 0x7f2b04108530 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.627+0000 7f2afaa00700 1 --2- 172.21.15.6:0/344598053 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f2ae80395b0 0x7f2ae803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:29.853 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.627+0000 7f2afaa00700 1 -- 172.21.15.6:0/344598053 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2af404dbc0 con 0x7f2b04108530 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.627+0000 7f2afbe00700 1 --2- 172.21.15.6:0/344598053 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f2ae80395b0 0x7f2ae803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.628+0000 7f2afbe00700 1 --2- 172.21.15.6:0/344598053 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f2ae80395b0 0x7f2ae803ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2aec006fd0 tx=0x7f2aec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.632+0000 7f2afaa00700 1 -- 172.21.15.6:0/344598053 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2af404d510 con 0x7f2b04108530 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.903+0000 7f2b09200700 1 -- 172.21.15.6:0/344598053 --> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f2b04104cc0 con 0x7f2ae80395b0 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.910+0000 7f2afaa00700 1 -- 172.21.15.6:0/344598053 <== mgr.14120 v2:172.21.15.6:6800/635066163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2b04104cc0 con 0x7f2ae80395b0 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.913+0000 7f2b09200700 1 -- 172.21.15.6:0/344598053 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f2ae80395b0 msgr2=0x7f2ae803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.913+0000 7f2b09200700 1 --2- 172.21.15.6:0/344598053 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f2ae80395b0 0x7f2ae803ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2aec006fd0 tx=0x7f2aec006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.913+0000 7f2b09200700 1 -- 172.21.15.6:0/344598053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 msgr2=0x7f2b0419e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.913+0000 7f2b09200700 1 --2- 172.21.15.6:0/344598053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 0x7f2b0419e6d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2af4003f40 tx=0x7f2af4011890 comp rx=0 tx=0).stop 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.913+0000 7f2b09200700 1 -- 172.21.15.6:0/344598053 shutdown_connections 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.913+0000 7f2b09200700 1 --2- 172.21.15.6:0/344598053 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f2ae80395b0 0x7f2ae803ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.913+0000 7f2b09200700 1 --2- 172.21.15.6:0/344598053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b04108530 0x7f2b0419e6d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.913+0000 7f2b09200700 1 -- 172.21.15.6:0/344598053 >> 172.21.15.6:0/344598053 conn(0x7f2b04103b50 msgr2=0x7f2b041045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:29.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.913+0000 7f2b09200700 1 -- 172.21.15.6:0/344598053 shutdown_connections 2024-04-26T01:33:29.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:28.913+0000 7f2b09200700 1 -- 172.21.15.6:0/344598053 wait complete. 2024-04-26T01:33:29.855 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying unmanaged mgr service... 2024-04-26T01:33:29.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:29 smithi006 ceph-mon[28419]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:29.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:29 smithi006 ceph-mon[28419]: Saving service mon spec with placement count:5 2024-04-26T01:33:29.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:29 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:31.772 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:31 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:31.772 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:31 smithi006 ceph-mon[28419]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:31.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:31 smithi006 ceph-mon[28419]: Saving service mgr spec with placement count:2 2024-04-26T01:33:31.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:31 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:31.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:31 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:32.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T01:33:32.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.167+0000 7fd1ab000700 1 Processor -- start 2024-04-26T01:33:32.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.167+0000 7fd1ab000700 1 -- start start 2024-04-26T01:33:32.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.168+0000 7fd1ab000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a4106da0 0x7fd1a41071c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:32.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.168+0000 7fd1ab000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a4107700 con 0x7fd1a4106da0 2024-04-26T01:33:32.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.168+0000 7fd1a3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a4106da0 0x7fd1a41071c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:32.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.168+0000 7fd1a3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a4106da0 0x7fd1a41071c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48800/0 (socket says 172.21.15.6:48800) 2024-04-26T01:33:32.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.168+0000 7fd1a3e00700 1 -- 172.21.15.6:0/4172184376 learned_addr learned my addr 172.21.15.6:0/4172184376 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.169+0000 7fd1a3e00700 1 -- 172.21.15.6:0/4172184376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a4107880 con 0x7fd1a4106da0 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.169+0000 7fd1a3e00700 1 --2- 172.21.15.6:0/4172184376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a4106da0 0x7fd1a41071c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd1940098a0 tx=0x7fd194009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4aee3b63916fcc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.169+0000 7fd1a2a00700 1 -- 172.21.15.6:0/4172184376 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd194004160 con 0x7fd1a4106da0 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.170+0000 7fd1a2a00700 1 -- 172.21.15.6:0/4172184376 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd194004300 con 0x7fd1a4106da0 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.170+0000 7fd1a2a00700 1 -- 172.21.15.6:0/4172184376 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd194011a20 con 0x7fd1a4106da0 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.170+0000 7fd1ab000700 1 -- 172.21.15.6:0/4172184376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a4106da0 msgr2=0x7fd1a41071c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.170+0000 7fd1ab000700 1 --2- 172.21.15.6:0/4172184376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a4106da0 0x7fd1a41071c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd1940098a0 tx=0x7fd194009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.170+0000 7fd1ab000700 1 -- 172.21.15.6:0/4172184376 shutdown_connections 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.170+0000 7fd1ab000700 1 --2- 172.21.15.6:0/4172184376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a4106da0 0x7fd1a41071c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.170+0000 7fd1ab000700 1 -- 172.21.15.6:0/4172184376 >> 172.21.15.6:0/4172184376 conn(0x7fd1a4102a60 msgr2=0x7fd1a4104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.171+0000 7fd1ab000700 1 -- 172.21.15.6:0/4172184376 shutdown_connections 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.171+0000 7fd1ab000700 1 -- 172.21.15.6:0/4172184376 wait complete. 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.172+0000 7fd1ab000700 1 Processor -- start 2024-04-26T01:33:32.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.172+0000 7fd1ab000700 1 -- start start 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.173+0000 7fd1ab000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a419e960 0x7fd1a419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.173+0000 7fd1ab000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a41088f0 con 0x7fd1a419e960 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.173+0000 7fd1a3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a419e960 0x7fd1a419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.173+0000 7fd1a3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a419e960 0x7fd1a419ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48816/0 (socket says 172.21.15.6:48816) 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.173+0000 7fd1a3e00700 1 -- 172.21.15.6:0/800807637 learned_addr learned my addr 172.21.15.6:0/800807637 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.173+0000 7fd1a3e00700 1 -- 172.21.15.6:0/800807637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a419f2c0 con 0x7fd1a419e960 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.174+0000 7fd1a3e00700 1 --2- 172.21.15.6:0/800807637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a419e960 0x7fd1a419ed80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd194000c00 tx=0x7fd194012fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.174+0000 7fd1a0c00700 1 -- 172.21.15.6:0/800807637 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd194011a20 con 0x7fd1a419e960 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.174+0000 7fd1a0c00700 1 -- 172.21.15.6:0/800807637 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd194003b50 con 0x7fd1a419e960 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.174+0000 7fd1a0c00700 1 -- 172.21.15.6:0/800807637 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd194025db0 con 0x7fd1a419e960 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.174+0000 7fd1ab000700 1 -- 172.21.15.6:0/800807637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a419f550 con 0x7fd1a419e960 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.174+0000 7fd1ab000700 1 -- 172.21.15.6:0/800807637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1a41a2030 con 0x7fd1a419e960 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.175+0000 7fd1a0c00700 1 -- 172.21.15.6:0/800807637 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fd1940255a0 con 0x7fd1a419e960 2024-04-26T01:33:32.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.175+0000 7fd1ab000700 1 -- 172.21.15.6:0/800807637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1a40640a0 con 0x7fd1a419e960 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.176+0000 7fd1a0c00700 1 --2- 172.21.15.6:0/800807637 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7fd1880395b0 0x7fd18803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.176+0000 7fd1a0c00700 1 -- 172.21.15.6:0/800807637 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd19404dc10 con 0x7fd1a419e960 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.176+0000 7fd1a3400700 1 --2- 172.21.15.6:0/800807637 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7fd1880395b0 0x7fd18803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.177+0000 7fd1a3400700 1 --2- 172.21.15.6:0/800807637 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7fd1880395b0 0x7fd18803ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd190006fd0 tx=0x7fd190006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.182+0000 7fd1a0c00700 1 -- 172.21.15.6:0/800807637 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd19400b3e0 con 0x7fd1a419e960 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.432+0000 7fd1ab000700 1 -- 172.21.15.6:0/800807637 --> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fd1a402e240 con 0x7fd1880395b0 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.439+0000 7fd1a0c00700 1 -- 172.21.15.6:0/800807637 <== mgr.14120 v2:172.21.15.6:6800/635066163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd1a402e240 con 0x7fd1880395b0 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.443+0000 7fd1ab000700 1 -- 172.21.15.6:0/800807637 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7fd1880395b0 msgr2=0x7fd18803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.443+0000 7fd1ab000700 1 --2- 172.21.15.6:0/800807637 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7fd1880395b0 0x7fd18803ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd190006fd0 tx=0x7fd190006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.443+0000 7fd1ab000700 1 -- 172.21.15.6:0/800807637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a419e960 msgr2=0x7fd1a419ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.443+0000 7fd1ab000700 1 --2- 172.21.15.6:0/800807637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a419e960 0x7fd1a419ed80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd194000c00 tx=0x7fd194012fc0 comp rx=0 tx=0).stop 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.444+0000 7fd1ab000700 1 -- 172.21.15.6:0/800807637 shutdown_connections 2024-04-26T01:33:32.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.444+0000 7fd1ab000700 1 --2- 172.21.15.6:0/800807637 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7fd1880395b0 0x7fd18803ba70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:32.180 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.444+0000 7fd1ab000700 1 --2- 172.21.15.6:0/800807637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1a419e960 0x7fd1a419ed80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:32.180 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.444+0000 7fd1ab000700 1 -- 172.21.15.6:0/800807637 >> 172.21.15.6:0/800807637 conn(0x7fd1a4102a60 msgr2=0x7fd1a4195440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:32.180 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.444+0000 7fd1ab000700 1 -- 172.21.15.6:0/800807637 shutdown_connections 2024-04-26T01:33:32.180 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:31.444+0000 7fd1ab000700 1 -- 172.21.15.6:0/800807637 wait complete. 2024-04-26T01:33:32.756 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:32 smithi006 ceph-mon[28419]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:34.438 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:34 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/762996732' entity='client.admin' 2024-04-26T01:33:34.450 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.837+0000 7f9ab2e00700 1 Processor -- start 2024-04-26T01:33:34.450 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.838+0000 7f9ab2e00700 1 -- start start 2024-04-26T01:33:34.450 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.838+0000 7f9ab2e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 0x7f9aac07b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:34.450 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.838+0000 7f9ab2e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac07b790 con 0x7f9aac07cd60 2024-04-26T01:33:34.450 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.838+0000 7f9aabe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 0x7f9aac07b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:34.450 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.838+0000 7f9aabe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 0x7f9aac07b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48824/0 (socket says 172.21.15.6:48824) 2024-04-26T01:33:34.450 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.838+0000 7f9aabe00700 1 -- 172.21.15.6:0/1461560973 learned_addr learned my addr 172.21.15.6:0/1461560973 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.839+0000 7f9aabe00700 1 -- 172.21.15.6:0/1461560973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aac07b910 con 0x7f9aac07cd60 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.839+0000 7f9aabe00700 1 --2- 172.21.15.6:0/1461560973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 0x7f9aac07b1c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0098a0 tx=0x7f9a9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d433046cf20678a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.839+0000 7f9aaaa00700 1 -- 172.21.15.6:0/1461560973 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a9c004160 con 0x7f9aac07cd60 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.839+0000 7f9aaaa00700 1 -- 172.21.15.6:0/1461560973 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a9c004300 con 0x7f9aac07cd60 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.839+0000 7f9aaaa00700 1 -- 172.21.15.6:0/1461560973 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a9c011a20 con 0x7f9aac07cd60 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.840+0000 7f9ab2e00700 1 -- 172.21.15.6:0/1461560973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 msgr2=0x7f9aac07b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.840+0000 7f9ab2e00700 1 --2- 172.21.15.6:0/1461560973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 0x7f9aac07b1c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0098a0 tx=0x7f9a9c009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.840+0000 7f9ab2e00700 1 -- 172.21.15.6:0/1461560973 shutdown_connections 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.840+0000 7f9ab2e00700 1 --2- 172.21.15.6:0/1461560973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 0x7f9aac07b1c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.840+0000 7f9ab2e00700 1 -- 172.21.15.6:0/1461560973 >> 172.21.15.6:0/1461560973 conn(0x7f9aac103b50 msgr2=0x7f9aac105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.840+0000 7f9ab2e00700 1 -- 172.21.15.6:0/1461560973 shutdown_connections 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.840+0000 7f9ab2e00700 1 -- 172.21.15.6:0/1461560973 wait complete. 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.841+0000 7f9ab2e00700 1 Processor -- start 2024-04-26T01:33:34.451 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.842+0000 7f9ab2e00700 1 -- start start 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.842+0000 7f9ab2e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 0x7f9aac19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.842+0000 7f9ab2e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac10eb10 con 0x7f9aac07cd60 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.842+0000 7f9aabe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 0x7f9aac19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.842+0000 7f9aabe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 0x7f9aac19e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48832/0 (socket says 172.21.15.6:48832) 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.842+0000 7f9aabe00700 1 -- 172.21.15.6:0/762996732 learned_addr learned my addr 172.21.15.6:0/762996732 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.843+0000 7f9aabe00700 1 -- 172.21.15.6:0/762996732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aac19eb20 con 0x7f9aac07cd60 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.843+0000 7f9aabe00700 1 --2- 172.21.15.6:0/762996732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 0x7f9aac19e5e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c000c00 tx=0x7f9a9c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.843+0000 7f9aa8c00700 1 -- 172.21.15.6:0/762996732 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a9c003aa0 con 0x7f9aac07cd60 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.843+0000 7f9aa8c00700 1 -- 172.21.15.6:0/762996732 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a9c01b8f0 con 0x7f9aac07cd60 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.843+0000 7f9ab2e00700 1 -- 172.21.15.6:0/762996732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aac19edb0 con 0x7f9aac07cd60 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.844+0000 7f9aa8c00700 1 -- 172.21.15.6:0/762996732 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a9c023870 con 0x7f9aac07cd60 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.844+0000 7f9ab2e00700 1 -- 172.21.15.6:0/762996732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aac1a1aa0 con 0x7f9aac07cd60 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.845+0000 7f9aa8c00700 1 -- 172.21.15.6:0/762996732 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f9a9c023b70 con 0x7f9aac07cd60 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.845+0000 7f9ab2e00700 1 -- 172.21.15.6:0/762996732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aac0640a0 con 0x7f9aac07cd60 2024-04-26T01:33:34.452 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.845+0000 7f9aa8c00700 1 --2- 172.21.15.6:0/762996732 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f9a900395b0 0x7f9a9003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.846+0000 7f9aa8c00700 1 -- 172.21.15.6:0/762996732 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9a9c04de60 con 0x7f9aac07cd60 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.846+0000 7f9aab400700 1 --2- 172.21.15.6:0/762996732 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f9a900395b0 0x7f9a9003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.846+0000 7f9aab400700 1 --2- 172.21.15.6:0/762996732 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f9a900395b0 0x7f9a9003ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9a98006fd0 tx=0x7f9a98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:32.851+0000 7f9aa8c00700 1 -- 172.21.15.6:0/762996732 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a9c01f030 con 0x7f9aac07cd60 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:33.097+0000 7f9ab2e00700 1 -- 172.21.15.6:0/762996732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f9aac1a2090 con 0x7f9aac07cd60 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:33.103+0000 7f9aa8c00700 1 -- 172.21.15.6:0/762996732 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f9a9c01a3d0 con 0x7f9aac07cd60 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:33.106+0000 7f9ab2e00700 1 -- 172.21.15.6:0/762996732 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f9a900395b0 msgr2=0x7f9a9003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:33.106+0000 7f9ab2e00700 1 --2- 172.21.15.6:0/762996732 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f9a900395b0 0x7f9a9003ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9a98006fd0 tx=0x7f9a98006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:33.106+0000 7f9ab2e00700 1 -- 172.21.15.6:0/762996732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 msgr2=0x7f9aac19e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:33.106+0000 7f9ab2e00700 1 --2- 172.21.15.6:0/762996732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 0x7f9aac19e5e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c000c00 tx=0x7f9a9c011aa0 comp rx=0 tx=0).stop 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:33.107+0000 7f9ab2e00700 1 -- 172.21.15.6:0/762996732 shutdown_connections 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:33.107+0000 7f9ab2e00700 1 --2- 172.21.15.6:0/762996732 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f9a900395b0 0x7f9a9003ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:33.107+0000 7f9ab2e00700 1 --2- 172.21.15.6:0/762996732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac07cd60 0x7f9aac19e5e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:33.107+0000 7f9ab2e00700 1 -- 172.21.15.6:0/762996732 >> 172.21.15.6:0/762996732 conn(0x7f9aac103b50 msgr2=0x7f9aac104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:34.453 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:33.107+0000 7f9ab2e00700 1 -- 172.21.15.6:0/762996732 shutdown_connections 2024-04-26T01:33:34.454 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:33.107+0000 7f9ab2e00700 1 -- 172.21.15.6:0/762996732 wait complete. 2024-04-26T01:33:35.563 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:35 smithi006 ceph-mon[28419]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:35.563 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:35 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:36.152 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.141+0000 7f513a000700 1 Processor -- start 2024-04-26T01:33:36.158 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.142+0000 7f513a000700 1 -- start start 2024-04-26T01:33:36.158 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.142+0000 7f513a000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 0x7f5134108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.142+0000 7f513a000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5134108f20 con 0x7f5134108530 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.142+0000 7f5133400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 0x7f5134108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.143+0000 7f5133400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 0x7f5134108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48848/0 (socket says 172.21.15.6:48848) 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.143+0000 7f5133400700 1 -- 172.21.15.6:0/2468974947 learned_addr learned my addr 172.21.15.6:0/2468974947 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.143+0000 7f5133400700 1 -- 172.21.15.6:0/2468974947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5134109730 con 0x7f5134108530 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.143+0000 7f5133400700 1 --2- 172.21.15.6:0/2468974947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 0x7f5134108950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f51240098a0 tx=0x7f5124009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856907574f1ff889 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.144+0000 7f5132000700 1 -- 172.21.15.6:0/2468974947 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5124004160 con 0x7f5134108530 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.144+0000 7f5132000700 1 -- 172.21.15.6:0/2468974947 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5124004300 con 0x7f5134108530 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.144+0000 7f5132000700 1 -- 172.21.15.6:0/2468974947 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5124011a20 con 0x7f5134108530 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.145+0000 7f513a000700 1 -- 172.21.15.6:0/2468974947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 msgr2=0x7f5134108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.145+0000 7f513a000700 1 --2- 172.21.15.6:0/2468974947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 0x7f5134108950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f51240098a0 tx=0x7f5124009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.145+0000 7f513a000700 1 -- 172.21.15.6:0/2468974947 shutdown_connections 2024-04-26T01:33:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.145+0000 7f513a000700 1 --2- 172.21.15.6:0/2468974947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 0x7f5134108950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.145+0000 7f513a000700 1 -- 172.21.15.6:0/2468974947 >> 172.21.15.6:0/2468974947 conn(0x7f5134103b50 msgr2=0x7f5134105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.145+0000 7f513a000700 1 -- 172.21.15.6:0/2468974947 shutdown_connections 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.145+0000 7f513a000700 1 -- 172.21.15.6:0/2468974947 wait complete. 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.147+0000 7f513a000700 1 Processor -- start 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.147+0000 7f513a000700 1 -- start start 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.148+0000 7f513a000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 0x7f513407ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.148+0000 7f513a000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f513410cae0 con 0x7f5134108530 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.148+0000 7f5133400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 0x7f513407ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.148+0000 7f5133400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 0x7f513407ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48864/0 (socket says 172.21.15.6:48864) 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.148+0000 7f5133400700 1 -- 172.21.15.6:0/2260407193 learned_addr learned my addr 172.21.15.6:0/2260407193 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.149+0000 7f5133400700 1 -- 172.21.15.6:0/2260407193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513407cfb0 con 0x7f5134108530 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.149+0000 7f5133400700 1 --2- 172.21.15.6:0/2260407193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 0x7f513407ca70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5124009520 tx=0x7f5124011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.150+0000 7f512be00700 1 -- 172.21.15.6:0/2260407193 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51240039e0 con 0x7f5134108530 2024-04-26T01:33:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.150+0000 7f512be00700 1 -- 172.21.15.6:0/2260407193 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f512401b8f0 con 0x7f5134108530 2024-04-26T01:33:36.161 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.150+0000 7f513a000700 1 -- 172.21.15.6:0/2260407193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f513407b1a0 con 0x7f5134108530 2024-04-26T01:33:36.161 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.150+0000 7f512be00700 1 -- 172.21.15.6:0/2260407193 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5124019770 con 0x7f5134108530 2024-04-26T01:33:36.161 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.150+0000 7f513a000700 1 -- 172.21.15.6:0/2260407193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f513407b6e0 con 0x7f5134108530 2024-04-26T01:33:36.161 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.151+0000 7f513a000700 1 -- 172.21.15.6:0/2260407193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51340640a0 con 0x7f5134108530 2024-04-26T01:33:36.161 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.152+0000 7f512be00700 1 -- 172.21.15.6:0/2260407193 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f512401b440 con 0x7f5134108530 2024-04-26T01:33:36.161 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.152+0000 7f512be00700 1 --2- 172.21.15.6:0/2260407193 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f5118039600 0x7f511803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:36.161 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.152+0000 7f512be00700 1 -- 172.21.15.6:0/2260407193 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f512404df00 con 0x7f5134108530 2024-04-26T01:33:36.161 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.153+0000 7f5132a00700 1 --2- 172.21.15.6:0/2260407193 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f5118039600 0x7f511803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:36.161 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.153+0000 7f5132a00700 1 --2- 172.21.15.6:0/2260407193 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f5118039600 0x7f511803bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5120006fd0 tx=0x7f5120006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:36.161 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.158+0000 7f512be00700 1 -- 172.21.15.6:0/2260407193 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f512401f030 con 0x7f5134108530 2024-04-26T01:33:36.161 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.432+0000 7f513a000700 1 -- 172.21.15.6:0/2260407193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f513407bef0 con 0x7f5134108530 2024-04-26T01:33:36.162 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.437+0000 7f512be00700 1 -- 172.21.15.6:0/2260407193 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f512404d850 con 0x7f5134108530 2024-04-26T01:33:36.162 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.441+0000 7f513a000700 1 -- 172.21.15.6:0/2260407193 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f5118039600 msgr2=0x7f511803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:36.162 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.441+0000 7f513a000700 1 --2- 172.21.15.6:0/2260407193 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f5118039600 0x7f511803bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5120006fd0 tx=0x7f5120006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:36.162 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.441+0000 7f513a000700 1 -- 172.21.15.6:0/2260407193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 msgr2=0x7f513407ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:36.162 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.441+0000 7f513a000700 1 --2- 172.21.15.6:0/2260407193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 0x7f513407ca70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5124009520 tx=0x7f5124011aa0 comp rx=0 tx=0).stop 2024-04-26T01:33:36.162 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.441+0000 7f513a000700 1 -- 172.21.15.6:0/2260407193 shutdown_connections 2024-04-26T01:33:36.162 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.441+0000 7f513a000700 1 --2- 172.21.15.6:0/2260407193 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f5118039600 0x7f511803bac0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:36.162 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.441+0000 7f513a000700 1 --2- 172.21.15.6:0/2260407193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5134108530 0x7f513407ca70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:36.162 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.441+0000 7f513a000700 1 -- 172.21.15.6:0/2260407193 >> 172.21.15.6:0/2260407193 conn(0x7f5134103b50 msgr2=0x7f51341045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:36.162 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.441+0000 7f513a000700 1 -- 172.21.15.6:0/2260407193 shutdown_connections 2024-04-26T01:33:36.162 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:35.441+0000 7f513a000700 1 -- 172.21.15.6:0/2260407193 wait complete. 2024-04-26T01:33:36.163 INFO:teuthology.orchestra.run.smithi006.stdout:Enabling the dashboard module... 2024-04-26T01:33:36.715 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:36 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2260407193' entity='client.admin' 2024-04-26T01:33:37.563 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:37 smithi006 ceph-mon[28419]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:37.563 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:37 smithi006 ceph-mon[28419]: from='mgr.14120 172.21.15.6:0/1690632532' entity='mgr.x' 2024-04-26T01:33:37.563 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:37 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3703771806' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T01:33:38.061 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:37 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: ignoring --setuser ceph since I am not root 2024-04-26T01:33:38.061 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:37 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: ignoring --setgroup ceph since I am not root 2024-04-26T01:33:38.061 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:38 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:38.065+0000 7f1b5f0b7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:33:38.588 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:38 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:38.322+0000 7f1b5f0b7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:33:38.588 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:38 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:38.458+0000 7f1b5f0b7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:33:38.595 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.892+0000 7fdfc5400700 1 Processor -- start 2024-04-26T01:33:38.596 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.892+0000 7fdfc5400700 1 -- start start 2024-04-26T01:33:38.597 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.892+0000 7fdfc5400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 0x7fdfc010b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:38.597 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.892+0000 7fdfc5400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc010b8e0 con 0x7fdfc0108fb0 2024-04-26T01:33:38.597 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.893+0000 7fdfbe800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 0x7fdfc010b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:38.597 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.893+0000 7fdfbe800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 0x7fdfc010b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48880/0 (socket says 172.21.15.6:48880) 2024-04-26T01:33:38.597 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.893+0000 7fdfbe800700 1 -- 172.21.15.6:0/756504205 learned_addr learned my addr 172.21.15.6:0/756504205 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:38.597 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.893+0000 7fdfbe800700 1 -- 172.21.15.6:0/756504205 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc010ba60 con 0x7fdfc0108fb0 2024-04-26T01:33:38.597 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.894+0000 7fdfbe800700 1 --2- 172.21.15.6:0/756504205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 0x7fdfc010b3a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdfb00098a0 tx=0x7fdfb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05e09c61a32eded server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:38.600 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.894+0000 7fdfbde00700 1 -- 172.21.15.6:0/756504205 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfb0004160 con 0x7fdfc0108fb0 2024-04-26T01:33:38.600 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.894+0000 7fdfbde00700 1 -- 172.21.15.6:0/756504205 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfb0004300 con 0x7fdfc0108fb0 2024-04-26T01:33:38.600 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.894+0000 7fdfbde00700 1 -- 172.21.15.6:0/756504205 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfb0011a20 con 0x7fdfc0108fb0 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.895+0000 7fdfc5400700 1 -- 172.21.15.6:0/756504205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 msgr2=0x7fdfc010b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.895+0000 7fdfc5400700 1 --2- 172.21.15.6:0/756504205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 0x7fdfc010b3a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdfb00098a0 tx=0x7fdfb0009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.895+0000 7fdfc5400700 1 -- 172.21.15.6:0/756504205 shutdown_connections 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.895+0000 7fdfc5400700 1 --2- 172.21.15.6:0/756504205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 0x7fdfc010b3a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.895+0000 7fdfc5400700 1 -- 172.21.15.6:0/756504205 >> 172.21.15.6:0/756504205 conn(0x7fdfc0102a60 msgr2=0x7fdfc0104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.895+0000 7fdfc5400700 1 -- 172.21.15.6:0/756504205 shutdown_connections 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.895+0000 7fdfc5400700 1 -- 172.21.15.6:0/756504205 wait complete. 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.896+0000 7fdfc5400700 1 Processor -- start 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.896+0000 7fdfc5400700 1 -- start start 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.897+0000 7fdfc5400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 0x7fdfc0108830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.897+0000 7fdfc5400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc010ca00 con 0x7fdfc0108fb0 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.897+0000 7fdfbe800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 0x7fdfc0108830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:38.601 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.897+0000 7fdfbe800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 0x7fdfc0108830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48896/0 (socket says 172.21.15.6:48896) 2024-04-26T01:33:38.602 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.897+0000 7fdfbe800700 1 -- 172.21.15.6:0/3703771806 learned_addr learned my addr 172.21.15.6:0/3703771806 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:38.602 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.897+0000 7fdfbe800700 1 -- 172.21.15.6:0/3703771806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc0108d70 con 0x7fdfc0108fb0 2024-04-26T01:33:38.602 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.898+0000 7fdfbe800700 1 --2- 172.21.15.6:0/3703771806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 0x7fdfc0108830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdfb00116e0 tx=0x7fdfb0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:38.602 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.898+0000 7fdfb7400700 1 -- 172.21.15.6:0/3703771806 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfb0003b20 con 0x7fdfc0108fb0 2024-04-26T01:33:38.602 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.898+0000 7fdfb7400700 1 -- 172.21.15.6:0/3703771806 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfb001b8f0 con 0x7fdfc0108fb0 2024-04-26T01:33:38.602 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.898+0000 7fdfc5400700 1 -- 172.21.15.6:0/3703771806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc0106f60 con 0x7fdfc0108fb0 2024-04-26T01:33:38.602 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.898+0000 7fdfb7400700 1 -- 172.21.15.6:0/3703771806 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfb00238c0 con 0x7fdfc0108fb0 2024-04-26T01:33:38.602 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.898+0000 7fdfc5400700 1 -- 172.21.15.6:0/3703771806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc01074a0 con 0x7fdfc0108fb0 2024-04-26T01:33:38.602 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.899+0000 7fdfc5400700 1 -- 172.21.15.6:0/3703771806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfc00640a0 con 0x7fdfc0108fb0 2024-04-26T01:33:38.602 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.900+0000 7fdfb7400700 1 -- 172.21.15.6:0/3703771806 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fdfb0023b70 con 0x7fdfc0108fb0 2024-04-26T01:33:38.602 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.900+0000 7fdfb7400700 1 --2- 172.21.15.6:0/3703771806 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7fdfa4039560 0x7fdfa403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:38.602 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.900+0000 7fdfb7400700 1 -- 172.21.15.6:0/3703771806 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdfb004e1b0 con 0x7fdfc0108fb0 2024-04-26T01:33:38.603 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.900+0000 7fdfb7e00700 1 --2- 172.21.15.6:0/3703771806 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7fdfa4039560 0x7fdfa403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:38.603 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.901+0000 7fdfb7e00700 1 --2- 172.21.15.6:0/3703771806 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7fdfa4039560 0x7fdfa403ba20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8006fd0 tx=0x7fdfa8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:38.603 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:36.906+0000 7fdfb7400700 1 -- 172.21.15.6:0/3703771806 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdfb001f030 con 0x7fdfc0108fb0 2024-04-26T01:33:38.603 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.202+0000 7fdfc5400700 1 -- 172.21.15.6:0/3703771806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fdfc0107af0 con 0x7fdfc0108fb0 2024-04-26T01:33:38.603 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.742+0000 7fdfb7400700 1 -- 172.21.15.6:0/3703771806 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fdfb002c910 con 0x7fdfc0108fb0 2024-04-26T01:33:38.603 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.746+0000 7fdfb7400700 1 -- 172.21.15.6:0/3703771806 <== mon.0 v2:172.21.15.6:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fdfb002c3f0 con 0x7fdfc0108fb0 2024-04-26T01:33:38.603 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.748+0000 7fdfc5400700 1 -- 172.21.15.6:0/3703771806 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7fdfa4039560 msgr2=0x7fdfa403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:38.603 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.748+0000 7fdfc5400700 1 --2- 172.21.15.6:0/3703771806 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7fdfa4039560 0x7fdfa403ba20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8006fd0 tx=0x7fdfa8006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:38.604 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.748+0000 7fdfc5400700 1 -- 172.21.15.6:0/3703771806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 msgr2=0x7fdfc0108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:38.604 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.748+0000 7fdfc5400700 1 --2- 172.21.15.6:0/3703771806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 0x7fdfc0108830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdfb00116e0 tx=0x7fdfb0011aa0 comp rx=0 tx=0).stop 2024-04-26T01:33:38.604 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.748+0000 7fdfc5400700 1 -- 172.21.15.6:0/3703771806 shutdown_connections 2024-04-26T01:33:38.604 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.748+0000 7fdfc5400700 1 --2- 172.21.15.6:0/3703771806 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7fdfa4039560 0x7fdfa403ba20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:38.604 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.748+0000 7fdfc5400700 1 --2- 172.21.15.6:0/3703771806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfc0108fb0 0x7fdfc0108830 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:38.604 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.748+0000 7fdfc5400700 1 -- 172.21.15.6:0/3703771806 >> 172.21.15.6:0/3703771806 conn(0x7fdfc0102a60 msgr2=0x7fdfc0103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:38.604 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.748+0000 7fdfc5400700 1 -- 172.21.15.6:0/3703771806 shutdown_connections 2024-04-26T01:33:38.604 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:37.748+0000 7fdfc5400700 1 -- 172.21.15.6:0/3703771806 wait complete. 2024-04-26T01:33:38.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:38 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3703771806' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T01:33:38.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:38 smithi006 ceph-mon[28419]: mgrmap e9: x(active, since 26s) 2024-04-26T01:33:39.144 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:38 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:38.888+0000 7f1b5f0b7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:33:39.144 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:39 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:39.005+0000 7f1b5f0b7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:33:40.625 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:40 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2952702401' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T01:33:41.063 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:40 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:40.630+0000 7f1b5f0b7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:33:41.064 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:40 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:40.865+0000 7f1b5f0b7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:33:41.110 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:33:41.110 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T01:33:41.110 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T01:33:41.111 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T01:33:41.111 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T01:33:41.111 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:33:41.111 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.033+0000 7f8985e00700 1 Processor -- start 2024-04-26T01:33:41.111 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.033+0000 7f8985e00700 1 -- start start 2024-04-26T01:33:41.111 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.033+0000 7f8985e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 0x7f898007c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:41.111 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.033+0000 7f8985e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f898007c920 con 0x7f898007dea0 2024-04-26T01:33:41.111 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.034+0000 7f897ea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 0x7f898007c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:41.111 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.034+0000 7f897ea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 0x7f898007c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51880/0 (socket says 172.21.15.6:51880) 2024-04-26T01:33:41.111 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.034+0000 7f897ea00700 1 -- 172.21.15.6:0/2698176024 learned_addr learned my addr 172.21.15.6:0/2698176024 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:41.111 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.034+0000 7f897ea00700 1 -- 172.21.15.6:0/2698176024 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898007caa0 con 0x7f898007dea0 2024-04-26T01:33:41.111 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.035+0000 7f897ea00700 1 --2- 172.21.15.6:0/2698176024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 0x7f898007c350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f89700098a0 tx=0x7f8970009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edeb559232d43862 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.035+0000 7f897d600700 1 -- 172.21.15.6:0/2698176024 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8970004160 con 0x7f898007dea0 2024-04-26T01:33:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.035+0000 7f897d600700 1 -- 172.21.15.6:0/2698176024 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8970004300 con 0x7f898007dea0 2024-04-26T01:33:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.035+0000 7f897d600700 1 -- 172.21.15.6:0/2698176024 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8970011a20 con 0x7f898007dea0 2024-04-26T01:33:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.036+0000 7f8985e00700 1 -- 172.21.15.6:0/2698176024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 msgr2=0x7f898007c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.036+0000 7f8985e00700 1 --2- 172.21.15.6:0/2698176024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 0x7f898007c350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f89700098a0 tx=0x7f8970009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.036+0000 7f8985e00700 1 -- 172.21.15.6:0/2698176024 shutdown_connections 2024-04-26T01:33:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.036+0000 7f8985e00700 1 --2- 172.21.15.6:0/2698176024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 0x7f898007c350 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.036+0000 7f8985e00700 1 -- 172.21.15.6:0/2698176024 >> 172.21.15.6:0/2698176024 conn(0x7f8980103b40 msgr2=0x7f8980105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.036+0000 7f8985e00700 1 -- 172.21.15.6:0/2698176024 shutdown_connections 2024-04-26T01:33:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.036+0000 7f8985e00700 1 -- 172.21.15.6:0/2698176024 wait complete. 2024-04-26T01:33:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.037+0000 7f8985e00700 1 Processor -- start 2024-04-26T01:33:41.112 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.038+0000 7f8985e00700 1 -- start start 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.038+0000 7f8985e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 0x7f89801a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.038+0000 7f8985e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f898010a710 con 0x7f898007dea0 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.038+0000 7f897ea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 0x7f89801a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.038+0000 7f897ea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 0x7f89801a2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51886/0 (socket says 172.21.15.6:51886) 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.038+0000 7f897ea00700 1 -- 172.21.15.6:0/2952702401 learned_addr learned my addr 172.21.15.6:0/2952702401 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.039+0000 7f897ea00700 1 -- 172.21.15.6:0/2952702401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89801a2f70 con 0x7f898007dea0 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.039+0000 7f897ea00700 1 --2- 172.21.15.6:0/2952702401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 0x7f89801a2a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8970004660 tx=0x7f8970011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.039+0000 7f8984a00700 1 -- 172.21.15.6:0/2952702401 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8970003aa0 con 0x7f898007dea0 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.039+0000 7f8984a00700 1 -- 172.21.15.6:0/2952702401 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f897001b8f0 con 0x7f898007dea0 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.039+0000 7f8985e00700 1 -- 172.21.15.6:0/2952702401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89801a3200 con 0x7f898007dea0 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.039+0000 7f8984a00700 1 -- 172.21.15.6:0/2952702401 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8970023900 con 0x7f898007dea0 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.040+0000 7f8985e00700 1 -- 172.21.15.6:0/2952702401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89801a5ef0 con 0x7f898007dea0 2024-04-26T01:33:41.113 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.041+0000 7f8984a00700 1 -- 172.21.15.6:0/2952702401 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f8970023b70 con 0x7f898007dea0 2024-04-26T01:33:41.114 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.041+0000 7f8975600700 1 -- 172.21.15.6:0/2952702401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89800640a0 con 0x7f898007dea0 2024-04-26T01:33:41.114 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.041+0000 7f8984a00700 1 --2- 172.21.15.6:0/2952702401 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f8964039660 0x7f896403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:41.114 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.041+0000 7f8984a00700 1 -- 172.21.15.6:0/2952702401 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f897004dfa0 con 0x7f898007dea0 2024-04-26T01:33:41.114 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.041+0000 7f897e000700 1 -- 172.21.15.6:0/2952702401 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f8964039660 msgr2=0x7f896403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/635066163 2024-04-26T01:33:41.114 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.042+0000 7f897e000700 1 --2- 172.21.15.6:0/2952702401 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f8964039660 0x7f896403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:33:41.114 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.047+0000 7f8984a00700 1 -- 172.21.15.6:0/2952702401 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f897001f030 con 0x7f898007dea0 2024-04-26T01:33:41.114 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.243+0000 7f897e000700 1 -- 172.21.15.6:0/2952702401 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f8964039660 msgr2=0x7f896403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/635066163 2024-04-26T01:33:41.114 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.243+0000 7f897e000700 1 --2- 172.21.15.6:0/2952702401 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f8964039660 0x7f896403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:33:41.114 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.326+0000 7f8975600700 1 -- 172.21.15.6:0/2952702401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f8980075460 con 0x7f898007dea0 2024-04-26T01:33:41.114 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.327+0000 7f8984a00700 1 -- 172.21.15.6:0/2952702401 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f897001bde0 con 0x7f898007dea0 2024-04-26T01:33:41.114 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.330+0000 7f8975600700 1 -- 172.21.15.6:0/2952702401 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f8964039660 msgr2=0x7f896403bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:33:41.114 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.330+0000 7f8975600700 1 --2- 172.21.15.6:0/2952702401 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f8964039660 0x7f896403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:41.115 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.330+0000 7f8975600700 1 -- 172.21.15.6:0/2952702401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 msgr2=0x7f89801a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:41.115 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.330+0000 7f8975600700 1 --2- 172.21.15.6:0/2952702401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 0x7f89801a2a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8970004660 tx=0x7f8970011aa0 comp rx=0 tx=0).stop 2024-04-26T01:33:41.115 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.330+0000 7f8975600700 1 -- 172.21.15.6:0/2952702401 shutdown_connections 2024-04-26T01:33:41.115 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.330+0000 7f8975600700 1 --2- 172.21.15.6:0/2952702401 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7f8964039660 0x7f896403bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:41.115 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.330+0000 7f8975600700 1 --2- 172.21.15.6:0/2952702401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f898007dea0 0x7f89801a2a30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:41.115 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.330+0000 7f8975600700 1 -- 172.21.15.6:0/2952702401 >> 172.21.15.6:0/2952702401 conn(0x7f8980103b40 msgr2=0x7f8980105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:41.115 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.330+0000 7f8975600700 1 -- 172.21.15.6:0/2952702401 shutdown_connections 2024-04-26T01:33:41.115 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:40.330+0000 7f8975600700 1 -- 172.21.15.6:0/2952702401 wait complete. 2024-04-26T01:33:41.115 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for the mgr to restart... 2024-04-26T01:33:41.115 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr epoch 9... 2024-04-26T01:33:41.364 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:41 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:41.085+0000 7f1b5f0b7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:33:41.621 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:41 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:41.496+0000 7f1b5f0b7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:33:41.885 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:41 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:41.625+0000 7f1b5f0b7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:33:42.148 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:41 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:41.889+0000 7f1b5f0b7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:33:42.149 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:42 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:42.152+0000 7f1b5f0b7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:33:43.251 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:42 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:42.962+0000 7f1b5f0b7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:33:43.813 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:43 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:43.532+0000 7f1b5f0b7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:33:43.814 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:43 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:43.662+0000 7f1b5f0b7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:33:44.313 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:43 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:43.936+0000 7f1b5f0b7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:33:44.853 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:44 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:44.606+0000 7f1b5f0b7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:33:44.853 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:44 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:44.734+0000 7f1b5f0b7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:33:45.245 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:45.065+0000 7f1b5f0b7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:33:45.245 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:45.249+0000 7f1b5f0b7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:33:45.538 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:45.382+0000 7f1b5f0b7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:33:45.538 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:45.541+0000 7f1b5f0b7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:33:46.314 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:45.925+0000 7f1b5f0b7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:33:47.013 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:46 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:46.721+0000 7f1b5f0b7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:33:47.313 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:33:47 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:33:47.017+0000 7f1b5f0b7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:33:47.313 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:47 smithi006 ceph-mon[28419]: Active manager daemon x restarted 2024-04-26T01:33:47.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:47 smithi006 ceph-mon[28419]: Activating manager daemon x 2024-04-26T01:33:47.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:47 smithi006 ceph-mon[28419]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T01:33:47.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:47 smithi006 ceph-mon[28419]: mgrmap e10: x(active, starting, since 0.00468607s) 2024-04-26T01:33:47.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:33:47.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:33:47.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:33:47.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:33:47.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:33:47.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:47 smithi006 ceph-mon[28419]: Manager daemon x is now available 2024-04-26T01:33:48.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:48.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:33:48.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:33:48.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:48 smithi006 ceph-mon[28419]: [26/Apr/2024:01:33:47] ENGINE Bus STARTING 2024-04-26T01:33:48.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:48 smithi006 ceph-mon[28419]: [26/Apr/2024:01:33:47] ENGINE Serving on https://172.21.15.6:7150 2024-04-26T01:33:48.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:48 smithi006 ceph-mon[28419]: [26/Apr/2024:01:33:47] ENGINE Serving on http://172.21.15.6:8765 2024-04-26T01:33:48.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:48 smithi006 ceph-mon[28419]: [26/Apr/2024:01:33:47] ENGINE Bus STARTED 2024-04-26T01:33:48.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:48 smithi006 ceph-mon[28419]: mgrmap e11: x(active, since 1.01274s) 2024-04-26T01:33:48.761 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:33:48.761 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T01:33:48.761 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T01:33:48.761 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:33:48.761 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.844+0000 7ffb3b000700 1 Processor -- start 2024-04-26T01:33:48.761 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.844+0000 7ffb3b000700 1 -- start start 2024-04-26T01:33:48.761 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.845+0000 7ffb3b000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 0x7ffb34107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:48.761 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.845+0000 7ffb3b000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ffb34107e30 con 0x7ffb34107440 2024-04-26T01:33:48.761 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.845+0000 7ffb33e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 0x7ffb34107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:48.762 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.845+0000 7ffb33e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 0x7ffb34107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51892/0 (socket says 172.21.15.6:51892) 2024-04-26T01:33:48.762 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.845+0000 7ffb33e00700 1 -- 172.21.15.6:0/1736446962 learned_addr learned my addr 172.21.15.6:0/1736446962 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:48.762 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.846+0000 7ffb33e00700 1 -- 172.21.15.6:0/1736446962 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb34108640 con 0x7ffb34107440 2024-04-26T01:33:48.762 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.846+0000 7ffb33e00700 1 --2- 172.21.15.6:0/1736446962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 0x7ffb34107860 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffb240098a0 tx=0x7ffb24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56065a3e9e057067 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:48.762 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.846+0000 7ffb32a00700 1 -- 172.21.15.6:0/1736446962 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb24004160 con 0x7ffb34107440 2024-04-26T01:33:48.762 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.846+0000 7ffb32a00700 1 -- 172.21.15.6:0/1736446962 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb24004300 con 0x7ffb34107440 2024-04-26T01:33:48.762 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.846+0000 7ffb32a00700 1 -- 172.21.15.6:0/1736446962 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb24011a20 con 0x7ffb34107440 2024-04-26T01:33:48.762 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.847+0000 7ffb3b000700 1 -- 172.21.15.6:0/1736446962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 msgr2=0x7ffb34107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:48.762 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.847+0000 7ffb3b000700 1 --2- 172.21.15.6:0/1736446962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 0x7ffb34107860 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffb240098a0 tx=0x7ffb24009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:48.762 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.847+0000 7ffb3b000700 1 -- 172.21.15.6:0/1736446962 shutdown_connections 2024-04-26T01:33:48.762 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.847+0000 7ffb3b000700 1 --2- 172.21.15.6:0/1736446962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 0x7ffb34107860 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:48.762 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.847+0000 7ffb3b000700 1 -- 172.21.15.6:0/1736446962 >> 172.21.15.6:0/1736446962 conn(0x7ffb34102a40 msgr2=0x7ffb34104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.847+0000 7ffb3b000700 1 -- 172.21.15.6:0/1736446962 shutdown_connections 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.847+0000 7ffb3b000700 1 -- 172.21.15.6:0/1736446962 wait complete. 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.848+0000 7ffb3b000700 1 Processor -- start 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.848+0000 7ffb3b000700 1 -- start start 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.849+0000 7ffb3b000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 0x7ffb3419c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.849+0000 7ffb3b000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3410b9f0 con 0x7ffb34107440 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.849+0000 7ffb33e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 0x7ffb3419c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.849+0000 7ffb33e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 0x7ffb3419c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51908/0 (socket says 172.21.15.6:51908) 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.849+0000 7ffb33e00700 1 -- 172.21.15.6:0/4096023046 learned_addr learned my addr 172.21.15.6:0/4096023046 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.849+0000 7ffb33e00700 1 -- 172.21.15.6:0/4096023046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb3419c9d0 con 0x7ffb34107440 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.850+0000 7ffb33e00700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 0x7ffb3419c490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb24009520 tx=0x7ffb24011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.850+0000 7ffb30c00700 1 -- 172.21.15.6:0/4096023046 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb240039e0 con 0x7ffb34107440 2024-04-26T01:33:48.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.850+0000 7ffb30c00700 1 -- 172.21.15.6:0/4096023046 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb2401b8f0 con 0x7ffb34107440 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.850+0000 7ffb3b000700 1 -- 172.21.15.6:0/4096023046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb3419cc60 con 0x7ffb34107440 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.850+0000 7ffb30c00700 1 -- 172.21.15.6:0/4096023046 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb24019770 con 0x7ffb34107440 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.850+0000 7ffb3b000700 1 -- 172.21.15.6:0/4096023046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb3419f950 con 0x7ffb34107440 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.851+0000 7ffb30c00700 1 -- 172.21.15.6:0/4096023046 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7ffb2401b440 con 0x7ffb34107440 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.852+0000 7ffb30c00700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 0x7ffb1803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.852+0000 7ffb30c00700 1 -- 172.21.15.6:0/4096023046 --> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffb1803c220 con 0x7ffb18039650 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.852+0000 7ffb30c00700 1 -- 172.21.15.6:0/4096023046 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ffb2404df90 con 0x7ffb34107440 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.852+0000 7ffb33400700 1 -- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 msgr2=0x7ffb1803bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/635066163 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:41.852+0000 7ffb33400700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 0x7ffb1803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:42.053+0000 7ffb33400700 1 -- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 msgr2=0x7ffb1803bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/635066163 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:42.053+0000 7ffb33400700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 0x7ffb1803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:42.454+0000 7ffb33400700 1 -- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 msgr2=0x7ffb1803bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/635066163 2024-04-26T01:33:48.764 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:42.454+0000 7ffb33400700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 0x7ffb1803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:43.255+0000 7ffb33400700 1 -- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 msgr2=0x7ffb1803bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/635066163 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:43.255+0000 7ffb33400700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 0x7ffb1803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:44.857+0000 7ffb33400700 1 -- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 msgr2=0x7ffb1803bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/635066163 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:44.857+0000 7ffb33400700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 0x7ffb1803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:47.025+0000 7ffb30c00700 1 -- 172.21.15.6:0/4096023046 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7ffb24019a20 con 0x7ffb34107440 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:47.026+0000 7ffb30c00700 1 -- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 msgr2=0x7ffb1803bb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:47.026+0000 7ffb30c00700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 0x7ffb1803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.031+0000 7ffb30c00700 1 -- 172.21.15.6:0/4096023046 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7ffb2404fd70 con 0x7ffb34107440 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.031+0000 7ffb30c00700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ffb1803cfe0 0x7ffb1803f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.031+0000 7ffb30c00700 1 -- 172.21.15.6:0/4096023046 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffb24010580 con 0x7ffb1803cfe0 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.032+0000 7ffb33400700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ffb1803cfe0 0x7ffb1803f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.034+0000 7ffb33400700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ffb1803cfe0 0x7ffb1803f3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffb20003a10 tx=0x7ffb20009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:48.765 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.035+0000 7ffb30c00700 1 -- 172.21.15.6:0/4096023046 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7ffb24010580 con 0x7ffb1803cfe0 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.046+0000 7ffb2a000700 1 -- 172.21.15.6:0/4096023046 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ffb0c002600 con 0x7ffb1803cfe0 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.046+0000 7ffb30c00700 1 -- 172.21.15.6:0/4096023046 <== mgr.14152 v2:172.21.15.6:6800/809793211 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7ffb0c002600 con 0x7ffb1803cfe0 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.047+0000 7ffb2a000700 1 -- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ffb1803cfe0 msgr2=0x7ffb1803f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.047+0000 7ffb2a000700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ffb1803cfe0 0x7ffb1803f3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffb20003a10 tx=0x7ffb20009270 comp rx=0 tx=0).stop 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.047+0000 7ffb2a000700 1 -- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 msgr2=0x7ffb3419c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.047+0000 7ffb2a000700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 0x7ffb3419c490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb24009520 tx=0x7ffb24011aa0 comp rx=0 tx=0).stop 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.047+0000 7ffb2a000700 1 -- 172.21.15.6:0/4096023046 shutdown_connections 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.047+0000 7ffb2a000700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ffb1803cfe0 0x7ffb1803f3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.047+0000 7ffb2a000700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb34107440 0x7ffb3419c490 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.047+0000 7ffb2a000700 1 --2- 172.21.15.6:0/4096023046 >> [v2:172.21.15.6:6800/635066163,v1:172.21.15.6:6801/635066163] conn(0x7ffb18039650 0x7ffb1803bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.047+0000 7ffb2a000700 1 -- 172.21.15.6:0/4096023046 >> 172.21.15.6:0/4096023046 conn(0x7ffb34102a40 msgr2=0x7ffb34103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.048+0000 7ffb2a000700 1 -- 172.21.15.6:0/4096023046 shutdown_connections 2024-04-26T01:33:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:48.048+0000 7ffb2a000700 1 -- 172.21.15.6:0/4096023046 wait complete. 2024-04-26T01:33:48.767 INFO:teuthology.orchestra.run.smithi006.stdout:mgr epoch 9 is available 2024-04-26T01:33:48.767 INFO:teuthology.orchestra.run.smithi006.stdout:Generating a dashboard self-signed certificate... 2024-04-26T01:33:49.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:49 smithi006 ceph-mon[28419]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T01:33:49.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:49 smithi006 ceph-mon[28419]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T01:33:50.242 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:50 smithi006 ceph-mon[28419]: mgrmap e12: x(active, since 2s) 2024-04-26T01:33:50.242 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:50 smithi006 ceph-mon[28419]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:50.242 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:33:50.242 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:33:50.242 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:33:50.612 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T01:33:50.612 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.338+0000 7f18b7000700 1 Processor -- start 2024-04-26T01:33:50.612 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.338+0000 7f18b7000700 1 -- start start 2024-04-26T01:33:50.612 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.339+0000 7f18b7000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 0x7f18b0108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:50.613 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.339+0000 7f18b7000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18b0108f20 con 0x7f18b0108530 2024-04-26T01:33:50.613 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.339+0000 7f18afe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 0x7f18b0108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:50.613 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.339+0000 7f18afe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 0x7f18b0108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33086/0 (socket says 172.21.15.6:33086) 2024-04-26T01:33:50.613 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.339+0000 7f18afe00700 1 -- 172.21.15.6:0/2943454117 learned_addr learned my addr 172.21.15.6:0/2943454117 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:50.613 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.339+0000 7f18afe00700 1 -- 172.21.15.6:0/2943454117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b0109730 con 0x7f18b0108530 2024-04-26T01:33:50.613 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.340+0000 7f18afe00700 1 --2- 172.21.15.6:0/2943454117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 0x7f18b0108950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f18a00098a0 tx=0x7f18a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6544d90700f70125 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:50.614 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.340+0000 7f18aea00700 1 -- 172.21.15.6:0/2943454117 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18a0004160 con 0x7f18b0108530 2024-04-26T01:33:50.614 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.340+0000 7f18aea00700 1 -- 172.21.15.6:0/2943454117 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18a0004300 con 0x7f18b0108530 2024-04-26T01:33:50.614 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.340+0000 7f18aea00700 1 -- 172.21.15.6:0/2943454117 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18a0011a20 con 0x7f18b0108530 2024-04-26T01:33:50.614 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.341+0000 7f18b7000700 1 -- 172.21.15.6:0/2943454117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 msgr2=0x7f18b0108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:50.614 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.341+0000 7f18b7000700 1 --2- 172.21.15.6:0/2943454117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 0x7f18b0108950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f18a00098a0 tx=0x7f18a0009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.341+0000 7f18b7000700 1 -- 172.21.15.6:0/2943454117 shutdown_connections 2024-04-26T01:33:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.341+0000 7f18b7000700 1 --2- 172.21.15.6:0/2943454117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 0x7f18b0108950 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.341+0000 7f18b7000700 1 -- 172.21.15.6:0/2943454117 >> 172.21.15.6:0/2943454117 conn(0x7f18b0103b50 msgr2=0x7f18b0105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.341+0000 7f18b7000700 1 -- 172.21.15.6:0/2943454117 shutdown_connections 2024-04-26T01:33:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.341+0000 7f18b7000700 1 -- 172.21.15.6:0/2943454117 wait complete. 2024-04-26T01:33:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.342+0000 7f18b7000700 1 Processor -- start 2024-04-26T01:33:50.616 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.343+0000 7f18b7000700 1 -- start start 2024-04-26T01:33:50.616 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.343+0000 7f18b7000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 0x7f18b019c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:50.616 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.343+0000 7f18b7000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18b010cae0 con 0x7f18b0108530 2024-04-26T01:33:50.616 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.343+0000 7f18afe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 0x7f18b019c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:50.616 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.343+0000 7f18afe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 0x7f18b019c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33092/0 (socket says 172.21.15.6:33092) 2024-04-26T01:33:50.616 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.343+0000 7f18afe00700 1 -- 172.21.15.6:0/2427565963 learned_addr learned my addr 172.21.15.6:0/2427565963 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:50.617 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.344+0000 7f18afe00700 1 -- 172.21.15.6:0/2427565963 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b019c940 con 0x7f18b0108530 2024-04-26T01:33:50.617 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.344+0000 7f18afe00700 1 --2- 172.21.15.6:0/2427565963 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 0x7f18b019c400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f18a00116e0 tx=0x7f18a0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:50.617 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.344+0000 7f18acc00700 1 -- 172.21.15.6:0/2427565963 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18a00039e0 con 0x7f18b0108530 2024-04-26T01:33:50.617 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.344+0000 7f18b7000700 1 -- 172.21.15.6:0/2427565963 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b019cbd0 con 0x7f18b0108530 2024-04-26T01:33:50.617 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.344+0000 7f18acc00700 1 -- 172.21.15.6:0/2427565963 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18a001b8f0 con 0x7f18b0108530 2024-04-26T01:33:50.617 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.344+0000 7f18acc00700 1 -- 172.21.15.6:0/2427565963 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18a0023760 con 0x7f18b0108530 2024-04-26T01:33:50.617 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.345+0000 7f18b7000700 1 -- 172.21.15.6:0/2427565963 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b019f8c0 con 0x7f18b0108530 2024-04-26T01:33:50.618 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.346+0000 7f18acc00700 1 -- 172.21.15.6:0/2427565963 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f18a001b440 con 0x7f18b0108530 2024-04-26T01:33:50.618 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.346+0000 7f18b7000700 1 -- 172.21.15.6:0/2427565963 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b00640a0 con 0x7f18b0108530 2024-04-26T01:33:50.618 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.346+0000 7f18acc00700 1 --2- 172.21.15.6:0/2427565963 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f18940395b0 0x7f189403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:50.618 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.346+0000 7f18acc00700 1 -- 172.21.15.6:0/2427565963 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f18a004e1a0 con 0x7f18b0108530 2024-04-26T01:33:50.618 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.347+0000 7f18af400700 1 --2- 172.21.15.6:0/2427565963 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f18940395b0 0x7f189403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:50.619 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.347+0000 7f18af400700 1 --2- 172.21.15.6:0/2427565963 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f18940395b0 0x7f189403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f189c006fd0 tx=0x7f189c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:50.619 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.352+0000 7f18acc00700 1 -- 172.21.15.6:0/2427565963 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f18a001f030 con 0x7f18b0108530 2024-04-26T01:33:50.619 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.597+0000 7f18b7000700 1 -- 172.21.15.6:0/2427565963 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f18b0104cb0 con 0x7f18940395b0 2024-04-26T01:33:50.619 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.854+0000 7f18acc00700 1 -- 172.21.15.6:0/2427565963 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f18b0104cb0 con 0x7f18940395b0 2024-04-26T01:33:50.619 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.857+0000 7f18b7000700 1 -- 172.21.15.6:0/2427565963 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f18940395b0 msgr2=0x7f189403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:50.619 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.857+0000 7f18b7000700 1 --2- 172.21.15.6:0/2427565963 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f18940395b0 0x7f189403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f189c006fd0 tx=0x7f189c006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:50.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.857+0000 7f18b7000700 1 -- 172.21.15.6:0/2427565963 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 msgr2=0x7f18b019c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:50.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.857+0000 7f18b7000700 1 --2- 172.21.15.6:0/2427565963 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 0x7f18b019c400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f18a00116e0 tx=0x7f18a0011aa0 comp rx=0 tx=0).stop 2024-04-26T01:33:50.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.857+0000 7f18b7000700 1 -- 172.21.15.6:0/2427565963 shutdown_connections 2024-04-26T01:33:50.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.857+0000 7f18b7000700 1 --2- 172.21.15.6:0/2427565963 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f18940395b0 0x7f189403ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:50.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.857+0000 7f18b7000700 1 --2- 172.21.15.6:0/2427565963 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18b0108530 0x7f18b019c400 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:50.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.857+0000 7f18b7000700 1 -- 172.21.15.6:0/2427565963 >> 172.21.15.6:0/2427565963 conn(0x7f18b0103b50 msgr2=0x7f18b01045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:50.621 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.858+0000 7f18b7000700 1 -- 172.21.15.6:0/2427565963 shutdown_connections 2024-04-26T01:33:50.621 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:49.858+0000 7f18b7000700 1 -- 172.21.15.6:0/2427565963 wait complete. 2024-04-26T01:33:50.621 INFO:teuthology.orchestra.run.smithi006.stdout:Creating initial admin user... 2024-04-26T01:33:52.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$8UEf/fkhaUEEd1hiv3XbXOrR9ENVd8Wdfg4Ig6p0czcBB5teNP4.G", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714095231, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T01:33:52.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.245+0000 7f419a600700 1 Processor -- start 2024-04-26T01:33:52.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.245+0000 7f419a600700 1 -- start start 2024-04-26T01:33:52.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.246+0000 7f419a600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 0x7f419407c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:52.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.246+0000 7f419a600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f419407c920 con 0x7f419407dea0 2024-04-26T01:33:52.635 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.246+0000 7f4193400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 0x7f419407c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:52.635 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.246+0000 7f4193400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 0x7f419407c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33098/0 (socket says 172.21.15.6:33098) 2024-04-26T01:33:52.636 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.246+0000 7f4193400700 1 -- 172.21.15.6:0/2470765705 learned_addr learned my addr 172.21.15.6:0/2470765705 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:52.636 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.247+0000 7f4193400700 1 -- 172.21.15.6:0/2470765705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419407caa0 con 0x7f419407dea0 2024-04-26T01:33:52.636 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.247+0000 7f4193400700 1 --2- 172.21.15.6:0/2470765705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 0x7f419407c350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f41840098a0 tx=0x7f4184009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6a581fc1d64876 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:52.636 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.248+0000 7f4192000700 1 -- 172.21.15.6:0/2470765705 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4184004160 con 0x7f419407dea0 2024-04-26T01:33:52.636 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.248+0000 7f4192000700 1 -- 172.21.15.6:0/2470765705 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4184004300 con 0x7f419407dea0 2024-04-26T01:33:52.636 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.248+0000 7f4192000700 1 -- 172.21.15.6:0/2470765705 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4184011a20 con 0x7f419407dea0 2024-04-26T01:33:52.636 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.248+0000 7f419a600700 1 -- 172.21.15.6:0/2470765705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 msgr2=0x7f419407c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:52.636 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.248+0000 7f419a600700 1 --2- 172.21.15.6:0/2470765705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 0x7f419407c350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f41840098a0 tx=0x7f4184009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:52.636 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.249+0000 7f419a600700 1 -- 172.21.15.6:0/2470765705 shutdown_connections 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.249+0000 7f419a600700 1 --2- 172.21.15.6:0/2470765705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 0x7f419407c350 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.249+0000 7f419a600700 1 -- 172.21.15.6:0/2470765705 >> 172.21.15.6:0/2470765705 conn(0x7f4194103b10 msgr2=0x7f4194105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.249+0000 7f419a600700 1 -- 172.21.15.6:0/2470765705 shutdown_connections 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.249+0000 7f419a600700 1 -- 172.21.15.6:0/2470765705 wait complete. 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.250+0000 7f419a600700 1 Processor -- start 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.250+0000 7f419a600700 1 -- start start 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.250+0000 7f419a600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 0x7f41941a2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.250+0000 7f419a600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f419410a700 con 0x7f419407dea0 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.251+0000 7f4193400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 0x7f41941a2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.251+0000 7f4193400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 0x7f41941a2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33100/0 (socket says 172.21.15.6:33100) 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.251+0000 7f4193400700 1 -- 172.21.15.6:0/608392601 learned_addr learned my addr 172.21.15.6:0/608392601 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.251+0000 7f4193400700 1 -- 172.21.15.6:0/608392601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41941a2f80 con 0x7f419407dea0 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.252+0000 7f4193400700 1 --2- 172.21.15.6:0/608392601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 0x7f41941a2a40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4184004660 tx=0x7f4184011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:52.637 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.252+0000 7f418be00700 1 -- 172.21.15.6:0/608392601 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4184003aa0 con 0x7f419407dea0 2024-04-26T01:33:52.638 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.252+0000 7f418be00700 1 -- 172.21.15.6:0/608392601 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f418401b8f0 con 0x7f419407dea0 2024-04-26T01:33:52.640 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.252+0000 7f418be00700 1 -- 172.21.15.6:0/608392601 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4184023900 con 0x7f419407dea0 2024-04-26T01:33:52.640 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.252+0000 7f419a600700 1 -- 172.21.15.6:0/608392601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41941a3210 con 0x7f419407dea0 2024-04-26T01:33:52.640 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.252+0000 7f419a600700 1 -- 172.21.15.6:0/608392601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41941a5f00 con 0x7f419407dea0 2024-04-26T01:33:52.640 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.254+0000 7f418be00700 1 -- 172.21.15.6:0/608392601 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f418401baa0 con 0x7f419407dea0 2024-04-26T01:33:52.640 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.254+0000 7f419a600700 1 -- 172.21.15.6:0/608392601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41940640a0 con 0x7f419407dea0 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.254+0000 7f418be00700 1 --2- 172.21.15.6:0/608392601 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4178039660 0x7f417803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.254+0000 7f418be00700 1 -- 172.21.15.6:0/608392601 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f418404e080 con 0x7f419407dea0 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.255+0000 7f4192a00700 1 --2- 172.21.15.6:0/608392601 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4178039660 0x7f417803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.255+0000 7f4192a00700 1 --2- 172.21.15.6:0/608392601 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4178039660 0x7f417803bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4180006fd0 tx=0x7f4180006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.261+0000 7f418be00700 1 -- 172.21.15.6:0/608392601 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f418401f030 con 0x7f419407dea0 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.514+0000 7f419a600700 1 -- 172.21.15.6:0/608392601 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f419419bff0 con 0x7f4178039660 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.776+0000 7f418be00700 1 -- 172.21.15.6:0/608392601 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f419419bff0 con 0x7f4178039660 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.778+0000 7f419a600700 1 -- 172.21.15.6:0/608392601 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4178039660 msgr2=0x7f417803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.778+0000 7f419a600700 1 --2- 172.21.15.6:0/608392601 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4178039660 0x7f417803bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4180006fd0 tx=0x7f4180006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.778+0000 7f419a600700 1 -- 172.21.15.6:0/608392601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 msgr2=0x7f41941a2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.778+0000 7f419a600700 1 --2- 172.21.15.6:0/608392601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 0x7f41941a2a40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4184004660 tx=0x7f4184011aa0 comp rx=0 tx=0).stop 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.779+0000 7f419a600700 1 -- 172.21.15.6:0/608392601 shutdown_connections 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.779+0000 7f419a600700 1 --2- 172.21.15.6:0/608392601 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4178039660 0x7f417803bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.779+0000 7f419a600700 1 --2- 172.21.15.6:0/608392601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f419407dea0 0x7f41941a2a40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.779+0000 7f419a600700 1 -- 172.21.15.6:0/608392601 >> 172.21.15.6:0/608392601 conn(0x7f4194103b10 msgr2=0x7f4194105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:52.642 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.779+0000 7f419a600700 1 -- 172.21.15.6:0/608392601 shutdown_connections 2024-04-26T01:33:52.642 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:51.779+0000 7f419a600700 1 -- 172.21.15.6:0/608392601 wait complete. 2024-04-26T01:33:52.642 INFO:teuthology.orchestra.run.smithi006.stdout:Fetching dashboard port number... 2024-04-26T01:33:52.934 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:52 smithi006 ceph-mon[28419]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:52.934 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:52 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:33:52.934 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:52 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:33:53.813 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:53 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3923845080' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T01:33:54.138 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T01:33:54.139 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.276+0000 7ff753400700 1 Processor -- start 2024-04-26T01:33:54.139 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.276+0000 7ff753400700 1 -- start start 2024-04-26T01:33:54.139 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.277+0000 7ff753400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 0x7ff74c07b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:54.139 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.277+0000 7ff753400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff74c07b810 con 0x7ff74c07cde0 2024-04-26T01:33:54.139 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.277+0000 7ff750a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 0x7ff74c07b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:54.142 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.277+0000 7ff750a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 0x7ff74c07b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33116/0 (socket says 172.21.15.6:33116) 2024-04-26T01:33:54.142 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.277+0000 7ff750a00700 1 -- 172.21.15.6:0/1089820690 learned_addr learned my addr 172.21.15.6:0/1089820690 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.278+0000 7ff750a00700 1 -- 172.21.15.6:0/1089820690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff74c07b990 con 0x7ff74c07cde0 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.278+0000 7ff750a00700 1 --2- 172.21.15.6:0/1089820690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 0x7ff74c07b240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff7400098a0 tx=0x7ff740009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbca58f45406a896 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.278+0000 7ff74b400700 1 -- 172.21.15.6:0/1089820690 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff740004160 con 0x7ff74c07cde0 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.278+0000 7ff74b400700 1 -- 172.21.15.6:0/1089820690 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff740004300 con 0x7ff74c07cde0 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.278+0000 7ff74b400700 1 -- 172.21.15.6:0/1089820690 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff740011a20 con 0x7ff74c07cde0 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.279+0000 7ff753400700 1 -- 172.21.15.6:0/1089820690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 msgr2=0x7ff74c07b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.279+0000 7ff753400700 1 --2- 172.21.15.6:0/1089820690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 0x7ff74c07b240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff7400098a0 tx=0x7ff740009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.279+0000 7ff753400700 1 -- 172.21.15.6:0/1089820690 shutdown_connections 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.279+0000 7ff753400700 1 --2- 172.21.15.6:0/1089820690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 0x7ff74c07b240 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.279+0000 7ff753400700 1 -- 172.21.15.6:0/1089820690 >> 172.21.15.6:0/1089820690 conn(0x7ff74c103b50 msgr2=0x7ff74c105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.279+0000 7ff753400700 1 -- 172.21.15.6:0/1089820690 shutdown_connections 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.279+0000 7ff753400700 1 -- 172.21.15.6:0/1089820690 wait complete. 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.280+0000 7ff753400700 1 Processor -- start 2024-04-26T01:33:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.281+0000 7ff753400700 1 -- start start 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.281+0000 7ff753400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 0x7ff74c19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.281+0000 7ff753400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff74c10eb40 con 0x7ff74c07cde0 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.281+0000 7ff750a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 0x7ff74c19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.281+0000 7ff750a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 0x7ff74c19e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33122/0 (socket says 172.21.15.6:33122) 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.281+0000 7ff750a00700 1 -- 172.21.15.6:0/3923845080 learned_addr learned my addr 172.21.15.6:0/3923845080 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.282+0000 7ff750a00700 1 -- 172.21.15.6:0/3923845080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff74c19eb70 con 0x7ff74c07cde0 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.282+0000 7ff750a00700 1 --2- 172.21.15.6:0/3923845080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 0x7ff74c19e630 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff740009520 tx=0x7ff740011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.283+0000 7ff749600700 1 -- 172.21.15.6:0/3923845080 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff740003af0 con 0x7ff74c07cde0 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.283+0000 7ff749600700 1 -- 172.21.15.6:0/3923845080 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff74001a8e0 con 0x7ff74c07cde0 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.283+0000 7ff753400700 1 -- 172.21.15.6:0/3923845080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff74c19ee00 con 0x7ff74c07cde0 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.283+0000 7ff749600700 1 -- 172.21.15.6:0/3923845080 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7400247a0 con 0x7ff74c07cde0 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.283+0000 7ff753400700 1 -- 172.21.15.6:0/3923845080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff74c1a1af0 con 0x7ff74c07cde0 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.284+0000 7ff753400700 1 -- 172.21.15.6:0/3923845080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff74c0640a0 con 0x7ff74c07cde0 2024-04-26T01:33:54.144 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.284+0000 7ff749600700 1 -- 172.21.15.6:0/3923845080 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7ff740022020 con 0x7ff74c07cde0 2024-04-26T01:33:54.146 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.285+0000 7ff749600700 1 --2- 172.21.15.6:0/3923845080 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff734039660 0x7ff73403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:54.146 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.285+0000 7ff749600700 1 -- 172.21.15.6:0/3923845080 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff74004da90 con 0x7ff74c07cde0 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.285+0000 7ff74be00700 1 --2- 172.21.15.6:0/3923845080 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff734039660 0x7ff73403bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.286+0000 7ff74be00700 1 --2- 172.21.15.6:0/3923845080 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff734039660 0x7ff73403bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff738006fd0 tx=0x7ff738006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.291+0000 7ff749600700 1 -- 172.21.15.6:0/3923845080 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff74000b3e0 con 0x7ff74c07cde0 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.528+0000 7ff753400700 1 -- 172.21.15.6:0/3923845080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7ff74c1a1cc0 con 0x7ff74c07cde0 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.528+0000 7ff749600700 1 -- 172.21.15.6:0/3923845080 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7ff74002ddb0 con 0x7ff74c07cde0 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.531+0000 7ff753400700 1 -- 172.21.15.6:0/3923845080 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff734039660 msgr2=0x7ff73403bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.531+0000 7ff753400700 1 --2- 172.21.15.6:0/3923845080 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff734039660 0x7ff73403bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff738006fd0 tx=0x7ff738006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.531+0000 7ff753400700 1 -- 172.21.15.6:0/3923845080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 msgr2=0x7ff74c19e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.531+0000 7ff753400700 1 --2- 172.21.15.6:0/3923845080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 0x7ff74c19e630 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff740009520 tx=0x7ff740011aa0 comp rx=0 tx=0).stop 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.532+0000 7ff753400700 1 -- 172.21.15.6:0/3923845080 shutdown_connections 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.532+0000 7ff753400700 1 --2- 172.21.15.6:0/3923845080 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff734039660 0x7ff73403bb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.532+0000 7ff753400700 1 --2- 172.21.15.6:0/3923845080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff74c07cde0 0x7ff74c19e630 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.532+0000 7ff753400700 1 -- 172.21.15.6:0/3923845080 >> 172.21.15.6:0/3923845080 conn(0x7ff74c103b50 msgr2=0x7ff74c1045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:54.147 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.532+0000 7ff753400700 1 -- 172.21.15.6:0/3923845080 shutdown_connections 2024-04-26T01:33:54.148 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:53.532+0000 7ff753400700 1 -- 172.21.15.6:0/3923845080 wait complete. 2024-04-26T01:33:54.172 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T01:33:54.173 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-04-26T01:33:54.195 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T01:33:54.195 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-04-26T01:33:54.196 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-04-26T01:33:54.196 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T01:33:54.199 INFO:teuthology.orchestra.run.smithi006.stdout:Ceph Dashboard is now available at: 2024-04-26T01:33:54.199 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:33:54.199 INFO:teuthology.orchestra.run.smithi006.stdout: URL: https://smithi006.front.sepia.ceph.com:8443/ 2024-04-26T01:33:54.199 INFO:teuthology.orchestra.run.smithi006.stdout: User: admin 2024-04-26T01:33:54.199 INFO:teuthology.orchestra.run.smithi006.stdout: Password: tkqar34jzk 2024-04-26T01:33:54.199 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:33:54.200 INFO:teuthology.orchestra.run.smithi006.stdout:Saving cluster configuration to /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config directory 2024-04-26T01:33:54.201 INFO:teuthology.orchestra.run.smithi006.stdout:Enabling autotune for osd_memory_target 2024-04-26T01:33:55.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:54 smithi006 ceph-mon[28419]: mgrmap e13: x(active, since 6s) 2024-04-26T01:33:56.234 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.248+0000 7f3e43800700 1 Processor -- start 2024-04-26T01:33:56.235 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.248+0000 7f3e43800700 1 -- start start 2024-04-26T01:33:56.235 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.248+0000 7f3e43800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 0x7f3e3c108960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:56.235 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.248+0000 7f3e43800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3c108f30 con 0x7f3e3c108540 2024-04-26T01:33:56.235 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.249+0000 7f3e40e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 0x7f3e3c108960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:56.235 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.249+0000 7f3e40e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 0x7f3e3c108960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33128/0 (socket says 172.21.15.6:33128) 2024-04-26T01:33:56.235 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.249+0000 7f3e40e00700 1 -- 172.21.15.6:0/1771743142 learned_addr learned my addr 172.21.15.6:0/1771743142 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:56.235 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.249+0000 7f3e40e00700 1 -- 172.21.15.6:0/1771743142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3c1096f0 con 0x7f3e3c108540 2024-04-26T01:33:56.235 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.250+0000 7f3e40e00700 1 --2- 172.21.15.6:0/1771743142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 0x7f3e3c108960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3e300098a0 tx=0x7f3e30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=550b83a9f9b56af9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:56.235 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.250+0000 7f3e3b400700 1 -- 172.21.15.6:0/1771743142 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e30004160 con 0x7f3e3c108540 2024-04-26T01:33:56.236 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.250+0000 7f3e3b400700 1 -- 172.21.15.6:0/1771743142 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e30004300 con 0x7f3e3c108540 2024-04-26T01:33:56.236 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.250+0000 7f3e3b400700 1 -- 172.21.15.6:0/1771743142 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e30011a20 con 0x7f3e3c108540 2024-04-26T01:33:56.236 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.251+0000 7f3e43800700 1 -- 172.21.15.6:0/1771743142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 msgr2=0x7f3e3c108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:56.236 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.251+0000 7f3e43800700 1 --2- 172.21.15.6:0/1771743142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 0x7f3e3c108960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3e300098a0 tx=0x7f3e30009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:56.236 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.251+0000 7f3e43800700 1 -- 172.21.15.6:0/1771743142 shutdown_connections 2024-04-26T01:33:56.236 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.251+0000 7f3e43800700 1 --2- 172.21.15.6:0/1771743142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 0x7f3e3c108960 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:56.236 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.251+0000 7f3e43800700 1 -- 172.21.15.6:0/1771743142 >> 172.21.15.6:0/1771743142 conn(0x7f3e3c103b40 msgr2=0x7f3e3c105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:56.236 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.251+0000 7f3e43800700 1 -- 172.21.15.6:0/1771743142 shutdown_connections 2024-04-26T01:33:56.236 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.251+0000 7f3e43800700 1 -- 172.21.15.6:0/1771743142 wait complete. 2024-04-26T01:33:56.237 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.252+0000 7f3e43800700 1 Processor -- start 2024-04-26T01:33:56.237 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.253+0000 7f3e43800700 1 -- start start 2024-04-26T01:33:56.237 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.253+0000 7f3e43800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 0x7f3e3c19c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:56.237 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.253+0000 7f3e43800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3c10caa0 con 0x7f3e3c108540 2024-04-26T01:33:56.237 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.253+0000 7f3e40e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 0x7f3e3c19c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:56.237 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.253+0000 7f3e40e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 0x7f3e3c19c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33136/0 (socket says 172.21.15.6:33136) 2024-04-26T01:33:56.237 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.253+0000 7f3e40e00700 1 -- 172.21.15.6:0/266542621 learned_addr learned my addr 172.21.15.6:0/266542621 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:56.237 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.254+0000 7f3e40e00700 1 -- 172.21.15.6:0/266542621 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3c19c9e0 con 0x7f3e3c108540 2024-04-26T01:33:56.237 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.254+0000 7f3e40e00700 1 --2- 172.21.15.6:0/266542621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 0x7f3e3c19c4a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3e30000c00 tx=0x7f3e30011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:56.237 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.254+0000 7f3e39600700 1 -- 172.21.15.6:0/266542621 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e30003aa0 con 0x7f3e3c108540 2024-04-26T01:33:56.237 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.254+0000 7f3e39600700 1 -- 172.21.15.6:0/266542621 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e3001b8f0 con 0x7f3e3c108540 2024-04-26T01:33:56.238 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.254+0000 7f3e43800700 1 -- 172.21.15.6:0/266542621 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e3c19cc70 con 0x7f3e3c108540 2024-04-26T01:33:56.238 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.254+0000 7f3e39600700 1 -- 172.21.15.6:0/266542621 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e30023890 con 0x7f3e3c108540 2024-04-26T01:33:56.238 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.254+0000 7f3e43800700 1 -- 172.21.15.6:0/266542621 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e3c19f960 con 0x7f3e3c108540 2024-04-26T01:33:56.238 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.255+0000 7f3e39600700 1 -- 172.21.15.6:0/266542621 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f3e3001b440 con 0x7f3e3c108540 2024-04-26T01:33:56.238 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.255+0000 7f3e43800700 1 -- 172.21.15.6:0/266542621 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e3c0640a0 con 0x7f3e3c108540 2024-04-26T01:33:56.238 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.256+0000 7f3e39600700 1 --2- 172.21.15.6:0/266542621 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3e240396a0 0x7f3e2403bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:56.238 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.256+0000 7f3e39600700 1 -- 172.21.15.6:0/266542621 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3e3004e0b0 con 0x7f3e3c108540 2024-04-26T01:33:56.238 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.256+0000 7f3e3be00700 1 --2- 172.21.15.6:0/266542621 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3e240396a0 0x7f3e2403bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:56.238 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.257+0000 7f3e3be00700 1 --2- 172.21.15.6:0/266542621 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3e240396a0 0x7f3e2403bb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3e28006fd0 tx=0x7f3e28006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:56.238 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.262+0000 7f3e39600700 1 -- 172.21.15.6:0/266542621 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e3001f030 con 0x7f3e3c108540 2024-04-26T01:33:56.238 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.497+0000 7f3e43800700 1 -- 172.21.15.6:0/266542621 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f3e3c19ffc0 con 0x7f3e3c108540 2024-04-26T01:33:56.239 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.498+0000 7f3e39600700 1 -- 172.21.15.6:0/266542621 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f3e3000be50 con 0x7f3e3c108540 2024-04-26T01:33:56.239 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.500+0000 7f3e43800700 1 -- 172.21.15.6:0/266542621 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3e240396a0 msgr2=0x7f3e2403bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:56.239 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.500+0000 7f3e43800700 1 --2- 172.21.15.6:0/266542621 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3e240396a0 0x7f3e2403bb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3e28006fd0 tx=0x7f3e28006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:56.239 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.500+0000 7f3e43800700 1 -- 172.21.15.6:0/266542621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 msgr2=0x7f3e3c19c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:56.239 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.500+0000 7f3e43800700 1 --2- 172.21.15.6:0/266542621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 0x7f3e3c19c4a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3e30000c00 tx=0x7f3e30011aa0 comp rx=0 tx=0).stop 2024-04-26T01:33:56.239 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.501+0000 7f3e43800700 1 -- 172.21.15.6:0/266542621 shutdown_connections 2024-04-26T01:33:56.239 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.501+0000 7f3e43800700 1 --2- 172.21.15.6:0/266542621 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3e240396a0 0x7f3e2403bb60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:56.239 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.501+0000 7f3e43800700 1 --2- 172.21.15.6:0/266542621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e3c108540 0x7f3e3c19c4a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:56.239 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.501+0000 7f3e43800700 1 -- 172.21.15.6:0/266542621 >> 172.21.15.6:0/266542621 conn(0x7f3e3c103b40 msgr2=0x7f3e3c1046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:56.239 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.501+0000 7f3e43800700 1 -- 172.21.15.6:0/266542621 shutdown_connections 2024-04-26T01:33:56.239 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:55.501+0000 7f3e43800700 1 -- 172.21.15.6:0/266542621 wait complete. 2024-04-26T01:33:57.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.858+0000 7f02efa00700 1 Processor -- start 2024-04-26T01:33:57.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.858+0000 7f02efa00700 1 -- start start 2024-04-26T01:33:57.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.859+0000 7f02efa00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 0x7f02e8108960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:57.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.859+0000 7f02efa00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f02e8108f30 con 0x7f02e8108540 2024-04-26T01:33:57.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.859+0000 7f02ed000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 0x7f02e8108960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:57.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.859+0000 7f02ed000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 0x7f02e8108960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33142/0 (socket says 172.21.15.6:33142) 2024-04-26T01:33:57.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.859+0000 7f02ed000700 1 -- 172.21.15.6:0/1711893087 learned_addr learned my addr 172.21.15.6:0/1711893087 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:57.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.860+0000 7f02ed000700 1 -- 172.21.15.6:0/1711893087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e81096f0 con 0x7f02e8108540 2024-04-26T01:33:57.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.860+0000 7f02ed000700 1 --2- 172.21.15.6:0/1711893087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 0x7f02e8108960 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0098a0 tx=0x7f02dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b21f5f778da535 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:57.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.860+0000 7f02e7400700 1 -- 172.21.15.6:0/1711893087 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02dc004160 con 0x7f02e8108540 2024-04-26T01:33:57.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.860+0000 7f02e7400700 1 -- 172.21.15.6:0/1711893087 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02dc004300 con 0x7f02e8108540 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.860+0000 7f02e7400700 1 -- 172.21.15.6:0/1711893087 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02dc011a20 con 0x7f02e8108540 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.861+0000 7f02efa00700 1 -- 172.21.15.6:0/1711893087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 msgr2=0x7f02e8108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.861+0000 7f02efa00700 1 --2- 172.21.15.6:0/1711893087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 0x7f02e8108960 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0098a0 tx=0x7f02dc009bb0 comp rx=0 tx=0).stop 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.861+0000 7f02efa00700 1 -- 172.21.15.6:0/1711893087 shutdown_connections 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.861+0000 7f02efa00700 1 --2- 172.21.15.6:0/1711893087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 0x7f02e8108960 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.861+0000 7f02efa00700 1 -- 172.21.15.6:0/1711893087 >> 172.21.15.6:0/1711893087 conn(0x7f02e8103b40 msgr2=0x7f02e8105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.861+0000 7f02efa00700 1 -- 172.21.15.6:0/1711893087 shutdown_connections 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.861+0000 7f02efa00700 1 -- 172.21.15.6:0/1711893087 wait complete. 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.862+0000 7f02efa00700 1 Processor -- start 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.863+0000 7f02efa00700 1 -- start start 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.863+0000 7f02efa00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 0x7f02e819c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.863+0000 7f02efa00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f02e810caa0 con 0x7f02e8108540 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.863+0000 7f02ed000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 0x7f02e819c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.863+0000 7f02ed000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 0x7f02e819c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33146/0 (socket says 172.21.15.6:33146) 2024-04-26T01:33:57.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.863+0000 7f02ed000700 1 -- 172.21.15.6:0/2790159110 learned_addr learned my addr 172.21.15.6:0/2790159110 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.864+0000 7f02ed000700 1 -- 172.21.15.6:0/2790159110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e819c9e0 con 0x7f02e8108540 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.864+0000 7f02ed000700 1 --2- 172.21.15.6:0/2790159110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 0x7f02e819c4a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f02dc000c00 tx=0x7f02dc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.864+0000 7f02e5600700 1 -- 172.21.15.6:0/2790159110 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02dc003aa0 con 0x7f02e8108540 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.864+0000 7f02e5600700 1 -- 172.21.15.6:0/2790159110 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02dc01b8f0 con 0x7f02e8108540 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.864+0000 7f02efa00700 1 -- 172.21.15.6:0/2790159110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e819cc70 con 0x7f02e8108540 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.864+0000 7f02e5600700 1 -- 172.21.15.6:0/2790159110 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02dc023890 con 0x7f02e8108540 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.864+0000 7f02efa00700 1 -- 172.21.15.6:0/2790159110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e819f960 con 0x7f02e8108540 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.866+0000 7f02efa00700 1 -- 172.21.15.6:0/2790159110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02e80640a0 con 0x7f02e8108540 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.866+0000 7f02e5600700 1 -- 172.21.15.6:0/2790159110 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f02dc01b440 con 0x7f02e8108540 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.866+0000 7f02e5600700 1 --2- 172.21.15.6:0/2790159110 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f02d00396a0 0x7f02d003bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.866+0000 7f02e5600700 1 -- 172.21.15.6:0/2790159110 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f02dc04e130 con 0x7f02e8108540 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.867+0000 7f02e7e00700 1 --2- 172.21.15.6:0/2790159110 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f02d00396a0 0x7f02d003bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.867+0000 7f02e7e00700 1 --2- 172.21.15.6:0/2790159110 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f02d00396a0 0x7f02d003bb60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f02d4006fd0 tx=0x7f02d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:56.872+0000 7f02e5600700 1 -- 172.21.15.6:0/2790159110 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f02dc01f030 con 0x7f02e8108540 2024-04-26T01:33:57.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:57.224+0000 7f02efa00700 1 -- 172.21.15.6:0/2790159110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f02e807c5c0 con 0x7f02e8108540 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:57.228+0000 7f02e5600700 1 -- 172.21.15.6:0/2790159110 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f02dc016260 con 0x7f02e8108540 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:57.230+0000 7f02efa00700 1 -- 172.21.15.6:0/2790159110 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f02d00396a0 msgr2=0x7f02d003bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:57.230+0000 7f02efa00700 1 --2- 172.21.15.6:0/2790159110 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f02d00396a0 0x7f02d003bb60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f02d4006fd0 tx=0x7f02d4006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:57.231+0000 7f02efa00700 1 -- 172.21.15.6:0/2790159110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 msgr2=0x7f02e819c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:57.231+0000 7f02efa00700 1 --2- 172.21.15.6:0/2790159110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 0x7f02e819c4a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f02dc000c00 tx=0x7f02dc011aa0 comp rx=0 tx=0).stop 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:57.231+0000 7f02efa00700 1 -- 172.21.15.6:0/2790159110 shutdown_connections 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:57.231+0000 7f02efa00700 1 --2- 172.21.15.6:0/2790159110 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f02d00396a0 0x7f02d003bb60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:57.231+0000 7f02efa00700 1 --2- 172.21.15.6:0/2790159110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02e8108540 0x7f02e819c4a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:57.231+0000 7f02efa00700 1 -- 172.21.15.6:0/2790159110 >> 172.21.15.6:0/2790159110 conn(0x7f02e8103b40 msgr2=0x7f02e81046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:57.231+0000 7f02efa00700 1 -- 172.21.15.6:0/2790159110 shutdown_connections 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-04-26T01:33:57.231+0000 7f02efa00700 1 -- 172.21.15.6:0/2790159110 wait complete. 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:33:57.889 INFO:teuthology.orchestra.run.smithi006.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout: ceph telemetry on 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout:For more information see: 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:33:57.890 INFO:teuthology.orchestra.run.smithi006.stdout:Bootstrap complete. 2024-04-26T01:33:57.956 INFO:tasks.cephadm:Fetching config... 2024-04-26T01:33:57.957 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:33:57.957 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T01:33:58.005 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T01:33:58.005 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:33:58.006 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T01:33:58.086 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T01:33:58.086 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:33:58.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T01:33:58.201 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T01:33:58.201 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:33:58.201 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T01:33:58.299 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T01:33:58.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9/IoH5V/4QN52cB2pRQb7vo4jzMp2nlrc481f8N4iab7SfNWfUay7jRS0EXUjv+08XdAjLoF6PgmE1Bd9QVPVfM/7BYrC1nleXw0JLy/MK9vs2WMR32v4hVBcVH9cCwft+nXpgelFsaRCgEMU0O3hE6+WYUxPH6frcNc6sQDPGa5dDCq1qC8Mc1RFt2/wNNIz7ocTONVBiDDMla+F/E6xn376PNOp77vfALWHaTFhob3N8sVn/F4DXPAbdLonscPDru4xM01Mxc8r/oGlVlk/zow6XnFNM6J37UyTqs/WEwsmJezVCeIj3T/eOof7vnnZgW451ABzf0AUO+ilaZ0s7M8mncr76KFQG2zashvJ+L7HjOGCH/0WpcJop+xptAIxyT1L0D96K4a+Zb7vXHjo/HUSrR0IBo2g7EaA0uRCVRpavnFvuJGv6D5Jk92f2KtLV8fgD33YEQms1gPcA5GZVitYdyFKzG2zTWLINlJoIVZjm/Cru0kXTHlJLozv6XM= ceph-a5b72bee-036c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T01:33:58.405 INFO:teuthology.orchestra.run.smithi006.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9/IoH5V/4QN52cB2pRQb7vo4jzMp2nlrc481f8N4iab7SfNWfUay7jRS0EXUjv+08XdAjLoF6PgmE1Bd9QVPVfM/7BYrC1nleXw0JLy/MK9vs2WMR32v4hVBcVH9cCwft+nXpgelFsaRCgEMU0O3hE6+WYUxPH6frcNc6sQDPGa5dDCq1qC8Mc1RFt2/wNNIz7ocTONVBiDDMla+F/E6xn376PNOp77vfALWHaTFhob3N8sVn/F4DXPAbdLonscPDru4xM01Mxc8r/oGlVlk/zow6XnFNM6J37UyTqs/WEwsmJezVCeIj3T/eOof7vnnZgW451ABzf0AUO+ilaZ0s7M8mncr76KFQG2zashvJ+L7HjOGCH/0WpcJop+xptAIxyT1L0D96K4a+Zb7vXHjo/HUSrR0IBo2g7EaA0uRCVRpavnFvuJGv6D5Jk92f2KtLV8fgD33YEQms1gPcA5GZVitYdyFKzG2zTWLINlJoIVZjm/Cru0kXTHlJLozv6XM= ceph-a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:58.441 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:33:58 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2790159110' entity='client.admin' 2024-04-26T01:33:58.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9/IoH5V/4QN52cB2pRQb7vo4jzMp2nlrc481f8N4iab7SfNWfUay7jRS0EXUjv+08XdAjLoF6PgmE1Bd9QVPVfM/7BYrC1nleXw0JLy/MK9vs2WMR32v4hVBcVH9cCwft+nXpgelFsaRCgEMU0O3hE6+WYUxPH6frcNc6sQDPGa5dDCq1qC8Mc1RFt2/wNNIz7ocTONVBiDDMla+F/E6xn376PNOp77vfALWHaTFhob3N8sVn/F4DXPAbdLonscPDru4xM01Mxc8r/oGlVlk/zow6XnFNM6J37UyTqs/WEwsmJezVCeIj3T/eOof7vnnZgW451ABzf0AUO+ilaZ0s7M8mncr76KFQG2zashvJ+L7HjOGCH/0WpcJop+xptAIxyT1L0D96K4a+Zb7vXHjo/HUSrR0IBo2g7EaA0uRCVRpavnFvuJGv6D5Jk92f2KtLV8fgD33YEQms1gPcA5GZVitYdyFKzG2zTWLINlJoIVZjm/Cru0kXTHlJLozv6XM= ceph-a5b72bee-036c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T01:33:58.500 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9/IoH5V/4QN52cB2pRQb7vo4jzMp2nlrc481f8N4iab7SfNWfUay7jRS0EXUjv+08XdAjLoF6PgmE1Bd9QVPVfM/7BYrC1nleXw0JLy/MK9vs2WMR32v4hVBcVH9cCwft+nXpgelFsaRCgEMU0O3hE6+WYUxPH6frcNc6sQDPGa5dDCq1qC8Mc1RFt2/wNNIz7ocTONVBiDDMla+F/E6xn376PNOp77vfALWHaTFhob3N8sVn/F4DXPAbdLonscPDru4xM01Mxc8r/oGlVlk/zow6XnFNM6J37UyTqs/WEwsmJezVCeIj3T/eOof7vnnZgW451ABzf0AUO+ilaZ0s7M8mncr76KFQG2zashvJ+L7HjOGCH/0WpcJop+xptAIxyT1L0D96K4a+Zb7vXHjo/HUSrR0IBo2g7EaA0uRCVRpavnFvuJGv6D5Jk92f2KtLV8fgD33YEQms1gPcA5GZVitYdyFKzG2zTWLINlJoIVZjm/Cru0kXTHlJLozv6XM= ceph-a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:58.513 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9/IoH5V/4QN52cB2pRQb7vo4jzMp2nlrc481f8N4iab7SfNWfUay7jRS0EXUjv+08XdAjLoF6PgmE1Bd9QVPVfM/7BYrC1nleXw0JLy/MK9vs2WMR32v4hVBcVH9cCwft+nXpgelFsaRCgEMU0O3hE6+WYUxPH6frcNc6sQDPGa5dDCq1qC8Mc1RFt2/wNNIz7ocTONVBiDDMla+F/E6xn376PNOp77vfALWHaTFhob3N8sVn/F4DXPAbdLonscPDru4xM01Mxc8r/oGlVlk/zow6XnFNM6J37UyTqs/WEwsmJezVCeIj3T/eOof7vnnZgW451ABzf0AUO+ilaZ0s7M8mncr76KFQG2zashvJ+L7HjOGCH/0WpcJop+xptAIxyT1L0D96K4a+Zb7vXHjo/HUSrR0IBo2g7EaA0uRCVRpavnFvuJGv6D5Jk92f2KtLV8fgD33YEQms1gPcA5GZVitYdyFKzG2zTWLINlJoIVZjm/Cru0kXTHlJLozv6XM= ceph-a5b72bee-036c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T01:33:58.555 INFO:teuthology.orchestra.run.smithi188.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9/IoH5V/4QN52cB2pRQb7vo4jzMp2nlrc481f8N4iab7SfNWfUay7jRS0EXUjv+08XdAjLoF6PgmE1Bd9QVPVfM/7BYrC1nleXw0JLy/MK9vs2WMR32v4hVBcVH9cCwft+nXpgelFsaRCgEMU0O3hE6+WYUxPH6frcNc6sQDPGa5dDCq1qC8Mc1RFt2/wNNIz7ocTONVBiDDMla+F/E6xn376PNOp77vfALWHaTFhob3N8sVn/F4DXPAbdLonscPDru4xM01Mxc8r/oGlVlk/zow6XnFNM6J37UyTqs/WEwsmJezVCeIj3T/eOof7vnnZgW451ABzf0AUO+ilaZ0s7M8mncr76KFQG2zashvJ+L7HjOGCH/0WpcJop+xptAIxyT1L0D96K4a+Zb7vXHjo/HUSrR0IBo2g7EaA0uRCVRpavnFvuJGv6D5Jk92f2KtLV8fgD33YEQms1gPcA5GZVitYdyFKzG2zTWLINlJoIVZjm/Cru0kXTHlJLozv6XM= ceph-a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:58.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T01:33:58.996 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:34:01.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.711+0000 7f5bc5800700 1 -- 172.21.15.6:0/3642423389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bc0076a70 msgr2=0x7f5bc0074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:01.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.711+0000 7f5bc5800700 1 --2- 172.21.15.6:0/3642423389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bc0076a70 0x7f5bc0074ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009910 tx=0x7f5bb0009c20 comp rx=0 tx=0).stop 2024-04-26T01:34:01.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.711+0000 7f5bc5800700 1 -- 172.21.15.6:0/3642423389 shutdown_connections 2024-04-26T01:34:01.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.711+0000 7f5bc5800700 1 --2- 172.21.15.6:0/3642423389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bc0076a70 0x7f5bc0074ed0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.711+0000 7f5bc5800700 1 -- 172.21.15.6:0/3642423389 >> 172.21.15.6:0/3642423389 conn(0x7f5bc00fda60 msgr2=0x7f5bc00ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.711+0000 7f5bc5800700 1 -- 172.21.15.6:0/3642423389 shutdown_connections 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.712+0000 7f5bc5800700 1 -- 172.21.15.6:0/3642423389 wait complete. 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.712+0000 7f5bc5800700 1 Processor -- start 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.713+0000 7f5bc5800700 1 -- start start 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.713+0000 7f5bc5800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bc0076a70 0x7f5bc019e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.713+0000 7f5bc5800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc010eaa0 con 0x7f5bc0076a70 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.713+0000 7f5bbea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bc0076a70 0x7f5bc019e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.713+0000 7f5bbea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bc0076a70 0x7f5bc019e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55500/0 (socket says 172.21.15.6:55500) 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.713+0000 7f5bbea00700 1 -- 172.21.15.6:0/176072219 learned_addr learned my addr 172.21.15.6:0/176072219 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.714+0000 7f5bbea00700 1 -- 172.21.15.6:0/176072219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc019e630 con 0x7f5bc0076a70 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.714+0000 7f5bbea00700 1 --2- 172.21.15.6:0/176072219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bc0076a70 0x7f5bc019e0f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0005950 tx=0x7f5bb0004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.714+0000 7f5bb7400700 1 -- 172.21.15.6:0/176072219 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bb0026070 con 0x7f5bc0076a70 2024-04-26T01:34:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.714+0000 7f5bb7400700 1 -- 172.21.15.6:0/176072219 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bb000bc00 con 0x7f5bc0076a70 2024-04-26T01:34:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.714+0000 7f5bb7400700 1 -- 172.21.15.6:0/176072219 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bb0003a10 con 0x7f5bc0076a70 2024-04-26T01:34:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.714+0000 7f5bc5800700 1 -- 172.21.15.6:0/176072219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc019e8c0 con 0x7f5bc0076a70 2024-04-26T01:34:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.714+0000 7f5bc5800700 1 -- 172.21.15.6:0/176072219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc01a15b0 con 0x7f5bc0076a70 2024-04-26T01:34:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.716+0000 7f5bc5800700 1 -- 172.21.15.6:0/176072219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc010f180 con 0x7f5bc0076a70 2024-04-26T01:34:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.716+0000 7f5bb7400700 1 -- 172.21.15.6:0/176072219 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f5bb0029070 con 0x7f5bc0076a70 2024-04-26T01:34:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.716+0000 7f5bb7400700 1 --2- 172.21.15.6:0/176072219 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5ba40396b0 0x7f5ba403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.716+0000 7f5bb7400700 1 -- 172.21.15.6:0/176072219 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5bb001e070 con 0x7f5bc0076a70 2024-04-26T01:34:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.717+0000 7f5bbe000700 1 --2- 172.21.15.6:0/176072219 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5ba40396b0 0x7f5ba403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.718+0000 7f5bbe000700 1 --2- 172.21.15.6:0/176072219 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5ba40396b0 0x7f5ba403bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5bac006fd0 tx=0x7f5bac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.722+0000 7f5bb7400700 1 -- 172.21.15.6:0/176072219 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5bb0013070 con 0x7f5bc0076a70 2024-04-26T01:34:01.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.959+0000 7f5bc5800700 1 -- 172.21.15.6:0/176072219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f5bc002dff0 con 0x7f5bc0076a70 2024-04-26T01:34:01.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.963+0000 7f5bb7400700 1 -- 172.21.15.6:0/176072219 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f5bb00300c0 con 0x7f5bc0076a70 2024-04-26T01:34:01.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.966+0000 7f5bc5800700 1 -- 172.21.15.6:0/176072219 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5ba40396b0 msgr2=0x7f5ba403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:01.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.966+0000 7f5bc5800700 1 --2- 172.21.15.6:0/176072219 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5ba40396b0 0x7f5ba403bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5bac006fd0 tx=0x7f5bac006e40 comp rx=0 tx=0).stop 2024-04-26T01:34:01.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.966+0000 7f5bc5800700 1 -- 172.21.15.6:0/176072219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bc0076a70 msgr2=0x7f5bc019e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:01.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.966+0000 7f5bc5800700 1 --2- 172.21.15.6:0/176072219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bc0076a70 0x7f5bc019e0f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0005950 tx=0x7f5bb0004fa0 comp rx=0 tx=0).stop 2024-04-26T01:34:01.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.966+0000 7f5bc5800700 1 -- 172.21.15.6:0/176072219 shutdown_connections 2024-04-26T01:34:01.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.966+0000 7f5bc5800700 1 --2- 172.21.15.6:0/176072219 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5ba40396b0 0x7f5ba403bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:01.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.966+0000 7f5bc5800700 1 --2- 172.21.15.6:0/176072219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bc0076a70 0x7f5bc019e0f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:01.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.966+0000 7f5bc5800700 1 -- 172.21.15.6:0/176072219 >> 172.21.15.6:0/176072219 conn(0x7f5bc00fda60 msgr2=0x7f5bc00fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:01.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.967+0000 7f5bc5800700 1 -- 172.21.15.6:0/176072219 shutdown_connections 2024-04-26T01:34:01.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:01.967+0000 7f5bc5800700 1 -- 172.21.15.6:0/176072219 wait complete. 2024-04-26T01:34:02.761 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T01:34:02.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T01:34:03.178 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:34:03.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:02 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/176072219' entity='client.admin' 2024-04-26T01:34:05.525 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:05.525 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:05.525 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:05.525 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:05.525 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:05.525 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:05.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.711+0000 7f7ad3c00700 1 -- 172.21.15.6:0/2308122156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7acc108c50 msgr2=0x7f7acc109070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:05.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.711+0000 7f7ad3c00700 1 --2- 172.21.15.6:0/2308122156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7acc108c50 0x7f7acc109070 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009910 tx=0x7f7abc009c20 comp rx=0 tx=0).stop 2024-04-26T01:34:05.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.711+0000 7f7ad3c00700 1 -- 172.21.15.6:0/2308122156 shutdown_connections 2024-04-26T01:34:05.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.711+0000 7f7ad3c00700 1 --2- 172.21.15.6:0/2308122156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7acc108c50 0x7f7acc109070 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:05.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.711+0000 7f7ad3c00700 1 -- 172.21.15.6:0/2308122156 >> 172.21.15.6:0/2308122156 conn(0x7f7acc077b30 msgr2=0x7f7acc079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:05.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.711+0000 7f7ad3c00700 1 -- 172.21.15.6:0/2308122156 shutdown_connections 2024-04-26T01:34:05.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.711+0000 7f7ad3c00700 1 -- 172.21.15.6:0/2308122156 wait complete. 2024-04-26T01:34:05.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.713+0000 7f7ad3c00700 1 Processor -- start 2024-04-26T01:34:05.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.713+0000 7f7ad3c00700 1 -- start start 2024-04-26T01:34:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.713+0000 7f7ad3c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7acc108c50 0x7f7acc19e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.713+0000 7f7ad3c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc10ea20 con 0x7f7acc108c50 2024-04-26T01:34:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.714+0000 7f7ad1200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7acc108c50 0x7f7acc19e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.714+0000 7f7ad1200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7acc108c50 0x7f7acc19e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55526/0 (socket says 172.21.15.6:55526) 2024-04-26T01:34:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.714+0000 7f7ad1200700 1 -- 172.21.15.6:0/33919317 learned_addr learned my addr 172.21.15.6:0/33919317 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:34:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.714+0000 7f7ad1200700 1 -- 172.21.15.6:0/33919317 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc19e650 con 0x7f7acc108c50 2024-04-26T01:34:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.714+0000 7f7ad1200700 1 --2- 172.21.15.6:0/33919317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7acc108c50 0x7f7acc19e110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7abc005950 tx=0x7f7abc004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.715+0000 7f7aca000700 1 -- 172.21.15.6:0/33919317 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7abc026070 con 0x7f7acc108c50 2024-04-26T01:34:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.715+0000 7f7aca000700 1 -- 172.21.15.6:0/33919317 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7abc00bc00 con 0x7f7acc108c50 2024-04-26T01:34:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.715+0000 7f7aca000700 1 -- 172.21.15.6:0/33919317 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7abc003a10 con 0x7f7acc108c50 2024-04-26T01:34:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.715+0000 7f7ad3c00700 1 -- 172.21.15.6:0/33919317 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7acc19e8e0 con 0x7f7acc108c50 2024-04-26T01:34:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.715+0000 7f7ad3c00700 1 -- 172.21.15.6:0/33919317 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7acc1a15d0 con 0x7f7acc108c50 2024-04-26T01:34:05.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.716+0000 7f7aca000700 1 -- 172.21.15.6:0/33919317 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f7abc029070 con 0x7f7acc108c50 2024-04-26T01:34:05.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.716+0000 7f7ad3c00700 1 -- 172.21.15.6:0/33919317 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7acc10f1a0 con 0x7f7acc108c50 2024-04-26T01:34:05.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.717+0000 7f7aca000700 1 --2- 172.21.15.6:0/33919317 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7aac0396b0 0x7f7aac03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:05.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.717+0000 7f7aca000700 1 -- 172.21.15.6:0/33919317 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7abc01e070 con 0x7f7acc108c50 2024-04-26T01:34:05.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.717+0000 7f7ac3e00700 1 --2- 172.21.15.6:0/33919317 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7aac0396b0 0x7f7aac03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:05.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.718+0000 7f7ac3e00700 1 --2- 172.21.15.6:0/33919317 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7aac0396b0 0x7f7aac03bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4006fd0 tx=0x7f7ab4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:05.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.722+0000 7f7aca000700 1 -- 172.21.15.6:0/33919317 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7abc013070 con 0x7f7acc108c50 2024-04-26T01:34:05.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.957+0000 7f7ad3c00700 1 -- 172.21.15.6:0/33919317 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f7acc02e090 con 0x7f7aac0396b0 2024-04-26T01:34:05.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.966+0000 7f7aca000700 1 -- 172.21.15.6:0/33919317 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7acc02e090 con 0x7f7aac0396b0 2024-04-26T01:34:05.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.969+0000 7f7ad3c00700 1 -- 172.21.15.6:0/33919317 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7aac0396b0 msgr2=0x7f7aac03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.969+0000 7f7ad3c00700 1 --2- 172.21.15.6:0/33919317 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7aac0396b0 0x7f7aac03bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4006fd0 tx=0x7f7ab4006e40 comp rx=0 tx=0).stop 2024-04-26T01:34:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.969+0000 7f7ad3c00700 1 -- 172.21.15.6:0/33919317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7acc108c50 msgr2=0x7f7acc19e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.969+0000 7f7ad3c00700 1 --2- 172.21.15.6:0/33919317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7acc108c50 0x7f7acc19e110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7abc005950 tx=0x7f7abc004fa0 comp rx=0 tx=0).stop 2024-04-26T01:34:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.969+0000 7f7ad3c00700 1 -- 172.21.15.6:0/33919317 shutdown_connections 2024-04-26T01:34:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.969+0000 7f7ad3c00700 1 --2- 172.21.15.6:0/33919317 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7aac0396b0 0x7f7aac03bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.969+0000 7f7ad3c00700 1 --2- 172.21.15.6:0/33919317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7acc108c50 0x7f7acc19e110 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.969+0000 7f7ad3c00700 1 -- 172.21.15.6:0/33919317 >> 172.21.15.6:0/33919317 conn(0x7f7acc077b30 msgr2=0x7f7acc078640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.969+0000 7f7ad3c00700 1 -- 172.21.15.6:0/33919317 shutdown_connections 2024-04-26T01:34:05.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:05.969+0000 7f7ad3c00700 1 -- 172.21.15.6:0/33919317 wait complete. 2024-04-26T01:34:06.818 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi029 2024-04-26T01:34:06.819 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-26T01:34:06.819 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/etc/ceph/ceph.conf 2024-04-26T01:34:06.839 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-26T01:34:06.839 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:34:06.898 INFO:tasks.cephadm:Adding host smithi029 to orchestrator... 2024-04-26T01:34:06.898 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch host add smithi029 2024-04-26T01:34:07.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:06 smithi006 ceph-mon[28419]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:07.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:07.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:07.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:07.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:07.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:06 smithi006 ceph-mon[28419]: Updating smithi006:/etc/ceph/ceph.conf 2024-04-26T01:34:07.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:06 smithi006 ceph-mon[28419]: Updating smithi006:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:34:07.331 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:34:08.256 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:07 smithi006 ceph-mon[28419]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:08.256 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:07 smithi006 ceph-mon[28419]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:34:09.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.304+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1554156901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1edc074de0 msgr2=0x7f1edc0751c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:09.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.304+0000 7f1ee2a00700 1 --2- 172.21.15.6:0/1554156901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1edc074de0 0x7f1edc0751c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc009910 tx=0x7f1ecc009c20 comp rx=0 tx=0).stop 2024-04-26T01:34:09.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.304+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1554156901 shutdown_connections 2024-04-26T01:34:09.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.304+0000 7f1ee2a00700 1 --2- 172.21.15.6:0/1554156901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1edc074de0 0x7f1edc0751c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:09.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.304+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1554156901 >> 172.21.15.6:0/1554156901 conn(0x7f1edc0fdf80 msgr2=0x7f1edc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:09.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.304+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1554156901 shutdown_connections 2024-04-26T01:34:09.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.304+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1554156901 wait complete. 2024-04-26T01:34:09.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.305+0000 7f1ee2a00700 1 Processor -- start 2024-04-26T01:34:09.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.306+0000 7f1ee2a00700 1 -- start start 2024-04-26T01:34:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.306+0000 7f1ee2a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1edc074de0 0x7f1edc19e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.306+0000 7f1ee2a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc112fa0 con 0x7f1edc074de0 2024-04-26T01:34:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.306+0000 7f1edbe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1edc074de0 0x7f1edc19e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.306+0000 7f1edbe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1edc074de0 0x7f1edc19e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43430/0 (socket says 172.21.15.6:43430) 2024-04-26T01:34:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.306+0000 7f1edbe00700 1 -- 172.21.15.6:0/1310138574 learned_addr learned my addr 172.21.15.6:0/1310138574 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:34:09.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.307+0000 7f1edbe00700 1 -- 172.21.15.6:0/1310138574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1edc19e9e0 con 0x7f1edc074de0 2024-04-26T01:34:09.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.307+0000 7f1edbe00700 1 --2- 172.21.15.6:0/1310138574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1edc074de0 0x7f1edc19e4a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc006010 tx=0x7f1ecc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:09.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.307+0000 7f1ed8c00700 1 -- 172.21.15.6:0/1310138574 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ecc026070 con 0x7f1edc074de0 2024-04-26T01:34:09.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.307+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1310138574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1edc19ec70 con 0x7f1edc074de0 2024-04-26T01:34:09.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.307+0000 7f1ed8c00700 1 -- 172.21.15.6:0/1310138574 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ecc00bc00 con 0x7f1edc074de0 2024-04-26T01:34:09.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.307+0000 7f1ed8c00700 1 -- 172.21.15.6:0/1310138574 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ecc003910 con 0x7f1edc074de0 2024-04-26T01:34:09.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.308+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1310138574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1edc19f0d0 con 0x7f1edc074de0 2024-04-26T01:34:09.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.309+0000 7f1ed8c00700 1 -- 172.21.15.6:0/1310138574 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f1ecc029070 con 0x7f1edc074de0 2024-04-26T01:34:09.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.309+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1310138574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1edc076520 con 0x7f1edc074de0 2024-04-26T01:34:09.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.309+0000 7f1ed8c00700 1 --2- 172.21.15.6:0/1310138574 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ec0039650 0x7f1ec003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:09.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.309+0000 7f1ed8c00700 1 -- 172.21.15.6:0/1310138574 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1ecc01e070 con 0x7f1edc074de0 2024-04-26T01:34:09.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.309+0000 7f1edb400700 1 --2- 172.21.15.6:0/1310138574 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ec0039650 0x7f1ec003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:09.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.310+0000 7f1edb400700 1 --2- 172.21.15.6:0/1310138574 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ec0039650 0x7f1ec003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8006fd0 tx=0x7f1ec8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:09.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.315+0000 7f1ed8c00700 1 -- 172.21.15.6:0/1310138574 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1ecc013070 con 0x7f1edc074de0 2024-04-26T01:34:09.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:09.552+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1310138574 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi029", "target": ["mon-mgr", ""]}) -- 0x7f1edc19f870 con 0x7f1ec0039650 2024-04-26T01:34:09.813 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:09 smithi006 ceph-mon[28419]: Updating smithi006:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:34:09.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:09.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:09.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:11.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:10 smithi006 ceph-mon[28419]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:11.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:10 smithi006 ceph-mon[28419]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:12.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:11 smithi006 ceph-mon[28419]: Deploying cephadm binary to smithi029 2024-04-26T01:34:13.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:12 smithi006 ceph-mon[28419]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:14.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:14.526+0000 7f1ed8c00700 1 -- 172.21.15.6:0/1310138574 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f1edc19f870 con 0x7f1ec0039650 2024-04-26T01:34:14.522 INFO:teuthology.orchestra.run.smithi006.stdout:Added host 'smithi029' with addr '172.21.15.29' 2024-04-26T01:34:14.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:14.532+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1310138574 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ec0039650 msgr2=0x7f1ec003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:14.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:14.532+0000 7f1ee2a00700 1 --2- 172.21.15.6:0/1310138574 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ec0039650 0x7f1ec003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8006fd0 tx=0x7f1ec8006e40 comp rx=0 tx=0).stop 2024-04-26T01:34:14.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:14.532+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1310138574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1edc074de0 msgr2=0x7f1edc19e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:14.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:14.532+0000 7f1ee2a00700 1 --2- 172.21.15.6:0/1310138574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1edc074de0 0x7f1edc19e4a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc006010 tx=0x7f1ecc005780 comp rx=0 tx=0).stop 2024-04-26T01:34:14.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:14.532+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1310138574 shutdown_connections 2024-04-26T01:34:14.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:14.532+0000 7f1ee2a00700 1 --2- 172.21.15.6:0/1310138574 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ec0039650 0x7f1ec003bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:14.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:14.532+0000 7f1ee2a00700 1 --2- 172.21.15.6:0/1310138574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1edc074de0 0x7f1edc19e4a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:14.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:14.532+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1310138574 >> 172.21.15.6:0/1310138574 conn(0x7f1edc0fdf80 msgr2=0x7f1edc1041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:14.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:14.532+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1310138574 shutdown_connections 2024-04-26T01:34:14.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:14.532+0000 7f1ee2a00700 1 -- 172.21.15.6:0/1310138574 wait complete. 2024-04-26T01:34:14.813 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:14 smithi006 ceph-mon[28419]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:14.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:14.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:15.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T01:34:15.699 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:34:16.028 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:15 smithi006 ceph-mon[28419]: Added host smithi029 2024-04-26T01:34:16.028 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:15 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:16.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:15 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:16.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:16 smithi006 ceph-mon[28419]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:17.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.735+0000 7f05ca600700 1 -- 172.21.15.6:0/1262824451 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05c41084e0 msgr2=0x7f05c41088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:17.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.735+0000 7f05ca600700 1 --2- 172.21.15.6:0/1262824451 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05c41084e0 0x7f05c41088c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009910 tx=0x7f05b4009c20 comp rx=0 tx=0).stop 2024-04-26T01:34:17.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.736+0000 7f05ca600700 1 -- 172.21.15.6:0/1262824451 shutdown_connections 2024-04-26T01:34:17.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.736+0000 7f05ca600700 1 --2- 172.21.15.6:0/1262824451 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05c41084e0 0x7f05c41088c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:17.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.736+0000 7f05ca600700 1 -- 172.21.15.6:0/1262824451 >> 172.21.15.6:0/1262824451 conn(0x7f05c40fdf80 msgr2=0x7f05c41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:17.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.736+0000 7f05ca600700 1 -- 172.21.15.6:0/1262824451 shutdown_connections 2024-04-26T01:34:17.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.736+0000 7f05ca600700 1 -- 172.21.15.6:0/1262824451 wait complete. 2024-04-26T01:34:17.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.737+0000 7f05ca600700 1 Processor -- start 2024-04-26T01:34:17.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.737+0000 7f05ca600700 1 -- start start 2024-04-26T01:34:17.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.737+0000 7f05ca600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05c41084e0 0x7f05c419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:17.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.737+0000 7f05ca600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f05c410c940 con 0x7f05c41084e0 2024-04-26T01:34:17.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.737+0000 7f05c3400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05c41084e0 0x7f05c419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:17.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.738+0000 7f05c3400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05c41084e0 0x7f05c419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43440/0 (socket says 172.21.15.6:43440) 2024-04-26T01:34:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.738+0000 7f05c3400700 1 -- 172.21.15.6:0/1899199524 learned_addr learned my addr 172.21.15.6:0/1899199524 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:34:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.738+0000 7f05c3400700 1 -- 172.21.15.6:0/1899199524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c419a620 con 0x7f05c41084e0 2024-04-26T01:34:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.738+0000 7f05c3400700 1 --2- 172.21.15.6:0/1899199524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05c41084e0 0x7f05c419a0e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f05b4006010 tx=0x7f05b4005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.738+0000 7f05bbe00700 1 -- 172.21.15.6:0/1899199524 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05b4026070 con 0x7f05c41084e0 2024-04-26T01:34:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.738+0000 7f05bbe00700 1 -- 172.21.15.6:0/1899199524 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05b400bba0 con 0x7f05c41084e0 2024-04-26T01:34:17.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.738+0000 7f05ca600700 1 -- 172.21.15.6:0/1899199524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c419a910 con 0x7f05c41084e0 2024-04-26T01:34:17.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.739+0000 7f05bbe00700 1 -- 172.21.15.6:0/1899199524 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05b4003910 con 0x7f05c41084e0 2024-04-26T01:34:17.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.739+0000 7f05ca600700 1 -- 172.21.15.6:0/1899199524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c419e9b0 con 0x7f05c41084e0 2024-04-26T01:34:17.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.740+0000 7f05ca600700 1 -- 172.21.15.6:0/1899199524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05c4103b90 con 0x7f05c41084e0 2024-04-26T01:34:17.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.740+0000 7f05bbe00700 1 -- 172.21.15.6:0/1899199524 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f05b4029070 con 0x7f05c41084e0 2024-04-26T01:34:17.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.740+0000 7f05bbe00700 1 --2- 172.21.15.6:0/1899199524 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f05a80396b0 0x7f05a803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:17.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.740+0000 7f05bbe00700 1 -- 172.21.15.6:0/1899199524 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f05b401e070 con 0x7f05c41084e0 2024-04-26T01:34:17.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.741+0000 7f05c2a00700 1 --2- 172.21.15.6:0/1899199524 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f05a80396b0 0x7f05a803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:17.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.741+0000 7f05c2a00700 1 --2- 172.21.15.6:0/1899199524 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f05a80396b0 0x7f05a803bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f05b0006fd0 tx=0x7f05b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:17.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.750+0000 7f05bbe00700 1 -- 172.21.15.6:0/1899199524 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f05b4013070 con 0x7f05c41084e0 2024-04-26T01:34:17.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.985+0000 7f05ca600700 1 -- 172.21.15.6:0/1899199524 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f05c419b440 con 0x7f05a80396b0 2024-04-26T01:34:17.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.989+0000 7f05bbe00700 1 -- 172.21.15.6:0/1899199524 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+158 (secure 0 0 0) 0x7f05c419b440 con 0x7f05a80396b0 2024-04-26T01:34:17.984 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:34:17.984 INFO:teuthology.orchestra.run.smithi006.stdout:[{"addr": "172.21.15.6", "hostname": "smithi006", "labels": [], "status": ""}, {"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}] 2024-04-26T01:34:17.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.992+0000 7f05ca600700 1 -- 172.21.15.6:0/1899199524 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f05a80396b0 msgr2=0x7f05a803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:17.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.992+0000 7f05ca600700 1 --2- 172.21.15.6:0/1899199524 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f05a80396b0 0x7f05a803bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f05b0006fd0 tx=0x7f05b0006e40 comp rx=0 tx=0).stop 2024-04-26T01:34:17.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.992+0000 7f05ca600700 1 -- 172.21.15.6:0/1899199524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05c41084e0 msgr2=0x7f05c419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:17.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.992+0000 7f05ca600700 1 --2- 172.21.15.6:0/1899199524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05c41084e0 0x7f05c419a0e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f05b4006010 tx=0x7f05b4005780 comp rx=0 tx=0).stop 2024-04-26T01:34:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.993+0000 7f05ca600700 1 -- 172.21.15.6:0/1899199524 shutdown_connections 2024-04-26T01:34:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.993+0000 7f05ca600700 1 --2- 172.21.15.6:0/1899199524 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f05a80396b0 0x7f05a803bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.993+0000 7f05ca600700 1 --2- 172.21.15.6:0/1899199524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05c41084e0 0x7f05c419a0e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.993+0000 7f05ca600700 1 -- 172.21.15.6:0/1899199524 >> 172.21.15.6:0/1899199524 conn(0x7f05c40fdf80 msgr2=0x7f05c4109f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.993+0000 7f05ca600700 1 -- 172.21.15.6:0/1899199524 shutdown_connections 2024-04-26T01:34:17.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:17.993+0000 7f05ca600700 1 -- 172.21.15.6:0/1899199524 wait complete. 2024-04-26T01:34:18.690 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:18 smithi006 ceph-mon[28419]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:18.691 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:18 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:18.780 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi188 2024-04-26T01:34:18.781 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-26T01:34:18.781 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.conf 2024-04-26T01:34:18.798 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-26T01:34:18.798 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:34:18.856 INFO:tasks.cephadm:Adding host smithi188 to orchestrator... 2024-04-26T01:34:18.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch host add smithi188 2024-04-26T01:34:19.327 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:34:19.648 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:19 smithi006 ceph-mon[28419]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:34:20.653 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:20 smithi006 ceph-mon[28419]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:21.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.483+0000 7f2447c00700 1 -- 172.21.15.6:0/1282854283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24401084e0 msgr2=0x7f24401088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:21.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.483+0000 7f2447c00700 1 --2- 172.21.15.6:0/1282854283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24401084e0 0x7f24401088c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2434009910 tx=0x7f2434009c20 comp rx=0 tx=0).stop 2024-04-26T01:34:21.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.483+0000 7f2447c00700 1 -- 172.21.15.6:0/1282854283 shutdown_connections 2024-04-26T01:34:21.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.483+0000 7f2447c00700 1 --2- 172.21.15.6:0/1282854283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24401084e0 0x7f24401088c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:21.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.483+0000 7f2447c00700 1 -- 172.21.15.6:0/1282854283 >> 172.21.15.6:0/1282854283 conn(0x7f2440076ea0 msgr2=0x7f24400772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:21.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.483+0000 7f2447c00700 1 -- 172.21.15.6:0/1282854283 shutdown_connections 2024-04-26T01:34:21.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.483+0000 7f2447c00700 1 -- 172.21.15.6:0/1282854283 wait complete. 2024-04-26T01:34:21.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.484+0000 7f2447c00700 1 Processor -- start 2024-04-26T01:34:21.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.485+0000 7f2447c00700 1 -- start start 2024-04-26T01:34:21.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.485+0000 7f2447c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24401084e0 0x7f244019a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:21.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.485+0000 7f2447c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f244010c920 con 0x7f24401084e0 2024-04-26T01:34:21.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.485+0000 7f2445200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24401084e0 0x7f244019a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:21.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.485+0000 7f2445200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24401084e0 0x7f244019a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42064/0 (socket says 172.21.15.6:42064) 2024-04-26T01:34:21.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.485+0000 7f2445200700 1 -- 172.21.15.6:0/27110153 learned_addr learned my addr 172.21.15.6:0/27110153 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:34:21.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.486+0000 7f2445200700 1 -- 172.21.15.6:0/27110153 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244019a5e0 con 0x7f24401084e0 2024-04-26T01:34:21.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.486+0000 7f2445200700 1 --2- 172.21.15.6:0/27110153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24401084e0 0x7f244019a0a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2434006010 tx=0x7f2434005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:21.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.486+0000 7f243d600700 1 -- 172.21.15.6:0/27110153 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2434026070 con 0x7f24401084e0 2024-04-26T01:34:21.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.486+0000 7f2447c00700 1 -- 172.21.15.6:0/27110153 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244019a8d0 con 0x7f24401084e0 2024-04-26T01:34:21.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.486+0000 7f243d600700 1 -- 172.21.15.6:0/27110153 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f243400bba0 con 0x7f24401084e0 2024-04-26T01:34:21.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.486+0000 7f243d600700 1 -- 172.21.15.6:0/27110153 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2434003910 con 0x7f24401084e0 2024-04-26T01:34:21.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.487+0000 7f2447c00700 1 -- 172.21.15.6:0/27110153 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244019e970 con 0x7f24401084e0 2024-04-26T01:34:21.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.488+0000 7f243d600700 1 -- 172.21.15.6:0/27110153 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f2434029070 con 0x7f24401084e0 2024-04-26T01:34:21.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.488+0000 7f2447c00700 1 -- 172.21.15.6:0/27110153 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2440103b90 con 0x7f24401084e0 2024-04-26T01:34:21.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.488+0000 7f243d600700 1 --2- 172.21.15.6:0/27110153 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f24280396b0 0x7f242803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:21.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.488+0000 7f243d600700 1 -- 172.21.15.6:0/27110153 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f243401e070 con 0x7f24401084e0 2024-04-26T01:34:21.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.489+0000 7f243fe00700 1 --2- 172.21.15.6:0/27110153 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f24280396b0 0x7f242803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:21.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.489+0000 7f243fe00700 1 --2- 172.21.15.6:0/27110153 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f24280396b0 0x7f242803bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f242c006fd0 tx=0x7f242c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:21.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.494+0000 7f243d600700 1 -- 172.21.15.6:0/27110153 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2434013070 con 0x7f24401084e0 2024-04-26T01:34:21.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:21.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:21.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:21.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:21.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:21.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:21.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:21.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:21 smithi006 ceph-mon[28419]: Updating smithi029:/etc/ceph/ceph.conf 2024-04-26T01:34:21.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:21 smithi006 ceph-mon[28419]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:34:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:21.729+0000 7f2447c00700 1 -- 172.21.15.6:0/27110153 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}) -- 0x7f244019b400 con 0x7f24280396b0 2024-04-26T01:34:23.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:22 smithi006 ceph-mon[28419]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:23.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:22 smithi006 ceph-mon[28419]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:34:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:22 smithi006 ceph-mon[28419]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:34:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:22 smithi006 ceph-mon[28419]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:24.313 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:23 smithi006 ceph-mon[28419]: Deploying cephadm binary to smithi188 2024-04-26T01:34:25.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:24 smithi006 ceph-mon[28419]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:26.699+0000 7f243d600700 1 -- 172.21.15.6:0/27110153 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f244019b400 con 0x7f24280396b0 2024-04-26T01:34:26.695 INFO:teuthology.orchestra.run.smithi006.stdout:Added host 'smithi188' with addr '172.21.15.188' 2024-04-26T01:34:26.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:26.703+0000 7f2447c00700 1 -- 172.21.15.6:0/27110153 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f24280396b0 msgr2=0x7f242803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:26.703+0000 7f2447c00700 1 --2- 172.21.15.6:0/27110153 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f24280396b0 0x7f242803bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f242c006fd0 tx=0x7f242c006e40 comp rx=0 tx=0).stop 2024-04-26T01:34:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:26.703+0000 7f2447c00700 1 -- 172.21.15.6:0/27110153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24401084e0 msgr2=0x7f244019a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:26.703+0000 7f2447c00700 1 --2- 172.21.15.6:0/27110153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24401084e0 0x7f244019a0a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2434006010 tx=0x7f2434005780 comp rx=0 tx=0).stop 2024-04-26T01:34:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:26.704+0000 7f2447c00700 1 -- 172.21.15.6:0/27110153 shutdown_connections 2024-04-26T01:34:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:26.704+0000 7f2447c00700 1 --2- 172.21.15.6:0/27110153 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f24280396b0 0x7f242803bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:26.704+0000 7f2447c00700 1 --2- 172.21.15.6:0/27110153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24401084e0 0x7f244019a0a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:26.704+0000 7f2447c00700 1 -- 172.21.15.6:0/27110153 >> 172.21.15.6:0/27110153 conn(0x7f2440076ea0 msgr2=0x7f2440104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:26.704+0000 7f2447c00700 1 -- 172.21.15.6:0/27110153 shutdown_connections 2024-04-26T01:34:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:26.704+0000 7f2447c00700 1 -- 172.21.15.6:0/27110153 wait complete. 2024-04-26T01:34:27.039 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:26 smithi006 ceph-mon[28419]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:27.039 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:27.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:27.473 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T01:34:27.917 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:34:28.172 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:27 smithi006 ceph-mon[28419]: Added host smithi188 2024-04-26T01:34:28.173 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:27 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:28.173 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:27 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:28.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:28 smithi006 ceph-mon[28419]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.786+0000 7f748ae00700 1 -- 172.21.15.6:0/103177873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74841084e0 msgr2=0x7f74841088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.786+0000 7f748ae00700 1 --2- 172.21.15.6:0/103177873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74841084e0 0x7f74841088c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f747400b0a0 tx=0x7f747400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:34:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.786+0000 7f748ae00700 1 -- 172.21.15.6:0/103177873 shutdown_connections 2024-04-26T01:34:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.786+0000 7f748ae00700 1 --2- 172.21.15.6:0/103177873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74841084e0 0x7f74841088c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.786+0000 7f748ae00700 1 -- 172.21.15.6:0/103177873 >> 172.21.15.6:0/103177873 conn(0x7f7484076ea0 msgr2=0x7f74840772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.786+0000 7f748ae00700 1 -- 172.21.15.6:0/103177873 shutdown_connections 2024-04-26T01:34:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.787+0000 7f748ae00700 1 -- 172.21.15.6:0/103177873 wait complete. 2024-04-26T01:34:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.787+0000 7f748ae00700 1 Processor -- start 2024-04-26T01:34:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.788+0000 7f748ae00700 1 -- start start 2024-04-26T01:34:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.788+0000 7f748ae00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74841084e0 0x7f7484199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.788+0000 7f748ae00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f748410c920 con 0x7f74841084e0 2024-04-26T01:34:29.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.788+0000 7f7483e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74841084e0 0x7f7484199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:29.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.788+0000 7f7483e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74841084e0 0x7f7484199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56040/0 (socket says 172.21.15.6:56040) 2024-04-26T01:34:29.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.788+0000 7f7483e00700 1 -- 172.21.15.6:0/892300926 learned_addr learned my addr 172.21.15.6:0/892300926 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:34:29.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.789+0000 7f7483e00700 1 -- 172.21.15.6:0/892300926 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f748419a520 con 0x7f74841084e0 2024-04-26T01:34:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.789+0000 7f7483e00700 1 --2- 172.21.15.6:0/892300926 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74841084e0 0x7f7484199fe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f747400d010 tx=0x7f7474003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.789+0000 7f7480c00700 1 -- 172.21.15.6:0/892300926 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7474021020 con 0x7f74841084e0 2024-04-26T01:34:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.789+0000 7f7480c00700 1 -- 172.21.15.6:0/892300926 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74740044b0 con 0x7f74841084e0 2024-04-26T01:34:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.789+0000 7f748ae00700 1 -- 172.21.15.6:0/892300926 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f748419a7b0 con 0x7f74841084e0 2024-04-26T01:34:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.789+0000 7f7480c00700 1 -- 172.21.15.6:0/892300926 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7474017040 con 0x7f74841084e0 2024-04-26T01:34:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.790+0000 7f748ae00700 1 -- 172.21.15.6:0/892300926 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f748419e8b0 con 0x7f74841084e0 2024-04-26T01:34:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.791+0000 7f748ae00700 1 -- 172.21.15.6:0/892300926 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7484103b90 con 0x7f74841084e0 2024-04-26T01:34:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.791+0000 7f7480c00700 1 -- 172.21.15.6:0/892300926 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f7474024020 con 0x7f74841084e0 2024-04-26T01:34:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.791+0000 7f7480c00700 1 --2- 172.21.15.6:0/892300926 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f74680396a0 0x7f746803bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.791+0000 7f7480c00700 1 -- 172.21.15.6:0/892300926 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f747400f050 con 0x7f74841084e0 2024-04-26T01:34:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.792+0000 7f7483400700 1 --2- 172.21.15.6:0/892300926 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f74680396a0 0x7f746803bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.792+0000 7f7483400700 1 --2- 172.21.15.6:0/892300926 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f74680396a0 0x7f746803bb60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7470006fd0 tx=0x7f7470006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:29.797+0000 7f7480c00700 1 -- 172.21.15.6:0/892300926 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f747402b020 con 0x7f74841084e0 2024-04-26T01:34:30.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:30.035+0000 7f748ae00700 1 -- 172.21.15.6:0/892300926 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f74840640a0 con 0x7f74680396a0 2024-04-26T01:34:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:30.037+0000 7f7480c00700 1 -- 172.21.15.6:0/892300926 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+238 (secure 0 0 0) 0x7f74840640a0 con 0x7f74680396a0 2024-04-26T01:34:30.031 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:34:30.031 INFO:teuthology.orchestra.run.smithi006.stdout:[{"addr": "172.21.15.6", "hostname": "smithi006", "labels": [], "status": ""}, {"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.188", "hostname": "smithi188", "labels": [], "status": ""}] 2024-04-26T01:34:30.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:30.039+0000 7f748ae00700 1 -- 172.21.15.6:0/892300926 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f74680396a0 msgr2=0x7f746803bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:30.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:30.039+0000 7f748ae00700 1 --2- 172.21.15.6:0/892300926 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f74680396a0 0x7f746803bb60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7470006fd0 tx=0x7f7470006e40 comp rx=0 tx=0).stop 2024-04-26T01:34:30.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:30.039+0000 7f748ae00700 1 -- 172.21.15.6:0/892300926 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74841084e0 msgr2=0x7f7484199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:30.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:30.039+0000 7f748ae00700 1 --2- 172.21.15.6:0/892300926 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74841084e0 0x7f7484199fe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f747400d010 tx=0x7f7474003ce0 comp rx=0 tx=0).stop 2024-04-26T01:34:30.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:30.039+0000 7f748ae00700 1 -- 172.21.15.6:0/892300926 shutdown_connections 2024-04-26T01:34:30.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:30.039+0000 7f748ae00700 1 --2- 172.21.15.6:0/892300926 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f74680396a0 0x7f746803bb60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:30.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:30.039+0000 7f748ae00700 1 --2- 172.21.15.6:0/892300926 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74841084e0 0x7f7484199fe0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:30.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:30.039+0000 7f748ae00700 1 -- 172.21.15.6:0/892300926 >> 172.21.15.6:0/892300926 conn(0x7f7484076ea0 msgr2=0x7f7484104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:30.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:30.040+0000 7f748ae00700 1 -- 172.21.15.6:0/892300926 shutdown_connections 2024-04-26T01:34:30.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:30.040+0000 7f748ae00700 1 -- 172.21.15.6:0/892300926 wait complete. 2024-04-26T01:34:30.563 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:30 smithi006 ceph-mon[28419]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:30.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:30.686 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T01:34:30.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T01:34:31.181 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:34:31.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:31 smithi006 ceph-mon[28419]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:34:32.619 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:32 smithi006 ceph-mon[28419]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:33.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.099+0000 7f5bd1800700 1 -- 172.21.15.6:0/1780640661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bcc074e70 msgr2=0x7f5bcc075250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:33.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.099+0000 7f5bd1800700 1 --2- 172.21.15.6:0/1780640661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bcc074e70 0x7f5bcc075250 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009910 tx=0x7f5bbc009c20 comp rx=0 tx=0).stop 2024-04-26T01:34:33.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.099+0000 7f5bd1800700 1 -- 172.21.15.6:0/1780640661 shutdown_connections 2024-04-26T01:34:33.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.099+0000 7f5bd1800700 1 --2- 172.21.15.6:0/1780640661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bcc074e70 0x7f5bcc075250 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:33.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.099+0000 7f5bd1800700 1 -- 172.21.15.6:0/1780640661 >> 172.21.15.6:0/1780640661 conn(0x7f5bcc0fdda0 msgr2=0x7f5bcc1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:33.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.099+0000 7f5bd1800700 1 -- 172.21.15.6:0/1780640661 shutdown_connections 2024-04-26T01:34:33.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.099+0000 7f5bd1800700 1 -- 172.21.15.6:0/1780640661 wait complete. 2024-04-26T01:34:33.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.100+0000 7f5bd1800700 1 Processor -- start 2024-04-26T01:34:33.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.100+0000 7f5bd1800700 1 -- start start 2024-04-26T01:34:33.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.101+0000 7f5bd1800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bcc074e70 0x7f5bcc199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.101+0000 7f5bd1800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc10e9f0 con 0x7f5bcc074e70 2024-04-26T01:34:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.101+0000 7f5bcbe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bcc074e70 0x7f5bcc199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.101+0000 7f5bcbe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bcc074e70 0x7f5bcc199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56064/0 (socket says 172.21.15.6:56064) 2024-04-26T01:34:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.101+0000 7f5bcbe00700 1 -- 172.21.15.6:0/218326018 learned_addr learned my addr 172.21.15.6:0/218326018 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:34:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.101+0000 7f5bcbe00700 1 -- 172.21.15.6:0/218326018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bcc19a4a0 con 0x7f5bcc074e70 2024-04-26T01:34:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.102+0000 7f5bcbe00700 1 --2- 172.21.15.6:0/218326018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bcc074e70 0x7f5bcc199f60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc006010 tx=0x7f5bbc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.102+0000 7f5bc9600700 1 -- 172.21.15.6:0/218326018 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bbc026070 con 0x7f5bcc074e70 2024-04-26T01:34:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.102+0000 7f5bc9600700 1 -- 172.21.15.6:0/218326018 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bbc00bc00 con 0x7f5bcc074e70 2024-04-26T01:34:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.102+0000 7f5bc9600700 1 -- 172.21.15.6:0/218326018 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bbc003910 con 0x7f5bcc074e70 2024-04-26T01:34:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.102+0000 7f5bd1800700 1 -- 172.21.15.6:0/218326018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bcc19a730 con 0x7f5bcc074e70 2024-04-26T01:34:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.102+0000 7f5bd1800700 1 -- 172.21.15.6:0/218326018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bcc19ab90 con 0x7f5bcc074e70 2024-04-26T01:34:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.103+0000 7f5bc9600700 1 -- 172.21.15.6:0/218326018 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f5bbc029070 con 0x7f5bcc074e70 2024-04-26T01:34:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.103+0000 7f5bd1800700 1 -- 172.21.15.6:0/218326018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bcc076520 con 0x7f5bcc074e70 2024-04-26T01:34:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.104+0000 7f5bc9600700 1 --2- 172.21.15.6:0/218326018 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5bb00396b0 0x7f5bb003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.104+0000 7f5bc9600700 1 -- 172.21.15.6:0/218326018 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5bbc01e070 con 0x7f5bcc074e70 2024-04-26T01:34:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.104+0000 7f5bc3400700 1 --2- 172.21.15.6:0/218326018 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5bb00396b0 0x7f5bb003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.105+0000 7f5bc3400700 1 --2- 172.21.15.6:0/218326018 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5bb00396b0 0x7f5bb003bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4006fd0 tx=0x7f5bb4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:33.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.110+0000 7f5bc9600700 1 -- 172.21.15.6:0/218326018 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5bbc013070 con 0x7f5bcc074e70 2024-04-26T01:34:33.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.393+0000 7f5bd1800700 1 -- 172.21.15.6:0/218326018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f5bcc19b2c0 con 0x7f5bcc074e70 2024-04-26T01:34:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.532+0000 7f5bc9600700 1 -- 172.21.15.6:0/218326018 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f5bbc00fe20 con 0x7f5bcc074e70 2024-04-26T01:34:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.537+0000 7f5bd1800700 1 -- 172.21.15.6:0/218326018 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5bb00396b0 msgr2=0x7f5bb003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.537+0000 7f5bd1800700 1 --2- 172.21.15.6:0/218326018 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5bb00396b0 0x7f5bb003bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4006fd0 tx=0x7f5bb4006e40 comp rx=0 tx=0).stop 2024-04-26T01:34:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.537+0000 7f5bd1800700 1 -- 172.21.15.6:0/218326018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bcc074e70 msgr2=0x7f5bcc199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.537+0000 7f5bd1800700 1 --2- 172.21.15.6:0/218326018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bcc074e70 0x7f5bcc199f60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc006010 tx=0x7f5bbc005780 comp rx=0 tx=0).stop 2024-04-26T01:34:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.537+0000 7f5bd1800700 1 -- 172.21.15.6:0/218326018 shutdown_connections 2024-04-26T01:34:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.537+0000 7f5bd1800700 1 --2- 172.21.15.6:0/218326018 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5bb00396b0 0x7f5bb003bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.537+0000 7f5bd1800700 1 --2- 172.21.15.6:0/218326018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5bcc074e70 0x7f5bcc199f60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.537+0000 7f5bd1800700 1 -- 172.21.15.6:0/218326018 >> 172.21.15.6:0/218326018 conn(0x7f5bcc0fdda0 msgr2=0x7f5bcc108540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.537+0000 7f5bd1800700 1 -- 172.21.15.6:0/218326018 shutdown_connections 2024-04-26T01:34:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:34:33.537+0000 7f5bd1800700 1 -- 172.21.15.6:0/218326018 wait complete. 2024-04-26T01:34:33.533 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-04-26T01:34:33.813 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:33.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:33.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:33.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:33.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:33.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:33.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:33.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:33 smithi006 ceph-mon[28419]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-26T01:34:33.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:33 smithi006 ceph-mon[28419]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:34:33.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:33 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/218326018' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T01:34:34.245 INFO:tasks.cephadm:Adding mon.a on smithi006 2024-04-26T01:34:34.245 INFO:tasks.cephadm:Adding mon.b on smithi029 2024-04-26T01:34:34.245 INFO:tasks.cephadm:Adding mon.c on smithi188 2024-04-26T01:34:34.246 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi006:172.21.15.6=a;smithi029:172.21.15.29=b;smithi188:172.21.15.188=c' 2024-04-26T01:34:34.482 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:34:34.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:34 smithi006 ceph-mon[28419]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:34.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:34 smithi006 ceph-mon[28419]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:34:34.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:34 smithi006 ceph-mon[28419]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:34:34.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:34 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/218326018' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T01:34:34.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:34 smithi006 ceph-mon[28419]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:34:34.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:34.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:34.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:35.891 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.894+0000 7f7ec3199700 1 -- 172.21.15.188:0/3286761566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ebc1015b0 msgr2=0x7f7ebc101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:35.891 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.894+0000 7f7ec3199700 1 --2- 172.21.15.188:0/3286761566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ebc1015b0 0x7f7ebc101990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0009910 tx=0x7f7eb0009c20 comp rx=0 tx=0).stop 2024-04-26T01:34:35.891 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.894+0000 7f7ec3199700 1 -- 172.21.15.188:0/3286761566 shutdown_connections 2024-04-26T01:34:35.891 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.894+0000 7f7ec3199700 1 --2- 172.21.15.188:0/3286761566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ebc1015b0 0x7f7ebc101990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:35.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.894+0000 7f7ec3199700 1 -- 172.21.15.188:0/3286761566 >> 172.21.15.188:0/3286761566 conn(0x7f7ebc0fce40 msgr2=0x7f7ebc0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:35.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.894+0000 7f7ec3199700 1 -- 172.21.15.188:0/3286761566 shutdown_connections 2024-04-26T01:34:35.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.894+0000 7f7ec3199700 1 -- 172.21.15.188:0/3286761566 wait complete. 2024-04-26T01:34:35.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.895+0000 7f7ec3199700 1 Processor -- start 2024-04-26T01:34:35.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.895+0000 7f7ec3199700 1 -- start start 2024-04-26T01:34:35.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.895+0000 7f7ec3199700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ebc1015b0 0x7f7ebc196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:35.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.895+0000 7f7ec3199700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7ebc10b830 con 0x7f7ebc1015b0 2024-04-26T01:34:35.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.896+0000 7f7ec0f35700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ebc1015b0 0x7f7ebc196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:35.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.896+0000 7f7ec0f35700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ebc1015b0 0x7f7ebc196d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.188:55316/0 (socket says 172.21.15.188:55316) 2024-04-26T01:34:35.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.896+0000 7f7ec0f35700 1 -- 172.21.15.188:0/1694076846 learned_addr learned my addr 172.21.15.188:0/1694076846 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-26T01:34:35.893 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.896+0000 7f7ec0f35700 1 -- 172.21.15.188:0/1694076846 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ebc1972d0 con 0x7f7ebc1015b0 2024-04-26T01:34:35.893 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.897+0000 7f7ec0f35700 1 --2- 172.21.15.188:0/1694076846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ebc1015b0 0x7f7ebc196d90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7eb00056d0 tx=0x7f7eb00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:35.893 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.897+0000 7f7eb9ffb700 1 -- 172.21.15.188:0/1694076846 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7eb0028070 con 0x7f7ebc1015b0 2024-04-26T01:34:35.893 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.897+0000 7f7ec3199700 1 -- 172.21.15.188:0/1694076846 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ebc197560 con 0x7f7ebc1015b0 2024-04-26T01:34:35.893 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.897+0000 7f7ec3199700 1 -- 172.21.15.188:0/1694076846 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ebc19b660 con 0x7f7ebc1015b0 2024-04-26T01:34:35.893 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.897+0000 7f7eb9ffb700 1 -- 172.21.15.188:0/1694076846 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7eb000bba0 con 0x7f7ebc1015b0 2024-04-26T01:34:35.894 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.898+0000 7f7eb9ffb700 1 -- 172.21.15.188:0/1694076846 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7eb00037f0 con 0x7f7ebc1015b0 2024-04-26T01:34:35.894 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.898+0000 7f7ec3199700 1 -- 172.21.15.188:0/1694076846 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ebc102a90 con 0x7f7ebc1015b0 2024-04-26T01:34:35.895 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.899+0000 7f7eb9ffb700 1 -- 172.21.15.188:0/1694076846 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f7eb002b070 con 0x7f7ebc1015b0 2024-04-26T01:34:35.895 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.899+0000 7f7eb9ffb700 1 --2- 172.21.15.188:0/1694076846 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7eac0392b0 0x7f7eac03b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:35.895 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.899+0000 7f7eb9ffb700 1 -- 172.21.15.188:0/1694076846 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7eb0020070 con 0x7f7ebc1015b0 2024-04-26T01:34:35.895 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.900+0000 7f7ebbfff700 1 --2- 172.21.15.188:0/1694076846 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7eac0392b0 0x7f7eac03b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:35.896 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.900+0000 7f7ebbfff700 1 --2- 172.21.15.188:0/1694076846 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7eac0392b0 0x7f7eac03b770 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8006fd0 tx=0x7f7ea8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:35.912 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:35.917+0000 7f7eb9ffb700 1 -- 172.21.15.188:0/1694076846 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7eb0013070 con 0x7f7ebc1015b0 2024-04-26T01:34:36.119 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:36.124+0000 7f7ec3199700 1 -- 172.21.15.188:0/1694076846 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi006:172.21.15.6=a;smithi029:172.21.15.29=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}) -- 0x7f7ebc1980f0 con 0x7f7eac0392b0 2024-04-26T01:34:36.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:36.139+0000 7f7eb9ffb700 1 -- 172.21.15.188:0/1694076846 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7ebc1980f0 con 0x7f7eac0392b0 2024-04-26T01:34:36.135 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mon update... 2024-04-26T01:34:36.138 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:36.141+0000 7f7ec3199700 1 -- 172.21.15.188:0/1694076846 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7eac0392b0 msgr2=0x7f7eac03b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:36.138 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:36.141+0000 7f7ec3199700 1 --2- 172.21.15.188:0/1694076846 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7eac0392b0 0x7f7eac03b770 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8006fd0 tx=0x7f7ea8006e40 comp rx=0 tx=0).stop 2024-04-26T01:34:36.138 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:36.141+0000 7f7ec3199700 1 -- 172.21.15.188:0/1694076846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ebc1015b0 msgr2=0x7f7ebc196d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:36.138 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:36.141+0000 7f7ec3199700 1 --2- 172.21.15.188:0/1694076846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ebc1015b0 0x7f7ebc196d90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7eb00056d0 tx=0x7f7eb00057b0 comp rx=0 tx=0).stop 2024-04-26T01:34:36.138 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:36.141+0000 7f7ec3199700 1 -- 172.21.15.188:0/1694076846 shutdown_connections 2024-04-26T01:34:36.138 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:36.141+0000 7f7ec3199700 1 --2- 172.21.15.188:0/1694076846 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7eac0392b0 0x7f7eac03b770 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:36.138 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:36.142+0000 7f7ec3199700 1 --2- 172.21.15.188:0/1694076846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ebc1015b0 0x7f7ebc196d90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:36.138 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:36.142+0000 7f7ec3199700 1 -- 172.21.15.188:0/1694076846 >> 172.21.15.188:0/1694076846 conn(0x7f7ebc0fce40 msgr2=0x7f7ebc102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:36.138 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:36.142+0000 7f7ec3199700 1 -- 172.21.15.188:0/1694076846 shutdown_connections 2024-04-26T01:34:36.138 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:36.142+0000 7f7ec3199700 1 -- 172.21.15.188:0/1694076846 wait complete. 2024-04-26T01:34:36.773 DEBUG:teuthology.orchestra.run.smithi029:mon.b> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T01:34:36.776 DEBUG:teuthology.orchestra.run.smithi188:mon.c> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T01:34:36.778 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T01:34:36.778 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T01:34:36.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:36 smithi006 ceph-mon[28419]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:36.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:36 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:36.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:36 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:36.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:36 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:36.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:36 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:36.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:36 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:36.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:36 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:36.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:36 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:37.525 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:34:37.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:37 smithi006 ceph-mon[28419]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi006:172.21.15.6=a;smithi029:172.21.15.29=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:37.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:37 smithi006 ceph-mon[28419]: Saving service mon spec with placement smithi006:172.21.15.6=a;smithi029:172.21.15.29=b;smithi188:172.21.15.188=c;count:3 2024-04-26T01:34:37.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:37 smithi006 ceph-mon[28419]: Deploying daemon mon.c on smithi188 2024-04-26T01:34:38.813 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:38 smithi006 ceph-mon[28419]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:39.871 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.851+0000 7fae94efc700 1 -- 172.21.15.188:0/3991587629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae901015b0 msgr2=0x7fae90101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:39.871 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.851+0000 7fae94efc700 1 --2- 172.21.15.188:0/3991587629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae901015b0 0x7fae90101990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fae84009910 tx=0x7fae84009c20 comp rx=0 tx=0).stop 2024-04-26T01:34:39.871 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.851+0000 7fae94efc700 1 -- 172.21.15.188:0/3991587629 shutdown_connections 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.851+0000 7fae94efc700 1 --2- 172.21.15.188:0/3991587629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae901015b0 0x7fae90101990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.851+0000 7fae94efc700 1 -- 172.21.15.188:0/3991587629 >> 172.21.15.188:0/3991587629 conn(0x7fae900fce60 msgr2=0x7fae900ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.851+0000 7fae94efc700 1 -- 172.21.15.188:0/3991587629 shutdown_connections 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.851+0000 7fae94efc700 1 -- 172.21.15.188:0/3991587629 wait complete. 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.852+0000 7fae94efc700 1 Processor -- start 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.852+0000 7fae94efc700 1 -- start start 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.852+0000 7fae94efc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae901015b0 0x7fae90197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.852+0000 7fae94efc700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fae9010b830 con 0x7fae901015b0 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.852+0000 7fae8e59c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae901015b0 0x7fae90197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.853+0000 7fae8e59c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae901015b0 0x7fae90197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.188:55328/0 (socket says 172.21.15.188:55328) 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.853+0000 7fae8e59c700 1 -- 172.21.15.188:0/2986223335 learned_addr learned my addr 172.21.15.188:0/2986223335 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.853+0000 7fae8e59c700 1 -- 172.21.15.188:0/2986223335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae901983d0 con 0x7fae901015b0 2024-04-26T01:34:39.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.853+0000 7fae8e59c700 1 --2- 172.21.15.188:0/2986223335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae901015b0 0x7fae90197e90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fae840056d0 tx=0x7fae840057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:39.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.854+0000 7fae7f7fe700 1 -- 172.21.15.188:0/2986223335 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae84026070 con 0x7fae901015b0 2024-04-26T01:34:39.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.854+0000 7fae94efc700 1 -- 172.21.15.188:0/2986223335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae901986c0 con 0x7fae901015b0 2024-04-26T01:34:39.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.854+0000 7fae7f7fe700 1 -- 172.21.15.188:0/2986223335 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae8400bba0 con 0x7fae901015b0 2024-04-26T01:34:39.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.854+0000 7fae94efc700 1 -- 172.21.15.188:0/2986223335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae9019c760 con 0x7fae901015b0 2024-04-26T01:34:39.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.854+0000 7fae7f7fe700 1 -- 172.21.15.188:0/2986223335 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae84030060 con 0x7fae901015b0 2024-04-26T01:34:39.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.855+0000 7fae94efc700 1 -- 172.21.15.188:0/2986223335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae90102a90 con 0x7fae901015b0 2024-04-26T01:34:39.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.856+0000 7fae7f7fe700 1 -- 172.21.15.188:0/2986223335 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fae84029070 con 0x7fae901015b0 2024-04-26T01:34:39.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.856+0000 7fae7f7fe700 1 --2- 172.21.15.188:0/2986223335 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fae78039650 0x7fae7803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:39.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.856+0000 7fae7f7fe700 1 -- 172.21.15.188:0/2986223335 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fae8401e070 con 0x7fae901015b0 2024-04-26T01:34:39.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.857+0000 7fae8dd9b700 1 --2- 172.21.15.188:0/2986223335 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fae78039650 0x7fae7803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:39.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.858+0000 7fae8dd9b700 1 --2- 172.21.15.188:0/2986223335 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fae78039650 0x7fae7803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fae80006fd0 tx=0x7fae80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:39.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:39.874+0000 7fae7f7fe700 1 -- 172.21.15.188:0/2986223335 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fae84013070 con 0x7fae901015b0 2024-04-26T01:34:40.152 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:40.156+0000 7fae94efc700 1 -- 172.21.15.188:0/2986223335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fae900640a0 con 0x7fae901015b0 2024-04-26T01:34:40.153 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:40.157+0000 7fae7f7fe700 1 -- 172.21.15.188:0/2986223335 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+741 (secure 0 0 0) 0x7fae84004390 con 0x7fae901015b0 2024-04-26T01:34:40.153 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:34:40.154 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","modified":"2024-04-26T01:32:24.866051Z","created":"2024-04-26T01:32:24.866051Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T01:34:40.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:40.160+0000 7fae94efc700 1 -- 172.21.15.188:0/2986223335 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fae78039650 msgr2=0x7fae7803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:40.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:40.160+0000 7fae94efc700 1 --2- 172.21.15.188:0/2986223335 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fae78039650 0x7fae7803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fae80006fd0 tx=0x7fae80006e40 comp rx=0 tx=0).stop 2024-04-26T01:34:40.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:40.160+0000 7fae94efc700 1 -- 172.21.15.188:0/2986223335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae901015b0 msgr2=0x7fae90197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:40.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:40.160+0000 7fae94efc700 1 --2- 172.21.15.188:0/2986223335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae901015b0 0x7fae90197e90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fae840056d0 tx=0x7fae840057b0 comp rx=0 tx=0).stop 2024-04-26T01:34:40.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:40.160+0000 7fae94efc700 1 -- 172.21.15.188:0/2986223335 shutdown_connections 2024-04-26T01:34:40.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:40.160+0000 7fae94efc700 1 --2- 172.21.15.188:0/2986223335 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fae78039650 0x7fae7803bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:40.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:40.160+0000 7fae94efc700 1 --2- 172.21.15.188:0/2986223335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae901015b0 0x7fae90197e90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:40.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:40.160+0000 7fae94efc700 1 -- 172.21.15.188:0/2986223335 >> 172.21.15.188:0/2986223335 conn(0x7fae900fce60 msgr2=0x7fae9006a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:40.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:40.160+0000 7fae94efc700 1 -- 172.21.15.188:0/2986223335 shutdown_connections 2024-04-26T01:34:40.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:40.160+0000 7fae94efc700 1 -- 172.21.15.188:0/2986223335 wait complete. 2024-04-26T01:34:40.157 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-26T01:34:40.813 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:40 smithi006 ceph-mon[28419]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:40.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:40 smithi006 ceph-mon[28419]: from='client.? 172.21.15.188:0/2986223335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:34:41.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 podman[32733]: 2024-04-26 01:34:41.561141793 +0000 UTC m=+0.129727537 container create 344aa2641bc9ba02b95f7932791ffd516dca7f3b66847cb033e45af339296781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T01:34:42.031 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T01:34:42.032 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T01:34:42.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 podman[32733]: 2024-04-26 01:34:41.740642151 +0000 UTC m=+0.309227856 container init 344aa2641bc9ba02b95f7932791ffd516dca7f3b66847cb033e45af339296781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5) 2024-04-26T01:34:42.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 podman[32733]: 2024-04-26 01:34:41.743873921 +0000 UTC m=+0.312459623 container start 344aa2641bc9ba02b95f7932791ffd516dca7f3b66847cb033e45af339296781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-26T01:34:42.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T01:34:42.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-26T01:34:42.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pidfile_write: ignore empty --pid-file 2024-04-26T01:34:42.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: load: jerasure load: lrc 2024-04-26T01:34:42.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: RocksDB version: 7.9.2 2024-04-26T01:34:42.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Git sha 0 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: DB SUMMARY 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: DB Session ID: G4C89MJZPA4Z7Y4G1RCT 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: CURRENT file: CURRENT 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.error_if_exists: 0 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.create_if_missing: 0 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.paranoid_checks: 1 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.env: 0x55d4b1687cc0 2024-04-26T01:34:42.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T01:34:42.057 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.info_log: 0x55d4b2153be0 2024-04-26T01:34:42.057 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T01:34:42.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.statistics: (nil) 2024-04-26T01:34:42.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.use_fsync: 0 2024-04-26T01:34:42.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_log_file_size: 0 2024-04-26T01:34:42.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T01:34:42.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T01:34:42.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T01:34:42.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T01:34:42.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.allow_fallocate: 1 2024-04-26T01:34:42.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T01:34:42.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T01:34:42.059 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.use_direct_reads: 0 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.db_log_dir: 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.wal_dir: 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.write_buffer_manager: 0x55d4b20e70e0 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T01:34:42.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T01:34:42.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T01:34:42.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T01:34:42.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T01:34:42.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T01:34:42.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T01:34:42.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T01:34:42.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.unordered_write: 0 2024-04-26T01:34:42.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T01:34:42.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T01:34:42.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.row_cache: None 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.wal_filter: None 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.two_write_queues: 0 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.wal_compression: 0 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.atomic_flush: 0 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T01:34:42.062 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.log_readahead_size: 0 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_background_jobs: 2 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_background_compactions: -1 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_subcompactions: 1 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T01:34:42.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_open_files: -1 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_background_flushes: -1 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Compression algorithms supported: 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: kZSTD supported: 0 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: kXpressCompression supported: 0 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: kZlibCompression supported: 1 2024-04-26T01:34:42.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: kSnappyCompression supported: 1 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: kLZ4Compression supported: 1 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: kBZip2Compression supported: 0 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.merge_operator: 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_filter: None 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T01:34:42.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d4b2153dc0) 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: cache_index_and_filter_blocks: 1 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: pin_top_level_index_and_filter: 1 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_type: 0 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: data_block_index_type: 0 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_shortening: 1 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: checksum: 4 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: no_block_cache: 0 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache: 0x55d4b21191f0 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_name: BinnedLRUCache 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_options: 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: capacity : 536870912 2024-04-26T01:34:42.066 INFO:journalctl@ceph.mon.c.smithi188.stdout: num_shard_bits : 4 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: strict_capacity_limit : 0 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: high_pri_pool_ratio: 0.000 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_compressed: (nil) 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: persistent_cache: (nil) 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_size: 4096 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_size_deviation: 10 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_restart_interval: 16 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_block_restart_interval: 1 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: metadata_block_size: 4096 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: partition_filters: 0 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: use_delta_encoding: 1 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: filter_policy: bloomfilter 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: whole_key_filtering: 1 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: verify_compression: 0 2024-04-26T01:34:42.067 INFO:journalctl@ceph.mon.c.smithi188.stdout: read_amp_bytes_per_bit: 0 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout: format_version: 5 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout: enable_index_compression: 1 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_align: 0 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout: max_auto_readahead_size: 262144 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout: prepopulate_block_cache: 0 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout: initial_auto_readahead_size: 8192 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compression: NoCompression 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T01:34:42.068 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.num_levels: 7 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T01:34:42.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T01:34:42.070 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:34:42.070 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:34:42.070 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T01:34:42.070 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T01:34:42.070 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:34:42.070 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T01:34:42.070 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T01:34:42.070 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T01:34:42.070 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T01:34:42.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T01:34:42.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.inplace_update_support: 0 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.bloom_locality: 0 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.max_successive_merges: 0 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.ttl: 2592000 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T01:34:42.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.enable_blob_files: false 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.min_blob_size: 0 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-26T01:34:42.074 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e4c167a9-823b-4fa4-a722-9b0f8c9f02a6 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714095281773230, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714095281773663, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714095281, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e4c167a9-823b-4fa4-a722-9b0f8c9f02a6", "db_session_id": "G4C89MJZPA4Z7Y4G1RCT", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714095281773741, "job": 1, "event": "recovery_finished"} 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d4b222e000 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: DB pointer 0x55d4b2218000 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** DB Stats ** 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T01:34:42.075 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:34:42.076 INFO:journalctl@ceph.mon.c.smithi188.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: Block cache BinnedLRUCache@0x55d4b21191f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-26T01:34:42.077 INFO:journalctl@ceph.mon.c.smithi188.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: using public_addr v2:172.21.15.188:0/0 -> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: starting mon.c rank -1 at public addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] at bind addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(???) e0 preinit fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).mds e1 new map 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout: e1 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout: legacy client fscid: -1 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout: No filesystems configured 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T01:34:42.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Deploying cephadm binary to smithi029 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Added host smithi029 2024-04-26T01:34:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:34:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Updating smithi029:/etc/ceph/ceph.conf 2024-04-26T01:34:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:34:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:34:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:34:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Deploying cephadm binary to smithi188 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Added host smithi188 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/218326018' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/218326018' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi006:172.21.15.6=a;smithi029:172.21.15.29=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Saving service mon spec with placement smithi006:172.21.15.6=a;smithi029:172.21.15.29=b;smithi188:172.21.15.188=c;count:3 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: Deploying daemon mon.c on smithi188 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: from='client.? 172.21.15.188:0/2986223335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:34:42.084 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 ceph-mon[32751]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T01:34:42.085 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 bash[32733]: 344aa2641bc9ba02b95f7932791ffd516dca7f3b66847cb033e45af339296781 2024-04-26T01:34:42.085 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:41 smithi188 systemd[1]: Started Ceph mon.c for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:34:42.262 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:34:46.941 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.945+0000 7f26ac4ed700 1 -- 172.21.15.188:0/1662644437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26900035c0 msgr2=0x7f2690005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:46.941 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.945+0000 7f26ac4ed700 1 --2- 172.21.15.188:0/1662644437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26900035c0 0x7f2690005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f26a4101360 tx=0x7f269c013e90 comp rx=0 tx=0).stop 2024-04-26T01:34:46.941 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.945+0000 7f26ac4ed700 1 -- 172.21.15.188:0/1662644437 shutdown_connections 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.945+0000 7f26ac4ed700 1 --2- 172.21.15.188:0/1662644437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26900035c0 0x7f2690005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.945+0000 7f26ac4ed700 1 -- 172.21.15.188:0/1662644437 >> 172.21.15.188:0/1662644437 conn(0x7f26a40fcf20 msgr2=0x7f26a40ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.945+0000 7f26ac4ed700 1 -- 172.21.15.188:0/1662644437 shutdown_connections 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.945+0000 7f26ac4ed700 1 -- 172.21.15.188:0/1662644437 wait complete. 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.945+0000 7f26ac4ed700 1 Processor -- start 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26ac4ed700 1 -- start start 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26ac4ed700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26900035c0 0x7f26a4103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26ac4ed700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26a41035d0 0x7f26a4103a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26ac4ed700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f26a4109380 con 0x7f26a41035d0 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26ac4ed700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f26a4109200 con 0x7f26900035c0 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26aa289700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26900035c0 0x7f26a4103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26aa289700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26900035c0 0x7f26a4103090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:53398/0 (socket says 172.21.15.188:53398) 2024-04-26T01:34:46.942 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26aa289700 1 -- 172.21.15.188:0/2011526712 learned_addr learned my addr 172.21.15.188:0/2011526712 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-26T01:34:46.943 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26aa289700 1 -- 172.21.15.188:0/2011526712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26900035c0 msgr2=0x7f26a4103090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T01:34:46.943 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26aa289700 1 -- 172.21.15.188:0/2011526712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26900035c0 msgr2=0x7f26a4103090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T01:34:46.943 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26aa289700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26900035c0 0x7f26a4103090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T01:34:46.943 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26aa289700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26900035c0 0x7f26a4103090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:34:46.943 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.946+0000 7f26a9a88700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26a41035d0 0x7f26a4103a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:46.944 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.947+0000 7f26a9a88700 1 -- 172.21.15.188:0/2011526712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26900035c0 msgr2=0x7f26a4103090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:34:46.944 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.947+0000 7f26a9a88700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26900035c0 0x7f26a4103090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:46.944 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.947+0000 7f26a9a88700 1 -- 172.21.15.188:0/2011526712 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a4104fb0 con 0x7f26a41035d0 2024-04-26T01:34:46.944 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.948+0000 7f26a9a88700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26a41035d0 0x7f26a4103a50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f269400b4d0 tx=0x7f269400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:46.944 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.948+0000 7f269b7fe700 1 -- 172.21.15.188:0/2011526712 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2694011630 con 0x7f26a41035d0 2024-04-26T01:34:46.944 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.948+0000 7f269b7fe700 1 -- 172.21.15.188:0/2011526712 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2694011c80 con 0x7f26a41035d0 2024-04-26T01:34:46.944 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.948+0000 7f26ac4ed700 1 -- 172.21.15.188:0/2011526712 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a41a42c0 con 0x7f26a41035d0 2024-04-26T01:34:46.944 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.948+0000 7f269b7fe700 1 -- 172.21.15.188:0/2011526712 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2694010da0 con 0x7f26a41035d0 2024-04-26T01:34:46.944 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.948+0000 7f26ac4ed700 1 -- 172.21.15.188:0/2011526712 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a41a4800 con 0x7f26a41035d0 2024-04-26T01:34:46.944 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.949+0000 7f26ac4ed700 1 -- 172.21.15.188:0/2011526712 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a4102b70 con 0x7f26a41035d0 2024-04-26T01:34:46.945 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.950+0000 7f269b7fe700 1 -- 172.21.15.188:0/2011526712 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f2694018020 con 0x7f26a41035d0 2024-04-26T01:34:46.945 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.950+0000 7f269b7fe700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f26880393f0 0x7f268803b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:46.945 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.950+0000 7f269b7fe700 1 -- 172.21.15.188:0/2011526712 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f269404c390 con 0x7f26a41035d0 2024-04-26T01:34:46.945 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.951+0000 7f26aa289700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f26880393f0 0x7f268803b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:46.946 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.952+0000 7f26aa289700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f26880393f0 0x7f268803b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f269c013a40 tx=0x7f269c00f3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:46.960 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:46.966+0000 7f269b7fe700 1 -- 172.21.15.188:0/2011526712 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f269401f020 con 0x7f26a41035d0 2024-04-26T01:34:47.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:47.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:34:47.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: mon.a calling monitor election 2024-04-26T01:34:47.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:47.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:47.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:47.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:47.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: mon.c calling monitor election 2024-04-26T01:34:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:34:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: monmap epoch 2 2024-04-26T01:34:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:47.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: last_changed 2024-04-26T01:34:41.806428+0000 2024-04-26T01:34:47.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: created 2024-04-26T01:32:24.866051+0000 2024-04-26T01:34:47.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: min_mon_release 19 (squid) 2024-04-26T01:34:47.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: election_strategy: 1 2024-04-26T01:34:47.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-26T01:34:47.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-26T01:34:47.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: fsmap 2024-04-26T01:34:47.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:34:47.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: mgrmap e13: x(active, since 59s) 2024-04-26T01:34:47.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: overall HEALTH_OK 2024-04-26T01:34:47.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:47.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:47.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:47.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:47.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: mon.a calling monitor election 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: mon.c calling monitor election 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:47.156 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: monmap epoch 2 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: last_changed 2024-04-26T01:34:41.806428+0000 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: created 2024-04-26T01:32:24.866051+0000 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: min_mon_release 19 (squid) 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: election_strategy: 1 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: fsmap 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: mgrmap e13: x(active, since 59s) 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: overall HEALTH_OK 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:47.157 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:47.158 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:47.158 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:47.244 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.249+0000 7f26ac4ed700 1 -- 172.21.15.188:0/2011526712 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f26a41a4b10 con 0x7f26a41035d0 2024-04-26T01:34:47.244 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.250+0000 7f269b7fe700 1 -- 172.21.15.188:0/2011526712 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7f26940172c0 con 0x7f26a41035d0 2024-04-26T01:34:47.245 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:34:47.245 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":2,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","modified":"2024-04-26T01:34:41.806428Z","created":"2024-04-26T01:32:24.866051Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T01:34:47.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.252+0000 7f26ac4ed700 1 -- 172.21.15.188:0/2011526712 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f26880393f0 msgr2=0x7f268803b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:47.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.252+0000 7f26ac4ed700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f26880393f0 0x7f268803b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f269c013a40 tx=0x7f269c00f3c0 comp rx=0 tx=0).stop 2024-04-26T01:34:47.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.252+0000 7f26ac4ed700 1 -- 172.21.15.188:0/2011526712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26a41035d0 msgr2=0x7f26a4103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:47.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.252+0000 7f26ac4ed700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26a41035d0 0x7f26a4103a50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f269400b4d0 tx=0x7f269400b890 comp rx=0 tx=0).stop 2024-04-26T01:34:47.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.252+0000 7f26ac4ed700 1 -- 172.21.15.188:0/2011526712 shutdown_connections 2024-04-26T01:34:47.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.252+0000 7f26ac4ed700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f26880393f0 0x7f268803b8b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:47.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.252+0000 7f26ac4ed700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26900035c0 0x7f26a4103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:47.248 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.252+0000 7f26ac4ed700 1 --2- 172.21.15.188:0/2011526712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26a41035d0 0x7f26a4103a50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:47.248 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.252+0000 7f26ac4ed700 1 -- 172.21.15.188:0/2011526712 >> 172.21.15.188:0/2011526712 conn(0x7f26a40fcf20 msgr2=0x7f26a41067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:47.248 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.252+0000 7f26ac4ed700 1 -- 172.21.15.188:0/2011526712 shutdown_connections 2024-04-26T01:34:47.248 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:47.252+0000 7f26ac4ed700 1 -- 172.21.15.188:0/2011526712 wait complete. 2024-04-26T01:34:47.248 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 2 2024-04-26T01:34:48.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:47 smithi188 ceph-mon[32751]: Deploying daemon mon.b on smithi029 2024-04-26T01:34:48.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:47 smithi188 ceph-mon[32751]: from='client.? 172.21.15.188:0/2011526712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:34:48.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:48.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:47 smithi006 ceph-mon[28419]: Deploying daemon mon.b on smithi029 2024-04-26T01:34:48.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:47 smithi006 ceph-mon[28419]: from='client.? 172.21.15.188:0/2011526712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:34:48.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:48.836 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T01:34:48.837 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T01:34:49.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:48 smithi006 ceph-mon[28419]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:49.081 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:34:49.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:48 smithi188 ceph-mon[32751]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:50.196 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:49 smithi188 ceph-mon[32751]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:50.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:49 smithi006 ceph-mon[28419]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:50.604 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.608+0000 7f0316dae700 1 -- 172.21.15.188:0/4172311949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0310108630 msgr2=0x7f0310108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:50.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.608+0000 7f0316dae700 1 --2- 172.21.15.188:0/4172311949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0310108630 0x7f0310108a10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f030800b0a0 tx=0x7f030800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:34:50.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.608+0000 7f0316dae700 1 -- 172.21.15.188:0/4172311949 shutdown_connections 2024-04-26T01:34:50.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.608+0000 7f0316dae700 1 --2- 172.21.15.188:0/4172311949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0310108630 0x7f0310108a10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:50.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.608+0000 7f0316dae700 1 -- 172.21.15.188:0/4172311949 >> 172.21.15.188:0/4172311949 conn(0x7f03100fe050 msgr2=0x7f0310100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:50.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.608+0000 7f0316dae700 1 -- 172.21.15.188:0/4172311949 shutdown_connections 2024-04-26T01:34:50.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.608+0000 7f0316dae700 1 -- 172.21.15.188:0/4172311949 wait complete. 2024-04-26T01:34:50.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0316dae700 1 Processor -- start 2024-04-26T01:34:50.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0316dae700 1 -- start start 2024-04-26T01:34:50.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0316dae700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0310108630 0x7f0310074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:50.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0316dae700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0310075280 0x7f0310075700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:50.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0316dae700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f031010caa0 con 0x7f0310075280 2024-04-26T01:34:50.606 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0316dae700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f031010c920 con 0x7f0310108630 2024-04-26T01:34:50.606 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0314b4a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0310108630 0x7f0310074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:50.606 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0314b4a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0310108630 0x7f0310074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:53412/0 (socket says 172.21.15.188:53412) 2024-04-26T01:34:50.606 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0314b4a700 1 -- 172.21.15.188:0/1025400831 learned_addr learned my addr 172.21.15.188:0/1025400831 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-26T01:34:50.606 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0314b4a700 1 -- 172.21.15.188:0/1025400831 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0310108630 msgr2=0x7f0310074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T01:34:50.606 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0314b4a700 1 -- 172.21.15.188:0/1025400831 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0310108630 msgr2=0x7f0310074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T01:34:50.606 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0314b4a700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0310108630 0x7f0310074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T01:34:50.606 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.609+0000 7f0314b4a700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0310108630 0x7f0310074d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:34:50.606 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.610+0000 7f030ffff700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0310075280 0x7f0310075700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:50.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.610+0000 7f030ffff700 1 -- 172.21.15.188:0/1025400831 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0310108630 msgr2=0x7f0310074d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:34:50.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.610+0000 7f030ffff700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0310108630 0x7f0310074d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:50.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.610+0000 7f030ffff700 1 -- 172.21.15.188:0/1025400831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0310076c60 con 0x7f0310075280 2024-04-26T01:34:50.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.611+0000 7f030ffff700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0310075280 0x7f0310075700 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f030400b4d0 tx=0x7f030400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:50.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.611+0000 7f030dffb700 1 -- 172.21.15.188:0/1025400831 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0304011630 con 0x7f0310075280 2024-04-26T01:34:50.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.611+0000 7f0316dae700 1 -- 172.21.15.188:0/1025400831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03101a4350 con 0x7f0310075280 2024-04-26T01:34:50.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.611+0000 7f030dffb700 1 -- 172.21.15.188:0/1025400831 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0304011c80 con 0x7f0310075280 2024-04-26T01:34:50.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.611+0000 7f030dffb700 1 -- 172.21.15.188:0/1025400831 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f030401adc0 con 0x7f0310075280 2024-04-26T01:34:50.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.611+0000 7f0316dae700 1 -- 172.21.15.188:0/1025400831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03101a4890 con 0x7f0310075280 2024-04-26T01:34:50.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.612+0000 7f0316dae700 1 -- 172.21.15.188:0/1025400831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0310103ca0 con 0x7f0310075280 2024-04-26T01:34:50.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.613+0000 7f030dffb700 1 -- 172.21.15.188:0/1025400831 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f0304018020 con 0x7f0310075280 2024-04-26T01:34:50.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.613+0000 7f030dffb700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f02f8039410 0x7f02f803b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:50.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.613+0000 7f030dffb700 1 -- 172.21.15.188:0/1025400831 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f030404c5d0 con 0x7f0310075280 2024-04-26T01:34:50.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.614+0000 7f0314b4a700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f02f8039410 0x7f02f803b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:50.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.615+0000 7f0314b4a700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f02f8039410 0x7f02f803b8d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0308009e60 tx=0x7f030800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:50.624 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.630+0000 7f030dffb700 1 -- 172.21.15.188:0/1025400831 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f030401f020 con 0x7f0310075280 2024-04-26T01:34:50.795 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 podman[32756]: 2024-04-26 01:34:50.608122381 +0000 UTC m=+0.018658650 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:34:50.902 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.907+0000 7f0316dae700 1 -- 172.21.15.188:0/1025400831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f03101a4bf0 con 0x7f0310075280 2024-04-26T01:34:50.903 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.909+0000 7f030dffb700 1 -- 172.21.15.188:0/1025400831 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7f030400e260 con 0x7f0310075280 2024-04-26T01:34:50.904 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:34:50.904 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":2,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","modified":"2024-04-26T01:34:41.806428Z","created":"2024-04-26T01:32:24.866051Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T01:34:50.906 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.911+0000 7f0316dae700 1 -- 172.21.15.188:0/1025400831 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f02f8039410 msgr2=0x7f02f803b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:50.906 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.911+0000 7f0316dae700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f02f8039410 0x7f02f803b8d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0308009e60 tx=0x7f030800b7b0 comp rx=0 tx=0).stop 2024-04-26T01:34:50.906 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.911+0000 7f0316dae700 1 -- 172.21.15.188:0/1025400831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0310075280 msgr2=0x7f0310075700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:50.906 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.911+0000 7f0316dae700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0310075280 0x7f0310075700 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f030400b4d0 tx=0x7f030400b890 comp rx=0 tx=0).stop 2024-04-26T01:34:50.906 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.911+0000 7f0316dae700 1 -- 172.21.15.188:0/1025400831 shutdown_connections 2024-04-26T01:34:50.906 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.911+0000 7f0316dae700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f02f8039410 0x7f02f803b8d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:50.906 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.911+0000 7f0316dae700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0310108630 0x7f0310074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:50.906 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.911+0000 7f0316dae700 1 --2- 172.21.15.188:0/1025400831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0310075280 0x7f0310075700 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:50.906 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.911+0000 7f0316dae700 1 -- 172.21.15.188:0/1025400831 >> 172.21.15.188:0/1025400831 conn(0x7f03100fe050 msgr2=0x7f03100ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:50.907 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.911+0000 7f0316dae700 1 -- 172.21.15.188:0/1025400831 shutdown_connections 2024-04-26T01:34:50.907 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:50.911+0000 7f0316dae700 1 -- 172.21.15.188:0/1025400831 wait complete. 2024-04-26T01:34:50.907 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 2 2024-04-26T01:34:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 podman[32756]: 2024-04-26 01:34:50.884355546 +0000 UTC m=+0.294891797 container init 3724a8af3fb8d03f6fe626b460877764586fdb88d3fffc1d458f1ae7fb27aaf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:34:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 podman[32756]: 2024-04-26 01:34:50.887993397 +0000 UTC m=+0.298529648 container start 3724a8af3fb8d03f6fe626b460877764586fdb88d3fffc1d458f1ae7fb27aaf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-26T01:34:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T01:34:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-26T01:34:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pidfile_write: ignore empty --pid-file 2024-04-26T01:34:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: load: jerasure load: lrc 2024-04-26T01:34:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: RocksDB version: 7.9.2 2024-04-26T01:34:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Git sha 0 2024-04-26T01:34:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-26T01:34:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: DB SUMMARY 2024-04-26T01:34:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: DB Session ID: XJBQHOEIRRSKT3261IDH 2024-04-26T01:34:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: CURRENT file: CURRENT 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.error_if_exists: 0 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.create_if_missing: 0 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.paranoid_checks: 1 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.env: 0x55b0eb253cc0 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.info_log: 0x55b0ec863be0 2024-04-26T01:34:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T01:34:51.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.statistics: (nil) 2024-04-26T01:34:51.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.use_fsync: 0 2024-04-26T01:34:51.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_log_file_size: 0 2024-04-26T01:34:51.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T01:34:51.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T01:34:51.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.allow_fallocate: 1 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.use_direct_reads: 0 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.db_log_dir: 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.wal_dir: 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T01:34:51.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.write_buffer_manager: 0x55b0ec7f70e0 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.unordered_write: 0 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T01:34:51.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.row_cache: None 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.wal_filter: None 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.two_write_queues: 0 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.wal_compression: 0 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.atomic_flush: 0 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.log_readahead_size: 0 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T01:34:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_background_jobs: 2 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_background_compactions: -1 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_subcompactions: 1 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T01:34:51.195 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_open_files: -1 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_background_flushes: -1 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Compression algorithms supported: 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: kZSTD supported: 0 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: kXpressCompression supported: 0 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: kZlibCompression supported: 1 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: kSnappyCompression supported: 1 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: kLZ4Compression supported: 1 2024-04-26T01:34:51.196 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: kBZip2Compression supported: 0 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.merge_operator: 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_filter: None 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b0ec863dc0) 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T01:34:51.197 INFO:journalctl@ceph.mon.b.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: index_type: 0 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: data_block_index_type: 0 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: index_shortening: 1 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: checksum: 4 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: no_block_cache: 0 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: block_cache: 0x55b0ec8291f0 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: block_cache_options: 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: capacity : 536870912 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: num_shard_bits : 4 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: strict_capacity_limit : 0 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-04-26T01:34:51.198 INFO:journalctl@ceph.mon.b.smithi029.stdout: block_cache_compressed: (nil) 2024-04-26T01:34:51.199 INFO:journalctl@ceph.mon.b.smithi029.stdout: persistent_cache: (nil) 2024-04-26T01:34:51.199 INFO:journalctl@ceph.mon.b.smithi029.stdout: block_size: 4096 2024-04-26T01:34:51.201 INFO:journalctl@ceph.mon.b.smithi029.stdout: block_size_deviation: 10 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: block_restart_interval: 16 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: index_block_restart_interval: 1 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: metadata_block_size: 4096 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: partition_filters: 0 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: use_delta_encoding: 1 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: filter_policy: bloomfilter 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: whole_key_filtering: 1 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: verify_compression: 0 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: format_version: 5 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: enable_index_compression: 1 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: block_align: 0 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: max_auto_readahead_size: 262144 2024-04-26T01:34:51.202 INFO:journalctl@ceph.mon.b.smithi029.stdout: prepopulate_block_cache: 0 2024-04-26T01:34:51.203 INFO:journalctl@ceph.mon.b.smithi029.stdout: initial_auto_readahead_size: 8192 2024-04-26T01:34:51.203 INFO:journalctl@ceph.mon.b.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T01:34:51.204 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:50 smithi188 ceph-mon[32751]: from='client.? 172.21.15.188:0/1025400831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:34:51.205 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T01:34:51.205 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T01:34:51.205 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compression: NoCompression 2024-04-26T01:34:51.205 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.num_levels: 7 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T01:34:51.206 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T01:34:51.207 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T01:34:51.208 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T01:34:51.209 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.inplace_update_support: 0 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.bloom_locality: 0 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.max_successive_merges: 0 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.ttl: 2592000 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T01:34:51.210 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.enable_blob_files: false 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.min_blob_size: 0 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-26T01:34:51.211 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 411d2ec8-4859-4842-b3c2-0dc9e2570e75 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714095290918185, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714095290918597, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714095290, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "411d2ec8-4859-4842-b3c2-0dc9e2570e75", "db_session_id": "XJBQHOEIRRSKT3261IDH", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714095290918679, "job": 1, "event": "recovery_finished"} 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b0ec93e000 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: DB pointer 0x55b0ec928000 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout: ** DB Stats ** 2024-04-26T01:34:51.212 INFO:journalctl@ceph.mon.b.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi029.stdout: 2024-04-26T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi029.stdout: ** Compaction Stats [default] ** 2024-04-26T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi029.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi029.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T01:34:51.214 INFO:journalctl@ceph.mon.b.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T01:34:51.214 INFO:journalctl@ceph.mon.b.smithi029.stdout: 2024-04-26T01:34:51.214 INFO:journalctl@ceph.mon.b.smithi029.stdout: ** Compaction Stats [default] ** 2024-04-26T01:34:51.214 INFO:journalctl@ceph.mon.b.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:34:51.214 INFO:journalctl@ceph.mon.b.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T01:34:51.214 INFO:journalctl@ceph.mon.b.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T01:34:51.215 INFO:journalctl@ceph.mon.b.smithi029.stdout: 2024-04-26T01:34:51.215 INFO:journalctl@ceph.mon.b.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T01:34:51.215 INFO:journalctl@ceph.mon.b.smithi029.stdout: 2024-04-26T01:34:51.215 INFO:journalctl@ceph.mon.b.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:34:51.215 INFO:journalctl@ceph.mon.b.smithi029.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T01:34:51.215 INFO:journalctl@ceph.mon.b.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T01:34:51.216 INFO:journalctl@ceph.mon.b.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T01:34:51.216 INFO:journalctl@ceph.mon.b.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T01:34:51.216 INFO:journalctl@ceph.mon.b.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T01:34:51.216 INFO:journalctl@ceph.mon.b.smithi029.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:34:51.216 INFO:journalctl@ceph.mon.b.smithi029.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:34:51.216 INFO:journalctl@ceph.mon.b.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T01:34:51.216 INFO:journalctl@ceph.mon.b.smithi029.stdout: Block cache BinnedLRUCache@0x55b0ec8291f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-26T01:34:51.216 INFO:journalctl@ceph.mon.b.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-26T01:34:51.217 INFO:journalctl@ceph.mon.b.smithi029.stdout: 2024-04-26T01:34:51.217 INFO:journalctl@ceph.mon.b.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T01:34:51.217 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-26T01:34:51.217 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: using public_addr v2:172.21.15.29:0/0 -> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2024-04-26T01:34:51.217 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: starting mon.b rank -1 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:51.217 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(???) e0 preinit fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:51.217 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).mds e1 new map 2024-04-26T01:34:51.217 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-26T01:34:51.217 INFO:journalctl@ceph.mon.b.smithi029.stdout: e1 2024-04-26T01:34:51.217 INFO:journalctl@ceph.mon.b.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T01:34:51.217 INFO:journalctl@ceph.mon.b.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:34:51.218 INFO:journalctl@ceph.mon.b.smithi029.stdout: legacy client fscid: -1 2024-04-26T01:34:51.218 INFO:journalctl@ceph.mon.b.smithi029.stdout: 2024-04-26T01:34:51.218 INFO:journalctl@ceph.mon.b.smithi029.stdout: No filesystems configured 2024-04-26T01:34:51.218 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T01:34:51.218 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T01:34:51.218 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T01:34:51.218 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T01:34:51.218 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T01:34:51.218 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T01:34:51.218 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T01:34:51.218 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Deploying cephadm binary to smithi029 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Added host smithi029 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.219 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Updating smithi029:/etc/ceph/ceph.conf 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.220 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:34:51.221 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:34:51.221 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:51.221 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.221 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.221 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.221 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Deploying cephadm binary to smithi188 2024-04-26T01:34:51.221 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.221 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.221 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.221 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:51.221 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Added host smithi188 2024-04-26T01:34:51.221 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.222 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.222 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.222 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.222 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.222 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:34:51.222 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.222 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.222 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.222 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/218326018' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/218326018' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:34:51.223 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi006:172.21.15.6=a;smithi029:172.21.15.29=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Saving service mon spec with placement smithi006:172.21.15.6=a;smithi029:172.21.15.29=b;smithi188:172.21.15.188=c;count:3 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Deploying daemon mon.c on smithi188 2024-04-26T01:34:51.224 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.225 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.225 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='client.? 172.21.15.188:0/2986223335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:34:51.225 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.225 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:34:51.225 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.a calling monitor election 2024-04-26T01:34:51.225 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:51.226 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:51.226 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.226 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:51.226 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.c calling monitor election 2024-04-26T01:34:51.226 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:51.227 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.227 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:51.227 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:51.227 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:34:51.227 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: monmap epoch 2 2024-04-26T01:34:51.227 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:51.227 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: last_changed 2024-04-26T01:34:41.806428+0000 2024-04-26T01:34:51.227 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: created 2024-04-26T01:32:24.866051+0000 2024-04-26T01:34:51.227 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: min_mon_release 19 (squid) 2024-04-26T01:34:51.227 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: election_strategy: 1 2024-04-26T01:34:51.227 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-26T01:34:51.227 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: fsmap 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mgrmap e13: x(active, since 59s) 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: overall HEALTH_OK 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: Deploying daemon mon.b on smithi029 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='client.? 172.21.15.188:0/2011526712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:51.228 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.229 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:51.229 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: from='client.? 172.21.15.188:0/1025400831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:34:51.229 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 ceph-mon[32775]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T01:34:51.229 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 bash[32756]: 3724a8af3fb8d03f6fe626b460877764586fdb88d3fffc1d458f1ae7fb27aaf4 2024-04-26T01:34:51.229 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:50 smithi029 systemd[1]: Started Ceph mon.b for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:34:51.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:50 smithi006 ceph-mon[28419]: from='client.? 172.21.15.188:0/1025400831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:34:52.509 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T01:34:52.509 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T01:34:52.764 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:34:56.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: mon.c calling monitor election 2024-04-26T01:34:56.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:34:56.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: mon.a calling monitor election 2024-04-26T01:34:56.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:56.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:56.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:56.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:56.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:56.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:56.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:56.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:56.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:56.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:56.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:34:56.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: monmap epoch 3 2024-04-26T01:34:56.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:56.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: last_changed 2024-04-26T01:34:50.944133+0000 2024-04-26T01:34:56.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: created 2024-04-26T01:32:24.866051+0000 2024-04-26T01:34:56.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: min_mon_release 19 (squid) 2024-04-26T01:34:56.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: election_strategy: 1 2024-04-26T01:34:56.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-26T01:34:56.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-26T01:34:56.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: 2: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.b 2024-04-26T01:34:56.319 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: fsmap 2024-04-26T01:34:56.319 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:34:56.319 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: mgrmap e13: x(active, since 68s) 2024-04-26T01:34:56.319 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: overall HEALTH_OK 2024-04-26T01:34:56.319 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:56.319 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:56.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: mon.c calling monitor election 2024-04-26T01:34:56.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: mon.a calling monitor election 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:34:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: monmap epoch 3 2024-04-26T01:34:56.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:56.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: last_changed 2024-04-26T01:34:50.944133+0000 2024-04-26T01:34:56.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: created 2024-04-26T01:32:24.866051+0000 2024-04-26T01:34:56.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: min_mon_release 19 (squid) 2024-04-26T01:34:56.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: election_strategy: 1 2024-04-26T01:34:56.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-26T01:34:56.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-26T01:34:56.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: 2: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.b 2024-04-26T01:34:56.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: fsmap 2024-04-26T01:34:56.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:34:56.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: mgrmap e13: x(active, since 68s) 2024-04-26T01:34:56.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: overall HEALTH_OK 2024-04-26T01:34:56.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:56.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:57.163 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.163+0000 7f249b5be700 1 -- 172.21.15.188:0/3043128228 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24941074c0 msgr2=0x7f2480005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:57.163 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.163+0000 7f249b5be700 1 --2- 172.21.15.188:0/3043128228 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24941074c0 0x7f2480005610 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2484005fa0 tx=0x7f248400ffc0 comp rx=0 tx=0).stop 2024-04-26T01:34:57.163 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.163+0000 7f249b5be700 1 -- 172.21.15.188:0/3043128228 shutdown_connections 2024-04-26T01:34:57.163 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.163+0000 7f249b5be700 1 --2- 172.21.15.188:0/3043128228 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24941074c0 0x7f2480005610 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:57.163 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.163+0000 7f249b5be700 1 -- 172.21.15.188:0/3043128228 >> 172.21.15.188:0/3043128228 conn(0x7f24940fcf20 msgr2=0x7f24940ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:57.164 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.163+0000 7f249b5be700 1 -- 172.21.15.188:0/3043128228 shutdown_connections 2024-04-26T01:34:57.164 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.163+0000 7f249b5be700 1 -- 172.21.15.188:0/3043128228 wait complete. 2024-04-26T01:34:57.164 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f249b5be700 1 Processor -- start 2024-04-26T01:34:57.164 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f249b5be700 1 -- start start 2024-04-26T01:34:57.164 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f249b5be700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24941074c0 0x7f2494198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:57.164 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f249b5be700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2494199450 0x7f24941998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:57.164 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f249b5be700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24941a1f80 0x7f24941a4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:57.164 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f249b5be700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2494109340 con 0x7f24941074c0 2024-04-26T01:34:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f249b5be700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f24941091c0 con 0x7f24941a1f80 2024-04-26T01:34:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f249b5be700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f24941094c0 con 0x7f2494199450 2024-04-26T01:34:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f2498b59700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2494199450 0x7f24941998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f2498b59700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2494199450 0x7f24941998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:38058/0 (socket says 172.21.15.188:38058) 2024-04-26T01:34:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f2498b59700 1 -- 172.21.15.188:0/4100627078 learned_addr learned my addr 172.21.15.188:0/4100627078 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-26T01:34:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f2498b59700 1 -- 172.21.15.188:0/4100627078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24941a1f80 msgr2=0x7f24941a4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f2499b5b700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24941a1f80 0x7f24941a4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f2498b59700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24941a1f80 0x7f24941a4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f2498b59700 1 -- 172.21.15.188:0/4100627078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24941074c0 msgr2=0x7f2494198f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f2498b59700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24941074c0 0x7f2494198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f2498b59700 1 -- 172.21.15.188:0/4100627078 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24941a4be0 con 0x7f2494199450 2024-04-26T01:34:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.164+0000 7f2499b5b700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24941a1f80 0x7f24941a4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:34:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.165+0000 7f2498b59700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2494199450 0x7f24941998d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f249000b780 tx=0x7f249000bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.165+0000 7f248a7fc700 1 -- 172.21.15.188:0/4100627078 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490004230 con 0x7f2494199450 2024-04-26T01:34:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.165+0000 7f248a7fc700 1 -- 172.21.15.188:0/4100627078 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24900043d0 con 0x7f2494199450 2024-04-26T01:34:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.165+0000 7f248a7fc700 1 -- 172.21.15.188:0/4100627078 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490011bd0 con 0x7f2494199450 2024-04-26T01:34:57.167 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.165+0000 7f249b5be700 1 -- 172.21.15.188:0/4100627078 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24941a4ed0 con 0x7f2494199450 2024-04-26T01:34:57.167 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.165+0000 7f249b5be700 1 -- 172.21.15.188:0/4100627078 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24941a53b0 con 0x7f2494199450 2024-04-26T01:34:57.167 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.166+0000 7f248a7fc700 1 -- 172.21.15.188:0/4100627078 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f249001a400 con 0x7f2494199450 2024-04-26T01:34:57.167 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.166+0000 7f249b5be700 1 -- 172.21.15.188:0/4100627078 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24940640a0 con 0x7f2494199450 2024-04-26T01:34:57.167 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.166+0000 7f248a7fc700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f2478039ae0 0x7f247803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:57.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.166+0000 7f248a7fc700 1 -- 172.21.15.188:0/4100627078 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f249004c3c0 con 0x7f2494199450 2024-04-26T01:34:57.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.167+0000 7f249935a700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f2478039ae0 0x7f247803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:57.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.168+0000 7f249935a700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f2478039ae0 0x7f247803bfa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2484004fd0 tx=0x7f2484005e00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:57.169 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.171+0000 7f248a7fc700 1 -- 172.21.15.188:0/4100627078 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2490014160 con 0x7f2494199450 2024-04-26T01:34:57.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:57.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:57.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:57.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:57.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:57.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:57.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:57.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:57.452 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.457+0000 7f249b5be700 1 -- 172.21.15.188:0/4100627078 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f249402df80 con 0x7f2494199450 2024-04-26T01:34:57.452 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.458+0000 7f248a7fc700 1 -- 172.21.15.188:0/4100627078 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1275 (secure 0 0 0) 0x7f249003ae20 con 0x7f2494199450 2024-04-26T01:34:57.453 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-26T01:34:57.453 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":3,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","modified":"2024-04-26T01:34:50.944133Z","created":"2024-04-26T01:32:24.866051Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T01:34:57.455 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.460+0000 7f249b5be700 1 -- 172.21.15.188:0/4100627078 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f2478039ae0 msgr2=0x7f247803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:57.455 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.460+0000 7f249b5be700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f2478039ae0 0x7f247803bfa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2484004fd0 tx=0x7f2484005e00 comp rx=0 tx=0).stop 2024-04-26T01:34:57.455 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.460+0000 7f249b5be700 1 -- 172.21.15.188:0/4100627078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2494199450 msgr2=0x7f24941998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:57.455 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.460+0000 7f249b5be700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2494199450 0x7f24941998d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f249000b780 tx=0x7f249000bb40 comp rx=0 tx=0).stop 2024-04-26T01:34:57.455 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.460+0000 7f249b5be700 1 -- 172.21.15.188:0/4100627078 shutdown_connections 2024-04-26T01:34:57.456 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.460+0000 7f249b5be700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24941a1f80 0x7f24941a4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:57.456 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.460+0000 7f249b5be700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24941074c0 0x7f2494198f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:57.456 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.460+0000 7f249b5be700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f2478039ae0 0x7f247803bfa0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:57.456 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.460+0000 7f249b5be700 1 --2- 172.21.15.188:0/4100627078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2494199450 0x7f24941998d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:57.456 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.460+0000 7f249b5be700 1 -- 172.21.15.188:0/4100627078 >> 172.21.15.188:0/4100627078 conn(0x7f24940fcf20 msgr2=0x7f2494105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:57.456 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.460+0000 7f249b5be700 1 -- 172.21.15.188:0/4100627078 shutdown_connections 2024-04-26T01:34:57.456 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:34:57.460+0000 7f249b5be700 1 -- 172.21.15.188:0/4100627078 wait complete. 2024-04-26T01:34:57.456 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 3 2024-04-26T01:34:58.258 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T01:34:58.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T01:34:58.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: mon.c calling monitor election 2024-04-26T01:34:58.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:34:58.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: mon.a calling monitor election 2024-04-26T01:34:58.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:58.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:34:58.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:58.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:58.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:58.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:58.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:58.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:58.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:58.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:58.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:34:58.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: monmap epoch 3 2024-04-26T01:34:58.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:58.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: last_changed 2024-04-26T01:34:50.944133+0000 2024-04-26T01:34:58.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: created 2024-04-26T01:32:24.866051+0000 2024-04-26T01:34:58.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: min_mon_release 19 (squid) 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: election_strategy: 1 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: 2: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.b 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: fsmap 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: mgrmap e13: x(active, since 68s) 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: overall HEALTH_OK 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:58.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:58.684 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:34:58.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: mon.b calling monitor election 2024-04-26T01:34:58.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: mon.b calling monitor election 2024-04-26T01:34:58.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: mon.c calling monitor election 2024-04-26T01:34:58.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: mon.a calling monitor election 2024-04-26T01:34:58.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T01:34:58.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: monmap epoch 3 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: last_changed 2024-04-26T01:34:50.944133+0000 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: created 2024-04-26T01:32:24.866051+0000 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: min_mon_release 19 (squid) 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: election_strategy: 1 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: 2: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.b 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: fsmap 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: mgrmap e13: x(active, since 70s) 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: overall HEALTH_OK 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:58 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: mon.b calling monitor election 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: mon.b calling monitor election 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: mon.c calling monitor election 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: mon.a calling monitor election 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: monmap epoch 3 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: last_changed 2024-04-26T01:34:50.944133+0000 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: created 2024-04-26T01:32:24.866051+0000 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: min_mon_release 19 (squid) 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: election_strategy: 1 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-26T01:34:58.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: 2: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.b 2024-04-26T01:34:58.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: fsmap 2024-04-26T01:34:58.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:34:58.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: mgrmap e13: x(active, since 70s) 2024-04-26T01:34:58.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: overall HEALTH_OK 2024-04-26T01:34:58.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:58 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: mon.b calling monitor election 2024-04-26T01:34:58.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: mon.b calling monitor election 2024-04-26T01:34:58.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: mon.c calling monitor election 2024-04-26T01:34:58.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: mon.a calling monitor election 2024-04-26T01:34:58.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T01:34:58.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: monmap epoch 3 2024-04-26T01:34:58.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:34:58.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: last_changed 2024-04-26T01:34:50.944133+0000 2024-04-26T01:34:58.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: created 2024-04-26T01:32:24.866051+0000 2024-04-26T01:34:58.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: min_mon_release 19 (squid) 2024-04-26T01:34:58.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: election_strategy: 1 2024-04-26T01:34:58.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-26T01:34:58.978 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-26T01:34:58.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: 2: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.b 2024-04-26T01:34:58.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: fsmap 2024-04-26T01:34:58.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:34:58.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: mgrmap e13: x(active, since 70s) 2024-04-26T01:34:58.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: overall HEALTH_OK 2024-04-26T01:34:58.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:58.979 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:58 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:59.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:34:59.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:34:59.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: Updating smithi006:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:34:59.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:34:59.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:59.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:59.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:59.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:59.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:59.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:59.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:34:59.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:59.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:34:59.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:34:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:00.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:35:00.026 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:35:00.026 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: Updating smithi006:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:35:00.026 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:35:00.026 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.026 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.026 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.026 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.026 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.026 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.026 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.026 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:35:00.027 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:35:00.027 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:34:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:00.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:35:00.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:35:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: Updating smithi006:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:35:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:35:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:35:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:35:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:34:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:00.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:00 smithi029 ceph-mon[32775]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:00.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:00 smithi029 ceph-mon[32775]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T01:35:00.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:00 smithi029 ceph-mon[32775]: Reconfiguring daemon mon.a on smithi006 2024-04-26T01:35:00.972 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:00 smithi006 ceph-mon[28419]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:00.972 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:00 smithi006 ceph-mon[28419]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T01:35:00.972 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:00 smithi006 ceph-mon[28419]: Reconfiguring daemon mon.a on smithi006 2024-04-26T01:35:01.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:00 smithi188 ceph-mon[32751]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:01.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:00 smithi188 ceph-mon[32751]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T01:35:01.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:00 smithi188 ceph-mon[32751]: Reconfiguring daemon mon.a on smithi006 2024-04-26T01:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.685+0000 7f38fae00700 1 -- 172.21.15.6:0/4254822181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f41071e0 msgr2=0x7f38f4069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.685+0000 7f38fae00700 1 --2- 172.21.15.6:0/4254822181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f41071e0 0x7f38f4069e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009910 tx=0x7f38e8009c20 comp rx=0 tx=0).stop 2024-04-26T01:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.685+0000 7f38fae00700 1 -- 172.21.15.6:0/4254822181 shutdown_connections 2024-04-26T01:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.685+0000 7f38fae00700 1 --2- 172.21.15.6:0/4254822181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f41071e0 0x7f38f4069e60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.685+0000 7f38fae00700 1 -- 172.21.15.6:0/4254822181 >> 172.21.15.6:0/4254822181 conn(0x7f38f4076cf0 msgr2=0x7f38f4077100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.686+0000 7f38fae00700 1 -- 172.21.15.6:0/4254822181 shutdown_connections 2024-04-26T01:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.686+0000 7f38fae00700 1 -- 172.21.15.6:0/4254822181 wait complete. 2024-04-26T01:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.687+0000 7f38fae00700 1 Processor -- start 2024-04-26T01:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.687+0000 7f38fae00700 1 -- start start 2024-04-26T01:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.687+0000 7f38fae00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38f41071e0 0x7f38f419e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.687+0000 7f38fae00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f419eb10 0x7f38f41a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.687+0000 7f38fae00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38f41a33e0 0x7f38f41a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.687+0000 7f38fae00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f38f4112de0 con 0x7f38f419eb10 2024-04-26T01:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.688+0000 7f38fae00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f38f4112c60 con 0x7f38f41071e0 2024-04-26T01:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.688+0000 7f38fae00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f38f4112f60 con 0x7f38f41a33e0 2024-04-26T01:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.688+0000 7f38f9a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38f41071e0 0x7f38f419e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.688+0000 7f38f9a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38f41071e0 0x7f38f419e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55628/0 (socket says 172.21.15.6:55628) 2024-04-26T01:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.688+0000 7f38f9a00700 1 -- 172.21.15.6:0/2007162121 learned_addr learned my addr 172.21.15.6:0/2007162121 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.688+0000 7f38f9000700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f419eb10 0x7f38f41a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.688+0000 7f38fa400700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38f41a33e0 0x7f38f41a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.688+0000 7f38f9a00700 1 -- 172.21.15.6:0/2007162121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38f41a33e0 msgr2=0x7f38f41a3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.688+0000 7f38f9a00700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38f41a33e0 0x7f38f41a3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.688+0000 7f38f9a00700 1 -- 172.21.15.6:0/2007162121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f419eb10 msgr2=0x7f38f41a2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.688+0000 7f38f9a00700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f419eb10 0x7f38f41a2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.688+0000 7f38f9a00700 1 -- 172.21.15.6:0/2007162121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f41ab150 con 0x7f38f41071e0 2024-04-26T01:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.689+0000 7f38f9a00700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38f41071e0 0x7f38f419e5d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f38e8004f80 tx=0x7f38e8004fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.689+0000 7f38f2000700 1 -- 172.21.15.6:0/2007162121 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e8026070 con 0x7f38f41071e0 2024-04-26T01:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.689+0000 7f38f2000700 1 -- 172.21.15.6:0/2007162121 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e800b750 con 0x7f38f41071e0 2024-04-26T01:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.689+0000 7f38fae00700 1 -- 172.21.15.6:0/2007162121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38f41ab320 con 0x7f38f41071e0 2024-04-26T01:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.689+0000 7f38f2000700 1 -- 172.21.15.6:0/2007162121 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e80038e0 con 0x7f38f41071e0 2024-04-26T01:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.689+0000 7f38fae00700 1 -- 172.21.15.6:0/2007162121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38f41ab800 con 0x7f38f41071e0 2024-04-26T01:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.690+0000 7f38fae00700 1 -- 172.21.15.6:0/2007162121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38f41017d0 con 0x7f38f41071e0 2024-04-26T01:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.690+0000 7f38f2000700 1 -- 172.21.15.6:0/2007162121 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f38e8029070 con 0x7f38f41071e0 2024-04-26T01:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.691+0000 7f38f2000700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f38dc039ae0 0x7f38dc03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.691+0000 7f38f2000700 1 -- 172.21.15.6:0/2007162121 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f38e804d450 con 0x7f38f41071e0 2024-04-26T01:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.691+0000 7f38f9000700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f38dc039ae0 0x7f38dc03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.692+0000 7f38f9000700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f38dc039ae0 0x7f38dc03bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f38e0009ad0 tx=0x7f38e0009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.696+0000 7f38f2000700 1 -- 172.21.15.6:0/2007162121 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f38e8013070 con 0x7f38f41071e0 2024-04-26T01:35:01.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.947+0000 7f38fae00700 1 -- 172.21.15.6:0/2007162121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f38f41abac0 con 0x7f38f41071e0 2024-04-26T01:35:01.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.948+0000 7f38f2000700 1 -- 172.21.15.6:0/2007162121 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+271 (secure 0 0 0) 0x7f38e8030090 con 0x7f38f41071e0 2024-04-26T01:35:01.944 INFO:teuthology.orchestra.run.smithi006.stdout:# minimal ceph.conf for a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:35:01.944 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-04-26T01:35:01.944 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:35:01.944 INFO:teuthology.orchestra.run.smithi006.stdout: mon_host = [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2024-04-26T01:35:01.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.951+0000 7f38fae00700 1 -- 172.21.15.6:0/2007162121 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f38dc039ae0 msgr2=0x7f38dc03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:01.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.951+0000 7f38fae00700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f38dc039ae0 0x7f38dc03bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f38e0009ad0 tx=0x7f38e0009210 comp rx=0 tx=0).stop 2024-04-26T01:35:01.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.951+0000 7f38fae00700 1 -- 172.21.15.6:0/2007162121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38f41071e0 msgr2=0x7f38f419e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:01.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.951+0000 7f38fae00700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38f41071e0 0x7f38f419e5d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f38e8004f80 tx=0x7f38e8004fb0 comp rx=0 tx=0).stop 2024-04-26T01:35:01.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.951+0000 7f38fae00700 1 -- 172.21.15.6:0/2007162121 shutdown_connections 2024-04-26T01:35:01.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.951+0000 7f38fae00700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f38dc039ae0 0x7f38dc03bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:01.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.951+0000 7f38fae00700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38f41a33e0 0x7f38f41a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:01.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.951+0000 7f38fae00700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38f41071e0 0x7f38f419e5d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:01.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.951+0000 7f38fae00700 1 --2- 172.21.15.6:0/2007162121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f419eb10 0x7f38f41a2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:01.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.951+0000 7f38fae00700 1 -- 172.21.15.6:0/2007162121 >> 172.21.15.6:0/2007162121 conn(0x7f38f4076cf0 msgr2=0x7f38f4101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:01.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.951+0000 7f38fae00700 1 -- 172.21.15.6:0/2007162121 shutdown_connections 2024-04-26T01:35:01.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:01.951+0000 7f38fae00700 1 -- 172.21.15.6:0/2007162121 wait complete. 2024-04-26T01:35:02.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:02 smithi006 ceph-mon[28419]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:02.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:02.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:02.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:02 smithi006 ceph-mon[28419]: Reconfiguring mon.b (monmap changed)... 2024-04-26T01:35:02.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:35:02.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:35:02.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:02.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:02 smithi006 ceph-mon[28419]: Reconfiguring daemon mon.b on smithi029 2024-04-26T01:35:02.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:02 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2007162121' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:02.666 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T01:35:02.666 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:35:02.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T01:35:02.741 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:35:02.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:35:02.806 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:02 smithi029 ceph-mon[32775]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:02.807 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:02.807 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:02.807 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:02 smithi029 ceph-mon[32775]: Reconfiguring mon.b (monmap changed)... 2024-04-26T01:35:02.807 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:35:02.807 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:35:02.807 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:02.807 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:02 smithi029 ceph-mon[32775]: Reconfiguring daemon mon.b on smithi029 2024-04-26T01:35:02.807 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:02 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2007162121' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:02.854 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-26T01:35:02.854 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T01:35:02.884 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-26T01:35:02.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:35:02.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:02 smithi188 ceph-mon[32751]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:02.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:02.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:02.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:02 smithi188 ceph-mon[32751]: Reconfiguring mon.b (monmap changed)... 2024-04-26T01:35:02.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:35:02.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:35:02.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:02.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:02 smithi188 ceph-mon[32751]: Reconfiguring daemon mon.b on smithi029 2024-04-26T01:35:02.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:02 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2007162121' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:02.952 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-26T01:35:02.952 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T01:35:02.984 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-26T01:35:02.984 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:35:03.052 INFO:tasks.cephadm:Adding mgr.x on smithi006 2024-04-26T01:35:03.052 INFO:tasks.cephadm:Adding mgr.y on smithi029 2024-04-26T01:35:03.052 INFO:tasks.cephadm:Adding mgr.z on smithi188 2024-04-26T01:35:03.052 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi006=x;smithi029=y;smithi188=z' 2024-04-26T01:35:03.326 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:35:04.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:04 smithi188 ceph-mon[32751]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:04.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:04.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:04.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:04 smithi188 ceph-mon[32751]: Reconfiguring mon.c (monmap changed)... 2024-04-26T01:35:04.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:35:04.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:35:04.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:04.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:04 smithi188 ceph-mon[32751]: Reconfiguring daemon mon.c on smithi188 2024-04-26T01:35:04.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:04 smithi006 ceph-mon[28419]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:04.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:04.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:04.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:04 smithi006 ceph-mon[28419]: Reconfiguring mon.c (monmap changed)... 2024-04-26T01:35:04.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:35:04.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:35:04.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:04.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:04 smithi006 ceph-mon[28419]: Reconfiguring daemon mon.c on smithi188 2024-04-26T01:35:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:04 smithi029 ceph-mon[32775]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:04 smithi029 ceph-mon[32775]: Reconfiguring mon.c (monmap changed)... 2024-04-26T01:35:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:35:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:35:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:04 smithi029 ceph-mon[32775]: Reconfiguring daemon mon.c on smithi188 2024-04-26T01:35:04.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.857+0000 7f0e0e23b700 1 -- 172.21.15.188:0/122545993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e081025d0 msgr2=0x7f0e081029f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:04.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.857+0000 7f0e0e23b700 1 --2- 172.21.15.188:0/122545993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e081025d0 0x7f0e081029f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009910 tx=0x7f0df8009c20 comp rx=0 tx=0).stop 2024-04-26T01:35:04.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.857+0000 7f0e0e23b700 1 -- 172.21.15.188:0/122545993 shutdown_connections 2024-04-26T01:35:04.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.857+0000 7f0e0e23b700 1 --2- 172.21.15.188:0/122545993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e081025d0 0x7f0e081029f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:04.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.857+0000 7f0e0e23b700 1 -- 172.21.15.188:0/122545993 >> 172.21.15.188:0/122545993 conn(0x7f0e080fdb70 msgr2=0x7f0e080fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:04.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.857+0000 7f0e0e23b700 1 -- 172.21.15.188:0/122545993 shutdown_connections 2024-04-26T01:35:04.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.857+0000 7f0e0e23b700 1 -- 172.21.15.188:0/122545993 wait complete. 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e0e23b700 1 Processor -- start 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e0e23b700 1 -- start start 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e0e23b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e081025d0 0x7f0e08197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e0e23b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08198270 0x7f0e0819d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e0e23b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e081986f0 0x7f0e0819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e0e23b700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0e08108510 con 0x7f0e081025d0 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e0e23b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0e08108390 con 0x7f0e081986f0 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e0e23b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0e08108690 con 0x7f0e08198270 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e06ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08198270 0x7f0e0819d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e06ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08198270 0x7f0e0819d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:45200/0 (socket says 172.21.15.188:45200) 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e06ffd700 1 -- 172.21.15.188:0/4187459125 learned_addr learned my addr 172.21.15.188:0/4187459125 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e06ffd700 1 -- 172.21.15.188:0/4187459125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e081986f0 msgr2=0x7f0e0819f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e06ffd700 1 --2- 172.21.15.188:0/4187459125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e081986f0 0x7f0e0819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e06ffd700 1 -- 172.21.15.188:0/4187459125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e081025d0 msgr2=0x7f0e08197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e06ffd700 1 --2- 172.21.15.188:0/4187459125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e081025d0 0x7f0e08197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:04.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.858+0000 7f0e06ffd700 1 -- 172.21.15.188:0/4187459125 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e081a0150 con 0x7f0e08198270 2024-04-26T01:35:04.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.859+0000 7f0e06ffd700 1 --2- 172.21.15.188:0/4187459125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08198270 0x7f0e0819d2a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00b780 tx=0x7f0dfc00bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:04.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.859+0000 7f0e04ff9700 1 -- 172.21.15.188:0/4187459125 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc004230 con 0x7f0e08198270 2024-04-26T01:35:04.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.859+0000 7f0e04ff9700 1 -- 172.21.15.188:0/4187459125 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dfc0043d0 con 0x7f0e08198270 2024-04-26T01:35:04.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.859+0000 7f0e04ff9700 1 -- 172.21.15.188:0/4187459125 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc011bd0 con 0x7f0e08198270 2024-04-26T01:35:04.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.859+0000 7f0e0e23b700 1 -- 172.21.15.188:0/4187459125 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e081a0440 con 0x7f0e08198270 2024-04-26T01:35:04.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.859+0000 7f0e0e23b700 1 -- 172.21.15.188:0/4187459125 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e081a0a50 con 0x7f0e08198270 2024-04-26T01:35:04.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.860+0000 7f0e04ff9700 1 -- 172.21.15.188:0/4187459125 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f0dfc01a400 con 0x7f0e08198270 2024-04-26T01:35:04.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.860+0000 7f0e0e23b700 1 -- 172.21.15.188:0/4187459125 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e080640a0 con 0x7f0e08198270 2024-04-26T01:35:04.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.860+0000 7f0e04ff9700 1 --2- 172.21.15.188:0/4187459125 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f0df0039ae0 0x7f0df003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:04.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.860+0000 7f0e04ff9700 1 -- 172.21.15.188:0/4187459125 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0dfc04c410 con 0x7f0e08198270 2024-04-26T01:35:04.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.861+0000 7f0e077fe700 1 --2- 172.21.15.188:0/4187459125 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f0df0039ae0 0x7f0df003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:04.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.861+0000 7f0e077fe700 1 --2- 172.21.15.188:0/4187459125 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f0df0039ae0 0x7f0df003bfa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0df8005e00 tx=0x7f0df8005260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:04.860 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:04.865+0000 7f0e04ff9700 1 -- 172.21.15.188:0/4187459125 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0dfc014160 con 0x7f0e08198270 2024-04-26T01:35:05.078 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.083+0000 7f0e0e23b700 1 -- 172.21.15.188:0/4187459125 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi006=x;smithi029=y;smithi188=z", "target": ["mon-mgr", ""]}) -- 0x7f0e08062e80 con 0x7f0df0039ae0 2024-04-26T01:35:05.092 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.097+0000 7f0e04ff9700 1 -- 172.21.15.188:0/4187459125 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0e08062e80 con 0x7f0df0039ae0 2024-04-26T01:35:05.093 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mgr update... 2024-04-26T01:35:05.098 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.099+0000 7f0e0e23b700 1 -- 172.21.15.188:0/4187459125 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f0df0039ae0 msgr2=0x7f0df003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:05.098 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.099+0000 7f0e0e23b700 1 --2- 172.21.15.188:0/4187459125 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f0df0039ae0 0x7f0df003bfa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0df8005e00 tx=0x7f0df8005260 comp rx=0 tx=0).stop 2024-04-26T01:35:05.098 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.099+0000 7f0e0e23b700 1 -- 172.21.15.188:0/4187459125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08198270 msgr2=0x7f0e0819d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:05.098 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.099+0000 7f0e0e23b700 1 --2- 172.21.15.188:0/4187459125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08198270 0x7f0e0819d2a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00b780 tx=0x7f0dfc00bb40 comp rx=0 tx=0).stop 2024-04-26T01:35:05.098 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.099+0000 7f0e0e23b700 1 -- 172.21.15.188:0/4187459125 shutdown_connections 2024-04-26T01:35:05.098 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.099+0000 7f0e0e23b700 1 --2- 172.21.15.188:0/4187459125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e081986f0 0x7f0e0819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:05.098 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.099+0000 7f0e0e23b700 1 --2- 172.21.15.188:0/4187459125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e081025d0 0x7f0e08197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:05.098 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.099+0000 7f0e0e23b700 1 --2- 172.21.15.188:0/4187459125 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f0df0039ae0 0x7f0df003bfa0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:05.099 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.099+0000 7f0e0e23b700 1 --2- 172.21.15.188:0/4187459125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e08198270 0x7f0e0819d2a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:05.099 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.099+0000 7f0e0e23b700 1 -- 172.21.15.188:0/4187459125 >> 172.21.15.188:0/4187459125 conn(0x7f0e080fdb70 msgr2=0x7f0e080fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:05.099 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.100+0000 7f0e0e23b700 1 -- 172.21.15.188:0/4187459125 shutdown_connections 2024-04-26T01:35:05.099 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:35:05.100+0000 7f0e0e23b700 1 -- 172.21.15.188:0/4187459125 wait complete. 2024-04-26T01:35:05.763 DEBUG:teuthology.orchestra.run.smithi029:mgr.y> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T01:35:05.766 DEBUG:teuthology.orchestra.run.smithi188:mgr.z> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.z.service 2024-04-26T01:35:05.768 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T01:35:05.768 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:35:05.769 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:35:05.825 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:35:05.825 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-04-26T01:35:05.873 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:35:05.873 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:35:05.873 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-04-26T01:35:05.873 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:35:05.873 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:35:05.874 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-26 01:34:02.767432729 +0000 2024-04-26T01:35:05.874 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-26 01:28:22.181793377 +0000 2024-04-26T01:35:05.874 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-26 01:28:22.181793377 +0000 2024-04-26T01:35:05.874 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-04-26 01:28:22.181793377 +0000 2024-04-26T01:35:05.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:35:05.963 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-26T01:35:05.963 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-26T01:35:05.963 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000418948 s, 1.2 MB/s 2024-04-26T01:35:05.967 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:35:06.029 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-04-26T01:35:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:35:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:35:06.085 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-04-26T01:35:06.085 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:35:06.085 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:35:06.085 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-26 01:34:02.845431900 +0000 2024-04-26T01:35:06.085 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-26 01:28:22.793786957 +0000 2024-04-26T01:35:06.085 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-26 01:28:22.793786957 +0000 2024-04-26T01:35:06.085 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-04-26 01:28:22.793786957 +0000 2024-04-26T01:35:06.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:35:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-26T01:35:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-26T01:35:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000514164 s, 996 kB/s 2024-04-26T01:35:06.178 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:35:06.239 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-04-26T01:35:06.289 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:35:06.289 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:35:06.289 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-04-26T01:35:06.289 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:35:06.289 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:35:06.289 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-26 01:34:02.912431189 +0000 2024-04-26T01:35:06.290 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-26 01:28:23.407780516 +0000 2024-04-26T01:35:06.290 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-26 01:28:23.407780516 +0000 2024-04-26T01:35:06.290 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-04-26 01:28:23.407780516 +0000 2024-04-26T01:35:06.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:35:06.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:06 smithi006 ceph-mon[28419]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:06.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:06 smithi006 ceph-mon[28419]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi006=x;smithi029=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:06.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:06 smithi006 ceph-mon[28419]: Saving service mgr spec with placement smithi006=x;smithi029=y;smithi188=z;count:3 2024-04-26T01:35:06.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:06.372 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-26T01:35:06.372 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-26T01:35:06.372 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000370192 s, 1.4 MB/s 2024-04-26T01:35:06.375 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:35:06.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:06 smithi188 ceph-mon[32751]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:06.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:06 smithi188 ceph-mon[32751]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi006=x;smithi029=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:06.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:06 smithi188 ceph-mon[32751]: Saving service mgr spec with placement smithi006=x;smithi029=y;smithi188=z;count:3 2024-04-26T01:35:06.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:06 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:06.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:06 smithi029 ceph-mon[32775]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:06.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:06 smithi029 ceph-mon[32775]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi006=x;smithi029=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:06.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:06 smithi029 ceph-mon[32775]: Saving service mgr spec with placement smithi006=x;smithi029=y;smithi188=z;count:3 2024-04-26T01:35:06.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:06 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:06.478 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-04-26T01:35:06.535 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:35:06.535 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:35:06.535 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-04-26T01:35:06.536 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:35:06.536 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:35:06.536 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-26 01:34:02.978430488 +0000 2024-04-26T01:35:06.536 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-26 01:28:24.018774106 +0000 2024-04-26T01:35:06.536 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-26 01:28:24.018774106 +0000 2024-04-26T01:35:06.536 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-04-26 01:28:24.018774106 +0000 2024-04-26T01:35:06.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:35:06.618 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-26T01:35:06.618 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-26T01:35:06.618 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000358826 s, 1.4 MB/s 2024-04-26T01:35:06.622 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:35:06.682 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-26T01:35:06.683 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:35:06.705 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:35:06.705 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-04-26T01:35:06.764 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:35:06.764 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:35:06.764 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-26T01:35:06.764 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:35:06.764 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:35:06.764 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-26 01:34:50.398624168 +0000 2024-04-26T01:35:06.764 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-26 01:27:14.073951209 +0000 2024-04-26T01:35:06.764 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-26 01:27:14.073951209 +0000 2024-04-26T01:35:06.764 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-26 01:27:14.073951209 +0000 2024-04-26T01:35:06.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:35:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-26T01:35:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-26T01:35:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000171986 s, 3.0 MB/s 2024-04-26T01:35:06.833 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:35:06.893 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-04-26T01:35:06.951 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:35:06.951 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:35:06.951 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-26T01:35:06.951 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:35:06.951 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:35:06.952 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-26 01:34:50.398624168 +0000 2024-04-26T01:35:06.952 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-26 01:27:14.395945798 +0000 2024-04-26T01:35:06.952 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-26 01:27:14.395945798 +0000 2024-04-26T01:35:06.952 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-26 01:27:14.395945798 +0000 2024-04-26T01:35:06.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:35:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-26T01:35:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-26T01:35:07.018 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000120004 s, 4.3 MB/s 2024-04-26T01:35:07.019 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:35:07.077 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-04-26T01:35:07.134 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:35:07.134 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:35:07.134 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-26T01:35:07.134 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:35:07.134 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:35:07.134 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-26 01:34:50.399624152 +0000 2024-04-26T01:35:07.134 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-26 01:27:14.730940168 +0000 2024-04-26T01:35:07.134 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-26 01:27:14.730940168 +0000 2024-04-26T01:35:07.134 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-26 01:27:14.730940168 +0000 2024-04-26T01:35:07.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:35:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-26T01:35:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-26T01:35:07.207 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000119513 s, 4.3 MB/s 2024-04-26T01:35:07.208 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:35:07.265 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-04-26T01:35:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:35:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:35:07.322 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-26T01:35:07.322 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:35:07.322 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:35:07.322 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-26 01:34:50.399624152 +0000 2024-04-26T01:35:07.322 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-26 01:27:15.072934420 +0000 2024-04-26T01:35:07.322 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-26 01:27:15.072934420 +0000 2024-04-26T01:35:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-26 01:27:15.072934420 +0000 2024-04-26T01:35:07.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:35:07.386 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-26T01:35:07.386 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-26T01:35:07.386 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.5589e-05 s, 5.4 MB/s 2024-04-26T01:35:07.387 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:35:07.444 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-26T01:35:07.444 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:35:07.459 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:35:07.459 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2024-04-26T01:35:07.513 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:35:07.514 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:35:07.514 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-26T01:35:07.514 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:35:07.514 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:35:07.514 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-26 01:34:41.251547853 +0000 2024-04-26T01:35:07.514 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-26 01:27:08.717797219 +0000 2024-04-26T01:35:07.514 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-26 01:27:08.717797219 +0000 2024-04-26T01:35:07.514 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-26 01:27:08.717797219 +0000 2024-04-26T01:35:07.515 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:35:07.577 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-26T01:35:07.577 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-26T01:35:07.577 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000212481 s, 2.4 MB/s 2024-04-26T01:35:07.578 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:35:07.634 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2024-04-26T01:35:07.689 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:35:07.690 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:35:07.690 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-26T01:35:07.690 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:35:07.690 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:35:07.690 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-26 01:34:41.252547834 +0000 2024-04-26T01:35:07.690 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-26 01:27:09.054790862 +0000 2024-04-26T01:35:07.690 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-26 01:27:09.054790862 +0000 2024-04-26T01:35:07.690 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-26 01:27:09.054790862 +0000 2024-04-26T01:35:07.690 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:35:07.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:07 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:07.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:07 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:07.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:07 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:07.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:07 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:07.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:07 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:07.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:07 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:07.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:07 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:35:07.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:07 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:35:07.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:07 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:35:07.751 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:07 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:07.751 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:07 smithi188 ceph-mon[32751]: Deploying daemon mgr.z on smithi188 2024-04-26T01:35:07.751 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-26T01:35:07.751 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-26T01:35:07.752 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000173911 s, 2.9 MB/s 2024-04-26T01:35:07.753 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:35:07.809 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2024-04-26T01:35:07.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:07 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:07.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:07 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:07.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:07 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:07.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:07 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:07.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:07 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:07.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:07 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:07.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:07 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:35:07.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:07 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:35:07.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:07 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:35:07.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:07 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:07.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:07 smithi006 ceph-mon[28419]: Deploying daemon mgr.z on smithi188 2024-04-26T01:35:07.863 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:35:07.863 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:35:07.863 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-26T01:35:07.864 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:35:07.864 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:35:07.864 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-26 01:34:41.252547834 +0000 2024-04-26T01:35:07.864 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-26 01:27:09.407784204 +0000 2024-04-26T01:35:07.864 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-26 01:27:09.407784204 +0000 2024-04-26T01:35:07.864 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-26 01:27:09.407784204 +0000 2024-04-26T01:35:07.864 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:35:07.926 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-26T01:35:07.926 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-26T01:35:07.926 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.00024589 s, 2.1 MB/s 2024-04-26T01:35:07.927 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:35:07.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:07 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:07.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:07 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:07.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:07 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:07.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:07 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:07.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:07 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:07.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:07 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:07.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:07 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:35:07.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:07 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:35:07.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:07 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:35:07.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:07 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:07.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:07 smithi029 ceph-mon[32775]: Deploying daemon mgr.z on smithi188 2024-04-26T01:35:07.984 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2024-04-26T01:35:08.039 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:35:08.039 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:35:08.040 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-26T01:35:08.040 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:35:08.040 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:35:08.040 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-26 01:34:41.252547834 +0000 2024-04-26T01:35:08.040 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-26 01:27:09.738777961 +0000 2024-04-26T01:35:08.040 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-26 01:27:09.738777961 +0000 2024-04-26T01:35:08.040 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-26 01:27:09.738777961 +0000 2024-04-26T01:35:08.040 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:35:08.102 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-26T01:35:08.102 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-26T01:35:08.102 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000192652 s, 2.7 MB/s 2024-04-26T01:35:08.103 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:35:08.159 INFO:tasks.cephadm:Deploying osd.0 on smithi006 with /dev/vg_nvme/lv_4... 2024-04-26T01:35:08.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T01:35:08.596 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:35:08.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:08 smithi188 ceph-mon[32751]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:08.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:08 smithi006 ceph-mon[28419]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:08.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:08 smithi029 ceph-mon[32775]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:09.020 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:09 smithi188 podman[34756]: 2024-04-26 01:35:09.001457282 +0000 UTC m=+0.355472299 container start 8a425eb7f7ea4ab302a05eaef0ee92d1a31b5ee8895443e0417e8fbe77a169fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD) 2024-04-26T01:35:09.020 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:09 smithi188 ceph-mgr[34775]: -- 172.21.15.188:0/1271248189 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56488ebb05a0 con 0x56488eb84c00 2024-04-26T01:35:09.331 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:09 smithi188 bash[34756]: 8a425eb7f7ea4ab302a05eaef0ee92d1a31b5ee8895443e0417e8fbe77a169fb 2024-04-26T01:35:09.332 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:09 smithi188 systemd[1]: Started Ceph mgr.z for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:35:09.332 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:09 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:09.153+0000 7fed2b13a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:35:09.332 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:09 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:09.336+0000 7fed2b13a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:35:09.637 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:09 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:09.438+0000 7fed2b13a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:35:10.137 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:09 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:09.744+0000 7fed2b13a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:35:10.137 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:09 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:09.828+0000 7fed2b13a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:35:10.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:10 smithi029 ceph-mon[32775]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:10.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:10.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:10.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:10.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:35:10.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:35:10.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:35:10.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:10.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:10 smithi029 ceph-mon[32775]: Deploying daemon mgr.y on smithi029 2024-04-26T01:35:10.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:10 smithi188 ceph-mon[32751]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:10.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:10.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:10.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:10.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:35:10.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:35:10.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:35:10.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:10.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:10 smithi188 ceph-mon[32751]: Deploying daemon mgr.y on smithi029 2024-04-26T01:35:10.442 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:10 smithi006 ceph-mon[28419]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:10.442 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:10.442 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:10.442 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:10.443 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:35:10.443 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:35:10.443 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:35:10.443 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:10.443 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:10 smithi006 ceph-mon[28419]: Deploying daemon mgr.y on smithi029 2024-04-26T01:35:11.387 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:11.055+0000 7fed2b13a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:35:11.387 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:11.227+0000 7fed2b13a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:35:11.438 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:11 smithi029 systemd[1]: Starting Ceph mgr.y for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:35:11.552 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T01:35:11.552 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T01:35:11.552 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-04-26T01:35:11.552 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-04-26T01:35:11.552 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0348667 s, 301 MB/s 2024-04-26T01:35:11.552 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-04-26T01:35:11.637 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:11.422+0000 7fed2b13a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:35:11.767 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:11 smithi029 podman[34150]: 2024-04-26 01:35:11.367182031 +0000 UTC m=+0.018325169 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:35:11.768 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:11 smithi029 podman[34150]: 2024-04-26 01:35:11.466272314 +0000 UTC m=+0.117415479 container create c9b317ef545a63c8d52fb625add9e3205763ff29c1fb9ea3242bdbcb53d89e1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-26T01:35:11.768 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:11 smithi029 podman[34150]: 2024-04-26 01:35:11.671792714 +0000 UTC m=+0.322935846 container init c9b317ef545a63c8d52fb625add9e3205763ff29c1fb9ea3242bdbcb53d89e1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:35:11.768 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:11 smithi029 podman[34150]: 2024-04-26 01:35:11.675062302 +0000 UTC m=+0.326205426 container start c9b317ef545a63c8d52fb625add9e3205763ff29c1fb9ea3242bdbcb53d89e1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-26T01:35:11.768 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:11 smithi029 ceph-mgr[34169]: -- 172.21.15.29:0/1722981942 <== mon.2 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561aefd3e5a0 con 0x561aefd12c00 2024-04-26T01:35:11.768 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:11 smithi029 bash[34150]: c9b317ef545a63c8d52fb625add9e3205763ff29c1fb9ea3242bdbcb53d89e1d 2024-04-26T01:35:12.000 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:11.709+0000 7fed2b13a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:35:12.000 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:11.799+0000 7fed2b13a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:35:12.000 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:12 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:12.004+0000 7fed2b13a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:35:12.017 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:11 smithi029 systemd[1]: Started Ceph mgr.y for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:35:12.017 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:11 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:11.830+0000 7f71410ef200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:35:12.017 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:12 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:12.022+0000 7f71410ef200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:35:12.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi006:vg_nvme/lv_4 2024-04-26T01:35:12.387 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:12 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:12.201+0000 7fed2b13a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:35:12.438 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:12 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:12.129+0000 7f71410ef200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:35:12.842 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:12 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:12.458+0000 7f71410ef200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:35:12.842 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:12 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:12.548+0000 7f71410ef200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:35:12.849 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:35:13.114 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:12 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:12.844+0000 7fed2b13a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:35:13.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:12 smithi188 ceph-mon[32751]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:13.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:12 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:12 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:12 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:12 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:12 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:12 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:13.142 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:12 smithi029 ceph-mon[32775]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:13.142 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:12 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.142 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:12 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.142 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:12 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.142 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:12 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.142 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:12 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.142 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:12 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:13.195 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:12 smithi006 ceph-mon[28419]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:13.195 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:12 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.195 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:12 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.195 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:12 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.195 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:12 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.195 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:12 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:13.195 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:12 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:13.375 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:13.282+0000 7fed2b13a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:35:13.376 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:13.379+0000 7fed2b13a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:35:13.637 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:13.596+0000 7fed2b13a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:35:14.186 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:13 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:13.840+0000 7f71410ef200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:35:14.187 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:14.015+0000 7f71410ef200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:35:14.187 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:14.191+0000 7f71410ef200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:35:14.387 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:14.120+0000 7fed2b13a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:35:14.387 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:14.212+0000 7fed2b13a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:35:14.788 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:14.788 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.788 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.788 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.788 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.788 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:14.788 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:14.788 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.788 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T01:35:14.789 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:35:14.789 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:35:14.789 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:14.789 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-mon[32751]: Reconfiguring daemon mgr.x on smithi006 2024-04-26T01:35:14.789 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:14.452+0000 7fed2b13a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:35:14.789 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:14.583+0000 7fed2b13a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:35:14.789 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:14.678+0000 7fed2b13a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:35:14.789 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:14.791+0000 7fed2b13a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:35:14.790 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:14.488+0000 7f71410ef200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:35:14.790 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:14.584+0000 7f71410ef200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:35:14.790 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:14.793+0000 7f71410ef200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:35:14.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:14.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:14 smithi006 ceph-mon[28419]: Reconfiguring daemon mgr.x on smithi006 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.791 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.792 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:14.792 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:14.792 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:14.792 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T01:35:14.792 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:35:14.792 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:35:14.792 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:14.792 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-mon[32775]: Reconfiguring daemon mgr.x on smithi006 2024-04-26T01:35:14.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.965+0000 7f1f0f600700 1 -- 172.21.15.6:0/1402348192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f08100d50 msgr2=0x7f1f08101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:14.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.965+0000 7f1f0f600700 1 --2- 172.21.15.6:0/1402348192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f08100d50 0x7f1f08101170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1efc009990 tx=0x7f1efc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.966+0000 7f1f0f600700 1 -- 172.21.15.6:0/1402348192 shutdown_connections 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.966+0000 7f1f0f600700 1 --2- 172.21.15.6:0/1402348192 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f081046d0 0x7f1f0810af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.966+0000 7f1f0f600700 1 --2- 172.21.15.6:0/1402348192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f08100d50 0x7f1f08101170 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.966+0000 7f1f0f600700 1 --2- 172.21.15.6:0/1402348192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f08103d30 0x7f1f08104190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.966+0000 7f1f0f600700 1 -- 172.21.15.6:0/1402348192 >> 172.21.15.6:0/1402348192 conn(0x7f1f080fc6d0 msgr2=0x7f1f080feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.966+0000 7f1f0f600700 1 -- 172.21.15.6:0/1402348192 shutdown_connections 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.966+0000 7f1f0f600700 1 -- 172.21.15.6:0/1402348192 wait complete. 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.967+0000 7f1f0f600700 1 Processor -- start 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.967+0000 7f1f0f600700 1 -- start start 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.967+0000 7f1f0f600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f08100d50 0x7f1f08103690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.967+0000 7f1f0f600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f08103d30 0x7f1f08101ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.967+0000 7f1f0f600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f081046d0 0x7f1f08102220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.967+0000 7f1f0f600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0810e4f0 con 0x7f1f08103d30 2024-04-26T01:35:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.967+0000 7f1f0f600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0810e370 con 0x7f1f081046d0 2024-04-26T01:35:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.967+0000 7f1f0f600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0810e670 con 0x7f1f08100d50 2024-04-26T01:35:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.968+0000 7f1f0d600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f081046d0 0x7f1f08102220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.968+0000 7f1f0cc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f08100d50 0x7f1f08103690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.968+0000 7f1f0d600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f081046d0 0x7f1f08102220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53388/0 (socket says 172.21.15.6:53388) 2024-04-26T01:35:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.968+0000 7f1f0d600700 1 -- 172.21.15.6:0/4154461035 learned_addr learned my addr 172.21.15.6:0/4154461035 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:35:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.968+0000 7f1f0d600700 1 -- 172.21.15.6:0/4154461035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f08100d50 msgr2=0x7f1f08103690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.968+0000 7f1f0d600700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f08100d50 0x7f1f08103690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.968+0000 7f1f0d600700 1 -- 172.21.15.6:0/4154461035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f08103d30 msgr2=0x7f1f08101ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T01:35:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.968+0000 7f1f07e00700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f08103d30 0x7f1f08101ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.968+0000 7f1f0d600700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f08103d30 0x7f1f08101ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.968+0000 7f1f0d600700 1 -- 172.21.15.6:0/4154461035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f08102ad0 con 0x7f1f081046d0 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.969+0000 7f1f07e00700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f08103d30 0x7f1f08101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.969+0000 7f1f0d600700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f081046d0 0x7f1f08102220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1f00007d70 tx=0x7f1f0000a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.969+0000 7f1f05600700 1 -- 172.21.15.6:0/4154461035 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f00016020 con 0x7f1f081046d0 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.969+0000 7f1f05600700 1 -- 172.21.15.6:0/4154461035 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f0000ad40 con 0x7f1f081046d0 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.969+0000 7f1f05600700 1 -- 172.21.15.6:0/4154461035 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f000115e0 con 0x7f1f081046d0 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.969+0000 7f1f0f600700 1 -- 172.21.15.6:0/4154461035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f081a2cf0 con 0x7f1f081046d0 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.969+0000 7f1f0f600700 1 -- 172.21.15.6:0/4154461035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f081a31b0 con 0x7f1f081046d0 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.970+0000 7f1f05600700 1 -- 172.21.15.6:0/4154461035 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f1f00019020 con 0x7f1f081046d0 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.970+0000 7f1f0f600700 1 -- 172.21.15.6:0/4154461035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f08069730 con 0x7f1f081046d0 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.971+0000 7f1f05600700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ef0039890 0x7f1ef003bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.971+0000 7f1f05600700 1 -- 172.21.15.6:0/4154461035 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1f0004ca80 con 0x7f1f081046d0 2024-04-26T01:35:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.971+0000 7f1f0cc00700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ef0039890 0x7f1ef003bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:14.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.972+0000 7f1f0cc00700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ef0039890 0x7f1ef003bd50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0098e0 tx=0x7f1efc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:14.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:14.977+0000 7f1f05600700 1 -- 172.21.15.6:0/4154461035 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f00020020 con 0x7f1f081046d0 2024-04-26T01:35:15.188 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:14 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:14.992+0000 7f71410ef200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:35:15.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:15.220+0000 7f1f0f600700 1 -- 172.21.15.6:0/4154461035 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1f08062e80 con 0x7f1ef0039890 2024-04-26T01:35:15.387 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:15.097+0000 7fed2b13a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:35:15.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:15 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:15.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:15 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:15.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:15 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:15.887 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:15.698+0000 7fed2b13a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:35:15.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:15 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:15.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:15 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:15.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:15 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:15.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:15 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:15.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:15 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:15.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:15 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:15.939 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:15 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:15.616+0000 7f71410ef200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:35:16.352 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:16.050+0000 7f71410ef200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:35:16.352 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:16.149+0000 7f71410ef200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:35:16.352 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:16.356+0000 7f71410ef200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:35:16.387 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:35:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:35:15.912+0000 7fed2b13a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:35:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:16.528+0000 7f1f05600700 1 -- 172.21.15.6:0/4154461035 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 14) ==== 92601+0+0 (secure 0 0 0) 0x7f1f00011b00 con 0x7f1f081046d0 2024-04-26T01:35:16.622 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:16.622 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:16.622 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: Standby manager daemon z started 2024-04-26T01:35:16.622 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.188:0/1944687242' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:35:16.622 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.188:0/1944687242' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:35:16.622 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.188:0/1944687242' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:35:16.622 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.188:0/1944687242' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:35:16.622 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:16.622 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:16.622 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:16.623 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:16.623 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:16.623 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:16.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:16.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:16.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: Standby manager daemon z started 2024-04-26T01:35:16.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.188:0/1944687242' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:35:16.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.188:0/1944687242' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:35:16.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.188:0/1944687242' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:35:16.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.188:0/1944687242' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:35:16.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:16.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:16.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:16.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:16.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:16.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:16.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:16.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:16.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: Standby manager daemon z started 2024-04-26T01:35:16.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.188:0/1944687242' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:35:16.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.188:0/1944687242' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:35:16.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.188:0/1944687242' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:35:16.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.188:0/1944687242' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:35:16.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:16.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:16.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:16.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:16.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:16.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:17.188 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:16.902+0000 7f71410ef200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:35:17.188 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:16 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:16.997+0000 7f71410ef200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:35:17.528 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:17 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:17.247+0000 7f71410ef200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:35:17.528 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:17 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:17.385+0000 7f71410ef200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:35:17.528 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:17 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:17.482+0000 7f71410ef200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:35:17.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:17 smithi006 ceph-mon[28419]: mgrmap e14: x(active, since 89s), standbys: z 2024-04-26T01:35:17.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:17 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:35:17.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:17 smithi188 ceph-mon[32751]: mgrmap e14: x(active, since 89s), standbys: z 2024-04-26T01:35:17.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:17 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:35:17.910 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:17 smithi029 ceph-mon[32775]: mgrmap e14: x(active, since 89s), standbys: z 2024-04-26T01:35:17.910 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:17 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:35:17.910 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:17 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:17.602+0000 7f71410ef200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:35:18.188 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:17 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:17.915+0000 7f71410ef200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:35:18.766 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:18 smithi029 ceph-mon[32775]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:18.766 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:18 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:18.548+0000 7f71410ef200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:35:18.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:18 smithi006 ceph-mon[28419]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:18.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:18 smithi188 ceph-mon[32751]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:19.188 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:35:18 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:35:18.770+0000 7f71410ef200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:35:19.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:19.553+0000 7f1f05600700 1 -- 172.21.15.6:0/4154461035 <== mon.2 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7f1f0005f940 con 0x7f1f081046d0 2024-04-26T01:35:19.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:19 smithi006 ceph-mon[28419]: Standby manager daemon y started 2024-04-26T01:35:19.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:19 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.29:0/1857746939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:35:19.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:19 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.29:0/1857746939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:35:19.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:19 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.29:0/1857746939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:35:19.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:19 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.29:0/1857746939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:35:19.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:19 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2259371048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f164c51d-ce5d-4140-b9e3-a4b84088f5ae"}]: dispatch 2024-04-26T01:35:19.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:19 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f164c51d-ce5d-4140-b9e3-a4b84088f5ae"}]: dispatch 2024-04-26T01:35:19.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:19 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f164c51d-ce5d-4140-b9e3-a4b84088f5ae"}]': finished 2024-04-26T01:35:19.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:19 smithi006 ceph-mon[28419]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T01:35:19.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:19 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:19.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:19 smithi188 ceph-mon[32751]: Standby manager daemon y started 2024-04-26T01:35:19.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:19 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.29:0/1857746939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:35:19.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:19 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.29:0/1857746939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:35:19.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:19 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.29:0/1857746939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:35:19.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:19 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.29:0/1857746939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:35:19.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:19 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2259371048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f164c51d-ce5d-4140-b9e3-a4b84088f5ae"}]: dispatch 2024-04-26T01:35:19.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:19 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f164c51d-ce5d-4140-b9e3-a4b84088f5ae"}]: dispatch 2024-04-26T01:35:19.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:19 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f164c51d-ce5d-4140-b9e3-a4b84088f5ae"}]': finished 2024-04-26T01:35:19.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:19 smithi188 ceph-mon[32751]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T01:35:19.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:19 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:19.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:19 smithi029 ceph-mon[32775]: Standby manager daemon y started 2024-04-26T01:35:19.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:19 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.29:0/1857746939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:35:19.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:19 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.29:0/1857746939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:35:19.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:19 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.29:0/1857746939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:35:19.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:19 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.29:0/1857746939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:35:19.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:19 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2259371048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f164c51d-ce5d-4140-b9e3-a4b84088f5ae"}]: dispatch 2024-04-26T01:35:19.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:19 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f164c51d-ce5d-4140-b9e3-a4b84088f5ae"}]: dispatch 2024-04-26T01:35:19.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:19 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f164c51d-ce5d-4140-b9e3-a4b84088f5ae"}]': finished 2024-04-26T01:35:19.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:19 smithi029 ceph-mon[32775]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T01:35:19.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:19 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:20.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:20 smithi006 ceph-mon[28419]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:20.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:20 smithi006 ceph-mon[28419]: mgrmap e15: x(active, since 92s), standbys: z, y 2024-04-26T01:35:20.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:35:20.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:20 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3122010644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:20.879 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:20 smithi188 ceph-mon[32751]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:20.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:20 smithi188 ceph-mon[32751]: mgrmap e15: x(active, since 92s), standbys: z, y 2024-04-26T01:35:20.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:35:20.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:20 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3122010644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:20.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:20 smithi029 ceph-mon[32775]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:20.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:20 smithi029 ceph-mon[32775]: mgrmap e15: x(active, since 92s), standbys: z, y 2024-04-26T01:35:20.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:35:20.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:20 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3122010644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:22.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:22 smithi188 ceph-mon[32751]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:22.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:22 smithi029 ceph-mon[32775]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:23.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:22 smithi006 ceph-mon[28419]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:24.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:24 smithi006 ceph-mon[28419]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:24.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:24 smithi188 ceph-mon[32751]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:24.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:24 smithi029 ceph-mon[32775]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:26.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:26 smithi006 ceph-mon[28419]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:26.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:26 smithi188 ceph-mon[32751]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:26.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:26 smithi029 ceph-mon[32775]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:28.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:28 smithi006 ceph-mon[28419]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:28.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:28 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T01:35:28.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:28 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:28.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:28 smithi188 ceph-mon[32751]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:28.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:28 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T01:35:28.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:28 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:28.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:28 smithi029 ceph-mon[32775]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:28.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:28 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T01:35:28.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:28 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:29.757 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:29 smithi006 ceph-mon[28419]: Deploying daemon osd.0 on smithi006 2024-04-26T01:35:29.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:29 smithi188 ceph-mon[32751]: Deploying daemon osd.0 on smithi006 2024-04-26T01:35:29.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:29 smithi029 ceph-mon[32775]: Deploying daemon osd.0 on smithi006 2024-04-26T01:35:30.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:30 smithi006 ceph-mon[28419]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:30.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:30 smithi188 ceph-mon[32751]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:30.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:30 smithi029 ceph-mon[32775]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:32.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:32 smithi188 ceph-mon[32751]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:32.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:32 smithi029 ceph-mon[32775]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:33.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:32 smithi006 ceph-mon[28419]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:34.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:34 smithi188 ceph-mon[32751]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:34.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:34 smithi029 ceph-mon[32775]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:34.994 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:34 smithi006 ceph-mon[28419]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:36.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:36 smithi188 ceph-mon[32751]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:36.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:36 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:36.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:36 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:36.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:36 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:36.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:36 smithi029 ceph-mon[32775]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:36.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:36 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:36.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:36 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:36.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:36 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:36.939 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:36 smithi006 ceph-mon[28419]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:36.939 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:36 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:36.939 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:36 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:36.939 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:36 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:38.875 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:38 smithi006 ceph-mon[28419]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:38.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:38 smithi188 ceph-mon[32751]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:38.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:38 smithi029 ceph-mon[32775]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:40.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:40 smithi006 ceph-mon[28419]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:40.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:40 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:40.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:40 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:40.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:40 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:40.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:40 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:40.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:40 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:40.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:40 smithi188 ceph-mon[32751]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:40.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:40 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:40.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:40 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:40.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:40 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:40.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:40 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:40.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:40 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:40.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:40 smithi029 ceph-mon[32775]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:40.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:40 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:40.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:40 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:40.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:40 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:40.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:40 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:40.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:40 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:41.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.416+0000 7f1f05600700 1 -- 172.21.15.6:0/4154461035 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1f08062e80 con 0x7f1ef0039890 2024-04-26T01:35:41.414 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 0 on host 'smithi006' 2024-04-26T01:35:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.421+0000 7f1f0f600700 1 -- 172.21.15.6:0/4154461035 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ef0039890 msgr2=0x7f1ef003bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.421+0000 7f1f0f600700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ef0039890 0x7f1ef003bd50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0098e0 tx=0x7f1efc005960 comp rx=0 tx=0).stop 2024-04-26T01:35:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.421+0000 7f1f0f600700 1 -- 172.21.15.6:0/4154461035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f081046d0 msgr2=0x7f1f08102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.421+0000 7f1f0f600700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f081046d0 0x7f1f08102220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1f00007d70 tx=0x7f1f0000a340 comp rx=0 tx=0).stop 2024-04-26T01:35:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.421+0000 7f1f0f600700 1 -- 172.21.15.6:0/4154461035 shutdown_connections 2024-04-26T01:35:41.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.421+0000 7f1f0f600700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f081046d0 0x7f1f08102220 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:41.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.421+0000 7f1f0f600700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ef0039890 0x7f1ef003bd50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:41.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.421+0000 7f1f0f600700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f08100d50 0x7f1f08103690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:41.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.421+0000 7f1f0f600700 1 --2- 172.21.15.6:0/4154461035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f08103d30 0x7f1f08101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:41.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.421+0000 7f1f0f600700 1 -- 172.21.15.6:0/4154461035 >> 172.21.15.6:0/4154461035 conn(0x7f1f080fc6d0 msgr2=0x7f1f081070b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:41.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.421+0000 7f1f0f600700 1 -- 172.21.15.6:0/4154461035 shutdown_connections 2024-04-26T01:35:41.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:35:41.421+0000 7f1f0f600700 1 -- 172.21.15.6:0/4154461035 wait complete. 2024-04-26T01:35:41.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:41 smithi006 ceph-mon[28419]: from='osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T01:35:41.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:41 smithi188 ceph-mon[32751]: from='osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T01:35:41.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:41 smithi029 ceph-mon[32775]: from='osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T01:35:42.153 DEBUG:teuthology.orchestra.run.smithi006:osd.0> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T01:35:42.162 INFO:tasks.cephadm:Deploying osd.1 on smithi029 with /dev/vg_nvme/lv_4... 2024-04-26T01:35:42.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T01:35:42.381 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:42 smithi006 ceph-mon[28419]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:42.381 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:42 smithi006 ceph-mon[28419]: from='osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T01:35:42.381 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:42 smithi006 ceph-mon[28419]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:42 smithi006 ceph-mon[28419]: from='osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:42 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:42 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:42 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:42 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:42.382 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 01:35:42 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0[43410]: 2024-04-26T01:35:42.326+0000 7f4c73800700 -1 osd.0 0 waiting for initial osdmap 2024-04-26T01:35:42.382 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 01:35:42 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0[43410]: 2024-04-26T01:35:42.331+0000 7f4c6e600700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:35:42.417 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:35:42.540 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:42 smithi029 ceph-mon[32775]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:42.540 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:42 smithi029 ceph-mon[32775]: from='osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T01:35:42.541 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:42 smithi029 ceph-mon[32775]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T01:35:42.541 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:42 smithi029 ceph-mon[32775]: from='osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:35:42.541 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:42 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:42.541 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:42 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:42.541 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:42 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:42.541 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:42 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:42.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:42 smithi188 ceph-mon[32751]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:42.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:42 smithi188 ceph-mon[32751]: from='osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T01:35:42.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:42 smithi188 ceph-mon[32751]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T01:35:42.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:42 smithi188 ceph-mon[32751]: from='osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:35:42.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:42 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:42.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:42 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:42.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:42 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:42.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:42 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:43.560 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:43 smithi006 ceph-mon[28419]: from='osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-04-26T01:35:43.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:43 smithi006 ceph-mon[28419]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T01:35:43.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:43 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:43.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:43 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:43.563 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:43 smithi029 ceph-mon[32775]: from='osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-04-26T01:35:43.563 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:43 smithi029 ceph-mon[32775]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T01:35:43.563 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:43 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:43.563 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:43 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:43.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:43 smithi188 ceph-mon[32751]: from='osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-04-26T01:35:43.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:43 smithi188 ceph-mon[32751]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T01:35:43.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:43 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:43.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:43 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:44.309 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T01:35:44.310 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T01:35:44.310 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-04-26T01:35:44.310 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-04-26T01:35:44.310 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00844123 s, 1.2 GB/s 2024-04-26T01:35:44.310 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-04-26T01:35:44.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:44 smithi006 ceph-mon[28419]: purged_snaps scrub starts 2024-04-26T01:35:44.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:44 smithi006 ceph-mon[28419]: purged_snaps scrub ok 2024-04-26T01:35:44.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:44 smithi006 ceph-mon[28419]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:44.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:44 smithi006 ceph-mon[28419]: osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486] boot 2024-04-26T01:35:44.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:44 smithi006 ceph-mon[28419]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T01:35:44.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:44 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:44.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:44 smithi188 ceph-mon[32751]: purged_snaps scrub starts 2024-04-26T01:35:44.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:44 smithi188 ceph-mon[32751]: purged_snaps scrub ok 2024-04-26T01:35:44.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:44 smithi188 ceph-mon[32751]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:44.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:44 smithi188 ceph-mon[32751]: osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486] boot 2024-04-26T01:35:44.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:44 smithi188 ceph-mon[32751]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T01:35:44.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:44 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:44.658 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:44 smithi029 ceph-mon[32775]: purged_snaps scrub starts 2024-04-26T01:35:44.658 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:44 smithi029 ceph-mon[32775]: purged_snaps scrub ok 2024-04-26T01:35:44.658 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:44 smithi029 ceph-mon[32775]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:35:44.658 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:44 smithi029 ceph-mon[32775]: osd.0 [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486] boot 2024-04-26T01:35:44.658 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:44 smithi029 ceph-mon[32775]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T01:35:44.658 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:44 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:35:44.994 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_4 2024-04-26T01:35:45.242 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:35:45.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:45 smithi188 ceph-mon[32751]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T01:35:45.676 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:45 smithi029 ceph-mon[32775]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T01:35:45.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:45 smithi006 ceph-mon[28419]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T01:35:46.574 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:46 smithi029 ceph-mon[32775]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:46.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:46 smithi188 ceph-mon[32751]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.790+0000 7f68ed699700 1 -- 172.21.15.29:0/4066700333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e8101380 msgr2=0x7f68e81017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.790+0000 7f68ed699700 1 --2- 172.21.15.29:0/4066700333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e8101380 0x7f68e81017a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f68d8009990 tx=0x7f68d8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.790+0000 7f68ed699700 1 -- 172.21.15.29:0/4066700333 shutdown_connections 2024-04-26T01:35:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.790+0000 7f68ed699700 1 --2- 172.21.15.29:0/4066700333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68e81026a0 0x7f68e8102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.790+0000 7f68ed699700 1 --2- 172.21.15.29:0/4066700333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e8101380 0x7f68e81017a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.790+0000 7f68ed699700 1 --2- 172.21.15.29:0/4066700333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68e8101ce0 0x7f68e8102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.790+0000 7f68ed699700 1 -- 172.21.15.29:0/4066700333 >> 172.21.15.29:0/4066700333 conn(0x7f68e80fcf40 msgr2=0x7f68e80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.790+0000 7f68ed699700 1 -- 172.21.15.29:0/4066700333 shutdown_connections 2024-04-26T01:35:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.790+0000 7f68ed699700 1 -- 172.21.15.29:0/4066700333 wait complete. 2024-04-26T01:35:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68ed699700 1 Processor -- start 2024-04-26T01:35:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68ed699700 1 -- start start 2024-04-26T01:35:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68ed699700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68e8101380 0x7f68e819ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68ed699700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e8101ce0 0x7f68e819b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68ed699700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68e81026a0 0x7f68e81a2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68ed699700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f68e8110790 con 0x7f68e8101380 2024-04-26T01:35:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68ed699700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f68e8110610 con 0x7f68e8101ce0 2024-04-26T01:35:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68ed699700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f68e8110910 con 0x7f68e81026a0 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68e67fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e8101ce0 0x7f68e819b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68e67fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e8101ce0 0x7f68e819b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46938/0 (socket says 172.21.15.29:46938) 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68e67fc700 1 -- 172.21.15.29:0/2405821537 learned_addr learned my addr 172.21.15.29:0/2405821537 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68e77fe700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68e81026a0 0x7f68e81a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68e67fc700 1 -- 172.21.15.29:0/2405821537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68e81026a0 msgr2=0x7f68e81a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68e67fc700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68e81026a0 0x7f68e81a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68e67fc700 1 -- 172.21.15.29:0/2405821537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68e8101380 msgr2=0x7f68e819ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68e67fc700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68e8101380 0x7f68e819ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68e67fc700 1 -- 172.21.15.29:0/2405821537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e81a2af0 con 0x7f68e8101ce0 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68e77fe700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68e81026a0 0x7f68e81a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68e67fc700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e8101ce0 0x7f68e819b3d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00d980 tx=0x7f68dc00dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68cffff700 1 -- 172.21.15.29:0/2405821537 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68dc017070 con 0x7f68e8101ce0 2024-04-26T01:35:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68cffff700 1 -- 172.21.15.29:0/2405821537 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68dc004830 con 0x7f68e8101ce0 2024-04-26T01:35:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68cffff700 1 -- 172.21.15.29:0/2405821537 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68dc005440 con 0x7f68e8101ce0 2024-04-26T01:35:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68ed699700 1 -- 172.21.15.29:0/2405821537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e81a2de0 con 0x7f68e8101ce0 2024-04-26T01:35:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.791+0000 7f68ed699700 1 -- 172.21.15.29:0/2405821537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e81a3310 con 0x7f68e8101ce0 2024-04-26T01:35:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.793+0000 7f68ed699700 1 -- 172.21.15.29:0/2405821537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68e8069730 con 0x7f68e8101ce0 2024-04-26T01:35:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.794+0000 7f68cffff700 1 -- 172.21.15.29:0/2405821537 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7f68dc0055e0 con 0x7f68e8101ce0 2024-04-26T01:35:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.795+0000 7f68cffff700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f68d00a3cb0 0x7f68d00a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.795+0000 7f68cffff700 1 -- 172.21.15.29:0/2405821537 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f68dc0ce8b0 con 0x7f68e8101ce0 2024-04-26T01:35:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.795+0000 7f68e6ffd700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f68d00a3cb0 0x7f68d00a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.796+0000 7f68e6ffd700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f68d00a3cb0 0x7f68d00a6170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68d8006010 tx=0x7f68d8003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:46.800+0000 7f68cffff700 1 -- 172.21.15.29:0/2405821537 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f68dc020020 con 0x7f68e8101ce0 2024-04-26T01:35:46.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:46 smithi006 ceph-mon[28419]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:47.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:35:47.018+0000 7f68ed699700 1 -- 172.21.15.29:0/2405821537 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f68e8062e80 con 0x7f68d00a3cb0 2024-04-26T01:35:48.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:47 smithi029 ceph-mon[32775]: Detected new or changed devices on smithi006 2024-04-26T01:35:48.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:48.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:48.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:48.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:48.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:48.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:48.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:48.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:48.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:48.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:48.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:47 smithi006 ceph-mon[28419]: Detected new or changed devices on smithi006 2024-04-26T01:35:48.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:48.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:48.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:48.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:48.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:48.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:48.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:48.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:48.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:48.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:48.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:47 smithi188 ceph-mon[32751]: Detected new or changed devices on smithi006 2024-04-26T01:35:48.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:48.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:48.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:48.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:48.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:48.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:48.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:35:48.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:48.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:48.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:49.034 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:48 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi006 to 17094M 2024-04-26T01:35:49.034 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:48 smithi006 ceph-mon[28419]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:49.034 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:48 smithi006 ceph-mon[28419]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:49.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:48 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi006 to 17094M 2024-04-26T01:35:49.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:48 smithi029 ceph-mon[32775]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:49.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:48 smithi029 ceph-mon[32775]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:49.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:48 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi006 to 17094M 2024-04-26T01:35:49.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:48 smithi188 ceph-mon[32751]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:49.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:48 smithi188 ceph-mon[32751]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:50.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:49 smithi188 ceph-mon[32751]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:50.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:49 smithi188 ceph-mon[32751]: from='client.? 172.21.15.29:0/2078012226' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57e875dd-ddf1-4ee4-a2a7-9589d7f42818"}]: dispatch 2024-04-26T01:35:50.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:49 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57e875dd-ddf1-4ee4-a2a7-9589d7f42818"}]: dispatch 2024-04-26T01:35:50.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:49 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57e875dd-ddf1-4ee4-a2a7-9589d7f42818"}]': finished 2024-04-26T01:35:50.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:49 smithi188 ceph-mon[32751]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T01:35:50.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:50.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:49 smithi029 ceph-mon[32775]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:50.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:49 smithi029 ceph-mon[32775]: from='client.? 172.21.15.29:0/2078012226' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57e875dd-ddf1-4ee4-a2a7-9589d7f42818"}]: dispatch 2024-04-26T01:35:50.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:49 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57e875dd-ddf1-4ee4-a2a7-9589d7f42818"}]: dispatch 2024-04-26T01:35:50.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:49 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57e875dd-ddf1-4ee4-a2a7-9589d7f42818"}]': finished 2024-04-26T01:35:50.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:49 smithi029 ceph-mon[32775]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T01:35:50.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:50.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:49 smithi006 ceph-mon[28419]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:50.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:49 smithi006 ceph-mon[28419]: from='client.? 172.21.15.29:0/2078012226' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57e875dd-ddf1-4ee4-a2a7-9589d7f42818"}]: dispatch 2024-04-26T01:35:50.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:49 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57e875dd-ddf1-4ee4-a2a7-9589d7f42818"}]: dispatch 2024-04-26T01:35:50.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:49 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57e875dd-ddf1-4ee4-a2a7-9589d7f42818"}]': finished 2024-04-26T01:35:50.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:49 smithi006 ceph-mon[28419]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T01:35:50.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:50 smithi029 ceph-mon[32775]: from='client.? 172.21.15.29:0/4020198830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:51.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:50 smithi006 ceph-mon[28419]: from='client.? 172.21.15.29:0/4020198830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:51.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:50 smithi188 ceph-mon[32751]: from='client.? 172.21.15.29:0/4020198830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:52.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:51 smithi188 ceph-mon[32751]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:52.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:51 smithi006 ceph-mon[28419]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:52.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:51 smithi029 ceph-mon[32775]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:54.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:54 smithi188 ceph-mon[32751]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:54.393 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:54 smithi029 ceph-mon[32775]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:54.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:54 smithi006 ceph-mon[28419]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:56.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:56 smithi188 ceph-mon[32751]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:56.419 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:56 smithi029 ceph-mon[32775]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:56.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:56 smithi006 ceph-mon[28419]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:57.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T01:35:57.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:57.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T01:35:57.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:57.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T01:35:57.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:58.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:58 smithi188 ceph-mon[32751]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:58.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:35:58 smithi188 ceph-mon[32751]: Deploying daemon osd.1 on smithi029 2024-04-26T01:35:58.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:58 smithi029 ceph-mon[32775]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:58.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:35:58 smithi029 ceph-mon[32775]: Deploying daemon osd.1 on smithi029 2024-04-26T01:35:58.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:58 smithi006 ceph-mon[28419]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:58.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:35:58 smithi006 ceph-mon[28419]: Deploying daemon osd.1 on smithi029 2024-04-26T01:36:00.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:00 smithi188 ceph-mon[32751]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:00.408 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:00 smithi029 ceph-mon[32775]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:00.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:00 smithi006 ceph-mon[28419]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:02.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:02 smithi029 ceph-mon[32775]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:02.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:02 smithi006 ceph-mon[28419]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:02.637 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:02 smithi188 ceph-mon[32751]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:03.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:03 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:03.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:03 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:03.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:03 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:04.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:03 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:04.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:03 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:04.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:03 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:04.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:03 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:04.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:03 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:04.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:03 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:05.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:04 smithi006 ceph-mon[28419]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:05.099 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:04 smithi029 ceph-mon[32775]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:05.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:04 smithi188 ceph-mon[32751]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.560+0000 7f68cffff700 1 -- 172.21.15.29:0/2405821537 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f68e8062e80 con 0x7f68d00a3cb0 2024-04-26T01:36:06.557 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 1 on host 'smithi029' 2024-04-26T01:36:06.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.563+0000 7f68ed699700 1 -- 172.21.15.29:0/2405821537 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f68d00a3cb0 msgr2=0x7f68d00a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:06.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.563+0000 7f68ed699700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f68d00a3cb0 0x7f68d00a6170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68d8006010 tx=0x7f68d8003860 comp rx=0 tx=0).stop 2024-04-26T01:36:06.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.563+0000 7f68ed699700 1 -- 172.21.15.29:0/2405821537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e8101ce0 msgr2=0x7f68e819b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:06.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.563+0000 7f68ed699700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e8101ce0 0x7f68e819b3d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00d980 tx=0x7f68dc00dd40 comp rx=0 tx=0).stop 2024-04-26T01:36:06.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.563+0000 7f68ed699700 1 -- 172.21.15.29:0/2405821537 shutdown_connections 2024-04-26T01:36:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.563+0000 7f68ed699700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f68d00a3cb0 0x7f68d00a6170 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.563+0000 7f68ed699700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68e81026a0 0x7f68e81a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.563+0000 7f68ed699700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68e8101380 0x7f68e819ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.563+0000 7f68ed699700 1 --2- 172.21.15.29:0/2405821537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68e8101ce0 0x7f68e819b3d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.563+0000 7f68ed699700 1 -- 172.21.15.29:0/2405821537 >> 172.21.15.29:0/2405821537 conn(0x7f68e80fcf40 msgr2=0x7f68e80fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.564+0000 7f68ed699700 1 -- 172.21.15.29:0/2405821537 shutdown_connections 2024-04-26T01:36:06.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:36:06.564+0000 7f68ed699700 1 -- 172.21.15.29:0/2405821537 wait complete. 2024-04-26T01:36:06.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:06 smithi188 ceph-mon[32751]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:06.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:06 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:06.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:06 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:06.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:06 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:06.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:06 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:06.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:06 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:06.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:06 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:06.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:06 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:06.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:06 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:06.914 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:06 smithi029 ceph-mon[32775]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:06.914 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:06 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:06.914 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:06 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:06.914 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:06 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:06.914 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:06 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:06.914 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:06 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:06.914 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:06 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:06.914 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:06 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:06.914 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:06 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:07.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:06 smithi006 ceph-mon[28419]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:07.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:07.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:07.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:07.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:07.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:07.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:07.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:07.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:07.226 DEBUG:teuthology.orchestra.run.smithi029:osd.1> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T01:36:07.228 INFO:tasks.cephadm:Deploying osd.2 on smithi188 with /dev/vg_nvme/lv_4... 2024-04-26T01:36:07.229 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T01:36:07.476 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:36:07.883 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:07 smithi188 ceph-mon[32751]: from='osd.1 [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:36:07.883 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:07 smithi188 ceph-mon[32751]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:36:07.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:07 smithi029 ceph-mon[32775]: from='osd.1 [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:36:07.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:07 smithi029 ceph-mon[32775]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:36:08.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:07 smithi006 ceph-mon[28419]: from='osd.1 [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:36:08.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:07 smithi006 ceph-mon[28419]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:36:08.736 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:08 smithi188 ceph-mon[32751]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:08.736 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:08 smithi188 ceph-mon[32751]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T01:36:08.736 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:08 smithi188 ceph-mon[32751]: from='osd.1 [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:36:08.736 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:08 smithi188 ceph-mon[32751]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T01:36:08.736 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:08.736 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:08 smithi188 ceph-mon[32751]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:36:08.936 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:08 smithi006 ceph-mon[28419]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:08.936 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:08 smithi006 ceph-mon[28419]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T01:36:08.936 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:08 smithi006 ceph-mon[28419]: from='osd.1 [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:36:08.936 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:08 smithi006 ceph-mon[28419]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T01:36:08.936 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:08.936 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:08 smithi006 ceph-mon[28419]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:36:08.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:08 smithi029 ceph-mon[32775]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:08.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:08 smithi029 ceph-mon[32775]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T01:36:08.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:08 smithi029 ceph-mon[32775]: from='osd.1 [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:36:08.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:08 smithi029 ceph-mon[32775]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T01:36:08.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:08.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:08 smithi029 ceph-mon[32775]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:36:09.310 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T01:36:09.310 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T01:36:09.310 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-26T01:36:09.310 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-26T01:36:09.310 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148759 s, 705 MB/s 2024-04-26T01:36:09.310 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-26T01:36:09.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:09 smithi188 ceph-mon[32751]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-26T01:36:09.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:09 smithi188 ceph-mon[32751]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T01:36:09.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:09.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:09.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:09 smithi188 ceph-mon[32751]: from='osd.1 ' entity='osd.1' 2024-04-26T01:36:09.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:09.897 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:09 smithi029 ceph-mon[32775]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-26T01:36:09.897 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:09 smithi029 ceph-mon[32775]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T01:36:09.897 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:09.897 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:09.897 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:09 smithi029 ceph-mon[32775]: from='osd.1 ' entity='osd.1' 2024-04-26T01:36:09.897 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:09.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 01:36:09 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1[37375]: 2024-04-26T01:36:09.457+0000 7f38ddfb7700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T01:36:09.898 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 01:36:09 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1[37375]: 2024-04-26T01:36:09.461+0000 7f38d5da2700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:36:10.004 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_4 2024-04-26T01:36:10.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:09 smithi006 ceph-mon[28419]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-26T01:36:10.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:09 smithi006 ceph-mon[28419]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T01:36:10.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:10.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:10.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:09 smithi006 ceph-mon[28419]: from='osd.1 ' entity='osd.1' 2024-04-26T01:36:10.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:10.241 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:36:10.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: purged_snaps scrub starts 2024-04-26T01:36:10.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: purged_snaps scrub ok 2024-04-26T01:36:10.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:10.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:10.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:10.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:10.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:10.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:10.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:10.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:10.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: osd.1 [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116] boot 2024-04-26T01:36:10.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T01:36:10.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:10 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:10.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: purged_snaps scrub starts 2024-04-26T01:36:10.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: purged_snaps scrub ok 2024-04-26T01:36:10.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:10.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:10.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:10.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:10.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:10.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:10.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:10.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:10.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: osd.1 [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116] boot 2024-04-26T01:36:10.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T01:36:10.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:10 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: purged_snaps scrub starts 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: purged_snaps scrub ok 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: osd.1 [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116] boot 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T01:36:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:10 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:36:11.623 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.625+0000 7f1af2d11700 1 -- 172.21.15.188:0/3007571957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aec103cc0 msgr2=0x7f1aec104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:11.624 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.625+0000 7f1af2d11700 1 --2- 172.21.15.188:0/3007571957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aec103cc0 0x7f1aec104120 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1adc009990 tx=0x7f1adc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:36:11.624 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.625+0000 7f1af2d11700 1 -- 172.21.15.188:0/3007571957 shutdown_connections 2024-04-26T01:36:11.624 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.625+0000 7f1af2d11700 1 --2- 172.21.15.188:0/3007571957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aec104660 0x7f1aec10af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:11.624 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.625+0000 7f1af2d11700 1 --2- 172.21.15.188:0/3007571957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec102ac0 0x7f1aec102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:11.624 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.625+0000 7f1af2d11700 1 --2- 172.21.15.188:0/3007571957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aec103cc0 0x7f1aec104120 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:11.624 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.625+0000 7f1af2d11700 1 -- 172.21.15.188:0/3007571957 >> 172.21.15.188:0/3007571957 conn(0x7f1aec0fe060 msgr2=0x7f1aec1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:11.624 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.625+0000 7f1af2d11700 1 -- 172.21.15.188:0/3007571957 shutdown_connections 2024-04-26T01:36:11.624 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.625+0000 7f1af2d11700 1 -- 172.21.15.188:0/3007571957 wait complete. 2024-04-26T01:36:11.624 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.625+0000 7f1af2d11700 1 Processor -- start 2024-04-26T01:36:11.624 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1af2d11700 1 -- start start 2024-04-26T01:36:11.624 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1af2d11700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec102ac0 0x7f1aec198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1af2d11700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aec103cc0 0x7f1aec1991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1af2d11700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aec104660 0x7f1aec1a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1af2d11700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec10e650 con 0x7f1aec104660 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1af2d11700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec10e4d0 con 0x7f1aec102ac0 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1af2d11700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec10e7d0 con 0x7f1aec103cc0 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1ae3fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aec103cc0 0x7f1aec1991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1ae3fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aec103cc0 0x7f1aec1991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:36394/0 (socket says 172.21.15.188:36394) 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1ae3fff700 1 -- 172.21.15.188:0/653896672 learned_addr learned my addr 172.21.15.188:0/653896672 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1af0aad700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec102ac0 0x7f1aec198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1ae3fff700 1 -- 172.21.15.188:0/653896672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec102ac0 msgr2=0x7f1aec198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1ae3fff700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec102ac0 0x7f1aec198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1ae3fff700 1 -- 172.21.15.188:0/653896672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aec104660 msgr2=0x7f1aec1a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1ae3fff700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aec104660 0x7f1aec1a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:11.625 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1ae3fff700 1 -- 172.21.15.188:0/653896672 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aec1a0910 con 0x7f1aec103cc0 2024-04-26T01:36:11.626 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1ae3fff700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aec103cc0 0x7f1aec1991d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0036f0 tx=0x7f1adc0039f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:11.626 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1ae1ffb700 1 -- 172.21.15.188:0/653896672 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1adc027070 con 0x7f1aec103cc0 2024-04-26T01:36:11.626 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1ae1ffb700 1 -- 172.21.15.188:0/653896672 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1adc003f60 con 0x7f1aec103cc0 2024-04-26T01:36:11.626 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1af2d11700 1 -- 172.21.15.188:0/653896672 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aec1a0ba0 con 0x7f1aec103cc0 2024-04-26T01:36:11.626 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.626+0000 7f1ae1ffb700 1 -- 172.21.15.188:0/653896672 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1adc022e60 con 0x7f1aec103cc0 2024-04-26T01:36:11.626 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.627+0000 7f1af2d11700 1 -- 172.21.15.188:0/653896672 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aec1a1070 con 0x7f1aec103cc0 2024-04-26T01:36:11.627 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.628+0000 7f1af2d11700 1 -- 172.21.15.188:0/653896672 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aec02df80 con 0x7f1aec103cc0 2024-04-26T01:36:11.627 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.629+0000 7f1ae1ffb700 1 -- 172.21.15.188:0/653896672 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7f1adc02a070 con 0x7f1aec103cc0 2024-04-26T01:36:11.627 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.630+0000 7f1ae1ffb700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ad40a3bb0 0x7f1ad40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:11.627 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.630+0000 7f1ae1ffb700 1 -- 172.21.15.188:0/653896672 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7f1adc0d0130 con 0x7f1aec103cc0 2024-04-26T01:36:11.627 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.630+0000 7f1af0aad700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ad40a3bb0 0x7f1ad40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:11.627 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.631+0000 7f1af0aad700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ad40a3bb0 0x7f1ad40a6070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1aec199b70 tx=0x7f1ae80094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:11.630 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.634+0000 7f1ae1ffb700 1 -- 172.21.15.188:0/653896672 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1adc013070 con 0x7f1aec103cc0 2024-04-26T01:36:11.836 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:11.840+0000 7f1af2d11700 1 -- 172.21.15.188:0/653896672 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1aec062e80 con 0x7f1ad40a3bb0 2024-04-26T01:36:11.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:11 smithi188 ceph-mon[32751]: Detected new or changed devices on smithi029 2024-04-26T01:36:11.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:11 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi029 to 17093M 2024-04-26T01:36:11.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:11 smithi029 ceph-mon[32775]: Detected new or changed devices on smithi029 2024-04-26T01:36:11.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:11 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi029 to 17093M 2024-04-26T01:36:12.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:11 smithi006 ceph-mon[28419]: Detected new or changed devices on smithi029 2024-04-26T01:36:12.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:11 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi029 to 17093M 2024-04-26T01:36:12.764 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:12 smithi188 ceph-mon[32751]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:12.764 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:12 smithi188 ceph-mon[32751]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T01:36:12.765 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:12 smithi188 ceph-mon[32751]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:36:12.765 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:12 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:36:12.765 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:12 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:36:12.765 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:12 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:12.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:12 smithi029 ceph-mon[32775]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:12.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:12 smithi029 ceph-mon[32775]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T01:36:12.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:12 smithi029 ceph-mon[32775]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:36:12.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:12 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:36:12.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:12 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:36:12.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:12 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:13.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:12 smithi006 ceph-mon[28419]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:13.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:12 smithi006 ceph-mon[28419]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T01:36:13.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:12 smithi006 ceph-mon[28419]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:36:13.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:12 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:36:13.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:12 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:36:13.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:12 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:14.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:14 smithi029 ceph-mon[32775]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:14.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:14 smithi029 ceph-mon[32775]: from='client.? 172.21.15.188:0/1302338253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77e0c294-c48a-4318-938b-4604a533a2d6"}]: dispatch 2024-04-26T01:36:14.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:14 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77e0c294-c48a-4318-938b-4604a533a2d6"}]: dispatch 2024-04-26T01:36:14.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:14 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77e0c294-c48a-4318-938b-4604a533a2d6"}]': finished 2024-04-26T01:36:14.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:14 smithi029 ceph-mon[32775]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T01:36:14.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:14 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:15.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:14 smithi006 ceph-mon[28419]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:15.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:14 smithi006 ceph-mon[28419]: from='client.? 172.21.15.188:0/1302338253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77e0c294-c48a-4318-938b-4604a533a2d6"}]: dispatch 2024-04-26T01:36:15.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:14 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77e0c294-c48a-4318-938b-4604a533a2d6"}]: dispatch 2024-04-26T01:36:15.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:14 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77e0c294-c48a-4318-938b-4604a533a2d6"}]': finished 2024-04-26T01:36:15.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:14 smithi006 ceph-mon[28419]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T01:36:15.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:14 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:15.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:14 smithi188 ceph-mon[32751]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:15.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:14 smithi188 ceph-mon[32751]: from='client.? 172.21.15.188:0/1302338253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77e0c294-c48a-4318-938b-4604a533a2d6"}]: dispatch 2024-04-26T01:36:15.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:14 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77e0c294-c48a-4318-938b-4604a533a2d6"}]: dispatch 2024-04-26T01:36:15.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:14 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77e0c294-c48a-4318-938b-4604a533a2d6"}]': finished 2024-04-26T01:36:15.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:14 smithi188 ceph-mon[32751]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T01:36:15.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:14 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:15.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:15 smithi029 ceph-mon[32775]: from='client.? 172.21.15.188:0/3554833985' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:36:16.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:15 smithi006 ceph-mon[28419]: from='client.? 172.21.15.188:0/3554833985' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:36:16.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:15 smithi188 ceph-mon[32751]: from='client.? 172.21.15.188:0/3554833985' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:36:17.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:16 smithi006 ceph-mon[28419]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:17.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:16 smithi188 ceph-mon[32751]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:17.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:16 smithi029 ceph-mon[32775]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:18.901 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:18 smithi188 ceph-mon[32751]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:19.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:18 smithi006 ceph-mon[28419]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:19.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:18 smithi029 ceph-mon[32775]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:20.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:20 smithi006 ceph-mon[28419]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:20.957 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:20 smithi029 ceph-mon[32775]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:21.034 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:20 smithi188 ceph-mon[32751]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:21.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:21 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T01:36:21.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:21 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:22.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T01:36:22.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:22.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:21 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T01:36:22.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:21 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:22.795 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:22 smithi188 ceph-mon[32751]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:22.795 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:22 smithi188 ceph-mon[32751]: Deploying daemon osd.2 on smithi188 2024-04-26T01:36:23.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:22 smithi006 ceph-mon[28419]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:23.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:22 smithi006 ceph-mon[28419]: Deploying daemon osd.2 on smithi188 2024-04-26T01:36:23.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:22 smithi029 ceph-mon[32775]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:23.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:22 smithi029 ceph-mon[32775]: Deploying daemon osd.2 on smithi188 2024-04-26T01:36:24.869 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:24 smithi188 ceph-mon[32751]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:25.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:24 smithi006 ceph-mon[28419]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:25.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:24 smithi029 ceph-mon[32775]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:26 smithi188 ceph-mon[32751]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:27.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:26 smithi006 ceph-mon[28419]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:27.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:26 smithi029 ceph-mon[32775]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:28.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:27 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:28.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:27 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:28.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:27 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:28.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:27 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:28.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:27 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:28.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:27 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:28.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:27 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:28.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:27 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:28.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:27 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:29.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:28 smithi188 ceph-mon[32751]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:29.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:28 smithi029 ceph-mon[32775]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:29.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:28 smithi006 ceph-mon[28419]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:30.272 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 2 on host 'smithi188' 2024-04-26T01:36:30.272 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.276+0000 7f1ae1ffb700 1 -- 172.21.15.188:0/653896672 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1aec062e80 con 0x7f1ad40a3bb0 2024-04-26T01:36:30.274 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.279+0000 7f1af2d11700 1 -- 172.21.15.188:0/653896672 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ad40a3bb0 msgr2=0x7f1ad40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:30.274 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.279+0000 7f1af2d11700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ad40a3bb0 0x7f1ad40a6070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1aec199b70 tx=0x7f1ae80094f0 comp rx=0 tx=0).stop 2024-04-26T01:36:30.274 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.279+0000 7f1af2d11700 1 -- 172.21.15.188:0/653896672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aec103cc0 msgr2=0x7f1aec1991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:30.275 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.279+0000 7f1af2d11700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aec103cc0 0x7f1aec1991d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0036f0 tx=0x7f1adc0039f0 comp rx=0 tx=0).stop 2024-04-26T01:36:30.275 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.279+0000 7f1af2d11700 1 -- 172.21.15.188:0/653896672 shutdown_connections 2024-04-26T01:36:30.275 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.279+0000 7f1af2d11700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1aec104660 0x7f1aec1a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:30.275 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.279+0000 7f1af2d11700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec102ac0 0x7f1aec198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:30.275 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.279+0000 7f1af2d11700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1ad40a3bb0 0x7f1ad40a6070 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:30.275 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.279+0000 7f1af2d11700 1 --2- 172.21.15.188:0/653896672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aec103cc0 0x7f1aec1991d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:30.275 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.279+0000 7f1af2d11700 1 -- 172.21.15.188:0/653896672 >> 172.21.15.188:0/653896672 conn(0x7f1aec0fe060 msgr2=0x7f1aec107040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:30.275 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.279+0000 7f1af2d11700 1 -- 172.21.15.188:0/653896672 shutdown_connections 2024-04-26T01:36:30.275 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:36:30.279+0000 7f1af2d11700 1 -- 172.21.15.188:0/653896672 wait complete. 2024-04-26T01:36:30.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:30 smithi188 ceph-mon[32751]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:30.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:30.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:30.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:30.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:30.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:30 smithi188 ceph-mon[32751]: from='osd.2 [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:36:30.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:30 smithi188 ceph-mon[32751]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:36:30.949 DEBUG:teuthology.orchestra.run.smithi188:osd.2> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T01:36:30.951 INFO:tasks.cephadm:Deploying osd.3 on smithi006 with /dev/vg_nvme/lv_3... 2024-04-26T01:36:30.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T01:36:31.036 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:30 smithi006 ceph-mon[28419]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:31.036 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:31.036 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:31.036 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:31.036 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:31.036 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:30 smithi006 ceph-mon[28419]: from='osd.2 [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:36:31.036 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:30 smithi006 ceph-mon[28419]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:36:31.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:30 smithi029 ceph-mon[32775]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:31.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:31.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:31.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:31.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:31.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:30 smithi029 ceph-mon[32775]: from='osd.2 [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:36:31.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:30 smithi029 ceph-mon[32775]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:36:31.400 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:32.505 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:32 smithi006 ceph-mon[28419]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:32.505 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:32 smithi006 ceph-mon[28419]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T01:36:32.505 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:32 smithi006 ceph-mon[28419]: from='osd.2 [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:36:32.506 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:32 smithi006 ceph-mon[28419]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T01:36:32.506 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:32 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:32.506 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:32 smithi006 ceph-mon[28419]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:36:32.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:32 smithi188 ceph-mon[32751]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:32.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:32 smithi188 ceph-mon[32751]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T01:36:32.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:32 smithi188 ceph-mon[32751]: from='osd.2 [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:36:32.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:32 smithi188 ceph-mon[32751]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T01:36:32.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:32 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:32.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:32 smithi188 ceph-mon[32751]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:36:32.639 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 01:36:32 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2[38071]: 2024-04-26T01:36:32.325+0000 7f528094a700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T01:36:32.639 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 01:36:32 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2[38071]: 2024-04-26T01:36:32.330+0000 7f527c73d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:36:32.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:32 smithi029 ceph-mon[32775]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:32.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:32 smithi029 ceph-mon[32775]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T01:36:32.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:32 smithi029 ceph-mon[32775]: from='osd.2 [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:36:32.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:32 smithi029 ceph-mon[32775]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T01:36:32.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:32 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:32.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:32 smithi029 ceph-mon[32775]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:36:33.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:33 smithi188 ceph-mon[32751]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-26T01:36:33.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:33 smithi188 ceph-mon[32751]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T01:36:33.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:33.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:33.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:33.397 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:33 smithi006 ceph-mon[28419]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-26T01:36:33.397 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:33 smithi006 ceph-mon[28419]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T01:36:33.397 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:33.397 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:33.397 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:33.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:33 smithi029 ceph-mon[32775]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-26T01:36:33.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:33 smithi029 ceph-mon[32775]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T01:36:33.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:33.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:33.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:34.010 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T01:36:34.010 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T01:36:34.010 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-04-26T01:36:34.010 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-04-26T01:36:34.011 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0378202 s, 277 MB/s 2024-04-26T01:36:34.011 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-04-26T01:36:34.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: purged_snaps scrub starts 2024-04-26T01:36:34.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: purged_snaps scrub ok 2024-04-26T01:36:34.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:34.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: osd.2 [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672] boot 2024-04-26T01:36:34.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T01:36:34.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: Detected new or changed devices on smithi188 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi188 to 17088M 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: purged_snaps scrub starts 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: purged_snaps scrub ok 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: osd.2 [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672] boot 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: Detected new or changed devices on smithi188 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi188 to 17088M 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:34.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:34.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:34.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:34.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.696 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: purged_snaps scrub starts 2024-04-26T01:36:34.696 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: purged_snaps scrub ok 2024-04-26T01:36:34.696 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:36:34.696 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: osd.2 [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672] boot 2024-04-26T01:36:34.696 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T01:36:34.696 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:36:34.696 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: Detected new or changed devices on smithi188 2024-04-26T01:36:34.696 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.696 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.696 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:34.696 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi188 to 17088M 2024-04-26T01:36:34.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:34.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:34.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:34.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:34.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:34.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:34.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi006:vg_nvme/lv_3 2024-04-26T01:36:35.214 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:35.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:35 smithi188 ceph-mon[32751]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T01:36:35.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:36:35.674 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:35 smithi006 ceph-mon[28419]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T01:36:35.674 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:36:35.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:35 smithi029 ceph-mon[32775]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T01:36:35.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:36:36.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:36 smithi188 ceph-mon[32751]: pgmap v93: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:36.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:36 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T01:36:36.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:36 smithi188 ceph-mon[32751]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T01:36:36.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:36 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:36:36.652 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:36 smithi006 ceph-mon[28419]: pgmap v93: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:36.653 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:36 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T01:36:36.653 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:36 smithi006 ceph-mon[28419]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T01:36:36.653 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:36 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:36:36.653 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47896]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:36:36.653 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47896]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:36:36.653 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47896]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:36:36.653 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47896]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:36:36.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:36 smithi029 ceph-mon[32775]: pgmap v93: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:36.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:36 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T01:36:36.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:36 smithi029 ceph-mon[32775]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T01:36:36.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:36 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:36:36.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47912]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:36:36.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47912]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:36:36.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47912]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:36:36.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47912]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:36:37.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 01:36:36 smithi029 sudo[41519]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:36:37.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 01:36:36 smithi029 sudo[41519]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:36:37.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 01:36:36 smithi029 sudo[41519]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:36:37.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 01:36:36 smithi029 sudo[41519]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:36:37.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 01:36:36 smithi029 sudo[41522]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:36:37.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 01:36:36 smithi029 sudo[41522]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:36:37.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 01:36:36 smithi029 sudo[41522]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:36:37.033 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 01:36:36 smithi029 sudo[41522]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:36:37.138 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 01:36:36 smithi188 sudo[41823]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:36:37.138 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 01:36:36 smithi188 sudo[41823]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:36:37.138 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 01:36:36 smithi188 sudo[41823]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:36:37.138 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 01:36:36 smithi188 sudo[41823]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:36:37.138 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 01:36:36 smithi188 sudo[41826]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:36:37.138 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 01:36:36 smithi188 sudo[41826]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:36:37.138 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 01:36:36 smithi188 sudo[41826]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:36:37.138 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 01:36:36 smithi188 sudo[41826]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:36:37.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:36 smithi188 sudo[41829]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:36:37.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:36 smithi188 sudo[41829]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:36:37.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:36 smithi188 sudo[41829]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:36:37.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 sudo[41829]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:36:37.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 sudo[41832]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:36:37.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 sudo[41832]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:36:37.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 sudo[41832]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:36:37.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 sudo[41832]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:36:37.179 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47916]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:36:37.179 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47916]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:36:37.179 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47916]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:36:37.180 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47916]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:36:37.180 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47923]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:36:37.180 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47923]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:36:37.180 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47923]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:36:37.180 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:36 smithi006 sudo[47923]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:36:37.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.249+0000 7f9630c00700 1 -- 172.21.15.6:0/2644645377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f962c104cf0 msgr2=0x7f962c105110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:37.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.249+0000 7f9630c00700 1 --2- 172.21.15.6:0/2644645377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f962c104cf0 0x7f962c105110 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f961c009920 tx=0x7f961c009c30 comp rx=0 tx=0).stop 2024-04-26T01:36:37.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.249+0000 7f9630c00700 1 -- 172.21.15.6:0/2644645377 shutdown_connections 2024-04-26T01:36:37.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.249+0000 7f9630c00700 1 --2- 172.21.15.6:0/2644645377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f962c106890 0x7f962c10d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:37.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.249+0000 7f9630c00700 1 --2- 172.21.15.6:0/2644645377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f962c104cf0 0x7f962c105110 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:37.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.249+0000 7f9630c00700 1 --2- 172.21.15.6:0/2644645377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f962c105ef0 0x7f962c106350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:37.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.249+0000 7f9630c00700 1 -- 172.21.15.6:0/2644645377 >> 172.21.15.6:0/2644645377 conn(0x7f962c100290 msgr2=0x7f962c1026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:37.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.249+0000 7f9630c00700 1 -- 172.21.15.6:0/2644645377 shutdown_connections 2024-04-26T01:36:37.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.249+0000 7f9630c00700 1 -- 172.21.15.6:0/2644645377 wait complete. 2024-04-26T01:36:37.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.250+0000 7f9630c00700 1 Processor -- start 2024-04-26T01:36:37.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.251+0000 7f9630c00700 1 -- start start 2024-04-26T01:36:37.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.251+0000 7f9630c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f962c104cf0 0x7f962c19aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:37.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.251+0000 7f9630c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f962c105ef0 0x7f962c19b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:37.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.251+0000 7f9630c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f962c106890 0x7f962c1a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:37.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.251+0000 7f9630c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f962c1107b0 con 0x7f962c105ef0 2024-04-26T01:36:37.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.251+0000 7f9630c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f962c110630 con 0x7f962c104cf0 2024-04-26T01:36:37.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.251+0000 7f9630c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f962c110930 con 0x7f962c106890 2024-04-26T01:36:37.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.251+0000 7f9629e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f962c104cf0 0x7f962c19aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:37.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.251+0000 7f9629e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f962c104cf0 0x7f962c19aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:38254/0 (socket says 172.21.15.6:38254) 2024-04-26T01:36:37.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.251+0000 7f9629e00700 1 -- 172.21.15.6:0/1815934263 learned_addr learned my addr 172.21.15.6:0/1815934263 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:36:37.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.251+0000 7f9629400700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f962c105ef0 0x7f962c19b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:37.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.251+0000 7f962a800700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f962c106890 0x7f962c1a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:37.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.252+0000 7f9629e00700 1 -- 172.21.15.6:0/1815934263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f962c106890 msgr2=0x7f962c1a24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:37.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.252+0000 7f9629e00700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f962c106890 0x7f962c1a24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:37.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.252+0000 7f9629e00700 1 -- 172.21.15.6:0/1815934263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f962c105ef0 msgr2=0x7f962c19b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:37.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.252+0000 7f9629e00700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f962c105ef0 0x7f962c19b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:37.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.252+0000 7f9629e00700 1 -- 172.21.15.6:0/1815934263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962c1a2b40 con 0x7f962c104cf0 2024-04-26T01:36:37.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.252+0000 7f9629e00700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f962c104cf0 0x7f962c19aee0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f961c009230 tx=0x7f961c005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:37.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.252+0000 7f9622a00700 1 -- 172.21.15.6:0/1815934263 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c027070 con 0x7f962c104cf0 2024-04-26T01:36:37.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.252+0000 7f9630c00700 1 -- 172.21.15.6:0/1815934263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f962c1a2dd0 con 0x7f962c104cf0 2024-04-26T01:36:37.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.253+0000 7f9630c00700 1 -- 172.21.15.6:0/1815934263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f962c1a3170 con 0x7f962c104cf0 2024-04-26T01:36:37.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.253+0000 7f9622a00700 1 -- 172.21.15.6:0/1815934263 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f961c009400 con 0x7f962c104cf0 2024-04-26T01:36:37.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.253+0000 7f9622a00700 1 -- 172.21.15.6:0/1815934263 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c004da0 con 0x7f962c104cf0 2024-04-26T01:36:37.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.254+0000 7f9630c00700 1 -- 172.21.15.6:0/1815934263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f960c005360 con 0x7f962c104cf0 2024-04-26T01:36:37.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.256+0000 7f9622a00700 1 -- 172.21.15.6:0/1815934263 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7f961c02a070 con 0x7f962c104cf0 2024-04-26T01:36:37.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.257+0000 7f9622a00700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f96100a3cb0 0x7f96100a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:37.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.258+0000 7f9622a00700 1 -- 172.21.15.6:0/1815934263 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f961c0cf8d0 con 0x7f962c104cf0 2024-04-26T01:36:37.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.258+0000 7f9629400700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f96100a3cb0 0x7f96100a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:37.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.258+0000 7f9629400700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f96100a3cb0 0x7f96100a6170 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9618009750 tx=0x7f9618009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:37.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.260+0000 7f9622a00700 1 -- 172.21.15.6:0/1815934263 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f961c05fff0 con 0x7f962c104cf0 2024-04-26T01:36:37.385 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 sudo[41525]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:36:37.386 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 sudo[41525]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:36:37.386 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 sudo[41525]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:36:37.386 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 sudo[41525]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:36:37.386 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 sudo[41528]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:36:37.386 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 sudo[41528]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:36:37.386 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 sudo[41528]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:36:37.386 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 sudo[41528]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:36:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.506+0000 7f9630c00700 1 -- 172.21.15.6:0/1815934263 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f960c000bf0 con 0x7f96100a3cb0 2024-04-26T01:36:37.503 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T01:36:37.503 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T01:36:37.503 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:37.503 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:37.503 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:37.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:36:37.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:36:37.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:36:37.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:36:37.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:36:37.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:36:37.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:36:37.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:36:37.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:36:37.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:36:37.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:36:37.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:36:37.505 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:36:37.505 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:36:37.505 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:37 smithi006 ceph-mon[28419]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:36:37.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T01:36:37.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T01:36:37.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:37.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:37.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:37.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:36:37.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:36:37.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:36:37.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:36:37.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:36:37.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:36:37.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:36:37.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:36:37.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:36:37.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:36:37.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:36:37.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:36:37.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:36:37.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:36:37.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:37 smithi188 ceph-mon[32751]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:36:37.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T01:36:37.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T01:36:37.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:37.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:37.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:36:37.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:36:37.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:36:37.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:36:37.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:36:37.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:36:37.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:36:37.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:36:37.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:36:37.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:36:37.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:36:37.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:36:37.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:36:37.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:36:37.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:36:37.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:37 smithi029 ceph-mon[32775]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:36:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:36:37.891+0000 7f9622a00700 1 -- 172.21.15.6:0/1815934263 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f961c0997e0 con 0x7f962c104cf0 2024-04-26T01:36:38.661 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:38 smithi006 ceph-mon[28419]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:38.661 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:38 smithi006 ceph-mon[28419]: osdmap e22: 3 total, 3 up, 3 in 2024-04-26T01:36:38.661 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:38 smithi006 ceph-mon[28419]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:36:38.661 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:38 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:36:38.662 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:38 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:36:38.662 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:38 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:38.662 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:38 smithi006 ceph-mon[28419]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T01:36:38.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:38 smithi029 ceph-mon[32775]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:38.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:38 smithi029 ceph-mon[32775]: osdmap e22: 3 total, 3 up, 3 in 2024-04-26T01:36:38.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:38 smithi029 ceph-mon[32775]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:36:38.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:38 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:36:38.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:38 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:36:38.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:38 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:38.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:38 smithi029 ceph-mon[32775]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T01:36:38.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:38 smithi188 ceph-mon[32751]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:38.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:38 smithi188 ceph-mon[32751]: osdmap e22: 3 total, 3 up, 3 in 2024-04-26T01:36:38.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:38 smithi188 ceph-mon[32751]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:36:38.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:38 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:36:38.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:38 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:36:38.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:38 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:38.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:38 smithi188 ceph-mon[32751]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T01:36:40.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:40 smithi006 ceph-mon[28419]: pgmap v98: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:40.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:40 smithi029 ceph-mon[32775]: pgmap v98: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:40.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:40 smithi188 ceph-mon[32751]: pgmap v98: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:41.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:41 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3514583662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6393dac5-996c-47af-9687-0f1393ecc483"}]: dispatch 2024-04-26T01:36:41.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:41 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6393dac5-996c-47af-9687-0f1393ecc483"}]: dispatch 2024-04-26T01:36:41.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:41 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6393dac5-996c-47af-9687-0f1393ecc483"}]': finished 2024-04-26T01:36:41.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:41 smithi029 ceph-mon[32775]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T01:36:41.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:41 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:41.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:41 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3514583662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6393dac5-996c-47af-9687-0f1393ecc483"}]: dispatch 2024-04-26T01:36:41.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:41 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6393dac5-996c-47af-9687-0f1393ecc483"}]: dispatch 2024-04-26T01:36:41.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:41 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6393dac5-996c-47af-9687-0f1393ecc483"}]': finished 2024-04-26T01:36:41.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:41 smithi006 ceph-mon[28419]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T01:36:41.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:41 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:41.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:41 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3514583662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6393dac5-996c-47af-9687-0f1393ecc483"}]: dispatch 2024-04-26T01:36:41.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:41 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6393dac5-996c-47af-9687-0f1393ecc483"}]: dispatch 2024-04-26T01:36:41.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:41 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6393dac5-996c-47af-9687-0f1393ecc483"}]': finished 2024-04-26T01:36:41.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:41 smithi188 ceph-mon[32751]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T01:36:41.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:42.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:42 smithi006 ceph-mon[28419]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:42.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:42 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/4135515396' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:36:42.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:42 smithi188 ceph-mon[32751]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:42.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:42 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/4135515396' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:36:42.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:42 smithi029 ceph-mon[32775]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:42.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:42 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/4135515396' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:36:44.798 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:44 smithi006 ceph-mon[28419]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:44.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:44 smithi188 ceph-mon[32751]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:44.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:44 smithi029 ceph-mon[32775]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:46.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:46 smithi006 ceph-mon[28419]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:46.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:46 smithi188 ceph-mon[32751]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:46.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:46 smithi029 ceph-mon[32775]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:48.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:48 smithi006 ceph-mon[28419]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:48.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:48 smithi188 ceph-mon[32751]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:48.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:48 smithi029 ceph-mon[32775]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:50.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:50 smithi006 ceph-mon[28419]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:50.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T01:36:50.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:50.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:50 smithi188 ceph-mon[32751]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:50.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T01:36:50.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:50.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:50 smithi029 ceph-mon[32775]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:50.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T01:36:50.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:51.749 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:51 smithi006 ceph-mon[28419]: Deploying daemon osd.3 on smithi006 2024-04-26T01:36:51.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:51 smithi188 ceph-mon[32751]: Deploying daemon osd.3 on smithi006 2024-04-26T01:36:51.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:51 smithi029 ceph-mon[32775]: Deploying daemon osd.3 on smithi006 2024-04-26T01:36:52.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:52 smithi006 ceph-mon[28419]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:52.789 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:52 smithi188 ceph-mon[32751]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:52.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:52 smithi029 ceph-mon[32775]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:54.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:54 smithi006 ceph-mon[28419]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:54.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:54 smithi188 ceph-mon[32751]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:54.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:54 smithi029 ceph-mon[32775]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:56.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:56 smithi006 ceph-mon[28419]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:56.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:56 smithi188 ceph-mon[32751]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:56.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:56 smithi029 ceph-mon[32775]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:58.553 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:58 smithi006 ceph-mon[28419]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:58.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:58 smithi188 ceph-mon[32751]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:58.938 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:58 smithi029 ceph-mon[32775]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:00.055 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:00.055 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:00.055 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:36:59 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:00.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:36:59 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:00.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:00.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:00.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:36:59 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:00.830 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:00 smithi006 ceph-mon[28419]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:01.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:00 smithi188 ceph-mon[32751]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:01.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:00 smithi029 ceph-mon[32775]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:03.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:02 smithi006 ceph-mon[28419]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:03.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:03.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:03.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:03.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:03.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:03.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:02 smithi006 ceph-mon[28419]: from='osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:37:03.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:02 smithi188 ceph-mon[32751]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:03.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:03.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:03.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:03.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:03.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:03.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:02 smithi188 ceph-mon[32751]: from='osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:37:03.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:02 smithi029 ceph-mon[32775]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:03.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:03.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:03.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:03.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:03.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:03.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:02 smithi029 ceph-mon[32775]: from='osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:37:03.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.555+0000 7f9622a00700 1 -- 172.21.15.6:0/1815934263 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f960c000bf0 con 0x7f96100a3cb0 2024-04-26T01:37:03.552 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 3 on host 'smithi006' 2024-04-26T01:37:03.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.559+0000 7f9630c00700 1 -- 172.21.15.6:0/1815934263 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f96100a3cb0 msgr2=0x7f96100a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:03.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.559+0000 7f9630c00700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f96100a3cb0 0x7f96100a6170 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9618009750 tx=0x7f9618009310 comp rx=0 tx=0).stop 2024-04-26T01:37:03.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.559+0000 7f9630c00700 1 -- 172.21.15.6:0/1815934263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f962c104cf0 msgr2=0x7f962c19aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:03.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.559+0000 7f9630c00700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f962c104cf0 0x7f962c19aee0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f961c009230 tx=0x7f961c005270 comp rx=0 tx=0).stop 2024-04-26T01:37:03.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.559+0000 7f9630c00700 1 -- 172.21.15.6:0/1815934263 shutdown_connections 2024-04-26T01:37:03.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.559+0000 7f9630c00700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f96100a3cb0 0x7f96100a6170 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:03.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.559+0000 7f9630c00700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f962c106890 0x7f962c1a24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:03.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.559+0000 7f9630c00700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f962c104cf0 0x7f962c19aee0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:03.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.559+0000 7f9630c00700 1 --2- 172.21.15.6:0/1815934263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f962c105ef0 0x7f962c19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:03.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.559+0000 7f9630c00700 1 -- 172.21.15.6:0/1815934263 >> 172.21.15.6:0/1815934263 conn(0x7f962c100290 msgr2=0x7f962c109300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:03.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.560+0000 7f9630c00700 1 -- 172.21.15.6:0/1815934263 shutdown_connections 2024-04-26T01:37:03.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:37:03.560+0000 7f9630c00700 1 -- 172.21.15.6:0/1815934263 wait complete. 2024-04-26T01:37:04.356 DEBUG:teuthology.orchestra.run.smithi006:osd.3> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T01:37:04.360 INFO:tasks.cephadm:Deploying osd.4 on smithi029 with /dev/vg_nvme/lv_3... 2024-04-26T01:37:04.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T01:37:04.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:04 smithi006 ceph-mon[28419]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:04.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:04 smithi006 ceph-mon[28419]: from='osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T01:37:04.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:04 smithi006 ceph-mon[28419]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T01:37:04.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:04 smithi006 ceph-mon[28419]: from='osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:37:04.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:37:04.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:04.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:04.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:04.616 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:37:04.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:04 smithi188 ceph-mon[32751]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:04.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:04 smithi188 ceph-mon[32751]: from='osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T01:37:04.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:04 smithi188 ceph-mon[32751]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T01:37:04.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:04 smithi188 ceph-mon[32751]: from='osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:37:04.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:37:04.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:04.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:04.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:04.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:04 smithi029 ceph-mon[32775]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:04.688 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:04 smithi029 ceph-mon[32775]: from='osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T01:37:04.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:04 smithi029 ceph-mon[32775]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T01:37:04.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:04 smithi029 ceph-mon[32775]: from='osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:37:04.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:37:04.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:04.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:04.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:05.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:05 smithi188 ceph-mon[32751]: from='osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-04-26T01:37:05.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:05 smithi188 ceph-mon[32751]: osdmap e25: 4 total, 3 up, 4 in 2024-04-26T01:37:05.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:37:05.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:37:05.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:05 smithi188 ceph-mon[32751]: osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118] boot 2024-04-26T01:37:05.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:05 smithi188 ceph-mon[32751]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T01:37:05.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:37:05.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:05 smithi006 ceph-mon[28419]: from='osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-04-26T01:37:05.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:05 smithi006 ceph-mon[28419]: osdmap e25: 4 total, 3 up, 4 in 2024-04-26T01:37:05.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:37:05.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:37:05.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:05 smithi006 ceph-mon[28419]: osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118] boot 2024-04-26T01:37:05.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:05 smithi006 ceph-mon[28419]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T01:37:05.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:37:05.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:05 smithi029 ceph-mon[32775]: from='osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-04-26T01:37:05.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:05 smithi029 ceph-mon[32775]: osdmap e25: 4 total, 3 up, 4 in 2024-04-26T01:37:05.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:37:05.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:37:05.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:05 smithi029 ceph-mon[32775]: osd.3 [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118] boot 2024-04-26T01:37:05.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:05 smithi029 ceph-mon[32775]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T01:37:05.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:37:06.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:06 smithi006 ceph-mon[28419]: purged_snaps scrub starts 2024-04-26T01:37:06.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:06 smithi006 ceph-mon[28419]: purged_snaps scrub ok 2024-04-26T01:37:06.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:06 smithi006 ceph-mon[28419]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:06.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:06 smithi006 ceph-mon[28419]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T01:37:06.632 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:06 smithi029 ceph-mon[32775]: purged_snaps scrub starts 2024-04-26T01:37:06.632 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:06 smithi029 ceph-mon[32775]: purged_snaps scrub ok 2024-04-26T01:37:06.632 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:06 smithi029 ceph-mon[32775]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:06.632 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:06 smithi029 ceph-mon[32775]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T01:37:06.633 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T01:37:06.633 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T01:37:06.633 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-04-26T01:37:06.633 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-04-26T01:37:06.633 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00835692 s, 1.3 GB/s 2024-04-26T01:37:06.633 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-04-26T01:37:06.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:06 smithi188 ceph-mon[32751]: purged_snaps scrub starts 2024-04-26T01:37:06.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:06 smithi188 ceph-mon[32751]: purged_snaps scrub ok 2024-04-26T01:37:06.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:06 smithi188 ceph-mon[32751]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:37:06.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:06 smithi188 ceph-mon[32751]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T01:37:07.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_3 2024-04-26T01:37:07.532 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:37:08.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:07 smithi188 ceph-mon[32751]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T01:37:08.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:07 smithi029 ceph-mon[32775]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T01:37:08.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:07 smithi006 ceph-mon[28419]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T01:37:09.087 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:08 smithi029 ceph-mon[32775]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:09.087 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:08 smithi029 ceph-mon[32775]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T01:37:09.087 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:08 smithi029 ceph-mon[32775]: osdmap e30: 4 total, 4 up, 4 in 2024-04-26T01:37:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.131+0000 7fb18880b700 1 -- 172.21.15.29:0/3397028875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184102aa0 msgr2=0x7fb184102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.131+0000 7fb18880b700 1 --2- 172.21.15.29:0/3397028875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184102aa0 0x7fb184102ec0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb1740099c0 tx=0x7fb174009cd0 comp rx=0 tx=0).stop 2024-04-26T01:37:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.131+0000 7fb18880b700 1 -- 172.21.15.29:0/3397028875 shutdown_connections 2024-04-26T01:37:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.131+0000 7fb18880b700 1 --2- 172.21.15.29:0/3397028875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb184104640 0x7fb18410aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.131+0000 7fb18880b700 1 --2- 172.21.15.29:0/3397028875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184102aa0 0x7fb184102ec0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.131+0000 7fb18880b700 1 --2- 172.21.15.29:0/3397028875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb184103ca0 0x7fb184104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.131+0000 7fb18880b700 1 -- 172.21.15.29:0/3397028875 >> 172.21.15.29:0/3397028875 conn(0x7fb1840fe040 msgr2=0x7fb184100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.131+0000 7fb18880b700 1 -- 172.21.15.29:0/3397028875 shutdown_connections 2024-04-26T01:37:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.131+0000 7fb18880b700 1 -- 172.21.15.29:0/3397028875 wait complete. 2024-04-26T01:37:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.132+0000 7fb18880b700 1 Processor -- start 2024-04-26T01:37:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.132+0000 7fb18880b700 1 -- start start 2024-04-26T01:37:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.132+0000 7fb18880b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb184102aa0 0x7fb1840767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.132+0000 7fb18880b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb184103ca0 0x7fb184074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.132+0000 7fb18880b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184104640 0x7fb184075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.132+0000 7fb18880b700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb18410e610 con 0x7fb184103ca0 2024-04-26T01:37:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.132+0000 7fb18880b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb18410e490 con 0x7fb184104640 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.132+0000 7fb18880b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb18410e790 con 0x7fb184102aa0 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.132+0000 7fb182d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184104640 0x7fb184075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.132+0000 7fb182d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184104640 0x7fb184075370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46934/0 (socket says 172.21.15.29:46934) 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.132+0000 7fb182d9d700 1 -- 172.21.15.29:0/571502140 learned_addr learned my addr 172.21.15.29:0/571502140 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb18259c700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb184102aa0 0x7fb1840767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb182d9d700 1 -- 172.21.15.29:0/571502140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb184102aa0 msgr2=0x7fb1840767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb182d9d700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb184102aa0 0x7fb1840767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb182d9d700 1 -- 172.21.15.29:0/571502140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb184103ca0 msgr2=0x7fb184074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb181d9b700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb184103ca0 0x7fb184074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb182d9d700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb184103ca0 0x7fb184074e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb182d9d700 1 -- 172.21.15.29:0/571502140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1840758b0 con 0x7fb184104640 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb182d9d700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184104640 0x7fb184075370 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00b970 tx=0x7fb17c00bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb1737fe700 1 -- 172.21.15.29:0/571502140 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c019070 con 0x7fb184104640 2024-04-26T01:37:09.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb1737fe700 1 -- 172.21.15.29:0/571502140 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb17c00cde0 con 0x7fb184104640 2024-04-26T01:37:09.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb18880b700 1 -- 172.21.15.29:0/571502140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb184075ba0 con 0x7fb184104640 2024-04-26T01:37:09.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb1737fe700 1 -- 172.21.15.29:0/571502140 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c007370 con 0x7fb184104640 2024-04-26T01:37:09.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.133+0000 7fb18880b700 1 -- 172.21.15.29:0/571502140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1841a77f0 con 0x7fb184104640 2024-04-26T01:37:09.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.134+0000 7fb18880b700 1 -- 172.21.15.29:0/571502140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb184069730 con 0x7fb184104640 2024-04-26T01:37:09.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.135+0000 7fb1737fe700 1 -- 172.21.15.29:0/571502140 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fb17c01c070 con 0x7fb184104640 2024-04-26T01:37:09.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.136+0000 7fb1737fe700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb16c0a3c40 0x7fb16c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:09.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.136+0000 7fb1737fe700 1 -- 172.21.15.29:0/571502140 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 3669+0+0 (secure 0 0 0) 0x7fb17c0ce6d0 con 0x7fb184104640 2024-04-26T01:37:09.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.136+0000 7fb18259c700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb16c0a3c40 0x7fb16c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:09.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.138+0000 7fb18259c700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb16c0a3c40 0x7fb16c0a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb1740038c0 tx=0x7fb17400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:09.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.141+0000 7fb1737fe700 1 -- 172.21.15.29:0/571502140 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb17c09c2d0 con 0x7fb184104640 2024-04-26T01:37:09.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:08 smithi188 ceph-mon[32751]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:09.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:08 smithi188 ceph-mon[32751]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T01:37:09.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:08 smithi188 ceph-mon[32751]: osdmap e30: 4 total, 4 up, 4 in 2024-04-26T01:37:09.261 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:08 smithi006 ceph-mon[28419]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:09.261 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:08 smithi006 ceph-mon[28419]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T01:37:09.261 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:08 smithi006 ceph-mon[28419]: osdmap e30: 4 total, 4 up, 4 in 2024-04-26T01:37:09.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:09.359+0000 7fb18880b700 1 -- 172.21.15.29:0/571502140 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb184062e80 con 0x7fb16c0a3c40 2024-04-26T01:37:10.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:10.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:10.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:10.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:10.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:10.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:10.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:10.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:10.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:10.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:10.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:09 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:10.167 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:10.167 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:10.167 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:10.167 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:10.167 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:10.167 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:10.167 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:10.167 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:10.167 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:10.167 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:10.167 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:09 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:10.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:10.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:10.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:10.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:10.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:10.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:10.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:10.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:10.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:10.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:10.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:09 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:11.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:10 smithi029 ceph-mon[32775]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:11.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:10 smithi029 ceph-mon[32775]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:11.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:10 smithi029 ceph-mon[32775]: Detected new or changed devices on smithi006 2024-04-26T01:37:11.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:10 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi006 to 8547M 2024-04-26T01:37:11.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:10 smithi006 ceph-mon[28419]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:11.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:10 smithi006 ceph-mon[28419]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:11.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:10 smithi006 ceph-mon[28419]: Detected new or changed devices on smithi006 2024-04-26T01:37:11.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:10 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi006 to 8547M 2024-04-26T01:37:11.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:10 smithi188 ceph-mon[32751]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:11.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:10 smithi188 ceph-mon[32751]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:11.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:10 smithi188 ceph-mon[32751]: Detected new or changed devices on smithi006 2024-04-26T01:37:11.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:10 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi006 to 8547M 2024-04-26T01:37:13.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:12 smithi029 ceph-mon[32775]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-26T01:37:13.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:12 smithi029 ceph-mon[32775]: from='client.? 172.21.15.29:0/656960476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2c1a535-26a0-46c6-a6d0-c2529c786739"}]: dispatch 2024-04-26T01:37:13.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:12 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2c1a535-26a0-46c6-a6d0-c2529c786739"}]: dispatch 2024-04-26T01:37:13.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:12 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2c1a535-26a0-46c6-a6d0-c2529c786739"}]': finished 2024-04-26T01:37:13.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:12 smithi029 ceph-mon[32775]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T01:37:13.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:12 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:13.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:12 smithi029 ceph-mon[32775]: from='client.? 172.21.15.29:0/1347420287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:37:13.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:12 smithi006 ceph-mon[28419]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-26T01:37:13.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:12 smithi006 ceph-mon[28419]: from='client.? 172.21.15.29:0/656960476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2c1a535-26a0-46c6-a6d0-c2529c786739"}]: dispatch 2024-04-26T01:37:13.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:12 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2c1a535-26a0-46c6-a6d0-c2529c786739"}]: dispatch 2024-04-26T01:37:13.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:12 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2c1a535-26a0-46c6-a6d0-c2529c786739"}]': finished 2024-04-26T01:37:13.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:12 smithi006 ceph-mon[28419]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T01:37:13.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:12 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:13.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:12 smithi006 ceph-mon[28419]: from='client.? 172.21.15.29:0/1347420287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:37:13.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:12 smithi188 ceph-mon[32751]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-26T01:37:13.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:12 smithi188 ceph-mon[32751]: from='client.? 172.21.15.29:0/656960476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2c1a535-26a0-46c6-a6d0-c2529c786739"}]: dispatch 2024-04-26T01:37:13.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:12 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2c1a535-26a0-46c6-a6d0-c2529c786739"}]: dispatch 2024-04-26T01:37:13.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:12 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2c1a535-26a0-46c6-a6d0-c2529c786739"}]': finished 2024-04-26T01:37:13.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:12 smithi188 ceph-mon[32751]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T01:37:13.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:12 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:13.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:12 smithi188 ceph-mon[32751]: from='client.? 172.21.15.29:0/1347420287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:37:15.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:14 smithi029 ceph-mon[32775]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:37:15.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:14 smithi006 ceph-mon[28419]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:37:15.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:14 smithi188 ceph-mon[32751]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:37:17.169 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:16 smithi029 ceph-mon[32775]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-26T01:37:17.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:16 smithi006 ceph-mon[28419]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-26T01:37:17.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:16 smithi188 ceph-mon[32751]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-26T01:37:19.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:18 smithi029 ceph-mon[32775]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-26T01:37:19.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:18 smithi006 ceph-mon[28419]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-26T01:37:19.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:18 smithi188 ceph-mon[32751]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-26T01:37:20.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:19 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T01:37:20.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:19 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:20.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:19 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T01:37:20.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:19 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:20.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:19 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T01:37:20.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:19 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:21.130 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:20 smithi029 ceph-mon[32775]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T01:37:21.131 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:20 smithi029 ceph-mon[32775]: Deploying daemon osd.4 on smithi029 2024-04-26T01:37:21.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:20 smithi006 ceph-mon[28419]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T01:37:21.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:20 smithi006 ceph-mon[28419]: Deploying daemon osd.4 on smithi029 2024-04-26T01:37:21.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:20 smithi188 ceph-mon[32751]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T01:37:21.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:20 smithi188 ceph-mon[32751]: Deploying daemon osd.4 on smithi029 2024-04-26T01:37:23.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:22 smithi029 ceph-mon[32775]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:23.189 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:22 smithi188 ceph-mon[32751]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:23.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:22 smithi006 ceph-mon[28419]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:25.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:24 smithi006 ceph-mon[28419]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:25.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:24 smithi188 ceph-mon[32751]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:25.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:24 smithi029 ceph-mon[32775]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:26.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:26 smithi006 ceph-mon[28419]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:26.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:26.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:26.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:26.616 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:26 smithi029 ceph-mon[32775]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:26.617 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:26.617 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:26.617 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:26.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:26 smithi188 ceph-mon[32751]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:26.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:26.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:26.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:28.406 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:28 smithi029 ceph-mon[32775]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:28.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.498+0000 7fb1737fe700 1 -- 172.21.15.29:0/571502140 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb184062e80 con 0x7fb16c0a3c40 2024-04-26T01:37:28.494 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 4 on host 'smithi029' 2024-04-26T01:37:28.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.501+0000 7fb18880b700 1 -- 172.21.15.29:0/571502140 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb16c0a3c40 msgr2=0x7fb16c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:28.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.501+0000 7fb18880b700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb16c0a3c40 0x7fb16c0a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb1740038c0 tx=0x7fb17400b540 comp rx=0 tx=0).stop 2024-04-26T01:37:28.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.501+0000 7fb18880b700 1 -- 172.21.15.29:0/571502140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184104640 msgr2=0x7fb184075370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:28.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.501+0000 7fb18880b700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184104640 0x7fb184075370 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00b970 tx=0x7fb17c00bc80 comp rx=0 tx=0).stop 2024-04-26T01:37:28.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.501+0000 7fb18880b700 1 -- 172.21.15.29:0/571502140 shutdown_connections 2024-04-26T01:37:28.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.501+0000 7fb18880b700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184104640 0x7fb184075370 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:28.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.501+0000 7fb18880b700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb16c0a3c40 0x7fb16c0a6100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:28.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.501+0000 7fb18880b700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb184102aa0 0x7fb1840767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:28.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.501+0000 7fb18880b700 1 --2- 172.21.15.29:0/571502140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb184103ca0 0x7fb184074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:28.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.501+0000 7fb18880b700 1 -- 172.21.15.29:0/571502140 >> 172.21.15.29:0/571502140 conn(0x7fb1840fe040 msgr2=0x7fb1841070b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:28.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.501+0000 7fb18880b700 1 -- 172.21.15.29:0/571502140 shutdown_connections 2024-04-26T01:37:28.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:37:28.501+0000 7fb18880b700 1 -- 172.21.15.29:0/571502140 wait complete. 2024-04-26T01:37:28.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:28 smithi006 ceph-mon[28419]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:28.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:28 smithi188 ceph-mon[32751]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:29.649 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:29 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:29.650 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:29 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:29.650 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:29 smithi029 ceph-mon[32775]: from='osd.4 [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:37:29.650 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:29 smithi029 ceph-mon[32775]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:37:29.650 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:29 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:29.650 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:29 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:29.678 DEBUG:teuthology.orchestra.run.smithi029:osd.4> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T01:37:29.681 INFO:tasks.cephadm:Deploying osd.5 on smithi188 with /dev/vg_nvme/lv_3... 2024-04-26T01:37:29.681 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T01:37:29.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:29 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:29.708 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:29 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:29.708 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:29 smithi188 ceph-mon[32751]: from='osd.4 [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:37:29.708 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:29 smithi188 ceph-mon[32751]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:37:29.708 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:29 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:29.708 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:29 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:29.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:29 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:29.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:29 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:29.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:29 smithi006 ceph-mon[28419]: from='osd.4 [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:37:29.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:29 smithi006 ceph-mon[28419]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:37:29.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:29 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:29.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:29 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:29.924 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:37:30.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:30 smithi188 ceph-mon[32751]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:30.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:30 smithi188 ceph-mon[32751]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T01:37:30.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:30 smithi188 ceph-mon[32751]: from='osd.4 [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:37:30.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:30 smithi188 ceph-mon[32751]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T01:37:30.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:30.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:30 smithi188 ceph-mon[32751]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:37:30.680 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:30 smithi029 ceph-mon[32775]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:30.680 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:30 smithi029 ceph-mon[32775]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T01:37:30.680 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:30 smithi029 ceph-mon[32775]: from='osd.4 [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:37:30.680 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:30 smithi029 ceph-mon[32775]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T01:37:30.680 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:30.680 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:30 smithi029 ceph-mon[32775]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:37:30.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:30 smithi006 ceph-mon[28419]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:30.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:30 smithi006 ceph-mon[28419]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T01:37:30.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:30 smithi006 ceph-mon[28419]: from='osd.4 [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:37:30.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:30 smithi006 ceph-mon[28419]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T01:37:30.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:30.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:30 smithi006 ceph-mon[28419]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:37:31.312 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 01:37:31 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4[44350]: 2024-04-26T01:37:31.176+0000 7f87b501f700 -1 osd.4 0 waiting for initial osdmap 2024-04-26T01:37:31.312 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 01:37:31 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4[44350]: 2024-04-26T01:37:31.182+0000 7f87ac609700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:37:31.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:31 smithi188 ceph-mon[32751]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-26T01:37:31.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:31 smithi188 ceph-mon[32751]: osdmap e33: 5 total, 4 up, 5 in 2024-04-26T01:37:31.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:31 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:31.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:31 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:31.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:31 smithi188 ceph-mon[32751]: from='osd.4 ' entity='osd.4' 2024-04-26T01:37:31.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:31 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:31.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:31 smithi029 ceph-mon[32775]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-26T01:37:31.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:31 smithi029 ceph-mon[32775]: osdmap e33: 5 total, 4 up, 5 in 2024-04-26T01:37:31.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:31 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:31.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:31 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:31.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:31 smithi029 ceph-mon[32775]: from='osd.4 ' entity='osd.4' 2024-04-26T01:37:31.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:31 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:31.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:31 smithi006 ceph-mon[28419]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-26T01:37:31.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:31 smithi006 ceph-mon[28419]: osdmap e33: 5 total, 4 up, 5 in 2024-04-26T01:37:31.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:31 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:31.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:31 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:31.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:31 smithi006 ceph-mon[28419]: from='osd.4 ' entity='osd.4' 2024-04-26T01:37:31.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:31 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:31.852 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T01:37:31.852 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T01:37:31.852 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-26T01:37:31.852 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-26T01:37:31.853 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112939 s, 928 MB/s 2024-04-26T01:37:31.853 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-26T01:37:32.444 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_3 2024-04-26T01:37:32.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:32 smithi188 ceph-mon[32751]: purged_snaps scrub starts 2024-04-26T01:37:32.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:32 smithi188 ceph-mon[32751]: purged_snaps scrub ok 2024-04-26T01:37:32.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:32 smithi188 ceph-mon[32751]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:32.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:32 smithi188 ceph-mon[32751]: osd.4 [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609] boot 2024-04-26T01:37:32.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:32 smithi188 ceph-mon[32751]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T01:37:32.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:32 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:32.736 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:37:32.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:32 smithi006 ceph-mon[28419]: purged_snaps scrub starts 2024-04-26T01:37:32.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:32 smithi006 ceph-mon[28419]: purged_snaps scrub ok 2024-04-26T01:37:32.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:32 smithi006 ceph-mon[28419]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:32.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:32 smithi006 ceph-mon[28419]: osd.4 [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609] boot 2024-04-26T01:37:32.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:32 smithi006 ceph-mon[28419]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T01:37:32.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:32 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:32.857 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:32 smithi029 ceph-mon[32775]: purged_snaps scrub starts 2024-04-26T01:37:32.857 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:32 smithi029 ceph-mon[32775]: purged_snaps scrub ok 2024-04-26T01:37:32.857 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:32 smithi029 ceph-mon[32775]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:37:32.857 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:32 smithi029 ceph-mon[32775]: osd.4 [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609] boot 2024-04-26T01:37:32.857 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:32 smithi029 ceph-mon[32775]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T01:37:32.857 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:32 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:37:34.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: Detected new or changed devices on smithi029 2024-04-26T01:37:34.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:34.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:34.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi029 to 8546M 2024-04-26T01:37:34.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:34.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:34.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:34.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:34.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:34.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:33 smithi029 ceph-mon[32775]: osdmap e35: 5 total, 5 up, 5 in 2024-04-26T01:37:34.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: Detected new or changed devices on smithi029 2024-04-26T01:37:34.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi029 to 8546M 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:33 smithi188 ceph-mon[32751]: osdmap e35: 5 total, 5 up, 5 in 2024-04-26T01:37:34.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: Detected new or changed devices on smithi029 2024-04-26T01:37:34.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi029 to 8546M 2024-04-26T01:37:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:33 smithi006 ceph-mon[28419]: osdmap e35: 5 total, 5 up, 5 in 2024-04-26T01:37:34.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 -- 172.21.15.188:0/1165897867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d8103ca0 msgr2=0x7f77d8104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:34.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 --2- 172.21.15.188:0/1165897867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d8103ca0 0x7f77d8104100 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009990 tx=0x7f77c8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:37:34.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 -- 172.21.15.188:0/1165897867 shutdown_connections 2024-04-26T01:37:34.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 --2- 172.21.15.188:0/1165897867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77d8104640 0x7f77d810aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:34.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 --2- 172.21.15.188:0/1165897867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d8102aa0 0x7f77d8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:34.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 --2- 172.21.15.188:0/1165897867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d8103ca0 0x7f77d8104100 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:34.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 -- 172.21.15.188:0/1165897867 >> 172.21.15.188:0/1165897867 conn(0x7f77d80fe040 msgr2=0x7f77d8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 -- 172.21.15.188:0/1165897867 shutdown_connections 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 -- 172.21.15.188:0/1165897867 wait complete. 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 Processor -- start 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 -- start start 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d8102aa0 0x7f77d8198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d8103ca0 0x7f77d8199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77d8104640 0x7f77d81a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.470+0000 7f77df5a9700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f77d810e590 con 0x7f77d8104640 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77df5a9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f77d810e410 con 0x7f77d8102aa0 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77df5a9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f77d810e710 con 0x7f77d8103ca0 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77dcb44700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d8103ca0 0x7f77d8199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77dcb44700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d8103ca0 0x7f77d8199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:37196/0 (socket says 172.21.15.188:37196) 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77dcb44700 1 -- 172.21.15.188:0/792401442 learned_addr learned my addr 172.21.15.188:0/792401442 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-26T01:37:34.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77dcb44700 1 -- 172.21.15.188:0/792401442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d8102aa0 msgr2=0x7f77d8198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:34.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77dd345700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d8102aa0 0x7f77d8198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:34.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77dcb44700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d8102aa0 0x7f77d8198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:34.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77dcb44700 1 -- 172.21.15.188:0/792401442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77d8104640 msgr2=0x7f77d81a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:34.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77dcb44700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77d8104640 0x7f77d81a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:34.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77dcb44700 1 -- 172.21.15.188:0/792401442 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d81a0880 con 0x7f77d8103ca0 2024-04-26T01:37:34.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77dcb44700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d8103ca0 0x7f77d8199170 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f77c8006010 tx=0x7f77c8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:34.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77ce7fc700 1 -- 172.21.15.188:0/792401442 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c8027070 con 0x7f77d8103ca0 2024-04-26T01:37:34.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77df5a9700 1 -- 172.21.15.188:0/792401442 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d81a0b10 con 0x7f77d8103ca0 2024-04-26T01:37:34.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77ce7fc700 1 -- 172.21.15.188:0/792401442 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77c8022470 con 0x7f77d8103ca0 2024-04-26T01:37:34.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.471+0000 7f77ce7fc700 1 -- 172.21.15.188:0/792401442 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c800f6e0 con 0x7f77d8103ca0 2024-04-26T01:37:34.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.472+0000 7f77df5a9700 1 -- 172.21.15.188:0/792401442 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d81a0f60 con 0x7f77d8103ca0 2024-04-26T01:37:34.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.473+0000 7f77df5a9700 1 -- 172.21.15.188:0/792401442 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77d802df80 con 0x7f77d8103ca0 2024-04-26T01:37:34.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.473+0000 7f77ce7fc700 1 -- 172.21.15.188:0/792401442 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f77c802a070 con 0x7f77d8103ca0 2024-04-26T01:37:34.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.474+0000 7f77ce7fc700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f77c40a3c40 0x7f77c40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:34.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.474+0000 7f77ce7fc700 1 -- 172.21.15.188:0/792401442 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7f77c80d0190 con 0x7f77d8103ca0 2024-04-26T01:37:34.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.475+0000 7f77dd345700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f77c40a3c40 0x7f77c40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:34.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.476+0000 7f77dd345700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f77c40a3c40 0x7f77c40a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77d4005950 tx=0x7f77d40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:34.474 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.479+0000 7f77ce7fc700 1 -- 172.21.15.188:0/792401442 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77c8013070 con 0x7f77d8103ca0 2024-04-26T01:37:34.681 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:34.686+0000 7f77df5a9700 1 -- 172.21.15.188:0/792401442 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f77d8062e80 con 0x7f77c40a3c40 2024-04-26T01:37:35.188 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:34 smithi029 ceph-mon[32775]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:35.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:35.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:35.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:35.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:34 smithi006 ceph-mon[28419]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:35.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:35.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:35.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:35.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:34 smithi188 ceph-mon[32751]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:35.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:35.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:35.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:36.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:35 smithi029 ceph-mon[32775]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:36.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:35 smithi006 ceph-mon[28419]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:36.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:35 smithi188 ceph-mon[32751]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:37.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:36 smithi029 ceph-mon[32775]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:37.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:36 smithi006 ceph-mon[28419]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:37.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:36 smithi188 ceph-mon[32751]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:38.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:37 smithi006 ceph-mon[28419]: from='client.? 172.21.15.188:0/4171320331' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9788bb0a-0d79-4743-bce6-80f5342e8f4a"}]: dispatch 2024-04-26T01:37:38.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:37 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9788bb0a-0d79-4743-bce6-80f5342e8f4a"}]: dispatch 2024-04-26T01:37:38.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:37 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9788bb0a-0d79-4743-bce6-80f5342e8f4a"}]': finished 2024-04-26T01:37:38.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:37 smithi006 ceph-mon[28419]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T01:37:38.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:38.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:37 smithi006 ceph-mon[28419]: from='client.? 172.21.15.188:0/3923825828' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:37:38.379 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:37 smithi029 ceph-mon[32775]: from='client.? 172.21.15.188:0/4171320331' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9788bb0a-0d79-4743-bce6-80f5342e8f4a"}]: dispatch 2024-04-26T01:37:38.379 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:37 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9788bb0a-0d79-4743-bce6-80f5342e8f4a"}]: dispatch 2024-04-26T01:37:38.380 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:37 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9788bb0a-0d79-4743-bce6-80f5342e8f4a"}]': finished 2024-04-26T01:37:38.380 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:37 smithi029 ceph-mon[32775]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T01:37:38.380 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:38.380 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:37 smithi029 ceph-mon[32775]: from='client.? 172.21.15.188:0/3923825828' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:37:38.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:37 smithi188 ceph-mon[32751]: from='client.? 172.21.15.188:0/4171320331' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9788bb0a-0d79-4743-bce6-80f5342e8f4a"}]: dispatch 2024-04-26T01:37:38.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:37 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9788bb0a-0d79-4743-bce6-80f5342e8f4a"}]: dispatch 2024-04-26T01:37:38.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:37 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9788bb0a-0d79-4743-bce6-80f5342e8f4a"}]': finished 2024-04-26T01:37:38.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:37 smithi188 ceph-mon[32751]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T01:37:38.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:38.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:37 smithi188 ceph-mon[32751]: from='client.? 172.21.15.188:0/3923825828' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:37:39.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:38 smithi006 ceph-mon[28419]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:39.350 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:38 smithi029 ceph-mon[32775]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:39.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:38 smithi188 ceph-mon[32751]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:40.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:40 smithi006 ceph-mon[28419]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:40.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:40 smithi188 ceph-mon[32751]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:40.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:40 smithi029 ceph-mon[32775]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:42.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:42 smithi029 ceph-mon[32775]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:42.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:42 smithi188 ceph-mon[32751]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:42.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:42 smithi006 ceph-mon[28419]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:44.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:44 smithi188 ceph-mon[32751]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:44.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:44 smithi029 ceph-mon[32775]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:44.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:44 smithi006 ceph-mon[28419]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:45.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:45 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T01:37:45.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:45 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:45.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:45 smithi029 ceph-mon[32775]: Deploying daemon osd.5 on smithi188 2024-04-26T01:37:45.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:45 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T01:37:45.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:45 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:45.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:45 smithi188 ceph-mon[32751]: Deploying daemon osd.5 on smithi188 2024-04-26T01:37:45.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:45 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T01:37:45.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:45 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:45.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:45 smithi006 ceph-mon[28419]: Deploying daemon osd.5 on smithi188 2024-04-26T01:37:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:46 smithi188 ceph-mon[32751]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:46.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:46 smithi029 ceph-mon[32775]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:46.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:46 smithi006 ceph-mon[28419]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:48.328 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:48 smithi188 ceph-mon[32751]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:48.438 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:48 smithi029 ceph-mon[32775]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:48.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:48 smithi006 ceph-mon[28419]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:50.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:50 smithi029 ceph-mon[32775]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:50.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:50.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:50.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:50.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:50 smithi188 ceph-mon[32751]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:50.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:50.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:50.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:50.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:50 smithi006 ceph-mon[28419]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:50.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:50.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:50.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:52.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:52 smithi188 ceph-mon[32751]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:52.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:52 smithi029 ceph-mon[32775]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:52.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:52 smithi006 ceph-mon[28419]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:53.769 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.772+0000 7f77ce7fc700 1 -- 172.21.15.188:0/792401442 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f77d8062e80 con 0x7f77c40a3c40 2024-04-26T01:37:53.769 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 5 on host 'smithi188' 2024-04-26T01:37:53.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.774+0000 7f77df5a9700 1 -- 172.21.15.188:0/792401442 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f77c40a3c40 msgr2=0x7f77c40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:53.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.774+0000 7f77df5a9700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f77c40a3c40 0x7f77c40a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77d4005950 tx=0x7f77d40058e0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.774+0000 7f77df5a9700 1 -- 172.21.15.188:0/792401442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d8103ca0 msgr2=0x7f77d8199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:53.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.774+0000 7f77df5a9700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d8103ca0 0x7f77d8199170 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f77c8006010 tx=0x7f77c8009f90 comp rx=0 tx=0).stop 2024-04-26T01:37:53.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.774+0000 7f77df5a9700 1 -- 172.21.15.188:0/792401442 shutdown_connections 2024-04-26T01:37:53.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.774+0000 7f77df5a9700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77d8104640 0x7f77d81a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.774+0000 7f77df5a9700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d8102aa0 0x7f77d8198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.774+0000 7f77df5a9700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f77c40a3c40 0x7f77c40a6100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.774+0000 7f77df5a9700 1 --2- 172.21.15.188:0/792401442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77d8103ca0 0x7f77d8199170 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:53.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.774+0000 7f77df5a9700 1 -- 172.21.15.188:0/792401442 >> 172.21.15.188:0/792401442 conn(0x7f77d80fe040 msgr2=0x7f77d81070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:53.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.774+0000 7f77df5a9700 1 -- 172.21.15.188:0/792401442 shutdown_connections 2024-04-26T01:37:53.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:37:53.774+0000 7f77df5a9700 1 -- 172.21.15.188:0/792401442 wait complete. 2024-04-26T01:37:53.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:53 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:53.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:53 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:53.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:53 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:53.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:53 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:53.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:53 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:53.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:53 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:53.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:53 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:53.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:53 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:53.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:53 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:53.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:53 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:54.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:53 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:54.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:53 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:54.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:53 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:54.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:53 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:54.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:53 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:54.325 DEBUG:teuthology.orchestra.run.smithi188:osd.5> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T01:37:54.327 INFO:tasks.cephadm:Deploying osd.6 on smithi006 with /dev/vg_nvme/lv_2... 2024-04-26T01:37:54.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T01:37:54.789 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:54.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:54 smithi188 ceph-mon[32751]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:54.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:54 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:54.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:54 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:54.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:54 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:54.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:54 smithi188 ceph-mon[32751]: from='osd.5 [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:37:54.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:54 smithi188 ceph-mon[32751]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:37:55.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:54 smithi006 ceph-mon[28419]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:55.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:54 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:55.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:54 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:55.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:54 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:55.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:54 smithi006 ceph-mon[28419]: from='osd.5 [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:37:55.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:54 smithi006 ceph-mon[28419]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:37:55.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:54 smithi029 ceph-mon[32775]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:55.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:54 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:55.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:54 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:55.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:54 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:55.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:54 smithi029 ceph-mon[32775]: from='osd.5 [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:37:55.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:54 smithi029 ceph-mon[32775]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:37:55.842 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:55 smithi188 ceph-mon[32751]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T01:37:55.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:55 smithi188 ceph-mon[32751]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T01:37:55.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:55 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:55.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:55 smithi188 ceph-mon[32751]: from='osd.5 [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:37:55.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:55 smithi188 ceph-mon[32751]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:37:56.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:55 smithi006 ceph-mon[28419]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T01:37:56.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:55 smithi006 ceph-mon[28419]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T01:37:56.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:55 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:56.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:55 smithi006 ceph-mon[28419]: from='osd.5 [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:37:56.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:55 smithi006 ceph-mon[28419]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:37:56.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:55 smithi029 ceph-mon[32775]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T01:37:56.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:55 smithi029 ceph-mon[32775]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T01:37:56.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:55 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:56.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:55 smithi029 ceph-mon[32775]: from='osd.5 [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:37:56.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:55 smithi029 ceph-mon[32775]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:37:56.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:56 smithi188 ceph-mon[32751]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:56.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:56 smithi188 ceph-mon[32751]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-26T01:37:56.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:56 smithi188 ceph-mon[32751]: osdmap e38: 6 total, 5 up, 6 in 2024-04-26T01:37:56.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:56.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:56.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:56 smithi188 ceph-mon[32751]: from='osd.5 ' entity='osd.5' 2024-04-26T01:37:56.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:56 smithi188 ceph-mon[32751]: osdmap e39: 6 total, 5 up, 6 in 2024-04-26T01:37:56.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:56.889 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 01:37:56 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T01:37:56.648+0000 7f34600ea700 -1 osd.5 0 waiting for initial osdmap 2024-04-26T01:37:56.889 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 01:37:56 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T01:37:56.747+0000 7f3457ed5700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:37:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:56 smithi006 ceph-mon[28419]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:56 smithi006 ceph-mon[28419]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-26T01:37:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:56 smithi006 ceph-mon[28419]: osdmap e38: 6 total, 5 up, 6 in 2024-04-26T01:37:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:56 smithi006 ceph-mon[28419]: from='osd.5 ' entity='osd.5' 2024-04-26T01:37:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:56 smithi006 ceph-mon[28419]: osdmap e39: 6 total, 5 up, 6 in 2024-04-26T01:37:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:57.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:56 smithi029 ceph-mon[32775]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:57.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:56 smithi029 ceph-mon[32775]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-26T01:37:57.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:56 smithi029 ceph-mon[32775]: osdmap e38: 6 total, 5 up, 6 in 2024-04-26T01:37:57.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:56 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:57.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:56 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:57.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:56 smithi029 ceph-mon[32775]: from='osd.5 ' entity='osd.5' 2024-04-26T01:37:57.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:56 smithi029 ceph-mon[32775]: osdmap e39: 6 total, 5 up, 6 in 2024-04-26T01:37:57.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:56 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:57.720 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T01:37:57.720 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T01:37:57.720 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-04-26T01:37:57.720 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-04-26T01:37:57.720 INFO:teuthology.orchestra.run.smithi006.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0372106 s, 282 MB/s 2024-04-26T01:37:57.720 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-04-26T01:37:58.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: purged_snaps scrub starts 2024-04-26T01:37:58.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: purged_snaps scrub ok 2024-04-26T01:37:58.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:58.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:58.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:58.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:58.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:58.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:58.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:58.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:58.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: osd.5 [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007] boot 2024-04-26T01:37:58.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T01:37:58.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:58.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: purged_snaps scrub starts 2024-04-26T01:37:58.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: purged_snaps scrub ok 2024-04-26T01:37:58.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:58.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:58.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:58.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:58.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:58.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:58.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:58.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:58.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: osd.5 [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007] boot 2024-04-26T01:37:58.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T01:37:58.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:58.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: purged_snaps scrub starts 2024-04-26T01:37:58.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: purged_snaps scrub ok 2024-04-26T01:37:58.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:58.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:58.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:58.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:58.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:58.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:58.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:58.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:37:58.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: osd.5 [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007] boot 2024-04-26T01:37:58.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T01:37:58.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:37:58.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi006:vg_nvme/lv_2 2024-04-26T01:37:59.010 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:37:59.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:58 smithi006 ceph-mon[28419]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:59.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:58 smithi006 ceph-mon[28419]: Detected new or changed devices on smithi188 2024-04-26T01:37:59.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:58 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi188 to 8544M 2024-04-26T01:37:59.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:37:58 smithi006 ceph-mon[28419]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T01:37:59.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:58 smithi188 ceph-mon[32751]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:59.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:58 smithi188 ceph-mon[32751]: Detected new or changed devices on smithi188 2024-04-26T01:37:59.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:58 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi188 to 8544M 2024-04-26T01:37:59.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:37:58 smithi188 ceph-mon[32751]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T01:37:59.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:58 smithi029 ceph-mon[32775]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:37:59.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:58 smithi029 ceph-mon[32775]: Detected new or changed devices on smithi188 2024-04-26T01:37:59.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:58 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi188 to 8544M 2024-04-26T01:37:59.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:37:58 smithi029 ceph-mon[32775]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T01:38:00.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:00 smithi006 ceph-mon[28419]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:00.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:00 smithi006 ceph-mon[28419]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T01:38:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.036+0000 7fd2c7800700 1 -- 172.21.15.6:0/3713783854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c00754a0 msgr2=0x7fd2c0075920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.036+0000 7fd2c7800700 1 --2- 172.21.15.6:0/3713783854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c00754a0 0x7fd2c0075920 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009990 tx=0x7fd2ac009ca0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.036+0000 7fd2c7800700 1 -- 172.21.15.6:0/3713783854 shutdown_connections 2024-04-26T01:38:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.036+0000 7fd2c7800700 1 --2- 172.21.15.6:0/3713783854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c010f080 0x7fd2c0111470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.036+0000 7fd2c7800700 1 --2- 172.21.15.6:0/3713783854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c0076a70 0x7fd2c0074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.036+0000 7fd2c7800700 1 --2- 172.21.15.6:0/3713783854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c00754a0 0x7fd2c0075920 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.036+0000 7fd2c7800700 1 -- 172.21.15.6:0/3713783854 >> 172.21.15.6:0/3713783854 conn(0x7fd2c00fe040 msgr2=0x7fd2c0100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.036+0000 7fd2c7800700 1 -- 172.21.15.6:0/3713783854 shutdown_connections 2024-04-26T01:38:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.036+0000 7fd2c7800700 1 -- 172.21.15.6:0/3713783854 wait complete. 2024-04-26T01:38:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.037+0000 7fd2c7800700 1 Processor -- start 2024-04-26T01:38:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.037+0000 7fd2c7800700 1 -- start start 2024-04-26T01:38:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.038+0000 7fd2c7800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c00754a0 0x7fd2c019f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.038+0000 7fd2c7800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c0076a70 0x7fd2c019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.038+0000 7fd2c7800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c010f080 0x7fd2c01a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.038+0000 7fd2c7800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c0114ba0 con 0x7fd2c00754a0 2024-04-26T01:38:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.038+0000 7fd2c7800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c0114a20 con 0x7fd2c0076a70 2024-04-26T01:38:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.038+0000 7fd2c7800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c0114d20 con 0x7fd2c010f080 2024-04-26T01:38:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.038+0000 7fd2c4e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c00754a0 0x7fd2c019f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.038+0000 7fd2bfe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c0076a70 0x7fd2c019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.038+0000 7fd2c4e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c00754a0 0x7fd2c019f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51912/0 (socket says 172.21.15.6:51912) 2024-04-26T01:38:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.038+0000 7fd2c4e00700 1 -- 172.21.15.6:0/2213519431 learned_addr learned my addr 172.21.15.6:0/2213519431 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:38:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.038+0000 7fd2c5800700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c010f080 0x7fd2c01a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.039+0000 7fd2c5800700 1 -- 172.21.15.6:0/2213519431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c0076a70 msgr2=0x7fd2c019f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.039+0000 7fd2c5800700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c0076a70 0x7fd2c019f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.039+0000 7fd2c5800700 1 -- 172.21.15.6:0/2213519431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c00754a0 msgr2=0x7fd2c019f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.039+0000 7fd2c5800700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c00754a0 0x7fd2c019f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.039+0000 7fd2c5800700 1 -- 172.21.15.6:0/2213519431 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c01a6ef0 con 0x7fd2c010f080 2024-04-26T01:38:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.039+0000 7fd2c4e00700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c00754a0 0x7fd2c019f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:38:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.039+0000 7fd2c5800700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c010f080 0x7fd2c01a6850 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800c0c0 tx=0x7fd2b800c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.039+0000 7fd2bd600700 1 -- 172.21.15.6:0/2213519431 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8003a00 con 0x7fd2c010f080 2024-04-26T01:38:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.039+0000 7fd2c7800700 1 -- 172.21.15.6:0/2213519431 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c01a71e0 con 0x7fd2c010f080 2024-04-26T01:38:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.039+0000 7fd2bd600700 1 -- 172.21.15.6:0/2213519431 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2b800ce80 con 0x7fd2c010f080 2024-04-26T01:38:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.039+0000 7fd2bd600700 1 -- 172.21.15.6:0/2213519431 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b80047c0 con 0x7fd2c010f080 2024-04-26T01:38:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.039+0000 7fd2c7800700 1 -- 172.21.15.6:0/2213519431 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c01a76d0 con 0x7fd2c010f080 2024-04-26T01:38:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.040+0000 7fd2c7800700 1 -- 172.21.15.6:0/2213519431 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c0069730 con 0x7fd2c010f080 2024-04-26T01:38:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.042+0000 7fd2bd600700 1 -- 172.21.15.6:0/2213519431 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fd2b8012050 con 0x7fd2c010f080 2024-04-26T01:38:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.044+0000 7fd2bd600700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd2a80a3c40 0x7fd2a80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.044+0000 7fd2bd600700 1 -- 172.21.15.6:0/2213519431 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4533+0+0 (secure 0 0 0) 0x7fd2b80cef20 con 0x7fd2c010f080 2024-04-26T01:38:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.044+0000 7fd2c4e00700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd2a80a3c40 0x7fd2a80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.045+0000 7fd2c4e00700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd2a80a3c40 0x7fd2a80a6100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4011550 tx=0x7fd2b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.046+0000 7fd2bd600700 1 -- 172.21.15.6:0/2213519431 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd2b8022060 con 0x7fd2c010f080 2024-04-26T01:38:01.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:00 smithi188 ceph-mon[32751]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:01.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:00 smithi188 ceph-mon[32751]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T01:38:01.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:00 smithi029 ceph-mon[32775]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:01.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:00 smithi029 ceph-mon[32775]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T01:38:01.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:01.287+0000 7fd2c7800700 1 -- 172.21.15.6:0/2213519431 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fd2c0062e80 con 0x7fd2a80a3c40 2024-04-26T01:38:02.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:01 smithi006 ceph-mon[28419]: osdmap e43: 6 total, 6 up, 6 in 2024-04-26T01:38:02.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:01 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:38:02.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:01 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:38:02.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:01 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:02.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:01 smithi188 ceph-mon[32751]: osdmap e43: 6 total, 6 up, 6 in 2024-04-26T01:38:02.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:01 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:38:02.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:01 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:38:02.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:01 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:02.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:01 smithi029 ceph-mon[32775]: osdmap e43: 6 total, 6 up, 6 in 2024-04-26T01:38:02.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:01 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:38:02.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:01 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:38:02.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:01 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:02.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:02 smithi006 ceph-mon[28419]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T01:38:02.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:02 smithi006 ceph-mon[28419]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:38:02.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:02 smithi006 ceph-mon[28419]: osdmap e44: 6 total, 6 up, 6 in 2024-04-26T01:38:03.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:02 smithi188 ceph-mon[32751]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T01:38:03.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:02 smithi188 ceph-mon[32751]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:38:03.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:02 smithi188 ceph-mon[32751]: osdmap e44: 6 total, 6 up, 6 in 2024-04-26T01:38:03.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:02 smithi029 ceph-mon[32775]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T01:38:03.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:02 smithi029 ceph-mon[32775]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:38:03.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:02 smithi029 ceph-mon[32775]: osdmap e44: 6 total, 6 up, 6 in 2024-04-26T01:38:05.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:04 smithi006 ceph-mon[28419]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-04-26T01:38:05.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:04 smithi188 ceph-mon[32751]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-04-26T01:38:05.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:04 smithi029 ceph-mon[32775]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-04-26T01:38:06.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:05 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3393610285' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd8584e1-69ee-4c30-85e4-a637357d1b6d"}]: dispatch 2024-04-26T01:38:06.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:05 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd8584e1-69ee-4c30-85e4-a637357d1b6d"}]: dispatch 2024-04-26T01:38:06.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:05 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd8584e1-69ee-4c30-85e4-a637357d1b6d"}]': finished 2024-04-26T01:38:06.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:05 smithi006 ceph-mon[28419]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T01:38:06.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:06.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:05 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3393610285' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd8584e1-69ee-4c30-85e4-a637357d1b6d"}]: dispatch 2024-04-26T01:38:06.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:05 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd8584e1-69ee-4c30-85e4-a637357d1b6d"}]: dispatch 2024-04-26T01:38:06.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:05 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd8584e1-69ee-4c30-85e4-a637357d1b6d"}]': finished 2024-04-26T01:38:06.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:05 smithi188 ceph-mon[32751]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T01:38:06.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:06.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:05 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3393610285' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd8584e1-69ee-4c30-85e4-a637357d1b6d"}]: dispatch 2024-04-26T01:38:06.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:05 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd8584e1-69ee-4c30-85e4-a637357d1b6d"}]: dispatch 2024-04-26T01:38:06.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:05 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd8584e1-69ee-4c30-85e4-a637357d1b6d"}]': finished 2024-04-26T01:38:06.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:05 smithi029 ceph-mon[32775]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T01:38:06.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:07.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:06 smithi188 ceph-mon[32751]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:38:07.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:06 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1398917588' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:38:07.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:06 smithi029 ceph-mon[32775]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:38:07.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:06 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1398917588' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:38:07.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:06 smithi006 ceph-mon[28419]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:38:07.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:06 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1398917588' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:38:09.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:08 smithi188 ceph-mon[32751]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:09.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:08 smithi029 ceph-mon[32775]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:09.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:08 smithi006 ceph-mon[28419]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:11.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:10 smithi006 ceph-mon[28419]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:11.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:10 smithi188 ceph-mon[32751]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:11.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:10 smithi029 ceph-mon[32775]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:13.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:12 smithi188 ceph-mon[32751]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:13.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:12 smithi029 ceph-mon[32775]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:13.269 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:12 smithi006 ceph-mon[28419]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:15.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:14 smithi188 ceph-mon[32751]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:15.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:14 smithi029 ceph-mon[32775]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:15.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:14 smithi006 ceph-mon[28419]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:16.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:15 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T01:38:16.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:15 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:16.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:15 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T01:38:16.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:15 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:16.314 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:15 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T01:38:16.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:15 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:17.000 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:16 smithi006 ceph-mon[28419]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:17.000 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:16 smithi006 ceph-mon[28419]: Deploying daemon osd.6 on smithi006 2024-04-26T01:38:17.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:16 smithi188 ceph-mon[32751]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:17.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:16 smithi188 ceph-mon[32751]: Deploying daemon osd.6 on smithi006 2024-04-26T01:38:17.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:16 smithi029 ceph-mon[32775]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:17.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:16 smithi029 ceph-mon[32775]: Deploying daemon osd.6 on smithi006 2024-04-26T01:38:19.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:18 smithi006 ceph-mon[28419]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:19.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:18 smithi188 ceph-mon[32751]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:19.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:18 smithi029 ceph-mon[32775]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:21.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:20 smithi188 ceph-mon[32751]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:21.158 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:20 smithi006 ceph-mon[28419]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:21.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:20 smithi029 ceph-mon[32775]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:23.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:22 smithi006 ceph-mon[28419]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:23.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:22 smithi188 ceph-mon[32751]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:23.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:22 smithi029 ceph-mon[32775]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:25.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:24 smithi029 ceph-mon[32775]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:25.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:25.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:25.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:25.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:24 smithi188 ceph-mon[32751]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:25.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:25.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:25.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:25.246 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:24 smithi006 ceph-mon[28419]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:25.247 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:25.247 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:25.247 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:27.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:26 smithi006 ceph-mon[28419]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:27.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:26 smithi029 ceph-mon[32775]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:27.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:26 smithi188 ceph-mon[32751]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:29.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:28 smithi006 ceph-mon[28419]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:29.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:28 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:29.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:28 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:29.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:28 smithi006 ceph-mon[28419]: from='osd.6 [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T01:38:29.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:28 smithi006 ceph-mon[28419]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T01:38:29.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:28 smithi188 ceph-mon[32751]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:29.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:28 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:29.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:28 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:29.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:28 smithi188 ceph-mon[32751]: from='osd.6 [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T01:38:29.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:28 smithi188 ceph-mon[32751]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T01:38:29.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:28 smithi029 ceph-mon[32775]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:29.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:28 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:29.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:28 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:29.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:28 smithi029 ceph-mon[32775]: from='osd.6 [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T01:38:29.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:28 smithi029 ceph-mon[32775]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T01:38:29.190 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 6 on host 'smithi006' 2024-04-26T01:38:29.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.193+0000 7fd2bd600700 1 -- 172.21.15.6:0/2213519431 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd2c0062e80 con 0x7fd2a80a3c40 2024-04-26T01:38:29.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.197+0000 7fd2c7800700 1 -- 172.21.15.6:0/2213519431 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd2a80a3c40 msgr2=0x7fd2a80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:29.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.197+0000 7fd2c7800700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd2a80a3c40 0x7fd2a80a6100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4011550 tx=0x7fd2b4009260 comp rx=0 tx=0).stop 2024-04-26T01:38:29.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.197+0000 7fd2c7800700 1 -- 172.21.15.6:0/2213519431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c010f080 msgr2=0x7fd2c01a6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:29.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.197+0000 7fd2c7800700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c010f080 0x7fd2c01a6850 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800c0c0 tx=0x7fd2b800c3d0 comp rx=0 tx=0).stop 2024-04-26T01:38:29.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.197+0000 7fd2c7800700 1 -- 172.21.15.6:0/2213519431 shutdown_connections 2024-04-26T01:38:29.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.197+0000 7fd2c7800700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd2a80a3c40 0x7fd2a80a6100 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:29.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.197+0000 7fd2c7800700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c010f080 0x7fd2c01a6850 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:29.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.197+0000 7fd2c7800700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2c00754a0 0x7fd2c019f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:29.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.197+0000 7fd2c7800700 1 --2- 172.21.15.6:0/2213519431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c0076a70 0x7fd2c019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:29.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.197+0000 7fd2c7800700 1 -- 172.21.15.6:0/2213519431 >> 172.21.15.6:0/2213519431 conn(0x7fd2c00fe040 msgr2=0x7fd2c0102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:29.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.197+0000 7fd2c7800700 1 -- 172.21.15.6:0/2213519431 shutdown_connections 2024-04-26T01:38:29.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:38:29.197+0000 7fd2c7800700 1 -- 172.21.15.6:0/2213519431 wait complete. 2024-04-26T01:38:30.050 DEBUG:teuthology.orchestra.run.smithi006:osd.6> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T01:38:30.056 INFO:tasks.cephadm:Deploying osd.7 on smithi029 with /dev/vg_nvme/lv_2... 2024-04-26T01:38:30.056 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T01:38:30.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:29 smithi029 ceph-mon[32775]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T01:38:30.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:29 smithi029 ceph-mon[32775]: from='osd.6 [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:38:30.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:29 smithi029 ceph-mon[32775]: osdmap e46: 7 total, 6 up, 7 in 2024-04-26T01:38:30.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:29 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:30.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:29 smithi029 ceph-mon[32775]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:38:30.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:29 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:30.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:29 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:30.298 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:29 smithi006 ceph-mon[28419]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T01:38:30.298 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:29 smithi006 ceph-mon[28419]: from='osd.6 [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:38:30.299 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:29 smithi006 ceph-mon[28419]: osdmap e46: 7 total, 6 up, 7 in 2024-04-26T01:38:30.299 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:29 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:30.299 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:29 smithi006 ceph-mon[28419]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:38:30.299 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:29 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:30.299 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:29 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:30.312 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:38:30.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:29 smithi188 ceph-mon[32751]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T01:38:30.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:29 smithi188 ceph-mon[32751]: from='osd.6 [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:38:30.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:29 smithi188 ceph-mon[32751]: osdmap e46: 7 total, 6 up, 7 in 2024-04-26T01:38:30.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:29 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:30.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:29 smithi188 ceph-mon[32751]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:38:30.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:29 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:30.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:29 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:31.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:30 smithi029 ceph-mon[32775]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:31.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:30 smithi029 ceph-mon[32775]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-04-26T01:38:31.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:30 smithi029 ceph-mon[32775]: osdmap e47: 7 total, 6 up, 7 in 2024-04-26T01:38:31.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:31.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:31.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:30 smithi006 ceph-mon[28419]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:31.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:30 smithi006 ceph-mon[28419]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-04-26T01:38:31.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:30 smithi006 ceph-mon[28419]: osdmap e47: 7 total, 6 up, 7 in 2024-04-26T01:38:31.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:31.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:31.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:30 smithi188 ceph-mon[32751]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:38:31.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:30 smithi188 ceph-mon[32751]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-04-26T01:38:31.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:30 smithi188 ceph-mon[32751]: osdmap e47: 7 total, 6 up, 7 in 2024-04-26T01:38:31.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:31.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:31.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:31 smithi006 ceph-mon[28419]: purged_snaps scrub starts 2024-04-26T01:38:31.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:31 smithi006 ceph-mon[28419]: purged_snaps scrub ok 2024-04-26T01:38:31.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:31 smithi006 ceph-mon[28419]: osd.6 [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695] boot 2024-04-26T01:38:31.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:31 smithi006 ceph-mon[28419]: osdmap e48: 7 total, 7 up, 7 in 2024-04-26T01:38:31.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:31 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:32.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:31 smithi029 ceph-mon[32775]: purged_snaps scrub starts 2024-04-26T01:38:32.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:31 smithi029 ceph-mon[32775]: purged_snaps scrub ok 2024-04-26T01:38:32.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:31 smithi029 ceph-mon[32775]: osd.6 [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695] boot 2024-04-26T01:38:32.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:31 smithi029 ceph-mon[32775]: osdmap e48: 7 total, 7 up, 7 in 2024-04-26T01:38:32.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:31 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:32.348 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T01:38:32.348 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T01:38:32.348 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-04-26T01:38:32.348 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-04-26T01:38:32.349 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00844245 s, 1.2 GB/s 2024-04-26T01:38:32.349 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-04-26T01:38:32.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:31 smithi188 ceph-mon[32751]: purged_snaps scrub starts 2024-04-26T01:38:32.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:31 smithi188 ceph-mon[32751]: purged_snaps scrub ok 2024-04-26T01:38:32.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:31 smithi188 ceph-mon[32751]: osd.6 [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695] boot 2024-04-26T01:38:32.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:31 smithi188 ceph-mon[32751]: osdmap e48: 7 total, 7 up, 7 in 2024-04-26T01:38:32.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:31 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:38:33.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_2 2024-04-26T01:38:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:38:33.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:32 smithi006 ceph-mon[28419]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:33.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:32 smithi006 ceph-mon[28419]: osdmap e49: 7 total, 7 up, 7 in 2024-04-26T01:38:33.388 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:32 smithi029 ceph-mon[32775]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:33.388 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:32 smithi029 ceph-mon[32775]: osdmap e49: 7 total, 7 up, 7 in 2024-04-26T01:38:33.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:32 smithi188 ceph-mon[32751]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:33.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:32 smithi188 ceph-mon[32751]: osdmap e49: 7 total, 7 up, 7 in 2024-04-26T01:38:34.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.934+0000 7fdd82a56700 1 -- 172.21.15.29:0/1124147748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd7c102ba0 msgr2=0x7fdd7c103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:34.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.934+0000 7fdd82a56700 1 --2- 172.21.15.29:0/1124147748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd7c102ba0 0x7fdd7c103000 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdd700099c0 tx=0x7fdd70009cd0 comp rx=0 tx=0).stop 2024-04-26T01:38:34.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.934+0000 7fdd82a56700 1 -- 172.21.15.29:0/1124147748 shutdown_connections 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.934+0000 7fdd82a56700 1 --2- 172.21.15.29:0/1124147748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd7c103540 0x7fdd7c109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.934+0000 7fdd82a56700 1 --2- 172.21.15.29:0/1124147748 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd7c1019a0 0x7fdd7c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.934+0000 7fdd82a56700 1 --2- 172.21.15.29:0/1124147748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd7c102ba0 0x7fdd7c103000 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.934+0000 7fdd82a56700 1 -- 172.21.15.29:0/1124147748 >> 172.21.15.29:0/1124147748 conn(0x7fdd7c0fcf40 msgr2=0x7fdd7c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.934+0000 7fdd82a56700 1 -- 172.21.15.29:0/1124147748 shutdown_connections 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.934+0000 7fdd82a56700 1 -- 172.21.15.29:0/1124147748 wait complete. 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.935+0000 7fdd82a56700 1 Processor -- start 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.935+0000 7fdd82a56700 1 -- start start 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.935+0000 7fdd82a56700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd7c1019a0 0x7fdd7c1058e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.935+0000 7fdd82a56700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd7c102ba0 0x7fdd7c103f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.935+0000 7fdd82a56700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd7c103540 0x7fdd7c1044c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.935+0000 7fdd82a56700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c10d530 con 0x7fdd7c103540 2024-04-26T01:38:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.935+0000 7fdd82a56700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c10d3b0 con 0x7fdd7c1019a0 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.935+0000 7fdd82a56700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c10d6b0 con 0x7fdd7c102ba0 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.935+0000 7fdd7bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd7c1019a0 0x7fdd7c1058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.935+0000 7fdd7bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd7c1019a0 0x7fdd7c1058e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47444/0 (socket says 172.21.15.29:47444) 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.935+0000 7fdd7bfff700 1 -- 172.21.15.29:0/2170467043 learned_addr learned my addr 172.21.15.29:0/2170467043 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.935+0000 7fdd7b7fe700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd7c102ba0 0x7fdd7c103f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.936+0000 7fdd7bfff700 1 -- 172.21.15.29:0/2170467043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd7c102ba0 msgr2=0x7fdd7c103f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.936+0000 7fdd7bfff700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd7c102ba0 0x7fdd7c103f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.936+0000 7fdd7bfff700 1 -- 172.21.15.29:0/2170467043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd7c103540 msgr2=0x7fdd7c1044c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.936+0000 7fdd7bfff700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd7c103540 0x7fdd7c1044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.936+0000 7fdd7bfff700 1 -- 172.21.15.29:0/2170467043 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd7c104d90 con 0x7fdd7c1019a0 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.936+0000 7fdd7bfff700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd7c1019a0 0x7fdd7c1058e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00b500 tx=0x7fdd6c00b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.936+0000 7fdd797fa700 1 -- 172.21.15.29:0/2170467043 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c015020 con 0x7fdd7c1019a0 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.936+0000 7fdd797fa700 1 -- 172.21.15.29:0/2170467043 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd6c011ce0 con 0x7fdd7c1019a0 2024-04-26T01:38:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.936+0000 7fdd82a56700 1 -- 172.21.15.29:0/2170467043 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd7c1a2eb0 con 0x7fdd7c1019a0 2024-04-26T01:38:34.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.936+0000 7fdd797fa700 1 -- 172.21.15.29:0/2170467043 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c010d20 con 0x7fdd7c1019a0 2024-04-26T01:38:34.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.936+0000 7fdd82a56700 1 -- 172.21.15.29:0/2170467043 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd7c1a3320 con 0x7fdd7c1019a0 2024-04-26T01:38:34.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.937+0000 7fdd82a56700 1 -- 172.21.15.29:0/2170467043 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd7c069730 con 0x7fdd7c1019a0 2024-04-26T01:38:34.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.938+0000 7fdd797fa700 1 -- 172.21.15.29:0/2170467043 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fdd6c018020 con 0x7fdd7c1019a0 2024-04-26T01:38:34.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.939+0000 7fdd797fa700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdd640a3c40 0x7fdd640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:34.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.939+0000 7fdd797fa700 1 -- 172.21.15.29:0/2170467043 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4965+0+0 (secure 0 0 0) 0x7fdd6c0cee80 con 0x7fdd7c1019a0 2024-04-26T01:38:34.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.939+0000 7fdd7b7fe700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdd640a3c40 0x7fdd640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:34.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.940+0000 7fdd7b7fe700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdd640a3c40 0x7fdd640a6100 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009990 tx=0x7fdd70005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:34.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:34.944+0000 7fdd797fa700 1 -- 172.21.15.29:0/2170467043 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd6c01f020 con 0x7fdd7c1019a0 2024-04-26T01:38:35.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:35.165+0000 7fdd82a56700 1 -- 172.21.15.29:0/2170467043 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fdd7c06ae30 con 0x7fdd640a3c40 2024-04-26T01:38:35.163 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:34 smithi029 ceph-mon[32775]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:35.163 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:35.163 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:35.163 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:35.163 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:35.163 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:35.163 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:35.163 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:35.163 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:35.163 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:34 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:35.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:34 smithi006 ceph-mon[28419]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:35.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:35.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:35.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:35.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:35.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:35.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:35.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:35.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:35.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:34 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:35.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:34 smithi188 ceph-mon[32751]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:35.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:35.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:35.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:35.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:35.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:35.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:35.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:35.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:35.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:34 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:36.087 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:35 smithi029 ceph-mon[32775]: Detected new or changed devices on smithi006 2024-04-26T01:38:36.087 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:35 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi006 to 5698M 2024-04-26T01:38:36.088 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:36.088 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:36.088 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:36.088 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:36.088 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:38:36.089 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:38:36.089 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:36.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:35 smithi006 ceph-mon[28419]: Detected new or changed devices on smithi006 2024-04-26T01:38:36.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:35 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi006 to 5698M 2024-04-26T01:38:36.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:36.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:36.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:36.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:36.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:38:36.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:38:36.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:36.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:35 smithi188 ceph-mon[32751]: Detected new or changed devices on smithi006 2024-04-26T01:38:36.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:35 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi006 to 5698M 2024-04-26T01:38:36.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:36.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:36.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:36.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:36.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:38:36.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:38:36.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:37.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:36 smithi029 ceph-mon[32775]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:37.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:36 smithi029 ceph-mon[32775]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:38:37.256 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:36 smithi006 ceph-mon[28419]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:37.257 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:36 smithi006 ceph-mon[28419]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:38:37.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:36 smithi188 ceph-mon[32751]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:37.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:36 smithi188 ceph-mon[32751]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:38:39.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:38 smithi006 ceph-mon[28419]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:39.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:38 smithi006 ceph-mon[28419]: from='client.? 172.21.15.29:0/2695003246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b7e6784-d574-42da-8416-a74a3b6b3755"}]: dispatch 2024-04-26T01:38:39.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:38 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b7e6784-d574-42da-8416-a74a3b6b3755"}]: dispatch 2024-04-26T01:38:39.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:38 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b7e6784-d574-42da-8416-a74a3b6b3755"}]': finished 2024-04-26T01:38:39.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:38 smithi006 ceph-mon[28419]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T01:38:39.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:38 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:38:39.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:38 smithi006 ceph-mon[28419]: from='client.? 172.21.15.29:0/250549422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:38:39.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:38 smithi188 ceph-mon[32751]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:39.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:38 smithi188 ceph-mon[32751]: from='client.? 172.21.15.29:0/2695003246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b7e6784-d574-42da-8416-a74a3b6b3755"}]: dispatch 2024-04-26T01:38:39.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:38 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b7e6784-d574-42da-8416-a74a3b6b3755"}]: dispatch 2024-04-26T01:38:39.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:38 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b7e6784-d574-42da-8416-a74a3b6b3755"}]': finished 2024-04-26T01:38:39.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:38 smithi188 ceph-mon[32751]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T01:38:39.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:38 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:38:39.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:38 smithi188 ceph-mon[32751]: from='client.? 172.21.15.29:0/250549422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:38:39.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:38 smithi029 ceph-mon[32775]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:39.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:38 smithi029 ceph-mon[32775]: from='client.? 172.21.15.29:0/2695003246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b7e6784-d574-42da-8416-a74a3b6b3755"}]: dispatch 2024-04-26T01:38:39.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:38 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b7e6784-d574-42da-8416-a74a3b6b3755"}]: dispatch 2024-04-26T01:38:39.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:38 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b7e6784-d574-42da-8416-a74a3b6b3755"}]': finished 2024-04-26T01:38:39.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:38 smithi029 ceph-mon[32775]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T01:38:39.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:38 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:38:39.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:38 smithi029 ceph-mon[32775]: from='client.? 172.21.15.29:0/250549422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:38:40.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:39 smithi006 ceph-mon[28419]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:40.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:39 smithi188 ceph-mon[32751]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:40.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:39 smithi029 ceph-mon[32775]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:42.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:42 smithi029 ceph-mon[32775]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:42.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:42 smithi006 ceph-mon[28419]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:42.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:42 smithi188 ceph-mon[32751]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:44.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:44 smithi029 ceph-mon[32775]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:44.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:44 smithi006 ceph-mon[28419]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:44.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:44 smithi188 ceph-mon[32751]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:46.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:46 smithi029 ceph-mon[32775]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:46.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:46 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T01:38:46.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:46 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:46.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:46 smithi029 ceph-mon[32775]: Deploying daemon osd.7 on smithi029 2024-04-26T01:38:46.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:46 smithi006 ceph-mon[28419]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:46.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T01:38:46.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:46.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:46 smithi006 ceph-mon[28419]: Deploying daemon osd.7 on smithi029 2024-04-26T01:38:46.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:46 smithi188 ceph-mon[32751]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:46.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T01:38:46.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:46.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:46 smithi188 ceph-mon[32751]: Deploying daemon osd.7 on smithi029 2024-04-26T01:38:48.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:48 smithi029 ceph-mon[32775]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:48.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:48 smithi006 ceph-mon[28419]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:48.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:48 smithi188 ceph-mon[32751]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:50.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:50 smithi029 ceph-mon[32775]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:50.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:50 smithi006 ceph-mon[28419]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:50.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:50 smithi188 ceph-mon[32751]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:52.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:52 smithi188 ceph-mon[32751]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:52.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:52 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:52.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:52 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:52.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:52 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:52.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:52 smithi029 ceph-mon[32775]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:52.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:52 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:52.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:52 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:52.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:52 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:52.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:52 smithi006 ceph-mon[28419]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:52.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:52 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:52.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:52 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:52.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:52 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:54.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:54 smithi188 ceph-mon[32751]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:54.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:54 smithi029 ceph-mon[32775]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:54.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:54 smithi006 ceph-mon[28419]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:55.446 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 7 on host 'smithi029' 2024-04-26T01:38:55.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.450+0000 7fdd797fa700 1 -- 172.21.15.29:0/2170467043 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdd7c06ae30 con 0x7fdd640a3c40 2024-04-26T01:38:55.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.452+0000 7fdd82a56700 1 -- 172.21.15.29:0/2170467043 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdd640a3c40 msgr2=0x7fdd640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:55.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.452+0000 7fdd82a56700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdd640a3c40 0x7fdd640a6100 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009990 tx=0x7fdd70005c20 comp rx=0 tx=0).stop 2024-04-26T01:38:55.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.452+0000 7fdd82a56700 1 -- 172.21.15.29:0/2170467043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd7c1019a0 msgr2=0x7fdd7c1058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.452+0000 7fdd82a56700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd7c1019a0 0x7fdd7c1058e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00b500 tx=0x7fdd6c00b810 comp rx=0 tx=0).stop 2024-04-26T01:38:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.452+0000 7fdd82a56700 1 -- 172.21.15.29:0/2170467043 shutdown_connections 2024-04-26T01:38:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.452+0000 7fdd82a56700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd7c103540 0x7fdd7c1044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.452+0000 7fdd82a56700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd7c1019a0 0x7fdd7c1058e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.452+0000 7fdd82a56700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdd640a3c40 0x7fdd640a6100 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.452+0000 7fdd82a56700 1 --2- 172.21.15.29:0/2170467043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd7c102ba0 0x7fdd7c103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.452+0000 7fdd82a56700 1 -- 172.21.15.29:0/2170467043 >> 172.21.15.29:0/2170467043 conn(0x7fdd7c0fcf40 msgr2=0x7fdd7c06a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.452+0000 7fdd82a56700 1 -- 172.21.15.29:0/2170467043 shutdown_connections 2024-04-26T01:38:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:38:55.452+0000 7fdd82a56700 1 -- 172.21.15.29:0/2170467043 wait complete. 2024-04-26T01:38:55.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:55 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:55 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:55 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:55.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:55 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:55.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:55 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:55 smithi029 ceph-mon[32775]: from='osd.7 [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:38:55.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:55 smithi029 ceph-mon[32775]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:38:55.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:55 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:55 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:55.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:55 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:55 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:55 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:55 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:55.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:55 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:55.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:55 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:55 smithi006 ceph-mon[28419]: from='osd.7 [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:38:55.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:55 smithi006 ceph-mon[28419]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:38:55.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:55 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:55 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:55.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:55 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:55 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:55 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:55 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:55.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:55 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:55.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:55 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:55 smithi188 ceph-mon[32751]: from='osd.7 [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:38:55.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:55 smithi188 ceph-mon[32751]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:38:55.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:55 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:55.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:55 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:55.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:55 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:38:56.114 DEBUG:teuthology.orchestra.run.smithi029:osd.7> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T01:38:56.117 INFO:tasks.cephadm:Deploying osd.8 on smithi188 with /dev/vg_nvme/lv_2... 2024-04-26T01:38:56.117 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T01:38:56.361 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:38:56.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:56 smithi029 ceph-mon[32775]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:56.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:56 smithi029 ceph-mon[32775]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T01:38:56.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:56 smithi029 ceph-mon[32775]: from='osd.7 [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:38:56.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:56 smithi029 ceph-mon[32775]: osdmap e51: 8 total, 7 up, 8 in 2024-04-26T01:38:56.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:56 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:38:56.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:56 smithi029 ceph-mon[32775]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:38:56.690 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 01:38:56 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T01:38:56.596+0000 7f9043945700 -1 osd.7 0 waiting for initial osdmap 2024-04-26T01:38:56.690 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 01:38:56 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T01:38:56.606+0000 7f903f738700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:38:56.768 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:56 smithi188 ceph-mon[32751]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:56.768 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:56 smithi188 ceph-mon[32751]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T01:38:56.768 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:56 smithi188 ceph-mon[32751]: from='osd.7 [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:38:56.768 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:56 smithi188 ceph-mon[32751]: osdmap e51: 8 total, 7 up, 8 in 2024-04-26T01:38:56.768 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:38:56.768 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:56 smithi188 ceph-mon[32751]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:38:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:56 smithi006 ceph-mon[28419]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:56 smithi006 ceph-mon[28419]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T01:38:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:56 smithi006 ceph-mon[28419]: from='osd.7 [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:38:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:56 smithi006 ceph-mon[28419]: osdmap e51: 8 total, 7 up, 8 in 2024-04-26T01:38:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:38:57.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:56 smithi006 ceph-mon[28419]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:38:57.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:57 smithi188 ceph-mon[32751]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-26T01:38:57.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:57 smithi188 ceph-mon[32751]: osdmap e52: 8 total, 7 up, 8 in 2024-04-26T01:38:57.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:38:57.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:57 smithi029 ceph-mon[32775]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-26T01:38:57.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:57 smithi029 ceph-mon[32775]: osdmap e52: 8 total, 7 up, 8 in 2024-04-26T01:38:57.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:38:58.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:57 smithi006 ceph-mon[28419]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-26T01:38:58.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:57 smithi006 ceph-mon[28419]: osdmap e52: 8 total, 7 up, 8 in 2024-04-26T01:38:58.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:38:58.293 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T01:38:58.294 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T01:38:58.294 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-26T01:38:58.294 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-26T01:38:58.294 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128579 s, 816 MB/s 2024-04-26T01:38:58.294 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-26T01:38:58.790 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:58 smithi029 ceph-mon[32775]: purged_snaps scrub starts 2024-04-26T01:38:58.790 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:58 smithi029 ceph-mon[32775]: purged_snaps scrub ok 2024-04-26T01:38:58.790 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:58 smithi029 ceph-mon[32775]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:58.790 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:58 smithi029 ceph-mon[32775]: osd.7 [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037] boot 2024-04-26T01:38:58.790 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:58 smithi029 ceph-mon[32775]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T01:38:58.791 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:58 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:38:58.791 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:38:58 smithi029 ceph-mon[32775]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T01:38:58.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:58 smithi188 ceph-mon[32751]: purged_snaps scrub starts 2024-04-26T01:38:58.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:58 smithi188 ceph-mon[32751]: purged_snaps scrub ok 2024-04-26T01:38:58.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:58 smithi188 ceph-mon[32751]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:58.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:58 smithi188 ceph-mon[32751]: osd.7 [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037] boot 2024-04-26T01:38:58.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:58 smithi188 ceph-mon[32751]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T01:38:58.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:58 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:38:58.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:38:58 smithi188 ceph-mon[32751]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T01:38:58.863 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_2 2024-04-26T01:38:59.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:58 smithi006 ceph-mon[28419]: purged_snaps scrub starts 2024-04-26T01:38:59.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:58 smithi006 ceph-mon[28419]: purged_snaps scrub ok 2024-04-26T01:38:59.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:58 smithi006 ceph-mon[28419]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:38:59.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:58 smithi006 ceph-mon[28419]: osd.7 [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037] boot 2024-04-26T01:38:59.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:58 smithi006 ceph-mon[28419]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T01:38:59.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:58 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:38:59.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:38:58 smithi006 ceph-mon[28419]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T01:38:59.101 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:39:00.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:00.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: Detected new or changed devices on smithi029 2024-04-26T01:39:00.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:00.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:00.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:00.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:00.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:00.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi029 to 5697M 2024-04-26T01:39:00.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:00.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:00.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:00.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:00.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T01:39:00.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-26T01:39:00.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:00 smithi188 ceph-mon[32751]: osdmap e56: 8 total, 8 up, 8 in 2024-04-26T01:39:00.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:00.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: Detected new or changed devices on smithi029 2024-04-26T01:39:00.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:00.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi029 to 5697M 2024-04-26T01:39:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T01:39:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-26T01:39:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:00 smithi029 ceph-mon[32775]: osdmap e56: 8 total, 8 up, 8 in 2024-04-26T01:39:00.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: Detected new or changed devices on smithi029 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi029 to 5697M 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T01:39:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-26T01:39:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:00 smithi006 ceph-mon[28419]: osdmap e56: 8 total, 8 up, 8 in 2024-04-26T01:39:00.716 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.719+0000 7f7e9cbda700 1 -- 172.21.15.188:0/2503067870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e98103ca0 msgr2=0x7f7e98104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:00.716 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.719+0000 7f7e9cbda700 1 --2- 172.21.15.188:0/2503067870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e98103ca0 0x7f7e98104100 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7e88009990 tx=0x7f7e88009ca0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.719+0000 7f7e9cbda700 1 -- 172.21.15.188:0/2503067870 shutdown_connections 2024-04-26T01:39:00.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.719+0000 7f7e9cbda700 1 --2- 172.21.15.188:0/2503067870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e98104640 0x7f7e9810aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.719+0000 7f7e9cbda700 1 --2- 172.21.15.188:0/2503067870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e98102aa0 0x7f7e98102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.719+0000 7f7e9cbda700 1 --2- 172.21.15.188:0/2503067870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e98103ca0 0x7f7e98104100 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.719+0000 7f7e9cbda700 1 -- 172.21.15.188:0/2503067870 >> 172.21.15.188:0/2503067870 conn(0x7f7e980fe040 msgr2=0x7f7e98100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:00.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.719+0000 7f7e9cbda700 1 -- 172.21.15.188:0/2503067870 shutdown_connections 2024-04-26T01:39:00.718 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.719+0000 7f7e9cbda700 1 -- 172.21.15.188:0/2503067870 wait complete. 2024-04-26T01:39:00.718 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.719+0000 7f7e9cbda700 1 Processor -- start 2024-04-26T01:39:00.718 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e9cbda700 1 -- start start 2024-04-26T01:39:00.720 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e9cbda700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e98102aa0 0x7f7e98198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:00.720 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e9cbda700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e98103ca0 0x7f7e981991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:00.720 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e9cbda700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e98104640 0x7f7e981a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:00.720 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e9cbda700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9810e5b0 con 0x7f7e98102aa0 2024-04-26T01:39:00.720 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e9cbda700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9810e430 con 0x7f7e98103ca0 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e9cbda700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9810e730 con 0x7f7e98104640 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e96d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e98104640 0x7f7e981a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e96d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e98104640 0x7f7e981a0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:35670/0 (socket says 172.21.15.188:35670) 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e96d9d700 1 -- 172.21.15.188:0/3980905573 learned_addr learned my addr 172.21.15.188:0/3980905573 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e95d9b700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e98103ca0 0x7f7e981991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e96d9d700 1 -- 172.21.15.188:0/3980905573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e98103ca0 msgr2=0x7f7e981991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e96d9d700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e98103ca0 0x7f7e981991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e96d9d700 1 -- 172.21.15.188:0/3980905573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e98102aa0 msgr2=0x7f7e98198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e96d9d700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e98102aa0 0x7f7e98198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e96d9d700 1 -- 172.21.15.188:0/3980905573 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e981a0900 con 0x7f7e98104640 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e96d9d700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e98104640 0x7f7e981a0230 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7e90007ee0 tx=0x7f7e9000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.720+0000 7f7e8f7fe700 1 -- 172.21.15.188:0/3980905573 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9000dbd0 con 0x7f7e98104640 2024-04-26T01:39:00.721 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.721+0000 7f7e8f7fe700 1 -- 172.21.15.188:0/3980905573 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e9000dd70 con 0x7f7e98104640 2024-04-26T01:39:00.722 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.721+0000 7f7e9cbda700 1 -- 172.21.15.188:0/3980905573 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e981a0bf0 con 0x7f7e98104640 2024-04-26T01:39:00.722 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.721+0000 7f7e8f7fe700 1 -- 172.21.15.188:0/3980905573 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e90005990 con 0x7f7e98104640 2024-04-26T01:39:00.722 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.721+0000 7f7e9cbda700 1 -- 172.21.15.188:0/3980905573 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e981a10e0 con 0x7f7e98104640 2024-04-26T01:39:00.722 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.722+0000 7f7e9cbda700 1 -- 172.21.15.188:0/3980905573 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e98069770 con 0x7f7e98104640 2024-04-26T01:39:00.722 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.723+0000 7f7e8f7fe700 1 -- 172.21.15.188:0/3980905573 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f7e9001a020 con 0x7f7e98104640 2024-04-26T01:39:00.722 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.723+0000 7f7e8f7fe700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7e840a3c40 0x7f7e840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:00.722 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.723+0000 7f7e8f7fe700 1 -- 172.21.15.188:0/3980905573 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5397+0+0 (secure 0 0 0) 0x7f7e900cf010 con 0x7f7e98104640 2024-04-26T01:39:00.723 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.724+0000 7f7e9659c700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7e840a3c40 0x7f7e840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:00.723 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.725+0000 7f7e9659c700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7e840a3c40 0x7f7e840a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e98199b60 tx=0x7f7e80006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:00.724 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.728+0000 7f7e8f7fe700 1 -- 172.21.15.188:0/3980905573 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e90010070 con 0x7f7e98104640 2024-04-26T01:39:00.933 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:00.936+0000 7f7e9cbda700 1 -- 172.21.15.188:0/3980905573 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7e98062e80 con 0x7f7e840a3c40 2024-04-26T01:39:02.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:01 smithi188 ceph-mon[32751]: osdmap e57: 8 total, 8 up, 8 in 2024-04-26T01:39:02.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:01 smithi188 ceph-mon[32751]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:39:02.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:01 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:39:02.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:01 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:39:02.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:01 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:02.186 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:01 smithi006 ceph-mon[28419]: osdmap e57: 8 total, 8 up, 8 in 2024-04-26T01:39:02.187 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:01 smithi006 ceph-mon[28419]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:39:02.187 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:01 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:39:02.187 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:01 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:39:02.187 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:01 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:02.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:01 smithi029 ceph-mon[32775]: osdmap e57: 8 total, 8 up, 8 in 2024-04-26T01:39:02.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:01 smithi029 ceph-mon[32775]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:39:02.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:01 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:39:02.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:01 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:39:02.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:01 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:03.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:02 smithi188 ceph-mon[32751]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:03.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:02 smithi029 ceph-mon[32775]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:03.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:02 smithi006 ceph-mon[28419]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:04.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:03 smithi029 ceph-mon[32775]: from='client.? 172.21.15.188:0/2557578242' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30cd2ea1-2767-4062-8bd9-47214e4b4b9e"}]: dispatch 2024-04-26T01:39:04.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:03 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30cd2ea1-2767-4062-8bd9-47214e4b4b9e"}]: dispatch 2024-04-26T01:39:04.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:03 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30cd2ea1-2767-4062-8bd9-47214e4b4b9e"}]': finished 2024-04-26T01:39:04.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:03 smithi029 ceph-mon[32775]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T01:39:04.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:03 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:04.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:03 smithi006 ceph-mon[28419]: from='client.? 172.21.15.188:0/2557578242' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30cd2ea1-2767-4062-8bd9-47214e4b4b9e"}]: dispatch 2024-04-26T01:39:04.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:03 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30cd2ea1-2767-4062-8bd9-47214e4b4b9e"}]: dispatch 2024-04-26T01:39:04.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:03 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30cd2ea1-2767-4062-8bd9-47214e4b4b9e"}]': finished 2024-04-26T01:39:04.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:03 smithi006 ceph-mon[28419]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T01:39:04.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:03 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:04.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:03 smithi188 ceph-mon[32751]: from='client.? 172.21.15.188:0/2557578242' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30cd2ea1-2767-4062-8bd9-47214e4b4b9e"}]: dispatch 2024-04-26T01:39:04.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:03 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30cd2ea1-2767-4062-8bd9-47214e4b4b9e"}]: dispatch 2024-04-26T01:39:04.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:03 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30cd2ea1-2767-4062-8bd9-47214e4b4b9e"}]': finished 2024-04-26T01:39:04.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:03 smithi188 ceph-mon[32751]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T01:39:04.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:03 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:05.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:04 smithi029 ceph-mon[32775]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:05.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:04 smithi029 ceph-mon[32775]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-26T01:39:05.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:04 smithi029 ceph-mon[32775]: Cluster is now healthy 2024-04-26T01:39:05.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:04 smithi029 ceph-mon[32775]: from='client.? 172.21.15.188:0/3628692002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:39:05.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:04 smithi006 ceph-mon[28419]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:05.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:04 smithi006 ceph-mon[28419]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-26T01:39:05.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:04 smithi006 ceph-mon[28419]: Cluster is now healthy 2024-04-26T01:39:05.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:04 smithi006 ceph-mon[28419]: from='client.? 172.21.15.188:0/3628692002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:39:05.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:04 smithi188 ceph-mon[32751]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:05.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:04 smithi188 ceph-mon[32751]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-26T01:39:05.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:04 smithi188 ceph-mon[32751]: Cluster is now healthy 2024-04-26T01:39:05.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:04 smithi188 ceph-mon[32751]: from='client.? 172.21.15.188:0/3628692002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:39:07.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:06 smithi029 ceph-mon[32775]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:07.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:06 smithi006 ceph-mon[28419]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:07.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:06 smithi188 ceph-mon[32751]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:09.045 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:08 smithi188 ceph-mon[32751]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:09.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:08 smithi029 ceph-mon[32775]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:09.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:08 smithi006 ceph-mon[28419]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:11.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:10 smithi188 ceph-mon[32751]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:11.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:10 smithi029 ceph-mon[32775]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:11.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:10 smithi006 ceph-mon[28419]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:12.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:11 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T01:39:12.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:11 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:12.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:11 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T01:39:12.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:11 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:12.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:11 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T01:39:12.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:11 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:13.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:12 smithi188 ceph-mon[32751]: Deploying daemon osd.8 on smithi188 2024-04-26T01:39:13.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:12 smithi188 ceph-mon[32751]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:13.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:12 smithi006 ceph-mon[28419]: Deploying daemon osd.8 on smithi188 2024-04-26T01:39:13.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:12 smithi006 ceph-mon[28419]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:13.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:12 smithi029 ceph-mon[32775]: Deploying daemon osd.8 on smithi188 2024-04-26T01:39:13.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:12 smithi029 ceph-mon[32775]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:15.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:14 smithi188 ceph-mon[32751]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:15.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:14 smithi006 ceph-mon[28419]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:15.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:14 smithi029 ceph-mon[32775]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:17.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:16 smithi188 ceph-mon[32751]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:17.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:17.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:17.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:17.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:16 smithi006 ceph-mon[28419]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:17.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:17.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:17.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:17.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:16 smithi029 ceph-mon[32775]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:17.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:17.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:17.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:18 smithi188 ceph-mon[32751]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:19.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:18 smithi006 ceph-mon[28419]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:19.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:18 smithi029 ceph-mon[32775]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:19.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.923+0000 7f7e8f7fe700 1 -- 172.21.15.188:0/3980905573 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7e98062e80 con 0x7f7e840a3c40 2024-04-26T01:39:19.920 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 8 on host 'smithi188' 2024-04-26T01:39:19.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.926+0000 7f7e9cbda700 1 -- 172.21.15.188:0/3980905573 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7e840a3c40 msgr2=0x7f7e840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:19.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.926+0000 7f7e9cbda700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7e840a3c40 0x7f7e840a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e98199b60 tx=0x7f7e80006d20 comp rx=0 tx=0).stop 2024-04-26T01:39:19.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.926+0000 7f7e9cbda700 1 -- 172.21.15.188:0/3980905573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e98104640 msgr2=0x7f7e981a0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:19.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.926+0000 7f7e9cbda700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e98104640 0x7f7e981a0230 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7e90007ee0 tx=0x7f7e9000d3b0 comp rx=0 tx=0).stop 2024-04-26T01:39:19.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.926+0000 7f7e9cbda700 1 -- 172.21.15.188:0/3980905573 shutdown_connections 2024-04-26T01:39:19.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.926+0000 7f7e9cbda700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7e840a3c40 0x7f7e840a6100 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:19.923 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.926+0000 7f7e9cbda700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e98104640 0x7f7e981a0230 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:19.923 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.926+0000 7f7e9cbda700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e98102aa0 0x7f7e98198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:19.923 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.926+0000 7f7e9cbda700 1 --2- 172.21.15.188:0/3980905573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e98103ca0 0x7f7e981991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:19.923 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.926+0000 7f7e9cbda700 1 -- 172.21.15.188:0/3980905573 >> 172.21.15.188:0/3980905573 conn(0x7f7e980fe040 msgr2=0x7f7e98107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:19.923 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.926+0000 7f7e9cbda700 1 -- 172.21.15.188:0/3980905573 shutdown_connections 2024-04-26T01:39:19.923 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:39:19.926+0000 7f7e9cbda700 1 -- 172.21.15.188:0/3980905573 wait complete. 2024-04-26T01:39:20.531 DEBUG:teuthology.orchestra.run.smithi188:osd.8> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.8.service 2024-04-26T01:39:20.533 INFO:tasks.cephadm:Deploying osd.9 on smithi006 with /dev/vg_nvme/lv_1... 2024-04-26T01:39:20.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T01:39:20.998 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:21.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:20 smithi188 ceph-mon[32751]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:21.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:21.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:21.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:21.135 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:21.135 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:20 smithi188 ceph-mon[32751]: from='osd.8 [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:39:21.135 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:20 smithi188 ceph-mon[32751]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:39:21.147 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:20 smithi006 ceph-mon[28419]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:21.147 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:21.148 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:21.148 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:21.148 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:21.148 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:20 smithi006 ceph-mon[28419]: from='osd.8 [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:39:21.148 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:20 smithi006 ceph-mon[28419]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:39:21.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:20 smithi029 ceph-mon[32775]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:21.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:21.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:21.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:21.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:21.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:20 smithi029 ceph-mon[32775]: from='osd.8 [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:39:21.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:20 smithi029 ceph-mon[32775]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:39:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:22 smithi188 ceph-mon[32751]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:22 smithi188 ceph-mon[32751]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T01:39:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:22 smithi188 ceph-mon[32751]: from='osd.8 [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:39:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:22 smithi188 ceph-mon[32751]: osdmap e59: 9 total, 8 up, 9 in 2024-04-26T01:39:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:22 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:22 smithi188 ceph-mon[32751]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:39:22.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:22 smithi006 ceph-mon[28419]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:22.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:22 smithi006 ceph-mon[28419]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T01:39:22.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:22 smithi006 ceph-mon[28419]: from='osd.8 [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:39:22.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:22 smithi006 ceph-mon[28419]: osdmap e59: 9 total, 8 up, 9 in 2024-04-26T01:39:22.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:22.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:22 smithi006 ceph-mon[28419]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:39:22.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:22 smithi029 ceph-mon[32775]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:22.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:22 smithi029 ceph-mon[32775]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T01:39:22.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:22 smithi029 ceph-mon[32775]: from='osd.8 [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:39:22.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:22 smithi029 ceph-mon[32775]: osdmap e59: 9 total, 8 up, 9 in 2024-04-26T01:39:22.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:22 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:22.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:22 smithi029 ceph-mon[32775]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:39:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:23 smithi006 ceph-mon[28419]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-26T01:39:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:23 smithi006 ceph-mon[28419]: osdmap e60: 9 total, 8 up, 9 in 2024-04-26T01:39:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:23 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:23 smithi006 ceph-mon[28419]: from='osd.8 ' entity='osd.8' 2024-04-26T01:39:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:23 smithi006 ceph-mon[28419]: osdmap e61: 9 total, 8 up, 9 in 2024-04-26T01:39:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:23 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:23.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:23 smithi188 ceph-mon[32751]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-26T01:39:23.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:23 smithi188 ceph-mon[32751]: osdmap e60: 9 total, 8 up, 9 in 2024-04-26T01:39:23.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:23 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:23.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:23 smithi188 ceph-mon[32751]: from='osd.8 ' entity='osd.8' 2024-04-26T01:39:23.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:23 smithi188 ceph-mon[32751]: osdmap e61: 9 total, 8 up, 9 in 2024-04-26T01:39:23.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:23 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:23.352 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 01:39:23 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T01:39:23.121+0000 7fd4aee5a700 -1 osd.8 0 waiting for initial osdmap 2024-04-26T01:39:23.352 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 01:39:23 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T01:39:23.245+0000 7fd4a6444700 -1 osd.8 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:39:23.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:23 smithi029 ceph-mon[32775]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-26T01:39:23.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:23 smithi029 ceph-mon[32775]: osdmap e60: 9 total, 8 up, 9 in 2024-04-26T01:39:23.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:23 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:23.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:23 smithi029 ceph-mon[32775]: from='osd.8 ' entity='osd.8' 2024-04-26T01:39:23.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:23 smithi029 ceph-mon[32775]: osdmap e61: 9 total, 8 up, 9 in 2024-04-26T01:39:23.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:23 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:23.978 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T01:39:23.978 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T01:39:23.978 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-04-26T01:39:23.978 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-04-26T01:39:23.978 INFO:teuthology.orchestra.run.smithi006.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0336725 s, 311 MB/s 2024-04-26T01:39:23.978 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-04-26T01:39:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: purged_snaps scrub starts 2024-04-26T01:39:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: purged_snaps scrub ok 2024-04-26T01:39:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: Detected new or changed devices on smithi188 2024-04-26T01:39:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi188 to 5696M 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: osd.8 [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376] boot 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T01:39:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: purged_snaps scrub starts 2024-04-26T01:39:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: purged_snaps scrub ok 2024-04-26T01:39:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: Detected new or changed devices on smithi188 2024-04-26T01:39:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi188 to 5696M 2024-04-26T01:39:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: osd.8 [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376] boot 2024-04-26T01:39:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T01:39:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:24.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: purged_snaps scrub starts 2024-04-26T01:39:24.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: purged_snaps scrub ok 2024-04-26T01:39:24.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:24.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: Detected new or changed devices on smithi188 2024-04-26T01:39:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi188 to 5696M 2024-04-26T01:39:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:24.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:24.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:24.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:24.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: osd.8 [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376] boot 2024-04-26T01:39:24.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T01:39:24.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:39:24.822 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi006:vg_nvme/lv_1 2024-04-26T01:39:25.287 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:26.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:25 smithi006 ceph-mon[28419]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T01:39:26.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:25 smithi029 ceph-mon[32775]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T01:39:26.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:25 smithi188 ceph-mon[32751]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T01:39:27.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:26 smithi029 ceph-mon[32775]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:27.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:26 smithi029 ceph-mon[32775]: osdmap e64: 9 total, 9 up, 9 in 2024-04-26T01:39:27.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:26 smithi006 ceph-mon[28419]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:27.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:26 smithi006 ceph-mon[28419]: osdmap e64: 9 total, 9 up, 9 in 2024-04-26T01:39:27.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:26 smithi188 ceph-mon[32751]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:39:27.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:26 smithi188 ceph-mon[32751]: osdmap e64: 9 total, 9 up, 9 in 2024-04-26T01:39:27.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.404+0000 7fb108200700 1 -- 172.21.15.6:0/2073889951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb100079000 msgr2=0x7fb100079880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:27.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.405+0000 7fb108200700 1 --2- 172.21.15.6:0/2073889951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb100079000 0x7fb100079880 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800b0a0 tx=0x7fb0f800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:39:27.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.405+0000 7fb108200700 1 -- 172.21.15.6:0/2073889951 shutdown_connections 2024-04-26T01:39:27.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.405+0000 7fb108200700 1 --2- 172.21.15.6:0/2073889951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb100079000 0x7fb100079880 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:27.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.405+0000 7fb108200700 1 --2- 172.21.15.6:0/2073889951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1000773f0 0x7fb100077810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:27.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.405+0000 7fb108200700 1 --2- 172.21.15.6:0/2073889951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb100078640 0x7fb100078ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:27.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.405+0000 7fb108200700 1 -- 172.21.15.6:0/2073889951 >> 172.21.15.6:0/2073889951 conn(0x7fb100100250 msgr2=0x7fb100102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:27.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.405+0000 7fb108200700 1 -- 172.21.15.6:0/2073889951 shutdown_connections 2024-04-26T01:39:27.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.405+0000 7fb108200700 1 -- 172.21.15.6:0/2073889951 wait complete. 2024-04-26T01:39:27.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.406+0000 7fb108200700 1 Processor -- start 2024-04-26T01:39:27.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.406+0000 7fb108200700 1 -- start start 2024-04-26T01:39:27.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.406+0000 7fb108200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1000773f0 0x7fb10019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:27.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.406+0000 7fb108200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb100078640 0x7fb10019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:27.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.406+0000 7fb108200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb100079000 0x7fb1001a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:27.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb108200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb100114b60 con 0x7fb1000773f0 2024-04-26T01:39:27.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb108200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb1001149e0 con 0x7fb100078640 2024-04-26T01:39:27.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb108200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb100114ce0 con 0x7fb100079000 2024-04-26T01:39:27.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb105800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1000773f0 0x7fb10019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:27.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb105800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1000773f0 0x7fb10019f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37016/0 (socket says 172.21.15.6:37016) 2024-04-26T01:39:27.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb105800700 1 -- 172.21.15.6:0/2766639351 learned_addr learned my addr 172.21.15.6:0/2766639351 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:39:27.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb104e00700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb100078640 0x7fb10019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:27.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb106200700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb100079000 0x7fb1001a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:27.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb105800700 1 -- 172.21.15.6:0/2766639351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb100079000 msgr2=0x7fb1001a6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:27.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb105800700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb100079000 0x7fb1001a6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:27.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb105800700 1 -- 172.21.15.6:0/2766639351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb100078640 msgr2=0x7fb10019f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:27.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb105800700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb100078640 0x7fb10019f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:27.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.407+0000 7fb105800700 1 -- 172.21.15.6:0/2766639351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1001a6da0 con 0x7fb1000773f0 2024-04-26T01:39:27.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.408+0000 7fb105800700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1000773f0 0x7fb10019f140 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400e9d0 tx=0x7fb0f400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:27.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.408+0000 7fb0fe000700 1 -- 172.21.15.6:0/2766639351 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f400cbd0 con 0x7fb1000773f0 2024-04-26T01:39:27.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.408+0000 7fb0fe000700 1 -- 172.21.15.6:0/2766639351 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0f400ee70 con 0x7fb1000773f0 2024-04-26T01:39:27.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.408+0000 7fb108200700 1 -- 172.21.15.6:0/2766639351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1001a7090 con 0x7fb1000773f0 2024-04-26T01:39:27.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.408+0000 7fb0fe000700 1 -- 172.21.15.6:0/2766639351 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4018680 con 0x7fb1000773f0 2024-04-26T01:39:27.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.408+0000 7fb108200700 1 -- 172.21.15.6:0/2766639351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1001a7620 con 0x7fb1000773f0 2024-04-26T01:39:27.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.409+0000 7fb108200700 1 -- 172.21.15.6:0/2766639351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb100069730 con 0x7fb1000773f0 2024-04-26T01:39:27.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.411+0000 7fb0fe000700 1 -- 172.21.15.6:0/2766639351 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fb0f4010430 con 0x7fb1000773f0 2024-04-26T01:39:27.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.413+0000 7fb0fe000700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb0e80a3cc0 0x7fb0e80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:27.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.413+0000 7fb0fe000700 1 -- 172.21.15.6:0/2766639351 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5829+0+0 (secure 0 0 0) 0x7fb0f40cffa0 con 0x7fb1000773f0 2024-04-26T01:39:27.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.413+0000 7fb104e00700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb0e80a3cc0 0x7fb0e80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:27.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.414+0000 7fb104e00700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb0e80a3cc0 0x7fb0e80a6180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec009c60 tx=0x7fb0ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.417+0000 7fb0fe000700 1 -- 172.21.15.6:0/2766639351 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb0f409d290 con 0x7fb1000773f0 2024-04-26T01:39:27.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:27.658+0000 7fb108200700 1 -- 172.21.15.6:0/2766639351 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb100062e80 con 0x7fb0e80a3cc0 2024-04-26T01:39:28.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:27 smithi029 ceph-mon[32775]: osdmap e65: 9 total, 9 up, 9 in 2024-04-26T01:39:28.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:27 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:39:28.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:27 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:39:28.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:27 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:28.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:27 smithi006 ceph-mon[28419]: osdmap e65: 9 total, 9 up, 9 in 2024-04-26T01:39:28.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:27 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:39:28.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:27 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:39:28.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:27 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:28.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:27 smithi188 ceph-mon[32751]: osdmap e65: 9 total, 9 up, 9 in 2024-04-26T01:39:28.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:27 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:39:28.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:27 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:39:28.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:27 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:29.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:28 smithi029 ceph-mon[32775]: pgmap v225: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:29.213 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:28 smithi029 ceph-mon[32775]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:39:29.213 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:28 smithi029 ceph-mon[32775]: osdmap e66: 9 total, 9 up, 9 in 2024-04-26T01:39:29.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:28 smithi006 ceph-mon[28419]: pgmap v225: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:29.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:28 smithi006 ceph-mon[28419]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:39:29.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:28 smithi006 ceph-mon[28419]: osdmap e66: 9 total, 9 up, 9 in 2024-04-26T01:39:29.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:28 smithi188 ceph-mon[32751]: pgmap v225: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:29.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:28 smithi188 ceph-mon[32751]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:39:29.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:28 smithi188 ceph-mon[32751]: osdmap e66: 9 total, 9 up, 9 in 2024-04-26T01:39:31.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:30 smithi006 ceph-mon[28419]: pgmap v227: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-04-26T01:39:31.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:30 smithi188 ceph-mon[32751]: pgmap v227: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-04-26T01:39:31.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:30 smithi029 ceph-mon[32775]: pgmap v227: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-04-26T01:39:32.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:31 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1766132368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f033cdc0-1178-4a7f-a338-69ff32416087"}]: dispatch 2024-04-26T01:39:32.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:31 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1766132368' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f033cdc0-1178-4a7f-a338-69ff32416087"}]': finished 2024-04-26T01:39:32.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:31 smithi006 ceph-mon[28419]: osdmap e67: 10 total, 9 up, 10 in 2024-04-26T01:39:32.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:31 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:32.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:31 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1766132368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f033cdc0-1178-4a7f-a338-69ff32416087"}]: dispatch 2024-04-26T01:39:32.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:31 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1766132368' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f033cdc0-1178-4a7f-a338-69ff32416087"}]': finished 2024-04-26T01:39:32.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:31 smithi188 ceph-mon[32751]: osdmap e67: 10 total, 9 up, 10 in 2024-04-26T01:39:32.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:31 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:32.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:31 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1766132368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f033cdc0-1178-4a7f-a338-69ff32416087"}]: dispatch 2024-04-26T01:39:32.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:31 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1766132368' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f033cdc0-1178-4a7f-a338-69ff32416087"}]': finished 2024-04-26T01:39:32.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:31 smithi029 ceph-mon[32775]: osdmap e67: 10 total, 9 up, 10 in 2024-04-26T01:39:32.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:31 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:33.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:32 smithi006 ceph-mon[28419]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:39:33.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:32 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/4009848179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:39:33.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:32 smithi188 ceph-mon[32751]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:39:33.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:32 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/4009848179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:39:33.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:32 smithi029 ceph-mon[32775]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:39:33.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:32 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/4009848179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:39:34.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:33 smithi006 ceph-mon[28419]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-26T01:39:34.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:33 smithi188 ceph-mon[32751]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-26T01:39:34.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:33 smithi029 ceph-mon[32775]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-26T01:39:36.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:36 smithi188 ceph-mon[32751]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:39:36.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:36 smithi029 ceph-mon[32775]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:39:36.535 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:36 smithi006 ceph-mon[28419]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:39:38.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:38 smithi029 ceph-mon[32775]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:38.459 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:38 smithi006 ceph-mon[28419]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:38.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:38 smithi188 ceph-mon[32751]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:40.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:40 smithi006 ceph-mon[28419]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:40.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:40 smithi029 ceph-mon[32775]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:40.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:40 smithi188 ceph-mon[32751]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:42.453 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:42 smithi006 ceph-mon[28419]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:42.453 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:42 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T01:39:42.453 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:42 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:42.453 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:42 smithi006 ceph-mon[28419]: Deploying daemon osd.9 on smithi006 2024-04-26T01:39:42.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:42 smithi188 ceph-mon[32751]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:42.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:42 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T01:39:42.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:42 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:42.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:42 smithi188 ceph-mon[32751]: Deploying daemon osd.9 on smithi006 2024-04-26T01:39:42.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:42 smithi029 ceph-mon[32775]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:42.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:42 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T01:39:42.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:42 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:42.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:42 smithi029 ceph-mon[32775]: Deploying daemon osd.9 on smithi006 2024-04-26T01:39:44.491 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:44 smithi006 ceph-mon[28419]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:44.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:44 smithi188 ceph-mon[32751]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:44.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:44 smithi029 ceph-mon[32775]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:46.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:46 smithi006 ceph-mon[28419]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:46.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:46 smithi188 ceph-mon[32751]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:46.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:46 smithi029 ceph-mon[32775]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:48.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:48 smithi006 ceph-mon[28419]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:48.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:48 smithi188 ceph-mon[32751]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:48.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:48 smithi029 ceph-mon[32775]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:50.660 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:50 smithi006 ceph-mon[28419]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:50.660 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:50.660 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:50.660 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:50.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:50 smithi188 ceph-mon[32751]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:50.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:50.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:50.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:50.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:50 smithi029 ceph-mon[32775]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:50.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:50.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:50.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:52.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:52 smithi006 ceph-mon[28419]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:52.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:52 smithi188 ceph-mon[32751]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:52.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:52 smithi029 ceph-mon[32775]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:54.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:53 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:54.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:53 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:54.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:53 smithi029 ceph-mon[32775]: from='osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T01:39:54.206 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:53 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:54.206 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:53 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:54.206 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:53 smithi006 ceph-mon[28419]: from='osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T01:39:54.352 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 9 on host 'smithi006' 2024-04-26T01:39:54.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.354+0000 7fb0fe000700 1 -- 172.21.15.6:0/2766639351 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb100062e80 con 0x7fb0e80a3cc0 2024-04-26T01:39:54.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.357+0000 7fb108200700 1 -- 172.21.15.6:0/2766639351 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb0e80a3cc0 msgr2=0x7fb0e80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:54.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.357+0000 7fb108200700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb0e80a3cc0 0x7fb0e80a6180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec009c60 tx=0x7fb0ec009260 comp rx=0 tx=0).stop 2024-04-26T01:39:54.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.357+0000 7fb108200700 1 -- 172.21.15.6:0/2766639351 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1000773f0 msgr2=0x7fb10019f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:54.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.357+0000 7fb108200700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1000773f0 0x7fb10019f140 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400e9d0 tx=0x7fb0f400ece0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.357+0000 7fb108200700 1 -- 172.21.15.6:0/2766639351 shutdown_connections 2024-04-26T01:39:54.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.357+0000 7fb108200700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb0e80a3cc0 0x7fb0e80a6180 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.357+0000 7fb108200700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb100079000 0x7fb1001a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.357+0000 7fb108200700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1000773f0 0x7fb10019f140 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.357+0000 7fb108200700 1 --2- 172.21.15.6:0/2766639351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb100078640 0x7fb10019f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.357+0000 7fb108200700 1 -- 172.21.15.6:0/2766639351 >> 172.21.15.6:0/2766639351 conn(0x7fb100100250 msgr2=0x7fb100101d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.358+0000 7fb108200700 1 -- 172.21.15.6:0/2766639351 shutdown_connections 2024-04-26T01:39:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:39:54.358+0000 7fb108200700 1 -- 172.21.15.6:0/2766639351 wait complete. 2024-04-26T01:39:54.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:53 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:54.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:53 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:54.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:53 smithi188 ceph-mon[32751]: from='osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T01:39:55.085 DEBUG:teuthology.orchestra.run.smithi006:osd.9> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.9.service 2024-04-26T01:39:55.090 INFO:tasks.cephadm:Deploying osd.10 on smithi029 with /dev/vg_nvme/lv_1... 2024-04-26T01:39:55.090 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T01:39:55.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:54 smithi006 ceph-mon[28419]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:55.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:54 smithi006 ceph-mon[28419]: from='osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T01:39:55.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:54 smithi006 ceph-mon[28419]: osdmap e68: 10 total, 9 up, 10 in 2024-04-26T01:39:55.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:54 smithi006 ceph-mon[28419]: from='osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:39:55.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:54 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:55.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:54 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:55.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:54 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:55.347 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:39:55.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:54 smithi188 ceph-mon[32751]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:55.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:54 smithi188 ceph-mon[32751]: from='osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T01:39:55.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:54 smithi188 ceph-mon[32751]: osdmap e68: 10 total, 9 up, 10 in 2024-04-26T01:39:55.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:54 smithi188 ceph-mon[32751]: from='osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:39:55.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:54 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:55.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:54 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:55.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:54 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:55.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:54 smithi029 ceph-mon[32775]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:55.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:54 smithi029 ceph-mon[32775]: from='osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T01:39:55.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:54 smithi029 ceph-mon[32775]: osdmap e68: 10 total, 9 up, 10 in 2024-04-26T01:39:55.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:54 smithi029 ceph-mon[32775]: from='osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-04-26T01:39:55.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:54 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:55.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:54 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:55.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:54 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:39:56.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:55 smithi029 ceph-mon[32775]: from='osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-04-26T01:39:56.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:55 smithi029 ceph-mon[32775]: osdmap e69: 10 total, 9 up, 10 in 2024-04-26T01:39:56.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:55 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:56.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:55 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:56.293 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:55 smithi006 ceph-mon[28419]: from='osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-04-26T01:39:56.293 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:55 smithi006 ceph-mon[28419]: osdmap e69: 10 total, 9 up, 10 in 2024-04-26T01:39:56.293 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:55 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:56.293 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:55 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:56.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:55 smithi188 ceph-mon[32751]: from='osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-04-26T01:39:56.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:55 smithi188 ceph-mon[32751]: osdmap e69: 10 total, 9 up, 10 in 2024-04-26T01:39:56.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:55 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:56.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:55 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:57.040 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:56 smithi029 ceph-mon[32775]: purged_snaps scrub starts 2024-04-26T01:39:57.040 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:56 smithi029 ceph-mon[32775]: purged_snaps scrub ok 2024-04-26T01:39:57.040 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:56 smithi029 ceph-mon[32775]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:57.040 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:56 smithi029 ceph-mon[32775]: osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912] boot 2024-04-26T01:39:57.040 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:56 smithi029 ceph-mon[32775]: osdmap e70: 10 total, 10 up, 10 in 2024-04-26T01:39:57.040 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:56 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:57.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:56 smithi006 ceph-mon[28419]: purged_snaps scrub starts 2024-04-26T01:39:57.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:56 smithi006 ceph-mon[28419]: purged_snaps scrub ok 2024-04-26T01:39:57.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:56 smithi006 ceph-mon[28419]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:57.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:56 smithi006 ceph-mon[28419]: osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912] boot 2024-04-26T01:39:57.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:56 smithi006 ceph-mon[28419]: osdmap e70: 10 total, 10 up, 10 in 2024-04-26T01:39:57.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:57.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:56 smithi188 ceph-mon[32751]: purged_snaps scrub starts 2024-04-26T01:39:57.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:56 smithi188 ceph-mon[32751]: purged_snaps scrub ok 2024-04-26T01:39:57.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:56 smithi188 ceph-mon[32751]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:39:57.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:56 smithi188 ceph-mon[32751]: osd.9 [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912] boot 2024-04-26T01:39:57.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:56 smithi188 ceph-mon[32751]: osdmap e70: 10 total, 10 up, 10 in 2024-04-26T01:39:57.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:39:57.523 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T01:39:57.524 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T01:39:57.524 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-04-26T01:39:57.524 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-04-26T01:39:57.524 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00855387 s, 1.2 GB/s 2024-04-26T01:39:57.524 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-04-26T01:39:58.173 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:58 smithi029 ceph-mon[32775]: osdmap e71: 10 total, 10 up, 10 in 2024-04-26T01:39:58.174 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:39:58 smithi029 ceph-mon[32775]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:58.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_1 2024-04-26T01:39:58.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:58 smithi006 ceph-mon[28419]: osdmap e71: 10 total, 10 up, 10 in 2024-04-26T01:39:58.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:39:58 smithi006 ceph-mon[28419]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:58.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:58 smithi188 ceph-mon[32751]: osdmap e71: 10 total, 10 up, 10 in 2024-04-26T01:39:58.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:39:58 smithi188 ceph-mon[32751]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:58.449 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:40:00.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.184+0000 7fea322fb700 1 -- 172.21.15.29:0/1557553169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea2c104640 msgr2=0x7fea2c10aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:00.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.184+0000 7fea322fb700 1 --2- 172.21.15.29:0/1557553169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea2c104640 0x7fea2c10aef0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fea2400b0d0 tx=0x7fea2400b3e0 comp rx=0 tx=0).stop 2024-04-26T01:40:00.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.184+0000 7fea322fb700 1 -- 172.21.15.29:0/1557553169 shutdown_connections 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.184+0000 7fea322fb700 1 --2- 172.21.15.29:0/1557553169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea2c104640 0x7fea2c10aef0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.184+0000 7fea322fb700 1 --2- 172.21.15.29:0/1557553169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea2c102aa0 0x7fea2c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.184+0000 7fea322fb700 1 --2- 172.21.15.29:0/1557553169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea2c103ca0 0x7fea2c104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.184+0000 7fea322fb700 1 -- 172.21.15.29:0/1557553169 >> 172.21.15.29:0/1557553169 conn(0x7fea2c0fe040 msgr2=0x7fea2c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.184+0000 7fea322fb700 1 -- 172.21.15.29:0/1557553169 shutdown_connections 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.184+0000 7fea322fb700 1 -- 172.21.15.29:0/1557553169 wait complete. 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.184+0000 7fea322fb700 1 Processor -- start 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea322fb700 1 -- start start 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea322fb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea2c102aa0 0x7fea2c0767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea322fb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea2c103ca0 0x7fea2c074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea322fb700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea2c104640 0x7fea2c075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea322fb700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c10e600 con 0x7fea2c104640 2024-04-26T01:40:00.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea322fb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c10e480 con 0x7fea2c102aa0 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea322fb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c10e780 con 0x7fea2c103ca0 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea2bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea2c102aa0 0x7fea2c0767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea2bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea2c102aa0 0x7fea2c0767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45388/0 (socket says 172.21.15.29:45388) 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea2bfff700 1 -- 172.21.15.29:0/3311606226 learned_addr learned my addr 172.21.15.29:0/3311606226 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea2b7fe700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea2c103ca0 0x7fea2c074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea2bfff700 1 -- 172.21.15.29:0/3311606226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea2c103ca0 msgr2=0x7fea2c074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea2bfff700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea2c103ca0 0x7fea2c074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea2bfff700 1 -- 172.21.15.29:0/3311606226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea2c104640 msgr2=0x7fea2c075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea2bfff700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea2c104640 0x7fea2c075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea2bfff700 1 -- 172.21.15.29:0/3311606226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2c075c20 con 0x7fea2c102aa0 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.185+0000 7fea2b7fe700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea2c103ca0 0x7fea2c074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.186+0000 7fea2bfff700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea2c102aa0 0x7fea2c0767e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00e950 tx=0x7fea1c00ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.186+0000 7fea297fa700 1 -- 172.21.15.29:0/3311606226 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c00cbd0 con 0x7fea2c102aa0 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.186+0000 7fea297fa700 1 -- 172.21.15.29:0/3311606226 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea1c0044e0 con 0x7fea2c102aa0 2024-04-26T01:40:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.186+0000 7fea322fb700 1 -- 172.21.15.29:0/3311606226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2c1a2dd0 con 0x7fea2c102aa0 2024-04-26T01:40:00.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.186+0000 7fea297fa700 1 -- 172.21.15.29:0/3311606226 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c010430 con 0x7fea2c102aa0 2024-04-26T01:40:00.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.186+0000 7fea322fb700 1 -- 172.21.15.29:0/3311606226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea2c1a3250 con 0x7fea2c102aa0 2024-04-26T01:40:00.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.187+0000 7fea322fb700 1 -- 172.21.15.29:0/3311606226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea2c02df80 con 0x7fea2c102aa0 2024-04-26T01:40:00.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.188+0000 7fea297fa700 1 -- 172.21.15.29:0/3311606226 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fea1c003ec0 con 0x7fea2c102aa0 2024-04-26T01:40:00.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.189+0000 7fea297fa700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fea140a3c40 0x7fea140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:00.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.189+0000 7fea297fa700 1 -- 172.21.15.29:0/3311606226 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6261+0+0 (secure 0 0 0) 0x7fea1c0d0090 con 0x7fea2c102aa0 2024-04-26T01:40:00.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.189+0000 7fea2b7fe700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fea140a3c40 0x7fea140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:00.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.190+0000 7fea2b7fe700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fea140a3c40 0x7fea140a6100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fea200099d0 tx=0x7fea20008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:00.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.193+0000 7fea297fa700 1 -- 172.21.15.29:0/3311606226 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea1c09d1d0 con 0x7fea2c102aa0 2024-04-26T01:40:00.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:00.412+0000 7fea322fb700 1 -- 172.21.15.29:0/3311606226 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fea2c062e80 con 0x7fea140a3c40 2024-04-26T01:40:01.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:01.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: Detected new or changed devices on smithi006 2024-04-26T01:40:01.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:01.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:01.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:01.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi006 to 4273M 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: overall HEALTH_OK 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:40:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:01.083 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:01.083 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: Detected new or changed devices on smithi006 2024-04-26T01:40:01.083 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi006 to 4273M 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: overall HEALTH_OK 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:01.084 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.085 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:40:01.085 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:40:01.085 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:01.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:01.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: Detected new or changed devices on smithi006 2024-04-26T01:40:01.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:01.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:01.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:01.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:01.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi006 to 4273M 2024-04-26T01:40:01.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:01.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: overall HEALTH_OK 2024-04-26T01:40:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:40:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:40:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:02.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:01 smithi006 ceph-mon[28419]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:02.086 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:01 smithi029 ceph-mon[32775]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:02.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:01 smithi188 ceph-mon[32751]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:03.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:02 smithi006 ceph-mon[28419]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:03.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:02 smithi188 ceph-mon[32751]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:03.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:02 smithi029 ceph-mon[32775]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:03.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:03 smithi029 ceph-mon[32775]: from='client.? 172.21.15.29:0/343523786' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04e9b5ef-ac0c-482b-a69e-a7121b36e7da"}]: dispatch 2024-04-26T01:40:03.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:03 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04e9b5ef-ac0c-482b-a69e-a7121b36e7da"}]: dispatch 2024-04-26T01:40:03.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:03 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "04e9b5ef-ac0c-482b-a69e-a7121b36e7da"}]': finished 2024-04-26T01:40:03.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:03 smithi029 ceph-mon[32775]: osdmap e72: 11 total, 10 up, 11 in 2024-04-26T01:40:03.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:03 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:04.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:03 smithi188 ceph-mon[32751]: from='client.? 172.21.15.29:0/343523786' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04e9b5ef-ac0c-482b-a69e-a7121b36e7da"}]: dispatch 2024-04-26T01:40:04.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:03 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04e9b5ef-ac0c-482b-a69e-a7121b36e7da"}]: dispatch 2024-04-26T01:40:04.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:03 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "04e9b5ef-ac0c-482b-a69e-a7121b36e7da"}]': finished 2024-04-26T01:40:04.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:03 smithi188 ceph-mon[32751]: osdmap e72: 11 total, 10 up, 11 in 2024-04-26T01:40:04.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:03 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:04.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:03 smithi006 ceph-mon[28419]: from='client.? 172.21.15.29:0/343523786' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04e9b5ef-ac0c-482b-a69e-a7121b36e7da"}]: dispatch 2024-04-26T01:40:04.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:03 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04e9b5ef-ac0c-482b-a69e-a7121b36e7da"}]: dispatch 2024-04-26T01:40:04.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:03 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "04e9b5ef-ac0c-482b-a69e-a7121b36e7da"}]': finished 2024-04-26T01:40:04.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:03 smithi006 ceph-mon[28419]: osdmap e72: 11 total, 10 up, 11 in 2024-04-26T01:40:04.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:03 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:05.092 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:04 smithi188 ceph-mon[32751]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:05.092 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:04 smithi188 ceph-mon[32751]: from='client.? 172.21.15.29:0/1325978011' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:40:05.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:04 smithi029 ceph-mon[32775]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:05.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:04 smithi029 ceph-mon[32775]: from='client.? 172.21.15.29:0/1325978011' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:40:05.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:04 smithi006 ceph-mon[28419]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:05.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:04 smithi006 ceph-mon[28419]: from='client.? 172.21.15.29:0/1325978011' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:40:07.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:06 smithi188 ceph-mon[32751]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:07.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:06 smithi029 ceph-mon[32775]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:07.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:06 smithi006 ceph-mon[28419]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:09.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:08 smithi188 ceph-mon[32751]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:09.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:08 smithi029 ceph-mon[32775]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:09.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:08 smithi006 ceph-mon[28419]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:11.091 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:10 smithi029 ceph-mon[32775]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:11.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:10 smithi188 ceph-mon[32751]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:11.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:10 smithi006 ceph-mon[28419]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:12.046 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:11 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T01:40:12.046 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:11 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:12.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:11 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T01:40:12.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:11 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:12.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:11 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T01:40:12.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:11 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:13.122 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:12 smithi029 ceph-mon[32775]: Deploying daemon osd.10 on smithi029 2024-04-26T01:40:13.123 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:12 smithi029 ceph-mon[32775]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:13.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:12 smithi188 ceph-mon[32751]: Deploying daemon osd.10 on smithi029 2024-04-26T01:40:13.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:12 smithi188 ceph-mon[32751]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:13.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:12 smithi006 ceph-mon[28419]: Deploying daemon osd.10 on smithi029 2024-04-26T01:40:13.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:12 smithi006 ceph-mon[28419]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:15.108 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:14 smithi029 ceph-mon[32775]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:15.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:14 smithi188 ceph-mon[32751]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:15.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:14 smithi006 ceph-mon[28419]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:17.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:16 smithi188 ceph-mon[32751]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:17.164 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:16 smithi029 ceph-mon[32775]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:17.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:16 smithi006 ceph-mon[28419]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:18.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:18 smithi006 ceph-mon[28419]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:18.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:18 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:18.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:18 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:18.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:18 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:18.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:18 smithi188 ceph-mon[32751]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:18.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:18 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:18.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:18 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:18.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:18 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:18.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:18 smithi029 ceph-mon[32775]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:18.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:18 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:18.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:18 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:18.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:18 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:20.312 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:20 smithi029 ceph-mon[32775]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:20.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:20 smithi006 ceph-mon[28419]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:20.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:20 smithi188 ceph-mon[32751]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:21.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.053+0000 7fea297fa700 1 -- 172.21.15.29:0/3311606226 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fea2c062e80 con 0x7fea140a3c40 2024-04-26T01:40:21.050 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 10 on host 'smithi029' 2024-04-26T01:40:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.055+0000 7fea322fb700 1 -- 172.21.15.29:0/3311606226 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fea140a3c40 msgr2=0x7fea140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.055+0000 7fea322fb700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fea140a3c40 0x7fea140a6100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fea200099d0 tx=0x7fea20008040 comp rx=0 tx=0).stop 2024-04-26T01:40:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.055+0000 7fea322fb700 1 -- 172.21.15.29:0/3311606226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea2c102aa0 msgr2=0x7fea2c0767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.055+0000 7fea322fb700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea2c102aa0 0x7fea2c0767e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00e950 tx=0x7fea1c00ed10 comp rx=0 tx=0).stop 2024-04-26T01:40:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.055+0000 7fea322fb700 1 -- 172.21.15.29:0/3311606226 shutdown_connections 2024-04-26T01:40:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.055+0000 7fea322fb700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea2c104640 0x7fea2c075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.055+0000 7fea322fb700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea2c102aa0 0x7fea2c0767e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.055+0000 7fea322fb700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fea140a3c40 0x7fea140a6100 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.055+0000 7fea322fb700 1 --2- 172.21.15.29:0/3311606226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea2c103ca0 0x7fea2c074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.055+0000 7fea322fb700 1 -- 172.21.15.29:0/3311606226 >> 172.21.15.29:0/3311606226 conn(0x7fea2c0fe040 msgr2=0x7fea2c107020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.056+0000 7fea322fb700 1 -- 172.21.15.29:0/3311606226 shutdown_connections 2024-04-26T01:40:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-26T01:40:21.056+0000 7fea322fb700 1 -- 172.21.15.29:0/3311606226 wait complete. 2024-04-26T01:40:21.816 DEBUG:teuthology.orchestra.run.smithi029:osd.10> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.10.service 2024-04-26T01:40:21.818 INFO:tasks.cephadm:Deploying osd.11 on smithi188 with /dev/vg_nvme/lv_1... 2024-04-26T01:40:21.818 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T01:40:22.066 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:40:22.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:22 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:22.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:22 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:22.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:22 smithi188 ceph-mon[32751]: from='osd.10 [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:40:22.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:22 smithi188 ceph-mon[32751]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:40:22.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:22 smithi188 ceph-mon[32751]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:22.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:22 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:22.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:22 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:22.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:22 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:22.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:22 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:22.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:22 smithi029 ceph-mon[32775]: from='osd.10 [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:40:22.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:22 smithi029 ceph-mon[32775]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:40:22.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:22 smithi029 ceph-mon[32775]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:22.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:22 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:22.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:22 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:22.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:22.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:22.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:22 smithi006 ceph-mon[28419]: from='osd.10 [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:40:22.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:22 smithi006 ceph-mon[28419]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:40:22.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:22 smithi006 ceph-mon[28419]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:22.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:22.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:23 smithi006 ceph-mon[28419]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T01:40:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:23 smithi006 ceph-mon[28419]: from='osd.10 [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:40:23.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:23 smithi006 ceph-mon[28419]: osdmap e73: 11 total, 10 up, 11 in 2024-04-26T01:40:23.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:23 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:23.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:23 smithi006 ceph-mon[28419]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:40:23.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:23 smithi188 ceph-mon[32751]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T01:40:23.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:23 smithi188 ceph-mon[32751]: from='osd.10 [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:40:23.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:23 smithi188 ceph-mon[32751]: osdmap e73: 11 total, 10 up, 11 in 2024-04-26T01:40:23.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:23 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:23.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:23 smithi188 ceph-mon[32751]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:40:23.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:23 smithi029 ceph-mon[32775]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T01:40:23.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:23 smithi029 ceph-mon[32775]: from='osd.10 [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:40:23.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:23 smithi029 ceph-mon[32775]: osdmap e73: 11 total, 10 up, 11 in 2024-04-26T01:40:23.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:23 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:23.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:23 smithi029 ceph-mon[32775]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-04-26T01:40:24.052 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 01:40:23 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T01:40:23.764+0000 7f733ab62700 -1 osd.10 0 waiting for initial osdmap 2024-04-26T01:40:24.052 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 01:40:23 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T01:40:23.782+0000 7f733214c700 -1 osd.10 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:40:24.086 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T01:40:24.086 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T01:40:24.086 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-26T01:40:24.086 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-26T01:40:24.086 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113464 s, 924 MB/s 2024-04-26T01:40:24.086 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-26T01:40:24.309 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:24 smithi029 ceph-mon[32775]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-26T01:40:24.309 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:24 smithi029 ceph-mon[32775]: osdmap e74: 11 total, 10 up, 11 in 2024-04-26T01:40:24.309 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:24.309 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:24 smithi029 ceph-mon[32775]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:24.309 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:24 smithi029 ceph-mon[32775]: from='osd.10 ' entity='osd.10' 2024-04-26T01:40:24.309 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:24 smithi029 ceph-mon[32775]: osd.10 [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729] boot 2024-04-26T01:40:24.309 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:24 smithi029 ceph-mon[32775]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T01:40:24.310 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:24.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:24 smithi006 ceph-mon[28419]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-26T01:40:24.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:24 smithi006 ceph-mon[28419]: osdmap e74: 11 total, 10 up, 11 in 2024-04-26T01:40:24.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:24.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:24 smithi006 ceph-mon[28419]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:24.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:24 smithi006 ceph-mon[28419]: from='osd.10 ' entity='osd.10' 2024-04-26T01:40:24.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:24 smithi006 ceph-mon[28419]: osd.10 [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729] boot 2024-04-26T01:40:24.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:24 smithi006 ceph-mon[28419]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T01:40:24.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:24.381 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:24 smithi188 ceph-mon[32751]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-04-26T01:40:24.381 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:24 smithi188 ceph-mon[32751]: osdmap e74: 11 total, 10 up, 11 in 2024-04-26T01:40:24.381 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:24.381 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:24 smithi188 ceph-mon[32751]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:40:24.381 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:24 smithi188 ceph-mon[32751]: from='osd.10 ' entity='osd.10' 2024-04-26T01:40:24.381 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:24 smithi188 ceph-mon[32751]: osd.10 [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729] boot 2024-04-26T01:40:24.381 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:24 smithi188 ceph-mon[32751]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T01:40:24.381 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:40:24.744 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_1 2024-04-26T01:40:24.985 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:40:25.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:25 smithi006 ceph-mon[28419]: purged_snaps scrub starts 2024-04-26T01:40:25.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:25 smithi006 ceph-mon[28419]: purged_snaps scrub ok 2024-04-26T01:40:25.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:25 smithi006 ceph-mon[28419]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T01:40:25.318 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:25 smithi188 ceph-mon[32751]: purged_snaps scrub starts 2024-04-26T01:40:25.318 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:25 smithi188 ceph-mon[32751]: purged_snaps scrub ok 2024-04-26T01:40:25.318 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:25 smithi188 ceph-mon[32751]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T01:40:25.391 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:25 smithi029 ceph-mon[32775]: purged_snaps scrub starts 2024-04-26T01:40:25.391 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:25 smithi029 ceph-mon[32775]: purged_snaps scrub ok 2024-04-26T01:40:25.391 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:25 smithi029 ceph-mon[32775]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T01:40:26.600 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.597+0000 7fe5c38e9700 1 -- 172.21.15.188:0/2533397917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5bc102ba0 msgr2=0x7fe5bc103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:26.600 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.597+0000 7fe5c38e9700 1 --2- 172.21.15.188:0/2533397917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5bc102ba0 0x7fe5bc103000 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac009990 tx=0x7fe5ac009ca0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.600 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.597+0000 7fe5c38e9700 1 -- 172.21.15.188:0/2533397917 shutdown_connections 2024-04-26T01:40:26.600 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.597+0000 7fe5c38e9700 1 --2- 172.21.15.188:0/2533397917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5bc103540 0x7fe5bc109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.600 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.597+0000 7fe5c38e9700 1 --2- 172.21.15.188:0/2533397917 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5bc1019a0 0x7fe5bc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.600 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c38e9700 1 --2- 172.21.15.188:0/2533397917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5bc102ba0 0x7fe5bc103000 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.600 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c38e9700 1 -- 172.21.15.188:0/2533397917 >> 172.21.15.188:0/2533397917 conn(0x7fe5bc0fcf40 msgr2=0x7fe5bc0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:26.600 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c38e9700 1 -- 172.21.15.188:0/2533397917 shutdown_connections 2024-04-26T01:40:26.601 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c38e9700 1 -- 172.21.15.188:0/2533397917 wait complete. 2024-04-26T01:40:26.601 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c38e9700 1 Processor -- start 2024-04-26T01:40:26.601 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c38e9700 1 -- start start 2024-04-26T01:40:26.601 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c38e9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5bc1019a0 0x7fe5bc198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:26.601 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c38e9700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5bc102ba0 0x7fe5bc1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:26.601 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c38e9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5bc103540 0x7fe5bc1a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:26.601 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c38e9700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe5bc10d4b0 con 0x7fe5bc102ba0 2024-04-26T01:40:26.601 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c38e9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe5bc10d330 con 0x7fe5bc103540 2024-04-26T01:40:26.601 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c38e9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe5bc10d630 con 0x7fe5bc1019a0 2024-04-26T01:40:26.601 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c1685700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5bc1019a0 0x7fe5bc198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:26.601 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c1685700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5bc1019a0 0x7fe5bc198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:56660/0 (socket says 172.21.15.188:56660) 2024-04-26T01:40:26.601 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.598+0000 7fe5c1685700 1 -- 172.21.15.188:0/3746363696 learned_addr learned my addr 172.21.15.188:0/3746363696 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-26T01:40:26.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.599+0000 7fe5c1685700 1 -- 172.21.15.188:0/3746363696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5bc103540 msgr2=0x7fe5bc1a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:26.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.599+0000 7fe5c1685700 1 --2- 172.21.15.188:0/3746363696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5bc103540 0x7fe5bc1a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.599+0000 7fe5c1685700 1 -- 172.21.15.188:0/3746363696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5bc102ba0 msgr2=0x7fe5bc1991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:26.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.599+0000 7fe5c1685700 1 --2- 172.21.15.188:0/3746363696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5bc102ba0 0x7fe5bc1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:26.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.599+0000 7fe5c1685700 1 -- 172.21.15.188:0/3746363696 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5bc1a08e0 con 0x7fe5bc1019a0 2024-04-26T01:40:26.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.599+0000 7fe5c1685700 1 --2- 172.21.15.188:0/3746363696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5bc1019a0 0x7fe5bc198c60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800b500 tx=0x7fe5b800b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:26.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.599+0000 7fe5b27fc700 1 -- 172.21.15.188:0/3746363696 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8015020 con 0x7fe5bc1019a0 2024-04-26T01:40:26.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.599+0000 7fe5c38e9700 1 -- 172.21.15.188:0/3746363696 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5bc1a0bd0 con 0x7fe5bc1019a0 2024-04-26T01:40:26.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.599+0000 7fe5b27fc700 1 -- 172.21.15.188:0/3746363696 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5b8011ce0 con 0x7fe5bc1019a0 2024-04-26T01:40:26.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.599+0000 7fe5b27fc700 1 -- 172.21.15.188:0/3746363696 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8010d60 con 0x7fe5bc1019a0 2024-04-26T01:40:26.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.599+0000 7fe5c38e9700 1 -- 172.21.15.188:0/3746363696 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5bc1a1110 con 0x7fe5bc1019a0 2024-04-26T01:40:26.602 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.600+0000 7fe5c38e9700 1 -- 172.21.15.188:0/3746363696 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5bc069760 con 0x7fe5bc1019a0 2024-04-26T01:40:26.603 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.601+0000 7fe5b27fc700 1 -- 172.21.15.188:0/3746363696 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fe5b8018020 con 0x7fe5bc1019a0 2024-04-26T01:40:26.603 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.602+0000 7fe5b27fc700 1 --2- 172.21.15.188:0/3746363696 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe5a80a3c40 0x7fe5a80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:26.603 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.602+0000 7fe5b27fc700 1 -- 172.21.15.188:0/3746363696 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6723+0+0 (secure 0 0 0) 0x7fe5b80cfbf0 con 0x7fe5bc1019a0 2024-04-26T01:40:26.603 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.603+0000 7fe5c0e84700 1 --2- 172.21.15.188:0/3746363696 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe5a80a3c40 0x7fe5a80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:26.603 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.604+0000 7fe5c0e84700 1 --2- 172.21.15.188:0/3746363696 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe5a80a3c40 0x7fe5a80a6100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac00b5c0 tx=0x7fe5ac003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:26.603 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.606+0000 7fe5b27fc700 1 -- 172.21.15.188:0/3746363696 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe5bc069760 con 0x7fe5bc1019a0 2024-04-26T01:40:26.806 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:26.809+0000 7fe5c38e9700 1 -- 172.21.15.188:0/3746363696 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe5bc0ff170 con 0x7fe5a80a3c40 2024-04-26T01:40:27.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 592 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:27.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: Detected new or changed devices on smithi029 2024-04-26T01:40:27.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:27.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:27.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:27.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:27.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi029 to 4273M 2024-04-26T01:40:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: osdmap e77: 11 total, 11 up, 11 in 2024-04-26T01:40:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:40:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:40:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:27.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 592 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: Detected new or changed devices on smithi029 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi029 to 4273M 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: osdmap e77: 11 total, 11 up, 11 in 2024-04-26T01:40:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:27.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:27.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:27.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:40:27.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:40:27.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:27.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 592 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:27.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: Detected new or changed devices on smithi029 2024-04-26T01:40:27.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:27.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:27.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi029 to 4273M 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: osdmap e77: 11 total, 11 up, 11 in 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:40:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:28.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:27 smithi188 ceph-mon[32751]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:28.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:27 smithi188 ceph-mon[32751]: osdmap e78: 11 total, 11 up, 11 in 2024-04-26T01:40:28.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:27 smithi029 ceph-mon[32775]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:28.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:27 smithi029 ceph-mon[32775]: osdmap e78: 11 total, 11 up, 11 in 2024-04-26T01:40:28.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:27 smithi006 ceph-mon[28419]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:28.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:27 smithi006 ceph-mon[28419]: osdmap e78: 11 total, 11 up, 11 in 2024-04-26T01:40:29.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:28 smithi188 ceph-mon[32751]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 592 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:29.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:28 smithi188 ceph-mon[32751]: osdmap e79: 11 total, 11 up, 11 in 2024-04-26T01:40:29.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:28 smithi029 ceph-mon[32775]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 592 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:29.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:28 smithi029 ceph-mon[32775]: osdmap e79: 11 total, 11 up, 11 in 2024-04-26T01:40:29.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:28 smithi006 ceph-mon[28419]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 592 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:29.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:28 smithi006 ceph-mon[28419]: osdmap e79: 11 total, 11 up, 11 in 2024-04-26T01:40:30.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:29 smithi188 ceph-mon[32751]: from='client.? 172.21.15.188:0/2622438845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb1638ce-4189-4241-b63e-b63c607e90ff"}]: dispatch 2024-04-26T01:40:30.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:29 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb1638ce-4189-4241-b63e-b63c607e90ff"}]: dispatch 2024-04-26T01:40:30.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:29 smithi188 ceph-mon[32751]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb1638ce-4189-4241-b63e-b63c607e90ff"}]': finished 2024-04-26T01:40:30.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:29 smithi188 ceph-mon[32751]: osdmap e80: 12 total, 11 up, 12 in 2024-04-26T01:40:30.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:29 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:30.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:29 smithi029 ceph-mon[32775]: from='client.? 172.21.15.188:0/2622438845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb1638ce-4189-4241-b63e-b63c607e90ff"}]: dispatch 2024-04-26T01:40:30.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:29 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb1638ce-4189-4241-b63e-b63c607e90ff"}]: dispatch 2024-04-26T01:40:30.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:29 smithi029 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb1638ce-4189-4241-b63e-b63c607e90ff"}]': finished 2024-04-26T01:40:30.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:29 smithi029 ceph-mon[32775]: osdmap e80: 12 total, 11 up, 12 in 2024-04-26T01:40:30.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:29 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:30.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:29 smithi006 ceph-mon[28419]: from='client.? 172.21.15.188:0/2622438845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb1638ce-4189-4241-b63e-b63c607e90ff"}]: dispatch 2024-04-26T01:40:30.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:29 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb1638ce-4189-4241-b63e-b63c607e90ff"}]: dispatch 2024-04-26T01:40:30.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:29 smithi006 ceph-mon[28419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb1638ce-4189-4241-b63e-b63c607e90ff"}]': finished 2024-04-26T01:40:30.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:29 smithi006 ceph-mon[28419]: osdmap e80: 12 total, 11 up, 12 in 2024-04-26T01:40:30.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:29 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:31.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:30 smithi188 ceph-mon[32751]: pgmap v270: 1 pgs: 1 peering; 577 KiB data, 338 MiB used, 983 GiB / 983 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-04-26T01:40:31.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:30 smithi188 ceph-mon[32751]: from='client.? 172.21.15.188:0/936510788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:40:31.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:30 smithi029 ceph-mon[32775]: pgmap v270: 1 pgs: 1 peering; 577 KiB data, 338 MiB used, 983 GiB / 983 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-04-26T01:40:31.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:30 smithi029 ceph-mon[32775]: from='client.? 172.21.15.188:0/936510788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:40:31.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:30 smithi006 ceph-mon[28419]: pgmap v270: 1 pgs: 1 peering; 577 KiB data, 338 MiB used, 983 GiB / 983 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-04-26T01:40:31.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:30 smithi006 ceph-mon[28419]: from='client.? 172.21.15.188:0/936510788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:40:33.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:32 smithi188 ceph-mon[32751]: pgmap v272: 1 pgs: 1 peering; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-26T01:40:33.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:32 smithi029 ceph-mon[32775]: pgmap v272: 1 pgs: 1 peering; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-26T01:40:33.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:32 smithi006 ceph-mon[28419]: pgmap v272: 1 pgs: 1 peering; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-26T01:40:34.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:34 smithi188 ceph-mon[32751]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-26T01:40:35.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:34 smithi029 ceph-mon[32775]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-26T01:40:35.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:34 smithi006 ceph-mon[28419]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-26T01:40:37.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:36 smithi188 ceph-mon[32751]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:40:37.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:36 smithi029 ceph-mon[32775]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:40:37.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:36 smithi006 ceph-mon[28419]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:40:38.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T01:40:38.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:37 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:38.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:37 smithi188 ceph-mon[32751]: Deploying daemon osd.11 on smithi188 2024-04-26T01:40:38.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T01:40:38.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:37 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:38.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:37 smithi029 ceph-mon[32775]: Deploying daemon osd.11 on smithi188 2024-04-26T01:40:38.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T01:40:38.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:37 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:38.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:37 smithi006 ceph-mon[28419]: Deploying daemon osd.11 on smithi188 2024-04-26T01:40:39.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:38 smithi188 ceph-mon[32751]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:39.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:38 smithi029 ceph-mon[32775]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:39.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:38 smithi006 ceph-mon[28419]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:41.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:40 smithi188 ceph-mon[32751]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:41.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:40 smithi029 ceph-mon[32775]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:41.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:40 smithi006 ceph-mon[28419]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:43.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:42 smithi029 ceph-mon[32775]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:43.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:42 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:43.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:42 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:43.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:42 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:43.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:42 smithi188 ceph-mon[32751]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:43.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:42 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:43.199 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:42 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:43.199 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:42 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:43.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:42 smithi006 ceph-mon[28419]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:43.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:42 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:43.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:42 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:43.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:42 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:45.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:44 smithi188 ceph-mon[32751]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:45.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:44 smithi029 ceph-mon[32775]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:45.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:44 smithi006 ceph-mon[28419]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:45.643 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.646+0000 7fe5b27fc700 1 -- 172.21.15.188:0/3746363696 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe5bc0ff170 con 0x7fe5a80a3c40 2024-04-26T01:40:45.643 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 11 on host 'smithi188' 2024-04-26T01:40:45.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.648+0000 7fe5c38e9700 1 -- 172.21.15.188:0/3746363696 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe5a80a3c40 msgr2=0x7fe5a80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:45.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.648+0000 7fe5c38e9700 1 --2- 172.21.15.188:0/3746363696 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe5a80a3c40 0x7fe5a80a6100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac00b5c0 tx=0x7fe5ac003860 comp rx=0 tx=0).stop 2024-04-26T01:40:45.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.648+0000 7fe5c38e9700 1 -- 172.21.15.188:0/3746363696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5bc1019a0 msgr2=0x7fe5bc198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:45.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.648+0000 7fe5c38e9700 1 --2- 172.21.15.188:0/3746363696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5bc1019a0 0x7fe5bc198c60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800b500 tx=0x7fe5b800b8c0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.648+0000 7fe5c38e9700 1 -- 172.21.15.188:0/3746363696 shutdown_connections 2024-04-26T01:40:45.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.648+0000 7fe5c38e9700 1 --2- 172.21.15.188:0/3746363696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5bc103540 0x7fe5bc1a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.648+0000 7fe5c38e9700 1 --2- 172.21.15.188:0/3746363696 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe5a80a3c40 0x7fe5a80a6100 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.648+0000 7fe5c38e9700 1 --2- 172.21.15.188:0/3746363696 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5bc1019a0 0x7fe5bc198c60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.648+0000 7fe5c38e9700 1 --2- 172.21.15.188:0/3746363696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5bc102ba0 0x7fe5bc1991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:45.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.648+0000 7fe5c38e9700 1 -- 172.21.15.188:0/3746363696 >> 172.21.15.188:0/3746363696 conn(0x7fe5bc0fcf40 msgr2=0x7fe5bc0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:45.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.648+0000 7fe5c38e9700 1 -- 172.21.15.188:0/3746363696 shutdown_connections 2024-04-26T01:40:45.647 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:40:45.648+0000 7fe5c38e9700 1 -- 172.21.15.188:0/3746363696 wait complete. 2024-04-26T01:40:46.502 DEBUG:teuthology.orchestra.run.smithi188:osd.11> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.11.service 2024-04-26T01:40:46.504 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-26T01:40:46.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T01:40:46.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:46 smithi188 ceph-mon[32751]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:46.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:46.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:46.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:46 smithi188 ceph-mon[32751]: from='osd.11 [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:40:46.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:46 smithi188 ceph-mon[32751]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:40:46.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:46 smithi029 ceph-mon[32775]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:46.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:46 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:46.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:46 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:46 smithi029 ceph-mon[32775]: from='osd.11 [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:40:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:46 smithi029 ceph-mon[32775]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:40:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:47.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:46 smithi006 ceph-mon[28419]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:47.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:47.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:47.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:46 smithi006 ceph-mon[28419]: from='osd.11 [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:40:47.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:46 smithi006 ceph-mon[28419]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:40:48.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:48.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:48.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:47 smithi006 ceph-mon[28419]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T01:40:48.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:47 smithi006 ceph-mon[28419]: from='osd.11 [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:40:48.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:47 smithi006 ceph-mon[28419]: osdmap e81: 12 total, 11 up, 12 in 2024-04-26T01:40:48.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:47 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:48.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:47 smithi006 ceph-mon[28419]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:40:48.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:48.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:48.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:47 smithi188 ceph-mon[32751]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T01:40:48.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:47 smithi188 ceph-mon[32751]: from='osd.11 [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:40:48.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:47 smithi188 ceph-mon[32751]: osdmap e81: 12 total, 11 up, 12 in 2024-04-26T01:40:48.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:47 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:48.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:47 smithi188 ceph-mon[32751]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:40:48.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:48.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:48.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:47 smithi029 ceph-mon[32775]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T01:40:48.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:47 smithi029 ceph-mon[32775]: from='osd.11 [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:40:48.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:47 smithi029 ceph-mon[32775]: osdmap e81: 12 total, 11 up, 12 in 2024-04-26T01:40:48.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:47 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:48.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:47 smithi029 ceph-mon[32775]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-26T01:40:48.914 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 01:40:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T01:40:48.834+0000 7f836be9d700 -1 osd.11 0 waiting for initial osdmap 2024-04-26T01:40:49.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.098+0000 7fc3d4a00700 1 -- 172.21.15.6:0/3695639937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3d010a2e0 msgr2=0x7fc3d010a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.098+0000 7fc3d4a00700 1 --2- 172.21.15.6:0/3695639937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3d010a2e0 0x7fc3d010a740 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80099c0 tx=0x7fc3b8009cd0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.098+0000 7fc3d4a00700 1 -- 172.21.15.6:0/3695639937 shutdown_connections 2024-04-26T01:40:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.098+0000 7fc3d4a00700 1 --2- 172.21.15.6:0/3695639937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3d010ac80 0x7fc3d0111530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.098+0000 7fc3d4a00700 1 --2- 172.21.15.6:0/3695639937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3d01090e0 0x7fc3d0109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.098+0000 7fc3d4a00700 1 --2- 172.21.15.6:0/3695639937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3d010a2e0 0x7fc3d010a740 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.098+0000 7fc3d4a00700 1 -- 172.21.15.6:0/3695639937 >> 172.21.15.6:0/3695639937 conn(0x7fc3d00780b0 msgr2=0x7fc3d007a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.098+0000 7fc3d4a00700 1 -- 172.21.15.6:0/3695639937 shutdown_connections 2024-04-26T01:40:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.098+0000 7fc3d4a00700 1 -- 172.21.15.6:0/3695639937 wait complete. 2024-04-26T01:40:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.099+0000 7fc3d4a00700 1 Processor -- start 2024-04-26T01:40:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.099+0000 7fc3d4a00700 1 -- start start 2024-04-26T01:40:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.099+0000 7fc3d4a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3d01090e0 0x7fc3d019f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.099+0000 7fc3d4a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3d010a2e0 0x7fc3d019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.099+0000 7fc3d4a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3d010ac80 0x7fc3d01a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.099+0000 7fc3d4a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d0114bb0 con 0x7fc3d010ac80 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.099+0000 7fc3d4a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d0114a30 con 0x7fc3d010a2e0 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.099+0000 7fc3d4a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d0114d30 con 0x7fc3d01090e0 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.100+0000 7fc3c7e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3d010a2e0 0x7fc3d019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.100+0000 7fc3c7e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3d010a2e0 0x7fc3d019f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55764/0 (socket says 172.21.15.6:55764) 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.100+0000 7fc3c7e00700 1 -- 172.21.15.6:0/1723369920 learned_addr learned my addr 172.21.15.6:0/1723369920 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.100+0000 7fc3ce800700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3d010ac80 0x7fc3d01a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.100+0000 7fc3cde00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3d01090e0 0x7fc3d019f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.100+0000 7fc3c7e00700 1 -- 172.21.15.6:0/1723369920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3d01090e0 msgr2=0x7fc3d019f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.100+0000 7fc3c7e00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3d01090e0 0x7fc3d019f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.100+0000 7fc3c7e00700 1 -- 172.21.15.6:0/1723369920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3d010ac80 msgr2=0x7fc3d01a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.100+0000 7fc3c7e00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3d010ac80 0x7fc3d01a67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.100+0000 7fc3c7e00700 1 -- 172.21.15.6:0/1723369920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d01a6e80 con 0x7fc3d010a2e0 2024-04-26T01:40:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.100+0000 7fc3c7e00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3d010a2e0 0x7fc3d019f760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80036f0 tx=0x7fc3b8003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.101+0000 7fc3c6a00700 1 -- 172.21.15.6:0/1723369920 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b800be50 con 0x7fc3d010a2e0 2024-04-26T01:40:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.101+0000 7fc3c6a00700 1 -- 172.21.15.6:0/1723369920 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b800f460 con 0x7fc3d010a2e0 2024-04-26T01:40:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.101+0000 7fc3d4a00700 1 -- 172.21.15.6:0/1723369920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d01a7110 con 0x7fc3d010a2e0 2024-04-26T01:40:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.101+0000 7fc3c6a00700 1 -- 172.21.15.6:0/1723369920 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8017650 con 0x7fc3d010a2e0 2024-04-26T01:40:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.101+0000 7fc3d4a00700 1 -- 172.21.15.6:0/1723369920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3d01a75c0 con 0x7fc3d010a2e0 2024-04-26T01:40:49.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.102+0000 7fc3d4a00700 1 -- 172.21.15.6:0/1723369920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3d0069730 con 0x7fc3d010a2e0 2024-04-26T01:40:49.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.104+0000 7fc3c6a00700 1 -- 172.21.15.6:0/1723369920 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fc3b800f910 con 0x7fc3d010a2e0 2024-04-26T01:40:49.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.105+0000 7fc3c6a00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc3b40a3cc0 0x7fc3b40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:49.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.105+0000 7fc3c6a00700 1 -- 172.21.15.6:0/1723369920 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6847+0+0 (secure 0 0 0) 0x7fc3b80d0c30 con 0x7fc3d010a2e0 2024-04-26T01:40:49.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.106+0000 7fc3cde00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc3b40a3cc0 0x7fc3b40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:49.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.106+0000 7fc3cde00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc3b40a3cc0 0x7fc3b40a6180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc3d010a140 tx=0x7fc3c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.110+0000 7fc3c6a00700 1 -- 172.21.15.6:0/1723369920 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3b8013070 con 0x7fc3d010a2e0 2024-04-26T01:40:49.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:48 smithi029 ceph-mon[32775]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:49.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:48 smithi029 ceph-mon[32775]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-26T01:40:49.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:48 smithi029 ceph-mon[32775]: osdmap e82: 12 total, 11 up, 12 in 2024-04-26T01:40:49.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:48 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:49.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:48 smithi029 ceph-mon[32775]: from='osd.11 ' entity='osd.11' 2024-04-26T01:40:49.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:48 smithi006 ceph-mon[28419]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:49.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:48 smithi006 ceph-mon[28419]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-26T01:40:49.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:48 smithi006 ceph-mon[28419]: osdmap e82: 12 total, 11 up, 12 in 2024-04-26T01:40:49.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:49.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:48 smithi006 ceph-mon[28419]: from='osd.11 ' entity='osd.11' 2024-04-26T01:40:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.341+0000 7fc3d4a00700 1 -- 172.21.15.6:0/1723369920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fc3d0063650 con 0x7fc3d010a2e0 2024-04-26T01:40:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.342+0000 7fc3c6a00700 1 -- 172.21.15.6:0/1723369920 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7fc3b809dbc0 con 0x7fc3d010a2e0 2024-04-26T01:40:49.338 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:40:49.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.344+0000 7fc3d4a00700 1 -- 172.21.15.6:0/1723369920 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc3b40a3cc0 msgr2=0x7fc3b40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.345+0000 7fc3d4a00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc3b40a3cc0 0x7fc3b40a6180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc3d010a140 tx=0x7fc3c0009260 comp rx=0 tx=0).stop 2024-04-26T01:40:49.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.345+0000 7fc3d4a00700 1 -- 172.21.15.6:0/1723369920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3d010a2e0 msgr2=0x7fc3d019f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.345+0000 7fc3d4a00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3d010a2e0 0x7fc3d019f760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80036f0 tx=0x7fc3b8003f10 comp rx=0 tx=0).stop 2024-04-26T01:40:49.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.345+0000 7fc3d4a00700 1 -- 172.21.15.6:0/1723369920 shutdown_connections 2024-04-26T01:40:49.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.345+0000 7fc3d4a00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3d010ac80 0x7fc3d01a67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.345+0000 7fc3d4a00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc3b40a3cc0 0x7fc3b40a6180 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.345+0000 7fc3d4a00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3d01090e0 0x7fc3d019f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.345+0000 7fc3d4a00700 1 --2- 172.21.15.6:0/1723369920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3d010a2e0 0x7fc3d019f760 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.345+0000 7fc3d4a00700 1 -- 172.21.15.6:0/1723369920 >> 172.21.15.6:0/1723369920 conn(0x7fc3d00780b0 msgr2=0x7fc3d010d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:49.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.345+0000 7fc3d4a00700 1 -- 172.21.15.6:0/1723369920 shutdown_connections 2024-04-26T01:40:49.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:49.345+0000 7fc3d4a00700 1 -- 172.21.15.6:0/1723369920 wait complete. 2024-04-26T01:40:49.388 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 01:40:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T01:40:49.001+0000 7f8363c88700 -1 osd.11 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:40:49.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:48 smithi188 ceph-mon[32751]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:49.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:48 smithi188 ceph-mon[32751]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-26T01:40:49.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:48 smithi188 ceph-mon[32751]: osdmap e82: 12 total, 11 up, 12 in 2024-04-26T01:40:49.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:48 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:49.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:48 smithi188 ceph-mon[32751]: from='osd.11 ' entity='osd.11' 2024-04-26T01:40:49.961 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:49 smithi006 ceph-mon[28419]: purged_snaps scrub starts 2024-04-26T01:40:49.961 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:49 smithi006 ceph-mon[28419]: purged_snaps scrub ok 2024-04-26T01:40:49.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:49.962 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:49 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1723369920' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:40:50.030 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":82,"num_osds":12,"num_up_osds":11,"osd_up_since":1714095624,"num_in_osds":12,"osd_in_since":1714095629,"num_remapped_pgs":0} 2024-04-26T01:40:50.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:49 smithi188 ceph-mon[32751]: purged_snaps scrub starts 2024-04-26T01:40:50.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:49 smithi188 ceph-mon[32751]: purged_snaps scrub ok 2024-04-26T01:40:50.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:50.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:49 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1723369920' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:40:50.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:49 smithi029 ceph-mon[32775]: purged_snaps scrub starts 2024-04-26T01:40:50.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:49 smithi029 ceph-mon[32775]: purged_snaps scrub ok 2024-04-26T01:40:50.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:50.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:49 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1723369920' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:40:51.032 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T01:40:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:51.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: osd.11 [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880] boot 2024-04-26T01:40:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T01:40:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:51.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:51.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:51.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:51.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: osd.11 [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880] boot 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: osd.11 [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880] boot 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:51.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:51.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:51.446 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:52.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:51 smithi006 ceph-mon[28419]: Detected new or changed devices on smithi188 2024-04-26T01:40:52.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:51 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi188 to 4272M 2024-04-26T01:40:52.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:51 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:52.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:51 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:52.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:51 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:52.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:51 smithi006 ceph-mon[28419]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T01:40:52.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:51 smithi188 ceph-mon[32751]: Detected new or changed devices on smithi188 2024-04-26T01:40:52.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:51 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi188 to 4272M 2024-04-26T01:40:52.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:51 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:52.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:51 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:52.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:51 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:52.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:51 smithi188 ceph-mon[32751]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T01:40:52.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:51 smithi029 ceph-mon[32775]: Detected new or changed devices on smithi188 2024-04-26T01:40:52.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:51 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi188 to 4272M 2024-04-26T01:40:52.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:51 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:52.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:51 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:52.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:51 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:40:52.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:51 smithi029 ceph-mon[32775]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T01:40:53.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:52 smithi006 ceph-mon[28419]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:53.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:52 smithi188 ceph-mon[32751]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:53.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:52 smithi029 ceph-mon[32775]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.585+0000 7f9e17a00700 1 -- 172.21.15.6:0/1555074978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e10075440 msgr2=0x7f9e100758c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.585+0000 7f9e17a00700 1 --2- 172.21.15.6:0/1555074978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e10075440 0x7f9e100758c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0098f0 tx=0x7f9dfc009c00 comp rx=0 tx=0).stop 2024-04-26T01:40:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.585+0000 7f9e17a00700 1 -- 172.21.15.6:0/1555074978 shutdown_connections 2024-04-26T01:40:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.585+0000 7f9e17a00700 1 --2- 172.21.15.6:0/1555074978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e1010a9d0 0x7f9e1010ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.585+0000 7f9e17a00700 1 --2- 172.21.15.6:0/1555074978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e10076a10 0x7f9e10074e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.585+0000 7f9e17a00700 1 --2- 172.21.15.6:0/1555074978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e10075440 0x7f9e100758c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.585+0000 7f9e17a00700 1 -- 172.21.15.6:0/1555074978 >> 172.21.15.6:0/1555074978 conn(0x7f9e100fe090 msgr2=0x7f9e101004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.585+0000 7f9e17a00700 1 -- 172.21.15.6:0/1555074978 shutdown_connections 2024-04-26T01:40:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.585+0000 7f9e17a00700 1 -- 172.21.15.6:0/1555074978 wait complete. 2024-04-26T01:40:53.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.586+0000 7f9e17a00700 1 Processor -- start 2024-04-26T01:40:53.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.586+0000 7f9e17a00700 1 -- start start 2024-04-26T01:40:53.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.587+0000 7f9e17a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e10075440 0x7f9e1019add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:53.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.587+0000 7f9e17a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e10076a10 0x7f9e1019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:53.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.587+0000 7f9e17a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e1010a9d0 0x7f9e101a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:53.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.587+0000 7f9e17a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9e10110790 con 0x7f9e10075440 2024-04-26T01:40:53.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.587+0000 7f9e17a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9e10110610 con 0x7f9e10076a10 2024-04-26T01:40:53.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.587+0000 7f9e17a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9e10110910 con 0x7f9e1010a9d0 2024-04-26T01:40:53.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.587+0000 7f9e0fe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e10076a10 0x7f9e1019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:53.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.587+0000 7f9e15000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e10075440 0x7f9e1019add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:53.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.587+0000 7f9e0fe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e10076a10 0x7f9e1019b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55790/0 (socket says 172.21.15.6:55790) 2024-04-26T01:40:53.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.587+0000 7f9e0fe00700 1 -- 172.21.15.6:0/2161101427 learned_addr learned my addr 172.21.15.6:0/2161101427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:40:53.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.587+0000 7f9e15000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e10075440 0x7f9e1019add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56942/0 (socket says 172.21.15.6:56942) 2024-04-26T01:40:53.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.587+0000 7f9e15a00700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e1010a9d0 0x7f9e101a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:53.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.588+0000 7f9e0fe00700 1 -- 172.21.15.6:0/2161101427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e1010a9d0 msgr2=0x7f9e101a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:53.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.588+0000 7f9e0fe00700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e1010a9d0 0x7f9e101a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.588+0000 7f9e0fe00700 1 -- 172.21.15.6:0/2161101427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e10075440 msgr2=0x7f9e1019add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:53.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.588+0000 7f9e0fe00700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e10075440 0x7f9e1019add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.588+0000 7f9e0fe00700 1 -- 172.21.15.6:0/2161101427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e101a2a30 con 0x7f9e10076a10 2024-04-26T01:40:53.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.588+0000 7f9e0fe00700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e10076a10 0x7f9e1019b310 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00c010 tx=0x7f9dfc005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:53.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.588+0000 7f9e0d600700 1 -- 172.21.15.6:0/2161101427 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc005460 con 0x7f9e10076a10 2024-04-26T01:40:53.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.588+0000 7f9e0d600700 1 -- 172.21.15.6:0/2161101427 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dfc005600 con 0x7f9e10076a10 2024-04-26T01:40:53.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.588+0000 7f9e0d600700 1 -- 172.21.15.6:0/2161101427 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc003b60 con 0x7f9e10076a10 2024-04-26T01:40:53.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.588+0000 7f9e17a00700 1 -- 172.21.15.6:0/2161101427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e101a2cc0 con 0x7f9e10076a10 2024-04-26T01:40:53.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.589+0000 7f9e17a00700 1 -- 172.21.15.6:0/2161101427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e101a30c0 con 0x7f9e10076a10 2024-04-26T01:40:53.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.590+0000 7f9e17a00700 1 -- 172.21.15.6:0/2161101427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e1002df80 con 0x7f9e10076a10 2024-04-26T01:40:53.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.592+0000 7f9e0d600700 1 -- 172.21.15.6:0/2161101427 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f9dfc003d00 con 0x7f9e10076a10 2024-04-26T01:40:53.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.593+0000 7f9e0d600700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9df80a3c40 0x7f9df80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:53.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.593+0000 7f9e0d600700 1 -- 172.21.15.6:0/2161101427 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f9dfc0d1290 con 0x7f9e10076a10 2024-04-26T01:40:53.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.593+0000 7f9e15000700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9df80a3c40 0x7f9df80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:53.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.594+0000 7f9e15000700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9df80a3c40 0x7f9df80a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009750 tx=0x7f9e04009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:53.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.595+0000 7f9e0d600700 1 -- 172.21.15.6:0/2161101427 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9dfc05eff0 con 0x7f9e10076a10 2024-04-26T01:40:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.828+0000 7f9e17a00700 1 -- 172.21.15.6:0/2161101427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f9e100636f0 con 0x7f9e10076a10 2024-04-26T01:40:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.829+0000 7f9e0d600700 1 -- 172.21.15.6:0/2161101427 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v84) ==== 74+0+133 (secure 0 0 0) 0x7f9dfc013030 con 0x7f9e10076a10 2024-04-26T01:40:53.825 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:40:53.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.832+0000 7f9e17a00700 1 -- 172.21.15.6:0/2161101427 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9df80a3c40 msgr2=0x7f9df80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.832+0000 7f9e17a00700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9df80a3c40 0x7f9df80a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009750 tx=0x7f9e04009310 comp rx=0 tx=0).stop 2024-04-26T01:40:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.832+0000 7f9e17a00700 1 -- 172.21.15.6:0/2161101427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e10076a10 msgr2=0x7f9e1019b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.832+0000 7f9e17a00700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e10076a10 0x7f9e1019b310 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00c010 tx=0x7f9dfc005270 comp rx=0 tx=0).stop 2024-04-26T01:40:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.832+0000 7f9e17a00700 1 -- 172.21.15.6:0/2161101427 shutdown_connections 2024-04-26T01:40:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.832+0000 7f9e17a00700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9df80a3c40 0x7f9df80a6100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.832+0000 7f9e17a00700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e1010a9d0 0x7f9e101a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.832+0000 7f9e17a00700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e10075440 0x7f9e1019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.832+0000 7f9e17a00700 1 --2- 172.21.15.6:0/2161101427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e10076a10 0x7f9e1019b310 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.832+0000 7f9e17a00700 1 -- 172.21.15.6:0/2161101427 >> 172.21.15.6:0/2161101427 conn(0x7f9e100fe090 msgr2=0x7f9e10106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.832+0000 7f9e17a00700 1 -- 172.21.15.6:0/2161101427 shutdown_connections 2024-04-26T01:40:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:53.832+0000 7f9e17a00700 1 -- 172.21.15.6:0/2161101427 wait complete. 2024-04-26T01:40:54.199 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:53 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2161101427' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:40:54.290 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:53 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2161101427' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:40:54.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:53 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2161101427' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:40:54.566 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"num_osds":12,"num_up_osds":12,"osd_up_since":1714095649,"num_in_osds":12,"osd_in_since":1714095629,"num_remapped_pgs":0} 2024-04-26T01:40:54.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:40:55.041 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:55.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:54 smithi006 ceph-mon[28419]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:55.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:54 smithi188 ceph-mon[32751]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:55.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:54 smithi029 ceph-mon[32775]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.072+0000 7fc886a00700 1 -- 172.21.15.6:0/111861125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88010f030 msgr2=0x7fc88010f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.072+0000 7fc886a00700 1 --2- 172.21.15.6:0/111861125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88010f030 0x7fc88010f410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc874009990 tx=0x7fc874009ca0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.072+0000 7fc886a00700 1 -- 172.21.15.6:0/111861125 shutdown_connections 2024-04-26T01:40:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.072+0000 7fc886a00700 1 --2- 172.21.15.6:0/111861125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc880109940 0x7fc880115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.072+0000 7fc886a00700 1 --2- 172.21.15.6:0/111861125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88010f030 0x7fc88010f410 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.072+0000 7fc886a00700 1 --2- 172.21.15.6:0/111861125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc880108f80 0x7fc880109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.072+0000 7fc886a00700 1 -- 172.21.15.6:0/111861125 >> 172.21.15.6:0/111861125 conn(0x7fc8800787d0 msgr2=0x7fc880107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.072+0000 7fc886a00700 1 -- 172.21.15.6:0/111861125 shutdown_connections 2024-04-26T01:40:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.072+0000 7fc886a00700 1 -- 172.21.15.6:0/111861125 wait complete. 2024-04-26T01:40:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.073+0000 7fc886a00700 1 Processor -- start 2024-04-26T01:40:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.073+0000 7fc886a00700 1 -- start start 2024-04-26T01:40:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc886a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880108f80 0x7fc88019f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc886a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc880109940 0x7fc88019fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc886a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc88010f030 0x7fc8801a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc886a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc880119120 con 0x7fc880109940 2024-04-26T01:40:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc886a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc880118fa0 con 0x7fc880108f80 2024-04-26T01:40:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc886a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc8801192a0 con 0x7fc88010f030 2024-04-26T01:40:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc884c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc880109940 0x7fc88019fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc885600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880108f80 0x7fc88019f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc884c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc880109940 0x7fc88019fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56956/0 (socket says 172.21.15.6:56956) 2024-04-26T01:40:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc884c00700 1 -- 172.21.15.6:0/3524275286 learned_addr learned my addr 172.21.15.6:0/3524275286 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:40:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc885600700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880108f80 0x7fc88019f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55806/0 (socket says 172.21.15.6:55806) 2024-04-26T01:40:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc886000700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc88010f030 0x7fc8801a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc885600700 1 -- 172.21.15.6:0/3524275286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc88010f030 msgr2=0x7fc8801a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc885600700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc88010f030 0x7fc8801a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.074+0000 7fc885600700 1 -- 172.21.15.6:0/3524275286 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc880109940 msgr2=0x7fc88019fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.075+0000 7fc885600700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc880109940 0x7fc88019fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.075+0000 7fc885600700 1 -- 172.21.15.6:0/3524275286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8801a4580 con 0x7fc880108f80 2024-04-26T01:40:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.075+0000 7fc884c00700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc880109940 0x7fc88019fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.075+0000 7fc885600700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880108f80 0x7fc88019f570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc874006010 tx=0x7fc8740057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.075+0000 7fc87e000700 1 -- 172.21.15.6:0/3524275286 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc874027070 con 0x7fc880108f80 2024-04-26T01:40:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.075+0000 7fc87e000700 1 -- 172.21.15.6:0/3524275286 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc874022470 con 0x7fc880108f80 2024-04-26T01:40:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.075+0000 7fc886a00700 1 -- 172.21.15.6:0/3524275286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8801a4810 con 0x7fc880108f80 2024-04-26T01:40:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.075+0000 7fc87e000700 1 -- 172.21.15.6:0/3524275286 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc874017650 con 0x7fc880108f80 2024-04-26T01:40:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.075+0000 7fc886a00700 1 -- 172.21.15.6:0/3524275286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8801ac0f0 con 0x7fc880108f80 2024-04-26T01:40:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.076+0000 7fc886a00700 1 -- 172.21.15.6:0/3524275286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc880069730 con 0x7fc880108f80 2024-04-26T01:40:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.078+0000 7fc87e000700 1 -- 172.21.15.6:0/3524275286 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fc87402a070 con 0x7fc880108f80 2024-04-26T01:40:57.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.080+0000 7fc87e000700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc8680a3cc0 0x7fc8680a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:57.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.080+0000 7fc87e000700 1 -- 172.21.15.6:0/3524275286 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fc8740d0be0 con 0x7fc880108f80 2024-04-26T01:40:57.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.080+0000 7fc884c00700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc8680a3cc0 0x7fc8680a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:57.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.081+0000 7fc884c00700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc8680a3cc0 0x7fc8680a6180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc88010e7f0 tx=0x7fc86c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:57.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.084+0000 7fc87e000700 1 -- 172.21.15.6:0/3524275286 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc874013070 con 0x7fc880108f80 2024-04-26T01:40:57.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.316+0000 7fc886a00700 1 -- 172.21.15.6:0/3524275286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc88002df80 con 0x7fc880108f80 2024-04-26T01:40:57.313 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:56 smithi006 ceph-mon[28419]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:57.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.318+0000 7fc87e000700 1 -- 172.21.15.6:0/3524275286 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+18346 (secure 0 0 0) 0x7fc87409da50 con 0x7fc880108f80 2024-04-26T01:40:57.315 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:40:57.315 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:32:29.071544+0000","modified":"2024-04-26T01:40:51.230784+0000","last_up_change":"2024-04-26T01:40:49.915924+0000","last_in_change":"2024-04-26T01:40:29.559647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:36:35.182726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f164c51d-ce5d-4140-b9e3-a4b84088f5ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6803","nonce":76518486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6805","nonce":76518486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6809","nonce":76518486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6807","nonce":76518486}]},"public_addr":"172.21.15.6:6803/76518486","cluster_addr":"172.21.15.6:6805/76518486","heartbeat_back_addr":"172.21.15.6:6809/76518486","heartbeat_front_addr":"172.21.15.6:6807/76518486","state":["exists","up"]},{"osd":1,"uuid":"57e875dd-ddf1-4ee4-a2a7-9589d7f42818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6801","nonce":400769116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6803","nonce":400769116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6807","nonce":400769116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6805","nonce":400769116}]},"public_addr":"172.21.15.29:6801/400769116","cluster_addr":"172.21.15.29:6803/400769116","heartbeat_back_addr":"172.21.15.29:6807/400769116","heartbeat_front_addr":"172.21.15.29:6805/400769116","state":["exists","up"]},{"osd":2,"uuid":"77e0c294-c48a-4318-938b-4604a533a2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6801","nonce":3508067672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6803","nonce":3508067672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6807","nonce":3508067672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6805","nonce":3508067672}]},"public_addr":"172.21.15.188:6801/3508067672","cluster_addr":"172.21.15.188:6803/3508067672","heartbeat_back_addr":"172.21.15.188:6807/3508067672","heartbeat_front_addr":"172.21.15.188:6805/3508067672","state":["exists","up"]},{"osd":3,"uuid":"6393dac5-996c-47af-9687-0f1393ecc483","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6811","nonce":1987311118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6813","nonce":1987311118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6817","nonce":1987311118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6815","nonce":1987311118}]},"public_addr":"172.21.15.6:6811/1987311118","cluster_addr":"172.21.15.6:6813/1987311118","heartbeat_back_addr":"172.21.15.6:6817/1987311118","heartbeat_front_addr":"172.21.15.6:6815/1987311118","state":["exists","up"]},{"osd":4,"uuid":"f2c1a535-26a0-46c6-a6d0-c2529c786739","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6809","nonce":2125876609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6811","nonce":2125876609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6815","nonce":2125876609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6813","nonce":2125876609}]},"public_addr":"172.21.15.29:6809/2125876609","cluster_addr":"172.21.15.29:6811/2125876609","heartbeat_back_addr":"172.21.15.29:6815/2125876609","heartbeat_front_addr":"172.21.15.29:6813/2125876609","state":["exists","up"]},{"osd":5,"uuid":"9788bb0a-0d79-4743-bce6-80f5342e8f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6809","nonce":2092077007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6811","nonce":2092077007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6815","nonce":2092077007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6813","nonce":2092077007}]},"public_addr":"172.21.15.188:6809/2092077007","cluster_addr":"172.21.15.188:6811/2092077007","heartbeat_back_addr":"172.21.15.188:6815/2092077007","heartbeat_front_addr":"172.21.15.188:6813/2092077007","state":["exists","up"]},{"osd":6,"uuid":"fd8584e1-69ee-4c30-85e4-a637357d1b6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6819","nonce":3256016695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6821","nonce":3256016695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6825","nonce":3256016695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6823","nonce":3256016695}]},"public_addr":"172.21.15.6:6819/3256016695","cluster_addr":"172.21.15.6:6821/3256016695","heartbeat_back_addr":"172.21.15.6:6825/3256016695","heartbeat_front_addr":"172.21.15.6:6823/3256016695","state":["exists","up"]},{"osd":7,"uuid":"6b7e6784-d574-42da-8416-a74a3b6b3755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6817","nonce":3478594037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6819","nonce":3478594037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6823","nonce":3478594037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6821","nonce":3478594037}]},"public_addr":"172.21.15.29:6817/3478594037","cluster_addr":"172.21.15.29:6819/3478594037","heartbeat_back_addr":"172.21.15.29:6823/3478594037","heartbeat_front_addr":"172.21.15.29:6821/3478594037","state":["exists","up"]},{"osd":8,"uuid":"30cd2ea1-2767-4062-8bd9-47214e4b4b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6817","nonce":3290364376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6819","nonce":3290364376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6823","nonce":3290364376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6821","nonce":3290364376}]},"public_addr":"172.21.15.188:6817/3290364376","cluster_addr":"172.21.15.188:6819/3290364376","heartbeat_back_addr":"172.21.15.188:6823/3290364376","heartbeat_front_addr":"172.21.15.188:6821/3290364376","state":["exists","up"]},{"osd":9,"uuid":"f033cdc0-1178-4a7f-a338-69ff32416087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6827","nonce":1918893912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6829","nonce":1918893912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6833","nonce":1918893912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6831","nonce":1918893912}]},"public_addr":"172.21.15.6:6827/1918893912","cluster_addr":"172.21.15.6:6829/1918893912","heartbeat_back_addr":"172.21.15.6:6833/1918893912","heartbeat_front_addr":"172.21.15.6:6831/1918893912","state":["exists","up"]},{"osd":10,"uuid":"04e9b5ef-ac0c-482b-a69e-a7121b36e7da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6825","nonce":1395777729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6827","nonce":1395777729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6831","nonce":1395777729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6829","nonce":1395777729}]},"public_addr":"172.21.15.29:6825/1395777729","cluster_addr":"172.21.15.29:6827/1395777729","heartbeat_back_addr":"172.21.15.29:6831/1395777729","heartbeat_front_addr":"172.21.15.29:6829/1395777729","state":["exists","up"]},{"osd":11,"uuid":"cb1638ce-4189-4241-b63e-b63c607e90ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6825","nonce":3322197880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6827","nonce":3322197880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6831","nonce":3322197880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6829","nonce":3322197880}]},"public_addr":"172.21.15.188:6825/3322197880","cluster_addr":"172.21.15.188:6827/3322197880","heartbeat_back_addr":"172.21.15.188:6831/3322197880","heartbeat_front_addr":"172.21.15.188:6829/3322197880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:41.553455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:07.641467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:31.748891+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:03.743553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:30.065262+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:54.976527+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:29.348206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:56.285928+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:21.246409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:54.468561+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:22.093015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:47.342379+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/3143272062":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1325707596":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6800/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6801/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1359518071":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6801/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/2813598711":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/3140150861":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6800/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/65701414":"2024-04-27T01:33:47.021882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:40:57.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.321+0000 7fc886a00700 1 -- 172.21.15.6:0/3524275286 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc8680a3cc0 msgr2=0x7fc8680a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:57.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.321+0000 7fc886a00700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc8680a3cc0 0x7fc8680a6180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc88010e7f0 tx=0x7fc86c009260 comp rx=0 tx=0).stop 2024-04-26T01:40:57.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.321+0000 7fc886a00700 1 -- 172.21.15.6:0/3524275286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880108f80 msgr2=0x7fc88019f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:57.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.321+0000 7fc886a00700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880108f80 0x7fc88019f570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc874006010 tx=0x7fc8740057c0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.321+0000 7fc886a00700 1 -- 172.21.15.6:0/3524275286 shutdown_connections 2024-04-26T01:40:57.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.321+0000 7fc886a00700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc8680a3cc0 0x7fc8680a6180 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.321+0000 7fc886a00700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc88010f030 0x7fc8801a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.321+0000 7fc886a00700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880108f80 0x7fc88019f570 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.321+0000 7fc886a00700 1 --2- 172.21.15.6:0/3524275286 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc880109940 0x7fc88019fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.321+0000 7fc886a00700 1 -- 172.21.15.6:0/3524275286 >> 172.21.15.6:0/3524275286 conn(0x7fc8800787d0 msgr2=0x7fc880079a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:57.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.322+0000 7fc886a00700 1 -- 172.21.15.6:0/3524275286 shutdown_connections 2024-04-26T01:40:57.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:40:57.322+0000 7fc886a00700 1 -- 172.21.15.6:0/3524275286 wait complete. 2024-04-26T01:40:57.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:56 smithi188 ceph-mon[32751]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:57.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:56 smithi029 ceph-mon[32775]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:58.143 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T01:36:35.182726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T01:40:58.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T01:40:58.222 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:57 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3524275286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:40:58.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:57 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3524275286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:40:58.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:57 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3524275286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:40:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:59.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:40:58 smithi006 ceph-mon[28419]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:59.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:40:58 smithi188 ceph-mon[32751]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:59.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:40:58 smithi029 ceph-mon[32775]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:00.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.674+0000 7f4ef0c00700 1 -- 172.21.15.6:0/1448487171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eec103390 msgr2=0x7f4eec10b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:00.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.674+0000 7f4ef0c00700 1 --2- 172.21.15.6:0/1448487171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eec103390 0x7f4eec10b7f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009990 tx=0x7f4ed8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.675+0000 7f4ef0c00700 1 -- 172.21.15.6:0/1448487171 shutdown_connections 2024-04-26T01:41:00.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.675+0000 7f4ef0c00700 1 --2- 172.21.15.6:0/1448487171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4eec10be60 0x7f4eec10e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.675+0000 7f4ef0c00700 1 --2- 172.21.15.6:0/1448487171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4eec1029e0 0x7f4eec102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.675+0000 7f4ef0c00700 1 --2- 172.21.15.6:0/1448487171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eec103390 0x7f4eec10b7f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.675+0000 7f4ef0c00700 1 -- 172.21.15.6:0/1448487171 >> 172.21.15.6:0/1448487171 conn(0x7f4eec0766e0 msgr2=0x7f4eec076af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:00.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.675+0000 7f4ef0c00700 1 -- 172.21.15.6:0/1448487171 shutdown_connections 2024-04-26T01:41:00.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.675+0000 7f4ef0c00700 1 -- 172.21.15.6:0/1448487171 wait complete. 2024-04-26T01:41:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.676+0000 7f4ef0c00700 1 Processor -- start 2024-04-26T01:41:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.676+0000 7f4ef0c00700 1 -- start start 2024-04-26T01:41:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.676+0000 7f4ef0c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4eec1029e0 0x7f4eec19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.676+0000 7f4ef0c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4eec103390 0x7f4eec19b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.676+0000 7f4ef0c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eec10be60 0x7f4eec19fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.676+0000 7f4ef0c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec1022c0 con 0x7f4eec103390 2024-04-26T01:41:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.676+0000 7f4ef0c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec102140 con 0x7f4eec10be60 2024-04-26T01:41:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.676+0000 7f4ef0c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec102440 con 0x7f4eec1029e0 2024-04-26T01:41:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.677+0000 7f4eebe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eec10be60 0x7f4eec19fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.677+0000 7f4eeaa00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4eec103390 0x7f4eec19b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.677+0000 7f4eebe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eec10be60 0x7f4eec19fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:45412/0 (socket says 172.21.15.6:45412) 2024-04-26T01:41:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.677+0000 7f4eebe00700 1 -- 172.21.15.6:0/535990315 learned_addr learned my addr 172.21.15.6:0/535990315 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:41:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.677+0000 7f4eeb400700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4eec1029e0 0x7f4eec19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.677+0000 7f4eebe00700 1 -- 172.21.15.6:0/535990315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4eec1029e0 msgr2=0x7f4eec19b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.677+0000 7f4eebe00700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4eec1029e0 0x7f4eec19b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.677+0000 7f4eebe00700 1 -- 172.21.15.6:0/535990315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4eec103390 msgr2=0x7f4eec19b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.677+0000 7f4eebe00700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4eec103390 0x7f4eec19b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.677+0000 7f4eebe00700 1 -- 172.21.15.6:0/535990315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eec1a0300 con 0x7f4eec10be60 2024-04-26T01:41:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.678+0000 7f4eebe00700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eec10be60 0x7f4eec19fbc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400ef30 tx=0x7f4ee400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.678+0000 7f4ee3e00700 1 -- 172.21.15.6:0/535990315 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee40092e0 con 0x7f4eec10be60 2024-04-26T01:41:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.678+0000 7f4ef0c00700 1 -- 172.21.15.6:0/535990315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eec1a0590 con 0x7f4eec10be60 2024-04-26T01:41:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.678+0000 7f4ee3e00700 1 -- 172.21.15.6:0/535990315 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ee4004bb0 con 0x7f4eec10be60 2024-04-26T01:41:00.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.678+0000 7f4ee3e00700 1 -- 172.21.15.6:0/535990315 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4004020 con 0x7f4eec10be60 2024-04-26T01:41:00.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.678+0000 7f4ef0c00700 1 -- 172.21.15.6:0/535990315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eec1a7e70 con 0x7f4eec10be60 2024-04-26T01:41:00.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.679+0000 7f4ef0c00700 1 -- 172.21.15.6:0/535990315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eec069730 con 0x7f4eec10be60 2024-04-26T01:41:00.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.681+0000 7f4ee3e00700 1 -- 172.21.15.6:0/535990315 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f4ee401b030 con 0x7f4eec10be60 2024-04-26T01:41:00.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.683+0000 7f4ee3e00700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4ed00a3c40 0x7f4ed00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:00.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.683+0000 7f4ee3e00700 1 -- 172.21.15.6:0/535990315 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f4ee40cf9b0 con 0x7f4eec10be60 2024-04-26T01:41:00.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.683+0000 7f4eeb400700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4ed00a3c40 0x7f4ed00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:00.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.684+0000 7f4eeb400700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4ed00a3c40 0x7f4ed00a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009c60 tx=0x7f4edc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:00.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.687+0000 7f4ee3e00700 1 -- 172.21.15.6:0/535990315 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ee4022020 con 0x7f4eec10be60 2024-04-26T01:41:00.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.916+0000 7f4ef0c00700 1 -- 172.21.15.6:0/535990315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4eec1a8040 con 0x7f4eec10be60 2024-04-26T01:41:00.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.917+0000 7f4ee3e00700 1 -- 172.21.15.6:0/535990315 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7f4ee4010040 con 0x7f4eec10be60 2024-04-26T01:41:00.913 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-26T01:41:00.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.920+0000 7f4ef0c00700 1 -- 172.21.15.6:0/535990315 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4ed00a3c40 msgr2=0x7f4ed00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:00.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.920+0000 7f4ef0c00700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4ed00a3c40 0x7f4ed00a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009c60 tx=0x7f4edc009260 comp rx=0 tx=0).stop 2024-04-26T01:41:00.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.920+0000 7f4ef0c00700 1 -- 172.21.15.6:0/535990315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eec10be60 msgr2=0x7f4eec19fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:00.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.920+0000 7f4ef0c00700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eec10be60 0x7f4eec19fbc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400ef30 tx=0x7f4ee400c3a0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.920+0000 7f4ef0c00700 1 -- 172.21.15.6:0/535990315 shutdown_connections 2024-04-26T01:41:00.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.920+0000 7f4ef0c00700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eec10be60 0x7f4eec19fbc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.920+0000 7f4ef0c00700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4ed00a3c40 0x7f4ed00a6100 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.920+0000 7f4ef0c00700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4eec1029e0 0x7f4eec19b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.920+0000 7f4ef0c00700 1 --2- 172.21.15.6:0/535990315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4eec103390 0x7f4eec19b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:00.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.920+0000 7f4ef0c00700 1 -- 172.21.15.6:0/535990315 >> 172.21.15.6:0/535990315 conn(0x7f4eec0766e0 msgr2=0x7f4eec110e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:00.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.920+0000 7f4ef0c00700 1 -- 172.21.15.6:0/535990315 shutdown_connections 2024-04-26T01:41:00.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:00.921+0000 7f4ef0c00700 1 -- 172.21.15.6:0/535990315 wait complete. 2024-04-26T01:41:01.191 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:00 smithi006 ceph-mon[28419]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:01.191 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:00 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/535990315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T01:41:01.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:00 smithi188 ceph-mon[32751]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:01.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:00 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/535990315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T01:41:01.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:00 smithi029 ceph-mon[32775]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:01.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:00 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/535990315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T01:41:01.680 INFO:tasks.cephadm:Adding mds.a on smithi006 2024-04-26T01:41:01.681 INFO:tasks.cephadm:Adding mds.d on smithi006 2024-04-26T01:41:01.681 INFO:tasks.cephadm:Adding mds.g on smithi006 2024-04-26T01:41:01.681 INFO:tasks.cephadm:Adding mds.j on smithi006 2024-04-26T01:41:01.681 INFO:tasks.cephadm:Adding mds.b on smithi029 2024-04-26T01:41:01.681 INFO:tasks.cephadm:Adding mds.e on smithi029 2024-04-26T01:41:01.681 INFO:tasks.cephadm:Adding mds.h on smithi029 2024-04-26T01:41:01.681 INFO:tasks.cephadm:Adding mds.k on smithi029 2024-04-26T01:41:01.681 INFO:tasks.cephadm:Adding mds.c on smithi188 2024-04-26T01:41:01.681 INFO:tasks.cephadm:Adding mds.f on smithi188 2024-04-26T01:41:01.681 INFO:tasks.cephadm:Adding mds.i on smithi188 2024-04-26T01:41:01.681 INFO:tasks.cephadm:Adding mds.l on smithi188 2024-04-26T01:41:01.681 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi006=a;smithi006=d;smithi006=g;smithi006=j;smithi029=b;smithi029=e;smithi029=h;smithi029=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l' 2024-04-26T01:41:01.929 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:41:03.266 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:02 smithi188 ceph-mon[32751]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:03.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:02 smithi006 ceph-mon[28419]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:03.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:02 smithi029 ceph-mon[32775]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.472+0000 7f075dec7700 1 -- 172.21.15.188:0/718822112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0758075420 msgr2=0x7f07580758a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.472+0000 7f075dec7700 1 --2- 172.21.15.188:0/718822112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0758075420 0x7f07580758a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f074c009990 tx=0x7f074c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.472+0000 7f075dec7700 1 -- 172.21.15.188:0/718822112 shutdown_connections 2024-04-26T01:41:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.472+0000 7f075dec7700 1 --2- 172.21.15.188:0/718822112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f075810a9a0 0x7f075810ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.472+0000 7f075dec7700 1 --2- 172.21.15.188:0/718822112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07580769f0 0x7f0758074e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.472+0000 7f075dec7700 1 --2- 172.21.15.188:0/718822112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0758075420 0x7f07580758a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.472+0000 7f075dec7700 1 -- 172.21.15.188:0/718822112 >> 172.21.15.188:0/718822112 conn(0x7f07580fe040 msgr2=0x7f0758100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.472+0000 7f075dec7700 1 -- 172.21.15.188:0/718822112 shutdown_connections 2024-04-26T01:41:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.472+0000 7f075dec7700 1 -- 172.21.15.188:0/718822112 wait complete. 2024-04-26T01:41:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.472+0000 7f075dec7700 1 Processor -- start 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.472+0000 7f075dec7700 1 -- start start 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f075dec7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0758075420 0x7f075819ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f075dec7700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07580769f0 0x7f075819b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f075dec7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f075810a9a0 0x7f07581a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f075dec7700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0758110770 con 0x7f07580769f0 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f075dec7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f07581105f0 con 0x7f0758075420 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f075dec7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f07581108f0 con 0x7f075810a9a0 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f0757fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f075810a9a0 0x7f07581a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f0757fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f075810a9a0 0x7f07581a23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:49362/0 (socket says 172.21.15.188:49362) 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f0757fff700 1 -- 172.21.15.188:0/3684663348 learned_addr learned my addr 172.21.15.188:0/3684663348 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f07577fe700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0758075420 0x7f075819ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f0757fff700 1 -- 172.21.15.188:0/3684663348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0758075420 msgr2=0x7f075819ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f0757fff700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0758075420 0x7f075819ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.472 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f0757fff700 1 -- 172.21.15.188:0/3684663348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07580769f0 msgr2=0x7f075819b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:03.472 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f0757fff700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07580769f0 0x7f075819b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.472 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f0757fff700 1 -- 172.21.15.188:0/3684663348 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07581a2a90 con 0x7f075810a9a0 2024-04-26T01:41:03.472 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f07577fe700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0758075420 0x7f075819ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:03.472 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f0757fff700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f075810a9a0 0x7f07581a23f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f075000b580 tx=0x7f075000b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:03.472 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f0754ff9700 1 -- 172.21.15.188:0/3684663348 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0750011830 con 0x7f075810a9a0 2024-04-26T01:41:03.472 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f0754ff9700 1 -- 172.21.15.188:0/3684663348 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0750016070 con 0x7f075810a9a0 2024-04-26T01:41:03.472 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f075dec7700 1 -- 172.21.15.188:0/3684663348 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07581a2d80 con 0x7f075810a9a0 2024-04-26T01:41:03.472 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f0754ff9700 1 -- 172.21.15.188:0/3684663348 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0750010d40 con 0x7f075810a9a0 2024-04-26T01:41:03.472 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.473+0000 7f075dec7700 1 -- 172.21.15.188:0/3684663348 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07581a32e0 con 0x7f075810a9a0 2024-04-26T01:41:03.473 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.474+0000 7f075dec7700 1 -- 172.21.15.188:0/3684663348 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f075802df80 con 0x7f075810a9a0 2024-04-26T01:41:03.473 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.476+0000 7f0754ff9700 1 -- 172.21.15.188:0/3684663348 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f07500119d0 con 0x7f075810a9a0 2024-04-26T01:41:03.473 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.477+0000 7f0754ff9700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07400a3cc0 0x7f07400a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:03.473 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.477+0000 7f0754ff9700 1 -- 172.21.15.188:0/3684663348 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f07500cfaf0 con 0x7f075810a9a0 2024-04-26T01:41:03.473 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.477+0000 7f07577fe700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07400a3cc0 0x7f07400a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:03.474 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.478+0000 7f07577fe700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07400a3cc0 0x7f07400a6180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0748007fd0 tx=0x7f0748007830 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:03.477 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.480+0000 7f0754ff9700 1 -- 172.21.15.188:0/3684663348 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f075009c970 con 0x7f075810a9a0 2024-04-26T01:41:03.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.689+0000 7f075dec7700 1 -- 172.21.15.188:0/3684663348 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi006=a;smithi006=d;smithi006=g;smithi006=j;smithi029=b;smithi029=e;smithi029=h;smithi029=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}) -- 0x7f0758062f80 con 0x7f07400a3cc0 2024-04-26T01:41:03.702 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.706+0000 7f0754ff9700 1 -- 172.21.15.188:0/3684663348 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f0758062f80 con 0x7f07400a3cc0 2024-04-26T01:41:03.702 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mds.all update... 2024-04-26T01:41:03.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.708+0000 7f075dec7700 1 -- 172.21.15.188:0/3684663348 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07400a3cc0 msgr2=0x7f07400a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:03.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.708+0000 7f075dec7700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07400a3cc0 0x7f07400a6180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0748007fd0 tx=0x7f0748007830 comp rx=0 tx=0).stop 2024-04-26T01:41:03.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.708+0000 7f075dec7700 1 -- 172.21.15.188:0/3684663348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f075810a9a0 msgr2=0x7f07581a23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:03.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.708+0000 7f075dec7700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f075810a9a0 0x7f07581a23f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f075000b580 tx=0x7f075000b940 comp rx=0 tx=0).stop 2024-04-26T01:41:03.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.708+0000 7f075dec7700 1 -- 172.21.15.188:0/3684663348 shutdown_connections 2024-04-26T01:41:03.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.708+0000 7f075dec7700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07400a3cc0 0x7f07400a6180 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.706 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.708+0000 7f075dec7700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f075810a9a0 0x7f07581a23f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.706 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.708+0000 7f075dec7700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0758075420 0x7f075819ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.706 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.708+0000 7f075dec7700 1 --2- 172.21.15.188:0/3684663348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07580769f0 0x7f075819b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:03.706 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.708+0000 7f075dec7700 1 -- 172.21.15.188:0/3684663348 >> 172.21.15.188:0/3684663348 conn(0x7f07580fe040 msgr2=0x7f07581068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:03.706 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.709+0000 7f075dec7700 1 -- 172.21.15.188:0/3684663348 shutdown_connections 2024-04-26T01:41:03.706 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-26T01:41:03.709+0000 7f075dec7700 1 -- 172.21.15.188:0/3684663348 wait complete. 2024-04-26T01:41:04.439 DEBUG:teuthology.orchestra.run.smithi006:mds.a> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.a.service 2024-04-26T01:41:04.445 DEBUG:teuthology.orchestra.run.smithi006:mds.d> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.d.service 2024-04-26T01:41:04.491 DEBUG:teuthology.orchestra.run.smithi006:mds.g> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.g.service 2024-04-26T01:41:04.538 DEBUG:teuthology.orchestra.run.smithi006:mds.j> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.j.service 2024-04-26T01:41:04.584 DEBUG:teuthology.orchestra.run.smithi029:mds.b> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.b.service 2024-04-26T01:41:04.587 DEBUG:teuthology.orchestra.run.smithi029:mds.e> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.e.service 2024-04-26T01:41:04.630 DEBUG:teuthology.orchestra.run.smithi029:mds.h> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.h.service 2024-04-26T01:41:04.673 DEBUG:teuthology.orchestra.run.smithi029:mds.k> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.k.service 2024-04-26T01:41:04.716 DEBUG:teuthology.orchestra.run.smithi188:mds.c> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.c.service 2024-04-26T01:41:04.719 DEBUG:teuthology.orchestra.run.smithi188:mds.f> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.f.service 2024-04-26T01:41:04.762 DEBUG:teuthology.orchestra.run.smithi188:mds.i> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.i.service 2024-04-26T01:41:04.805 DEBUG:teuthology.orchestra.run.smithi188:mds.l> sudo journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.l.service 2024-04-26T01:41:04.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:04 smithi006 ceph-mon[28419]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:04.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:04 smithi006 ceph-mon[28419]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi006=a;smithi006=d;smithi006=g;smithi006=j;smithi029=b;smithi029=e;smithi029=h;smithi029=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:04.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:04 smithi006 ceph-mon[28419]: Saving service mds.all spec with placement smithi006=a;smithi006=d;smithi006=g;smithi006=j;smithi029=b;smithi029=e;smithi029=h;smithi029=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2024-04-26T01:41:04.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:04.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:04.848 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-26T01:41:04.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:41:04.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:04 smithi029 ceph-mon[32775]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:04.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:04 smithi029 ceph-mon[32775]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi006=a;smithi006=d;smithi006=g;smithi006=j;smithi029=b;smithi029=e;smithi029=h;smithi029=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:04.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:04 smithi029 ceph-mon[32775]: Saving service mds.all spec with placement smithi006=a;smithi006=d;smithi006=g;smithi006=j;smithi029=b;smithi029=e;smithi029=h;smithi029=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2024-04-26T01:41:04.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:04.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:05.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:04 smithi188 ceph-mon[32751]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:05.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:04 smithi188 ceph-mon[32751]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi006=a;smithi006=d;smithi006=g;smithi006=j;smithi029=b;smithi029=e;smithi029=h;smithi029=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:05.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:04 smithi188 ceph-mon[32751]: Saving service mds.all spec with placement smithi006=a;smithi006=d;smithi006=g;smithi006=j;smithi029=b;smithi029=e;smithi029=h;smithi029=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2024-04-26T01:41:05.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:05.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:05.778 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:05.778 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:41:05.778 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:05.778 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:05.778 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:05.778 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:05.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:05.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:41:05.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:05.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:05.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:05.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:06.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:06.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:41:06.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:06.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:06.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:06.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:07.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:06 smithi188 ceph-mon[32751]: Deploying daemon mds.l on smithi188 2024-04-26T01:41:07.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:06 smithi188 ceph-mon[32751]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:07.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:06 smithi006 ceph-mon[28419]: Deploying daemon mds.l on smithi188 2024-04-26T01:41:07.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:06 smithi006 ceph-mon[28419]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:07.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:06 smithi029 ceph-mon[32775]: Deploying daemon mds.l on smithi188 2024-04-26T01:41:07.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:06 smithi029 ceph-mon[32775]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:07.448 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 26 01:41:07 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-l[63295]: starting mds.l at 2024-04-26T01:41:07.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.504+0000 7fcc08800700 1 -- 172.21.15.6:0/4169148435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc0006b900 msgr2=0x7fcc0010efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:07.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.504+0000 7fcc08800700 1 --2- 172.21.15.6:0/4169148435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc0006b900 0x7fcc0010efa0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0009990 tx=0x7fcbf0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.505+0000 7fcc08800700 1 -- 172.21.15.6:0/4169148435 shutdown_connections 2024-04-26T01:41:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.505+0000 7fcc08800700 1 --2- 172.21.15.6:0/4169148435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0010f4e0 0x7fcc001118d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.505+0000 7fcc08800700 1 --2- 172.21.15.6:0/4169148435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc0006afe0 0x7fcc0006b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.505+0000 7fcc08800700 1 --2- 172.21.15.6:0/4169148435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc0006b900 0x7fcc0010efa0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.505+0000 7fcc08800700 1 -- 172.21.15.6:0/4169148435 >> 172.21.15.6:0/4169148435 conn(0x7fcc000788b0 msgr2=0x7fcc00078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.505+0000 7fcc08800700 1 -- 172.21.15.6:0/4169148435 shutdown_connections 2024-04-26T01:41:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.505+0000 7fcc08800700 1 -- 172.21.15.6:0/4169148435 wait complete. 2024-04-26T01:41:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.506+0000 7fcc08800700 1 Processor -- start 2024-04-26T01:41:07.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.506+0000 7fcc08800700 1 -- start start 2024-04-26T01:41:07.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.507+0000 7fcc08800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc0006afe0 0x7fcc0019b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.507+0000 7fcc08800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc0006b900 0x7fcc0019b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.507+0000 7fcc08800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0010f4e0 0x7fcc0019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.507+0000 7fcc08800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcc00114df0 con 0x7fcc0006b900 2024-04-26T01:41:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.507+0000 7fcc08800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcc00114c70 con 0x7fcc0010f4e0 2024-04-26T01:41:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.507+0000 7fcc08800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcc00114f70 con 0x7fcc0006afe0 2024-04-26T01:41:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.507+0000 7fcc05e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc0006afe0 0x7fcc0019b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.507+0000 7fcc05e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc0006afe0 0x7fcc0019b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50804/0 (socket says 172.21.15.6:50804) 2024-04-26T01:41:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.507+0000 7fcc05e00700 1 -- 172.21.15.6:0/371544125 learned_addr learned my addr 172.21.15.6:0/371544125 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:41:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.507+0000 7fcc06800700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0010f4e0 0x7fcc0019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.508+0000 7fcc05400700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc0006b900 0x7fcc0019b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.508+0000 7fcc05e00700 1 -- 172.21.15.6:0/371544125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0010f4e0 msgr2=0x7fcc0019fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.508+0000 7fcc05e00700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0010f4e0 0x7fcc0019fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.508+0000 7fcc05e00700 1 -- 172.21.15.6:0/371544125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc0006b900 msgr2=0x7fcc0019b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.508+0000 7fcc05e00700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc0006b900 0x7fcc0019b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.508+0000 7fcc05e00700 1 -- 172.21.15.6:0/371544125 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc001a0330 con 0x7fcc0006afe0 2024-04-26T01:41:07.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.508+0000 7fcc05400700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc0006b900 0x7fcc0019b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:07.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.508+0000 7fcc05e00700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc0006afe0 0x7fcc0019b320 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400eb30 tx=0x7fcbf400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:07.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.509+0000 7fcbfaa00700 1 -- 172.21.15.6:0/371544125 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf400cc90 con 0x7fcc0006afe0 2024-04-26T01:41:07.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.509+0000 7fcbfaa00700 1 -- 172.21.15.6:0/371544125 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbf40044d0 con 0x7fcc0006afe0 2024-04-26T01:41:07.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.509+0000 7fcc08800700 1 -- 172.21.15.6:0/371544125 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc001a0620 con 0x7fcc0006afe0 2024-04-26T01:41:07.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.509+0000 7fcbfaa00700 1 -- 172.21.15.6:0/371544125 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4010430 con 0x7fcc0006afe0 2024-04-26T01:41:07.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.509+0000 7fcc08800700 1 -- 172.21.15.6:0/371544125 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc001a7ea0 con 0x7fcc0006afe0 2024-04-26T01:41:07.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.510+0000 7fcc08800700 1 -- 172.21.15.6:0/371544125 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc00069730 con 0x7fcc0006afe0 2024-04-26T01:41:07.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.512+0000 7fcbfaa00700 1 -- 172.21.15.6:0/371544125 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fcbf4004020 con 0x7fcc0006afe0 2024-04-26T01:41:07.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.513+0000 7fcbfaa00700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fcbe80a3cc0 0x7fcbe80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:07.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.513+0000 7fcbfaa00700 1 -- 172.21.15.6:0/371544125 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fcbf40d0230 con 0x7fcc0006afe0 2024-04-26T01:41:07.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.513+0000 7fcc05400700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fcbe80a3cc0 0x7fcbe80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:07.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.514+0000 7fcc05400700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fcbe80a3cc0 0x7fcbe80a6180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0006010 tx=0x7fcbf001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:07.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.517+0000 7fcbfaa00700 1 -- 172.21.15.6:0/371544125 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcbf409d010 con 0x7fcc0006afe0 2024-04-26T01:41:07.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.798+0000 7fcc08800700 1 -- 172.21.15.6:0/371544125 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc0002df80 con 0x7fcc0006afe0 2024-04-26T01:41:07.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.799+0000 7fcbfaa00700 1 -- 172.21.15.6:0/371544125 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fcbf409c910 con 0x7fcc0006afe0 2024-04-26T01:41:07.796 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:41:07.796 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T01:41:07.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.803+0000 7fcc08800700 1 -- 172.21.15.6:0/371544125 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fcbe80a3cc0 msgr2=0x7fcbe80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:07.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.803+0000 7fcc08800700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fcbe80a3cc0 0x7fcbe80a6180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0006010 tx=0x7fcbf001a040 comp rx=0 tx=0).stop 2024-04-26T01:41:07.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.803+0000 7fcc08800700 1 -- 172.21.15.6:0/371544125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc0006afe0 msgr2=0x7fcc0019b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:07.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.803+0000 7fcc08800700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc0006afe0 0x7fcc0019b320 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400eb30 tx=0x7fcbf400ee40 comp rx=0 tx=0).stop 2024-04-26T01:41:07.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.803+0000 7fcc08800700 1 -- 172.21.15.6:0/371544125 shutdown_connections 2024-04-26T01:41:07.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.803+0000 7fcc08800700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0010f4e0 0x7fcc0019fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.803+0000 7fcc08800700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fcbe80a3cc0 0x7fcbe80a6180 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.803+0000 7fcc08800700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc0006afe0 0x7fcc0019b320 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.803+0000 7fcc08800700 1 --2- 172.21.15.6:0/371544125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc0006b900 0x7fcc0019b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.803+0000 7fcc08800700 1 -- 172.21.15.6:0/371544125 >> 172.21.15.6:0/371544125 conn(0x7fcc000788b0 msgr2=0x7fcc00110930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:07.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.803+0000 7fcc08800700 1 -- 172.21.15.6:0/371544125 shutdown_connections 2024-04-26T01:41:07.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:07.804+0000 7fcc08800700 1 -- 172.21.15.6:0/371544125 wait complete. 2024-04-26T01:41:07.802 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 2 2024-04-26T01:41:07.890 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 26 01:41:07 smithi188 bash[63280]: 109c8f8d7b6c79a016e44feeae75fa4e22fffa7d9ff2c125d404373d1bcdc60a 2024-04-26T01:41:07.890 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 26 01:41:07 smithi188 systemd[1]: Started Ceph mds.l for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:41:08.471 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:08 smithi006 ceph-mon[28419]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:08.472 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:08 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6832/3510817210,v1:172.21.15.188:6833/3510817210] up:boot 2024-04-26T01:41:08.472 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:08 smithi006 ceph-mon[28419]: fsmap 1 up:standby 2024-04-26T01:41:08.472 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:41:08.472 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:08.472 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:08.472 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:08.472 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:08.472 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:08.472 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:08.472 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:08 smithi006 ceph-mon[28419]: Deploying daemon mds.k on smithi029 2024-04-26T01:41:08.472 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:08 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/371544125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:08.579 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:08 smithi029 ceph-mon[32775]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:08.579 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:08 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6832/3510817210,v1:172.21.15.188:6833/3510817210] up:boot 2024-04-26T01:41:08.579 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:08 smithi029 ceph-mon[32775]: fsmap 1 up:standby 2024-04-26T01:41:08.579 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:41:08.579 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:08.579 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:08.579 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:08.580 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:08.580 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:08.580 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:08.580 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:08 smithi029 ceph-mon[32775]: Deploying daemon mds.k on smithi029 2024-04-26T01:41:08.580 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:08 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/371544125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:08 smithi188 ceph-mon[32751]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:08 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6832/3510817210,v1:172.21.15.188:6833/3510817210] up:boot 2024-04-26T01:41:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:08 smithi188 ceph-mon[32751]: fsmap 1 up:standby 2024-04-26T01:41:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:41:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:08.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:08 smithi188 ceph-mon[32751]: Deploying daemon mds.k on smithi029 2024-04-26T01:41:08.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:08 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/371544125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:10.161 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 01:41:09 smithi029 systemd[1]: Starting Ceph mds.k for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:41:10.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:10 smithi029 ceph-mon[32775]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:10.440 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 01:41:10 smithi029 podman[62664]: 2024-04-26 01:41:10.164945612 +0000 UTC m=+0.104557008 container create a4fa7301454115d8f20386df0122966991b395c0df9075179282478358c7c4d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:41:10.440 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 01:41:10 smithi029 podman[62664]: 2024-04-26 01:41:10.078684821 +0000 UTC m=+0.018296195 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:41:10.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:41:10.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:10 smithi006 ceph-mon[28419]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:10.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:10 smithi188 ceph-mon[32751]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:10.939 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 01:41:10 smithi029 podman[62664]: 2024-04-26 01:41:10.477349302 +0000 UTC m=+0.416960657 container init a4fa7301454115d8f20386df0122966991b395c0df9075179282478358c7c4d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:41:10.940 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 01:41:10 smithi029 podman[62664]: 2024-04-26 01:41:10.480264003 +0000 UTC m=+0.419875357 container start a4fa7301454115d8f20386df0122966991b395c0df9075179282478358c7c4d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:41:10.940 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 01:41:10 smithi029 ceph-mds[62683]: -- 172.21.15.29:0/3637352446 <== mon.2 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d3f1fb9a40 con 0x55d3f1f8d000 2024-04-26T01:41:10.940 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 01:41:10 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-k[62679]: starting mds.k at 2024-04-26T01:41:10.940 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 01:41:10 smithi029 bash[62664]: a4fa7301454115d8f20386df0122966991b395c0df9075179282478358c7c4d4 2024-04-26T01:41:10.940 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 01:41:10 smithi029 systemd[1]: Started Ceph mds.k for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:41:10.995 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:11.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:11 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:11.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:11 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:11.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:11 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:11.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:11 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:11.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:11 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:11.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:11 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:11.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:11 smithi006 ceph-mon[28419]: Deploying daemon mds.j on smithi006 2024-04-26T01:41:11.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:11 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.29:6832/3500196993,v1:172.21.15.29:6833/3500196993] up:boot 2024-04-26T01:41:11.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:11 smithi006 ceph-mon[28419]: fsmap 2 up:standby 2024-04-26T01:41:11.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:11 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:41:11.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:11 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:11.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:11 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:11.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:11 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:11.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:11 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:11.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:11 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:11.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:11 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:11.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:11 smithi188 ceph-mon[32751]: Deploying daemon mds.j on smithi006 2024-04-26T01:41:11.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:11 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.29:6832/3500196993,v1:172.21.15.29:6833/3500196993] up:boot 2024-04-26T01:41:11.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:11 smithi188 ceph-mon[32751]: fsmap 2 up:standby 2024-04-26T01:41:11.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:11 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:41:11.917 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:11 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:11.917 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:11 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:11.917 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:11 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:11.918 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:11 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:11.918 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:11 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:11.918 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:11 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:11.918 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:11 smithi029 ceph-mon[32775]: Deploying daemon mds.j on smithi006 2024-04-26T01:41:11.918 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:11 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.29:6832/3500196993,v1:172.21.15.29:6833/3500196993] up:boot 2024-04-26T01:41:11.918 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:11 smithi029 ceph-mon[32775]: fsmap 2 up:standby 2024-04-26T01:41:11.918 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:11 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:41:12.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:12 smithi029 ceph-mon[32775]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:12.971 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:12 smithi006 ceph-mon[28419]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:13.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:12 smithi188 ceph-mon[32751]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:13.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.137+0000 7fe237200700 1 -- 172.21.15.6:0/836264899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe238077890 msgr2=0x7fe2381132b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:13.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.137+0000 7fe237200700 1 --2- 172.21.15.6:0/836264899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe238077890 0x7fe2381132b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe2200098f0 tx=0x7fe220009c00 comp rx=0 tx=0).stop 2024-04-26T01:41:13.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.138+0000 7fe237200700 1 -- 172.21.15.6:0/836264899 shutdown_connections 2024-04-26T01:41:13.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.138+0000 7fe237200700 1 --2- 172.21.15.6:0/836264899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2381137f0 0x7fe238115be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.138+0000 7fe237200700 1 --2- 172.21.15.6:0/836264899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe238076f70 0x7fe238077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.138+0000 7fe237200700 1 --2- 172.21.15.6:0/836264899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe238077890 0x7fe2381132b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.138+0000 7fe237200700 1 -- 172.21.15.6:0/836264899 >> 172.21.15.6:0/836264899 conn(0x7fe238100620 msgr2=0x7fe238102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.138+0000 7fe237200700 1 -- 172.21.15.6:0/836264899 shutdown_connections 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.138+0000 7fe237200700 1 -- 172.21.15.6:0/836264899 wait complete. 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.139+0000 7fe237200700 1 Processor -- start 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.139+0000 7fe237200700 1 -- start start 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.139+0000 7fe237200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe238076f70 0x7fe23819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.139+0000 7fe237200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe238077890 0x7fe23819fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.139+0000 7fe237200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2381137f0 0x7fe2381a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.139+0000 7fe237200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe2381191c0 con 0x7fe238077890 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.139+0000 7fe237200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe238119040 con 0x7fe2381137f0 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.139+0000 7fe237200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe238119340 con 0x7fe238076f70 2024-04-26T01:41:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.140+0000 7fe22d400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe238077890 0x7fe23819fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.140+0000 7fe22d400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe238077890 0x7fe23819fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45576/0 (socket says 172.21.15.6:45576) 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.140+0000 7fe22d400700 1 -- 172.21.15.6:0/1441531289 learned_addr learned my addr 172.21.15.6:0/1441531289 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.140+0000 7fe235e00700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe238076f70 0x7fe23819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.140+0000 7fe236800700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2381137f0 0x7fe2381a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.140+0000 7fe22d400700 1 -- 172.21.15.6:0/1441531289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe238076f70 msgr2=0x7fe23819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.140+0000 7fe22d400700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe238076f70 0x7fe23819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.140+0000 7fe22d400700 1 -- 172.21.15.6:0/1441531289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2381137f0 msgr2=0x7fe2381a4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.140+0000 7fe22d400700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2381137f0 0x7fe2381a4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.140+0000 7fe22d400700 1 -- 172.21.15.6:0/1441531289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2381a4750 con 0x7fe238077890 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.140+0000 7fe236800700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2381137f0 0x7fe2381a4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.140+0000 7fe22d400700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe238077890 0x7fe23819fcd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe220009230 tx=0x7fe2200094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.141+0000 7fe22f400700 1 -- 172.21.15.6:0/1441531289 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe220005220 con 0x7fe238077890 2024-04-26T01:41:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.141+0000 7fe22f400700 1 -- 172.21.15.6:0/1441531289 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe22000b560 con 0x7fe238077890 2024-04-26T01:41:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.141+0000 7fe22f400700 1 -- 172.21.15.6:0/1441531289 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe220003a60 con 0x7fe238077890 2024-04-26T01:41:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.141+0000 7fe237200700 1 -- 172.21.15.6:0/1441531289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2381a49e0 con 0x7fe238077890 2024-04-26T01:41:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.141+0000 7fe237200700 1 -- 172.21.15.6:0/1441531289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2381ac2c0 con 0x7fe238077890 2024-04-26T01:41:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.142+0000 7fe237200700 1 -- 172.21.15.6:0/1441531289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2380786b0 con 0x7fe238077890 2024-04-26T01:41:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.144+0000 7fe22f400700 1 -- 172.21.15.6:0/1441531289 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fe22000ba10 con 0x7fe238077890 2024-04-26T01:41:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.146+0000 7fe22f400700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe21c0a3c40 0x7fe21c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.146+0000 7fe22f400700 1 -- 172.21.15.6:0/1441531289 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fe2200d1180 con 0x7fe238077890 2024-04-26T01:41:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.146+0000 7fe235e00700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe21c0a3c40 0x7fe21c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.147+0000 7fe235e00700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe21c0a3c40 0x7fe21c0a6100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe22800f540 tx=0x7fe22800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.149+0000 7fe22f400700 1 -- 172.21.15.6:0/1441531289 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe22005eff0 con 0x7fe238077890 2024-04-26T01:41:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.436+0000 7fe237200700 1 -- 172.21.15.6:0/1441531289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2381a0aa0 con 0x7fe238077890 2024-04-26T01:41:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.437+0000 7fe22f400700 1 -- 172.21.15.6:0/1441531289 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fe220013030 con 0x7fe238077890 2024-04-26T01:41:13.520 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:41:13.520 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T01:41:13.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.440+0000 7fe237200700 1 -- 172.21.15.6:0/1441531289 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe21c0a3c40 msgr2=0x7fe21c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:13.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.440+0000 7fe237200700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe21c0a3c40 0x7fe21c0a6100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe22800f540 tx=0x7fe22800b040 comp rx=0 tx=0).stop 2024-04-26T01:41:13.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.440+0000 7fe237200700 1 -- 172.21.15.6:0/1441531289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe238077890 msgr2=0x7fe23819fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:13.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.440+0000 7fe237200700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe238077890 0x7fe23819fcd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe220009230 tx=0x7fe2200094a0 comp rx=0 tx=0).stop 2024-04-26T01:41:13.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.440+0000 7fe237200700 1 -- 172.21.15.6:0/1441531289 shutdown_connections 2024-04-26T01:41:13.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.440+0000 7fe237200700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2381137f0 0x7fe2381a4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:13.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.440+0000 7fe237200700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe21c0a3c40 0x7fe21c0a6100 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:13.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.440+0000 7fe237200700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe238076f70 0x7fe23819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:13.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.440+0000 7fe237200700 1 --2- 172.21.15.6:0/1441531289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe238077890 0x7fe23819fcd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:13.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.440+0000 7fe237200700 1 -- 172.21.15.6:0/1441531289 >> 172.21.15.6:0/1441531289 conn(0x7fe238100620 msgr2=0x7fe238102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:13.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.441+0000 7fe237200700 1 -- 172.21.15.6:0/1441531289 shutdown_connections 2024-04-26T01:41:13.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:13.441+0000 7fe237200700 1 -- 172.21.15.6:0/1441531289 wait complete. 2024-04-26T01:41:13.521 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 3 2024-04-26T01:41:13.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:13 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1441531289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:14.036 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:13 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1441531289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:14.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:13 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1441531289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:14.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:14 smithi006 ceph-mon[28419]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:14.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:14 smithi188 ceph-mon[32751]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:14.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:14 smithi029 ceph-mon[32775]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:15.242 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 01:41:14 smithi006 systemd[1]: Starting Ceph mds.j for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:41:15.563 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 01:41:15 smithi006 podman[69185]: 2024-04-26 01:41:15.150115653 +0000 UTC m=+0.037132594 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:41:15.563 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 01:41:15 smithi006 podman[69185]: 2024-04-26 01:41:15.27076006 +0000 UTC m=+0.157777031 container create 1eef28c3762a032050a3979688988c00e670bce79ca381e6760f54333486e581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-26T01:41:15.564 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 01:41:15 smithi006 podman[69185]: 2024-04-26 01:41:15.566081155 +0000 UTC m=+0.453098133 container init 1eef28c3762a032050a3979688988c00e670bce79ca381e6760f54333486e581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=) 2024-04-26T01:41:15.816 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 01:41:15 smithi006 podman[69185]: 2024-04-26 01:41:15.575519117 +0000 UTC m=+0.462536070 container start 1eef28c3762a032050a3979688988c00e670bce79ca381e6760f54333486e581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T01:41:15.817 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 01:41:15 smithi006 ceph-mds[69203]: -- 172.21.15.6:0/3036806770 <== mon.0 v2:172.21.15.6:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e11b9d9a40 con 0x55e11b9acc00 2024-04-26T01:41:15.817 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 01:41:15 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-j[69199]: starting mds.j at 2024-04-26T01:41:15.817 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 01:41:15 smithi006 bash[69185]: 1eef28c3762a032050a3979688988c00e670bce79ca381e6760f54333486e581 2024-04-26T01:41:15.817 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 01:41:15 smithi006 systemd[1]: Started Ceph mds.j for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:41:16.631 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:41:17.022 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:16 smithi006 ceph-mon[28419]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:17.023 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:16 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.6:6834/3962248183,v1:172.21.15.6:6835/3962248183] up:boot 2024-04-26T01:41:17.023 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:16 smithi006 ceph-mon[28419]: fsmap 3 up:standby 2024-04-26T01:41:17.023 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:41:17.023 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:17.023 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:17.023 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:17.023 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:17.023 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:17.023 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:16 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:17.023 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:16 smithi006 ceph-mon[28419]: Deploying daemon mds.d on smithi006 2024-04-26T01:41:17.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:16 smithi188 ceph-mon[32751]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:17.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:16 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.6:6834/3962248183,v1:172.21.15.6:6835/3962248183] up:boot 2024-04-26T01:41:17.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:16 smithi188 ceph-mon[32751]: fsmap 3 up:standby 2024-04-26T01:41:17.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:41:17.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:17.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:17.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:17.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:17.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:17.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:16 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:17.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:16 smithi188 ceph-mon[32751]: Deploying daemon mds.d on smithi006 2024-04-26T01:41:17.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:16 smithi029 ceph-mon[32775]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:17.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:16 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.6:6834/3962248183,v1:172.21.15.6:6835/3962248183] up:boot 2024-04-26T01:41:17.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:16 smithi029 ceph-mon[32775]: fsmap 3 up:standby 2024-04-26T01:41:17.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:41:17.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:17.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:17.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:17.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:17.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:17.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:16 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:17.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:16 smithi029 ceph-mon[32775]: Deploying daemon mds.d on smithi006 2024-04-26T01:41:17.547 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:18.807 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:18 smithi006 ceph-mon[28419]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:19.066 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 01:41:18 smithi006 systemd[1]: Starting Ceph mds.d for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:41:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:18 smithi188 ceph-mon[32751]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:19.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:18 smithi029 ceph-mon[32775]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:19.816 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 01:41:19 smithi006 podman[69577]: 2024-04-26 01:41:19.625160448 +0000 UTC m=+0.040346755 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:41:19.816 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 01:41:19 smithi006 podman[69577]: 2024-04-26 01:41:19.781093149 +0000 UTC m=+0.196279478 container create f7c3779515b9b0434484859310f831f2ea40ae5731c372d6dce2c4a09cdae79a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T01:41:20.429 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 01:41:20 smithi006 podman[69577]: 2024-04-26 01:41:20.21997194 +0000 UTC m=+0.635158248 container init f7c3779515b9b0434484859310f831f2ea40ae5731c372d6dce2c4a09cdae79a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-26T01:41:20.429 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 01:41:20 smithi006 podman[69577]: 2024-04-26 01:41:20.232471621 +0000 UTC m=+0.647657915 container start f7c3779515b9b0434484859310f831f2ea40ae5731c372d6dce2c4a09cdae79a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-26T01:41:20.429 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 01:41:20 smithi006 ceph-mds[69618]: -- 172.21.15.6:0/3590794795 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5599f5b97a40 con 0x5599f5b6b000 2024-04-26T01:41:20.429 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 01:41:20 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-d[69610]: starting mds.d at 2024-04-26T01:41:20.429 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 01:41:20 smithi006 bash[69577]: f7c3779515b9b0434484859310f831f2ea40ae5731c372d6dce2c4a09cdae79a 2024-04-26T01:41:20.429 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 01:41:20 smithi006 systemd[1]: Started Ceph mds.d for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:41:20.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.601+0000 7f4b96e00700 1 -- 172.21.15.6:0/1879292946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b90102af0 msgr2=0x7f4b90102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:20.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.601+0000 7f4b96e00700 1 --2- 172.21.15.6:0/1879292946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b90102af0 0x7f4b90102f70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0099c0 tx=0x7f4b7c009cd0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.601+0000 7f4b96e00700 1 -- 172.21.15.6:0/1879292946 shutdown_connections 2024-04-26T01:41:20.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.601+0000 7f4b96e00700 1 --2- 172.21.15.6:0/1879292946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b901034b0 0x7f4b9010f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.601+0000 7f4b96e00700 1 --2- 172.21.15.6:0/1879292946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b90107940 0x7f4b90107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.601+0000 7f4b96e00700 1 --2- 172.21.15.6:0/1879292946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b90102af0 0x7f4b90102f70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.601+0000 7f4b96e00700 1 -- 172.21.15.6:0/1879292946 >> 172.21.15.6:0/1879292946 conn(0x7f4b90079ba0 msgr2=0x7f4b90079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:20.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.602+0000 7f4b96e00700 1 -- 172.21.15.6:0/1879292946 shutdown_connections 2024-04-26T01:41:20.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.602+0000 7f4b96e00700 1 -- 172.21.15.6:0/1879292946 wait complete. 2024-04-26T01:41:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.603+0000 7f4b96e00700 1 Processor -- start 2024-04-26T01:41:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.603+0000 7f4b96e00700 1 -- start start 2024-04-26T01:41:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.604+0000 7f4b96e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b90102af0 0x7f4b901990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.604+0000 7f4b96e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b901034b0 0x7f4b90199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.604+0000 7f4b96e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b90107940 0x7f4b9019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.604+0000 7f4b96e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4b90112c30 con 0x7f4b90102af0 2024-04-26T01:41:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.604+0000 7f4b96e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4b90112ab0 con 0x7f4b90107940 2024-04-26T01:41:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.604+0000 7f4b96e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4b90112db0 con 0x7f4b901034b0 2024-04-26T01:41:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.604+0000 7f4b8fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b90102af0 0x7f4b901990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.604+0000 7f4b8fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b90102af0 0x7f4b901990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37736/0 (socket says 172.21.15.6:37736) 2024-04-26T01:41:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.604+0000 7f4b8fe00700 1 -- 172.21.15.6:0/2723347076 learned_addr learned my addr 172.21.15.6:0/2723347076 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:41:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.604+0000 7f4b94e00700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b90107940 0x7f4b9019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.604+0000 7f4b8f400700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b901034b0 0x7f4b90199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:20.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.605+0000 7f4b8f400700 1 -- 172.21.15.6:0/2723347076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b90107940 msgr2=0x7f4b9019d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:20.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.605+0000 7f4b8f400700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b90107940 0x7f4b9019d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.605+0000 7f4b8f400700 1 -- 172.21.15.6:0/2723347076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b90102af0 msgr2=0x7f4b901990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:20.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.605+0000 7f4b8f400700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b90102af0 0x7f4b901990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.605+0000 7f4b8f400700 1 -- 172.21.15.6:0/2723347076 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9019e0d0 con 0x7f4b901034b0 2024-04-26T01:41:20.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.605+0000 7f4b8f400700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b901034b0 0x7f4b90199600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00b580 tx=0x7f4b7c0055f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:20.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.606+0000 7f4b8cc00700 1 -- 172.21.15.6:0/2723347076 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c026070 con 0x7f4b901034b0 2024-04-26T01:41:20.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.606+0000 7f4b8cc00700 1 -- 172.21.15.6:0/2723347076 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b7c00f460 con 0x7f4b901034b0 2024-04-26T01:41:20.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.606+0000 7f4b8cc00700 1 -- 172.21.15.6:0/2723347076 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c009e60 con 0x7f4b901034b0 2024-04-26T01:41:20.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.606+0000 7f4b96e00700 1 -- 172.21.15.6:0/2723347076 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9019e360 con 0x7f4b901034b0 2024-04-26T01:41:20.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.606+0000 7f4b96e00700 1 -- 172.21.15.6:0/2723347076 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b901a5c40 con 0x7f4b901034b0 2024-04-26T01:41:20.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.608+0000 7f4b96e00700 1 -- 172.21.15.6:0/2723347076 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b90069730 con 0x7f4b901034b0 2024-04-26T01:41:20.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.609+0000 7f4b8cc00700 1 -- 172.21.15.6:0/2723347076 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f4b7c029070 con 0x7f4b901034b0 2024-04-26T01:41:20.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.612+0000 7f4b8cc00700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4b740a3c40 0x7f4b740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:20.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.612+0000 7f4b8cc00700 1 -- 172.21.15.6:0/2723347076 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f4b7c0d06e0 con 0x7f4b901034b0 2024-04-26T01:41:20.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.612+0000 7f4b8fe00700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4b740a3c40 0x7f4b740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:20.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.613+0000 7f4b8fe00700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4b740a3c40 0x7f4b740a6100 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4b80007c10 tx=0x7f4b80007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:20.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.615+0000 7f4b8cc00700 1 -- 172.21.15.6:0/2723347076 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b7c05dff0 con 0x7f4b901034b0 2024-04-26T01:41:20.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.909+0000 7f4b96e00700 1 -- 172.21.15.6:0/2723347076 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b9002df80 con 0x7f4b901034b0 2024-04-26T01:41:20.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.910+0000 7f4b8cc00700 1 -- 172.21.15.6:0/2723347076 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3862 (secure 0 0 0) 0x7f4b7c013070 con 0x7f4b901034b0 2024-04-26T01:41:20.907 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:41:20.907 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-26T01:41:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.913+0000 7f4b96e00700 1 -- 172.21.15.6:0/2723347076 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4b740a3c40 msgr2=0x7f4b740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.913+0000 7f4b96e00700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4b740a3c40 0x7f4b740a6100 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4b80007c10 tx=0x7f4b80007480 comp rx=0 tx=0).stop 2024-04-26T01:41:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.913+0000 7f4b96e00700 1 -- 172.21.15.6:0/2723347076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b901034b0 msgr2=0x7f4b90199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.913+0000 7f4b96e00700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b901034b0 0x7f4b90199600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00b580 tx=0x7f4b7c0055f0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.913+0000 7f4b96e00700 1 -- 172.21.15.6:0/2723347076 shutdown_connections 2024-04-26T01:41:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.913+0000 7f4b96e00700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b90107940 0x7f4b9019d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.913+0000 7f4b96e00700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b90102af0 0x7f4b901990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.913+0000 7f4b96e00700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4b740a3c40 0x7f4b740a6100 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.913+0000 7f4b96e00700 1 --2- 172.21.15.6:0/2723347076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b901034b0 0x7f4b90199600 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.913+0000 7f4b96e00700 1 -- 172.21.15.6:0/2723347076 >> 172.21.15.6:0/2723347076 conn(0x7f4b90079ba0 msgr2=0x7f4b90100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:20.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.913+0000 7f4b96e00700 1 -- 172.21.15.6:0/2723347076 shutdown_connections 2024-04-26T01:41:20.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:20.913+0000 7f4b96e00700 1 -- 172.21.15.6:0/2723347076 wait complete. 2024-04-26T01:41:20.912 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 5 2024-04-26T01:41:20.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:20 smithi006 ceph-mon[28419]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:20.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:20.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:20.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:20.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:20.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:20.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:21.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:20 smithi188 ceph-mon[32751]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:21.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:21.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:21.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:21.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:21.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:21.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:21.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:20 smithi029 ceph-mon[32775]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:21.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:21.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:21.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:21.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:21.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:21.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:21.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:21 smithi006 ceph-mon[28419]: Deploying daemon mds.e on smithi029 2024-04-26T01:41:21.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:21 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.6:6836/680270961,v1:172.21.15.6:6837/680270961] up:boot 2024-04-26T01:41:21.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:21 smithi006 ceph-mon[28419]: fsmap 4 up:standby 2024-04-26T01:41:21.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:41:21.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:21 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2723347076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:21.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:21 smithi029 ceph-mon[32775]: Deploying daemon mds.e on smithi029 2024-04-26T01:41:21.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:21 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.6:6836/680270961,v1:172.21.15.6:6837/680270961] up:boot 2024-04-26T01:41:21.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:21 smithi029 ceph-mon[32775]: fsmap 4 up:standby 2024-04-26T01:41:21.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:21 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:41:21.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:21 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2723347076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:22.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:21 smithi188 ceph-mon[32751]: Deploying daemon mds.e on smithi029 2024-04-26T01:41:22.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:21 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.6:6836/680270961,v1:172.21.15.6:6837/680270961] up:boot 2024-04-26T01:41:22.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:21 smithi188 ceph-mon[32751]: fsmap 4 up:standby 2024-04-26T01:41:22.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:21 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:41:22.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:21 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2723347076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:22.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:22 smithi029 ceph-mon[32775]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:23.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:22 smithi006 ceph-mon[28419]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:23.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:22 smithi188 ceph-mon[32751]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:23.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:41:23.689 INFO:journalctl@ceph.mds.e.smithi029.stdout:Apr 26 01:41:23 smithi029 systemd[1]: Started Ceph mds.e for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:41:24.070 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:24.481 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:24 smithi006 ceph-mon[28419]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:24.481 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:24.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:24.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:24.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:24.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:24.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:24.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:24 smithi006 ceph-mon[28419]: Deploying daemon mds.f on smithi188 2024-04-26T01:41:24.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:24 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.29:6834/2373773979,v1:172.21.15.29:6835/2373773979] up:boot 2024-04-26T01:41:24.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:24 smithi006 ceph-mon[28419]: fsmap 5 up:standby 2024-04-26T01:41:24.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:24 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:41:24.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:24 smithi029 ceph-mon[32775]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:24 smithi029 ceph-mon[32775]: Deploying daemon mds.f on smithi188 2024-04-26T01:41:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:24 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.29:6834/2373773979,v1:172.21.15.29:6835/2373773979] up:boot 2024-04-26T01:41:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:24 smithi029 ceph-mon[32775]: fsmap 5 up:standby 2024-04-26T01:41:24.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:24 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:41:24.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:24 smithi188 ceph-mon[32751]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:24.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:24.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:24.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:24.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:24.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:24.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:24.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:24 smithi188 ceph-mon[32751]: Deploying daemon mds.f on smithi188 2024-04-26T01:41:24.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:24 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.29:6834/2373773979,v1:172.21.15.29:6835/2373773979] up:boot 2024-04-26T01:41:24.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:24 smithi188 ceph-mon[32751]: fsmap 5 up:standby 2024-04-26T01:41:24.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:24 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:41:25.828 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:41:25 smithi188 systemd[1]: Starting Ceph mds.f for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:41:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.025+0000 7fd514200700 1 -- 172.21.15.6:0/123780158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50c108a30 msgr2=0x7fd50c108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.025+0000 7fd514200700 1 --2- 172.21.15.6:0/123780158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50c108a30 0x7fd50c108e10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd500009990 tx=0x7fd500009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.025+0000 7fd514200700 1 -- 172.21.15.6:0/123780158 shutdown_connections 2024-04-26T01:41:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.025+0000 7fd514200700 1 --2- 172.21.15.6:0/123780158 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd50c06b0f0 0x7fd50c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.025+0000 7fd514200700 1 --2- 172.21.15.6:0/123780158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50c108a30 0x7fd50c108e10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.025+0000 7fd514200700 1 --2- 172.21.15.6:0/123780158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50c06a750 0x7fd50c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.025+0000 7fd514200700 1 -- 172.21.15.6:0/123780158 >> 172.21.15.6:0/123780158 conn(0x7fd50c0788f0 msgr2=0x7fd50c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.025+0000 7fd514200700 1 -- 172.21.15.6:0/123780158 shutdown_connections 2024-04-26T01:41:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.025+0000 7fd514200700 1 -- 172.21.15.6:0/123780158 wait complete. 2024-04-26T01:41:26.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.026+0000 7fd514200700 1 Processor -- start 2024-04-26T01:41:26.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.027+0000 7fd514200700 1 -- start start 2024-04-26T01:41:26.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.027+0000 7fd514200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50c06a750 0x7fd50c199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:26.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.027+0000 7fd514200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50c06b0f0 0x7fd50c199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:26.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.027+0000 7fd514200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd50c108a30 0x7fd50c19da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:26.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.027+0000 7fd514200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c112c00 con 0x7fd50c108a30 2024-04-26T01:41:26.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.027+0000 7fd514200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c112a80 con 0x7fd50c06a750 2024-04-26T01:41:26.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.027+0000 7fd514200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c112d80 con 0x7fd50c06b0f0 2024-04-26T01:41:26.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.027+0000 7fd511800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50c06a750 0x7fd50c199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:26.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.027+0000 7fd511800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50c06a750 0x7fd50c199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47436/0 (socket says 172.21.15.6:47436) 2024-04-26T01:41:26.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.027+0000 7fd511800700 1 -- 172.21.15.6:0/3459357569 learned_addr learned my addr 172.21.15.6:0/3459357569 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:41:26.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.027+0000 7fd512200700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd50c108a30 0x7fd50c19da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:26.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.028+0000 7fd510e00700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50c06b0f0 0x7fd50c199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:26.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.028+0000 7fd511800700 1 -- 172.21.15.6:0/3459357569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50c06b0f0 msgr2=0x7fd50c199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:26.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.028+0000 7fd511800700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50c06b0f0 0x7fd50c199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.028+0000 7fd511800700 1 -- 172.21.15.6:0/3459357569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd50c108a30 msgr2=0x7fd50c19da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:26.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.028+0000 7fd511800700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd50c108a30 0x7fd50c19da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.028+0000 7fd511800700 1 -- 172.21.15.6:0/3459357569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd50c19e140 con 0x7fd50c06a750 2024-04-26T01:41:26.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.028+0000 7fd511800700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50c06a750 0x7fd50c199130 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd500000c00 tx=0x7fd5000057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:26.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.029+0000 7fd50a000700 1 -- 172.21.15.6:0/3459357569 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500027070 con 0x7fd50c06a750 2024-04-26T01:41:26.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.029+0000 7fd514200700 1 -- 172.21.15.6:0/3459357569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd50c19e3d0 con 0x7fd50c06a750 2024-04-26T01:41:26.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.029+0000 7fd514200700 1 -- 172.21.15.6:0/3459357569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd50c1a5cb0 con 0x7fd50c06a750 2024-04-26T01:41:26.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.029+0000 7fd50a000700 1 -- 172.21.15.6:0/3459357569 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd500022470 con 0x7fd50c06a750 2024-04-26T01:41:26.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.029+0000 7fd50a000700 1 -- 172.21.15.6:0/3459357569 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500022cc0 con 0x7fd50c06a750 2024-04-26T01:41:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.030+0000 7fd514200700 1 -- 172.21.15.6:0/3459357569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4ec005330 con 0x7fd50c06a750 2024-04-26T01:41:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.032+0000 7fd50a000700 1 -- 172.21.15.6:0/3459357569 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fd50002a070 con 0x7fd50c06a750 2024-04-26T01:41:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.034+0000 7fd50a000700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd4f40a3cc0 0x7fd4f40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.034+0000 7fd50a000700 1 -- 172.21.15.6:0/3459357569 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fd5000d0c40 con 0x7fd50c06a750 2024-04-26T01:41:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.034+0000 7fd510e00700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd4f40a3cc0 0x7fd4f40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:26.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.035+0000 7fd510e00700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd4f40a3cc0 0x7fd4f40a6180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd50c078f60 tx=0x7fd4f80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:26.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.038+0000 7fd50a000700 1 -- 172.21.15.6:0/3459357569 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd500013070 con 0x7fd50c06a750 2024-04-26T01:41:26.084 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:41:25 smithi188 podman[63587]: 2024-04-26 01:41:25.734951954 +0000 UTC m=+0.017352258 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:41:26.084 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:41:25 smithi188 podman[63587]: 2024-04-26 01:41:25.834419608 +0000 UTC m=+0.116819930 container create 146c931191428cac4535f6747a2e2f9d41c14c7eb00ebeab027ab3b83b2517da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-26T01:41:26.084 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:41:26 smithi188 podman[63587]: 2024-04-26 01:41:26.062138525 +0000 UTC m=+0.344538828 container init 146c931191428cac4535f6747a2e2f9d41c14c7eb00ebeab027ab3b83b2517da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:41:26.084 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:41:26 smithi188 podman[63587]: 2024-04-26 01:41:26.06501511 +0000 UTC m=+0.347415391 container start 146c931191428cac4535f6747a2e2f9d41c14c7eb00ebeab027ab3b83b2517da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:41:26.085 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:41:26 smithi188 ceph-mds[63606]: -- 172.21.15.188:0/574282669 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ecfccf3a40 con 0x55ecfccc7000 2024-04-26T01:41:26.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.322+0000 7fd514200700 1 -- 172.21.15.6:0/3459357569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4ec005620 con 0x7fd50c06a750 2024-04-26T01:41:26.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.323+0000 7fd50a000700 1 -- 172.21.15.6:0/3459357569 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4687 (secure 0 0 0) 0x7fd500022620 con 0x7fd50c06a750 2024-04-26T01:41:26.320 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:41:26.320 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-26T01:41:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.325+0000 7fd514200700 1 -- 172.21.15.6:0/3459357569 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd4f40a3cc0 msgr2=0x7fd4f40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.325+0000 7fd514200700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd4f40a3cc0 0x7fd4f40a6180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd50c078f60 tx=0x7fd4f80073d0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.325+0000 7fd514200700 1 -- 172.21.15.6:0/3459357569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50c06a750 msgr2=0x7fd50c199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.325+0000 7fd514200700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50c06a750 0x7fd50c199130 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd500000c00 tx=0x7fd5000057c0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.326+0000 7fd514200700 1 -- 172.21.15.6:0/3459357569 shutdown_connections 2024-04-26T01:41:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.326+0000 7fd514200700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd50c108a30 0x7fd50c19da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.326+0000 7fd514200700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50c06a750 0x7fd50c199130 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.326+0000 7fd514200700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd4f40a3cc0 0x7fd4f40a6180 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.326+0000 7fd514200700 1 --2- 172.21.15.6:0/3459357569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd50c06b0f0 0x7fd50c199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.326+0000 7fd514200700 1 -- 172.21.15.6:0/3459357569 >> 172.21.15.6:0/3459357569 conn(0x7fd50c0788f0 msgr2=0x7fd50c0ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.326+0000 7fd514200700 1 -- 172.21.15.6:0/3459357569 shutdown_connections 2024-04-26T01:41:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:26.326+0000 7fd514200700 1 -- 172.21.15.6:0/3459357569 wait complete. 2024-04-26T01:41:26.325 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 6 2024-04-26T01:41:26.389 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:41:26 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: starting mds.f at 2024-04-26T01:41:26.389 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:41:26 smithi188 bash[63587]: 146c931191428cac4535f6747a2e2f9d41c14c7eb00ebeab027ab3b83b2517da 2024-04-26T01:41:26.389 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:41:26 smithi188 systemd[1]: Started Ceph mds.f for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:41:26.661 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:26 smithi006 ceph-mon[28419]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:26.661 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:26.661 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:26.661 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:26.661 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:26.661 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:26.661 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:26 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:26.661 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:26 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3459357569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:26.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:26 smithi029 ceph-mon[32775]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:26.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:26 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:26 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3459357569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:26 smithi188 ceph-mon[32751]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:26 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:26 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3459357569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:27.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:27 smithi188 ceph-mon[32751]: Deploying daemon mds.c on smithi188 2024-04-26T01:41:27.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:27 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6834/1255666766,v1:172.21.15.188:6835/1255666766] up:boot 2024-04-26T01:41:27.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:27 smithi188 ceph-mon[32751]: fsmap 6 up:standby 2024-04-26T01:41:27.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:27 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:41:27.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:27 smithi029 ceph-mon[32775]: Deploying daemon mds.c on smithi188 2024-04-26T01:41:27.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:27 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6834/1255666766,v1:172.21.15.188:6835/1255666766] up:boot 2024-04-26T01:41:27.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:27 smithi029 ceph-mon[32775]: fsmap 6 up:standby 2024-04-26T01:41:27.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:27 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:41:27.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:27 smithi006 ceph-mon[28419]: Deploying daemon mds.c on smithi188 2024-04-26T01:41:27.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:27 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6834/1255666766,v1:172.21.15.188:6835/1255666766] up:boot 2024-04-26T01:41:27.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:27 smithi006 ceph-mon[28419]: fsmap 6 up:standby 2024-04-26T01:41:27.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:27 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:41:28.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:28 smithi188 ceph-mon[32751]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:28.639 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 01:41:28 smithi188 systemd[1]: Starting Ceph mds.c for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:41:28.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:28 smithi029 ceph-mon[32775]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:28.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:28 smithi006 ceph-mon[28419]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:28.937 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 01:41:28 smithi188 podman[63905]: 2024-04-26 01:41:28.581326525 +0000 UTC m=+0.017881271 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:41:28.938 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 01:41:28 smithi188 podman[63905]: 2024-04-26 01:41:28.705881049 +0000 UTC m=+0.142435783 container create 9a2bf011e344b905b07fd3daaff01bbafafcf168360d0ee3365410cc8dd22c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T01:41:28.938 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 01:41:28 smithi188 podman[63905]: 2024-04-26 01:41:28.941700884 +0000 UTC m=+0.378255614 container init 9a2bf011e344b905b07fd3daaff01bbafafcf168360d0ee3365410cc8dd22c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-26T01:41:29.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:41:29.389 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 01:41:28 smithi188 podman[63905]: 2024-04-26 01:41:28.94428153 +0000 UTC m=+0.380836260 container start 9a2bf011e344b905b07fd3daaff01bbafafcf168360d0ee3365410cc8dd22c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:41:29.389 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 01:41:28 smithi188 ceph-mds[63924]: -- 172.21.15.188:0/3664736226 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564da8ce1a40 con 0x564da8cb4800 2024-04-26T01:41:29.389 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 01:41:28 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-c[63920]: starting mds.c at 2024-04-26T01:41:29.389 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 01:41:29 smithi188 bash[63905]: 9a2bf011e344b905b07fd3daaff01bbafafcf168360d0ee3365410cc8dd22c0d 2024-04-26T01:41:29.389 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 01:41:29 smithi188 systemd[1]: Started Ceph mds.c for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:41:29.502 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:30.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:30.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:30.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:30.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:30.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:30.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:30.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:30 smithi006 ceph-mon[28419]: Deploying daemon mds.h on smithi029 2024-04-26T01:41:30.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:30 smithi006 ceph-mon[28419]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:30.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:30 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] up:boot 2024-04-26T01:41:30.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:30 smithi006 ceph-mon[28419]: fsmap 7 up:standby 2024-04-26T01:41:30.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:30 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:41:30.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:30.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:30.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:30.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:30.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:30.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:30.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:30 smithi188 ceph-mon[32751]: Deploying daemon mds.h on smithi029 2024-04-26T01:41:30.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:30 smithi188 ceph-mon[32751]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:30.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:30 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] up:boot 2024-04-26T01:41:30.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:30 smithi188 ceph-mon[32751]: fsmap 7 up:standby 2024-04-26T01:41:30.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:30 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:41:30.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:30.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:30.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:30.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:30.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:30.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:30.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:30 smithi029 ceph-mon[32775]: Deploying daemon mds.h on smithi029 2024-04-26T01:41:30.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:30 smithi029 ceph-mon[32775]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:30.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:30 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] up:boot 2024-04-26T01:41:30.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:30 smithi029 ceph-mon[32775]: fsmap 7 up:standby 2024-04-26T01:41:30.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:30 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:41:31.440 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 01:41:31 smithi029 systemd[1]: Starting Ceph mds.h for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:41:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.648+0000 7f6022e00700 1 -- 172.21.15.6:0/2527413932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c109fa0 msgr2=0x7f601c111b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.648+0000 7f6022e00700 1 --2- 172.21.15.6:0/2527413932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c109fa0 0x7f601c111b70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f601400b0d0 tx=0x7f601400b3e0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.649+0000 7f6022e00700 1 -- 172.21.15.6:0/2527413932 shutdown_connections 2024-04-26T01:41:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.649+0000 7f6022e00700 1 --2- 172.21.15.6:0/2527413932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c109fa0 0x7f601c111b70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.649+0000 7f6022e00700 1 --2- 172.21.15.6:0/2527413932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601c104ec0 0x7f601c1052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.649+0000 7f6022e00700 1 --2- 172.21.15.6:0/2527413932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f601c105870 0x7f601c1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.649+0000 7f6022e00700 1 -- 172.21.15.6:0/2527413932 >> 172.21.15.6:0/2527413932 conn(0x7f601c100750 msgr2=0x7f601c102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.649+0000 7f6022e00700 1 -- 172.21.15.6:0/2527413932 shutdown_connections 2024-04-26T01:41:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.649+0000 7f6022e00700 1 -- 172.21.15.6:0/2527413932 wait complete. 2024-04-26T01:41:31.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.650+0000 7f6022e00700 1 Processor -- start 2024-04-26T01:41:31.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.650+0000 7f6022e00700 1 -- start start 2024-04-26T01:41:31.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.651+0000 7f6022e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c104ec0 0x7f601c19b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:31.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.651+0000 7f6022e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601c105870 0x7f601c19b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:31.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.651+0000 7f6022e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f601c109fa0 0x7f601c19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:31.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.651+0000 7f6022e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f601c114e50 con 0x7f601c104ec0 2024-04-26T01:41:31.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.651+0000 7f6022e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f601c114cd0 con 0x7f601c105870 2024-04-26T01:41:31.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.651+0000 7f6022e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f601c114fd0 con 0x7f601c109fa0 2024-04-26T01:41:31.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.651+0000 7f601b400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601c105870 0x7f601c19b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:31.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.651+0000 7f601be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c104ec0 0x7f601c19b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:31.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.651+0000 7f601b400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601c105870 0x7f601c19b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43804/0 (socket says 172.21.15.6:43804) 2024-04-26T01:41:31.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.651+0000 7f601b400700 1 -- 172.21.15.6:0/3931186336 learned_addr learned my addr 172.21.15.6:0/3931186336 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:41:31.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.652+0000 7f6020e00700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f601c109fa0 0x7f601c19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:31.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.652+0000 7f601b400700 1 -- 172.21.15.6:0/3931186336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f601c109fa0 msgr2=0x7f601c19fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:31.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.652+0000 7f601b400700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f601c109fa0 0x7f601c19fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.652+0000 7f601b400700 1 -- 172.21.15.6:0/3931186336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c104ec0 msgr2=0x7f601c19b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:31.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.652+0000 7f601b400700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c104ec0 0x7f601c19b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.652+0000 7f601b400700 1 -- 172.21.15.6:0/3931186336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c1a02a0 con 0x7f601c105870 2024-04-26T01:41:31.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.652+0000 7f601b400700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601c105870 0x7f601c19b820 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f600800d950 tx=0x7f600800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:31.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.653+0000 7f6018c00700 1 -- 172.21.15.6:0/3931186336 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6008016070 con 0x7f601c105870 2024-04-26T01:41:31.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.653+0000 7f6018c00700 1 -- 172.21.15.6:0/3931186336 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6008004830 con 0x7f601c105870 2024-04-26T01:41:31.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.653+0000 7f6022e00700 1 -- 172.21.15.6:0/3931186336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601c1a0590 con 0x7f601c105870 2024-04-26T01:41:31.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.653+0000 7f6022e00700 1 -- 172.21.15.6:0/3931186336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601c1a7e10 con 0x7f601c105870 2024-04-26T01:41:31.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.653+0000 7f6018c00700 1 -- 172.21.15.6:0/3931186336 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6008005520 con 0x7f601c105870 2024-04-26T01:41:31.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.654+0000 7f6022e00700 1 -- 172.21.15.6:0/3931186336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ffc005330 con 0x7f601c105870 2024-04-26T01:41:31.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.656+0000 7f6018c00700 1 -- 172.21.15.6:0/3931186336 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f6008004220 con 0x7f601c105870 2024-04-26T01:41:31.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.657+0000 7f6018c00700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f60000a3d90 0x7f60000a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:31.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.658+0000 7f6018c00700 1 -- 172.21.15.6:0/3931186336 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f60080cfa30 con 0x7f601c105870 2024-04-26T01:41:31.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.658+0000 7f601be00700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f60000a3d90 0x7f60000a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:31.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.658+0000 7f601be00700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f60000a3d90 0x7f60000a6250 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f600c009c60 tx=0x7f600c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:31.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.662+0000 7f6018c00700 1 -- 172.21.15.6:0/3931186336 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f600801f020 con 0x7f601c105870 2024-04-26T01:41:31.796 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 01:41:31 smithi029 podman[63296]: 2024-04-26 01:41:31.441491072 +0000 UTC m=+0.018462934 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:41:31.796 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 01:41:31 smithi029 podman[63296]: 2024-04-26 01:41:31.55424095 +0000 UTC m=+0.131212811 container create cbe1eaab51c05ef85dea0cbbeefd99211211dfc8c7f78dfa6aa573480c13c3e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:41:31.796 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 01:41:31 smithi029 podman[63296]: 2024-04-26 01:41:31.800215068 +0000 UTC m=+0.377186932 container init cbe1eaab51c05ef85dea0cbbeefd99211211dfc8c7f78dfa6aa573480c13c3e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:41:31.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.951+0000 7f6022e00700 1 -- 172.21.15.6:0/3931186336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ffc005620 con 0x7f601c105870 2024-04-26T01:41:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.952+0000 7f6018c00700 1 -- 172.21.15.6:0/3931186336 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6340 (secure 0 0 0) 0x7f600809c810 con 0x7f601c105870 2024-04-26T01:41:31.949 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:41:31.949 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T01:41:31.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.955+0000 7f6022e00700 1 -- 172.21.15.6:0/3931186336 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f60000a3d90 msgr2=0x7f60000a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:31.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.955+0000 7f6022e00700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f60000a3d90 0x7f60000a6250 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f600c009c60 tx=0x7f600c009260 comp rx=0 tx=0).stop 2024-04-26T01:41:31.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.955+0000 7f6022e00700 1 -- 172.21.15.6:0/3931186336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601c105870 msgr2=0x7f601c19b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:31.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.955+0000 7f6022e00700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601c105870 0x7f601c19b820 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f600800d950 tx=0x7f600800dc60 comp rx=0 tx=0).stop 2024-04-26T01:41:31.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.955+0000 7f6022e00700 1 -- 172.21.15.6:0/3931186336 shutdown_connections 2024-04-26T01:41:31.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.955+0000 7f6022e00700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f60000a3d90 0x7f60000a6250 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.955+0000 7f6022e00700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f601c109fa0 0x7f601c19fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.955+0000 7f6022e00700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c104ec0 0x7f601c19b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.955+0000 7f6022e00700 1 --2- 172.21.15.6:0/3931186336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601c105870 0x7f601c19b820 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.955+0000 7f6022e00700 1 -- 172.21.15.6:0/3931186336 >> 172.21.15.6:0/3931186336 conn(0x7f601c100750 msgr2=0x7f601c102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:31.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.955+0000 7f6022e00700 1 -- 172.21.15.6:0/3931186336 shutdown_connections 2024-04-26T01:41:31.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:31.956+0000 7f6022e00700 1 -- 172.21.15.6:0/3931186336 wait complete. 2024-04-26T01:41:31.954 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-04-26T01:41:32.190 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 01:41:31 smithi029 podman[63296]: 2024-04-26 01:41:31.802977585 +0000 UTC m=+0.379949456 container start cbe1eaab51c05ef85dea0cbbeefd99211211dfc8c7f78dfa6aa573480c13c3e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:41:32.190 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 01:41:31 smithi029 ceph-mds[63315]: -- 172.21.15.29:0/3002449714 <== mon.2 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564c80d6da40 con 0x564c80d41000 2024-04-26T01:41:32.190 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 01:41:31 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-h[63311]: starting mds.h at 2024-04-26T01:41:32.190 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 01:41:31 smithi029 bash[63296]: cbe1eaab51c05ef85dea0cbbeefd99211211dfc8c7f78dfa6aa573480c13c3e6 2024-04-26T01:41:32.190 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 01:41:31 smithi029 systemd[1]: Started Ceph mds.h for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:41:32.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:32 smithi006 ceph-mon[28419]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:32.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:32 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3931186336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:32.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:32 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:32.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:32 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:32.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:32 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:32.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:32 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:32.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:32 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:32.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:32 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:32.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:32 smithi188 ceph-mon[32751]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:32.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:32 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3931186336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:32.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:32 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:32.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:32 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:32.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:32 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:32.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:32 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:32.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:32 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:32.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:32 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:32.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:32 smithi029 ceph-mon[32775]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:32.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:32 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3931186336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:32.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:32 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:32.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:32 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:32.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:32 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:32.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:32 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:32.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:32 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:32.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:32 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:33.553 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:33 smithi188 ceph-mon[32751]: Deploying daemon mds.i on smithi188 2024-04-26T01:41:33.553 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:33 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.29:6836/402171153,v1:172.21.15.29:6837/402171153] up:boot 2024-04-26T01:41:33.553 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:33 smithi188 ceph-mon[32751]: fsmap 8 up:standby 2024-04-26T01:41:33.553 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:33 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:41:33.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:33 smithi006 ceph-mon[28419]: Deploying daemon mds.i on smithi188 2024-04-26T01:41:33.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:33 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.29:6836/402171153,v1:172.21.15.29:6837/402171153] up:boot 2024-04-26T01:41:33.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:33 smithi006 ceph-mon[28419]: fsmap 8 up:standby 2024-04-26T01:41:33.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:33 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:41:33.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:33 smithi029 ceph-mon[32775]: Deploying daemon mds.i on smithi188 2024-04-26T01:41:33.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:33 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.29:6836/402171153,v1:172.21.15.29:6837/402171153] up:boot 2024-04-26T01:41:33.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:33 smithi029 ceph-mon[32775]: fsmap 8 up:standby 2024-04-26T01:41:33.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:33 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:41:34.139 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 01:41:33 smithi188 systemd[1]: Starting Ceph mds.i for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:41:34.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:34 smithi188 ceph-mon[32751]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:34.466 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 01:41:34 smithi188 podman[64212]: 2024-04-26 01:41:34.184502224 +0000 UTC m=+0.092033313 container create 70a79c79ae13c666d4753ba2f1fa776d60d941edb58a01589972b6496d113c44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T01:41:34.466 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 01:41:34 smithi188 podman[64212]: 2024-04-26 01:41:34.110072045 +0000 UTC m=+0.017603088 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:41:34.466 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 01:41:34 smithi188 podman[64212]: 2024-04-26 01:41:34.395098691 +0000 UTC m=+0.302629734 container init 70a79c79ae13c666d4753ba2f1fa776d60d941edb58a01589972b6496d113c44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:41:34.466 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 01:41:34 smithi188 podman[64212]: 2024-04-26 01:41:34.397648426 +0000 UTC m=+0.305179478 container start 70a79c79ae13c666d4753ba2f1fa776d60d941edb58a01589972b6496d113c44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-26T01:41:34.467 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 01:41:34 smithi188 ceph-mds[64231]: -- 172.21.15.188:0/1638449722 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5596fe18fa40 con 0x5596fe162c00 2024-04-26T01:41:34.467 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 01:41:34 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i[64227]: starting mds.i at 2024-04-26T01:41:34.467 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 01:41:34 smithi188 bash[64212]: 70a79c79ae13c666d4753ba2f1fa776d60d941edb58a01589972b6496d113c44 2024-04-26T01:41:34.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:34 smithi006 ceph-mon[28419]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:34.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:34 smithi029 ceph-mon[32775]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:34.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:41:34.888 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 01:41:34 smithi188 systemd[1]: Started Ceph mds.i for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:41:35.181 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:35.688 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:35.688 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:35.688 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:35.688 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:35.688 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:35.688 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:35.688 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:35 smithi006 ceph-mon[28419]: Deploying daemon mds.g on smithi006 2024-04-26T01:41:35.688 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:35 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] up:boot 2024-04-26T01:41:35.689 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:35 smithi006 ceph-mon[28419]: fsmap 9 up:standby 2024-04-26T01:41:35.689 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:35 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:41:35.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:35.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:35.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:35.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:35.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:35.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:35.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:35 smithi188 ceph-mon[32751]: Deploying daemon mds.g on smithi006 2024-04-26T01:41:35.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:35 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] up:boot 2024-04-26T01:41:35.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:35 smithi188 ceph-mon[32751]: fsmap 9 up:standby 2024-04-26T01:41:35.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:35 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:41:35.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:35.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:35.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:35.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:35.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:35.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:35.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:35 smithi029 ceph-mon[32775]: Deploying daemon mds.g on smithi006 2024-04-26T01:41:35.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:35 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] up:boot 2024-04-26T01:41:35.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:35 smithi029 ceph-mon[32775]: fsmap 9 up:standby 2024-04-26T01:41:35.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:35 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:41:36.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:36 smithi188 ceph-mon[32751]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:36.919 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:36 smithi006 ceph-mon[28419]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:36.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:36 smithi029 ceph-mon[32775]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.668+0000 7f7b03c00700 1 -- 172.21.15.6:0/1370140061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7afc074c70 msgr2=0x7f7afc075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.668+0000 7f7b03c00700 1 --2- 172.21.15.6:0/1370140061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7afc074c70 0x7f7afc075050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009990 tx=0x7f7af0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.669+0000 7f7b03c00700 1 -- 172.21.15.6:0/1370140061 shutdown_connections 2024-04-26T01:41:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.669+0000 7f7b03c00700 1 --2- 172.21.15.6:0/1370140061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7afc10f470 0x7f7afc111860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.669+0000 7f7b03c00700 1 --2- 172.21.15.6:0/1370140061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7afc074c70 0x7f7afc075050 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.669+0000 7f7b03c00700 1 --2- 172.21.15.6:0/1370140061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc075590 0x7f7afc10ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.669+0000 7f7b03c00700 1 -- 172.21.15.6:0/1370140061 >> 172.21.15.6:0/1370140061 conn(0x7f7afc079bf0 msgr2=0x7f7afc07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.669+0000 7f7b03c00700 1 -- 172.21.15.6:0/1370140061 shutdown_connections 2024-04-26T01:41:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.669+0000 7f7b03c00700 1 -- 172.21.15.6:0/1370140061 wait complete. 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.670+0000 7f7b03c00700 1 Processor -- start 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.670+0000 7f7b03c00700 1 -- start start 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.670+0000 7f7b03c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc074c70 0x7f7afc19b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.670+0000 7f7b03c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7afc075590 0x7f7afc19b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.670+0000 7f7b03c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7afc10f470 0x7f7afc19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.670+0000 7f7b03c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7afc114ec0 con 0x7f7afc10f470 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.670+0000 7f7b03c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7afc114d40 con 0x7f7afc074c70 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.670+0000 7f7b03c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7afc115040 con 0x7f7afc075590 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7b01c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7afc10f470 0x7f7afc19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7afbe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7afc075590 0x7f7afc19b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7afbe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7afc075590 0x7f7afc19b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:55160/0 (socket says 172.21.15.6:55160) 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7afbe00700 1 -- 172.21.15.6:0/1525992323 learned_addr learned my addr 172.21.15.6:0/1525992323 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7b01200700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc074c70 0x7f7afc19b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7b01c00700 1 -- 172.21.15.6:0/1525992323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7afc075590 msgr2=0x7f7afc19b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7b01c00700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7afc075590 0x7f7afc19b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7b01c00700 1 -- 172.21.15.6:0/1525992323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc074c70 msgr2=0x7f7afc19b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7b01c00700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc074c70 0x7f7afc19b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7b01c00700 1 -- 172.21.15.6:0/1525992323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7afc1a03e0 con 0x7f7afc10f470 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7afbe00700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7afc075590 0x7f7afc19b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7b01200700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc074c70 0x7f7afc19b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.671+0000 7f7b01c00700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7afc10f470 0x7f7afc19fca0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7af4007f10 tx=0x7f7af400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.672+0000 7f7af9600700 1 -- 172.21.15.6:0/1525992323 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af4004db0 con 0x7f7afc10f470 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.672+0000 7f7b03c00700 1 -- 172.21.15.6:0/1525992323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7afc1a06d0 con 0x7f7afc10f470 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.672+0000 7f7af9600700 1 -- 172.21.15.6:0/1525992323 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7af4003a00 con 0x7f7afc10f470 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.672+0000 7f7af9600700 1 -- 172.21.15.6:0/1525992323 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af4004580 con 0x7f7afc10f470 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.672+0000 7f7b03c00700 1 -- 172.21.15.6:0/1525992323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7afc1a7f50 con 0x7f7afc10f470 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.673+0000 7f7b03c00700 1 -- 172.21.15.6:0/1525992323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7afc069730 con 0x7f7afc10f470 2024-04-26T01:41:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.676+0000 7f7af9600700 1 -- 172.21.15.6:0/1525992323 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f7af401c020 con 0x7f7afc10f470 2024-04-26T01:41:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.679+0000 7f7af9600700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7ae40a3c40 0x7f7ae40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.679+0000 7f7af9600700 1 -- 172.21.15.6:0/1525992323 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f7af40cf6d0 con 0x7f7afc10f470 2024-04-26T01:41:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.679+0000 7f7b01200700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7ae40a3c40 0x7f7ae40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.680+0000 7f7b01200700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7ae40a3c40 0x7f7ae40a6100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7af000b2e0 tx=0x7f7af001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.685+0000 7f7af9600700 1 -- 172.21.15.6:0/1525992323 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7af4010070 con 0x7f7afc10f470 2024-04-26T01:41:37.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.973+0000 7f7b03c00700 1 -- 172.21.15.6:0/1525992323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7afc02df80 con 0x7f7afc10f470 2024-04-26T01:41:37.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.974+0000 7f7af9600700 1 -- 172.21.15.6:0/1525992323 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7989 (secure 0 0 0) 0x7f7af409c4b0 con 0x7f7afc10f470 2024-04-26T01:41:37.972 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:41:37.972 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T01:41:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.978+0000 7f7b03c00700 1 -- 172.21.15.6:0/1525992323 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7ae40a3c40 msgr2=0x7f7ae40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.978+0000 7f7b03c00700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7ae40a3c40 0x7f7ae40a6100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7af000b2e0 tx=0x7f7af001a040 comp rx=0 tx=0).stop 2024-04-26T01:41:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.978+0000 7f7b03c00700 1 -- 172.21.15.6:0/1525992323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7afc10f470 msgr2=0x7f7afc19fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.978+0000 7f7b03c00700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7afc10f470 0x7f7afc19fca0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7af4007f10 tx=0x7f7af400d3b0 comp rx=0 tx=0).stop 2024-04-26T01:41:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.978+0000 7f7b03c00700 1 -- 172.21.15.6:0/1525992323 shutdown_connections 2024-04-26T01:41:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.978+0000 7f7b03c00700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7afc10f470 0x7f7afc19fca0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.978+0000 7f7b03c00700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc074c70 0x7f7afc19b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.978+0000 7f7b03c00700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7ae40a3c40 0x7f7ae40a6100 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.978+0000 7f7b03c00700 1 --2- 172.21.15.6:0/1525992323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7afc075590 0x7f7afc19b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.978+0000 7f7b03c00700 1 -- 172.21.15.6:0/1525992323 >> 172.21.15.6:0/1525992323 conn(0x7f7afc079bf0 msgr2=0x7f7afc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.978+0000 7f7b03c00700 1 -- 172.21.15.6:0/1525992323 shutdown_connections 2024-04-26T01:41:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:37.978+0000 7f7b03c00700 1 -- 172.21.15.6:0/1525992323 wait complete. 2024-04-26T01:41:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 10 2024-04-26T01:41:38.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:38 smithi006 ceph-mon[28419]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:38.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:38 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1525992323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:38.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:38 smithi188 ceph-mon[32751]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:38.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:38 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1525992323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:38.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:38 smithi029 ceph-mon[32775]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:38.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:38 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1525992323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:40.066 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 01:41:39 smithi006 systemd[1]: Starting Ceph mds.g for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:41:40.412 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 01:41:40 smithi006 podman[70421]: 2024-04-26 01:41:40.010503961 +0000 UTC m=+0.036683355 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:41:40.412 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 01:41:40 smithi006 podman[70421]: 2024-04-26 01:41:40.111270484 +0000 UTC m=+0.137449932 container create c3d95a031c7f9d83e5b3280455c843a94cdcbf26e2798a3c1c43cc336f251de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T01:41:40.413 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 01:41:40 smithi006 podman[70421]: 2024-04-26 01:41:40.415153682 +0000 UTC m=+0.441333133 container init c3d95a031c7f9d83e5b3280455c843a94cdcbf26e2798a3c1c43cc336f251de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:41:40.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:40 smithi006 ceph-mon[28419]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:40.816 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 01:41:40 smithi006 podman[70421]: 2024-04-26 01:41:40.42593498 +0000 UTC m=+0.452114380 container start c3d95a031c7f9d83e5b3280455c843a94cdcbf26e2798a3c1c43cc336f251de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-26T01:41:40.816 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 01:41:40 smithi006 ceph-mds[70440]: -- 172.21.15.6:0/2197006561 <== mon.2 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562e9d53ba40 con 0x562e9d50e800 2024-04-26T01:41:40.816 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 01:41:40 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g[70436]: starting mds.g at 2024-04-26T01:41:40.816 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 01:41:40 smithi006 bash[70421]: c3d95a031c7f9d83e5b3280455c843a94cdcbf26e2798a3c1c43cc336f251de8 2024-04-26T01:41:40.816 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 01:41:40 smithi006 systemd[1]: Started Ceph mds.g for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:41:40.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:40 smithi188 ceph-mon[32751]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:40.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:40 smithi029 ceph-mon[32775]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:41.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:41:41.657 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:41.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:41 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.6:6838/3936238112,v1:172.21.15.6:6839/3936238112] up:boot 2024-04-26T01:41:41.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:41 smithi188 ceph-mon[32751]: fsmap 10 up:standby 2024-04-26T01:41:41.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:41:41.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:41.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:41.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:41.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:41.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:41.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:41 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:41.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:41 smithi188 ceph-mon[32751]: Deploying daemon mds.a on smithi006 2024-04-26T01:41:41.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:41 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.6:6838/3936238112,v1:172.21.15.6:6839/3936238112] up:boot 2024-04-26T01:41:41.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:41 smithi029 ceph-mon[32775]: fsmap 10 up:standby 2024-04-26T01:41:41.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:41 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:41:41.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:41 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:41.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:41 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:41.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:41 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:41.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:41 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:41.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:41 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:41.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:41 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:41.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:41 smithi029 ceph-mon[32775]: Deploying daemon mds.a on smithi006 2024-04-26T01:41:41.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:41 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.6:6838/3936238112,v1:172.21.15.6:6839/3936238112] up:boot 2024-04-26T01:41:41.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:41 smithi006 ceph-mon[28419]: fsmap 10 up:standby 2024-04-26T01:41:41.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:41 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:41:41.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:41 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:41.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:41 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:41.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:41 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:41.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:41 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:41.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:41 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:41.948 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:41 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:41.948 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:41 smithi006 ceph-mon[28419]: Deploying daemon mds.a on smithi006 2024-04-26T01:41:43.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:42 smithi006 ceph-mon[28419]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:43.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:42 smithi188 ceph-mon[32751]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:43.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:42 smithi029 ceph-mon[32775]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:44.779 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:44 smithi006 ceph-mon[28419]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:44.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.832+0000 7f6f0c600700 1 -- 172.21.15.6:0/1905699603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f04103450 msgr2=0x7f6f0410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:44.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.832+0000 7f6f0c600700 1 --2- 172.21.15.6:0/1905699603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f04103450 0x7f6f0410f8b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6efc009990 tx=0x7f6efc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.832+0000 7f6f0c600700 1 -- 172.21.15.6:0/1905699603 shutdown_connections 2024-04-26T01:41:44.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.832+0000 7f6f0c600700 1 --2- 172.21.15.6:0/1905699603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f04103450 0x7f6f0410f8b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.832+0000 7f6f0c600700 1 --2- 172.21.15.6:0/1905699603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f04108a50 0x7f6f04108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.832+0000 7f6f0c600700 1 --2- 172.21.15.6:0/1905699603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f04102a90 0x7f6f04102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.832+0000 7f6f0c600700 1 -- 172.21.15.6:0/1905699603 >> 172.21.15.6:0/1905699603 conn(0x7f6f04079ba0 msgr2=0x7f6f04079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:44.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.832+0000 7f6f0c600700 1 -- 172.21.15.6:0/1905699603 shutdown_connections 2024-04-26T01:41:44.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.832+0000 7f6f0c600700 1 -- 172.21.15.6:0/1905699603 wait complete. 2024-04-26T01:41:44.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.834+0000 7f6f0c600700 1 Processor -- start 2024-04-26T01:41:44.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.834+0000 7f6f0c600700 1 -- start start 2024-04-26T01:41:44.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.834+0000 7f6f0c600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f04102a90 0x7f6f04199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.834+0000 7f6f0c600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f04103450 0x7f6f04199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.834+0000 7f6f0c600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f04108a50 0x7f6f0419d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.834+0000 7f6f0c600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04112c50 con 0x7f6f04108a50 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.834+0000 7f6f0c600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04112ad0 con 0x7f6f04103450 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.835+0000 7f6f0c600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04112dd0 con 0x7f6f04102a90 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.835+0000 7f6f0a600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f04108a50 0x7f6f0419d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.835+0000 7f6f09c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f04102a90 0x7f6f04199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.835+0000 7f6f0a600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f04108a50 0x7f6f0419d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43432/0 (socket says 172.21.15.6:43432) 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.835+0000 7f6f0a600700 1 -- 172.21.15.6:0/2620152028 learned_addr learned my addr 172.21.15.6:0/2620152028 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.835+0000 7f6f09200700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f04103450 0x7f6f04199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.835+0000 7f6f09c00700 1 -- 172.21.15.6:0/2620152028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f04103450 msgr2=0x7f6f04199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.835+0000 7f6f09c00700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f04103450 0x7f6f04199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.835+0000 7f6f09c00700 1 -- 172.21.15.6:0/2620152028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f04108a50 msgr2=0x7f6f0419d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.835+0000 7f6f09c00700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f04108a50 0x7f6f0419d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.835+0000 7f6f09c00700 1 -- 172.21.15.6:0/2620152028 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0419e100 con 0x7f6f04102a90 2024-04-26T01:41:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.836+0000 7f6f09c00700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f04102a90 0x7f6f04199140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800efc0 tx=0x7f6ef800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.836+0000 7f6f02000700 1 -- 172.21.15.6:0/2620152028 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef80092e0 con 0x7f6f04102a90 2024-04-26T01:41:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.836+0000 7f6f02000700 1 -- 172.21.15.6:0/2620152028 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ef8004bb0 con 0x7f6f04102a90 2024-04-26T01:41:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.836+0000 7f6f0c600700 1 -- 172.21.15.6:0/2620152028 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0419e390 con 0x7f6f04102a90 2024-04-26T01:41:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.836+0000 7f6f02000700 1 -- 172.21.15.6:0/2620152028 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8004040 con 0x7f6f04102a90 2024-04-26T01:41:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.836+0000 7f6f0c600700 1 -- 172.21.15.6:0/2620152028 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f041a5c70 con 0x7f6f04102a90 2024-04-26T01:41:44.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.837+0000 7f6f0c600700 1 -- 172.21.15.6:0/2620152028 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f04069730 con 0x7f6f04102a90 2024-04-26T01:41:44.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.839+0000 7f6f02000700 1 -- 172.21.15.6:0/2620152028 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f6ef801b030 con 0x7f6f04102a90 2024-04-26T01:41:44.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.841+0000 7f6f02000700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6eec0a3cc0 0x7f6eec0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:44.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.841+0000 7f6f02000700 1 -- 172.21.15.6:0/2620152028 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f6ef80cfa60 con 0x7f6f04102a90 2024-04-26T01:41:44.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.841+0000 7f6f09200700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6eec0a3cc0 0x7f6eec0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:44.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.842+0000 7f6f09200700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6eec0a3cc0 0x7f6eec0a6180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6f0407a1a0 tx=0x7f6ef0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:44.845+0000 7f6f02000700 1 -- 172.21.15.6:0/2620152028 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ef8022020 con 0x7f6f04102a90 2024-04-26T01:41:45.066 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 01:41:44 smithi006 systemd[1]: Starting Ceph mds.a for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:41:45.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.140+0000 7f6f0c600700 1 -- 172.21.15.6:0/2620152028 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f0402df80 con 0x7f6f04102a90 2024-04-26T01:41:45.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.141+0000 7f6f02000700 1 -- 172.21.15.6:0/2620152028 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8812 (secure 0 0 0) 0x7f6ef8010040 con 0x7f6f04102a90 2024-04-26T01:41:45.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:44 smithi188 ceph-mon[32751]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:45.139 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:41:45.139 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24494,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T01:41:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.144+0000 7f6f0c600700 1 -- 172.21.15.6:0/2620152028 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6eec0a3cc0 msgr2=0x7f6eec0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.144+0000 7f6f0c600700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6eec0a3cc0 0x7f6eec0a6180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6f0407a1a0 tx=0x7f6ef0009260 comp rx=0 tx=0).stop 2024-04-26T01:41:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.144+0000 7f6f0c600700 1 -- 172.21.15.6:0/2620152028 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f04102a90 msgr2=0x7f6f04199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.144+0000 7f6f0c600700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f04102a90 0x7f6f04199140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800efc0 tx=0x7f6ef800c3a0 comp rx=0 tx=0).stop 2024-04-26T01:41:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.145+0000 7f6f0c600700 1 -- 172.21.15.6:0/2620152028 shutdown_connections 2024-04-26T01:41:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.145+0000 7f6f0c600700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f04108a50 0x7f6f0419d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.145+0000 7f6f0c600700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6eec0a3cc0 0x7f6eec0a6180 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.145+0000 7f6f0c600700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f04102a90 0x7f6f04199140 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.145+0000 7f6f0c600700 1 --2- 172.21.15.6:0/2620152028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f04103450 0x7f6f04199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.145+0000 7f6f0c600700 1 -- 172.21.15.6:0/2620152028 >> 172.21.15.6:0/2620152028 conn(0x7f6f04079ba0 msgr2=0x7f6f041017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.145+0000 7f6f0c600700 1 -- 172.21.15.6:0/2620152028 shutdown_connections 2024-04-26T01:41:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:45.145+0000 7f6f0c600700 1 -- 172.21.15.6:0/2620152028 wait complete. 2024-04-26T01:41:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 11 2024-04-26T01:41:45.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:44 smithi029 ceph-mon[32775]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:45.405 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 01:41:45 smithi006 podman[70866]: 2024-04-26 01:41:45.131797094 +0000 UTC m=+0.033471363 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:41:45.405 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 01:41:45 smithi006 podman[70866]: 2024-04-26 01:41:45.250738533 +0000 UTC m=+0.152412835 container create 359947f8e3430bbe05223c419c29c8370d84876503fdb688c71dc4051997879d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T01:41:45.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:45 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2620152028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:45.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:45 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2620152028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:45.968 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 01:41:45 smithi006 podman[70866]: 2024-04-26 01:41:45.674682717 +0000 UTC m=+0.576357029 container init 359947f8e3430bbe05223c419c29c8370d84876503fdb688c71dc4051997879d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-26T01:41:45.968 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 01:41:45 smithi006 podman[70866]: 2024-04-26 01:41:45.685523019 +0000 UTC m=+0.587197327 container start 359947f8e3430bbe05223c419c29c8370d84876503fdb688c71dc4051997879d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0) 2024-04-26T01:41:45.968 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 01:41:45 smithi006 ceph-mds[70897]: -- 172.21.15.6:0/202210453 <== mon.0 v2:172.21.15.6:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5597b180fa40 con 0x5597b17e3000 2024-04-26T01:41:45.968 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 01:41:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-a[70893]: starting mds.a at 2024-04-26T01:41:45.968 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 01:41:45 smithi006 bash[70866]: 359947f8e3430bbe05223c419c29c8370d84876503fdb688c71dc4051997879d 2024-04-26T01:41:45.968 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 01:41:45 smithi006 systemd[1]: Started Ceph mds.a for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:41:46.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:45 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2620152028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:46 smithi029 ceph-mon[32775]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:46 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.6:6840/1062358442,v1:172.21.15.6:6841/1062358442] up:boot 2024-04-26T01:41:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:46 smithi029 ceph-mon[32775]: fsmap 11 up:standby 2024-04-26T01:41:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:46 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:41:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:46 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:46 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:46 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:46 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:46 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:46 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:47.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:46 smithi006 ceph-mon[28419]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:47.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:46 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.6:6840/1062358442,v1:172.21.15.6:6841/1062358442] up:boot 2024-04-26T01:41:47.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:46 smithi006 ceph-mon[28419]: fsmap 11 up:standby 2024-04-26T01:41:47.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:41:47.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:47.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:47.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:47.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:47.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:47.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:46 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:46 smithi188 ceph-mon[32751]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:46 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.6:6840/1062358442,v1:172.21.15.6:6841/1062358442] up:boot 2024-04-26T01:41:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:46 smithi188 ceph-mon[32751]: fsmap 11 up:standby 2024-04-26T01:41:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:41:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:41:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:41:47.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:46 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:47.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:47 smithi029 ceph-mon[32775]: Deploying daemon mds.b on smithi029 2024-04-26T01:41:48.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:47 smithi006 ceph-mon[28419]: Deploying daemon mds.b on smithi029 2024-04-26T01:41:48.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:47 smithi188 ceph-mon[32751]: Deploying daemon mds.b on smithi029 2024-04-26T01:41:48.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:41:48.690 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 01:41:48 smithi029 podman[63613]: 2024-04-26 01:41:48.511223578 +0000 UTC m=+0.018412148 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:41:48.899 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:49.050 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:48 smithi006 ceph-mon[28419]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:49.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:48 smithi188 ceph-mon[32751]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:49.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:48 smithi029 ceph-mon[32775]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:49.190 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 01:41:48 smithi029 podman[63613]: 2024-04-26 01:41:48.861622372 +0000 UTC m=+0.368810944 container init e0537abc3b1fbdfab894b944c36e2765e4c233e89c6268b6fb6347833c736b0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T01:41:49.190 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 01:41:48 smithi029 podman[63613]: 2024-04-26 01:41:48.864542137 +0000 UTC m=+0.371730715 container start e0537abc3b1fbdfab894b944c36e2765e4c233e89c6268b6fb6347833c736b0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-26T01:41:49.190 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 01:41:48 smithi029 ceph-mds[63632]: -- 172.21.15.29:0/3447539741 <== mon.2 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5618d69f1a40 con 0x5618d69c4800 2024-04-26T01:41:49.190 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 01:41:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-b[63628]: starting mds.b at 2024-04-26T01:41:49.190 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 01:41:48 smithi029 bash[63613]: e0537abc3b1fbdfab894b944c36e2765e4c233e89c6268b6fb6347833c736b0b 2024-04-26T01:41:49.190 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 01:41:48 smithi029 systemd[1]: Started Ceph mds.b for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:41:50.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:50 smithi006 ceph-mon[28419]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:50.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:50.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:50 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] up:boot 2024-04-26T01:41:50.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:50 smithi006 ceph-mon[28419]: fsmap 12 up:standby 2024-04-26T01:41:50.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:50 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:41:50.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:50 smithi188 ceph-mon[32751]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:50.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:50.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:50 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] up:boot 2024-04-26T01:41:50.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:50 smithi188 ceph-mon[32751]: fsmap 12 up:standby 2024-04-26T01:41:50.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:50 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:41:50.356 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.356 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.357 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.357 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.357 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:50.357 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:50 smithi029 ceph-mon[32775]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:50.357 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:50.357 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:50 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] up:boot 2024-04-26T01:41:50.357 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:50 smithi029 ceph-mon[32775]: fsmap 12 up:standby 2024-04-26T01:41:50.357 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:50 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:41:51.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.070+0000 7fd5a2600700 1 -- 172.21.15.6:0/172318206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c103450 msgr2=0x7fd59c10f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.070+0000 7fd5a2600700 1 --2- 172.21.15.6:0/172318206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c103450 0x7fd59c10f8e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd59400b0a0 tx=0x7fd59400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:41:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.070+0000 7fd5a2600700 1 -- 172.21.15.6:0/172318206 shutdown_connections 2024-04-26T01:41:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.070+0000 7fd5a2600700 1 --2- 172.21.15.6:0/172318206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c103450 0x7fd59c10f8e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.070+0000 7fd5a2600700 1 --2- 172.21.15.6:0/172318206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd59c108a80 0x7fd59c108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.070+0000 7fd5a2600700 1 --2- 172.21.15.6:0/172318206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd59c102a90 0x7fd59c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.070+0000 7fd5a2600700 1 -- 172.21.15.6:0/172318206 >> 172.21.15.6:0/172318206 conn(0x7fd59c079bf0 msgr2=0x7fd59c07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.070+0000 7fd5a2600700 1 -- 172.21.15.6:0/172318206 shutdown_connections 2024-04-26T01:41:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.070+0000 7fd5a2600700 1 -- 172.21.15.6:0/172318206 wait complete. 2024-04-26T01:41:51.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.072+0000 7fd5a2600700 1 Processor -- start 2024-04-26T01:41:51.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.072+0000 7fd5a2600700 1 -- start start 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.072+0000 7fd5a2600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd59c102a90 0x7fd59c19b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd5a2600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c103450 0x7fd59c19b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd5a2600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd59c108a80 0x7fd59c19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd5a2600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c112c80 con 0x7fd59c108a80 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd5a2600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c112b00 con 0x7fd59c103450 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd5a2600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c112e00 con 0x7fd59c102a90 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd59b400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd59c102a90 0x7fd59c19b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd59aa00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c103450 0x7fd59c19b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd59b400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd59c102a90 0x7fd59c19b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:39024/0 (socket says 172.21.15.6:39024) 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd59b400700 1 -- 172.21.15.6:0/1689368472 learned_addr learned my addr 172.21.15.6:0/1689368472 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd59be00700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd59c108a80 0x7fd59c19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd59b400700 1 -- 172.21.15.6:0/1689368472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c103450 msgr2=0x7fd59c19b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd59b400700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c103450 0x7fd59c19b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:51.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd59b400700 1 -- 172.21.15.6:0/1689368472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd59c108a80 msgr2=0x7fd59c19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:51.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd59b400700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd59c108a80 0x7fd59c19fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:51.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd59b400700 1 -- 172.21.15.6:0/1689368472 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd59c1a0340 con 0x7fd59c102a90 2024-04-26T01:41:51.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.073+0000 7fd59be00700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd59c108a80 0x7fd59c19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:51.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.074+0000 7fd59b400700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd59c102a90 0x7fd59c19b330 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00e9c0 tx=0x7fd58c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:51.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.075+0000 7fd593e00700 1 -- 172.21.15.6:0/1689368472 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58c00cbd0 con 0x7fd59c102a90 2024-04-26T01:41:51.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.075+0000 7fd593e00700 1 -- 172.21.15.6:0/1689368472 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd58c00cd70 con 0x7fd59c102a90 2024-04-26T01:41:51.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.075+0000 7fd5a2600700 1 -- 172.21.15.6:0/1689368472 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd59c1a0630 con 0x7fd59c102a90 2024-04-26T01:41:51.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.075+0000 7fd593e00700 1 -- 172.21.15.6:0/1689368472 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58c005610 con 0x7fd59c102a90 2024-04-26T01:41:51.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.075+0000 7fd5a2600700 1 -- 172.21.15.6:0/1689368472 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd59c1a7eb0 con 0x7fd59c102a90 2024-04-26T01:41:51.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.077+0000 7fd5a2600700 1 -- 172.21.15.6:0/1689368472 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd59c069730 con 0x7fd59c102a90 2024-04-26T01:41:51.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.078+0000 7fd593e00700 1 -- 172.21.15.6:0/1689368472 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fd58c004830 con 0x7fd59c102a90 2024-04-26T01:41:51.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.080+0000 7fd593e00700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd5800a3cc0 0x7fd5800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:51.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.080+0000 7fd593e00700 1 -- 172.21.15.6:0/1689368472 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fd58c0d0230 con 0x7fd59c102a90 2024-04-26T01:41:51.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.080+0000 7fd59aa00700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd5800a3cc0 0x7fd5800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:51.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.081+0000 7fd59aa00700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd5800a3cc0 0x7fd5800a6180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd59c07a4f0 tx=0x7fd588009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:51.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.084+0000 7fd593e00700 1 -- 172.21.15.6:0/1689368472 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd58c09d010 con 0x7fd59c102a90 2024-04-26T01:41:51.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.382+0000 7fd5a2600700 1 -- 172.21.15.6:0/1689368472 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd59c1a8080 con 0x7fd59c102a90 2024-04-26T01:41:51.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.383+0000 7fd593e00700 1 -- 172.21.15.6:0/1689368472 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10461 (secure 0 0 0) 0x7fd58c09c910 con 0x7fd59c102a90 2024-04-26T01:41:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:41:51.381 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24494,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T01:41:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.386+0000 7fd5a2600700 1 -- 172.21.15.6:0/1689368472 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd5800a3cc0 msgr2=0x7fd5800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.386+0000 7fd5a2600700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd5800a3cc0 0x7fd5800a6180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd59c07a4f0 tx=0x7fd588009260 comp rx=0 tx=0).stop 2024-04-26T01:41:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.386+0000 7fd5a2600700 1 -- 172.21.15.6:0/1689368472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd59c102a90 msgr2=0x7fd59c19b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.386+0000 7fd5a2600700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd59c102a90 0x7fd59c19b330 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00e9c0 tx=0x7fd58c00ecd0 comp rx=0 tx=0).stop 2024-04-26T01:41:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.386+0000 7fd5a2600700 1 -- 172.21.15.6:0/1689368472 shutdown_connections 2024-04-26T01:41:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.386+0000 7fd5a2600700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd59c108a80 0x7fd59c19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.386+0000 7fd5a2600700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd5800a3cc0 0x7fd5800a6180 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.386+0000 7fd5a2600700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd59c102a90 0x7fd59c19b330 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.386+0000 7fd5a2600700 1 --2- 172.21.15.6:0/1689368472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c103450 0x7fd59c19b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.386+0000 7fd5a2600700 1 -- 172.21.15.6:0/1689368472 >> 172.21.15.6:0/1689368472 conn(0x7fd59c079bf0 msgr2=0x7fd59c10dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.386+0000 7fd5a2600700 1 -- 172.21.15.6:0/1689368472 shutdown_connections 2024-04-26T01:41:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:51.386+0000 7fd5a2600700 1 -- 172.21.15.6:0/1689368472 wait complete. 2024-04-26T01:41:51.385 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 13 2024-04-26T01:41:52.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:52 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:52.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:52 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:52.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:52 smithi188 ceph-mon[32751]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:52.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:52 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:52.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:52 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:52.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:52 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1689368472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:52.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:52 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:52.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:52 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:52.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:52 smithi029 ceph-mon[32775]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:52.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:52 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:52.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:52 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:52.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:52 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1689368472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:52.515 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:52 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:52.515 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:52 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:52.516 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:52 smithi006 ceph-mon[28419]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:52.516 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:52 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:52.516 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:52 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:52.516 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:52 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1689368472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:52.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-04-26T01:41:53.057 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:53.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:53 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:53.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:53 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:53.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:53 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:54.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:54 smithi029 ceph-mon[32775]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:54.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:54 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:54.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:54 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:54.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:54 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:54.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:54 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:41:54.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:54 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:54.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:54 smithi188 ceph-mon[32751]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:54.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:54 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:54.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:54 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:54.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:54 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:54.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:54 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:41:54.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:54 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:54.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:54 smithi006 ceph-mon[28419]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:54.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:54 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:54.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:54 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:54.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:54 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:41:54.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:54 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:41:54.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:54 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.300+0000 7fa511400700 1 -- 172.21.15.6:0/2872817413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa50c075590 msgr2=0x7fa50c10ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.300+0000 7fa511400700 1 --2- 172.21.15.6:0/2872817413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa50c075590 0x7fa50c10ef00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009990 tx=0x7fa4f8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.300+0000 7fa511400700 1 -- 172.21.15.6:0/2872817413 shutdown_connections 2024-04-26T01:41:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.300+0000 7fa511400700 1 --2- 172.21.15.6:0/2872817413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c10f440 0x7fa50c111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.300+0000 7fa511400700 1 --2- 172.21.15.6:0/2872817413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa50c074c70 0x7fa50c075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.300+0000 7fa511400700 1 --2- 172.21.15.6:0/2872817413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa50c075590 0x7fa50c10ef00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.300+0000 7fa511400700 1 -- 172.21.15.6:0/2872817413 >> 172.21.15.6:0/2872817413 conn(0x7fa50c079ba0 msgr2=0x7fa50c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.300+0000 7fa511400700 1 -- 172.21.15.6:0/2872817413 shutdown_connections 2024-04-26T01:41:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.300+0000 7fa511400700 1 -- 172.21.15.6:0/2872817413 wait complete. 2024-04-26T01:41:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.301+0000 7fa511400700 1 Processor -- start 2024-04-26T01:41:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.301+0000 7fa511400700 1 -- start start 2024-04-26T01:41:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.302+0000 7fa511400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c074c70 0x7fa50c19b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.302+0000 7fa511400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa50c075590 0x7fa50c19b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.302+0000 7fa511400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa50c10f440 0x7fa50c19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.302+0000 7fa511400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c114e50 con 0x7fa50c075590 2024-04-26T01:41:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.302+0000 7fa511400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c114cd0 con 0x7fa50c10f440 2024-04-26T01:41:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.302+0000 7fa511400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c114fd0 con 0x7fa50c074c70 2024-04-26T01:41:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.302+0000 7fa50a800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c074c70 0x7fa50c19b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.302+0000 7fa50a800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c074c70 0x7fa50c19b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:39042/0 (socket says 172.21.15.6:39042) 2024-04-26T01:41:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.302+0000 7fa50a800700 1 -- 172.21.15.6:0/861814432 learned_addr learned my addr 172.21.15.6:0/861814432 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:41:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.302+0000 7fa509e00700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa50c075590 0x7fa50c19b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.302+0000 7fa50b200700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa50c10f440 0x7fa50c19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.303+0000 7fa50a800700 1 -- 172.21.15.6:0/861814432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa50c10f440 msgr2=0x7fa50c19fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.303+0000 7fa50a800700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa50c10f440 0x7fa50c19fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.303+0000 7fa50a800700 1 -- 172.21.15.6:0/861814432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa50c075590 msgr2=0x7fa50c19b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.303+0000 7fa50a800700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa50c075590 0x7fa50c19b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.303+0000 7fa50a800700 1 -- 172.21.15.6:0/861814432 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50c1a0370 con 0x7fa50c074c70 2024-04-26T01:41:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.303+0000 7fa509e00700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa50c075590 0x7fa50c19b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.303+0000 7fa50a800700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c074c70 0x7fa50c19b360 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00e9a0 tx=0x7fa4fc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.303+0000 7fa503400700 1 -- 172.21.15.6:0/861814432 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc00cbd0 con 0x7fa50c074c70 2024-04-26T01:41:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.304+0000 7fa503400700 1 -- 172.21.15.6:0/861814432 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4fc00cd70 con 0x7fa50c074c70 2024-04-26T01:41:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.304+0000 7fa503400700 1 -- 172.21.15.6:0/861814432 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc018680 con 0x7fa50c074c70 2024-04-26T01:41:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.304+0000 7fa511400700 1 -- 172.21.15.6:0/861814432 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50c1a0660 con 0x7fa50c074c70 2024-04-26T01:41:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.304+0000 7fa511400700 1 -- 172.21.15.6:0/861814432 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa50c1a7ee0 con 0x7fa50c074c70 2024-04-26T01:41:55.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.305+0000 7fa511400700 1 -- 172.21.15.6:0/861814432 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa50c069730 con 0x7fa50c074c70 2024-04-26T01:41:55.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.308+0000 7fa503400700 1 -- 172.21.15.6:0/861814432 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fa4fc010430 con 0x7fa50c074c70 2024-04-26T01:41:55.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.310+0000 7fa503400700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa4f00a3cc0 0x7fa4f00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:55.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.310+0000 7fa503400700 1 -- 172.21.15.6:0/861814432 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fa4fc0d01a0 con 0x7fa50c074c70 2024-04-26T01:41:55.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.310+0000 7fa509e00700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa4f00a3cc0 0x7fa4f00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:55.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.311+0000 7fa509e00700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa4f00a3cc0 0x7fa4f00a6180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa50c1110b0 tx=0x7fa4f800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:55.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.314+0000 7fa503400700 1 -- 172.21.15.6:0/861814432 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa4fc09cf80 con 0x7fa50c074c70 2024-04-26T01:41:55.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:55 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:55.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:55 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:55.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:55 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:55.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.568+0000 7fa511400700 1 -- 172.21.15.6:0/861814432 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa50c02df80 con 0x7fa50c074c70 2024-04-26T01:41:55.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.569+0000 7fa503400700 1 -- 172.21.15.6:0/861814432 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v84) ==== 84+0+65 (secure 0 0 0) 0x7fa4fc09c880 con 0x7fa50c074c70 2024-04-26T01:41:55.567 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:41:55.567 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-04-26T01:41:55.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T01:41:55.567 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-04-26T01:41:55.567 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-04-26T01:41:55.567 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:41:55.567 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-04-26T01:41:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.571+0000 7fa511400700 1 -- 172.21.15.6:0/861814432 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa4f00a3cc0 msgr2=0x7fa4f00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.571+0000 7fa511400700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa4f00a3cc0 0x7fa4f00a6180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa50c1110b0 tx=0x7fa4f800b260 comp rx=0 tx=0).stop 2024-04-26T01:41:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.571+0000 7fa511400700 1 -- 172.21.15.6:0/861814432 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c074c70 msgr2=0x7fa50c19b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.571+0000 7fa511400700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c074c70 0x7fa50c19b360 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00e9a0 tx=0x7fa4fc00ecb0 comp rx=0 tx=0).stop 2024-04-26T01:41:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.572+0000 7fa511400700 1 -- 172.21.15.6:0/861814432 shutdown_connections 2024-04-26T01:41:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.572+0000 7fa511400700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa50c10f440 0x7fa50c19fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:55.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.572+0000 7fa511400700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa4f00a3cc0 0x7fa4f00a6180 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:55.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.572+0000 7fa511400700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c074c70 0x7fa50c19b360 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:55.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.572+0000 7fa511400700 1 --2- 172.21.15.6:0/861814432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa50c075590 0x7fa50c19b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:55.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.572+0000 7fa511400700 1 -- 172.21.15.6:0/861814432 >> 172.21.15.6:0/861814432 conn(0x7fa50c079ba0 msgr2=0x7fa50c0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:55.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.573+0000 7fa511400700 1 -- 172.21.15.6:0/861814432 shutdown_connections 2024-04-26T01:41:55.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:55.573+0000 7fa511400700 1 -- 172.21.15.6:0/861814432 wait complete. 2024-04-26T01:41:56.248 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:56 smithi006 ceph-mon[28419]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:56.249 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:56 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/861814432' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T01:41:56.249 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:56.249 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:56 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:56.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:56 smithi029 ceph-mon[32775]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:56.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:56 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/861814432' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T01:41:56.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:56 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:56.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:56 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:56.630 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T01:41:56.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T01:41:56.638 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:56 smithi188 ceph-mon[32751]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:56.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:56 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/861814432' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T01:41:56.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:56.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:56 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:57.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:57.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:57 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:57.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:57.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:57 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:57.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:57.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:57 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:41:58.525 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:41:58 smithi006 ceph-mon[28419]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:58.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:41:58 smithi188 ceph-mon[32751]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:58.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:41:58 smithi029 ceph-mon[32775]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:59.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.353+0000 7f4546400700 1 -- 172.21.15.6:0/3317919669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4540108a30 msgr2=0x7f4540108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:59.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.353+0000 7f4546400700 1 --2- 172.21.15.6:0/3317919669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4540108a30 0x7f4540108e10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4530009990 tx=0x7f4530009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:59.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.353+0000 7f4546400700 1 -- 172.21.15.6:0/3317919669 shutdown_connections 2024-04-26T01:41:59.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.353+0000 7f4546400700 1 --2- 172.21.15.6:0/3317919669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f454006b0f0 0x7f454010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:59.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.353+0000 7f4546400700 1 --2- 172.21.15.6:0/3317919669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4540108a30 0x7f4540108e10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:59.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.353+0000 7f4546400700 1 --2- 172.21.15.6:0/3317919669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f454006a750 0x7f454006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:59.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.353+0000 7f4546400700 1 -- 172.21.15.6:0/3317919669 >> 172.21.15.6:0/3317919669 conn(0x7f45400788f0 msgr2=0x7f4540078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:59.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.354+0000 7f4546400700 1 -- 172.21.15.6:0/3317919669 shutdown_connections 2024-04-26T01:41:59.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.354+0000 7f4546400700 1 -- 172.21.15.6:0/3317919669 wait complete. 2024-04-26T01:41:59.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.355+0000 7f4546400700 1 Processor -- start 2024-04-26T01:41:59.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.355+0000 7f4546400700 1 -- start start 2024-04-26T01:41:59.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.355+0000 7f4546400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f454006a750 0x7f454019b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:59.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.355+0000 7f4546400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f454006b0f0 0x7f454019b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:59.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.355+0000 7f4546400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4540108a30 0x7f454019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.355+0000 7f4546400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4540112c00 con 0x7f454006a750 2024-04-26T01:41:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.355+0000 7f4546400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4540112a80 con 0x7f4540108a30 2024-04-26T01:41:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.355+0000 7f4546400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4540112d80 con 0x7f454006b0f0 2024-04-26T01:41:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.356+0000 7f453f400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f454006a750 0x7f454019b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.356+0000 7f453f400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f454006a750 0x7f454019b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37956/0 (socket says 172.21.15.6:37956) 2024-04-26T01:41:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.356+0000 7f453f400700 1 -- 172.21.15.6:0/526492205 learned_addr learned my addr 172.21.15.6:0/526492205 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:41:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.356+0000 7f453ea00700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f454006b0f0 0x7f454019b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.356+0000 7f453fe00700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4540108a30 0x7f454019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:59.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.356+0000 7f453ea00700 1 -- 172.21.15.6:0/526492205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4540108a30 msgr2=0x7f454019fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:59.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.356+0000 7f453ea00700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4540108a30 0x7f454019fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:59.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.356+0000 7f453ea00700 1 -- 172.21.15.6:0/526492205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f454006a750 msgr2=0x7f454019b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:59.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.356+0000 7f453ea00700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f454006a750 0x7f454019b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:59.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.356+0000 7f453ea00700 1 -- 172.21.15.6:0/526492205 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45401a0380 con 0x7f454006b0f0 2024-04-26T01:41:59.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.356+0000 7f453f400700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f454006a750 0x7f454019b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:59.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.357+0000 7f453ea00700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f454006b0f0 0x7f454019b8b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f452c00e9f0 tx=0x7f452c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:59.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.357+0000 7f4537e00700 1 -- 172.21.15.6:0/526492205 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452c00c9c0 con 0x7f454006b0f0 2024-04-26T01:41:59.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.357+0000 7f4537e00700 1 -- 172.21.15.6:0/526492205 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f452c00cb60 con 0x7f454006b0f0 2024-04-26T01:41:59.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.357+0000 7f4546400700 1 -- 172.21.15.6:0/526492205 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45401a0670 con 0x7f454006b0f0 2024-04-26T01:41:59.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.357+0000 7f4537e00700 1 -- 172.21.15.6:0/526492205 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452c018680 con 0x7f454006b0f0 2024-04-26T01:41:59.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.357+0000 7f4546400700 1 -- 172.21.15.6:0/526492205 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45401a7ef0 con 0x7f454006b0f0 2024-04-26T01:41:59.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.359+0000 7f4546400700 1 -- 172.21.15.6:0/526492205 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4540069730 con 0x7f454006b0f0 2024-04-26T01:41:59.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.360+0000 7f4537e00700 1 -- 172.21.15.6:0/526492205 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f452c010430 con 0x7f454006b0f0 2024-04-26T01:41:59.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.362+0000 7f4537e00700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f45240a3cc0 0x7f45240a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:59.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.363+0000 7f4537e00700 1 -- 172.21.15.6:0/526492205 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f452c0d0120 con 0x7f454006b0f0 2024-04-26T01:41:59.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.363+0000 7f453f400700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f45240a3cc0 0x7f45240a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:59.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.363+0000 7f453f400700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f45240a3cc0 0x7f45240a6180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4530006010 tx=0x7f453001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:59.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.366+0000 7f4537e00700 1 -- 172.21.15.6:0/526492205 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f452c09cf00 con 0x7f454006b0f0 2024-04-26T01:41:59.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:41:59.597+0000 7f4546400700 1 -- 172.21.15.6:0/526492205 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f454019c6d0 con 0x7f454006b0f0 2024-04-26T01:42:00.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.302+0000 7f4537e00700 1 -- 172.21.15.6:0/526492205 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v85) ==== 138+0+0 (secure 0 0 0) 0x7f452c09c800 con 0x7f454006b0f0 2024-04-26T01:42:00.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.311+0000 7f4546400700 1 -- 172.21.15.6:0/526492205 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f45240a3cc0 msgr2=0x7f45240a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:00.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.311+0000 7f4546400700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f45240a3cc0 0x7f45240a6180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4530006010 tx=0x7f453001a040 comp rx=0 tx=0).stop 2024-04-26T01:42:00.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.311+0000 7f4546400700 1 -- 172.21.15.6:0/526492205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f454006b0f0 msgr2=0x7f454019b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:00.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.311+0000 7f4546400700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f454006b0f0 0x7f454019b8b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f452c00e9f0 tx=0x7f452c00ed00 comp rx=0 tx=0).stop 2024-04-26T01:42:00.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.311+0000 7f4546400700 1 -- 172.21.15.6:0/526492205 shutdown_connections 2024-04-26T01:42:00.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.311+0000 7f4546400700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4540108a30 0x7f454019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:00.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.311+0000 7f4546400700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f454006a750 0x7f454019b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.311+0000 7f4546400700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f45240a3cc0 0x7f45240a6180 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.311+0000 7f4546400700 1 --2- 172.21.15.6:0/526492205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f454006b0f0 0x7f454019b8b0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f452c00e9f0 tx=0x7f452c00ed00 comp rx=0 tx=0).stop 2024-04-26T01:42:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.311+0000 7f4546400700 1 -- 172.21.15.6:0/526492205 >> 172.21.15.6:0/526492205 conn(0x7f45400788f0 msgr2=0x7f454010df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.312+0000 7f4546400700 1 -- 172.21.15.6:0/526492205 shutdown_connections 2024-04-26T01:42:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:00.313+0000 7f4546400700 1 -- 172.21.15.6:0/526492205 wait complete. 2024-04-26T01:42:00.312 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-04-26T01:42:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:00 smithi006 ceph-mon[28419]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:42:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:42:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:42:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:42:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:42:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:00 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/526492205' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:42:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:00 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:42:00.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:00 smithi188 ceph-mon[32751]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:00.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:42:00.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:42:00.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:42:00.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:42:00.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:42:00.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:00 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/526492205' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:42:00.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:00 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:42:00.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:00 smithi029 ceph-mon[32775]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:00.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:42:00.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:42:00.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:42:00.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:42:00.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:42:00.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:00 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/526492205' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:42:00.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:00 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:42:01.173 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T01:42:01.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:01 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T01:42:01.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:01 smithi006 ceph-mon[28419]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T01:42:01.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:01 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T01:42:01.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:01 smithi188 ceph-mon[32751]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T01:42:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:01.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:01 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T01:42:01.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:01 smithi029 ceph-mon[32775]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T01:42:02.455 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:02 smithi006 ceph-mon[28419]: pgmap v322: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:02.455 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:02 smithi006 ceph-mon[28419]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:42:02.456 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:02 smithi006 ceph-mon[28419]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T01:42:02.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:02 smithi188 ceph-mon[32751]: pgmap v322: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:02.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:02 smithi188 ceph-mon[32751]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:42:02.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:02 smithi188 ceph-mon[32751]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T01:42:02.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:02 smithi029 ceph-mon[32775]: pgmap v322: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:02.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:02 smithi029 ceph-mon[32775]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:42:02.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:02 smithi029 ceph-mon[32775]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T01:42:03.515 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:03 smithi006 ceph-mon[28419]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T01:42:03.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:03 smithi188 ceph-mon[32751]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T01:42:03.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:03 smithi029 ceph-mon[32775]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T01:42:03.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.844+0000 7fec8a600700 1 -- 172.21.15.6:0/901783931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec84076f70 msgr2=0x7fec84077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:03.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.844+0000 7fec8a600700 1 --2- 172.21.15.6:0/901783931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec84076f70 0x7fec84077350 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fec74009990 tx=0x7fec74009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:03.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.844+0000 7fec8a600700 1 -- 172.21.15.6:0/901783931 shutdown_connections 2024-04-26T01:42:03.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.844+0000 7fec8a600700 1 --2- 172.21.15.6:0/901783931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec841138d0 0x7fec84115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:03.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.844+0000 7fec8a600700 1 --2- 172.21.15.6:0/901783931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec84076f70 0x7fec84077350 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:03.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.844+0000 7fec8a600700 1 --2- 172.21.15.6:0/901783931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec84077890 0x7fec84113390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:03.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.844+0000 7fec8a600700 1 -- 172.21.15.6:0/901783931 >> 172.21.15.6:0/901783931 conn(0x7fec84100720 msgr2=0x7fec84102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:03.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.845+0000 7fec8a600700 1 -- 172.21.15.6:0/901783931 shutdown_connections 2024-04-26T01:42:03.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.845+0000 7fec8a600700 1 -- 172.21.15.6:0/901783931 wait complete. 2024-04-26T01:42:03.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.846+0000 7fec8a600700 1 Processor -- start 2024-04-26T01:42:03.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.846+0000 7fec8a600700 1 -- start start 2024-04-26T01:42:03.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.846+0000 7fec8a600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec84076f70 0x7fec8419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:03.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.846+0000 7fec8a600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec84077890 0x7fec8419fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:03.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.846+0000 7fec8a600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec841138d0 0x7fec841a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:03.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.846+0000 7fec8a600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fec84119240 con 0x7fec841138d0 2024-04-26T01:42:03.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.846+0000 7fec8a600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fec841190c0 con 0x7fec84076f70 2024-04-26T01:42:03.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.846+0000 7fec8a600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fec841193c0 con 0x7fec84077890 2024-04-26T01:42:03.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.847+0000 7fec83400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec84076f70 0x7fec8419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:03.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.847+0000 7fec83400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec84076f70 0x7fec8419f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:42268/0 (socket says 172.21.15.6:42268) 2024-04-26T01:42:03.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.847+0000 7fec83400700 1 -- 172.21.15.6:0/2528673573 learned_addr learned my addr 172.21.15.6:0/2528673573 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:03.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.847+0000 7fec83e00700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec841138d0 0x7fec841a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:03.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.847+0000 7fec82a00700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec84077890 0x7fec8419fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:03.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.847+0000 7fec83400700 1 -- 172.21.15.6:0/2528673573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec84077890 msgr2=0x7fec8419fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:03.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.847+0000 7fec83400700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec84077890 0x7fec8419fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:03.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.847+0000 7fec83400700 1 -- 172.21.15.6:0/2528673573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec841138d0 msgr2=0x7fec841a4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:03.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.847+0000 7fec83400700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec841138d0 0x7fec841a4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:03.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.847+0000 7fec83400700 1 -- 172.21.15.6:0/2528673573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec841a4750 con 0x7fec84076f70 2024-04-26T01:42:03.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.847+0000 7fec83400700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec84076f70 0x7fec8419f790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fec740057f0 tx=0x7fec7400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:03.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.848+0000 7fec7be00700 1 -- 172.21.15.6:0/2528673573 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec74026070 con 0x7fec84076f70 2024-04-26T01:42:03.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.848+0000 7fec8a600700 1 -- 172.21.15.6:0/2528673573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec841a49e0 con 0x7fec84076f70 2024-04-26T01:42:03.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.848+0000 7fec7be00700 1 -- 172.21.15.6:0/2528673573 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec74021410 con 0x7fec84076f70 2024-04-26T01:42:03.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.848+0000 7fec7be00700 1 -- 172.21.15.6:0/2528673573 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec74017650 con 0x7fec84076f70 2024-04-26T01:42:03.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.848+0000 7fec8a600700 1 -- 172.21.15.6:0/2528673573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec841ac2c0 con 0x7fec84076f70 2024-04-26T01:42:03.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.849+0000 7fec8a600700 1 -- 172.21.15.6:0/2528673573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec84069730 con 0x7fec84076f70 2024-04-26T01:42:03.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.851+0000 7fec7be00700 1 -- 172.21.15.6:0/2528673573 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fec74029070 con 0x7fec84076f70 2024-04-26T01:42:03.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.853+0000 7fec7be00700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fec680a3c40 0x7fec680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:03.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.853+0000 7fec7be00700 1 -- 172.21.15.6:0/2528673573 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7497+0+0 (secure 0 0 0) 0x7fec740d0d70 con 0x7fec84076f70 2024-04-26T01:42:03.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.853+0000 7fec82a00700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fec680a3c40 0x7fec680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:03.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.854+0000 7fec82a00700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fec680a3c40 0x7fec680a6100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec84079b00 tx=0x7fec70009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:03.857+0000 7fec7be00700 1 -- 172.21.15.6:0/2528673573 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec7405dff0 con 0x7fec84076f70 2024-04-26T01:42:04.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.090+0000 7fec8a600700 1 -- 172.21.15.6:0/2528673573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fec8402df80 con 0x7fec84076f70 2024-04-26T01:42:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.360+0000 7fec7be00700 1 -- 172.21.15.6:0/2528673573 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v88) ==== 170+0+0 (secure 0 0 0) 0x7fec74013070 con 0x7fec84076f70 2024-04-26T01:42:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.366+0000 7fec8a600700 1 -- 172.21.15.6:0/2528673573 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fec680a3c40 msgr2=0x7fec680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.366+0000 7fec8a600700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fec680a3c40 0x7fec680a6100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec84079b00 tx=0x7fec70009260 comp rx=0 tx=0).stop 2024-04-26T01:42:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.366+0000 7fec8a600700 1 -- 172.21.15.6:0/2528673573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec84076f70 msgr2=0x7fec8419f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.366+0000 7fec8a600700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec84076f70 0x7fec8419f790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fec740057f0 tx=0x7fec7400bfd0 comp rx=0 tx=0).stop 2024-04-26T01:42:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.367+0000 7fec8a600700 1 -- 172.21.15.6:0/2528673573 shutdown_connections 2024-04-26T01:42:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.367+0000 7fec8a600700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec841138d0 0x7fec841a4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.367+0000 7fec8a600700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec84076f70 0x7fec8419f790 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.367+0000 7fec8a600700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fec680a3c40 0x7fec680a6100 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.367+0000 7fec8a600700 1 --2- 172.21.15.6:0/2528673573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec84077890 0x7fec8419fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.367+0000 7fec8a600700 1 -- 172.21.15.6:0/2528673573 >> 172.21.15.6:0/2528673573 conn(0x7fec84100720 msgr2=0x7fec84102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:04.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.368+0000 7fec8a600700 1 -- 172.21.15.6:0/2528673573 shutdown_connections 2024-04-26T01:42:04.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:04.368+0000 7fec8a600700 1 -- 172.21.15.6:0/2528673573 wait complete. 2024-04-26T01:42:04.370 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-04-26T01:42:04.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:04 smithi188 ceph-mon[32751]: pgmap v325: 33 pgs: 1 creating+activating, 6 creating+peering, 9 active+clean, 17 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:04.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:04 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2528673573' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:42:04.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:04 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:42:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:04 smithi029 ceph-mon[32775]: pgmap v325: 33 pgs: 1 creating+activating, 6 creating+peering, 9 active+clean, 17 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:04 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2528673573' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:42:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:04 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:42:04.734 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:04 smithi006 ceph-mon[28419]: pgmap v325: 33 pgs: 1 creating+activating, 6 creating+peering, 9 active+clean, 17 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:04.734 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:04 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2528673573' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:42:04.734 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:04 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:42:05.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T01:42:05.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:05 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T01:42:05.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:05 smithi006 ceph-mon[28419]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T01:42:05.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:05 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T01:42:05.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:05 smithi188 ceph-mon[32751]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T01:42:05.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:05 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T01:42:05.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:05 smithi029 ceph-mon[32775]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T01:42:05.743 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:06.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:06 smithi006 ceph-mon[28419]: pgmap v327: 97 pgs: 1 creating+activating, 6 creating+peering, 20 active+clean, 70 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:06.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:06 smithi006 ceph-mon[28419]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T01:42:06.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:06 smithi188 ceph-mon[32751]: pgmap v327: 97 pgs: 1 creating+activating, 6 creating+peering, 20 active+clean, 70 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:06.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:06 smithi188 ceph-mon[32751]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T01:42:06.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:06 smithi029 ceph-mon[32775]: pgmap v327: 97 pgs: 1 creating+activating, 6 creating+peering, 20 active+clean, 70 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:06.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:06 smithi029 ceph-mon[32775]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T01:42:07.494 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:07 smithi006 ceph-mon[28419]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T01:42:07.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:07 smithi188 ceph-mon[32751]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T01:42:07.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:07 smithi029 ceph-mon[32775]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T01:42:08.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.005+0000 7feca8a00700 1 -- 172.21.15.6:0/2254984032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feca4105760 msgr2=0x7feca41097b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:08.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.005+0000 7feca8a00700 1 --2- 172.21.15.6:0/2254984032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feca4105760 0x7feca41097b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fec900099c0 tx=0x7fec90009cd0 comp rx=0 tx=0).stop 2024-04-26T01:42:08.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.005+0000 7feca8a00700 1 -- 172.21.15.6:0/2254984032 shutdown_connections 2024-04-26T01:42:08.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.005+0000 7feca8a00700 1 --2- 172.21.15.6:0/2254984032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca4109e90 0x7feca4111a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:08.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.005+0000 7feca8a00700 1 --2- 172.21.15.6:0/2254984032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feca4104db0 0x7feca4105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:08.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.005+0000 7feca8a00700 1 --2- 172.21.15.6:0/2254984032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feca4105760 0x7feca41097b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:08.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.005+0000 7feca8a00700 1 -- 172.21.15.6:0/2254984032 >> 172.21.15.6:0/2254984032 conn(0x7feca4100620 msgr2=0x7feca4102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:08.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.005+0000 7feca8a00700 1 -- 172.21.15.6:0/2254984032 shutdown_connections 2024-04-26T01:42:08.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.005+0000 7feca8a00700 1 -- 172.21.15.6:0/2254984032 wait complete. 2024-04-26T01:42:08.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.006+0000 7feca8a00700 1 Processor -- start 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.007+0000 7feca8a00700 1 -- start start 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.007+0000 7feca8a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feca4104db0 0x7feca419b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.007+0000 7feca8a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feca4105760 0x7feca419b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.007+0000 7feca8a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca4109e90 0x7feca419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.007+0000 7feca8a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feca4114d60 con 0x7feca4104db0 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.007+0000 7feca8a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feca4114be0 con 0x7feca4109e90 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.007+0000 7feca8a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7feca4114ee0 con 0x7feca4105760 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.007+0000 7feca3400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feca4104db0 0x7feca419b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.007+0000 7feca2a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feca4105760 0x7feca419b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.007+0000 7feca3400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feca4104db0 0x7feca419b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33306/0 (socket says 172.21.15.6:33306) 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.007+0000 7feca3400700 1 -- 172.21.15.6:0/1620873117 learned_addr learned my addr 172.21.15.6:0/1620873117 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.007+0000 7feca3e00700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca4109e90 0x7feca419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:08.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.008+0000 7feca3400700 1 -- 172.21.15.6:0/1620873117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feca4105760 msgr2=0x7feca419b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:08.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.008+0000 7feca3400700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feca4105760 0x7feca419b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:08.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.008+0000 7feca3400700 1 -- 172.21.15.6:0/1620873117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca4109e90 msgr2=0x7feca419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:08.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.008+0000 7feca3400700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca4109e90 0x7feca419fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:08.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.008+0000 7feca3400700 1 -- 172.21.15.6:0/1620873117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca41a0290 con 0x7feca4104db0 2024-04-26T01:42:08.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.008+0000 7feca3400700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feca4104db0 0x7feca419b2d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fec9400d950 tx=0x7fec9400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:08.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.009+0000 7fec9be00700 1 -- 172.21.15.6:0/1620873117 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec94016070 con 0x7feca4104db0 2024-04-26T01:42:08.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.009+0000 7feca8a00700 1 -- 172.21.15.6:0/1620873117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca41a0580 con 0x7feca4104db0 2024-04-26T01:42:08.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.009+0000 7fec9be00700 1 -- 172.21.15.6:0/1620873117 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec9400fd40 con 0x7feca4104db0 2024-04-26T01:42:08.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.009+0000 7fec9be00700 1 -- 172.21.15.6:0/1620873117 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9400e3b0 con 0x7feca4104db0 2024-04-26T01:42:08.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.009+0000 7feca8a00700 1 -- 172.21.15.6:0/1620873117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca41a7e00 con 0x7feca4104db0 2024-04-26T01:42:08.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.010+0000 7feca8a00700 1 -- 172.21.15.6:0/1620873117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feca4069730 con 0x7feca4104db0 2024-04-26T01:42:08.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.014+0000 7fec9be00700 1 -- 172.21.15.6:0/1620873117 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fec94019070 con 0x7feca4104db0 2024-04-26T01:42:08.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.015+0000 7fec9be00700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fec880a3c40 0x7fec880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:08.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.015+0000 7fec9be00700 1 -- 172.21.15.6:0/1620873117 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 7879+0+0 (secure 0 0 0) 0x7fec940cfd60 con 0x7feca4104db0 2024-04-26T01:42:08.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.015+0000 7feca2a00700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fec880a3c40 0x7fec880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:08.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.016+0000 7feca2a00700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fec880a3c40 0x7fec880a6100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fec90006010 tx=0x7fec90005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:08.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.019+0000 7fec9be00700 1 -- 172.21.15.6:0/1620873117 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec9409c850 con 0x7feca4104db0 2024-04-26T01:42:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.298+0000 7feca8a00700 1 -- 172.21.15.6:0/1620873117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7feca402df80 con 0x7feca4104db0 2024-04-26T01:42:08.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.404+0000 7fec9be00700 1 -- 172.21.15.6:0/1620873117 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fec9409c150 con 0x7feca4104db0 2024-04-26T01:42:08.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.407+0000 7feca8a00700 1 -- 172.21.15.6:0/1620873117 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fec880a3c40 msgr2=0x7fec880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:08.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.407+0000 7feca8a00700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fec880a3c40 0x7fec880a6100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fec90006010 tx=0x7fec90005c20 comp rx=0 tx=0).stop 2024-04-26T01:42:08.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.407+0000 7feca8a00700 1 -- 172.21.15.6:0/1620873117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feca4104db0 msgr2=0x7feca419b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:08.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.407+0000 7feca8a00700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feca4104db0 0x7feca419b2d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fec9400d950 tx=0x7fec9400dc60 comp rx=0 tx=0).stop 2024-04-26T01:42:08.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.408+0000 7feca8a00700 1 -- 172.21.15.6:0/1620873117 shutdown_connections 2024-04-26T01:42:08.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.408+0000 7feca8a00700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca4109e90 0x7feca419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:08.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.408+0000 7feca8a00700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feca4104db0 0x7feca419b2d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:08.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.408+0000 7feca8a00700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fec880a3c40 0x7fec880a6100 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.408+0000 7feca8a00700 1 --2- 172.21.15.6:0/1620873117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feca4105760 0x7feca419b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.408+0000 7feca8a00700 1 -- 172.21.15.6:0/1620873117 >> 172.21.15.6:0/1620873117 conn(0x7feca4100620 msgr2=0x7feca41029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.408+0000 7feca8a00700 1 -- 172.21.15.6:0/1620873117 shutdown_connections 2024-04-26T01:42:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:08.408+0000 7feca8a00700 1 -- 172.21.15.6:0/1620873117 wait complete. 2024-04-26T01:42:08.407 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T01:42:08.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:08 smithi006 ceph-mon[28419]: pgmap v330: 97 pgs: 12 creating+peering, 54 active+clean, 31 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:08.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:08 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1620873117' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T01:42:08.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:08 smithi006 ceph-mon[28419]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-26T01:42:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:08 smithi188 ceph-mon[32751]: pgmap v330: 97 pgs: 12 creating+peering, 54 active+clean, 31 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:08 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1620873117' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T01:42:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:08 smithi188 ceph-mon[32751]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-26T01:42:08.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:08 smithi029 ceph-mon[32775]: pgmap v330: 97 pgs: 12 creating+peering, 54 active+clean, 31 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:42:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:08 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1620873117' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T01:42:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:08 smithi029 ceph-mon[32775]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-26T01:42:09.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:42:09.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:09 smithi006 ceph-mon[28419]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T01:42:09.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:09 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1620873117' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T01:42:09.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:09 smithi006 ceph-mon[28419]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-26T01:42:09.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:09 smithi006 ceph-mon[28419]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-26T01:42:09.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:09 smithi029 ceph-mon[32775]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T01:42:09.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:09 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1620873117' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T01:42:09.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:09 smithi029 ceph-mon[32775]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-26T01:42:09.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:09 smithi029 ceph-mon[32775]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-26T01:42:09.825 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:09.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:09 smithi188 ceph-mon[32751]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T01:42:09.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:09 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1620873117' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T01:42:09.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:09 smithi188 ceph-mon[32751]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-26T01:42:09.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:09 smithi188 ceph-mon[32751]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-26T01:42:10.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:10 smithi006 ceph-mon[28419]: pgmap v332: 97 pgs: 12 creating+peering, 73 active+clean, 12 unknown; 578 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s wr, 2 op/s 2024-04-26T01:42:10.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:10 smithi006 ceph-mon[28419]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T01:42:10.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:10 smithi006 ceph-mon[28419]: Cluster is now healthy 2024-04-26T01:42:10.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:10 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] up:active 2024-04-26T01:42:10.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:10 smithi006 ceph-mon[28419]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:42:10.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:10 smithi006 ceph-mon[28419]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:42:10.641 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 01:42:10 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i[64227]: 2024-04-26T01:42:10.249+0000 7f1cff6dd700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T01:42:10.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:10 smithi188 ceph-mon[32751]: pgmap v332: 97 pgs: 12 creating+peering, 73 active+clean, 12 unknown; 578 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s wr, 2 op/s 2024-04-26T01:42:10.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:10 smithi188 ceph-mon[32751]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T01:42:10.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:10 smithi188 ceph-mon[32751]: Cluster is now healthy 2024-04-26T01:42:10.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:10 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] up:active 2024-04-26T01:42:10.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:10 smithi188 ceph-mon[32751]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:42:10.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:10 smithi188 ceph-mon[32751]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:42:10.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:10 smithi029 ceph-mon[32775]: pgmap v332: 97 pgs: 12 creating+peering, 73 active+clean, 12 unknown; 578 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s wr, 2 op/s 2024-04-26T01:42:10.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:10 smithi029 ceph-mon[32775]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T01:42:10.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:10 smithi029 ceph-mon[32775]: Cluster is now healthy 2024-04-26T01:42:10.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:10 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] up:active 2024-04-26T01:42:10.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:10 smithi029 ceph-mon[32775]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:42:10.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:10 smithi029 ceph-mon[32775]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:42:11.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.910+0000 7f0560a00700 1 -- 172.21.15.6:0/3687518411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f055c10e3b0 msgr2=0x7f055c115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:11.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.910+0000 7f0560a00700 1 --2- 172.21.15.6:0/3687518411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f055c10e3b0 0x7f055c115f80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f055400b0d0 tx=0x7f055400b3e0 comp rx=0 tx=0).stop 2024-04-26T01:42:11.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.910+0000 7f0560a00700 1 -- 172.21.15.6:0/3687518411 shutdown_connections 2024-04-26T01:42:11.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.910+0000 7f0560a00700 1 --2- 172.21.15.6:0/3687518411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f055c10e3b0 0x7f055c115f80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:11.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.910+0000 7f0560a00700 1 --2- 172.21.15.6:0/3687518411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f055c1092d0 0x7f055c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:11.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.910+0000 7f0560a00700 1 --2- 172.21.15.6:0/3687518411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f055c109c80 0x7f055c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:11.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.910+0000 7f0560a00700 1 -- 172.21.15.6:0/3687518411 >> 172.21.15.6:0/3687518411 conn(0x7f055c076210 msgr2=0x7f055c107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:11.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.910+0000 7f0560a00700 1 -- 172.21.15.6:0/3687518411 shutdown_connections 2024-04-26T01:42:11.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.910+0000 7f0560a00700 1 -- 172.21.15.6:0/3687518411 wait complete. 2024-04-26T01:42:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.911+0000 7f0560a00700 1 Processor -- start 2024-04-26T01:42:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.911+0000 7f0560a00700 1 -- start start 2024-04-26T01:42:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.912+0000 7f0560a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f055c1092d0 0x7f055c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.912+0000 7f0560a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f055c109c80 0x7f055c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.912+0000 7f0560a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f055c10e3b0 0x7f055c1a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.912+0000 7f0560a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f055c1192a0 con 0x7f055c1092d0 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.912+0000 7f0560a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f055c119120 con 0x7f055c10e3b0 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.912+0000 7f0560a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f055c119420 con 0x7f055c109c80 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.912+0000 7f0559e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f055c1092d0 0x7f055c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.912+0000 7f0559400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f055c109c80 0x7f055c19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.912+0000 7f0559e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f055c1092d0 0x7f055c19f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33334/0 (socket says 172.21.15.6:33334) 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.912+0000 7f0559e00700 1 -- 172.21.15.6:0/2097035169 learned_addr learned my addr 172.21.15.6:0/2097035169 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.912+0000 7f055a800700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f055c10e3b0 0x7f055c1a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.912+0000 7f0559400700 1 -- 172.21.15.6:0/2097035169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f055c10e3b0 msgr2=0x7f055c1a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.913+0000 7f0559400700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f055c10e3b0 0x7f055c1a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.913+0000 7f0559400700 1 -- 172.21.15.6:0/2097035169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f055c1092d0 msgr2=0x7f055c19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.913+0000 7f0559400700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f055c1092d0 0x7f055c19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.913+0000 7f0559400700 1 -- 172.21.15.6:0/2097035169 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f055c1a4730 con 0x7f055c109c80 2024-04-26T01:42:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.913+0000 7f0559400700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f055c109c80 0x7f055c19fcb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f054800e9c0 tx=0x7f054800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:11.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.913+0000 7f0552a00700 1 -- 172.21.15.6:0/2097035169 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054800c9c0 con 0x7f055c109c80 2024-04-26T01:42:11.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.913+0000 7f0560a00700 1 -- 172.21.15.6:0/2097035169 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f055c1a4a20 con 0x7f055c109c80 2024-04-26T01:42:11.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.913+0000 7f0552a00700 1 -- 172.21.15.6:0/2097035169 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f054800cb60 con 0x7f055c109c80 2024-04-26T01:42:11.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.913+0000 7f0552a00700 1 -- 172.21.15.6:0/2097035169 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548018680 con 0x7f055c109c80 2024-04-26T01:42:11.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.913+0000 7f0560a00700 1 -- 172.21.15.6:0/2097035169 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f055c1ac2a0 con 0x7f055c109c80 2024-04-26T01:42:11.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.914+0000 7f0560a00700 1 -- 172.21.15.6:0/2097035169 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f055c069730 con 0x7f055c109c80 2024-04-26T01:42:11.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.917+0000 7f0552a00700 1 -- 172.21.15.6:0/2097035169 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f0548010430 con 0x7f055c109c80 2024-04-26T01:42:11.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.918+0000 7f0552a00700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f05400a3c40 0x7f05400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:11.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.918+0000 7f0552a00700 1 -- 172.21.15.6:0/2097035169 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7979+0+0 (secure 0 0 0) 0x7f05480d0450 con 0x7f055c109c80 2024-04-26T01:42:11.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.918+0000 7f0559e00700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f05400a3c40 0x7f05400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:11.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.919+0000 7f0559e00700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f05400a3c40 0x7f05400a6100 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f054c009d20 tx=0x7f054c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:11.922+0000 7f0552a00700 1 -- 172.21.15.6:0/2097035169 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f054809cee0 con 0x7f055c109c80 2024-04-26T01:42:12.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.158+0000 7f0560a00700 1 -- 172.21.15.6:0/2097035169 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f055c02df80 con 0x7f055c109c80 2024-04-26T01:42:12.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.160+0000 7f0552a00700 1 -- 172.21.15.6:0/2097035169 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22109 (secure 0 0 0) 0x7f054809c7e0 con 0x7f055c109c80 2024-04-26T01:42:12.158 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:42:12.158 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":91,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:32:29.071544+0000","modified":"2024-04-26T01:42:08.299651+0000","last_up_change":"2024-04-26T01:40:49.915924+0000","last_in_change":"2024-04-26T01:40:29.559647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:36:35.182726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:41:59.612167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:42:04.106467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f164c51d-ce5d-4140-b9e3-a4b84088f5ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6803","nonce":76518486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6805","nonce":76518486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6809","nonce":76518486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6807","nonce":76518486}]},"public_addr":"172.21.15.6:6803/76518486","cluster_addr":"172.21.15.6:6805/76518486","heartbeat_back_addr":"172.21.15.6:6809/76518486","heartbeat_front_addr":"172.21.15.6:6807/76518486","state":["exists","up"]},{"osd":1,"uuid":"57e875dd-ddf1-4ee4-a2a7-9589d7f42818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6801","nonce":400769116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6803","nonce":400769116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6807","nonce":400769116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6805","nonce":400769116}]},"public_addr":"172.21.15.29:6801/400769116","cluster_addr":"172.21.15.29:6803/400769116","heartbeat_back_addr":"172.21.15.29:6807/400769116","heartbeat_front_addr":"172.21.15.29:6805/400769116","state":["exists","up"]},{"osd":2,"uuid":"77e0c294-c48a-4318-938b-4604a533a2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6801","nonce":3508067672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6803","nonce":3508067672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6807","nonce":3508067672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6805","nonce":3508067672}]},"public_addr":"172.21.15.188:6801/3508067672","cluster_addr":"172.21.15.188:6803/3508067672","heartbeat_back_addr":"172.21.15.188:6807/3508067672","heartbeat_front_addr":"172.21.15.188:6805/3508067672","state":["exists","up"]},{"osd":3,"uuid":"6393dac5-996c-47af-9687-0f1393ecc483","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6811","nonce":1987311118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6813","nonce":1987311118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6817","nonce":1987311118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6815","nonce":1987311118}]},"public_addr":"172.21.15.6:6811/1987311118","cluster_addr":"172.21.15.6:6813/1987311118","heartbeat_back_addr":"172.21.15.6:6817/1987311118","heartbeat_front_addr":"172.21.15.6:6815/1987311118","state":["exists","up"]},{"osd":4,"uuid":"f2c1a535-26a0-46c6-a6d0-c2529c786739","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6809","nonce":2125876609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6811","nonce":2125876609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6815","nonce":2125876609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6813","nonce":2125876609}]},"public_addr":"172.21.15.29:6809/2125876609","cluster_addr":"172.21.15.29:6811/2125876609","heartbeat_back_addr":"172.21.15.29:6815/2125876609","heartbeat_front_addr":"172.21.15.29:6813/2125876609","state":["exists","up"]},{"osd":5,"uuid":"9788bb0a-0d79-4743-bce6-80f5342e8f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6809","nonce":2092077007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6811","nonce":2092077007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6815","nonce":2092077007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6813","nonce":2092077007}]},"public_addr":"172.21.15.188:6809/2092077007","cluster_addr":"172.21.15.188:6811/2092077007","heartbeat_back_addr":"172.21.15.188:6815/2092077007","heartbeat_front_addr":"172.21.15.188:6813/2092077007","state":["exists","up"]},{"osd":6,"uuid":"fd8584e1-69ee-4c30-85e4-a637357d1b6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6819","nonce":3256016695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6821","nonce":3256016695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6825","nonce":3256016695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6823","nonce":3256016695}]},"public_addr":"172.21.15.6:6819/3256016695","cluster_addr":"172.21.15.6:6821/3256016695","heartbeat_back_addr":"172.21.15.6:6825/3256016695","heartbeat_front_addr":"172.21.15.6:6823/3256016695","state":["exists","up"]},{"osd":7,"uuid":"6b7e6784-d574-42da-8416-a74a3b6b3755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6817","nonce":3478594037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6819","nonce":3478594037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6823","nonce":3478594037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6821","nonce":3478594037}]},"public_addr":"172.21.15.29:6817/3478594037","cluster_addr":"172.21.15.29:6819/3478594037","heartbeat_back_addr":"172.21.15.29:6823/3478594037","heartbeat_front_addr":"172.21.15.29:6821/3478594037","state":["exists","up"]},{"osd":8,"uuid":"30cd2ea1-2767-4062-8bd9-47214e4b4b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6817","nonce":3290364376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6819","nonce":3290364376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6823","nonce":3290364376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6821","nonce":3290364376}]},"public_addr":"172.21.15.188:6817/3290364376","cluster_addr":"172.21.15.188:6819/3290364376","heartbeat_back_addr":"172.21.15.188:6823/3290364376","heartbeat_front_addr":"172.21.15.188:6821/3290364376","state":["exists","up"]},{"osd":9,"uuid":"f033cdc0-1178-4a7f-a338-69ff32416087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6827","nonce":1918893912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6829","nonce":1918893912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6833","nonce":1918893912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6831","nonce":1918893912}]},"public_addr":"172.21.15.6:6827/1918893912","cluster_addr":"172.21.15.6:6829/1918893912","heartbeat_back_addr":"172.21.15.6:6833/1918893912","heartbeat_front_addr":"172.21.15.6:6831/1918893912","state":["exists","up"]},{"osd":10,"uuid":"04e9b5ef-ac0c-482b-a69e-a7121b36e7da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6825","nonce":1395777729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6827","nonce":1395777729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6831","nonce":1395777729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6829","nonce":1395777729}]},"public_addr":"172.21.15.29:6825/1395777729","cluster_addr":"172.21.15.29:6827/1395777729","heartbeat_back_addr":"172.21.15.29:6831/1395777729","heartbeat_front_addr":"172.21.15.29:6829/1395777729","state":["exists","up"]},{"osd":11,"uuid":"cb1638ce-4189-4241-b63e-b63c607e90ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6825","nonce":3322197880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6827","nonce":3322197880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6831","nonce":3322197880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6829","nonce":3322197880}]},"public_addr":"172.21.15.188:6825/3322197880","cluster_addr":"172.21.15.188:6827/3322197880","heartbeat_back_addr":"172.21.15.188:6831/3322197880","heartbeat_front_addr":"172.21.15.188:6829/3322197880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:41.553455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:07.641467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:31.748891+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:03.743553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:30.065262+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:54.976527+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:29.348206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:56.285928+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:21.246409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:54.468561+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:22.093015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:47.342379+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/3143272062":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1325707596":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6800/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6801/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1359518071":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6801/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/2813598711":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/3140150861":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6800/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/65701414":"2024-04-27T01:33:47.021882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:42:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.164+0000 7f0560a00700 1 -- 172.21.15.6:0/2097035169 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f05400a3c40 msgr2=0x7f05400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.164+0000 7f0560a00700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f05400a3c40 0x7f05400a6100 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f054c009d20 tx=0x7f054c009260 comp rx=0 tx=0).stop 2024-04-26T01:42:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.164+0000 7f0560a00700 1 -- 172.21.15.6:0/2097035169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f055c109c80 msgr2=0x7f055c19fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.164+0000 7f0560a00700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f055c109c80 0x7f055c19fcb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f054800e9c0 tx=0x7f054800ecd0 comp rx=0 tx=0).stop 2024-04-26T01:42:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.164+0000 7f0560a00700 1 -- 172.21.15.6:0/2097035169 shutdown_connections 2024-04-26T01:42:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.164+0000 7f0560a00700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f055c10e3b0 0x7f055c1a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.164+0000 7f0560a00700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f055c1092d0 0x7f055c19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.164+0000 7f0560a00700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f05400a3c40 0x7f05400a6100 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.164+0000 7f0560a00700 1 --2- 172.21.15.6:0/2097035169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f055c109c80 0x7f055c19fcb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.164+0000 7f0560a00700 1 -- 172.21.15.6:0/2097035169 >> 172.21.15.6:0/2097035169 conn(0x7f055c076210 msgr2=0x7f055c0767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:12.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.164+0000 7f0560a00700 1 -- 172.21.15.6:0/2097035169 shutdown_connections 2024-04-26T01:42:12.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:12.164+0000 7f0560a00700 1 -- 172.21.15.6:0/2097035169 wait complete. 2024-04-26T01:42:12.533 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:12 smithi006 ceph-mon[28419]: pgmap v333: 97 pgs: 97 active+clean; 578 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-04-26T01:42:12.533 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:12 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2097035169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:12.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:12 smithi029 ceph-mon[32775]: pgmap v333: 97 pgs: 97 active+clean; 578 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-04-26T01:42:12.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:12 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2097035169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:12.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:12 smithi188 ceph-mon[32751]: pgmap v333: 97 pgs: 97 active+clean; 578 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-04-26T01:42:12.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:12 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2097035169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:12.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:42:13.409 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:14.660 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:14 smithi006 ceph-mon[28419]: pgmap v334: 97 pgs: 97 active+clean; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-26T01:42:14.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:14 smithi029 ceph-mon[32775]: pgmap v334: 97 pgs: 97 active+clean; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-26T01:42:14.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:14 smithi188 ceph-mon[32751]: pgmap v334: 97 pgs: 97 active+clean; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-26T01:42:15.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.502+0000 7fa144400700 1 -- 172.21.15.6:0/147928667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa13c108a50 msgr2=0x7fa13c108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:15.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.502+0000 7fa144400700 1 --2- 172.21.15.6:0/147928667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa13c108a50 0x7fa13c108e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa130009990 tx=0x7fa130009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:15.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.502+0000 7fa144400700 1 -- 172.21.15.6:0/147928667 shutdown_connections 2024-04-26T01:42:15.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.502+0000 7fa144400700 1 --2- 172.21.15.6:0/147928667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa13c103450 0x7fa13c10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:15.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.502+0000 7fa144400700 1 --2- 172.21.15.6:0/147928667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa13c108a50 0x7fa13c108e30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:15.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.502+0000 7fa144400700 1 --2- 172.21.15.6:0/147928667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa13c102a90 0x7fa13c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:15.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.502+0000 7fa144400700 1 -- 172.21.15.6:0/147928667 >> 172.21.15.6:0/147928667 conn(0x7fa13c079ba0 msgr2=0x7fa13c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:15.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.502+0000 7fa144400700 1 -- 172.21.15.6:0/147928667 shutdown_connections 2024-04-26T01:42:15.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.502+0000 7fa144400700 1 -- 172.21.15.6:0/147928667 wait complete. 2024-04-26T01:42:15.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.504+0000 7fa144400700 1 Processor -- start 2024-04-26T01:42:15.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.504+0000 7fa144400700 1 -- start start 2024-04-26T01:42:15.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.504+0000 7fa144400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa13c102a90 0x7fa13c19b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:15.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.504+0000 7fa144400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa13c103450 0x7fa13c19b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:15.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.504+0000 7fa144400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa13c108a50 0x7fa13c19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:15.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa144400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa13c112c30 con 0x7fa13c103450 2024-04-26T01:42:15.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa144400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa13c112ab0 con 0x7fa13c102a90 2024-04-26T01:42:15.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa144400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa13c112db0 con 0x7fa13c108a50 2024-04-26T01:42:15.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa141000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa13c103450 0x7fa13c19b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:15.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa142400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa13c108a50 0x7fa13c19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:15.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa141a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa13c102a90 0x7fa13c19b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:15.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa141000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa13c103450 0x7fa13c19b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33360/0 (socket says 172.21.15.6:33360) 2024-04-26T01:42:15.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa142400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa13c108a50 0x7fa13c19fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:54150/0 (socket says 172.21.15.6:54150) 2024-04-26T01:42:15.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa141000700 1 -- 172.21.15.6:0/2722037725 learned_addr learned my addr 172.21.15.6:0/2722037725 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:15.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa142400700 1 -- 172.21.15.6:0/2722037725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa13c102a90 msgr2=0x7fa13c19b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:15.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa142400700 1 --2- 172.21.15.6:0/2722037725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa13c102a90 0x7fa13c19b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:15.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa142400700 1 -- 172.21.15.6:0/2722037725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa13c103450 msgr2=0x7fa13c19b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:15.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa142400700 1 --2- 172.21.15.6:0/2722037725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa13c103450 0x7fa13c19b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:15.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.505+0000 7fa142400700 1 -- 172.21.15.6:0/2722037725 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13c1a0370 con 0x7fa13c108a50 2024-04-26T01:42:15.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.506+0000 7fa142400700 1 --2- 172.21.15.6:0/2722037725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa13c108a50 0x7fa13c19fc30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa13400efc0 tx=0x7fa13400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:15.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.506+0000 7fa13a000700 1 -- 172.21.15.6:0/2722037725 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1340092e0 con 0x7fa13c108a50 2024-04-26T01:42:15.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.506+0000 7fa144400700 1 -- 172.21.15.6:0/2722037725 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13c1a0600 con 0x7fa13c108a50 2024-04-26T01:42:15.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.506+0000 7fa13a000700 1 -- 172.21.15.6:0/2722037725 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa134004bb0 con 0x7fa13c108a50 2024-04-26T01:42:15.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.506+0000 7fa13a000700 1 -- 172.21.15.6:0/2722037725 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa134004020 con 0x7fa13c108a50 2024-04-26T01:42:15.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.506+0000 7fa144400700 1 -- 172.21.15.6:0/2722037725 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa13c1a7ee0 con 0x7fa13c108a50 2024-04-26T01:42:15.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.507+0000 7fa144400700 1 -- 172.21.15.6:0/2722037725 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa13c069730 con 0x7fa13c108a50 2024-04-26T01:42:15.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.509+0000 7fa13a000700 1 -- 172.21.15.6:0/2722037725 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fa13401b030 con 0x7fa13c108a50 2024-04-26T01:42:15.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.511+0000 7fa13a000700 1 --2- 172.21.15.6:0/2722037725 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa1240a3cc0 0x7fa1240a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:15.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.511+0000 7fa13a000700 1 -- 172.21.15.6:0/2722037725 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7979+0+0 (secure 0 0 0) 0x7fa1340cfcb0 con 0x7fa13c108a50 2024-04-26T01:42:15.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.511+0000 7fa141a00700 1 --2- 172.21.15.6:0/2722037725 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa1240a3cc0 0x7fa1240a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:15.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.512+0000 7fa141a00700 1 --2- 172.21.15.6:0/2722037725 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa1240a3cc0 0x7fa1240a6180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa130006010 tx=0x7fa13000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:15.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.515+0000 7fa13a000700 1 -- 172.21.15.6:0/2722037725 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa134022020 con 0x7fa13c108a50 2024-04-26T01:42:15.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.746+0000 7fa144400700 1 -- 172.21.15.6:0/2722037725 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa13c02df80 con 0x7fa13c108a50 2024-04-26T01:42:15.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.749+0000 7fa13a000700 1 -- 172.21.15.6:0/2722037725 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22109 (secure 0 0 0) 0x7fa134010040 con 0x7fa13c108a50 2024-04-26T01:42:15.746 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:42:15.746 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":91,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:32:29.071544+0000","modified":"2024-04-26T01:42:08.299651+0000","last_up_change":"2024-04-26T01:40:49.915924+0000","last_in_change":"2024-04-26T01:40:29.559647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:36:35.182726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:41:59.612167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:42:04.106467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f164c51d-ce5d-4140-b9e3-a4b84088f5ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6803","nonce":76518486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6805","nonce":76518486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6809","nonce":76518486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6807","nonce":76518486}]},"public_addr":"172.21.15.6:6803/76518486","cluster_addr":"172.21.15.6:6805/76518486","heartbeat_back_addr":"172.21.15.6:6809/76518486","heartbeat_front_addr":"172.21.15.6:6807/76518486","state":["exists","up"]},{"osd":1,"uuid":"57e875dd-ddf1-4ee4-a2a7-9589d7f42818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6801","nonce":400769116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6803","nonce":400769116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6807","nonce":400769116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6805","nonce":400769116}]},"public_addr":"172.21.15.29:6801/400769116","cluster_addr":"172.21.15.29:6803/400769116","heartbeat_back_addr":"172.21.15.29:6807/400769116","heartbeat_front_addr":"172.21.15.29:6805/400769116","state":["exists","up"]},{"osd":2,"uuid":"77e0c294-c48a-4318-938b-4604a533a2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6801","nonce":3508067672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6803","nonce":3508067672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6807","nonce":3508067672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6805","nonce":3508067672}]},"public_addr":"172.21.15.188:6801/3508067672","cluster_addr":"172.21.15.188:6803/3508067672","heartbeat_back_addr":"172.21.15.188:6807/3508067672","heartbeat_front_addr":"172.21.15.188:6805/3508067672","state":["exists","up"]},{"osd":3,"uuid":"6393dac5-996c-47af-9687-0f1393ecc483","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6811","nonce":1987311118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6813","nonce":1987311118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6817","nonce":1987311118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6815","nonce":1987311118}]},"public_addr":"172.21.15.6:6811/1987311118","cluster_addr":"172.21.15.6:6813/1987311118","heartbeat_back_addr":"172.21.15.6:6817/1987311118","heartbeat_front_addr":"172.21.15.6:6815/1987311118","state":["exists","up"]},{"osd":4,"uuid":"f2c1a535-26a0-46c6-a6d0-c2529c786739","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6809","nonce":2125876609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6811","nonce":2125876609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6815","nonce":2125876609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6813","nonce":2125876609}]},"public_addr":"172.21.15.29:6809/2125876609","cluster_addr":"172.21.15.29:6811/2125876609","heartbeat_back_addr":"172.21.15.29:6815/2125876609","heartbeat_front_addr":"172.21.15.29:6813/2125876609","state":["exists","up"]},{"osd":5,"uuid":"9788bb0a-0d79-4743-bce6-80f5342e8f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6809","nonce":2092077007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6811","nonce":2092077007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6815","nonce":2092077007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6813","nonce":2092077007}]},"public_addr":"172.21.15.188:6809/2092077007","cluster_addr":"172.21.15.188:6811/2092077007","heartbeat_back_addr":"172.21.15.188:6815/2092077007","heartbeat_front_addr":"172.21.15.188:6813/2092077007","state":["exists","up"]},{"osd":6,"uuid":"fd8584e1-69ee-4c30-85e4-a637357d1b6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6819","nonce":3256016695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6821","nonce":3256016695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6825","nonce":3256016695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6823","nonce":3256016695}]},"public_addr":"172.21.15.6:6819/3256016695","cluster_addr":"172.21.15.6:6821/3256016695","heartbeat_back_addr":"172.21.15.6:6825/3256016695","heartbeat_front_addr":"172.21.15.6:6823/3256016695","state":["exists","up"]},{"osd":7,"uuid":"6b7e6784-d574-42da-8416-a74a3b6b3755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6817","nonce":3478594037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6819","nonce":3478594037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6823","nonce":3478594037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6821","nonce":3478594037}]},"public_addr":"172.21.15.29:6817/3478594037","cluster_addr":"172.21.15.29:6819/3478594037","heartbeat_back_addr":"172.21.15.29:6823/3478594037","heartbeat_front_addr":"172.21.15.29:6821/3478594037","state":["exists","up"]},{"osd":8,"uuid":"30cd2ea1-2767-4062-8bd9-47214e4b4b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6817","nonce":3290364376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6819","nonce":3290364376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6823","nonce":3290364376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6821","nonce":3290364376}]},"public_addr":"172.21.15.188:6817/3290364376","cluster_addr":"172.21.15.188:6819/3290364376","heartbeat_back_addr":"172.21.15.188:6823/3290364376","heartbeat_front_addr":"172.21.15.188:6821/3290364376","state":["exists","up"]},{"osd":9,"uuid":"f033cdc0-1178-4a7f-a338-69ff32416087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6827","nonce":1918893912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6829","nonce":1918893912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6833","nonce":1918893912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6831","nonce":1918893912}]},"public_addr":"172.21.15.6:6827/1918893912","cluster_addr":"172.21.15.6:6829/1918893912","heartbeat_back_addr":"172.21.15.6:6833/1918893912","heartbeat_front_addr":"172.21.15.6:6831/1918893912","state":["exists","up"]},{"osd":10,"uuid":"04e9b5ef-ac0c-482b-a69e-a7121b36e7da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6825","nonce":1395777729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6827","nonce":1395777729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6831","nonce":1395777729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6829","nonce":1395777729}]},"public_addr":"172.21.15.29:6825/1395777729","cluster_addr":"172.21.15.29:6827/1395777729","heartbeat_back_addr":"172.21.15.29:6831/1395777729","heartbeat_front_addr":"172.21.15.29:6829/1395777729","state":["exists","up"]},{"osd":11,"uuid":"cb1638ce-4189-4241-b63e-b63c607e90ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6825","nonce":3322197880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6827","nonce":3322197880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6831","nonce":3322197880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6829","nonce":3322197880}]},"public_addr":"172.21.15.188:6825/3322197880","cluster_addr":"172.21.15.188:6827/3322197880","heartbeat_back_addr":"172.21.15.188:6831/3322197880","heartbeat_front_addr":"172.21.15.188:6829/3322197880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:41.553455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:07.641467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:31.748891+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:03.743553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:30.065262+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:54.976527+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:29.348206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:56.285928+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:21.246409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:54.468561+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:22.093015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:47.342379+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/3143272062":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1325707596":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6800/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6801/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1359518071":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6801/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/2813598711":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/3140150861":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6800/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/65701414":"2024-04-27T01:33:47.021882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:42:15.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.752+0000 7fa144400700 1 -- 172.21.15.6:0/2722037725 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa1240a3cc0 msgr2=0x7fa1240a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:15.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.752+0000 7fa144400700 1 --2- 172.21.15.6:0/2722037725 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa1240a3cc0 0x7fa1240a6180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa130006010 tx=0x7fa13000b260 comp rx=0 tx=0).stop 2024-04-26T01:42:15.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.752+0000 7fa144400700 1 -- 172.21.15.6:0/2722037725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa13c108a50 msgr2=0x7fa13c19fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:15.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.752+0000 7fa144400700 1 --2- 172.21.15.6:0/2722037725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa13c108a50 0x7fa13c19fc30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa13400efc0 tx=0x7fa13400c3a0 comp rx=0 tx=0).stop 2024-04-26T01:42:15.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.753+0000 7fa144400700 1 -- 172.21.15.6:0/2722037725 shutdown_connections 2024-04-26T01:42:15.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.753+0000 7fa144400700 1 --2- 172.21.15.6:0/2722037725 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa1240a3cc0 0x7fa1240a6180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:15.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.753+0000 7fa144400700 1 --2- 172.21.15.6:0/2722037725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa13c108a50 0x7fa13c19fc30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:15.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.753+0000 7fa144400700 1 --2- 172.21.15.6:0/2722037725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa13c102a90 0x7fa13c19b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:15.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.753+0000 7fa144400700 1 --2- 172.21.15.6:0/2722037725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa13c103450 0x7fa13c19b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:15.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.753+0000 7fa144400700 1 -- 172.21.15.6:0/2722037725 >> 172.21.15.6:0/2722037725 conn(0x7fa13c079ba0 msgr2=0x7fa13c10dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:15.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.753+0000 7fa144400700 1 -- 172.21.15.6:0/2722037725 shutdown_connections 2024-04-26T01:42:15.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:15.753+0000 7fa144400700 1 -- 172.21.15.6:0/2722037725 wait complete. 2024-04-26T01:42:16.533 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:16 smithi006 ceph-mon[28419]: pgmap v335: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-26T01:42:16.534 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:16 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2722037725' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:16.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-04-26T01:42:16.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:16 smithi029 ceph-mon[32775]: pgmap v335: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-26T01:42:16.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:16 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2722037725' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:16.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:16 smithi188 ceph-mon[32751]: pgmap v335: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-26T01:42:16.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:16 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2722037725' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:17.055 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:18.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:18 smithi006 ceph-mon[28419]: pgmap v336: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T01:42:18.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:18 smithi029 ceph-mon[32775]: pgmap v336: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T01:42:18.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:18 smithi188 ceph-mon[32751]: pgmap v336: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T01:42:19.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.101+0000 7ff528000700 1 -- 172.21.15.6:0/4199430352 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5201092d0 msgr2=0x7ff5201096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:19.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.101+0000 7ff528000700 1 --2- 172.21.15.6:0/4199430352 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5201092d0 0x7ff5201096b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff5140099c0 tx=0x7ff514009cd0 comp rx=0 tx=0).stop 2024-04-26T01:42:19.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.101+0000 7ff528000700 1 -- 172.21.15.6:0/4199430352 shutdown_connections 2024-04-26T01:42:19.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.101+0000 7ff528000700 1 --2- 172.21.15.6:0/4199430352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff52010e3b0 0x7ff520115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:19.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.101+0000 7ff528000700 1 --2- 172.21.15.6:0/4199430352 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5201092d0 0x7ff5201096b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:19.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.101+0000 7ff528000700 1 --2- 172.21.15.6:0/4199430352 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff520109c80 0x7ff52010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:19.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.101+0000 7ff528000700 1 -- 172.21.15.6:0/4199430352 >> 172.21.15.6:0/4199430352 conn(0x7ff520076210 msgr2=0x7ff520107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:19.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.101+0000 7ff528000700 1 -- 172.21.15.6:0/4199430352 shutdown_connections 2024-04-26T01:42:19.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.101+0000 7ff528000700 1 -- 172.21.15.6:0/4199430352 wait complete. 2024-04-26T01:42:19.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.102+0000 7ff528000700 1 Processor -- start 2024-04-26T01:42:19.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.103+0000 7ff528000700 1 -- start start 2024-04-26T01:42:19.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.103+0000 7ff528000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5201092d0 0x7ff52019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:19.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.103+0000 7ff528000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff520109c80 0x7ff52019fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:19.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.103+0000 7ff528000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff52010e3b0 0x7ff5201a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:19.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.103+0000 7ff528000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff520119260 con 0x7ff52010e3b0 2024-04-26T01:42:19.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.103+0000 7ff528000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff5201190e0 con 0x7ff5201092d0 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.103+0000 7ff528000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff5201193e0 con 0x7ff520109c80 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.103+0000 7ff525600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5201092d0 0x7ff52019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.103+0000 7ff526000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff52010e3b0 0x7ff5201a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.103+0000 7ff525600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5201092d0 0x7ff52019f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59246/0 (socket says 172.21.15.6:59246) 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.103+0000 7ff525600700 1 -- 172.21.15.6:0/2439843167 learned_addr learned my addr 172.21.15.6:0/2439843167 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.104+0000 7ff524c00700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff520109c80 0x7ff52019fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.104+0000 7ff525600700 1 -- 172.21.15.6:0/2439843167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff520109c80 msgr2=0x7ff52019fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.104+0000 7ff525600700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff520109c80 0x7ff52019fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.104+0000 7ff525600700 1 -- 172.21.15.6:0/2439843167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff52010e3b0 msgr2=0x7ff5201a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.104+0000 7ff525600700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff52010e3b0 0x7ff5201a4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.104+0000 7ff525600700 1 -- 172.21.15.6:0/2439843167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5201a4740 con 0x7ff5201092d0 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.104+0000 7ff525600700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5201092d0 0x7ff52019f730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff514006010 tx=0x7ff514005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:19.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.104+0000 7ff51e000700 1 -- 172.21.15.6:0/2439843167 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff514026070 con 0x7ff5201092d0 2024-04-26T01:42:19.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.104+0000 7ff528000700 1 -- 172.21.15.6:0/2439843167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5201a49d0 con 0x7ff5201092d0 2024-04-26T01:42:19.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.104+0000 7ff51e000700 1 -- 172.21.15.6:0/2439843167 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff514005680 con 0x7ff5201092d0 2024-04-26T01:42:19.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.104+0000 7ff51e000700 1 -- 172.21.15.6:0/2439843167 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51400fb20 con 0x7ff5201092d0 2024-04-26T01:42:19.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.105+0000 7ff528000700 1 -- 172.21.15.6:0/2439843167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5201ac2b0 con 0x7ff5201092d0 2024-04-26T01:42:19.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.106+0000 7ff528000700 1 -- 172.21.15.6:0/2439843167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff520069730 con 0x7ff5201092d0 2024-04-26T01:42:19.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.108+0000 7ff51e000700 1 -- 172.21.15.6:0/2439843167 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7ff514029070 con 0x7ff5201092d0 2024-04-26T01:42:19.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.109+0000 7ff51e000700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff5080a3cc0 0x7ff5080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:19.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.109+0000 7ff51e000700 1 -- 172.21.15.6:0/2439843167 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7979+0+0 (secure 0 0 0) 0x7ff5140d0d60 con 0x7ff5201092d0 2024-04-26T01:42:19.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.109+0000 7ff524c00700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff5080a3cc0 0x7ff5080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:19.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.110+0000 7ff524c00700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff5080a3cc0 0x7ff5080a6180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff520078230 tx=0x7ff50c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:19.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.112+0000 7ff51e000700 1 -- 172.21.15.6:0/2439843167 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff514013070 con 0x7ff5201092d0 2024-04-26T01:42:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.398+0000 7ff528000700 1 -- 172.21.15.6:0/2439843167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff5201a0a90 con 0x7ff5201092d0 2024-04-26T01:42:19.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.430+0000 7ff51e000700 1 -- 172.21.15.6:0/2439843167 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7ff51409d880 con 0x7ff5201092d0 2024-04-26T01:42:19.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.433+0000 7ff528000700 1 -- 172.21.15.6:0/2439843167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff5080a3cc0 msgr2=0x7ff5080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:19.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.433+0000 7ff528000700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff5080a3cc0 0x7ff5080a6180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff520078230 tx=0x7ff50c009260 comp rx=0 tx=0).stop 2024-04-26T01:42:19.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.433+0000 7ff528000700 1 -- 172.21.15.6:0/2439843167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5201092d0 msgr2=0x7ff52019f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:19.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.433+0000 7ff528000700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5201092d0 0x7ff52019f730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff514006010 tx=0x7ff514005280 comp rx=0 tx=0).stop 2024-04-26T01:42:19.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.433+0000 7ff528000700 1 -- 172.21.15.6:0/2439843167 shutdown_connections 2024-04-26T01:42:19.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.433+0000 7ff528000700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff52010e3b0 0x7ff5201a4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:19.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.434+0000 7ff528000700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5201092d0 0x7ff52019f730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:19.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.434+0000 7ff528000700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff5080a3cc0 0x7ff5080a6180 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:19.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.434+0000 7ff528000700 1 --2- 172.21.15.6:0/2439843167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff520109c80 0x7ff52019fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:19.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.434+0000 7ff528000700 1 -- 172.21.15.6:0/2439843167 >> 172.21.15.6:0/2439843167 conn(0x7ff520076210 msgr2=0x7ff52010cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:19.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.434+0000 7ff528000700 1 -- 172.21.15.6:0/2439843167 shutdown_connections 2024-04-26T01:42:19.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:19.434+0000 7ff528000700 1 -- 172.21.15.6:0/2439843167 wait complete. 2024-04-26T01:42:19.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:19 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2439843167' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:42:19.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:19 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:42:19.768 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:19 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2439843167' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:42:19.768 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:19 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:42:19.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:19 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2439843167' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:42:19.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:19 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:42:20.122 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-26T01:42:20.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-04-26T01:42:20.587 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:20.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:20 smithi029 ceph-mon[32775]: pgmap v337: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T01:42:20.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:20 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T01:42:20.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:20 smithi029 ceph-mon[32775]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:42:20.734 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:20 smithi006 ceph-mon[28419]: pgmap v337: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T01:42:20.734 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:20 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T01:42:20.734 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:20 smithi006 ceph-mon[28419]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:42:20.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:20 smithi188 ceph-mon[32751]: pgmap v337: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T01:42:20.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:20 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T01:42:20.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:20 smithi188 ceph-mon[32751]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.521+0000 7f47bee00700 1 -- 172.21.15.6:0/3934287926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b806a750 msgr2=0x7f47b806ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.521+0000 7f47bee00700 1 --2- 172.21.15.6:0/3934287926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b806a750 0x7f47b806ab30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009990 tx=0x7f47a8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.521+0000 7f47bee00700 1 -- 172.21.15.6:0/3934287926 shutdown_connections 2024-04-26T01:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.521+0000 7f47bee00700 1 --2- 172.21.15.6:0/3934287926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b810f4f0 0x7f47b81118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.521+0000 7f47bee00700 1 --2- 172.21.15.6:0/3934287926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b806a750 0x7f47b806ab30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.521+0000 7f47bee00700 1 --2- 172.21.15.6:0/3934287926 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47b806b070 0x7f47b810efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.521+0000 7f47bee00700 1 -- 172.21.15.6:0/3934287926 >> 172.21.15.6:0/3934287926 conn(0x7f47b80788f0 msgr2=0x7f47b8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.521+0000 7f47bee00700 1 -- 172.21.15.6:0/3934287926 shutdown_connections 2024-04-26T01:42:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.521+0000 7f47bee00700 1 -- 172.21.15.6:0/3934287926 wait complete. 2024-04-26T01:42:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.522+0000 7f47bee00700 1 Processor -- start 2024-04-26T01:42:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.522+0000 7f47bee00700 1 -- start start 2024-04-26T01:42:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.523+0000 7f47bee00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b806a750 0x7f47b819b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.523+0000 7f47bee00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47b806b070 0x7f47b819b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.523+0000 7f47bee00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b810f4f0 0x7f47b819fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.523+0000 7f47bee00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f47b8114e50 con 0x7f47b806b070 2024-04-26T01:42:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.523+0000 7f47bee00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f47b8114cd0 con 0x7f47b806a750 2024-04-26T01:42:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.523+0000 7f47bee00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f47b8114fd0 con 0x7f47b810f4f0 2024-04-26T01:42:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.523+0000 7f47b7e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b806a750 0x7f47b819b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.523+0000 7f47b7e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b806a750 0x7f47b819b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59264/0 (socket says 172.21.15.6:59264) 2024-04-26T01:42:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.523+0000 7f47b7e00700 1 -- 172.21.15.6:0/4007194142 learned_addr learned my addr 172.21.15.6:0/4007194142 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.523+0000 7f47bce00700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b810f4f0 0x7f47b819fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.523+0000 7f47b7400700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47b806b070 0x7f47b819b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.524+0000 7f47b7e00700 1 -- 172.21.15.6:0/4007194142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b810f4f0 msgr2=0x7f47b819fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.524+0000 7f47b7e00700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b810f4f0 0x7f47b819fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.524+0000 7f47b7e00700 1 -- 172.21.15.6:0/4007194142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47b806b070 msgr2=0x7f47b819b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.524+0000 7f47b7e00700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47b806b070 0x7f47b819b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.524+0000 7f47b7e00700 1 -- 172.21.15.6:0/4007194142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b81a0360 con 0x7f47b806a750 2024-04-26T01:42:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.524+0000 7f47b7400700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47b806b070 0x7f47b819b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:42:22.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.524+0000 7f47b7e00700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b806a750 0x7f47b819b3a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f47a80057c0 tx=0x7f47a80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:22.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.524+0000 7f47b4c00700 1 -- 172.21.15.6:0/4007194142 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a800fad0 con 0x7f47b806a750 2024-04-26T01:42:22.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.524+0000 7f47bee00700 1 -- 172.21.15.6:0/4007194142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b81a05f0 con 0x7f47b806a750 2024-04-26T01:42:22.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.525+0000 7f47b4c00700 1 -- 172.21.15.6:0/4007194142 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47a800fc70 con 0x7f47b806a750 2024-04-26T01:42:22.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.525+0000 7f47b4c00700 1 -- 172.21.15.6:0/4007194142 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8017650 con 0x7f47b806a750 2024-04-26T01:42:22.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.525+0000 7f47bee00700 1 -- 172.21.15.6:0/4007194142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b81a7ed0 con 0x7f47b806a750 2024-04-26T01:42:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.526+0000 7f47bee00700 1 -- 172.21.15.6:0/4007194142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47b8101f60 con 0x7f47b806a750 2024-04-26T01:42:22.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.528+0000 7f47b4c00700 1 -- 172.21.15.6:0/4007194142 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f47a8009d10 con 0x7f47b806a750 2024-04-26T01:42:22.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.529+0000 7f47b4c00700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f479c0a3cc0 0x7f479c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:22.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.530+0000 7f47b4c00700 1 -- 172.21.15.6:0/4007194142 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7979+0+0 (secure 0 0 0) 0x7f47a80d1200 con 0x7f47b806a750 2024-04-26T01:42:22.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.530+0000 7f47b7400700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f479c0a3cc0 0x7f479c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:22.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.530+0000 7f47b7400700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f479c0a3cc0 0x7f479c0a6180 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f47b8102f70 tx=0x7f47a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:22.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.534+0000 7f47b4c00700 1 -- 172.21.15.6:0/4007194142 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f47a8013070 con 0x7f47b806a750 2024-04-26T01:42:22.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:22 smithi029 ceph-mon[32775]: pgmap v338: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2024-04-26T01:42:22.713 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:22 smithi006 ceph-mon[28419]: pgmap v338: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2024-04-26T01:42:22.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:22.815+0000 7f47bee00700 1 -- 172.21.15.6:0/4007194142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f47b802df80 con 0x7f47b806a750 2024-04-26T01:42:22.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:22 smithi188 ceph-mon[32751]: pgmap v338: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2024-04-26T01:42:23.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.451+0000 7f47b4c00700 1 -- 172.21.15.6:0/4007194142 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f47a809dd20 con 0x7f47b806a750 2024-04-26T01:42:23.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.457+0000 7f47bee00700 1 -- 172.21.15.6:0/4007194142 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f479c0a3cc0 msgr2=0x7f479c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:23.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.457+0000 7f47bee00700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f479c0a3cc0 0x7f479c0a6180 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f47b8102f70 tx=0x7f47a4009260 comp rx=0 tx=0).stop 2024-04-26T01:42:23.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.457+0000 7f47bee00700 1 -- 172.21.15.6:0/4007194142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b806a750 msgr2=0x7f47b819b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:23.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.457+0000 7f47bee00700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b806a750 0x7f47b819b3a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f47a80057c0 tx=0x7f47a80057f0 comp rx=0 tx=0).stop 2024-04-26T01:42:23.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.457+0000 7f47bee00700 1 -- 172.21.15.6:0/4007194142 shutdown_connections 2024-04-26T01:42:23.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.457+0000 7f47bee00700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f479c0a3cc0 0x7f479c0a6180 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:23.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.457+0000 7f47bee00700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b810f4f0 0x7f47b819fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:23.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.457+0000 7f47bee00700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b806a750 0x7f47b819b3a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:23.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.457+0000 7f47bee00700 1 --2- 172.21.15.6:0/4007194142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47b806b070 0x7f47b819b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:23.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.457+0000 7f47bee00700 1 -- 172.21.15.6:0/4007194142 >> 172.21.15.6:0/4007194142 conn(0x7f47b80788f0 msgr2=0x7f47b81109e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:23.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.458+0000 7f47bee00700 1 -- 172.21.15.6:0/4007194142 shutdown_connections 2024-04-26T01:42:23.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:23.458+0000 7f47bee00700 1 -- 172.21.15.6:0/4007194142 wait complete. 2024-04-26T01:42:23.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:23 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/4007194142' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:42:23.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:23 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:42:23.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:23 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/4007194142' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:42:23.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:23 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:42:23.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:23 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/4007194142' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:42:23.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:23 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:42:24.244 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-04-26T01:42:24.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:24 smithi006 ceph-mon[28419]: pgmap v339: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T01:42:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:24 smithi006 ceph-mon[28419]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T01:42:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:24 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T01:42:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:24 smithi006 ceph-mon[28419]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T01:42:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:24 smithi006 ceph-mon[28419]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:42:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:24 smithi006 ceph-mon[28419]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-04-26T01:42:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:24 smithi006 ceph-mon[28419]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-26T01:42:24.689 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:24 smithi029 ceph-mon[32775]: pgmap v339: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T01:42:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:24 smithi029 ceph-mon[32775]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T01:42:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:24 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T01:42:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:24 smithi029 ceph-mon[32775]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T01:42:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:24 smithi029 ceph-mon[32775]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:42:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:24 smithi029 ceph-mon[32775]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-04-26T01:42:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:24 smithi029 ceph-mon[32775]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-26T01:42:24.706 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:24.790 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:24 smithi188 ceph-mon[32751]: pgmap v339: 97 pgs: 97 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T01:42:24.790 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:24 smithi188 ceph-mon[32751]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T01:42:24.791 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:24 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T01:42:24.791 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:24 smithi188 ceph-mon[32751]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T01:42:24.791 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:24 smithi188 ceph-mon[32751]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-26T01:42:24.791 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:24 smithi188 ceph-mon[32751]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-04-26T01:42:24.791 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:24 smithi188 ceph-mon[32751]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-26T01:42:25.639 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 01:42:25 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i[64227]: 2024-04-26T01:42:25.248+0000 7f1cff6dd700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T01:42:25.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:25 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] up:active 2024-04-26T01:42:25.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:25 smithi188 ceph-mon[32751]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T01:42:25.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:25 smithi188 ceph-mon[32751]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T01:42:25.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:25 smithi188 ceph-mon[32751]: Cluster is now healthy 2024-04-26T01:42:25.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:25 smithi188 ceph-mon[32751]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-04-26T01:42:25.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:25 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-04-26T01:42:25.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:25 smithi188 ceph-mon[32751]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-26T01:42:25.745 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:25 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] up:active 2024-04-26T01:42:25.745 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:25 smithi006 ceph-mon[28419]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T01:42:25.745 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:25 smithi006 ceph-mon[28419]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T01:42:25.745 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:25 smithi006 ceph-mon[28419]: Cluster is now healthy 2024-04-26T01:42:25.745 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:25 smithi006 ceph-mon[28419]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-04-26T01:42:25.745 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:25 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-04-26T01:42:25.745 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:25 smithi006 ceph-mon[28419]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-26T01:42:25.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:25 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] up:active 2024-04-26T01:42:25.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:25 smithi029 ceph-mon[32775]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T01:42:25.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:25 smithi029 ceph-mon[32775]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T01:42:25.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:25 smithi029 ceph-mon[32775]: Cluster is now healthy 2024-04-26T01:42:25.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:25 smithi029 ceph-mon[32775]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-04-26T01:42:25.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:25 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-04-26T01:42:25.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:25 smithi029 ceph-mon[32775]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-26T01:42:26.688 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:26 smithi006 ceph-mon[28419]: pgmap v340: 97 pgs: 97 active+clean; 580 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-04-26T01:42:26.688 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:26 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] up:active 2024-04-26T01:42:26.688 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:26 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-26T01:42:26.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.869+0000 7f5691600700 1 -- 172.21.15.6:0/227609779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f568c108f80 msgr2=0x7f568c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:26.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.869+0000 7f5691600700 1 --2- 172.21.15.6:0/227609779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f568c108f80 0x7f568c109400 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5674009990 tx=0x7f5674009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:26.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.869+0000 7f5691600700 1 -- 172.21.15.6:0/227609779 shutdown_connections 2024-04-26T01:42:26.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.869+0000 7f5691600700 1 --2- 172.21.15.6:0/227609779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f568c109940 0x7f568c115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:26.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.869+0000 7f5691600700 1 --2- 172.21.15.6:0/227609779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f568c10f030 0x7f568c10f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:26.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.869+0000 7f5691600700 1 --2- 172.21.15.6:0/227609779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f568c108f80 0x7f568c109400 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:26.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.869+0000 7f5691600700 1 -- 172.21.15.6:0/227609779 >> 172.21.15.6:0/227609779 conn(0x7f568c0787d0 msgr2=0x7f568c107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:26.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.869+0000 7f5691600700 1 -- 172.21.15.6:0/227609779 shutdown_connections 2024-04-26T01:42:26.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.869+0000 7f5691600700 1 -- 172.21.15.6:0/227609779 wait complete. 2024-04-26T01:42:26.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.870+0000 7f5691600700 1 Processor -- start 2024-04-26T01:42:26.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.870+0000 7f5691600700 1 -- start start 2024-04-26T01:42:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.871+0000 7f5691600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f568c108f80 0x7f568c19f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.871+0000 7f5691600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f568c109940 0x7f568c19fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.871+0000 7f5691600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f568c10f030 0x7f568c1a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.871+0000 7f5691600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f568c119120 con 0x7f568c10f030 2024-04-26T01:42:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.871+0000 7f5691600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f568c118fa0 con 0x7f568c109940 2024-04-26T01:42:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.871+0000 7f5691600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f568c1192a0 con 0x7f568c108f80 2024-04-26T01:42:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.871+0000 7f5683400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f568c109940 0x7f568c19fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.871+0000 7f5683400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f568c109940 0x7f568c19fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59296/0 (socket says 172.21.15.6:59296) 2024-04-26T01:42:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.871+0000 7f5683400700 1 -- 172.21.15.6:0/1375312932 learned_addr learned my addr 172.21.15.6:0/1375312932 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.871+0000 7f5690c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f568c10f030 0x7f568c1a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.871+0000 7f568be00700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f568c108f80 0x7f568c19f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:26.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.872+0000 7f5683400700 1 -- 172.21.15.6:0/1375312932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f568c108f80 msgr2=0x7f568c19f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:26.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.872+0000 7f5683400700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f568c108f80 0x7f568c19f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.872+0000 7f5683400700 1 -- 172.21.15.6:0/1375312932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f568c10f030 msgr2=0x7f568c1a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.872+0000 7f5683400700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f568c10f030 0x7f568c1a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.872+0000 7f5683400700 1 -- 172.21.15.6:0/1375312932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568c1a4580 con 0x7f568c109940 2024-04-26T01:42:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.872+0000 7f5683400700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f568c109940 0x7f568c19fab0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5674000c00 tx=0x7f567400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.873+0000 7f5689600700 1 -- 172.21.15.6:0/1375312932 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5674027070 con 0x7f568c109940 2024-04-26T01:42:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.873+0000 7f5689600700 1 -- 172.21.15.6:0/1375312932 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5674022470 con 0x7f568c109940 2024-04-26T01:42:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.873+0000 7f5689600700 1 -- 172.21.15.6:0/1375312932 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5674017650 con 0x7f568c109940 2024-04-26T01:42:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.873+0000 7f5691600700 1 -- 172.21.15.6:0/1375312932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f568c1a4810 con 0x7f568c109940 2024-04-26T01:42:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.873+0000 7f5691600700 1 -- 172.21.15.6:0/1375312932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f568c1ac0f0 con 0x7f568c109940 2024-04-26T01:42:26.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.874+0000 7f5691600700 1 -- 172.21.15.6:0/1375312932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f568c10a670 con 0x7f568c109940 2024-04-26T01:42:26.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.876+0000 7f5689600700 1 -- 172.21.15.6:0/1375312932 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f567402a070 con 0x7f568c109940 2024-04-26T01:42:26.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.877+0000 7f5689600700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f56700a3cc0 0x7f56700a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:26.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.878+0000 7f5689600700 1 -- 172.21.15.6:0/1375312932 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7979+0+0 (secure 0 0 0) 0x7f56740d1180 con 0x7f568c109940 2024-04-26T01:42:26.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.878+0000 7f568be00700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f56700a3cc0 0x7f56700a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:26.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.878+0000 7f568be00700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f56700a3cc0 0x7f56700a6180 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f567c009da0 tx=0x7f567c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:26.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:26.880+0000 7f5689600700 1 -- 172.21.15.6:0/1375312932 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5674013070 con 0x7f568c109940 2024-04-26T01:42:26.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:26 smithi188 ceph-mon[32751]: pgmap v340: 97 pgs: 97 active+clean; 580 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-04-26T01:42:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:26 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] up:active 2024-04-26T01:42:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:26 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-26T01:42:26.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:26 smithi029 ceph-mon[32775]: pgmap v340: 97 pgs: 97 active+clean; 580 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-04-26T01:42:26.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:26 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] up:active 2024-04-26T01:42:26.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:26 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-26T01:42:27.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.165+0000 7f5691600700 1 -- 172.21.15.6:0/1375312932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f568c02df80 con 0x7f568c109940 2024-04-26T01:42:27.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.494+0000 7f5689600700 1 -- 172.21.15.6:0/1375312932 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f567409dca0 con 0x7f568c109940 2024-04-26T01:42:27.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.499+0000 7f5691600700 1 -- 172.21.15.6:0/1375312932 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f56700a3cc0 msgr2=0x7f56700a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:27.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.499+0000 7f5691600700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f56700a3cc0 0x7f56700a6180 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f567c009da0 tx=0x7f567c00b040 comp rx=0 tx=0).stop 2024-04-26T01:42:27.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.499+0000 7f5691600700 1 -- 172.21.15.6:0/1375312932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f568c109940 msgr2=0x7f568c19fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:27.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.499+0000 7f5691600700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f568c109940 0x7f568c19fab0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5674000c00 tx=0x7f567400bfd0 comp rx=0 tx=0).stop 2024-04-26T01:42:27.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.499+0000 7f5691600700 1 -- 172.21.15.6:0/1375312932 shutdown_connections 2024-04-26T01:42:27.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.499+0000 7f5691600700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f568c10f030 0x7f568c1a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:27.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.499+0000 7f5691600700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f56700a3cc0 0x7f56700a6180 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:27.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.499+0000 7f5691600700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f568c108f80 0x7f568c19f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:27.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.499+0000 7f5691600700 1 --2- 172.21.15.6:0/1375312932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f568c109940 0x7f568c19fab0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:27.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.499+0000 7f5691600700 1 -- 172.21.15.6:0/1375312932 >> 172.21.15.6:0/1375312932 conn(0x7f568c0787d0 msgr2=0x7f568c079a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:27.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.499+0000 7f5691600700 1 -- 172.21.15.6:0/1375312932 shutdown_connections 2024-04-26T01:42:27.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:27.500+0000 7f5691600700 1 -- 172.21.15.6:0/1375312932 wait complete. 2024-04-26T01:42:27.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:27 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1375312932' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:42:27.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:27 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:42:27.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:27 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1375312932' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:42:27.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:27 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:42:27.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:27 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1375312932' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:42:27.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:27 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:42:28.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-04-26T01:42:28.744 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:28.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:28 smithi006 ceph-mon[28419]: pgmap v341: 97 pgs: 97 active+clean; 580 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-26T01:42:28.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:28 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T01:42:28.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:28 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-26T01:42:28.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:28 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:42:28.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:28 smithi188 ceph-mon[32751]: pgmap v341: 97 pgs: 97 active+clean; 580 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-26T01:42:28.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:28 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T01:42:28.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:28 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-26T01:42:28.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:28 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:42:28.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:28 smithi029 ceph-mon[32775]: pgmap v341: 97 pgs: 97 active+clean; 580 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-26T01:42:28.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:28 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T01:42:28.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:28 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-26T01:42:28.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:28 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:42:30.677 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:30 smithi006 ceph-mon[28419]: pgmap v342: 97 pgs: 97 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-26T01:42:30.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:30 smithi188 ceph-mon[32751]: pgmap v342: 97 pgs: 97 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-26T01:42:30.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:30 smithi029 ceph-mon[32775]: pgmap v342: 97 pgs: 97 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-26T01:42:30.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.001+0000 7f3577200700 1 -- 172.21.15.6:0/22650467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3578105780 msgr2=0x7f35781097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:30.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.001+0000 7f3577200700 1 --2- 172.21.15.6:0/22650467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3578105780 0x7f35781097d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f35640099c0 tx=0x7f3564009cd0 comp rx=0 tx=0).stop 2024-04-26T01:42:30.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.001+0000 7f3577200700 1 -- 172.21.15.6:0/22650467 shutdown_connections 2024-04-26T01:42:30.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.001+0000 7f3577200700 1 --2- 172.21.15.6:0/22650467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3578109eb0 0x7f3578111a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:30.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.001+0000 7f3577200700 1 --2- 172.21.15.6:0/22650467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3578104dd0 0x7f35781051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:30.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.001+0000 7f3577200700 1 --2- 172.21.15.6:0/22650467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3578105780 0x7f35781097d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:30.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.001+0000 7f3577200700 1 -- 172.21.15.6:0/22650467 >> 172.21.15.6:0/22650467 conn(0x7f3578100660 msgr2=0x7f3578102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:30.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.001+0000 7f3577200700 1 -- 172.21.15.6:0/22650467 shutdown_connections 2024-04-26T01:42:30.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.001+0000 7f3577200700 1 -- 172.21.15.6:0/22650467 wait complete. 2024-04-26T01:42:30.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.002+0000 7f3577200700 1 Processor -- start 2024-04-26T01:42:30.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.002+0000 7f3577200700 1 -- start start 2024-04-26T01:42:31.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3577200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3578104dd0 0x7f357819b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:31.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3577200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3578105780 0x7f357819b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:31.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3577200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3578109eb0 0x7f357819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:31.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3577200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3578114d30 con 0x7f3578105780 2024-04-26T01:42:31.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3577200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3578114bb0 con 0x7f3578104dd0 2024-04-26T01:42:31.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3577200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3578114eb0 con 0x7f3578109eb0 2024-04-26T01:42:31.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3575400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3578105780 0x7f357819b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:31.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3576800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3578109eb0 0x7f357819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:31.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3575400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3578105780 0x7f357819b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49176/0 (socket says 172.21.15.6:49176) 2024-04-26T01:42:31.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3575400700 1 -- 172.21.15.6:0/1797290178 learned_addr learned my addr 172.21.15.6:0/1797290178 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:31.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3575e00700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3578104dd0 0x7f357819b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:31.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3576800700 1 -- 172.21.15.6:0/1797290178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3578104dd0 msgr2=0x7f357819b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:31.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3576800700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3578104dd0 0x7f357819b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:31.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3576800700 1 -- 172.21.15.6:0/1797290178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3578105780 msgr2=0x7f357819b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:31.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3576800700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3578105780 0x7f357819b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:31.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.003+0000 7f3576800700 1 -- 172.21.15.6:0/1797290178 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35781a0220 con 0x7f3578109eb0 2024-04-26T01:42:31.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.004+0000 7f3576800700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3578109eb0 0x7f357819fae0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f357000efc0 tx=0x7f357000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:31.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.004+0000 7f356ea00700 1 -- 172.21.15.6:0/1797290178 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35700092e0 con 0x7f3578109eb0 2024-04-26T01:42:31.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.004+0000 7f356ea00700 1 -- 172.21.15.6:0/1797290178 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3570004bb0 con 0x7f3578109eb0 2024-04-26T01:42:31.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.004+0000 7f356ea00700 1 -- 172.21.15.6:0/1797290178 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570004020 con 0x7f3578109eb0 2024-04-26T01:42:31.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.004+0000 7f3577200700 1 -- 172.21.15.6:0/1797290178 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35781a04b0 con 0x7f3578109eb0 2024-04-26T01:42:31.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.004+0000 7f3577200700 1 -- 172.21.15.6:0/1797290178 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35781a7d90 con 0x7f3578109eb0 2024-04-26T01:42:31.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.005+0000 7f3577200700 1 -- 172.21.15.6:0/1797290178 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3578069730 con 0x7f3578109eb0 2024-04-26T01:42:31.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.008+0000 7f356ea00700 1 -- 172.21.15.6:0/1797290178 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f357001b030 con 0x7f3578109eb0 2024-04-26T01:42:31.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.009+0000 7f356ea00700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f355c0a3cc0 0x7f355c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:31.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.009+0000 7f356ea00700 1 -- 172.21.15.6:0/1797290178 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7979+0+0 (secure 0 0 0) 0x7f35700d0690 con 0x7f3578109eb0 2024-04-26T01:42:31.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.009+0000 7f3575e00700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f355c0a3cc0 0x7f355c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:31.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.010+0000 7f3575e00700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f355c0a3cc0 0x7f355c0a6180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3568007be0 tx=0x7f35680073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:31.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.012+0000 7f356ea00700 1 -- 172.21.15.6:0/1797290178 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3570022020 con 0x7f3578109eb0 2024-04-26T01:42:31.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.292+0000 7f3577200700 1 -- 172.21.15.6:0/1797290178 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f357802df80 con 0x7f3578109eb0 2024-04-26T01:42:31.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.517+0000 7f356ea00700 1 -- 172.21.15.6:0/1797290178 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v25) ==== 115+0+0 (secure 0 0 0) 0x7f3570010040 con 0x7f3578109eb0 2024-04-26T01:42:31.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.521+0000 7f3577200700 1 -- 172.21.15.6:0/1797290178 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f355c0a3cc0 msgr2=0x7f355c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:31.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.521+0000 7f3577200700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f355c0a3cc0 0x7f355c0a6180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3568007be0 tx=0x7f35680073d0 comp rx=0 tx=0).stop 2024-04-26T01:42:31.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.521+0000 7f3577200700 1 -- 172.21.15.6:0/1797290178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3578109eb0 msgr2=0x7f357819fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:31.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.521+0000 7f3577200700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3578109eb0 0x7f357819fae0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f357000efc0 tx=0x7f357000c3a0 comp rx=0 tx=0).stop 2024-04-26T01:42:31.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.521+0000 7f3577200700 1 -- 172.21.15.6:0/1797290178 shutdown_connections 2024-04-26T01:42:31.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.521+0000 7f3577200700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f355c0a3cc0 0x7f355c0a6180 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:31.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.521+0000 7f3577200700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3578109eb0 0x7f357819fae0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:31.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.521+0000 7f3577200700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3578104dd0 0x7f357819b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:31.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.521+0000 7f3577200700 1 --2- 172.21.15.6:0/1797290178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3578105780 0x7f357819b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:31.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.521+0000 7f3577200700 1 -- 172.21.15.6:0/1797290178 >> 172.21.15.6:0/1797290178 conn(0x7f3578100660 msgr2=0x7f3578102980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:31.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.522+0000 7f3577200700 1 -- 172.21.15.6:0/1797290178 shutdown_connections 2024-04-26T01:42:31.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:31.522+0000 7f3577200700 1 -- 172.21.15.6:0/1797290178 wait complete. 2024-04-26T01:42:31.540 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:31 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1797290178' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:42:31.540 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:31 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:42:31.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:31 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1797290178' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:42:31.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:31 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:42:31.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:31 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1797290178' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:42:31.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:31 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:42:32.225 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-26T01:42:32.225 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-04-26T01:42:32.722 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:32.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:32 smithi006 ceph-mon[28419]: pgmap v343: 97 pgs: 97 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-26T01:42:32.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:32 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T01:42:32.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:32 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:42:32.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:32 smithi188 ceph-mon[32751]: pgmap v343: 97 pgs: 97 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-26T01:42:32.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:32 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T01:42:32.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:32 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:42:32.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:32 smithi029 ceph-mon[32775]: pgmap v343: 97 pgs: 97 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-26T01:42:32.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:32 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T01:42:32.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:32 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:42:34.748 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:34 smithi006 ceph-mon[28419]: pgmap v344: 97 pgs: 97 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-26T01:42:34.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:34 smithi188 ceph-mon[32751]: pgmap v344: 97 pgs: 97 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-26T01:42:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.934+0000 7f7aa2a00700 1 -- 172.21.15.6:0/22402140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a9c10f4e0 msgr2=0x7f7a9c1118d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.934+0000 7f7aa2a00700 1 --2- 172.21.15.6:0/22402140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a9c10f4e0 0x7f7a9c1118d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400b0a0 tx=0x7f7a9400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:42:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.934+0000 7f7aa2a00700 1 -- 172.21.15.6:0/22402140 shutdown_connections 2024-04-26T01:42:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.934+0000 7f7aa2a00700 1 --2- 172.21.15.6:0/22402140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a9c10f4e0 0x7f7a9c1118d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.934+0000 7f7aa2a00700 1 --2- 172.21.15.6:0/22402140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a9c06afe0 0x7f7a9c06b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.934+0000 7f7aa2a00700 1 --2- 172.21.15.6:0/22402140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c06b900 0x7f7a9c10efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.934+0000 7f7aa2a00700 1 -- 172.21.15.6:0/22402140 >> 172.21.15.6:0/22402140 conn(0x7f7a9c0788b0 msgr2=0x7f7a9c078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.934+0000 7f7aa2a00700 1 -- 172.21.15.6:0/22402140 shutdown_connections 2024-04-26T01:42:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.934+0000 7f7aa2a00700 1 -- 172.21.15.6:0/22402140 wait complete. 2024-04-26T01:42:34.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.935+0000 7f7aa2a00700 1 Processor -- start 2024-04-26T01:42:34.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.935+0000 7f7aa2a00700 1 -- start start 2024-04-26T01:42:34.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7aa2a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c06afe0 0x7f7a9c19b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:34.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7aa2a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a9c06b900 0x7f7a9c19b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:34.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7aa2a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a9c10f4e0 0x7f7a9c19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:34.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7aa2a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c114e30 con 0x7f7a9c06afe0 2024-04-26T01:42:34.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7aa2a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c114cb0 con 0x7f7a9c06b900 2024-04-26T01:42:34.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7aa2a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c114fb0 con 0x7f7a9c10f4e0 2024-04-26T01:42:34.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7a9be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c06afe0 0x7f7a9c19b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:34.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7aa0a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a9c10f4e0 0x7f7a9c19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:34.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7a9be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c06afe0 0x7f7a9c19b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49192/0 (socket says 172.21.15.6:49192) 2024-04-26T01:42:34.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7a9be00700 1 -- 172.21.15.6:0/3080633226 learned_addr learned my addr 172.21.15.6:0/3080633226 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:34.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7a9b400700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a9c06b900 0x7f7a9c19b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:34.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7aa0a00700 1 -- 172.21.15.6:0/3080633226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a9c06b900 msgr2=0x7f7a9c19b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:34.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7aa0a00700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a9c06b900 0x7f7a9c19b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:34.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7aa0a00700 1 -- 172.21.15.6:0/3080633226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c06afe0 msgr2=0x7f7a9c19b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:34.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7aa0a00700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c06afe0 0x7f7a9c19b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:34.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.936+0000 7f7aa0a00700 1 -- 172.21.15.6:0/3080633226 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a9c1a0350 con 0x7f7a9c10f4e0 2024-04-26T01:42:34.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.937+0000 7f7aa0a00700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a9c10f4e0 0x7f7a9c19fc10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400b070 tx=0x7f7a94004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:34.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.937+0000 7f7a98c00700 1 -- 172.21.15.6:0/3080633226 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94021070 con 0x7f7a9c10f4e0 2024-04-26T01:42:34.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.937+0000 7f7a98c00700 1 -- 172.21.15.6:0/3080633226 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a94007e30 con 0x7f7a9c10f4e0 2024-04-26T01:42:34.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.937+0000 7f7a98c00700 1 -- 172.21.15.6:0/3080633226 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a940045f0 con 0x7f7a9c10f4e0 2024-04-26T01:42:34.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.937+0000 7f7aa2a00700 1 -- 172.21.15.6:0/3080633226 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a9c1a05e0 con 0x7f7a9c10f4e0 2024-04-26T01:42:34.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.937+0000 7f7aa2a00700 1 -- 172.21.15.6:0/3080633226 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a9c1a7ec0 con 0x7f7a9c10f4e0 2024-04-26T01:42:34.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.938+0000 7f7aa2a00700 1 -- 172.21.15.6:0/3080633226 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a9c101f50 con 0x7f7a9c10f4e0 2024-04-26T01:42:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.941+0000 7f7a98c00700 1 -- 172.21.15.6:0/3080633226 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f7a94007980 con 0x7f7a9c10f4e0 2024-04-26T01:42:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.942+0000 7f7a98c00700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7a800a3cc0 0x7f7a800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.942+0000 7f7a98c00700 1 -- 172.21.15.6:0/3080633226 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7979+0+0 (secure 0 0 0) 0x7f7a9400f020 con 0x7f7a9c10f4e0 2024-04-26T01:42:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.942+0000 7f7a9be00700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7a800a3cc0 0x7f7a800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.943+0000 7f7a9be00700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7a800a3cc0 0x7f7a800a6180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c009d60 tx=0x7f7a8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:34.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:34 smithi029 ceph-mon[32775]: pgmap v344: 97 pgs: 97 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-26T01:42:34.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:34.944+0000 7f7a98c00700 1 -- 172.21.15.6:0/3080633226 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7a9402a020 con 0x7f7a9c10f4e0 2024-04-26T01:42:35.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.209+0000 7f7aa2a00700 1 -- 172.21.15.6:0/3080633226 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f7a9c062e80 con 0x7f7a800a3cc0 2024-04-26T01:42:35.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.247+0000 7f7a98c00700 1 -- 172.21.15.6:0/3080633226 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7a9c062e80 con 0x7f7a800a3cc0 2024-04-26T01:42:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.250+0000 7f7aa2a00700 1 -- 172.21.15.6:0/3080633226 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7a800a3cc0 msgr2=0x7f7a800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.250+0000 7f7aa2a00700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7a800a3cc0 0x7f7a800a6180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c009d60 tx=0x7f7a8c009260 comp rx=0 tx=0).stop 2024-04-26T01:42:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.250+0000 7f7aa2a00700 1 -- 172.21.15.6:0/3080633226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a9c10f4e0 msgr2=0x7f7a9c19fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.250+0000 7f7aa2a00700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a9c10f4e0 0x7f7a9c19fc10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400b070 tx=0x7f7a94004c80 comp rx=0 tx=0).stop 2024-04-26T01:42:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.250+0000 7f7aa2a00700 1 -- 172.21.15.6:0/3080633226 shutdown_connections 2024-04-26T01:42:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.250+0000 7f7aa2a00700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7a800a3cc0 0x7f7a800a6180 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.250+0000 7f7aa2a00700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a9c10f4e0 0x7f7a9c19fc10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.250+0000 7f7aa2a00700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a9c06afe0 0x7f7a9c19b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.250+0000 7f7aa2a00700 1 --2- 172.21.15.6:0/3080633226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a9c06b900 0x7f7a9c19b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.250+0000 7f7aa2a00700 1 -- 172.21.15.6:0/3080633226 >> 172.21.15.6:0/3080633226 conn(0x7f7a9c0788b0 msgr2=0x7f7a9c110990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.250+0000 7f7aa2a00700 1 -- 172.21.15.6:0/3080633226 shutdown_connections 2024-04-26T01:42:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:35.250+0000 7f7aa2a00700 1 -- 172.21.15.6:0/3080633226 wait complete. 2024-04-26T01:42:35.566 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:42:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:42:35.220+0000 7f1ab4a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:42:35.566 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:42:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:42:35.220+0000 7f1ab4a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:42:35.566 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:42:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:42:35.220+0000 7f1ab4a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:42:35.566 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:42:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:42:35.220+0000 7f1ab4a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:42:35.567 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:42:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:42:35.220+0000 7f1ab4a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:42:36.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-26T01:42:36.514 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:36.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:36 smithi006 ceph-mon[28419]: pgmap v345: 97 pgs: 97 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-26T01:42:36.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:36 smithi006 ceph-mon[28419]: from='client.24622 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:36.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:36 smithi006 ceph-mon[28419]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T01:42:36.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:36 smithi188 ceph-mon[32751]: pgmap v345: 97 pgs: 97 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-26T01:42:36.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:36 smithi188 ceph-mon[32751]: from='client.24622 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:36.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:36 smithi188 ceph-mon[32751]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T01:42:36.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:36 smithi029 ceph-mon[32775]: pgmap v345: 97 pgs: 97 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-26T01:42:36.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:36 smithi029 ceph-mon[32775]: from='client.24622 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:36.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:36 smithi029 ceph-mon[32775]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T01:42:38.707 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:38 smithi006 ceph-mon[28419]: pgmap v346: 97 pgs: 97 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-26T01:42:38.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:38 smithi188 ceph-mon[32751]: pgmap v346: 97 pgs: 97 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-26T01:42:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.893+0000 7fdb59a00700 1 -- 172.21.15.6:0/262633663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb54077890 msgr2=0x7fdb541132b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.893+0000 7fdb59a00700 1 --2- 172.21.15.6:0/262633663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb54077890 0x7fdb541132b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009990 tx=0x7fdb40009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.893+0000 7fdb59a00700 1 -- 172.21.15.6:0/262633663 shutdown_connections 2024-04-26T01:42:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.893+0000 7fdb59a00700 1 --2- 172.21.15.6:0/262633663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb541137f0 0x7fdb54115be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.893+0000 7fdb59a00700 1 --2- 172.21.15.6:0/262633663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb54076f70 0x7fdb54077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.893+0000 7fdb59a00700 1 --2- 172.21.15.6:0/262633663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb54077890 0x7fdb541132b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.893+0000 7fdb59a00700 1 -- 172.21.15.6:0/262633663 >> 172.21.15.6:0/262633663 conn(0x7fdb54100680 msgr2=0x7fdb54102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.893+0000 7fdb59a00700 1 -- 172.21.15.6:0/262633663 shutdown_connections 2024-04-26T01:42:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.893+0000 7fdb59a00700 1 -- 172.21.15.6:0/262633663 wait complete. 2024-04-26T01:42:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.894+0000 7fdb59a00700 1 Processor -- start 2024-04-26T01:42:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.894+0000 7fdb59a00700 1 -- start start 2024-04-26T01:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.895+0000 7fdb59a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb54076f70 0x7fdb5419f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.895+0000 7fdb59a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb54077890 0x7fdb5419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.895+0000 7fdb59a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb541137f0 0x7fdb541a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.895+0000 7fdb59a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdb541191a0 con 0x7fdb541137f0 2024-04-26T01:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.895+0000 7fdb59a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdb54119020 con 0x7fdb54076f70 2024-04-26T01:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.895+0000 7fdb59a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdb54119320 con 0x7fdb54077890 2024-04-26T01:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.895+0000 7fdb53e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb54076f70 0x7fdb5419f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.895+0000 7fdb59000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb541137f0 0x7fdb541a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.895+0000 7fdb53e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb54076f70 0x7fdb5419f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36556/0 (socket says 172.21.15.6:36556) 2024-04-26T01:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.895+0000 7fdb53400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb54077890 0x7fdb5419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.895+0000 7fdb53e00700 1 -- 172.21.15.6:0/84108639 learned_addr learned my addr 172.21.15.6:0/84108639 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.895+0000 7fdb59000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb541137f0 0x7fdb541a3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44182/0 (socket says 172.21.15.6:44182) 2024-04-26T01:42:38.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.896+0000 7fdb53e00700 1 -- 172.21.15.6:0/84108639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb54077890 msgr2=0x7fdb5419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:38.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.896+0000 7fdb53e00700 1 --2- 172.21.15.6:0/84108639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb54077890 0x7fdb5419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.896+0000 7fdb53e00700 1 -- 172.21.15.6:0/84108639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb541137f0 msgr2=0x7fdb541a3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:38.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.896+0000 7fdb53e00700 1 --2- 172.21.15.6:0/84108639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb541137f0 0x7fdb541a3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.896+0000 7fdb53e00700 1 -- 172.21.15.6:0/84108639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb541a4660 con 0x7fdb54076f70 2024-04-26T01:42:38.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.896+0000 7fdb53e00700 1 --2- 172.21.15.6:0/84108639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb54076f70 0x7fdb5419f650 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400d980 tx=0x7fdb4400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:38.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.896+0000 7fdb50c00700 1 -- 172.21.15.6:0/84108639 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44017070 con 0x7fdb54076f70 2024-04-26T01:42:38.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.896+0000 7fdb50c00700 1 -- 172.21.15.6:0/84108639 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb44005020 con 0x7fdb54076f70 2024-04-26T01:42:38.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.896+0000 7fdb50c00700 1 -- 172.21.15.6:0/84108639 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4400fa10 con 0x7fdb54076f70 2024-04-26T01:42:38.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.896+0000 7fdb59a00700 1 -- 172.21.15.6:0/84108639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb541a4950 con 0x7fdb54076f70 2024-04-26T01:42:38.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.896+0000 7fdb59a00700 1 -- 172.21.15.6:0/84108639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb541ac1d0 con 0x7fdb54076f70 2024-04-26T01:42:38.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.897+0000 7fdb59a00700 1 -- 172.21.15.6:0/84108639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb54069730 con 0x7fdb54076f70 2024-04-26T01:42:38.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.900+0000 7fdb50c00700 1 -- 172.21.15.6:0/84108639 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fdb44004a30 con 0x7fdb54076f70 2024-04-26T01:42:38.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.901+0000 7fdb50c00700 1 --2- 172.21.15.6:0/84108639 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdb380a3cd0 0x7fdb380a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.901+0000 7fdb50c00700 1 -- 172.21.15.6:0/84108639 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7979+0+0 (secure 0 0 0) 0x7fdb440d0380 con 0x7fdb54076f70 2024-04-26T01:42:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.901+0000 7fdb53400700 1 --2- 172.21.15.6:0/84108639 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdb380a3cd0 0x7fdb380a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.902+0000 7fdb53400700 1 --2- 172.21.15.6:0/84108639 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdb380a3cd0 0x7fdb380a6190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000b5c0 tx=0x7fdb40005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:38.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:38.910+0000 7fdb50c00700 1 -- 172.21.15.6:0/84108639 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb44020020 con 0x7fdb54076f70 2024-04-26T01:42:38.939 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:38 smithi029 ceph-mon[32775]: pgmap v346: 97 pgs: 97 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-26T01:42:39.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.185+0000 7fdb59a00700 1 -- 172.21.15.6:0/84108639 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fdb541a09b0 con 0x7fdb380a3cd0 2024-04-26T01:42:39.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.225+0000 7fdb50c00700 1 -- 172.21.15.6:0/84108639 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdb541a09b0 con 0x7fdb380a3cd0 2024-04-26T01:42:39.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.228+0000 7fdb59a00700 1 -- 172.21.15.6:0/84108639 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdb380a3cd0 msgr2=0x7fdb380a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.228+0000 7fdb59a00700 1 --2- 172.21.15.6:0/84108639 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdb380a3cd0 0x7fdb380a6190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000b5c0 tx=0x7fdb40005960 comp rx=0 tx=0).stop 2024-04-26T01:42:39.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.228+0000 7fdb59a00700 1 -- 172.21.15.6:0/84108639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb54076f70 msgr2=0x7fdb5419f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.228+0000 7fdb59a00700 1 --2- 172.21.15.6:0/84108639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb54076f70 0x7fdb5419f650 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400d980 tx=0x7fdb4400dc90 comp rx=0 tx=0).stop 2024-04-26T01:42:39.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.228+0000 7fdb59a00700 1 -- 172.21.15.6:0/84108639 shutdown_connections 2024-04-26T01:42:39.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.228+0000 7fdb59a00700 1 --2- 172.21.15.6:0/84108639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb541137f0 0x7fdb541a3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.228+0000 7fdb59a00700 1 --2- 172.21.15.6:0/84108639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb54076f70 0x7fdb5419f650 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.228+0000 7fdb59a00700 1 --2- 172.21.15.6:0/84108639 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdb380a3cd0 0x7fdb380a6190 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.228+0000 7fdb59a00700 1 --2- 172.21.15.6:0/84108639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb54077890 0x7fdb5419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.228+0000 7fdb59a00700 1 -- 172.21.15.6:0/84108639 >> 172.21.15.6:0/84108639 conn(0x7fdb54100680 msgr2=0x7fdb54102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:39.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.229+0000 7fdb59a00700 1 -- 172.21.15.6:0/84108639 shutdown_connections 2024-04-26T01:42:39.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:39.229+0000 7fdb59a00700 1 -- 172.21.15.6:0/84108639 wait complete. 2024-04-26T01:42:40.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-26T01:42:40.507 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:41.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:40 smithi006 ceph-mon[28419]: pgmap v347: 97 pgs: 97 active+clean; 589 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-26T01:42:41.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:40 smithi006 ceph-mon[28419]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:41.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:40 smithi188 ceph-mon[32751]: pgmap v347: 97 pgs: 97 active+clean; 589 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-26T01:42:41.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:40 smithi188 ceph-mon[32751]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:41.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:40 smithi029 ceph-mon[32775]: pgmap v347: 97 pgs: 97 active+clean; 589 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-26T01:42:41.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:40 smithi029 ceph-mon[32775]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:42.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.671+0000 7f3189a00700 1 -- 172.21.15.6:0/2084397787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3184104ec0 msgr2=0x7f31841052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.671+0000 7f3189a00700 1 --2- 172.21.15.6:0/2084397787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3184104ec0 0x7f31841052a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f31740099c0 tx=0x7f3174009cd0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.672+0000 7f3189a00700 1 -- 172.21.15.6:0/2084397787 shutdown_connections 2024-04-26T01:42:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.672+0000 7f3189a00700 1 --2- 172.21.15.6:0/2084397787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3184109fa0 0x7f3184111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.672+0000 7f3189a00700 1 --2- 172.21.15.6:0/2084397787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3184104ec0 0x7f31841052a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.672+0000 7f3189a00700 1 --2- 172.21.15.6:0/2084397787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3184105870 0x7f31841098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.672+0000 7f3189a00700 1 -- 172.21.15.6:0/2084397787 >> 172.21.15.6:0/2084397787 conn(0x7f3184100750 msgr2=0x7f3184102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.672+0000 7f3189a00700 1 -- 172.21.15.6:0/2084397787 shutdown_connections 2024-04-26T01:42:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.672+0000 7f3189a00700 1 -- 172.21.15.6:0/2084397787 wait complete. 2024-04-26T01:42:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.673+0000 7f3189a00700 1 Processor -- start 2024-04-26T01:42:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.673+0000 7f3189a00700 1 -- start start 2024-04-26T01:42:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.673+0000 7f3189a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3184104ec0 0x7f318419b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.673+0000 7f3189a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3184105870 0x7f318419b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.673+0000 7f3189a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3184109fa0 0x7f318419fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.673+0000 7f3189a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3184114e50 con 0x7f3184104ec0 2024-04-26T01:42:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.673+0000 7f3189a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3184114cd0 con 0x7f3184105870 2024-04-26T01:42:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.673+0000 7f3189a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3184114fd0 con 0x7f3184109fa0 2024-04-26T01:42:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.674+0000 7f3182a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3184104ec0 0x7f318419b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.674+0000 7f3182a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3184104ec0 0x7f318419b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44210/0 (socket says 172.21.15.6:44210) 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.674+0000 7f3182a00700 1 -- 172.21.15.6:0/2447083361 learned_addr learned my addr 172.21.15.6:0/2447083361 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.674+0000 7f3182000700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3184105870 0x7f318419b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.674+0000 7f3183400700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3184109fa0 0x7f318419fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.674+0000 7f3182a00700 1 -- 172.21.15.6:0/2447083361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3184109fa0 msgr2=0x7f318419fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.674+0000 7f3182a00700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3184109fa0 0x7f318419fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.674+0000 7f3182a00700 1 -- 172.21.15.6:0/2447083361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3184105870 msgr2=0x7f318419b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.674+0000 7f3182a00700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3184105870 0x7f318419b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.674+0000 7f3182a00700 1 -- 172.21.15.6:0/2447083361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31841a02f0 con 0x7f3184104ec0 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.674+0000 7f3182000700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3184105870 0x7f318419b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.675+0000 7f3182a00700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3184104ec0 0x7f318419b2e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3174006010 tx=0x7f3174005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.675+0000 7f317b400700 1 -- 172.21.15.6:0/2447083361 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3174026070 con 0x7f3184104ec0 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.675+0000 7f317b400700 1 -- 172.21.15.6:0/2447083361 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3174005680 con 0x7f3184104ec0 2024-04-26T01:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.675+0000 7f3189a00700 1 -- 172.21.15.6:0/2447083361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31841a0580 con 0x7f3184104ec0 2024-04-26T01:42:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.675+0000 7f317b400700 1 -- 172.21.15.6:0/2447083361 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317400fb20 con 0x7f3184104ec0 2024-04-26T01:42:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.675+0000 7f3189a00700 1 -- 172.21.15.6:0/2447083361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31841a7e60 con 0x7f3184104ec0 2024-04-26T01:42:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.677+0000 7f3189a00700 1 -- 172.21.15.6:0/2447083361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3184069730 con 0x7f3184104ec0 2024-04-26T01:42:42.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.679+0000 7f317b400700 1 -- 172.21.15.6:0/2447083361 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f3174029070 con 0x7f3184104ec0 2024-04-26T01:42:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.680+0000 7f317b400700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f31680a3d50 0x7f31680a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.680+0000 7f317b400700 1 -- 172.21.15.6:0/2447083361 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7979+0+0 (secure 0 0 0) 0x7f31740d0b40 con 0x7f3184104ec0 2024-04-26T01:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.680+0000 7f3182000700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f31680a3d50 0x7f31680a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.681+0000 7f3182000700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f31680a3d50 0x7f31680a6210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3184101b70 tx=0x7f3170009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.685+0000 7f317b400700 1 -- 172.21.15.6:0/2447083361 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f317405dff0 con 0x7f3184104ec0 2024-04-26T01:42:42.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.951+0000 7f3189a00700 1 -- 172.21.15.6:0/2447083361 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f318419c640 con 0x7f31680a3d50 2024-04-26T01:42:42.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.962+0000 7f317b400700 1 -- 172.21.15.6:0/2447083361 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f318419c640 con 0x7f31680a3d50 2024-04-26T01:42:42.960 INFO:teuthology.orchestra.run.smithi006.stdout:/volumes/qa/sv_0/7e0c82a1-47fc-4ef8-a747-cd42e83a80ab 2024-04-26T01:42:42.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.965+0000 7f3189a00700 1 -- 172.21.15.6:0/2447083361 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f31680a3d50 msgr2=0x7f31680a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:42.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.965+0000 7f3189a00700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f31680a3d50 0x7f31680a6210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3184101b70 tx=0x7f3170009260 comp rx=0 tx=0).stop 2024-04-26T01:42:42.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.965+0000 7f3189a00700 1 -- 172.21.15.6:0/2447083361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3184104ec0 msgr2=0x7f318419b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:42.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.965+0000 7f3189a00700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3184104ec0 0x7f318419b2e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3174006010 tx=0x7f3174005280 comp rx=0 tx=0).stop 2024-04-26T01:42:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.966+0000 7f3189a00700 1 -- 172.21.15.6:0/2447083361 shutdown_connections 2024-04-26T01:42:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.966+0000 7f3189a00700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f31680a3d50 0x7f31680a6210 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.966+0000 7f3189a00700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3184109fa0 0x7f318419fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.966+0000 7f3189a00700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3184104ec0 0x7f318419b2e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.966+0000 7f3189a00700 1 --2- 172.21.15.6:0/2447083361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3184105870 0x7f318419b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.966+0000 7f3189a00700 1 -- 172.21.15.6:0/2447083361 >> 172.21.15.6:0/2447083361 conn(0x7f3184100750 msgr2=0x7f3184109130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.966+0000 7f3189a00700 1 -- 172.21.15.6:0/2447083361 shutdown_connections 2024-04-26T01:42:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:42.966+0000 7f3189a00700 1 -- 172.21.15.6:0/2447083361 wait complete. 2024-04-26T01:42:43.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:42 smithi006 ceph-mon[28419]: pgmap v348: 97 pgs: 97 active+clean; 589 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-26T01:42:43.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:42 smithi188 ceph-mon[32751]: pgmap v348: 97 pgs: 97 active+clean; 589 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-26T01:42:43.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:42 smithi029 ceph-mon[32775]: pgmap v348: 97 pgs: 97 active+clean; 589 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-26T01:42:43.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-26T01:42:44.246 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:44.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:44 smithi006 ceph-mon[28419]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:44.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:44 smithi188 ceph-mon[32751]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:44.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:44 smithi029 ceph-mon[32775]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:45.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:45 smithi006 ceph-mon[28419]: pgmap v349: 97 pgs: 97 active+clean; 589 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:42:45.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:45 smithi188 ceph-mon[32751]: pgmap v349: 97 pgs: 97 active+clean; 589 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:42:45.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:45 smithi029 ceph-mon[32775]: pgmap v349: 97 pgs: 97 active+clean; 589 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:42:46.114 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:46 smithi006 ceph-mon[28419]: pgmap v350: 97 pgs: 97 active+clean; 607 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-26T01:42:46.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:46 smithi188 ceph-mon[32751]: pgmap v350: 97 pgs: 97 active+clean; 607 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-26T01:42:46.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.411+0000 7fc30b200700 1 -- 172.21.15.6:0/3607034101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc30c104dd0 msgr2=0x7fc30c1051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.411+0000 7fc30b200700 1 --2- 172.21.15.6:0/3607034101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc30c104dd0 0x7fc30c1051b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009990 tx=0x7fc2fc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.411+0000 7fc30b200700 1 -- 172.21.15.6:0/3607034101 shutdown_connections 2024-04-26T01:42:46.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.411+0000 7fc30b200700 1 --2- 172.21.15.6:0/3607034101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc30c109eb0 0x7fc30c111a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.411+0000 7fc30b200700 1 --2- 172.21.15.6:0/3607034101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc30c104dd0 0x7fc30c1051b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.411+0000 7fc30b200700 1 --2- 172.21.15.6:0/3607034101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc30c105780 0x7fc30c1097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.411+0000 7fc30b200700 1 -- 172.21.15.6:0/3607034101 >> 172.21.15.6:0/3607034101 conn(0x7fc30c100660 msgr2=0x7fc30c102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:46.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.411+0000 7fc30b200700 1 -- 172.21.15.6:0/3607034101 shutdown_connections 2024-04-26T01:42:46.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.411+0000 7fc30b200700 1 -- 172.21.15.6:0/3607034101 wait complete. 2024-04-26T01:42:46.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.413+0000 7fc30b200700 1 Processor -- start 2024-04-26T01:42:46.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.413+0000 7fc30b200700 1 -- start start 2024-04-26T01:42:46.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.413+0000 7fc30b200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc30c104dd0 0x7fc30c19b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:46.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.413+0000 7fc30b200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc30c105780 0x7fc30c19b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:46.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.413+0000 7fc30b200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc30c109eb0 0x7fc30c19fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:46.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.413+0000 7fc30b200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c114db0 con 0x7fc30c105780 2024-04-26T01:42:46.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.413+0000 7fc30b200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c114c30 con 0x7fc30c109eb0 2024-04-26T01:42:46.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.414+0000 7fc30b200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c114f30 con 0x7fc30c104dd0 2024-04-26T01:42:46.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.414+0000 7fc309e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc30c104dd0 0x7fc30c19b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:46.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.414+0000 7fc309e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc30c104dd0 0x7fc30c19b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:53276/0 (socket says 172.21.15.6:53276) 2024-04-26T01:42:46.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.414+0000 7fc309e00700 1 -- 172.21.15.6:0/3839187637 learned_addr learned my addr 172.21.15.6:0/3839187637 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:46.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.414+0000 7fc303e00700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc30c105780 0x7fc30c19b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:46.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.414+0000 7fc30a800700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc30c109eb0 0x7fc30c19fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:46.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.414+0000 7fc309e00700 1 -- 172.21.15.6:0/3839187637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc30c109eb0 msgr2=0x7fc30c19fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.414+0000 7fc309e00700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc30c109eb0 0x7fc30c19fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.414+0000 7fc309e00700 1 -- 172.21.15.6:0/3839187637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc30c105780 msgr2=0x7fc30c19b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.414+0000 7fc309e00700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc30c105780 0x7fc30c19b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.414+0000 7fc309e00700 1 -- 172.21.15.6:0/3839187637 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c1a0270 con 0x7fc30c104dd0 2024-04-26T01:42:46.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.414+0000 7fc30a800700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc30c109eb0 0x7fc30c19fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:46.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.415+0000 7fc309e00700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc30c104dd0 0x7fc30c19b260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc000c00 tx=0x7fc2fc0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:46.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.415+0000 7fc302a00700 1 -- 172.21.15.6:0/3839187637 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc027070 con 0x7fc30c104dd0 2024-04-26T01:42:46.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.415+0000 7fc302a00700 1 -- 172.21.15.6:0/3839187637 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2fc00b610 con 0x7fc30c104dd0 2024-04-26T01:42:46.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.415+0000 7fc30b200700 1 -- 172.21.15.6:0/3839187637 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30c1a0500 con 0x7fc30c104dd0 2024-04-26T01:42:46.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.415+0000 7fc302a00700 1 -- 172.21.15.6:0/3839187637 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0173f0 con 0x7fc30c104dd0 2024-04-26T01:42:46.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.415+0000 7fc30b200700 1 -- 172.21.15.6:0/3839187637 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30c1a7de0 con 0x7fc30c104dd0 2024-04-26T01:42:46.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.416+0000 7fc30b200700 1 -- 172.21.15.6:0/3839187637 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc30c1062b0 con 0x7fc30c104dd0 2024-04-26T01:42:46.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.418+0000 7fc302a00700 1 -- 172.21.15.6:0/3839187637 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fc2fc02a070 con 0x7fc30c104dd0 2024-04-26T01:42:46.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.420+0000 7fc302a00700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc2f00a3cd0 0x7fc2f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:46.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.420+0000 7fc302a00700 1 -- 172.21.15.6:0/3839187637 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7979+0+0 (secure 0 0 0) 0x7fc2fc0d1370 con 0x7fc30c104dd0 2024-04-26T01:42:46.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.420+0000 7fc303e00700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc2f00a3cd0 0x7fc2f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:46.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.421+0000 7fc303e00700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc2f00a3cd0 0x7fc2f00a6190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc30c101a80 tx=0x7fc2f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:46.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.424+0000 7fc302a00700 1 -- 172.21.15.6:0/3839187637 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2fc05eff0 con 0x7fc30c104dd0 2024-04-26T01:42:46.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:46 smithi029 ceph-mon[32775]: pgmap v350: 97 pgs: 97 active+clean; 607 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-26T01:42:46.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.699+0000 7fc30b200700 1 -- 172.21.15.6:0/3839187637 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc30c19c5c0 con 0x7fc2f00a3cd0 2024-04-26T01:42:46.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.742+0000 7fc302a00700 1 -- 172.21.15.6:0/3839187637 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc30c19c5c0 con 0x7fc2f00a3cd0 2024-04-26T01:42:46.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.745+0000 7fc30b200700 1 -- 172.21.15.6:0/3839187637 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc2f00a3cd0 msgr2=0x7fc2f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.745+0000 7fc30b200700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc2f00a3cd0 0x7fc2f00a6190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc30c101a80 tx=0x7fc2f4009260 comp rx=0 tx=0).stop 2024-04-26T01:42:46.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.746+0000 7fc30b200700 1 -- 172.21.15.6:0/3839187637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc30c104dd0 msgr2=0x7fc30c19b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.746+0000 7fc30b200700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc30c104dd0 0x7fc30c19b260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc000c00 tx=0x7fc2fc0057c0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.746+0000 7fc30b200700 1 -- 172.21.15.6:0/3839187637 shutdown_connections 2024-04-26T01:42:46.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.746+0000 7fc30b200700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc30c109eb0 0x7fc30c19fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.746+0000 7fc30b200700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc2f00a3cd0 0x7fc2f00a6190 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.746+0000 7fc30b200700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc30c104dd0 0x7fc30c19b260 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.746+0000 7fc30b200700 1 --2- 172.21.15.6:0/3839187637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc30c105780 0x7fc30c19b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.746+0000 7fc30b200700 1 -- 172.21.15.6:0/3839187637 >> 172.21.15.6:0/3839187637 conn(0x7fc30c100660 msgr2=0x7fc30c109040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:46.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.746+0000 7fc30b200700 1 -- 172.21.15.6:0/3839187637 shutdown_connections 2024-04-26T01:42:46.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:46.746+0000 7fc30b200700 1 -- 172.21.15.6:0/3839187637 wait complete. 2024-04-26T01:42:47.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:47 smithi006 ceph-mon[28419]: from='client.24646 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:47.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:47 smithi188 ceph-mon[32751]: from='client.24646 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:47.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:47 smithi029 ceph-mon[32775]: from='client.24646 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:47.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-26T01:42:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:48.292 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-04-26T01:42:48.292 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-26T01:42:48.292 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-26T01:42:48.292 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-26T01:42:48.292 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-26T01:42:48.292 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-04-26T01:42:48.292 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 8]}]: dispatch 2024-04-26T01:42:48.292 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-04-26T01:42:48.292 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-26T01:42:48.293 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-04-26T01:42:48.293 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:48 smithi006 ceph-mon[28419]: pgmap v351: 97 pgs: 97 active+clean; 607 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-04-26T01:42:48.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:48 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-04-26T01:42:48.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:48 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-26T01:42:48.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:48 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-26T01:42:48.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:48 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-26T01:42:48.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:48 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-26T01:42:48.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:48 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-04-26T01:42:48.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:48 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 8]}]: dispatch 2024-04-26T01:42:48.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:48 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-04-26T01:42:48.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:48 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-26T01:42:48.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:48 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-04-26T01:42:48.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:48 smithi188 ceph-mon[32751]: pgmap v351: 97 pgs: 97 active+clean; 607 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-04-26T01:42:48.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:48 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-04-26T01:42:48.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:48 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-26T01:42:48.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:48 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-26T01:42:48.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:48 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-26T01:42:48.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:48 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-26T01:42:48.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:48 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-04-26T01:42:48.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:48 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 8]}]: dispatch 2024-04-26T01:42:48.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:48 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-04-26T01:42:48.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:48 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-26T01:42:48.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:48 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-04-26T01:42:48.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:48 smithi029 ceph-mon[32775]: pgmap v351: 97 pgs: 97 active+clean; 607 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-04-26T01:42:49.171 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-04-26T01:42:49.171 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-26T01:42:49.171 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-26T01:42:49.171 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-26T01:42:49.171 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-26T01:42:49.171 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-04-26T01:42:49.171 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 8]}]': finished 2024-04-26T01:42:49.171 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-04-26T01:42:49.171 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-26T01:42:49.171 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-04-26T01:42:49.171 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:49 smithi006 ceph-mon[28419]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T01:42:49.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-04-26T01:42:49.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-26T01:42:49.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-26T01:42:49.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-26T01:42:49.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-26T01:42:49.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-04-26T01:42:49.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 8]}]': finished 2024-04-26T01:42:49.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-04-26T01:42:49.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-26T01:42:49.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-04-26T01:42:49.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:49 smithi188 ceph-mon[32751]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T01:42:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-04-26T01:42:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-26T01:42:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-26T01:42:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-26T01:42:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-26T01:42:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-04-26T01:42:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 8]}]': finished 2024-04-26T01:42:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-04-26T01:42:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-26T01:42:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-04-26T01:42:49.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:49 smithi029 ceph-mon[32775]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T01:42:49.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.975+0000 7ff8a1000700 1 -- 172.21.15.6:0/3703612859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff89c1092d0 msgr2=0x7ff89c1096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:49.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.975+0000 7ff8a1000700 1 --2- 172.21.15.6:0/3703612859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff89c1092d0 0x7ff89c1096b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009990 tx=0x7ff88c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:49.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.975+0000 7ff8a1000700 1 -- 172.21.15.6:0/3703612859 shutdown_connections 2024-04-26T01:42:49.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.975+0000 7ff8a1000700 1 --2- 172.21.15.6:0/3703612859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff89c10e3b0 0x7ff89c115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:49.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.975+0000 7ff8a1000700 1 --2- 172.21.15.6:0/3703612859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff89c1092d0 0x7ff89c1096b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:49.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.975+0000 7ff8a1000700 1 --2- 172.21.15.6:0/3703612859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff89c109c80 0x7ff89c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:49.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.975+0000 7ff8a1000700 1 -- 172.21.15.6:0/3703612859 >> 172.21.15.6:0/3703612859 conn(0x7ff89c076210 msgr2=0x7ff89c107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:49.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.975+0000 7ff8a1000700 1 -- 172.21.15.6:0/3703612859 shutdown_connections 2024-04-26T01:42:49.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.975+0000 7ff8a1000700 1 -- 172.21.15.6:0/3703612859 wait complete. 2024-04-26T01:42:49.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.976+0000 7ff8a1000700 1 Processor -- start 2024-04-26T01:42:49.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.977+0000 7ff8a1000700 1 -- start start 2024-04-26T01:42:49.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.977+0000 7ff8a1000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff89c1092d0 0x7ff89c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:49.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.977+0000 7ff8a1000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff89c109c80 0x7ff89c19fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:49.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.977+0000 7ff8a1000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff89c10e3b0 0x7ff89c1a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:49.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.977+0000 7ff8a1000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c119270 con 0x7ff89c109c80 2024-04-26T01:42:49.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.977+0000 7ff8a1000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c1190f0 con 0x7ff89c10e3b0 2024-04-26T01:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.977+0000 7ff8a1000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c1193f0 con 0x7ff89c1092d0 2024-04-26T01:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff899400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff89c109c80 0x7ff89c19fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff899e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff89c1092d0 0x7ff89c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff899400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff89c109c80 0x7ff89c19fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35414/0 (socket says 172.21.15.6:35414) 2024-04-26T01:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff899400700 1 -- 172.21.15.6:0/164779078 learned_addr learned my addr 172.21.15.6:0/164779078 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff899e00700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff89c1092d0 0x7ff89c19f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:57786/0 (socket says 172.21.15.6:57786) 2024-04-26T01:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff89a800700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff89c10e3b0 0x7ff89c1a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:49.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff899e00700 1 -- 172.21.15.6:0/164779078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff89c10e3b0 msgr2=0x7ff89c1a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:49.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff899e00700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff89c10e3b0 0x7ff89c1a3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:49.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff899e00700 1 -- 172.21.15.6:0/164779078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff89c109c80 msgr2=0x7ff89c19fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:49.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff899e00700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff89c109c80 0x7ff89c19fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:49.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff899e00700 1 -- 172.21.15.6:0/164779078 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff89c1a46f0 con 0x7ff89c1092d0 2024-04-26T01:42:49.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff89a800700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff89c10e3b0 0x7ff89c1a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:42:49.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.978+0000 7ff899e00700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff89c1092d0 0x7ff89c19f6e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff88c006010 tx=0x7ff88c0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:49.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.979+0000 7ff892a00700 1 -- 172.21.15.6:0/164779078 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c027070 con 0x7ff89c1092d0 2024-04-26T01:42:49.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.979+0000 7ff8a1000700 1 -- 172.21.15.6:0/164779078 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff89c1a4980 con 0x7ff89c1092d0 2024-04-26T01:42:49.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.979+0000 7ff8a1000700 1 -- 172.21.15.6:0/164779078 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff89c1ac260 con 0x7ff89c1092d0 2024-04-26T01:42:49.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.979+0000 7ff892a00700 1 -- 172.21.15.6:0/164779078 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff88c022470 con 0x7ff89c1092d0 2024-04-26T01:42:49.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.979+0000 7ff892a00700 1 -- 172.21.15.6:0/164779078 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c017650 con 0x7ff89c1092d0 2024-04-26T01:42:49.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.980+0000 7ff8a1000700 1 -- 172.21.15.6:0/164779078 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff87c005330 con 0x7ff89c1092d0 2024-04-26T01:42:49.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.983+0000 7ff892a00700 1 -- 172.21.15.6:0/164779078 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7ff88c009d10 con 0x7ff89c1092d0 2024-04-26T01:42:49.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.984+0000 7ff892a00700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff8800a3cd0 0x7ff8800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:49.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.984+0000 7ff892a00700 1 -- 172.21.15.6:0/164779078 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7ff88c0d11b0 con 0x7ff89c1092d0 2024-04-26T01:42:49.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.984+0000 7ff899400700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff8800a3cd0 0x7ff8800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:49.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.985+0000 7ff899400700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff8800a3cd0 0x7ff8800a6190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff89c077060 tx=0x7ff888009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:49.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:49.986+0000 7ff892a00700 1 -- 172.21.15.6:0/164779078 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff88c013070 con 0x7ff89c1092d0 2024-04-26T01:42:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.261+0000 7ff8a1000700 1 -- 172.21.15.6:0/164779078 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7ff87c000bc0 con 0x7ff8800a3cd0 2024-04-26T01:42:50.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.270+0000 7ff892a00700 1 -- 172.21.15.6:0/164779078 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7ff87c000bc0 con 0x7ff8800a3cd0 2024-04-26T01:42:50.268 INFO:teuthology.orchestra.run.smithi006.stdout:/volumes/qa/sv_1/29257dce-2e4e-437a-81bf-7b5b8d792c03 2024-04-26T01:42:50.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.273+0000 7ff8a1000700 1 -- 172.21.15.6:0/164779078 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff8800a3cd0 msgr2=0x7ff8800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:50.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.273+0000 7ff8a1000700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff8800a3cd0 0x7ff8800a6190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff89c077060 tx=0x7ff888009260 comp rx=0 tx=0).stop 2024-04-26T01:42:50.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.273+0000 7ff8a1000700 1 -- 172.21.15.6:0/164779078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff89c1092d0 msgr2=0x7ff89c19f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:50.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.273+0000 7ff8a1000700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff89c1092d0 0x7ff89c19f6e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff88c006010 tx=0x7ff88c0057c0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.274+0000 7ff8a1000700 1 -- 172.21.15.6:0/164779078 shutdown_connections 2024-04-26T01:42:50.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.274+0000 7ff8a1000700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff89c10e3b0 0x7ff89c1a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.274+0000 7ff8a1000700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff8800a3cd0 0x7ff8800a6190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.274+0000 7ff8a1000700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff89c1092d0 0x7ff89c19f6e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.274+0000 7ff8a1000700 1 --2- 172.21.15.6:0/164779078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff89c109c80 0x7ff89c19fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.274+0000 7ff8a1000700 1 -- 172.21.15.6:0/164779078 >> 172.21.15.6:0/164779078 conn(0x7ff89c076210 msgr2=0x7ff89c0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:50.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.274+0000 7ff8a1000700 1 -- 172.21.15.6:0/164779078 shutdown_connections 2024-04-26T01:42:50.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:50.274+0000 7ff8a1000700 1 -- 172.21.15.6:0/164779078 wait complete. 2024-04-26T01:42:50.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:50 smithi006 ceph-mon[28419]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T01:42:50.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:50 smithi006 ceph-mon[28419]: pgmap v354: 97 pgs: 97 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 9 op/s 2024-04-26T01:42:50.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:50 smithi188 ceph-mon[32751]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T01:42:50.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:50 smithi188 ceph-mon[32751]: pgmap v354: 97 pgs: 97 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 9 op/s 2024-04-26T01:42:50.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:50 smithi029 ceph-mon[32775]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T01:42:50.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:50 smithi029 ceph-mon[32775]: pgmap v354: 97 pgs: 97 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 9 op/s 2024-04-26T01:42:51.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:42:51.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:51 smithi006 ceph-mon[28419]: from='client.24652 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:51 smithi188 ceph-mon[32751]: from='client.24652 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:51.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:51 smithi029 ceph-mon[32775]: from='client.24652 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:42:51.553 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:52.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:52 smithi006 ceph-mon[28419]: pgmap v355: 97 pgs: 97 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.1 KiB/s wr, 9 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T01:42:52.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:52 smithi188 ceph-mon[32751]: pgmap v355: 97 pgs: 97 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.1 KiB/s wr, 9 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T01:42:52.439 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:52 smithi029 ceph-mon[32775]: pgmap v355: 97 pgs: 97 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.1 KiB/s wr, 9 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T01:42:53.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.775+0000 7fee90400700 1 -- 172.21.15.6:0/186390201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee8806b0f0 msgr2=0x7fee8810f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:53.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.775+0000 7fee90400700 1 --2- 172.21.15.6:0/186390201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee8806b0f0 0x7fee8810f900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fee8000b0a0 tx=0x7fee8000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:42:53.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.775+0000 7fee90400700 1 -- 172.21.15.6:0/186390201 shutdown_connections 2024-04-26T01:42:53.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.775+0000 7fee90400700 1 --2- 172.21.15.6:0/186390201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee8806b0f0 0x7fee8810f900 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:53.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.775+0000 7fee90400700 1 --2- 172.21.15.6:0/186390201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee88108a30 0x7fee88108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:53.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.775+0000 7fee90400700 1 --2- 172.21.15.6:0/186390201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806a750 0x7fee8806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:53.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.775+0000 7fee90400700 1 -- 172.21.15.6:0/186390201 >> 172.21.15.6:0/186390201 conn(0x7fee880788f0 msgr2=0x7fee88078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:53.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.775+0000 7fee90400700 1 -- 172.21.15.6:0/186390201 shutdown_connections 2024-04-26T01:42:53.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.775+0000 7fee90400700 1 -- 172.21.15.6:0/186390201 wait complete. 2024-04-26T01:42:53.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.776+0000 7fee90400700 1 Processor -- start 2024-04-26T01:42:53.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.776+0000 7fee90400700 1 -- start start 2024-04-26T01:42:53.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.777+0000 7fee90400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806a750 0x7fee88101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:53.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.777+0000 7fee90400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee8806b0f0 0x7fee88101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:53.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.777+0000 7fee90400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee88108a30 0x7fee88102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:53.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.777+0000 7fee90400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fee88112c20 con 0x7fee8806a750 2024-04-26T01:42:53.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.777+0000 7fee90400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fee88112aa0 con 0x7fee88108a30 2024-04-26T01:42:53.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.777+0000 7fee90400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fee88112da0 con 0x7fee8806b0f0 2024-04-26T01:42:53.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.777+0000 7fee8d000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee8806b0f0 0x7fee88101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:53.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.777+0000 7fee8da00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806a750 0x7fee88101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:53.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.777+0000 7fee8d000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee8806b0f0 0x7fee88101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:57800/0 (socket says 172.21.15.6:57800) 2024-04-26T01:42:53.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.777+0000 7fee8d000700 1 -- 172.21.15.6:0/2319613292 learned_addr learned my addr 172.21.15.6:0/2319613292 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:53.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.777+0000 7fee8e400700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee88108a30 0x7fee88102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:53.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.778+0000 7fee8d000700 1 -- 172.21.15.6:0/2319613292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee88108a30 msgr2=0x7fee88102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:53.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.778+0000 7fee8d000700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee88108a30 0x7fee88102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:53.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.778+0000 7fee8d000700 1 -- 172.21.15.6:0/2319613292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806a750 msgr2=0x7fee88101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:53.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.778+0000 7fee8d000700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806a750 0x7fee88101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:53.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.778+0000 7fee8d000700 1 -- 172.21.15.6:0/2319613292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee881aba80 con 0x7fee8806b0f0 2024-04-26T01:42:53.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.778+0000 7fee8e400700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee88108a30 0x7fee88102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:53.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.778+0000 7fee8d000700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee8806b0f0 0x7fee88101f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fee7400e9a0 tx=0x7fee7400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:53.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.778+0000 7fee86000700 1 -- 172.21.15.6:0/2319613292 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7400ca10 con 0x7fee8806b0f0 2024-04-26T01:42:53.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.778+0000 7fee90400700 1 -- 172.21.15.6:0/2319613292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee881abd10 con 0x7fee8806b0f0 2024-04-26T01:42:53.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.778+0000 7fee86000700 1 -- 172.21.15.6:0/2319613292 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee7400cbb0 con 0x7fee8806b0f0 2024-04-26T01:42:53.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.778+0000 7fee86000700 1 -- 172.21.15.6:0/2319613292 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee74018680 con 0x7fee8806b0f0 2024-04-26T01:42:53.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.779+0000 7fee90400700 1 -- 172.21.15.6:0/2319613292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee881ac1e0 con 0x7fee8806b0f0 2024-04-26T01:42:53.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.780+0000 7fee90400700 1 -- 172.21.15.6:0/2319613292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee88104170 con 0x7fee8806b0f0 2024-04-26T01:42:53.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.782+0000 7fee86000700 1 -- 172.21.15.6:0/2319613292 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fee74010430 con 0x7fee8806b0f0 2024-04-26T01:42:53.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.783+0000 7fee86000700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fee700a3cd0 0x7fee700a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:53.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.783+0000 7fee86000700 1 -- 172.21.15.6:0/2319613292 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7fee740d0aa0 con 0x7fee8806b0f0 2024-04-26T01:42:53.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.784+0000 7fee8da00700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fee700a3cd0 0x7fee700a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:53.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.784+0000 7fee8da00700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fee700a3cd0 0x7fee700a6190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fee7c009be0 tx=0x7fee7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:53.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:53.785+0000 7fee86000700 1 -- 172.21.15.6:0/2319613292 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee7409d2a0 con 0x7fee8806b0f0 2024-04-26T01:42:54.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.067+0000 7fee90400700 1 -- 172.21.15.6:0/2319613292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee88102f70 con 0x7fee8806b0f0 2024-04-26T01:42:54.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.068+0000 7fee86000700 1 -- 172.21.15.6:0/2319613292 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11850 (secure 0 0 0) 0x7fee88102f70 con 0x7fee8806b0f0 2024-04-26T01:42:54.066 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:42:54.066 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:42:31.294242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:42:54.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.071+0000 7fee90400700 1 -- 172.21.15.6:0/2319613292 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fee700a3cd0 msgr2=0x7fee700a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:54.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.071+0000 7fee90400700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fee700a3cd0 0x7fee700a6190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fee7c009be0 tx=0x7fee7c009260 comp rx=0 tx=0).stop 2024-04-26T01:42:54.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.071+0000 7fee90400700 1 -- 172.21.15.6:0/2319613292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee8806b0f0 msgr2=0x7fee88101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:54.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.071+0000 7fee90400700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee8806b0f0 0x7fee88101f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fee7400e9a0 tx=0x7fee7400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:42:54.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.071+0000 7fee90400700 1 -- 172.21.15.6:0/2319613292 shutdown_connections 2024-04-26T01:42:54.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.071+0000 7fee90400700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee88108a30 0x7fee88102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:54.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.071+0000 7fee90400700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806a750 0x7fee88101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:54.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.071+0000 7fee90400700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fee700a3cd0 0x7fee700a6190 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:54.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.071+0000 7fee90400700 1 --2- 172.21.15.6:0/2319613292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee8806b0f0 0x7fee88101f50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:54.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.071+0000 7fee90400700 1 -- 172.21.15.6:0/2319613292 >> 172.21.15.6:0/2319613292 conn(0x7fee880788f0 msgr2=0x7fee880ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:54.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.072+0000 7fee90400700 1 -- 172.21.15.6:0/2319613292 shutdown_connections 2024-04-26T01:42:54.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:54.072+0000 7fee90400700 1 -- 172.21.15.6:0/2319613292 wait complete. 2024-04-26T01:42:54.071 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-26T01:42:54.497 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:54 smithi006 ceph-mon[28419]: pgmap v356: 97 pgs: 97 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.9 KiB/s wr, 7 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T01:42:54.497 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:54 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2319613292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:42:54.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:54 smithi188 ceph-mon[32751]: pgmap v356: 97 pgs: 97 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.9 KiB/s wr, 7 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T01:42:54.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:54 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2319613292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:42:54.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:54 smithi029 ceph-mon[32775]: pgmap v356: 97 pgs: 97 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.9 KiB/s wr, 7 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T01:42:54.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:54 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2319613292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:42:54.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:42:55.360 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:56.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:56 smithi006 ceph-mon[28419]: pgmap v357: 97 pgs: 97 active+clean; 634 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.0 KiB/s wr, 8 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T01:42:56.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:56 smithi188 ceph-mon[32751]: pgmap v357: 97 pgs: 97 active+clean; 634 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.0 KiB/s wr, 8 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T01:42:56.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:56 smithi029 ceph-mon[32775]: pgmap v357: 97 pgs: 97 active+clean; 634 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.0 KiB/s wr, 8 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T01:42:57.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.585+0000 7fce42200700 1 -- 172.21.15.6:0/3846148091 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c109cc0 msgr2=0x7fce3c10dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:57.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.585+0000 7fce42200700 1 --2- 172.21.15.6:0/3846148091 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c109cc0 0x7fce3c10dcc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fce240099c0 tx=0x7fce24009cd0 comp rx=0 tx=0).stop 2024-04-26T01:42:57.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.586+0000 7fce42200700 1 -- 172.21.15.6:0/3846148091 shutdown_connections 2024-04-26T01:42:57.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.586+0000 7fce42200700 1 --2- 172.21.15.6:0/3846148091 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce3c10e3a0 0x7fce3c115f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:57.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.586+0000 7fce42200700 1 --2- 172.21.15.6:0/3846148091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce3c109310 0x7fce3c1096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:57.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.586+0000 7fce42200700 1 --2- 172.21.15.6:0/3846148091 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c109cc0 0x7fce3c10dcc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:57.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.586+0000 7fce42200700 1 -- 172.21.15.6:0/3846148091 >> 172.21.15.6:0/3846148091 conn(0x7fce3c076210 msgr2=0x7fce3c078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:57.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.586+0000 7fce42200700 1 -- 172.21.15.6:0/3846148091 shutdown_connections 2024-04-26T01:42:57.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.586+0000 7fce42200700 1 -- 172.21.15.6:0/3846148091 wait complete. 2024-04-26T01:42:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.587+0000 7fce42200700 1 Processor -- start 2024-04-26T01:42:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.587+0000 7fce42200700 1 -- start start 2024-04-26T01:42:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.587+0000 7fce42200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce3c109310 0x7fce3c1a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.587+0000 7fce42200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c109cc0 0x7fce3c1a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.587+0000 7fce42200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce3c10e3a0 0x7fce3c19f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.587+0000 7fce42200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fce3c119230 con 0x7fce3c109cc0 2024-04-26T01:42:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.587+0000 7fce42200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fce3c1190b0 con 0x7fce3c10e3a0 2024-04-26T01:42:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.587+0000 7fce42200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fce3c1193b0 con 0x7fce3c109310 2024-04-26T01:42:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.588+0000 7fce32a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c109cc0 0x7fce3c1a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.588+0000 7fce3b400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce3c109310 0x7fce3c1a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.588+0000 7fce32a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c109cc0 0x7fce3c1a5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35450/0 (socket says 172.21.15.6:35450) 2024-04-26T01:42:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.588+0000 7fce32a00700 1 -- 172.21.15.6:0/839210339 learned_addr learned my addr 172.21.15.6:0/839210339 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:42:57.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.588+0000 7fce3be00700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce3c10e3a0 0x7fce3c19f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:57.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.588+0000 7fce3b400700 1 -- 172.21.15.6:0/839210339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce3c10e3a0 msgr2=0x7fce3c19f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:57.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.588+0000 7fce3b400700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce3c10e3a0 0x7fce3c19f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:57.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.588+0000 7fce3b400700 1 -- 172.21.15.6:0/839210339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c109cc0 msgr2=0x7fce3c1a5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:57.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.588+0000 7fce3b400700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c109cc0 0x7fce3c1a5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:57.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.588+0000 7fce3b400700 1 -- 172.21.15.6:0/839210339 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce3c19fd50 con 0x7fce3c109310 2024-04-26T01:42:57.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.588+0000 7fce32a00700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c109cc0 0x7fce3c1a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:57.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.589+0000 7fce3b400700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce3c109310 0x7fce3c1a53e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce2c00ef10 tx=0x7fce2c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:57.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.589+0000 7fce38c00700 1 -- 172.21.15.6:0/839210339 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2c00c9c0 con 0x7fce3c109310 2024-04-26T01:42:57.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.589+0000 7fce38c00700 1 -- 172.21.15.6:0/839210339 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce2c019070 con 0x7fce3c109310 2024-04-26T01:42:57.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.589+0000 7fce38c00700 1 -- 172.21.15.6:0/839210339 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2c014480 con 0x7fce3c109310 2024-04-26T01:42:57.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.589+0000 7fce42200700 1 -- 172.21.15.6:0/839210339 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce3c19ffe0 con 0x7fce3c109310 2024-04-26T01:42:57.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.589+0000 7fce42200700 1 -- 172.21.15.6:0/839210339 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce3c1ac2b0 con 0x7fce3c109310 2024-04-26T01:42:57.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.590+0000 7fce42200700 1 -- 172.21.15.6:0/839210339 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce3c069730 con 0x7fce3c109310 2024-04-26T01:42:57.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.592+0000 7fce38c00700 1 -- 172.21.15.6:0/839210339 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fce2c012050 con 0x7fce3c109310 2024-04-26T01:42:57.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.594+0000 7fce38c00700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fce200a3d50 0x7fce200a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:57.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.594+0000 7fce32a00700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fce200a3d50 0x7fce200a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:57.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.594+0000 7fce38c00700 1 -- 172.21.15.6:0/839210339 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7fce2c0d0b70 con 0x7fce3c109310 2024-04-26T01:42:57.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.595+0000 7fce32a00700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fce200a3d50 0x7fce200a6210 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce24009990 tx=0x7fce24005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:57.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.596+0000 7fce38c00700 1 -- 172.21.15.6:0/839210339 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce2c010040 con 0x7fce3c109310 2024-04-26T01:42:57.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.831+0000 7fce42200700 1 -- 172.21.15.6:0/839210339 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce3c02df80 con 0x7fce3c109310 2024-04-26T01:42:57.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.833+0000 7fce38c00700 1 -- 172.21.15.6:0/839210339 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+22636 (secure 0 0 0) 0x7fce2c09d370 con 0x7fce3c109310 2024-04-26T01:42:57.831 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:42:57.832 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":93,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:32:29.071544+0000","modified":"2024-04-26T01:42:49.034928+0000","last_up_change":"2024-04-26T01:40:49.915924+0000","last_in_change":"2024-04-26T01:40:29.559647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:36:35.182726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:41:59.612167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:42:04.106467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f164c51d-ce5d-4140-b9e3-a4b84088f5ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6803","nonce":76518486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6805","nonce":76518486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6809","nonce":76518486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6807","nonce":76518486}]},"public_addr":"172.21.15.6:6803/76518486","cluster_addr":"172.21.15.6:6805/76518486","heartbeat_back_addr":"172.21.15.6:6809/76518486","heartbeat_front_addr":"172.21.15.6:6807/76518486","state":["exists","up"]},{"osd":1,"uuid":"57e875dd-ddf1-4ee4-a2a7-9589d7f42818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6801","nonce":400769116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6803","nonce":400769116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6807","nonce":400769116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6805","nonce":400769116}]},"public_addr":"172.21.15.29:6801/400769116","cluster_addr":"172.21.15.29:6803/400769116","heartbeat_back_addr":"172.21.15.29:6807/400769116","heartbeat_front_addr":"172.21.15.29:6805/400769116","state":["exists","up"]},{"osd":2,"uuid":"77e0c294-c48a-4318-938b-4604a533a2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6801","nonce":3508067672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6803","nonce":3508067672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6807","nonce":3508067672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6805","nonce":3508067672}]},"public_addr":"172.21.15.188:6801/3508067672","cluster_addr":"172.21.15.188:6803/3508067672","heartbeat_back_addr":"172.21.15.188:6807/3508067672","heartbeat_front_addr":"172.21.15.188:6805/3508067672","state":["exists","up"]},{"osd":3,"uuid":"6393dac5-996c-47af-9687-0f1393ecc483","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6811","nonce":1987311118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6813","nonce":1987311118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6817","nonce":1987311118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6815","nonce":1987311118}]},"public_addr":"172.21.15.6:6811/1987311118","cluster_addr":"172.21.15.6:6813/1987311118","heartbeat_back_addr":"172.21.15.6:6817/1987311118","heartbeat_front_addr":"172.21.15.6:6815/1987311118","state":["exists","up"]},{"osd":4,"uuid":"f2c1a535-26a0-46c6-a6d0-c2529c786739","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6809","nonce":2125876609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6811","nonce":2125876609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6815","nonce":2125876609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6813","nonce":2125876609}]},"public_addr":"172.21.15.29:6809/2125876609","cluster_addr":"172.21.15.29:6811/2125876609","heartbeat_back_addr":"172.21.15.29:6815/2125876609","heartbeat_front_addr":"172.21.15.29:6813/2125876609","state":["exists","up"]},{"osd":5,"uuid":"9788bb0a-0d79-4743-bce6-80f5342e8f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6809","nonce":2092077007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6811","nonce":2092077007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6815","nonce":2092077007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6813","nonce":2092077007}]},"public_addr":"172.21.15.188:6809/2092077007","cluster_addr":"172.21.15.188:6811/2092077007","heartbeat_back_addr":"172.21.15.188:6815/2092077007","heartbeat_front_addr":"172.21.15.188:6813/2092077007","state":["exists","up"]},{"osd":6,"uuid":"fd8584e1-69ee-4c30-85e4-a637357d1b6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6819","nonce":3256016695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6821","nonce":3256016695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6825","nonce":3256016695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6823","nonce":3256016695}]},"public_addr":"172.21.15.6:6819/3256016695","cluster_addr":"172.21.15.6:6821/3256016695","heartbeat_back_addr":"172.21.15.6:6825/3256016695","heartbeat_front_addr":"172.21.15.6:6823/3256016695","state":["exists","up"]},{"osd":7,"uuid":"6b7e6784-d574-42da-8416-a74a3b6b3755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6817","nonce":3478594037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6819","nonce":3478594037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6823","nonce":3478594037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6821","nonce":3478594037}]},"public_addr":"172.21.15.29:6817/3478594037","cluster_addr":"172.21.15.29:6819/3478594037","heartbeat_back_addr":"172.21.15.29:6823/3478594037","heartbeat_front_addr":"172.21.15.29:6821/3478594037","state":["exists","up"]},{"osd":8,"uuid":"30cd2ea1-2767-4062-8bd9-47214e4b4b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6817","nonce":3290364376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6819","nonce":3290364376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6823","nonce":3290364376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6821","nonce":3290364376}]},"public_addr":"172.21.15.188:6817/3290364376","cluster_addr":"172.21.15.188:6819/3290364376","heartbeat_back_addr":"172.21.15.188:6823/3290364376","heartbeat_front_addr":"172.21.15.188:6821/3290364376","state":["exists","up"]},{"osd":9,"uuid":"f033cdc0-1178-4a7f-a338-69ff32416087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6827","nonce":1918893912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6829","nonce":1918893912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6833","nonce":1918893912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6831","nonce":1918893912}]},"public_addr":"172.21.15.6:6827/1918893912","cluster_addr":"172.21.15.6:6829/1918893912","heartbeat_back_addr":"172.21.15.6:6833/1918893912","heartbeat_front_addr":"172.21.15.6:6831/1918893912","state":["exists","up"]},{"osd":10,"uuid":"04e9b5ef-ac0c-482b-a69e-a7121b36e7da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6825","nonce":1395777729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6827","nonce":1395777729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6831","nonce":1395777729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6829","nonce":1395777729}]},"public_addr":"172.21.15.29:6825/1395777729","cluster_addr":"172.21.15.29:6827/1395777729","heartbeat_back_addr":"172.21.15.29:6831/1395777729","heartbeat_front_addr":"172.21.15.29:6829/1395777729","state":["exists","up"]},{"osd":11,"uuid":"cb1638ce-4189-4241-b63e-b63c607e90ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6825","nonce":3322197880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6827","nonce":3322197880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6831","nonce":3322197880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6829","nonce":3322197880}]},"public_addr":"172.21.15.188:6825/3322197880","cluster_addr":"172.21.15.188:6827/3322197880","heartbeat_back_addr":"172.21.15.188:6831/3322197880","heartbeat_front_addr":"172.21.15.188:6829/3322197880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:41.553455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:07.641467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:31.748891+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:03.743553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:30.065262+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:54.976527+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:29.348206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:56.285928+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:21.246409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:54.468561+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:22.093015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:47.342379+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.19","mappings":[{"from":3,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/3143272062":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1325707596":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6800/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6801/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1359518071":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6801/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/2813598711":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/3140150861":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6800/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/65701414":"2024-04-27T01:33:47.021882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:42:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.837+0000 7fce42200700 1 -- 172.21.15.6:0/839210339 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fce200a3d50 msgr2=0x7fce200a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.837+0000 7fce42200700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fce200a3d50 0x7fce200a6210 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce24009990 tx=0x7fce24005960 comp rx=0 tx=0).stop 2024-04-26T01:42:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.837+0000 7fce42200700 1 -- 172.21.15.6:0/839210339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce3c109310 msgr2=0x7fce3c1a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.837+0000 7fce42200700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce3c109310 0x7fce3c1a53e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce2c00ef10 tx=0x7fce2c00c3a0 comp rx=0 tx=0).stop 2024-04-26T01:42:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.837+0000 7fce42200700 1 -- 172.21.15.6:0/839210339 shutdown_connections 2024-04-26T01:42:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.837+0000 7fce42200700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce3c10e3a0 0x7fce3c19f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.837+0000 7fce42200700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fce200a3d50 0x7fce200a6210 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.837+0000 7fce42200700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce3c109310 0x7fce3c1a53e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.837+0000 7fce42200700 1 --2- 172.21.15.6:0/839210339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c109cc0 0x7fce3c1a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.837+0000 7fce42200700 1 -- 172.21.15.6:0/839210339 >> 172.21.15.6:0/839210339 conn(0x7fce3c076210 msgr2=0x7fce3c078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.837+0000 7fce42200700 1 -- 172.21.15.6:0/839210339 shutdown_connections 2024-04-26T01:42:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:42:57.837+0000 7fce42200700 1 -- 172.21.15.6:0/839210339 wait complete. 2024-04-26T01:42:58.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:58 smithi006 ceph-mon[28419]: pgmap v358: 97 pgs: 97 active+clean; 634 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 8 op/s; 12 B/s, 1 objects/s recovering 2024-04-26T01:42:58.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:42:58 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/839210339' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:58.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:58 smithi188 ceph-mon[32751]: pgmap v358: 97 pgs: 97 active+clean; 634 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 8 op/s; 12 B/s, 1 objects/s recovering 2024-04-26T01:42:58.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:42:58 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/839210339' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:58.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:58 smithi029 ceph-mon[32775]: pgmap v358: 97 pgs: 97 active+clean; 634 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 8 op/s; 12 B/s, 1 objects/s recovering 2024-04-26T01:42:58.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:42:58 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/839210339' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:58.811 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T01:42:58.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:42:59.291 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:00.549 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:00 smithi029 ceph-mon[32775]: pgmap v359: 97 pgs: 97 active+clean; 638 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.7 KiB/s wr, 9 op/s; 11 B/s, 1 objects/s recovering 2024-04-26T01:43:00.549 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:00 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:43:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:00 smithi006 ceph-mon[28419]: pgmap v359: 97 pgs: 97 active+clean; 638 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.7 KiB/s wr, 9 op/s; 11 B/s, 1 objects/s recovering 2024-04-26T01:43:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:00 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:43:00.613 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:00 smithi188 ceph-mon[32751]: pgmap v359: 97 pgs: 97 active+clean; 638 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.7 KiB/s wr, 9 op/s; 11 B/s, 1 objects/s recovering 2024-04-26T01:43:00.614 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:00 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:43:01.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.447+0000 7f075a800700 1 -- 172.21.15.6:0/1130691344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0754107940 msgr2=0x7f0754107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:01.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.447+0000 7f075a800700 1 --2- 172.21.15.6:0/1130691344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0754107940 0x7f0754107d20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f07440099c0 tx=0x7f0744009cd0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.447+0000 7f075a800700 1 -- 172.21.15.6:0/1130691344 shutdown_connections 2024-04-26T01:43:01.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.447+0000 7f075a800700 1 --2- 172.21.15.6:0/1130691344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07541034b0 0x7f075410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.447+0000 7f075a800700 1 --2- 172.21.15.6:0/1130691344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0754107940 0x7f0754107d20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.447+0000 7f075a800700 1 --2- 172.21.15.6:0/1130691344 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0754102af0 0x7f0754102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.447+0000 7f075a800700 1 -- 172.21.15.6:0/1130691344 >> 172.21.15.6:0/1130691344 conn(0x7f0754079ba0 msgr2=0x7f0754079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:01.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.448+0000 7f075a800700 1 -- 172.21.15.6:0/1130691344 shutdown_connections 2024-04-26T01:43:01.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.448+0000 7f075a800700 1 -- 172.21.15.6:0/1130691344 wait complete. 2024-04-26T01:43:01.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.449+0000 7f075a800700 1 Processor -- start 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.449+0000 7f075a800700 1 -- start start 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.449+0000 7f075a800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0754102af0 0x7f075419b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.449+0000 7f075a800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07541034b0 0x7f075419baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.449+0000 7f075a800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f075419fe30 0x7f07541a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.449+0000 7f075a800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0754112cb0 con 0x7f0754102af0 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.449+0000 7f075a800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0754112b30 con 0x7f075419fe30 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.449+0000 7f075a800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0754112e30 con 0x7f07541034b0 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.450+0000 7f0753e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f075419fe30 0x7f07541a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.450+0000 7f074be00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07541034b0 0x7f075419baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.450+0000 7f0753400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0754102af0 0x7f075419b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.450+0000 7f0753e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f075419fe30 0x7f07541a02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:46474/0 (socket says 172.21.15.6:46474) 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.450+0000 7f074be00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07541034b0 0x7f075419baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:54830/0 (socket says 172.21.15.6:54830) 2024-04-26T01:43:01.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.450+0000 7f0753e00700 1 -- 172.21.15.6:0/3763729712 learned_addr learned my addr 172.21.15.6:0/3763729712 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:01.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.450+0000 7f0753e00700 1 -- 172.21.15.6:0/3763729712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07541034b0 msgr2=0x7f075419baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:01.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.450+0000 7f0753e00700 1 --2- 172.21.15.6:0/3763729712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07541034b0 0x7f075419baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.450+0000 7f0753e00700 1 -- 172.21.15.6:0/3763729712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0754102af0 msgr2=0x7f075419b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:01.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.450+0000 7f0753e00700 1 --2- 172.21.15.6:0/3763729712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0754102af0 0x7f075419b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.450+0000 7f0753e00700 1 -- 172.21.15.6:0/3763729712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07541a0950 con 0x7f075419fe30 2024-04-26T01:43:01.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.451+0000 7f0753e00700 1 --2- 172.21.15.6:0/3763729712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f075419fe30 0x7f07541a02b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f074c00efc0 tx=0x7f074c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:01.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.451+0000 7f0750c00700 1 -- 172.21.15.6:0/3763729712 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c0092e0 con 0x7f075419fe30 2024-04-26T01:43:01.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.451+0000 7f0750c00700 1 -- 172.21.15.6:0/3763729712 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f074c004bb0 con 0x7f075419fe30 2024-04-26T01:43:01.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.451+0000 7f0750c00700 1 -- 172.21.15.6:0/3763729712 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c004020 con 0x7f075419fe30 2024-04-26T01:43:01.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.451+0000 7f075a800700 1 -- 172.21.15.6:0/3763729712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07541a0be0 con 0x7f075419fe30 2024-04-26T01:43:01.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.451+0000 7f075a800700 1 -- 172.21.15.6:0/3763729712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07541a8370 con 0x7f075419fe30 2024-04-26T01:43:01.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.453+0000 7f075a800700 1 -- 172.21.15.6:0/3763729712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0754069730 con 0x7f075419fe30 2024-04-26T01:43:01.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.454+0000 7f0750c00700 1 -- 172.21.15.6:0/3763729712 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f074c01b030 con 0x7f075419fe30 2024-04-26T01:43:01.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.455+0000 7f0750c00700 1 --2- 172.21.15.6:0/3763729712 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07380a3d50 0x7f07380a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:01.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.456+0000 7f0750c00700 1 -- 172.21.15.6:0/3763729712 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7f074c0cfdf0 con 0x7f075419fe30 2024-04-26T01:43:01.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.456+0000 7f0753400700 1 --2- 172.21.15.6:0/3763729712 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07380a3d50 0x7f07380a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:01.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.456+0000 7f0753400700 1 --2- 172.21.15.6:0/3763729712 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07380a3d50 0x7f07380a6210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0744009990 tx=0x7f0744005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:01.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.460+0000 7f0750c00700 1 -- 172.21.15.6:0/3763729712 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f074c022020 con 0x7f075419fe30 2024-04-26T01:43:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.690+0000 7f075a800700 1 -- 172.21.15.6:0/3763729712 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0754101070 con 0x7f07380a3d50 2024-04-26T01:43:01.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.717+0000 7f0750c00700 1 -- 172.21.15.6:0/3763729712 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258753 (secure 0 0 0) 0x7f0754101070 con 0x7f07380a3d50 2024-04-26T01:43:01.722 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:43:01.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.724+0000 7f075a800700 1 -- 172.21.15.6:0/3763729712 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07380a3d50 msgr2=0x7f07380a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:01.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.724+0000 7f075a800700 1 --2- 172.21.15.6:0/3763729712 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07380a3d50 0x7f07380a6210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0744009990 tx=0x7f0744005fb0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.725+0000 7f075a800700 1 -- 172.21.15.6:0/3763729712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f075419fe30 msgr2=0x7f07541a02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:01.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.725+0000 7f075a800700 1 --2- 172.21.15.6:0/3763729712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f075419fe30 0x7f07541a02b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f074c00efc0 tx=0x7f074c00c3a0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.725+0000 7f075a800700 1 -- 172.21.15.6:0/3763729712 shutdown_connections 2024-04-26T01:43:01.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.725+0000 7f075a800700 1 --2- 172.21.15.6:0/3763729712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f075419fe30 0x7f07541a02b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.725+0000 7f075a800700 1 --2- 172.21.15.6:0/3763729712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0754102af0 0x7f075419b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.725+0000 7f075a800700 1 --2- 172.21.15.6:0/3763729712 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07380a3d50 0x7f07380a6210 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.725+0000 7f075a800700 1 --2- 172.21.15.6:0/3763729712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07541034b0 0x7f075419baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.725+0000 7f075a800700 1 -- 172.21.15.6:0/3763729712 >> 172.21.15.6:0/3763729712 conn(0x7f0754079ba0 msgr2=0x7f0754100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:01.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.725+0000 7f075a800700 1 -- 172.21.15.6:0/3763729712 shutdown_connections 2024-04-26T01:43:01.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:01.725+0000 7f075a800700 1 -- 172.21.15.6:0/3763729712 wait complete. 2024-04-26T01:43:01.725 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-26T01:43:02.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:02 smithi006 ceph-mon[28419]: pgmap v360: 97 pgs: 97 active+clean; 638 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s; 9 B/s, 0 objects/s recovering 2024-04-26T01:43:02.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:02.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:02.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:02.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:02 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:02.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:02 smithi006 ceph-mon[28419]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:43:02.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:02 smithi188 ceph-mon[32751]: pgmap v360: 97 pgs: 97 active+clean; 638 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s; 9 B/s, 0 objects/s recovering 2024-04-26T01:43:02.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:02.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:02.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:02.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:02 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:02.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:02 smithi188 ceph-mon[32751]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:43:02.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:02 smithi029 ceph-mon[32775]: pgmap v360: 97 pgs: 97 active+clean; 638 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s; 9 B/s, 0 objects/s recovering 2024-04-26T01:43:02.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:02.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:02.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:02.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:02 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:02.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:02 smithi029 ceph-mon[32775]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:43:02.900 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-04-26T01:43:01.195561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653809,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":355,"num_read_kb":297,"num_write":349,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392396,"kb_used_data":14080,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549428,"statfs":{"total":1151940427776,"available":1151538614272,"internally_reserved":0,"allocated":14417920,"data_stored":7879668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13304,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":48,"num_write":15,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004702"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058393+0000","last_change":"2024-04-26T01:42:05.417304+0000","last_active":"2024-04-26T01:42:49.058393+0000","last_peered":"2024-04-26T01:42:49.058393+0000","last_clean":"2024-04-26T01:42:49.058393+0000","last_became_active":"2024-04-26T01:42:05.416999+0000","last_became_peered":"2024-04-26T01:42:05.416999+0000","last_unstale":"2024-04-26T01:42:49.058393+0000","last_undegraded":"2024-04-26T01:42:49.058393+0000","last_fullsized":"2024-04-26T01:42:49.058393+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:33:30.958943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.061430+0000","last_change":"2024-04-26T01:42:05.413089+0000","last_active":"2024-04-26T01:42:49.061430+0000","last_peered":"2024-04-26T01:42:49.061430+0000","last_clean":"2024-04-26T01:42:49.061430+0000","last_became_active":"2024-04-26T01:42:05.412776+0000","last_became_peered":"2024-04-26T01:42:05.412776+0000","last_unstale":"2024-04-26T01:42:49.061430+0000","last_undegraded":"2024-04-26T01:42:49.061430+0000","last_fullsized":"2024-04-26T01:42:49.061430+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:47.861340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.060501+0000","last_change":"2024-04-26T01:42:05.404864+0000","last_active":"2024-04-26T01:42:49.060501+0000","last_peered":"2024-04-26T01:42:49.060501+0000","last_clean":"2024-04-26T01:42:49.060501+0000","last_became_active":"2024-04-26T01:42:05.404529+0000","last_became_peered":"2024-04-26T01:42:05.404529+0000","last_unstale":"2024-04-26T01:42:49.060501+0000","last_undegraded":"2024-04-26T01:42:49.060501+0000","last_fullsized":"2024-04-26T01:42:49.060501+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:04:18.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.051174+0000","last_change":"2024-04-26T01:42:05.393582+0000","last_active":"2024-04-26T01:42:49.051174+0000","last_peered":"2024-04-26T01:42:49.051174+0000","last_clean":"2024-04-26T01:42:49.051174+0000","last_became_active":"2024-04-26T01:42:05.393193+0000","last_became_peered":"2024-04-26T01:42:05.393193+0000","last_unstale":"2024-04-26T01:42:49.051174+0000","last_undegraded":"2024-04-26T01:42:49.051174+0000","last_fullsized":"2024-04-26T01:42:49.051174+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:20:29.420459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.055773+0000","last_change":"2024-04-26T01:42:05.412979+0000","last_active":"2024-04-26T01:42:49.055773+0000","last_peered":"2024-04-26T01:42:49.055773+0000","last_clean":"2024-04-26T01:42:49.055773+0000","last_became_active":"2024-04-26T01:42:05.412679+0000","last_became_peered":"2024-04-26T01:42:05.412679+0000","last_unstale":"2024-04-26T01:42:49.055773+0000","last_undegraded":"2024-04-26T01:42:49.055773+0000","last_fullsized":"2024-04-26T01:42:49.055773+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:22.774960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.062507+0000","last_change":"2024-04-26T01:42:49.062507+0000","last_active":"2024-04-26T01:42:49.062507+0000","last_peered":"2024-04-26T01:42:49.062507+0000","last_clean":"2024-04-26T01:42:49.062507+0000","last_became_active":"2024-04-26T01:42:49.062095+0000","last_became_peered":"2024-04-26T01:42:49.062095+0000","last_unstale":"2024-04-26T01:42:49.062507+0000","last_undegraded":"2024-04-26T01:42:49.062507+0000","last_fullsized":"2024-04-26T01:42:49.062507+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:47.846697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.052858+0000","last_change":"2024-04-26T01:42:05.385221+0000","last_active":"2024-04-26T01:42:49.052858+0000","last_peered":"2024-04-26T01:42:49.052858+0000","last_clean":"2024-04-26T01:42:49.052858+0000","last_became_active":"2024-04-26T01:42:05.384856+0000","last_became_peered":"2024-04-26T01:42:05.384856+0000","last_unstale":"2024-04-26T01:42:49.052858+0000","last_undegraded":"2024-04-26T01:42:49.052858+0000","last_fullsized":"2024-04-26T01:42:49.052858+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:07:23.461324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070057+0000","last_change":"2024-04-26T01:42:05.404754+0000","last_active":"2024-04-26T01:42:49.070057+0000","last_peered":"2024-04-26T01:42:49.070057+0000","last_clean":"2024-04-26T01:42:49.070057+0000","last_became_active":"2024-04-26T01:42:05.404220+0000","last_became_peered":"2024-04-26T01:42:05.404220+0000","last_unstale":"2024-04-26T01:42:49.070057+0000","last_undegraded":"2024-04-26T01:42:49.070057+0000","last_fullsized":"2024-04-26T01:42:49.070057+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:50.243552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054504+0000","last_change":"2024-04-26T01:42:05.390972+0000","last_active":"2024-04-26T01:42:49.054504+0000","last_peered":"2024-04-26T01:42:49.054504+0000","last_clean":"2024-04-26T01:42:49.054504+0000","last_became_active":"2024-04-26T01:42:05.390599+0000","last_became_peered":"2024-04-26T01:42:05.390599+0000","last_unstale":"2024-04-26T01:42:49.054504+0000","last_undegraded":"2024-04-26T01:42:49.054504+0000","last_fullsized":"2024-04-26T01:42:49.054504+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:19:46.328640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054966+0000","last_change":"2024-04-26T01:42:05.392320+0000","last_active":"2024-04-26T01:42:49.054966+0000","last_peered":"2024-04-26T01:42:49.054966+0000","last_clean":"2024-04-26T01:42:49.054966+0000","last_became_active":"2024-04-26T01:42:05.391986+0000","last_became_peered":"2024-04-26T01:42:05.391986+0000","last_unstale":"2024-04-26T01:42:49.054966+0000","last_undegraded":"2024-04-26T01:42:49.054966+0000","last_fullsized":"2024-04-26T01:42:49.054966+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:17:27.302358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053828+0000","last_change":"2024-04-26T01:42:05.388077+0000","last_active":"2024-04-26T01:42:49.053828+0000","last_peered":"2024-04-26T01:42:49.053828+0000","last_clean":"2024-04-26T01:42:49.053828+0000","last_became_active":"2024-04-26T01:42:05.387545+0000","last_became_peered":"2024-04-26T01:42:05.387545+0000","last_unstale":"2024-04-26T01:42:49.053828+0000","last_undegraded":"2024-04-26T01:42:49.053828+0000","last_fullsized":"2024-04-26T01:42:49.053828+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:51.288177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.072023+0000","last_change":"2024-04-26T01:42:49.067449+0000","last_active":"2024-04-26T01:42:49.072023+0000","last_peered":"2024-04-26T01:42:49.072023+0000","last_clean":"2024-04-26T01:42:49.072023+0000","last_became_active":"2024-04-26T01:42:49.065991+0000","last_became_peered":"2024-04-26T01:42:49.065991+0000","last_unstale":"2024-04-26T01:42:49.072023+0000","last_undegraded":"2024-04-26T01:42:49.072023+0000","last_fullsized":"2024-04-26T01:42:49.072023+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:13:36.565257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.067608+0000","last_change":"2024-04-26T01:42:05.402847+0000","last_active":"2024-04-26T01:42:49.067608+0000","last_peered":"2024-04-26T01:42:49.067608+0000","last_clean":"2024-04-26T01:42:49.067608+0000","last_became_active":"2024-04-26T01:42:05.402503+0000","last_became_peered":"2024-04-26T01:42:05.402503+0000","last_unstale":"2024-04-26T01:42:49.067608+0000","last_undegraded":"2024-04-26T01:42:49.067608+0000","last_fullsized":"2024-04-26T01:42:49.067608+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:06.935060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058574+0000","last_change":"2024-04-26T01:42:05.405257+0000","last_active":"2024-04-26T01:42:49.058574+0000","last_peered":"2024-04-26T01:42:49.058574+0000","last_clean":"2024-04-26T01:42:49.058574+0000","last_became_active":"2024-04-26T01:42:05.404551+0000","last_became_peered":"2024-04-26T01:42:05.404551+0000","last_unstale":"2024-04-26T01:42:49.058574+0000","last_undegraded":"2024-04-26T01:42:49.058574+0000","last_fullsized":"2024-04-26T01:42:49.058574+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:20.754088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.056662+0000","last_change":"2024-04-26T01:42:05.408847+0000","last_active":"2024-04-26T01:42:49.056662+0000","last_peered":"2024-04-26T01:42:49.056662+0000","last_clean":"2024-04-26T01:42:49.056662+0000","last_became_active":"2024-04-26T01:42:05.406189+0000","last_became_peered":"2024-04-26T01:42:05.406189+0000","last_unstale":"2024-04-26T01:42:49.056662+0000","last_undegraded":"2024-04-26T01:42:49.056662+0000","last_fullsized":"2024-04-26T01:42:49.056662+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:59:58.633630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053634+0000","last_change":"2024-04-26T01:42:05.404647+0000","last_active":"2024-04-26T01:42:49.053634+0000","last_peered":"2024-04-26T01:42:49.053634+0000","last_clean":"2024-04-26T01:42:49.053634+0000","last_became_active":"2024-04-26T01:42:05.404265+0000","last_became_peered":"2024-04-26T01:42:05.404265+0000","last_unstale":"2024-04-26T01:42:49.053634+0000","last_undegraded":"2024-04-26T01:42:49.053634+0000","last_fullsized":"2024-04-26T01:42:49.053634+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:00:43.472541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058658+0000","last_change":"2024-04-26T01:42:05.417602+0000","last_active":"2024-04-26T01:42:49.058658+0000","last_peered":"2024-04-26T01:42:49.058658+0000","last_clean":"2024-04-26T01:42:49.058658+0000","last_became_active":"2024-04-26T01:42:05.417144+0000","last_became_peered":"2024-04-26T01:42:05.417144+0000","last_unstale":"2024-04-26T01:42:49.058658+0000","last_undegraded":"2024-04-26T01:42:49.058658+0000","last_fullsized":"2024-04-26T01:42:49.058658+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:34.043597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054097+0000","last_change":"2024-04-26T01:42:05.392948+0000","last_active":"2024-04-26T01:42:49.054097+0000","last_peered":"2024-04-26T01:42:49.054097+0000","last_clean":"2024-04-26T01:42:49.054097+0000","last_became_active":"2024-04-26T01:42:05.392040+0000","last_became_peered":"2024-04-26T01:42:05.392040+0000","last_unstale":"2024-04-26T01:42:49.054097+0000","last_undegraded":"2024-04-26T01:42:49.054097+0000","last_fullsized":"2024-04-26T01:42:49.054097+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:17.307979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058740+0000","last_change":"2024-04-26T01:42:05.406588+0000","last_active":"2024-04-26T01:42:49.058740+0000","last_peered":"2024-04-26T01:42:49.058740+0000","last_clean":"2024-04-26T01:42:49.058740+0000","last_became_active":"2024-04-26T01:42:05.406200+0000","last_became_peered":"2024-04-26T01:42:05.406200+0000","last_unstale":"2024-04-26T01:42:49.058740+0000","last_undegraded":"2024-04-26T01:42:49.058740+0000","last_fullsized":"2024-04-26T01:42:49.058740+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:14.095383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054175+0000","last_change":"2024-04-26T01:42:01.339289+0000","last_active":"2024-04-26T01:42:49.054175+0000","last_peered":"2024-04-26T01:42:49.054175+0000","last_clean":"2024-04-26T01:42:49.054175+0000","last_became_active":"2024-04-26T01:42:01.335402+0000","last_became_peered":"2024-04-26T01:42:01.335402+0000","last_unstale":"2024-04-26T01:42:49.054175+0000","last_undegraded":"2024-04-26T01:42:49.054175+0000","last_fullsized":"2024-04-26T01:42:49.054175+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:51.411524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.071724+0000","last_change":"2024-04-26T01:42:05.409569+0000","last_active":"2024-04-26T01:42:49.071724+0000","last_peered":"2024-04-26T01:42:49.071724+0000","last_clean":"2024-04-26T01:42:49.071724+0000","last_became_active":"2024-04-26T01:42:05.407441+0000","last_became_peered":"2024-04-26T01:42:05.407441+0000","last_unstale":"2024-04-26T01:42:49.071724+0000","last_undegraded":"2024-04-26T01:42:49.071724+0000","last_fullsized":"2024-04-26T01:42:49.071724+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:36:58.043359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.072731+0000","last_change":"2024-04-26T01:42:02.231871+0000","last_active":"2024-04-26T01:42:49.072731+0000","last_peered":"2024-04-26T01:42:49.072731+0000","last_clean":"2024-04-26T01:42:49.072731+0000","last_became_active":"2024-04-26T01:42:02.230813+0000","last_became_peered":"2024-04-26T01:42:02.230813+0000","last_unstale":"2024-04-26T01:42:49.072731+0000","last_undegraded":"2024-04-26T01:42:49.072731+0000","last_fullsized":"2024-04-26T01:42:49.072731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:53.117266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.062086+0000","last_change":"2024-04-26T01:42:05.412414+0000","last_active":"2024-04-26T01:42:49.062086+0000","last_peered":"2024-04-26T01:42:49.062086+0000","last_clean":"2024-04-26T01:42:49.062086+0000","last_became_active":"2024-04-26T01:42:05.412057+0000","last_became_peered":"2024-04-26T01:42:05.412057+0000","last_unstale":"2024-04-26T01:42:49.062086+0000","last_undegraded":"2024-04-26T01:42:49.062086+0000","last_fullsized":"2024-04-26T01:42:49.062086+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:51:46.331246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.056301+0000","last_change":"2024-04-26T01:42:01.329957+0000","last_active":"2024-04-26T01:42:49.056301+0000","last_peered":"2024-04-26T01:42:49.056301+0000","last_clean":"2024-04-26T01:42:49.056301+0000","last_became_active":"2024-04-26T01:42:01.329289+0000","last_became_peered":"2024-04-26T01:42:01.329289+0000","last_unstale":"2024-04-26T01:42:49.056301+0000","last_undegraded":"2024-04-26T01:42:49.056301+0000","last_fullsized":"2024-04-26T01:42:49.056301+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:46:38.983138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054482+0000","last_change":"2024-04-26T01:42:05.399438+0000","last_active":"2024-04-26T01:42:49.054482+0000","last_peered":"2024-04-26T01:42:49.054482+0000","last_clean":"2024-04-26T01:42:49.054482+0000","last_became_active":"2024-04-26T01:42:05.399082+0000","last_became_peered":"2024-04-26T01:42:05.399082+0000","last_unstale":"2024-04-26T01:42:49.054482+0000","last_undegraded":"2024-04-26T01:42:49.054482+0000","last_fullsized":"2024-04-26T01:42:49.054482+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:13.619893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"91'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.056135+0000","last_change":"2024-04-26T01:42:01.328230+0000","last_active":"2024-04-26T01:42:49.056135+0000","last_peered":"2024-04-26T01:42:49.056135+0000","last_clean":"2024-04-26T01:42:49.056135+0000","last_became_active":"2024-04-26T01:42:01.327695+0000","last_became_peered":"2024-04-26T01:42:01.327695+0000","last_unstale":"2024-04-26T01:42:49.056135+0000","last_undegraded":"2024-04-26T01:42:49.056135+0000","last_fullsized":"2024-04-26T01:42:49.056135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:06.379749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.071822+0000","last_change":"2024-04-26T01:42:49.066057+0000","last_active":"2024-04-26T01:42:49.071822+0000","last_peered":"2024-04-26T01:42:49.071822+0000","last_clean":"2024-04-26T01:42:49.071822+0000","last_became_active":"2024-04-26T01:42:49.064684+0000","last_became_peered":"2024-04-26T01:42:49.064684+0000","last_unstale":"2024-04-26T01:42:49.071822+0000","last_undegraded":"2024-04-26T01:42:49.071822+0000","last_fullsized":"2024-04-26T01:42:49.071822+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:39.159528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:57.574391+0000","last_change":"2024-04-26T01:42:01.332623+0000","last_active":"2024-04-26T01:42:57.574391+0000","last_peered":"2024-04-26T01:42:57.574391+0000","last_clean":"2024-04-26T01:42:57.574391+0000","last_became_active":"2024-04-26T01:42:01.332227+0000","last_became_peered":"2024-04-26T01:42:01.332227+0000","last_unstale":"2024-04-26T01:42:57.574391+0000","last_undegraded":"2024-04-26T01:42:57.574391+0000","last_fullsized":"2024-04-26T01:42:57.574391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:58.611464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.055737+0000","last_change":"2024-04-26T01:42:05.396133+0000","last_active":"2024-04-26T01:42:49.055737+0000","last_peered":"2024-04-26T01:42:49.055737+0000","last_clean":"2024-04-26T01:42:49.055737+0000","last_became_active":"2024-04-26T01:42:05.395553+0000","last_became_peered":"2024-04-26T01:42:05.395553+0000","last_unstale":"2024-04-26T01:42:49.055737+0000","last_undegraded":"2024-04-26T01:42:49.055737+0000","last_fullsized":"2024-04-26T01:42:49.055737+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:36:01.363909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054479+0000","last_change":"2024-04-26T01:42:01.340624+0000","last_active":"2024-04-26T01:42:49.054479+0000","last_peered":"2024-04-26T01:42:49.054479+0000","last_clean":"2024-04-26T01:42:49.054479+0000","last_became_active":"2024-04-26T01:42:01.340158+0000","last_became_peered":"2024-04-26T01:42:01.340158+0000","last_unstale":"2024-04-26T01:42:49.054479+0000","last_undegraded":"2024-04-26T01:42:49.054479+0000","last_fullsized":"2024-04-26T01:42:49.054479+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:47:05.529922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050739+0000","last_change":"2024-04-26T01:42:05.406354+0000","last_active":"2024-04-26T01:42:49.050739+0000","last_peered":"2024-04-26T01:42:49.050739+0000","last_clean":"2024-04-26T01:42:49.050739+0000","last_became_active":"2024-04-26T01:42:05.405989+0000","last_became_peered":"2024-04-26T01:42:05.405989+0000","last_unstale":"2024-04-26T01:42:49.050739+0000","last_undegraded":"2024-04-26T01:42:49.050739+0000","last_fullsized":"2024-04-26T01:42:49.050739+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:43.115225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070125+0000","last_change":"2024-04-26T01:42:01.340549+0000","last_active":"2024-04-26T01:42:49.070125+0000","last_peered":"2024-04-26T01:42:49.070125+0000","last_clean":"2024-04-26T01:42:49.070125+0000","last_became_active":"2024-04-26T01:42:01.339861+0000","last_became_peered":"2024-04-26T01:42:01.339861+0000","last_unstale":"2024-04-26T01:42:49.070125+0000","last_undegraded":"2024-04-26T01:42:49.070125+0000","last_fullsized":"2024-04-26T01:42:49.070125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:02:53.154367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070461+0000","last_change":"2024-04-26T01:42:05.402664+0000","last_active":"2024-04-26T01:42:49.070461+0000","last_peered":"2024-04-26T01:42:49.070461+0000","last_clean":"2024-04-26T01:42:49.070461+0000","last_became_active":"2024-04-26T01:42:05.400785+0000","last_became_peered":"2024-04-26T01:42:05.400785+0000","last_unstale":"2024-04-26T01:42:49.070461+0000","last_undegraded":"2024-04-26T01:42:49.070461+0000","last_fullsized":"2024-04-26T01:42:49.070461+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:40:32.137752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050300+0000","last_change":"2024-04-26T01:42:01.328600+0000","last_active":"2024-04-26T01:42:49.050300+0000","last_peered":"2024-04-26T01:42:49.050300+0000","last_clean":"2024-04-26T01:42:49.050300+0000","last_became_active":"2024-04-26T01:42:01.328149+0000","last_became_peered":"2024-04-26T01:42:01.328149+0000","last_unstale":"2024-04-26T01:42:49.050300+0000","last_undegraded":"2024-04-26T01:42:49.050300+0000","last_fullsized":"2024-04-26T01:42:49.050300+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:45:42.811088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.065824+0000","last_change":"2024-04-26T01:42:49.065824+0000","last_active":"2024-04-26T01:42:49.065824+0000","last_peered":"2024-04-26T01:42:49.065824+0000","last_clean":"2024-04-26T01:42:49.065824+0000","last_became_active":"2024-04-26T01:42:49.064670+0000","last_became_peered":"2024-04-26T01:42:49.064670+0000","last_unstale":"2024-04-26T01:42:49.065824+0000","last_undegraded":"2024-04-26T01:42:49.065824+0000","last_fullsized":"2024-04-26T01:42:49.065824+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:44.790184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:57.573648+0000","last_change":"2024-04-26T01:42:01.335438+0000","last_active":"2024-04-26T01:42:57.573648+0000","last_peered":"2024-04-26T01:42:57.573648+0000","last_clean":"2024-04-26T01:42:57.573648+0000","last_became_active":"2024-04-26T01:42:01.334962+0000","last_became_peered":"2024-04-26T01:42:01.334962+0000","last_unstale":"2024-04-26T01:42:57.573648+0000","last_undegraded":"2024-04-26T01:42:57.573648+0000","last_fullsized":"2024-04-26T01:42:57.573648+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:35:08.252274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054352+0000","last_change":"2024-04-26T01:42:05.393294+0000","last_active":"2024-04-26T01:42:49.054352+0000","last_peered":"2024-04-26T01:42:49.054352+0000","last_clean":"2024-04-26T01:42:49.054352+0000","last_became_active":"2024-04-26T01:42:05.392974+0000","last_became_peered":"2024-04-26T01:42:05.392974+0000","last_unstale":"2024-04-26T01:42:49.054352+0000","last_undegraded":"2024-04-26T01:42:49.054352+0000","last_fullsized":"2024-04-26T01:42:49.054352+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:58.188706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053917+0000","last_change":"2024-04-26T01:42:01.333505+0000","last_active":"2024-04-26T01:42:49.053917+0000","last_peered":"2024-04-26T01:42:49.053917+0000","last_clean":"2024-04-26T01:42:49.053917+0000","last_became_active":"2024-04-26T01:42:01.333062+0000","last_became_peered":"2024-04-26T01:42:01.333062+0000","last_unstale":"2024-04-26T01:42:49.053917+0000","last_undegraded":"2024-04-26T01:42:49.053917+0000","last_fullsized":"2024-04-26T01:42:49.053917+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:50:48.804443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.063948+0000","last_change":"2024-04-26T01:42:49.063948+0000","last_active":"2024-04-26T01:42:49.063948+0000","last_peered":"2024-04-26T01:42:49.063948+0000","last_clean":"2024-04-26T01:42:49.063948+0000","last_became_active":"2024-04-26T01:42:49.060852+0000","last_became_peered":"2024-04-26T01:42:49.060852+0000","last_unstale":"2024-04-26T01:42:49.063948+0000","last_undegraded":"2024-04-26T01:42:49.063948+0000","last_fullsized":"2024-04-26T01:42:49.063948+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.370431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050957+0000","last_change":"2024-04-26T01:42:02.232308+0000","last_active":"2024-04-26T01:42:49.050957+0000","last_peered":"2024-04-26T01:42:49.050957+0000","last_clean":"2024-04-26T01:42:49.050957+0000","last_became_active":"2024-04-26T01:42:02.231897+0000","last_became_peered":"2024-04-26T01:42:02.231897+0000","last_unstale":"2024-04-26T01:42:49.050957+0000","last_undegraded":"2024-04-26T01:42:49.050957+0000","last_fullsized":"2024-04-26T01:42:49.050957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:36:16.847087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070798+0000","last_change":"2024-04-26T01:42:49.070798+0000","last_active":"2024-04-26T01:42:49.070798+0000","last_peered":"2024-04-26T01:42:49.070798+0000","last_clean":"2024-04-26T01:42:49.070798+0000","last_became_active":"2024-04-26T01:42:49.070470+0000","last_became_peered":"2024-04-26T01:42:49.070470+0000","last_unstale":"2024-04-26T01:42:49.070798+0000","last_undegraded":"2024-04-26T01:42:49.070798+0000","last_fullsized":"2024-04-26T01:42:49.070798+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:04:22.103704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"93'29","reported_seq":127,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:58.565159+0000","last_change":"2024-04-26T01:42:01.330822+0000","last_active":"2024-04-26T01:42:58.565159+0000","last_peered":"2024-04-26T01:42:58.565159+0000","last_clean":"2024-04-26T01:42:58.565159+0000","last_became_active":"2024-04-26T01:42:01.330337+0000","last_became_peered":"2024-04-26T01:42:01.330337+0000","last_unstale":"2024-04-26T01:42:58.565159+0000","last_undegraded":"2024-04-26T01:42:58.565159+0000","last_fullsized":"2024-04-26T01:42:58.565159+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:04.024645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58815,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":93,"num_write":30,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.056114+0000","last_change":"2024-04-26T01:42:05.390601+0000","last_active":"2024-04-26T01:42:49.056114+0000","last_peered":"2024-04-26T01:42:49.056114+0000","last_clean":"2024-04-26T01:42:49.056114+0000","last_became_active":"2024-04-26T01:42:05.390279+0000","last_became_peered":"2024-04-26T01:42:05.390279+0000","last_unstale":"2024-04-26T01:42:49.056114+0000","last_undegraded":"2024-04-26T01:42:49.056114+0000","last_fullsized":"2024-04-26T01:42:49.056114+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:20.171769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058226+0000","last_change":"2024-04-26T01:42:01.337200+0000","last_active":"2024-04-26T01:42:49.058226+0000","last_peered":"2024-04-26T01:42:49.058226+0000","last_clean":"2024-04-26T01:42:49.058226+0000","last_became_active":"2024-04-26T01:42:01.336919+0000","last_became_peered":"2024-04-26T01:42:01.336919+0000","last_unstale":"2024-04-26T01:42:49.058226+0000","last_undegraded":"2024-04-26T01:42:49.058226+0000","last_fullsized":"2024-04-26T01:42:49.058226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:29.799903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.067939+0000","last_change":"2024-04-26T01:42:05.410081+0000","last_active":"2024-04-26T01:42:49.067939+0000","last_peered":"2024-04-26T01:42:49.067939+0000","last_clean":"2024-04-26T01:42:49.067939+0000","last_became_active":"2024-04-26T01:42:05.409341+0000","last_became_peered":"2024-04-26T01:42:05.409341+0000","last_unstale":"2024-04-26T01:42:49.067939+0000","last_undegraded":"2024-04-26T01:42:49.067939+0000","last_fullsized":"2024-04-26T01:42:49.067939+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:25:31.925296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050589+0000","last_change":"2024-04-26T01:42:01.331702+0000","last_active":"2024-04-26T01:42:49.050589+0000","last_peered":"2024-04-26T01:42:49.050589+0000","last_clean":"2024-04-26T01:42:49.050589+0000","last_became_active":"2024-04-26T01:42:01.331311+0000","last_became_peered":"2024-04-26T01:42:01.331311+0000","last_unstale":"2024-04-26T01:42:49.050589+0000","last_undegraded":"2024-04-26T01:42:49.050589+0000","last_fullsized":"2024-04-26T01:42:49.050589+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:54:49.513627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053620+0000","last_change":"2024-04-26T01:42:05.403957+0000","last_active":"2024-04-26T01:42:49.053620+0000","last_peered":"2024-04-26T01:42:49.053620+0000","last_clean":"2024-04-26T01:42:49.053620+0000","last_became_active":"2024-04-26T01:42:05.403583+0000","last_became_peered":"2024-04-26T01:42:05.403583+0000","last_unstale":"2024-04-26T01:42:49.053620+0000","last_undegraded":"2024-04-26T01:42:49.053620+0000","last_fullsized":"2024-04-26T01:42:49.053620+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:12:20.095407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:55.561022+0000","last_change":"2024-04-26T01:42:01.332456+0000","last_active":"2024-04-26T01:42:55.561022+0000","last_peered":"2024-04-26T01:42:55.561022+0000","last_clean":"2024-04-26T01:42:55.561022+0000","last_became_active":"2024-04-26T01:42:01.331910+0000","last_became_peered":"2024-04-26T01:42:01.331910+0000","last_unstale":"2024-04-26T01:42:55.561022+0000","last_undegraded":"2024-04-26T01:42:55.561022+0000","last_fullsized":"2024-04-26T01:42:55.561022+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:13:25.368205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.062130+0000","last_change":"2024-04-26T01:42:49.062130+0000","last_active":"2024-04-26T01:42:49.062130+0000","last_peered":"2024-04-26T01:42:49.062130+0000","last_clean":"2024-04-26T01:42:49.062130+0000","last_became_active":"2024-04-26T01:42:49.061627+0000","last_became_peered":"2024-04-26T01:42:49.061627+0000","last_unstale":"2024-04-26T01:42:49.062130+0000","last_undegraded":"2024-04-26T01:42:49.062130+0000","last_fullsized":"2024-04-26T01:42:49.062130+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:16.101940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.083062+0000","last_change":"2024-04-26T01:42:49.082618+0000","last_active":"2024-04-26T01:42:49.083062+0000","last_peered":"2024-04-26T01:42:49.083062+0000","last_clean":"2024-04-26T01:42:49.083062+0000","last_became_active":"2024-04-26T01:42:49.064654+0000","last_became_peered":"2024-04-26T01:42:49.064654+0000","last_unstale":"2024-04-26T01:42:49.083062+0000","last_undegraded":"2024-04-26T01:42:49.083062+0000","last_fullsized":"2024-04-26T01:42:49.083062+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:13:23.937059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050479+0000","last_change":"2024-04-26T01:42:05.391392+0000","last_active":"2024-04-26T01:42:49.050479+0000","last_peered":"2024-04-26T01:42:49.050479+0000","last_clean":"2024-04-26T01:42:49.050479+0000","last_became_active":"2024-04-26T01:42:05.390939+0000","last_became_peered":"2024-04-26T01:42:05.390939+0000","last_unstale":"2024-04-26T01:42:49.050479+0000","last_undegraded":"2024-04-26T01:42:49.050479+0000","last_fullsized":"2024-04-26T01:42:49.050479+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:36.141915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.056422+0000","last_change":"2024-04-26T01:42:01.332932+0000","last_active":"2024-04-26T01:42:49.056422+0000","last_peered":"2024-04-26T01:42:49.056422+0000","last_clean":"2024-04-26T01:42:49.056422+0000","last_became_active":"2024-04-26T01:42:01.332626+0000","last_became_peered":"2024-04-26T01:42:01.332626+0000","last_unstale":"2024-04-26T01:42:49.056422+0000","last_undegraded":"2024-04-26T01:42:49.056422+0000","last_fullsized":"2024-04-26T01:42:49.056422+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:20.593730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058056+0000","last_change":"2024-04-26T01:42:05.394925+0000","last_active":"2024-04-26T01:42:49.058056+0000","last_peered":"2024-04-26T01:42:49.058056+0000","last_clean":"2024-04-26T01:42:49.058056+0000","last_became_active":"2024-04-26T01:42:05.394531+0000","last_became_peered":"2024-04-26T01:42:05.394531+0000","last_unstale":"2024-04-26T01:42:49.058056+0000","last_undegraded":"2024-04-26T01:42:49.058056+0000","last_fullsized":"2024-04-26T01:42:49.058056+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:33.177780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054580+0000","last_change":"2024-04-26T01:42:02.237482+0000","last_active":"2024-04-26T01:42:49.054580+0000","last_peered":"2024-04-26T01:42:49.054580+0000","last_clean":"2024-04-26T01:42:49.054580+0000","last_became_active":"2024-04-26T01:42:02.236752+0000","last_became_peered":"2024-04-26T01:42:02.236752+0000","last_unstale":"2024-04-26T01:42:49.054580+0000","last_undegraded":"2024-04-26T01:42:49.054580+0000","last_fullsized":"2024-04-26T01:42:49.054580+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:27.211245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'1","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054655+0000","last_change":"2024-04-26T01:42:05.402157+0000","last_active":"2024-04-26T01:42:49.054655+0000","last_peered":"2024-04-26T01:42:49.054655+0000","last_clean":"2024-04-26T01:42:49.054655+0000","last_became_active":"2024-04-26T01:42:05.401889+0000","last_became_peered":"2024-04-26T01:42:05.401889+0000","last_unstale":"2024-04-26T01:42:49.054655+0000","last_undegraded":"2024-04-26T01:42:49.054655+0000","last_fullsized":"2024-04-26T01:42:49.054655+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:33.854901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053243+0000","last_change":"2024-04-26T01:42:01.339675+0000","last_active":"2024-04-26T01:42:49.053243+0000","last_peered":"2024-04-26T01:42:49.053243+0000","last_clean":"2024-04-26T01:42:49.053243+0000","last_became_active":"2024-04-26T01:42:01.339247+0000","last_became_peered":"2024-04-26T01:42:01.339247+0000","last_unstale":"2024-04-26T01:42:49.053243+0000","last_undegraded":"2024-04-26T01:42:49.053243+0000","last_fullsized":"2024-04-26T01:42:49.053243+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:32.724830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054507+0000","last_change":"2024-04-26T01:42:05.412913+0000","last_active":"2024-04-26T01:42:49.054507+0000","last_peered":"2024-04-26T01:42:49.054507+0000","last_clean":"2024-04-26T01:42:49.054507+0000","last_became_active":"2024-04-26T01:42:05.412556+0000","last_became_peered":"2024-04-26T01:42:05.412556+0000","last_unstale":"2024-04-26T01:42:49.054507+0000","last_undegraded":"2024-04-26T01:42:49.054507+0000","last_fullsized":"2024-04-26T01:42:49.054507+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:00:52.798069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050679+0000","last_change":"2024-04-26T01:42:05.391235+0000","last_active":"2024-04-26T01:42:49.050679+0000","last_peered":"2024-04-26T01:42:49.050679+0000","last_clean":"2024-04-26T01:42:49.050679+0000","last_became_active":"2024-04-26T01:42:05.390717+0000","last_became_peered":"2024-04-26T01:42:05.390717+0000","last_unstale":"2024-04-26T01:42:49.050679+0000","last_undegraded":"2024-04-26T01:42:49.050679+0000","last_fullsized":"2024-04-26T01:42:49.050679+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:07.541593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:59.567760+0000","last_change":"2024-04-26T01:42:02.232823+0000","last_active":"2024-04-26T01:42:59.567760+0000","last_peered":"2024-04-26T01:42:59.567760+0000","last_clean":"2024-04-26T01:42:59.567760+0000","last_became_active":"2024-04-26T01:42:02.231899+0000","last_became_peered":"2024-04-26T01:42:02.231899+0000","last_unstale":"2024-04-26T01:42:59.567760+0000","last_undegraded":"2024-04-26T01:42:59.567760+0000","last_fullsized":"2024-04-26T01:42:59.567760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:23.287305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.071180+0000","last_change":"2024-04-26T01:42:05.405690+0000","last_active":"2024-04-26T01:42:49.071180+0000","last_peered":"2024-04-26T01:42:49.071180+0000","last_clean":"2024-04-26T01:42:49.071180+0000","last_became_active":"2024-04-26T01:42:05.405345+0000","last_became_peered":"2024-04-26T01:42:05.405345+0000","last_unstale":"2024-04-26T01:42:49.071180+0000","last_undegraded":"2024-04-26T01:42:49.071180+0000","last_fullsized":"2024-04-26T01:42:49.071180+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:48.377916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050943+0000","last_change":"2024-04-26T01:42:01.335721+0000","last_active":"2024-04-26T01:42:49.050943+0000","last_peered":"2024-04-26T01:42:49.050943+0000","last_clean":"2024-04-26T01:42:49.050943+0000","last_became_active":"2024-04-26T01:42:01.335149+0000","last_became_peered":"2024-04-26T01:42:01.335149+0000","last_unstale":"2024-04-26T01:42:49.050943+0000","last_undegraded":"2024-04-26T01:42:49.050943+0000","last_fullsized":"2024-04-26T01:42:49.050943+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:36.372033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":361,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053278+0000","last_change":"2024-04-26T01:40:28.086333+0000","last_active":"2024-04-26T01:42:49.053278+0000","last_peered":"2024-04-26T01:42:49.053278+0000","last_clean":"2024-04-26T01:42:49.053278+0000","last_became_active":"2024-04-26T01:40:28.085902+0000","last_became_peered":"2024-04-26T01:40:28.085902+0000","last_unstale":"2024-04-26T01:42:49.053278+0000","last_undegraded":"2024-04-26T01:42:49.053278+0000","last_fullsized":"2024-04-26T01:42:49.053278+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_clean_scrub_stamp":"2024-04-26T01:36:35.357698+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:43.290117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.068625+0000","last_change":"2024-04-26T01:42:05.401896+0000","last_active":"2024-04-26T01:42:49.068625+0000","last_peered":"2024-04-26T01:42:49.068625+0000","last_clean":"2024-04-26T01:42:49.068625+0000","last_became_active":"2024-04-26T01:42:05.401496+0000","last_became_peered":"2024-04-26T01:42:05.401496+0000","last_unstale":"2024-04-26T01:42:49.068625+0000","last_undegraded":"2024-04-26T01:42:49.068625+0000","last_fullsized":"2024-04-26T01:42:49.068625+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:49.758444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053408+0000","last_change":"2024-04-26T01:42:01.329705+0000","last_active":"2024-04-26T01:42:49.053408+0000","last_peered":"2024-04-26T01:42:49.053408+0000","last_clean":"2024-04-26T01:42:49.053408+0000","last_became_active":"2024-04-26T01:42:01.329153+0000","last_became_peered":"2024-04-26T01:42:01.329153+0000","last_unstale":"2024-04-26T01:42:49.053408+0000","last_undegraded":"2024-04-26T01:42:49.053408+0000","last_fullsized":"2024-04-26T01:42:49.053408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:06.747378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.056180+0000","last_change":"2024-04-26T01:42:02.232148+0000","last_active":"2024-04-26T01:42:49.056180+0000","last_peered":"2024-04-26T01:42:49.056180+0000","last_clean":"2024-04-26T01:42:49.056180+0000","last_became_active":"2024-04-26T01:42:02.231739+0000","last_became_peered":"2024-04-26T01:42:02.231739+0000","last_unstale":"2024-04-26T01:42:49.056180+0000","last_undegraded":"2024-04-26T01:42:49.056180+0000","last_fullsized":"2024-04-26T01:42:49.056180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:09.752433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.068717+0000","last_change":"2024-04-26T01:42:05.399938+0000","last_active":"2024-04-26T01:42:49.068717+0000","last_peered":"2024-04-26T01:42:49.068717+0000","last_clean":"2024-04-26T01:42:49.068717+0000","last_became_active":"2024-04-26T01:42:05.399550+0000","last_became_peered":"2024-04-26T01:42:05.399550+0000","last_unstale":"2024-04-26T01:42:49.068717+0000","last_undegraded":"2024-04-26T01:42:49.068717+0000","last_fullsized":"2024-04-26T01:42:49.068717+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:44.142284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"93'12","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:52.418677+0000","last_change":"2024-04-26T01:42:01.337437+0000","last_active":"2024-04-26T01:42:52.418677+0000","last_peered":"2024-04-26T01:42:52.418677+0000","last_clean":"2024-04-26T01:42:52.418677+0000","last_became_active":"2024-04-26T01:42:01.336894+0000","last_became_peered":"2024-04-26T01:42:01.336894+0000","last_unstale":"2024-04-26T01:42:52.418677+0000","last_undegraded":"2024-04-26T01:42:52.418677+0000","last_fullsized":"2024-04-26T01:42:52.418677+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:04:31.093273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.072806+0000","last_change":"2024-04-26T01:42:05.411694+0000","last_active":"2024-04-26T01:42:49.072806+0000","last_peered":"2024-04-26T01:42:49.072806+0000","last_clean":"2024-04-26T01:42:49.072806+0000","last_became_active":"2024-04-26T01:42:05.411300+0000","last_became_peered":"2024-04-26T01:42:05.411300+0000","last_unstale":"2024-04-26T01:42:49.072806+0000","last_undegraded":"2024-04-26T01:42:49.072806+0000","last_fullsized":"2024-04-26T01:42:49.072806+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:42:08.116145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.051041+0000","last_change":"2024-04-26T01:42:01.427501+0000","last_active":"2024-04-26T01:42:49.051041+0000","last_peered":"2024-04-26T01:42:49.051041+0000","last_clean":"2024-04-26T01:42:49.051041+0000","last_became_active":"2024-04-26T01:42:01.426970+0000","last_became_peered":"2024-04-26T01:42:01.426970+0000","last_unstale":"2024-04-26T01:42:49.051041+0000","last_undegraded":"2024-04-26T01:42:49.051041+0000","last_fullsized":"2024-04-26T01:42:49.051041+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:53:53.118296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.061715+0000","last_change":"2024-04-26T01:42:05.414790+0000","last_active":"2024-04-26T01:42:49.061715+0000","last_peered":"2024-04-26T01:42:49.061715+0000","last_clean":"2024-04-26T01:42:49.061715+0000","last_became_active":"2024-04-26T01:42:05.411947+0000","last_became_peered":"2024-04-26T01:42:05.411947+0000","last_unstale":"2024-04-26T01:42:49.061715+0000","last_undegraded":"2024-04-26T01:42:49.061715+0000","last_fullsized":"2024-04-26T01:42:49.061715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:11:34.450815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053787+0000","last_change":"2024-04-26T01:42:02.230539+0000","last_active":"2024-04-26T01:42:49.053787+0000","last_peered":"2024-04-26T01:42:49.053787+0000","last_clean":"2024-04-26T01:42:49.053787+0000","last_became_active":"2024-04-26T01:42:02.230150+0000","last_became_peered":"2024-04-26T01:42:02.230150+0000","last_unstale":"2024-04-26T01:42:49.053787+0000","last_undegraded":"2024-04-26T01:42:49.053787+0000","last_fullsized":"2024-04-26T01:42:49.053787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:08:03.073069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.055881+0000","last_change":"2024-04-26T01:42:05.388877+0000","last_active":"2024-04-26T01:42:49.055881+0000","last_peered":"2024-04-26T01:42:49.055881+0000","last_clean":"2024-04-26T01:42:49.055881+0000","last_became_active":"2024-04-26T01:42:05.388457+0000","last_became_peered":"2024-04-26T01:42:05.388457+0000","last_unstale":"2024-04-26T01:42:49.055881+0000","last_undegraded":"2024-04-26T01:42:49.055881+0000","last_fullsized":"2024-04-26T01:42:49.055881+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:54.436859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070848+0000","last_change":"2024-04-26T01:42:01.343461+0000","last_active":"2024-04-26T01:42:49.070848+0000","last_peered":"2024-04-26T01:42:49.070848+0000","last_clean":"2024-04-26T01:42:49.070848+0000","last_became_active":"2024-04-26T01:42:01.342713+0000","last_became_peered":"2024-04-26T01:42:01.342713+0000","last_unstale":"2024-04-26T01:42:49.070848+0000","last_undegraded":"2024-04-26T01:42:49.070848+0000","last_fullsized":"2024-04-26T01:42:49.070848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:38.471858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058169+0000","last_change":"2024-04-26T01:42:05.415101+0000","last_active":"2024-04-26T01:42:49.058169+0000","last_peered":"2024-04-26T01:42:49.058169+0000","last_clean":"2024-04-26T01:42:49.058169+0000","last_became_active":"2024-04-26T01:42:05.414745+0000","last_became_peered":"2024-04-26T01:42:05.414745+0000","last_unstale":"2024-04-26T01:42:49.058169+0000","last_undegraded":"2024-04-26T01:42:49.058169+0000","last_fullsized":"2024-04-26T01:42:49.058169+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:13:08.573547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.057699+0000","last_change":"2024-04-26T01:42:01.339501+0000","last_active":"2024-04-26T01:42:49.057699+0000","last_peered":"2024-04-26T01:42:49.057699+0000","last_clean":"2024-04-26T01:42:49.057699+0000","last_became_active":"2024-04-26T01:42:01.339132+0000","last_became_peered":"2024-04-26T01:42:01.339132+0000","last_unstale":"2024-04-26T01:42:49.057699+0000","last_undegraded":"2024-04-26T01:42:49.057699+0000","last_fullsized":"2024-04-26T01:42:49.057699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:16.463099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070531+0000","last_change":"2024-04-26T01:42:05.417864+0000","last_active":"2024-04-26T01:42:49.070531+0000","last_peered":"2024-04-26T01:42:49.070531+0000","last_clean":"2024-04-26T01:42:49.070531+0000","last_became_active":"2024-04-26T01:42:05.417414+0000","last_became_peered":"2024-04-26T01:42:05.417414+0000","last_unstale":"2024-04-26T01:42:49.070531+0000","last_undegraded":"2024-04-26T01:42:49.070531+0000","last_fullsized":"2024-04-26T01:42:49.070531+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:23:12.793421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.063769+0000","last_change":"2024-04-26T01:42:49.063769+0000","last_active":"2024-04-26T01:42:49.063769+0000","last_peered":"2024-04-26T01:42:49.063769+0000","last_clean":"2024-04-26T01:42:49.063769+0000","last_became_active":"2024-04-26T01:42:49.057463+0000","last_became_peered":"2024-04-26T01:42:49.057463+0000","last_unstale":"2024-04-26T01:42:49.063769+0000","last_undegraded":"2024-04-26T01:42:49.063769+0000","last_fullsized":"2024-04-26T01:42:49.063769+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:34:54.511304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053377+0000","last_change":"2024-04-26T01:42:05.397980+0000","last_active":"2024-04-26T01:42:49.053377+0000","last_peered":"2024-04-26T01:42:49.053377+0000","last_clean":"2024-04-26T01:42:49.053377+0000","last_became_active":"2024-04-26T01:42:05.397589+0000","last_became_peered":"2024-04-26T01:42:05.397589+0000","last_unstale":"2024-04-26T01:42:49.053377+0000","last_undegraded":"2024-04-26T01:42:49.053377+0000","last_fullsized":"2024-04-26T01:42:49.053377+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:39:55.863566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058561+0000","last_change":"2024-04-26T01:42:02.234313+0000","last_active":"2024-04-26T01:42:49.058561+0000","last_peered":"2024-04-26T01:42:49.058561+0000","last_clean":"2024-04-26T01:42:49.058561+0000","last_became_active":"2024-04-26T01:42:02.233873+0000","last_became_peered":"2024-04-26T01:42:02.233873+0000","last_unstale":"2024-04-26T01:42:49.058561+0000","last_undegraded":"2024-04-26T01:42:49.058561+0000","last_fullsized":"2024-04-26T01:42:49.058561+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:34.092434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053962+0000","last_change":"2024-04-26T01:42:05.387435+0000","last_active":"2024-04-26T01:42:49.053962+0000","last_peered":"2024-04-26T01:42:49.053962+0000","last_clean":"2024-04-26T01:42:49.053962+0000","last_became_active":"2024-04-26T01:42:05.386964+0000","last_became_peered":"2024-04-26T01:42:05.386964+0000","last_unstale":"2024-04-26T01:42:49.053962+0000","last_undegraded":"2024-04-26T01:42:49.053962+0000","last_fullsized":"2024-04-26T01:42:49.053962+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:12:27.905842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050721+0000","last_change":"2024-04-26T01:42:01.326432+0000","last_active":"2024-04-26T01:42:49.050721+0000","last_peered":"2024-04-26T01:42:49.050721+0000","last_clean":"2024-04-26T01:42:49.050721+0000","last_became_active":"2024-04-26T01:42:01.325989+0000","last_became_peered":"2024-04-26T01:42:01.325989+0000","last_unstale":"2024-04-26T01:42:49.050721+0000","last_undegraded":"2024-04-26T01:42:49.050721+0000","last_fullsized":"2024-04-26T01:42:49.050721+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:10.049563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'1","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053551+0000","last_change":"2024-04-26T01:42:05.412802+0000","last_active":"2024-04-26T01:42:49.053551+0000","last_peered":"2024-04-26T01:42:49.053551+0000","last_clean":"2024-04-26T01:42:49.053551+0000","last_became_active":"2024-04-26T01:42:05.412443+0000","last_became_peered":"2024-04-26T01:42:05.412443+0000","last_unstale":"2024-04-26T01:42:49.053551+0000","last_undegraded":"2024-04-26T01:42:49.053551+0000","last_fullsized":"2024-04-26T01:42:49.053551+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:28:56.360778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.080305+0000","last_change":"2024-04-26T01:42:49.080305+0000","last_active":"2024-04-26T01:42:49.080305+0000","last_peered":"2024-04-26T01:42:49.080305+0000","last_clean":"2024-04-26T01:42:49.080305+0000","last_became_active":"2024-04-26T01:42:49.074476+0000","last_became_peered":"2024-04-26T01:42:49.074476+0000","last_unstale":"2024-04-26T01:42:49.080305+0000","last_undegraded":"2024-04-26T01:42:49.080305+0000","last_fullsized":"2024-04-26T01:42:49.080305+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:58:06.619961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054258+0000","last_change":"2024-04-26T01:42:05.387621+0000","last_active":"2024-04-26T01:42:49.054258+0000","last_peered":"2024-04-26T01:42:49.054258+0000","last_clean":"2024-04-26T01:42:49.054258+0000","last_became_active":"2024-04-26T01:42:05.387165+0000","last_became_peered":"2024-04-26T01:42:05.387165+0000","last_unstale":"2024-04-26T01:42:49.054258+0000","last_undegraded":"2024-04-26T01:42:49.054258+0000","last_fullsized":"2024-04-26T01:42:49.054258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:37.803330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058443+0000","last_change":"2024-04-26T01:42:05.386787+0000","last_active":"2024-04-26T01:42:49.058443+0000","last_peered":"2024-04-26T01:42:49.058443+0000","last_clean":"2024-04-26T01:42:49.058443+0000","last_became_active":"2024-04-26T01:42:05.386410+0000","last_became_peered":"2024-04-26T01:42:05.386410+0000","last_unstale":"2024-04-26T01:42:49.058443+0000","last_undegraded":"2024-04-26T01:42:49.058443+0000","last_fullsized":"2024-04-26T01:42:49.058443+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:42.137191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058087+0000","last_change":"2024-04-26T01:42:05.392142+0000","last_active":"2024-04-26T01:42:49.058087+0000","last_peered":"2024-04-26T01:42:49.058087+0000","last_clean":"2024-04-26T01:42:49.058087+0000","last_became_active":"2024-04-26T01:42:05.391829+0000","last_became_peered":"2024-04-26T01:42:05.391829+0000","last_unstale":"2024-04-26T01:42:49.058087+0000","last_undegraded":"2024-04-26T01:42:49.058087+0000","last_fullsized":"2024-04-26T01:42:49.058087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:00:25.934641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.057545+0000","last_change":"2024-04-26T01:42:05.409184+0000","last_active":"2024-04-26T01:42:49.057545+0000","last_peered":"2024-04-26T01:42:49.057545+0000","last_clean":"2024-04-26T01:42:49.057545+0000","last_became_active":"2024-04-26T01:42:05.408894+0000","last_became_peered":"2024-04-26T01:42:05.408894+0000","last_unstale":"2024-04-26T01:42:49.057545+0000","last_undegraded":"2024-04-26T01:42:49.057545+0000","last_fullsized":"2024-04-26T01:42:49.057545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:53.291844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058733+0000","last_change":"2024-04-26T01:42:05.406940+0000","last_active":"2024-04-26T01:42:49.058733+0000","last_peered":"2024-04-26T01:42:49.058733+0000","last_clean":"2024-04-26T01:42:49.058733+0000","last_became_active":"2024-04-26T01:42:05.406476+0000","last_became_peered":"2024-04-26T01:42:05.406476+0000","last_unstale":"2024-04-26T01:42:49.058733+0000","last_undegraded":"2024-04-26T01:42:49.058733+0000","last_fullsized":"2024-04-26T01:42:49.058733+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:21:37.734633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.068949+0000","last_change":"2024-04-26T01:42:05.413521+0000","last_active":"2024-04-26T01:42:49.068949+0000","last_peered":"2024-04-26T01:42:49.068949+0000","last_clean":"2024-04-26T01:42:49.068949+0000","last_became_active":"2024-04-26T01:42:05.412789+0000","last_became_peered":"2024-04-26T01:42:05.412789+0000","last_unstale":"2024-04-26T01:42:49.068949+0000","last_undegraded":"2024-04-26T01:42:49.068949+0000","last_fullsized":"2024-04-26T01:42:49.068949+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:13.471376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058084+0000","last_change":"2024-04-26T01:42:05.391395+0000","last_active":"2024-04-26T01:42:49.058084+0000","last_peered":"2024-04-26T01:42:49.058084+0000","last_clean":"2024-04-26T01:42:49.058084+0000","last_became_active":"2024-04-26T01:42:05.391048+0000","last_became_peered":"2024-04-26T01:42:05.391048+0000","last_unstale":"2024-04-26T01:42:49.058084+0000","last_undegraded":"2024-04-26T01:42:49.058084+0000","last_fullsized":"2024-04-26T01:42:49.058084+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:45:56.645454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058824+0000","last_change":"2024-04-26T01:42:05.386868+0000","last_active":"2024-04-26T01:42:49.058824+0000","last_peered":"2024-04-26T01:42:49.058824+0000","last_clean":"2024-04-26T01:42:49.058824+0000","last_became_active":"2024-04-26T01:42:05.386483+0000","last_became_peered":"2024-04-26T01:42:05.386483+0000","last_unstale":"2024-04-26T01:42:49.058824+0000","last_undegraded":"2024-04-26T01:42:49.058824+0000","last_fullsized":"2024-04-26T01:42:49.058824+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:42:44.453076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.072508+0000","last_change":"2024-04-26T01:42:05.406711+0000","last_active":"2024-04-26T01:42:49.072508+0000","last_peered":"2024-04-26T01:42:49.072508+0000","last_clean":"2024-04-26T01:42:49.072508+0000","last_became_active":"2024-04-26T01:42:05.406318+0000","last_became_peered":"2024-04-26T01:42:05.406318+0000","last_unstale":"2024-04-26T01:42:49.072508+0000","last_undegraded":"2024-04-26T01:42:49.072508+0000","last_fullsized":"2024-04-26T01:42:49.072508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:36:16.599678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050354+0000","last_change":"2024-04-26T01:42:05.406305+0000","last_active":"2024-04-26T01:42:49.050354+0000","last_peered":"2024-04-26T01:42:49.050354+0000","last_clean":"2024-04-26T01:42:49.050354+0000","last_became_active":"2024-04-26T01:42:05.405953+0000","last_became_peered":"2024-04-26T01:42:05.405953+0000","last_unstale":"2024-04-26T01:42:49.050354+0000","last_undegraded":"2024-04-26T01:42:49.050354+0000","last_fullsized":"2024-04-26T01:42:49.050354+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:36.588672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070537+0000","last_change":"2024-04-26T01:42:05.413757+0000","last_active":"2024-04-26T01:42:49.070537+0000","last_peered":"2024-04-26T01:42:49.070537+0000","last_clean":"2024-04-26T01:42:49.070537+0000","last_became_active":"2024-04-26T01:42:05.413410+0000","last_became_peered":"2024-04-26T01:42:05.413410+0000","last_unstale":"2024-04-26T01:42:49.070537+0000","last_undegraded":"2024-04-26T01:42:49.070537+0000","last_fullsized":"2024-04-26T01:42:49.070537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:30.298443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054260+0000","last_change":"2024-04-26T01:42:05.412268+0000","last_active":"2024-04-26T01:42:49.054260+0000","last_peered":"2024-04-26T01:42:49.054260+0000","last_clean":"2024-04-26T01:42:49.054260+0000","last_became_active":"2024-04-26T01:42:05.411804+0000","last_became_peered":"2024-04-26T01:42:05.411804+0000","last_unstale":"2024-04-26T01:42:49.054260+0000","last_undegraded":"2024-04-26T01:42:49.054260+0000","last_fullsized":"2024-04-26T01:42:49.054260+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:47.277766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.067407+0000","last_change":"2024-04-26T01:42:05.415563+0000","last_active":"2024-04-26T01:42:49.067407+0000","last_peered":"2024-04-26T01:42:49.067407+0000","last_clean":"2024-04-26T01:42:49.067407+0000","last_became_active":"2024-04-26T01:42:05.414818+0000","last_became_peered":"2024-04-26T01:42:05.414818+0000","last_unstale":"2024-04-26T01:42:49.067407+0000","last_undegraded":"2024-04-26T01:42:49.067407+0000","last_fullsized":"2024-04-26T01:42:49.067407+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:54.713595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.057531+0000","last_change":"2024-04-26T01:42:05.392280+0000","last_active":"2024-04-26T01:42:49.057531+0000","last_peered":"2024-04-26T01:42:49.057531+0000","last_clean":"2024-04-26T01:42:49.057531+0000","last_became_active":"2024-04-26T01:42:05.391943+0000","last_became_peered":"2024-04-26T01:42:05.391943+0000","last_unstale":"2024-04-26T01:42:49.057531+0000","last_undegraded":"2024-04-26T01:42:49.057531+0000","last_fullsized":"2024-04-26T01:42:49.057531+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:38:51.560596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63207,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":219,"num_read_kb":178,"num_write":166,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":193878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285597,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":493526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547232,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41172,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703980,"statfs":{"total":95995035648,"available":95952875520,"internally_reserved":0,"allocated":1716224,"data_stored":1144101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710759,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":493650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972399,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1028096,"data_stored":493458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266738,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1630208,"data_stored":1085891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430264,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":493039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40560,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704592,"statfs":{"total":95995035648,"available":95953502208,"internally_reserved":0,"allocated":1089536,"data_stored":555904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":494310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":553388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":493350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888131,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1028096,"data_stored":493867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691902,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36948,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708204,"statfs":{"total":95995035648,"available":95957200896,"internally_reserved":0,"allocated":1642496,"data_stored":1085184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:43:02.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:43:03.398 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:05.030 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:04 smithi006 ceph-mon[28419]: pgmap v361: 97 pgs: 97 active+clean; 638 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T01:43:05.030 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:05.030 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:04 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:05.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:04 smithi188 ceph-mon[32751]: pgmap v361: 97 pgs: 97 active+clean; 638 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T01:43:05.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:05.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:04 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:05.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:04 smithi029 ceph-mon[32775]: pgmap v361: 97 pgs: 97 active+clean; 638 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T01:43:05.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:05.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:04 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:05.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.508+0000 7fa125c00700 1 -- 172.21.15.6:0/1369899223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1201092d0 msgr2=0x7fa1201096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.508+0000 7fa125c00700 1 --2- 172.21.15.6:0/1369899223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1201092d0 0x7fa1201096b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa110009990 tx=0x7fa110009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.509+0000 7fa125c00700 1 -- 172.21.15.6:0/1369899223 shutdown_connections 2024-04-26T01:43:05.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.509+0000 7fa125c00700 1 --2- 172.21.15.6:0/1369899223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa12010e3b0 0x7fa120115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.509+0000 7fa125c00700 1 --2- 172.21.15.6:0/1369899223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1201092d0 0x7fa1201096b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.509+0000 7fa125c00700 1 --2- 172.21.15.6:0/1369899223 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa120109c80 0x7fa12010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.509+0000 7fa125c00700 1 -- 172.21.15.6:0/1369899223 >> 172.21.15.6:0/1369899223 conn(0x7fa120076210 msgr2=0x7fa120107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:05.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.509+0000 7fa125c00700 1 -- 172.21.15.6:0/1369899223 shutdown_connections 2024-04-26T01:43:05.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.509+0000 7fa125c00700 1 -- 172.21.15.6:0/1369899223 wait complete. 2024-04-26T01:43:05.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.511+0000 7fa125c00700 1 Processor -- start 2024-04-26T01:43:05.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.511+0000 7fa125c00700 1 -- start start 2024-04-26T01:43:05.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa125c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1201092d0 0x7fa1201a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:05.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa125c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa120109c80 0x7fa1201a5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:05.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa125c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa12010e3b0 0x7fa12019f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:05.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa125c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa1201192b0 con 0x7fa1201092d0 2024-04-26T01:43:05.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa125c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa120119130 con 0x7fa120109c80 2024-04-26T01:43:05.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa125c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa120119430 con 0x7fa12010e3b0 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa116000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa120109c80 0x7fa1201a5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa11ea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1201092d0 0x7fa1201a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa116000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa120109c80 0x7fa1201a5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:46494/0 (socket says 172.21.15.6:46494) 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa116000700 1 -- 172.21.15.6:0/3130992786 learned_addr learned my addr 172.21.15.6:0/3130992786 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa11f400700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa12010e3b0 0x7fa12019f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa116000700 1 -- 172.21.15.6:0/3130992786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa12010e3b0 msgr2=0x7fa12019f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa116000700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa12010e3b0 0x7fa12019f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa116000700 1 -- 172.21.15.6:0/3130992786 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1201092d0 msgr2=0x7fa1201a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.512+0000 7fa116000700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1201092d0 0x7fa1201a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.513+0000 7fa116000700 1 -- 172.21.15.6:0/3130992786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12019fd30 con 0x7fa120109c80 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.513+0000 7fa116000700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa120109c80 0x7fa1201a5950 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa10800ef90 tx=0x7fa10800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.513+0000 7fa117e00700 1 -- 172.21.15.6:0/3130992786 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1080092e0 con 0x7fa120109c80 2024-04-26T01:43:05.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.513+0000 7fa117e00700 1 -- 172.21.15.6:0/3130992786 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa108004bb0 con 0x7fa120109c80 2024-04-26T01:43:05.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.513+0000 7fa117e00700 1 -- 172.21.15.6:0/3130992786 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa108004280 con 0x7fa120109c80 2024-04-26T01:43:05.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.513+0000 7fa125c00700 1 -- 172.21.15.6:0/3130992786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa12019ffc0 con 0x7fa120109c80 2024-04-26T01:43:05.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.514+0000 7fa125c00700 1 -- 172.21.15.6:0/3130992786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1201ac2e0 con 0x7fa120109c80 2024-04-26T01:43:05.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.515+0000 7fa125c00700 1 -- 172.21.15.6:0/3130992786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa120069730 con 0x7fa120109c80 2024-04-26T01:43:05.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.517+0000 7fa117e00700 1 -- 172.21.15.6:0/3130992786 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fa10801b030 con 0x7fa120109c80 2024-04-26T01:43:05.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.518+0000 7fa117e00700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa1040a3d50 0x7fa1040a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:05.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.518+0000 7fa117e00700 1 -- 172.21.15.6:0/3130992786 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7fa1080d00b0 con 0x7fa120109c80 2024-04-26T01:43:05.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.518+0000 7fa11ea00700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa1040a3d50 0x7fa1040a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:05.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.519+0000 7fa11ea00700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa1040a3d50 0x7fa1040a6210 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa11000b790 tx=0x7fa11000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:05.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.523+0000 7fa117e00700 1 -- 172.21.15.6:0/3130992786 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa108022020 con 0x7fa120109c80 2024-04-26T01:43:05.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.750+0000 7fa125c00700 1 -- 172.21.15.6:0/3130992786 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa120062e80 con 0x7fa1040a3d50 2024-04-26T01:43:05.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.773+0000 7fa117e00700 1 -- 172.21.15.6:0/3130992786 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258727 (secure 0 0 0) 0x7fa1100672c0 con 0x7fa1040a3d50 2024-04-26T01:43:05.777 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:43:05.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.781+0000 7fa125c00700 1 -- 172.21.15.6:0/3130992786 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa1040a3d50 msgr2=0x7fa1040a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.781+0000 7fa125c00700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa1040a3d50 0x7fa1040a6210 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa11000b790 tx=0x7fa11000b260 comp rx=0 tx=0).stop 2024-04-26T01:43:05.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.781+0000 7fa125c00700 1 -- 172.21.15.6:0/3130992786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa120109c80 msgr2=0x7fa1201a5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.781+0000 7fa125c00700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa120109c80 0x7fa1201a5950 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa10800ef90 tx=0x7fa10800c3a0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.781+0000 7fa125c00700 1 -- 172.21.15.6:0/3130992786 shutdown_connections 2024-04-26T01:43:05.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.781+0000 7fa125c00700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa1040a3d50 0x7fa1040a6210 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.781+0000 7fa125c00700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa12010e3b0 0x7fa12019f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.781+0000 7fa125c00700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1201092d0 0x7fa1201a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.781+0000 7fa125c00700 1 --2- 172.21.15.6:0/3130992786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa120109c80 0x7fa1201a5950 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.781+0000 7fa125c00700 1 -- 172.21.15.6:0/3130992786 >> 172.21.15.6:0/3130992786 conn(0x7fa120076210 msgr2=0x7fa120077e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:05.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.781+0000 7fa125c00700 1 -- 172.21.15.6:0/3130992786 shutdown_connections 2024-04-26T01:43:05.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:05.781+0000 7fa125c00700 1 -- 172.21.15.6:0/3130992786 wait complete. 2024-04-26T01:43:05.780 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-26T01:43:06.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:43:06.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:43:06.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:05 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:06.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:43:06.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:43:06.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:05 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:06.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:43:06.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:43:06.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:05 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:43:06.571 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-04-26T01:43:05.196934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658066,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":381,"num_read_kb":312,"num_write":350,"num_write_kb":3054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":281,"ondisk_log_size":281,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392404,"kb_used_data":14088,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549420,"statfs":{"total":1151940427776,"available":1151538606080,"internally_reserved":0,"allocated":14426112,"data_stored":7888182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17561,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":59,"num_write":10,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004393"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058393+0000","last_change":"2024-04-26T01:42:05.417304+0000","last_active":"2024-04-26T01:42:49.058393+0000","last_peered":"2024-04-26T01:42:49.058393+0000","last_clean":"2024-04-26T01:42:49.058393+0000","last_became_active":"2024-04-26T01:42:05.416999+0000","last_became_peered":"2024-04-26T01:42:05.416999+0000","last_unstale":"2024-04-26T01:42:49.058393+0000","last_undegraded":"2024-04-26T01:42:49.058393+0000","last_fullsized":"2024-04-26T01:42:49.058393+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:33:30.958943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.061430+0000","last_change":"2024-04-26T01:42:05.413089+0000","last_active":"2024-04-26T01:42:49.061430+0000","last_peered":"2024-04-26T01:42:49.061430+0000","last_clean":"2024-04-26T01:42:49.061430+0000","last_became_active":"2024-04-26T01:42:05.412776+0000","last_became_peered":"2024-04-26T01:42:05.412776+0000","last_unstale":"2024-04-26T01:42:49.061430+0000","last_undegraded":"2024-04-26T01:42:49.061430+0000","last_fullsized":"2024-04-26T01:42:49.061430+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:47.861340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.060501+0000","last_change":"2024-04-26T01:42:05.404864+0000","last_active":"2024-04-26T01:42:49.060501+0000","last_peered":"2024-04-26T01:42:49.060501+0000","last_clean":"2024-04-26T01:42:49.060501+0000","last_became_active":"2024-04-26T01:42:05.404529+0000","last_became_peered":"2024-04-26T01:42:05.404529+0000","last_unstale":"2024-04-26T01:42:49.060501+0000","last_undegraded":"2024-04-26T01:42:49.060501+0000","last_fullsized":"2024-04-26T01:42:49.060501+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:04:18.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.051174+0000","last_change":"2024-04-26T01:42:05.393582+0000","last_active":"2024-04-26T01:42:49.051174+0000","last_peered":"2024-04-26T01:42:49.051174+0000","last_clean":"2024-04-26T01:42:49.051174+0000","last_became_active":"2024-04-26T01:42:05.393193+0000","last_became_peered":"2024-04-26T01:42:05.393193+0000","last_unstale":"2024-04-26T01:42:49.051174+0000","last_undegraded":"2024-04-26T01:42:49.051174+0000","last_fullsized":"2024-04-26T01:42:49.051174+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:20:29.420459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.055773+0000","last_change":"2024-04-26T01:42:05.412979+0000","last_active":"2024-04-26T01:42:49.055773+0000","last_peered":"2024-04-26T01:42:49.055773+0000","last_clean":"2024-04-26T01:42:49.055773+0000","last_became_active":"2024-04-26T01:42:05.412679+0000","last_became_peered":"2024-04-26T01:42:05.412679+0000","last_unstale":"2024-04-26T01:42:49.055773+0000","last_undegraded":"2024-04-26T01:42:49.055773+0000","last_fullsized":"2024-04-26T01:42:49.055773+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:22.774960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.062507+0000","last_change":"2024-04-26T01:42:49.062507+0000","last_active":"2024-04-26T01:42:49.062507+0000","last_peered":"2024-04-26T01:42:49.062507+0000","last_clean":"2024-04-26T01:42:49.062507+0000","last_became_active":"2024-04-26T01:42:49.062095+0000","last_became_peered":"2024-04-26T01:42:49.062095+0000","last_unstale":"2024-04-26T01:42:49.062507+0000","last_undegraded":"2024-04-26T01:42:49.062507+0000","last_fullsized":"2024-04-26T01:42:49.062507+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:47.846697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.052858+0000","last_change":"2024-04-26T01:42:05.385221+0000","last_active":"2024-04-26T01:42:49.052858+0000","last_peered":"2024-04-26T01:42:49.052858+0000","last_clean":"2024-04-26T01:42:49.052858+0000","last_became_active":"2024-04-26T01:42:05.384856+0000","last_became_peered":"2024-04-26T01:42:05.384856+0000","last_unstale":"2024-04-26T01:42:49.052858+0000","last_undegraded":"2024-04-26T01:42:49.052858+0000","last_fullsized":"2024-04-26T01:42:49.052858+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:07:23.461324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070057+0000","last_change":"2024-04-26T01:42:05.404754+0000","last_active":"2024-04-26T01:42:49.070057+0000","last_peered":"2024-04-26T01:42:49.070057+0000","last_clean":"2024-04-26T01:42:49.070057+0000","last_became_active":"2024-04-26T01:42:05.404220+0000","last_became_peered":"2024-04-26T01:42:05.404220+0000","last_unstale":"2024-04-26T01:42:49.070057+0000","last_undegraded":"2024-04-26T01:42:49.070057+0000","last_fullsized":"2024-04-26T01:42:49.070057+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:50.243552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054504+0000","last_change":"2024-04-26T01:42:05.390972+0000","last_active":"2024-04-26T01:42:49.054504+0000","last_peered":"2024-04-26T01:42:49.054504+0000","last_clean":"2024-04-26T01:42:49.054504+0000","last_became_active":"2024-04-26T01:42:05.390599+0000","last_became_peered":"2024-04-26T01:42:05.390599+0000","last_unstale":"2024-04-26T01:42:49.054504+0000","last_undegraded":"2024-04-26T01:42:49.054504+0000","last_fullsized":"2024-04-26T01:42:49.054504+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:19:46.328640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054966+0000","last_change":"2024-04-26T01:42:05.392320+0000","last_active":"2024-04-26T01:42:49.054966+0000","last_peered":"2024-04-26T01:42:49.054966+0000","last_clean":"2024-04-26T01:42:49.054966+0000","last_became_active":"2024-04-26T01:42:05.391986+0000","last_became_peered":"2024-04-26T01:42:05.391986+0000","last_unstale":"2024-04-26T01:42:49.054966+0000","last_undegraded":"2024-04-26T01:42:49.054966+0000","last_fullsized":"2024-04-26T01:42:49.054966+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:17:27.302358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053828+0000","last_change":"2024-04-26T01:42:05.388077+0000","last_active":"2024-04-26T01:42:49.053828+0000","last_peered":"2024-04-26T01:42:49.053828+0000","last_clean":"2024-04-26T01:42:49.053828+0000","last_became_active":"2024-04-26T01:42:05.387545+0000","last_became_peered":"2024-04-26T01:42:05.387545+0000","last_unstale":"2024-04-26T01:42:49.053828+0000","last_undegraded":"2024-04-26T01:42:49.053828+0000","last_fullsized":"2024-04-26T01:42:49.053828+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:51.288177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.072023+0000","last_change":"2024-04-26T01:42:49.067449+0000","last_active":"2024-04-26T01:42:49.072023+0000","last_peered":"2024-04-26T01:42:49.072023+0000","last_clean":"2024-04-26T01:42:49.072023+0000","last_became_active":"2024-04-26T01:42:49.065991+0000","last_became_peered":"2024-04-26T01:42:49.065991+0000","last_unstale":"2024-04-26T01:42:49.072023+0000","last_undegraded":"2024-04-26T01:42:49.072023+0000","last_fullsized":"2024-04-26T01:42:49.072023+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:13:36.565257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.067608+0000","last_change":"2024-04-26T01:42:05.402847+0000","last_active":"2024-04-26T01:42:49.067608+0000","last_peered":"2024-04-26T01:42:49.067608+0000","last_clean":"2024-04-26T01:42:49.067608+0000","last_became_active":"2024-04-26T01:42:05.402503+0000","last_became_peered":"2024-04-26T01:42:05.402503+0000","last_unstale":"2024-04-26T01:42:49.067608+0000","last_undegraded":"2024-04-26T01:42:49.067608+0000","last_fullsized":"2024-04-26T01:42:49.067608+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:06.935060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058574+0000","last_change":"2024-04-26T01:42:05.405257+0000","last_active":"2024-04-26T01:42:49.058574+0000","last_peered":"2024-04-26T01:42:49.058574+0000","last_clean":"2024-04-26T01:42:49.058574+0000","last_became_active":"2024-04-26T01:42:05.404551+0000","last_became_peered":"2024-04-26T01:42:05.404551+0000","last_unstale":"2024-04-26T01:42:49.058574+0000","last_undegraded":"2024-04-26T01:42:49.058574+0000","last_fullsized":"2024-04-26T01:42:49.058574+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:20.754088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.056662+0000","last_change":"2024-04-26T01:42:05.408847+0000","last_active":"2024-04-26T01:42:49.056662+0000","last_peered":"2024-04-26T01:42:49.056662+0000","last_clean":"2024-04-26T01:42:49.056662+0000","last_became_active":"2024-04-26T01:42:05.406189+0000","last_became_peered":"2024-04-26T01:42:05.406189+0000","last_unstale":"2024-04-26T01:42:49.056662+0000","last_undegraded":"2024-04-26T01:42:49.056662+0000","last_fullsized":"2024-04-26T01:42:49.056662+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:59:58.633630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053634+0000","last_change":"2024-04-26T01:42:05.404647+0000","last_active":"2024-04-26T01:42:49.053634+0000","last_peered":"2024-04-26T01:42:49.053634+0000","last_clean":"2024-04-26T01:42:49.053634+0000","last_became_active":"2024-04-26T01:42:05.404265+0000","last_became_peered":"2024-04-26T01:42:05.404265+0000","last_unstale":"2024-04-26T01:42:49.053634+0000","last_undegraded":"2024-04-26T01:42:49.053634+0000","last_fullsized":"2024-04-26T01:42:49.053634+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:00:43.472541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058658+0000","last_change":"2024-04-26T01:42:05.417602+0000","last_active":"2024-04-26T01:42:49.058658+0000","last_peered":"2024-04-26T01:42:49.058658+0000","last_clean":"2024-04-26T01:42:49.058658+0000","last_became_active":"2024-04-26T01:42:05.417144+0000","last_became_peered":"2024-04-26T01:42:05.417144+0000","last_unstale":"2024-04-26T01:42:49.058658+0000","last_undegraded":"2024-04-26T01:42:49.058658+0000","last_fullsized":"2024-04-26T01:42:49.058658+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:34.043597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054097+0000","last_change":"2024-04-26T01:42:05.392948+0000","last_active":"2024-04-26T01:42:49.054097+0000","last_peered":"2024-04-26T01:42:49.054097+0000","last_clean":"2024-04-26T01:42:49.054097+0000","last_became_active":"2024-04-26T01:42:05.392040+0000","last_became_peered":"2024-04-26T01:42:05.392040+0000","last_unstale":"2024-04-26T01:42:49.054097+0000","last_undegraded":"2024-04-26T01:42:49.054097+0000","last_fullsized":"2024-04-26T01:42:49.054097+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:17.307979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058740+0000","last_change":"2024-04-26T01:42:05.406588+0000","last_active":"2024-04-26T01:42:49.058740+0000","last_peered":"2024-04-26T01:42:49.058740+0000","last_clean":"2024-04-26T01:42:49.058740+0000","last_became_active":"2024-04-26T01:42:05.406200+0000","last_became_peered":"2024-04-26T01:42:05.406200+0000","last_unstale":"2024-04-26T01:42:49.058740+0000","last_undegraded":"2024-04-26T01:42:49.058740+0000","last_fullsized":"2024-04-26T01:42:49.058740+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:14.095383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054175+0000","last_change":"2024-04-26T01:42:01.339289+0000","last_active":"2024-04-26T01:42:49.054175+0000","last_peered":"2024-04-26T01:42:49.054175+0000","last_clean":"2024-04-26T01:42:49.054175+0000","last_became_active":"2024-04-26T01:42:01.335402+0000","last_became_peered":"2024-04-26T01:42:01.335402+0000","last_unstale":"2024-04-26T01:42:49.054175+0000","last_undegraded":"2024-04-26T01:42:49.054175+0000","last_fullsized":"2024-04-26T01:42:49.054175+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:51.411524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.071724+0000","last_change":"2024-04-26T01:42:05.409569+0000","last_active":"2024-04-26T01:42:49.071724+0000","last_peered":"2024-04-26T01:42:49.071724+0000","last_clean":"2024-04-26T01:42:49.071724+0000","last_became_active":"2024-04-26T01:42:05.407441+0000","last_became_peered":"2024-04-26T01:42:05.407441+0000","last_unstale":"2024-04-26T01:42:49.071724+0000","last_undegraded":"2024-04-26T01:42:49.071724+0000","last_fullsized":"2024-04-26T01:42:49.071724+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:36:58.043359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.072731+0000","last_change":"2024-04-26T01:42:02.231871+0000","last_active":"2024-04-26T01:42:49.072731+0000","last_peered":"2024-04-26T01:42:49.072731+0000","last_clean":"2024-04-26T01:42:49.072731+0000","last_became_active":"2024-04-26T01:42:02.230813+0000","last_became_peered":"2024-04-26T01:42:02.230813+0000","last_unstale":"2024-04-26T01:42:49.072731+0000","last_undegraded":"2024-04-26T01:42:49.072731+0000","last_fullsized":"2024-04-26T01:42:49.072731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:53.117266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.062086+0000","last_change":"2024-04-26T01:42:05.412414+0000","last_active":"2024-04-26T01:42:49.062086+0000","last_peered":"2024-04-26T01:42:49.062086+0000","last_clean":"2024-04-26T01:42:49.062086+0000","last_became_active":"2024-04-26T01:42:05.412057+0000","last_became_peered":"2024-04-26T01:42:05.412057+0000","last_unstale":"2024-04-26T01:42:49.062086+0000","last_undegraded":"2024-04-26T01:42:49.062086+0000","last_fullsized":"2024-04-26T01:42:49.062086+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:51:46.331246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.056301+0000","last_change":"2024-04-26T01:42:01.329957+0000","last_active":"2024-04-26T01:42:49.056301+0000","last_peered":"2024-04-26T01:42:49.056301+0000","last_clean":"2024-04-26T01:42:49.056301+0000","last_became_active":"2024-04-26T01:42:01.329289+0000","last_became_peered":"2024-04-26T01:42:01.329289+0000","last_unstale":"2024-04-26T01:42:49.056301+0000","last_undegraded":"2024-04-26T01:42:49.056301+0000","last_fullsized":"2024-04-26T01:42:49.056301+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:46:38.983138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054482+0000","last_change":"2024-04-26T01:42:05.399438+0000","last_active":"2024-04-26T01:42:49.054482+0000","last_peered":"2024-04-26T01:42:49.054482+0000","last_clean":"2024-04-26T01:42:49.054482+0000","last_became_active":"2024-04-26T01:42:05.399082+0000","last_became_peered":"2024-04-26T01:42:05.399082+0000","last_unstale":"2024-04-26T01:42:49.054482+0000","last_undegraded":"2024-04-26T01:42:49.054482+0000","last_fullsized":"2024-04-26T01:42:49.054482+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:13.619893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"91'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.056135+0000","last_change":"2024-04-26T01:42:01.328230+0000","last_active":"2024-04-26T01:42:49.056135+0000","last_peered":"2024-04-26T01:42:49.056135+0000","last_clean":"2024-04-26T01:42:49.056135+0000","last_became_active":"2024-04-26T01:42:01.327695+0000","last_became_peered":"2024-04-26T01:42:01.327695+0000","last_unstale":"2024-04-26T01:42:49.056135+0000","last_undegraded":"2024-04-26T01:42:49.056135+0000","last_fullsized":"2024-04-26T01:42:49.056135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:06.379749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.071822+0000","last_change":"2024-04-26T01:42:49.066057+0000","last_active":"2024-04-26T01:42:49.071822+0000","last_peered":"2024-04-26T01:42:49.071822+0000","last_clean":"2024-04-26T01:42:49.071822+0000","last_became_active":"2024-04-26T01:42:49.064684+0000","last_became_peered":"2024-04-26T01:42:49.064684+0000","last_unstale":"2024-04-26T01:42:49.071822+0000","last_undegraded":"2024-04-26T01:42:49.071822+0000","last_fullsized":"2024-04-26T01:42:49.071822+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:39.159528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:43:02.583974+0000","last_change":"2024-04-26T01:42:01.332623+0000","last_active":"2024-04-26T01:43:02.583974+0000","last_peered":"2024-04-26T01:43:02.583974+0000","last_clean":"2024-04-26T01:43:02.583974+0000","last_became_active":"2024-04-26T01:42:01.332227+0000","last_became_peered":"2024-04-26T01:42:01.332227+0000","last_unstale":"2024-04-26T01:43:02.583974+0000","last_undegraded":"2024-04-26T01:43:02.583974+0000","last_fullsized":"2024-04-26T01:43:02.583974+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:58.611464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.055737+0000","last_change":"2024-04-26T01:42:05.396133+0000","last_active":"2024-04-26T01:42:49.055737+0000","last_peered":"2024-04-26T01:42:49.055737+0000","last_clean":"2024-04-26T01:42:49.055737+0000","last_became_active":"2024-04-26T01:42:05.395553+0000","last_became_peered":"2024-04-26T01:42:05.395553+0000","last_unstale":"2024-04-26T01:42:49.055737+0000","last_undegraded":"2024-04-26T01:42:49.055737+0000","last_fullsized":"2024-04-26T01:42:49.055737+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:36:01.363909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054479+0000","last_change":"2024-04-26T01:42:01.340624+0000","last_active":"2024-04-26T01:42:49.054479+0000","last_peered":"2024-04-26T01:42:49.054479+0000","last_clean":"2024-04-26T01:42:49.054479+0000","last_became_active":"2024-04-26T01:42:01.340158+0000","last_became_peered":"2024-04-26T01:42:01.340158+0000","last_unstale":"2024-04-26T01:42:49.054479+0000","last_undegraded":"2024-04-26T01:42:49.054479+0000","last_fullsized":"2024-04-26T01:42:49.054479+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:47:05.529922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050739+0000","last_change":"2024-04-26T01:42:05.406354+0000","last_active":"2024-04-26T01:42:49.050739+0000","last_peered":"2024-04-26T01:42:49.050739+0000","last_clean":"2024-04-26T01:42:49.050739+0000","last_became_active":"2024-04-26T01:42:05.405989+0000","last_became_peered":"2024-04-26T01:42:05.405989+0000","last_unstale":"2024-04-26T01:42:49.050739+0000","last_undegraded":"2024-04-26T01:42:49.050739+0000","last_fullsized":"2024-04-26T01:42:49.050739+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:43.115225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070125+0000","last_change":"2024-04-26T01:42:01.340549+0000","last_active":"2024-04-26T01:42:49.070125+0000","last_peered":"2024-04-26T01:42:49.070125+0000","last_clean":"2024-04-26T01:42:49.070125+0000","last_became_active":"2024-04-26T01:42:01.339861+0000","last_became_peered":"2024-04-26T01:42:01.339861+0000","last_unstale":"2024-04-26T01:42:49.070125+0000","last_undegraded":"2024-04-26T01:42:49.070125+0000","last_fullsized":"2024-04-26T01:42:49.070125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:02:53.154367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070461+0000","last_change":"2024-04-26T01:42:05.402664+0000","last_active":"2024-04-26T01:42:49.070461+0000","last_peered":"2024-04-26T01:42:49.070461+0000","last_clean":"2024-04-26T01:42:49.070461+0000","last_became_active":"2024-04-26T01:42:05.400785+0000","last_became_peered":"2024-04-26T01:42:05.400785+0000","last_unstale":"2024-04-26T01:42:49.070461+0000","last_undegraded":"2024-04-26T01:42:49.070461+0000","last_fullsized":"2024-04-26T01:42:49.070461+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:40:32.137752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050300+0000","last_change":"2024-04-26T01:42:01.328600+0000","last_active":"2024-04-26T01:42:49.050300+0000","last_peered":"2024-04-26T01:42:49.050300+0000","last_clean":"2024-04-26T01:42:49.050300+0000","last_became_active":"2024-04-26T01:42:01.328149+0000","last_became_peered":"2024-04-26T01:42:01.328149+0000","last_unstale":"2024-04-26T01:42:49.050300+0000","last_undegraded":"2024-04-26T01:42:49.050300+0000","last_fullsized":"2024-04-26T01:42:49.050300+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:45:42.811088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.065824+0000","last_change":"2024-04-26T01:42:49.065824+0000","last_active":"2024-04-26T01:42:49.065824+0000","last_peered":"2024-04-26T01:42:49.065824+0000","last_clean":"2024-04-26T01:42:49.065824+0000","last_became_active":"2024-04-26T01:42:49.064670+0000","last_became_peered":"2024-04-26T01:42:49.064670+0000","last_unstale":"2024-04-26T01:42:49.065824+0000","last_undegraded":"2024-04-26T01:42:49.065824+0000","last_fullsized":"2024-04-26T01:42:49.065824+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:44.790184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:43:02.583219+0000","last_change":"2024-04-26T01:42:01.335438+0000","last_active":"2024-04-26T01:43:02.583219+0000","last_peered":"2024-04-26T01:43:02.583219+0000","last_clean":"2024-04-26T01:43:02.583219+0000","last_became_active":"2024-04-26T01:42:01.334962+0000","last_became_peered":"2024-04-26T01:42:01.334962+0000","last_unstale":"2024-04-26T01:43:02.583219+0000","last_undegraded":"2024-04-26T01:43:02.583219+0000","last_fullsized":"2024-04-26T01:43:02.583219+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:35:08.252274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054352+0000","last_change":"2024-04-26T01:42:05.393294+0000","last_active":"2024-04-26T01:42:49.054352+0000","last_peered":"2024-04-26T01:42:49.054352+0000","last_clean":"2024-04-26T01:42:49.054352+0000","last_became_active":"2024-04-26T01:42:05.392974+0000","last_became_peered":"2024-04-26T01:42:05.392974+0000","last_unstale":"2024-04-26T01:42:49.054352+0000","last_undegraded":"2024-04-26T01:42:49.054352+0000","last_fullsized":"2024-04-26T01:42:49.054352+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:58.188706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053917+0000","last_change":"2024-04-26T01:42:01.333505+0000","last_active":"2024-04-26T01:42:49.053917+0000","last_peered":"2024-04-26T01:42:49.053917+0000","last_clean":"2024-04-26T01:42:49.053917+0000","last_became_active":"2024-04-26T01:42:01.333062+0000","last_became_peered":"2024-04-26T01:42:01.333062+0000","last_unstale":"2024-04-26T01:42:49.053917+0000","last_undegraded":"2024-04-26T01:42:49.053917+0000","last_fullsized":"2024-04-26T01:42:49.053917+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:50:48.804443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.063948+0000","last_change":"2024-04-26T01:42:49.063948+0000","last_active":"2024-04-26T01:42:49.063948+0000","last_peered":"2024-04-26T01:42:49.063948+0000","last_clean":"2024-04-26T01:42:49.063948+0000","last_became_active":"2024-04-26T01:42:49.060852+0000","last_became_peered":"2024-04-26T01:42:49.060852+0000","last_unstale":"2024-04-26T01:42:49.063948+0000","last_undegraded":"2024-04-26T01:42:49.063948+0000","last_fullsized":"2024-04-26T01:42:49.063948+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.370431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050957+0000","last_change":"2024-04-26T01:42:02.232308+0000","last_active":"2024-04-26T01:42:49.050957+0000","last_peered":"2024-04-26T01:42:49.050957+0000","last_clean":"2024-04-26T01:42:49.050957+0000","last_became_active":"2024-04-26T01:42:02.231897+0000","last_became_peered":"2024-04-26T01:42:02.231897+0000","last_unstale":"2024-04-26T01:42:49.050957+0000","last_undegraded":"2024-04-26T01:42:49.050957+0000","last_fullsized":"2024-04-26T01:42:49.050957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:36:16.847087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070798+0000","last_change":"2024-04-26T01:42:49.070798+0000","last_active":"2024-04-26T01:42:49.070798+0000","last_peered":"2024-04-26T01:42:49.070798+0000","last_clean":"2024-04-26T01:42:49.070798+0000","last_became_active":"2024-04-26T01:42:49.070470+0000","last_became_peered":"2024-04-26T01:42:49.070470+0000","last_unstale":"2024-04-26T01:42:49.070798+0000","last_undegraded":"2024-04-26T01:42:49.070798+0000","last_fullsized":"2024-04-26T01:42:49.070798+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:04:22.103704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"93'30","reported_seq":139,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:43:03.573026+0000","last_change":"2024-04-26T01:42:01.330822+0000","last_active":"2024-04-26T01:43:03.573026+0000","last_peered":"2024-04-26T01:43:03.573026+0000","last_clean":"2024-04-26T01:43:03.573026+0000","last_became_active":"2024-04-26T01:42:01.330337+0000","last_became_peered":"2024-04-26T01:42:01.330337+0000","last_unstale":"2024-04-26T01:43:03.573026+0000","last_undegraded":"2024-04-26T01:43:03.573026+0000","last_fullsized":"2024-04-26T01:43:03.573026+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:04.024645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63072,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":83,"num_read_kb":103,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.056114+0000","last_change":"2024-04-26T01:42:05.390601+0000","last_active":"2024-04-26T01:42:49.056114+0000","last_peered":"2024-04-26T01:42:49.056114+0000","last_clean":"2024-04-26T01:42:49.056114+0000","last_became_active":"2024-04-26T01:42:05.390279+0000","last_became_peered":"2024-04-26T01:42:05.390279+0000","last_unstale":"2024-04-26T01:42:49.056114+0000","last_undegraded":"2024-04-26T01:42:49.056114+0000","last_fullsized":"2024-04-26T01:42:49.056114+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:20.171769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058226+0000","last_change":"2024-04-26T01:42:01.337200+0000","last_active":"2024-04-26T01:42:49.058226+0000","last_peered":"2024-04-26T01:42:49.058226+0000","last_clean":"2024-04-26T01:42:49.058226+0000","last_became_active":"2024-04-26T01:42:01.336919+0000","last_became_peered":"2024-04-26T01:42:01.336919+0000","last_unstale":"2024-04-26T01:42:49.058226+0000","last_undegraded":"2024-04-26T01:42:49.058226+0000","last_fullsized":"2024-04-26T01:42:49.058226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:29.799903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.067939+0000","last_change":"2024-04-26T01:42:05.410081+0000","last_active":"2024-04-26T01:42:49.067939+0000","last_peered":"2024-04-26T01:42:49.067939+0000","last_clean":"2024-04-26T01:42:49.067939+0000","last_became_active":"2024-04-26T01:42:05.409341+0000","last_became_peered":"2024-04-26T01:42:05.409341+0000","last_unstale":"2024-04-26T01:42:49.067939+0000","last_undegraded":"2024-04-26T01:42:49.067939+0000","last_fullsized":"2024-04-26T01:42:49.067939+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:25:31.925296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050589+0000","last_change":"2024-04-26T01:42:01.331702+0000","last_active":"2024-04-26T01:42:49.050589+0000","last_peered":"2024-04-26T01:42:49.050589+0000","last_clean":"2024-04-26T01:42:49.050589+0000","last_became_active":"2024-04-26T01:42:01.331311+0000","last_became_peered":"2024-04-26T01:42:01.331311+0000","last_unstale":"2024-04-26T01:42:49.050589+0000","last_undegraded":"2024-04-26T01:42:49.050589+0000","last_fullsized":"2024-04-26T01:42:49.050589+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:54:49.513627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053620+0000","last_change":"2024-04-26T01:42:05.403957+0000","last_active":"2024-04-26T01:42:49.053620+0000","last_peered":"2024-04-26T01:42:49.053620+0000","last_clean":"2024-04-26T01:42:49.053620+0000","last_became_active":"2024-04-26T01:42:05.403583+0000","last_became_peered":"2024-04-26T01:42:05.403583+0000","last_unstale":"2024-04-26T01:42:49.053620+0000","last_undegraded":"2024-04-26T01:42:49.053620+0000","last_fullsized":"2024-04-26T01:42:49.053620+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:12:20.095407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":68,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:43:00.570014+0000","last_change":"2024-04-26T01:42:01.332456+0000","last_active":"2024-04-26T01:43:00.570014+0000","last_peered":"2024-04-26T01:43:00.570014+0000","last_clean":"2024-04-26T01:43:00.570014+0000","last_became_active":"2024-04-26T01:42:01.331910+0000","last_became_peered":"2024-04-26T01:42:01.331910+0000","last_unstale":"2024-04-26T01:43:00.570014+0000","last_undegraded":"2024-04-26T01:43:00.570014+0000","last_fullsized":"2024-04-26T01:43:00.570014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:13:25.368205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.062130+0000","last_change":"2024-04-26T01:42:49.062130+0000","last_active":"2024-04-26T01:42:49.062130+0000","last_peered":"2024-04-26T01:42:49.062130+0000","last_clean":"2024-04-26T01:42:49.062130+0000","last_became_active":"2024-04-26T01:42:49.061627+0000","last_became_peered":"2024-04-26T01:42:49.061627+0000","last_unstale":"2024-04-26T01:42:49.062130+0000","last_undegraded":"2024-04-26T01:42:49.062130+0000","last_fullsized":"2024-04-26T01:42:49.062130+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:16.101940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.083062+0000","last_change":"2024-04-26T01:42:49.082618+0000","last_active":"2024-04-26T01:42:49.083062+0000","last_peered":"2024-04-26T01:42:49.083062+0000","last_clean":"2024-04-26T01:42:49.083062+0000","last_became_active":"2024-04-26T01:42:49.064654+0000","last_became_peered":"2024-04-26T01:42:49.064654+0000","last_unstale":"2024-04-26T01:42:49.083062+0000","last_undegraded":"2024-04-26T01:42:49.083062+0000","last_fullsized":"2024-04-26T01:42:49.083062+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:13:23.937059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050479+0000","last_change":"2024-04-26T01:42:05.391392+0000","last_active":"2024-04-26T01:42:49.050479+0000","last_peered":"2024-04-26T01:42:49.050479+0000","last_clean":"2024-04-26T01:42:49.050479+0000","last_became_active":"2024-04-26T01:42:05.390939+0000","last_became_peered":"2024-04-26T01:42:05.390939+0000","last_unstale":"2024-04-26T01:42:49.050479+0000","last_undegraded":"2024-04-26T01:42:49.050479+0000","last_fullsized":"2024-04-26T01:42:49.050479+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:36.141915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.056422+0000","last_change":"2024-04-26T01:42:01.332932+0000","last_active":"2024-04-26T01:42:49.056422+0000","last_peered":"2024-04-26T01:42:49.056422+0000","last_clean":"2024-04-26T01:42:49.056422+0000","last_became_active":"2024-04-26T01:42:01.332626+0000","last_became_peered":"2024-04-26T01:42:01.332626+0000","last_unstale":"2024-04-26T01:42:49.056422+0000","last_undegraded":"2024-04-26T01:42:49.056422+0000","last_fullsized":"2024-04-26T01:42:49.056422+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:20.593730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058056+0000","last_change":"2024-04-26T01:42:05.394925+0000","last_active":"2024-04-26T01:42:49.058056+0000","last_peered":"2024-04-26T01:42:49.058056+0000","last_clean":"2024-04-26T01:42:49.058056+0000","last_became_active":"2024-04-26T01:42:05.394531+0000","last_became_peered":"2024-04-26T01:42:05.394531+0000","last_unstale":"2024-04-26T01:42:49.058056+0000","last_undegraded":"2024-04-26T01:42:49.058056+0000","last_fullsized":"2024-04-26T01:42:49.058056+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:33.177780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054580+0000","last_change":"2024-04-26T01:42:02.237482+0000","last_active":"2024-04-26T01:42:49.054580+0000","last_peered":"2024-04-26T01:42:49.054580+0000","last_clean":"2024-04-26T01:42:49.054580+0000","last_became_active":"2024-04-26T01:42:02.236752+0000","last_became_peered":"2024-04-26T01:42:02.236752+0000","last_unstale":"2024-04-26T01:42:49.054580+0000","last_undegraded":"2024-04-26T01:42:49.054580+0000","last_fullsized":"2024-04-26T01:42:49.054580+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:27.211245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'1","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054655+0000","last_change":"2024-04-26T01:42:05.402157+0000","last_active":"2024-04-26T01:42:49.054655+0000","last_peered":"2024-04-26T01:42:49.054655+0000","last_clean":"2024-04-26T01:42:49.054655+0000","last_became_active":"2024-04-26T01:42:05.401889+0000","last_became_peered":"2024-04-26T01:42:05.401889+0000","last_unstale":"2024-04-26T01:42:49.054655+0000","last_undegraded":"2024-04-26T01:42:49.054655+0000","last_fullsized":"2024-04-26T01:42:49.054655+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:33.854901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053243+0000","last_change":"2024-04-26T01:42:01.339675+0000","last_active":"2024-04-26T01:42:49.053243+0000","last_peered":"2024-04-26T01:42:49.053243+0000","last_clean":"2024-04-26T01:42:49.053243+0000","last_became_active":"2024-04-26T01:42:01.339247+0000","last_became_peered":"2024-04-26T01:42:01.339247+0000","last_unstale":"2024-04-26T01:42:49.053243+0000","last_undegraded":"2024-04-26T01:42:49.053243+0000","last_fullsized":"2024-04-26T01:42:49.053243+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:32.724830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054507+0000","last_change":"2024-04-26T01:42:05.412913+0000","last_active":"2024-04-26T01:42:49.054507+0000","last_peered":"2024-04-26T01:42:49.054507+0000","last_clean":"2024-04-26T01:42:49.054507+0000","last_became_active":"2024-04-26T01:42:05.412556+0000","last_became_peered":"2024-04-26T01:42:05.412556+0000","last_unstale":"2024-04-26T01:42:49.054507+0000","last_undegraded":"2024-04-26T01:42:49.054507+0000","last_fullsized":"2024-04-26T01:42:49.054507+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:00:52.798069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050679+0000","last_change":"2024-04-26T01:42:05.391235+0000","last_active":"2024-04-26T01:42:49.050679+0000","last_peered":"2024-04-26T01:42:49.050679+0000","last_clean":"2024-04-26T01:42:49.050679+0000","last_became_active":"2024-04-26T01:42:05.390717+0000","last_became_peered":"2024-04-26T01:42:05.390717+0000","last_unstale":"2024-04-26T01:42:49.050679+0000","last_undegraded":"2024-04-26T01:42:49.050679+0000","last_fullsized":"2024-04-26T01:42:49.050679+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:07.541593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:59.567760+0000","last_change":"2024-04-26T01:42:02.232823+0000","last_active":"2024-04-26T01:42:59.567760+0000","last_peered":"2024-04-26T01:42:59.567760+0000","last_clean":"2024-04-26T01:42:59.567760+0000","last_became_active":"2024-04-26T01:42:02.231899+0000","last_became_peered":"2024-04-26T01:42:02.231899+0000","last_unstale":"2024-04-26T01:42:59.567760+0000","last_undegraded":"2024-04-26T01:42:59.567760+0000","last_fullsized":"2024-04-26T01:42:59.567760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:23.287305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.071180+0000","last_change":"2024-04-26T01:42:05.405690+0000","last_active":"2024-04-26T01:42:49.071180+0000","last_peered":"2024-04-26T01:42:49.071180+0000","last_clean":"2024-04-26T01:42:49.071180+0000","last_became_active":"2024-04-26T01:42:05.405345+0000","last_became_peered":"2024-04-26T01:42:05.405345+0000","last_unstale":"2024-04-26T01:42:49.071180+0000","last_undegraded":"2024-04-26T01:42:49.071180+0000","last_fullsized":"2024-04-26T01:42:49.071180+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:48.377916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050943+0000","last_change":"2024-04-26T01:42:01.335721+0000","last_active":"2024-04-26T01:42:49.050943+0000","last_peered":"2024-04-26T01:42:49.050943+0000","last_clean":"2024-04-26T01:42:49.050943+0000","last_became_active":"2024-04-26T01:42:01.335149+0000","last_became_peered":"2024-04-26T01:42:01.335149+0000","last_unstale":"2024-04-26T01:42:49.050943+0000","last_undegraded":"2024-04-26T01:42:49.050943+0000","last_fullsized":"2024-04-26T01:42:49.050943+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:36.372033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":362,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053333+0000","last_change":"2024-04-26T01:40:28.086333+0000","last_active":"2024-04-26T01:42:49.053333+0000","last_peered":"2024-04-26T01:42:49.053333+0000","last_clean":"2024-04-26T01:42:49.053333+0000","last_became_active":"2024-04-26T01:40:28.085902+0000","last_became_peered":"2024-04-26T01:40:28.085902+0000","last_unstale":"2024-04-26T01:42:49.053333+0000","last_undegraded":"2024-04-26T01:42:49.053333+0000","last_fullsized":"2024-04-26T01:42:49.053333+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_clean_scrub_stamp":"2024-04-26T01:36:35.357698+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:43.290117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.068625+0000","last_change":"2024-04-26T01:42:05.401896+0000","last_active":"2024-04-26T01:42:49.068625+0000","last_peered":"2024-04-26T01:42:49.068625+0000","last_clean":"2024-04-26T01:42:49.068625+0000","last_became_active":"2024-04-26T01:42:05.401496+0000","last_became_peered":"2024-04-26T01:42:05.401496+0000","last_unstale":"2024-04-26T01:42:49.068625+0000","last_undegraded":"2024-04-26T01:42:49.068625+0000","last_fullsized":"2024-04-26T01:42:49.068625+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:49.758444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053408+0000","last_change":"2024-04-26T01:42:01.329705+0000","last_active":"2024-04-26T01:42:49.053408+0000","last_peered":"2024-04-26T01:42:49.053408+0000","last_clean":"2024-04-26T01:42:49.053408+0000","last_became_active":"2024-04-26T01:42:01.329153+0000","last_became_peered":"2024-04-26T01:42:01.329153+0000","last_unstale":"2024-04-26T01:42:49.053408+0000","last_undegraded":"2024-04-26T01:42:49.053408+0000","last_fullsized":"2024-04-26T01:42:49.053408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:06.747378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.056180+0000","last_change":"2024-04-26T01:42:02.232148+0000","last_active":"2024-04-26T01:42:49.056180+0000","last_peered":"2024-04-26T01:42:49.056180+0000","last_clean":"2024-04-26T01:42:49.056180+0000","last_became_active":"2024-04-26T01:42:02.231739+0000","last_became_peered":"2024-04-26T01:42:02.231739+0000","last_unstale":"2024-04-26T01:42:49.056180+0000","last_undegraded":"2024-04-26T01:42:49.056180+0000","last_fullsized":"2024-04-26T01:42:49.056180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:09.752433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.068717+0000","last_change":"2024-04-26T01:42:05.399938+0000","last_active":"2024-04-26T01:42:49.068717+0000","last_peered":"2024-04-26T01:42:49.068717+0000","last_clean":"2024-04-26T01:42:49.068717+0000","last_became_active":"2024-04-26T01:42:05.399550+0000","last_became_peered":"2024-04-26T01:42:05.399550+0000","last_unstale":"2024-04-26T01:42:49.068717+0000","last_undegraded":"2024-04-26T01:42:49.068717+0000","last_fullsized":"2024-04-26T01:42:49.068717+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:44.142284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"93'12","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:52.418677+0000","last_change":"2024-04-26T01:42:01.337437+0000","last_active":"2024-04-26T01:42:52.418677+0000","last_peered":"2024-04-26T01:42:52.418677+0000","last_clean":"2024-04-26T01:42:52.418677+0000","last_became_active":"2024-04-26T01:42:01.336894+0000","last_became_peered":"2024-04-26T01:42:01.336894+0000","last_unstale":"2024-04-26T01:42:52.418677+0000","last_undegraded":"2024-04-26T01:42:52.418677+0000","last_fullsized":"2024-04-26T01:42:52.418677+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:04:31.093273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.072806+0000","last_change":"2024-04-26T01:42:05.411694+0000","last_active":"2024-04-26T01:42:49.072806+0000","last_peered":"2024-04-26T01:42:49.072806+0000","last_clean":"2024-04-26T01:42:49.072806+0000","last_became_active":"2024-04-26T01:42:05.411300+0000","last_became_peered":"2024-04-26T01:42:05.411300+0000","last_unstale":"2024-04-26T01:42:49.072806+0000","last_undegraded":"2024-04-26T01:42:49.072806+0000","last_fullsized":"2024-04-26T01:42:49.072806+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:42:08.116145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.051041+0000","last_change":"2024-04-26T01:42:01.427501+0000","last_active":"2024-04-26T01:42:49.051041+0000","last_peered":"2024-04-26T01:42:49.051041+0000","last_clean":"2024-04-26T01:42:49.051041+0000","last_became_active":"2024-04-26T01:42:01.426970+0000","last_became_peered":"2024-04-26T01:42:01.426970+0000","last_unstale":"2024-04-26T01:42:49.051041+0000","last_undegraded":"2024-04-26T01:42:49.051041+0000","last_fullsized":"2024-04-26T01:42:49.051041+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:53:53.118296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.061715+0000","last_change":"2024-04-26T01:42:05.414790+0000","last_active":"2024-04-26T01:42:49.061715+0000","last_peered":"2024-04-26T01:42:49.061715+0000","last_clean":"2024-04-26T01:42:49.061715+0000","last_became_active":"2024-04-26T01:42:05.411947+0000","last_became_peered":"2024-04-26T01:42:05.411947+0000","last_unstale":"2024-04-26T01:42:49.061715+0000","last_undegraded":"2024-04-26T01:42:49.061715+0000","last_fullsized":"2024-04-26T01:42:49.061715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:11:34.450815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053787+0000","last_change":"2024-04-26T01:42:02.230539+0000","last_active":"2024-04-26T01:42:49.053787+0000","last_peered":"2024-04-26T01:42:49.053787+0000","last_clean":"2024-04-26T01:42:49.053787+0000","last_became_active":"2024-04-26T01:42:02.230150+0000","last_became_peered":"2024-04-26T01:42:02.230150+0000","last_unstale":"2024-04-26T01:42:49.053787+0000","last_undegraded":"2024-04-26T01:42:49.053787+0000","last_fullsized":"2024-04-26T01:42:49.053787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:08:03.073069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.055881+0000","last_change":"2024-04-26T01:42:05.388877+0000","last_active":"2024-04-26T01:42:49.055881+0000","last_peered":"2024-04-26T01:42:49.055881+0000","last_clean":"2024-04-26T01:42:49.055881+0000","last_became_active":"2024-04-26T01:42:05.388457+0000","last_became_peered":"2024-04-26T01:42:05.388457+0000","last_unstale":"2024-04-26T01:42:49.055881+0000","last_undegraded":"2024-04-26T01:42:49.055881+0000","last_fullsized":"2024-04-26T01:42:49.055881+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:54.436859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070848+0000","last_change":"2024-04-26T01:42:01.343461+0000","last_active":"2024-04-26T01:42:49.070848+0000","last_peered":"2024-04-26T01:42:49.070848+0000","last_clean":"2024-04-26T01:42:49.070848+0000","last_became_active":"2024-04-26T01:42:01.342713+0000","last_became_peered":"2024-04-26T01:42:01.342713+0000","last_unstale":"2024-04-26T01:42:49.070848+0000","last_undegraded":"2024-04-26T01:42:49.070848+0000","last_fullsized":"2024-04-26T01:42:49.070848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:38.471858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058169+0000","last_change":"2024-04-26T01:42:05.415101+0000","last_active":"2024-04-26T01:42:49.058169+0000","last_peered":"2024-04-26T01:42:49.058169+0000","last_clean":"2024-04-26T01:42:49.058169+0000","last_became_active":"2024-04-26T01:42:05.414745+0000","last_became_peered":"2024-04-26T01:42:05.414745+0000","last_unstale":"2024-04-26T01:42:49.058169+0000","last_undegraded":"2024-04-26T01:42:49.058169+0000","last_fullsized":"2024-04-26T01:42:49.058169+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:13:08.573547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.057699+0000","last_change":"2024-04-26T01:42:01.339501+0000","last_active":"2024-04-26T01:42:49.057699+0000","last_peered":"2024-04-26T01:42:49.057699+0000","last_clean":"2024-04-26T01:42:49.057699+0000","last_became_active":"2024-04-26T01:42:01.339132+0000","last_became_peered":"2024-04-26T01:42:01.339132+0000","last_unstale":"2024-04-26T01:42:49.057699+0000","last_undegraded":"2024-04-26T01:42:49.057699+0000","last_fullsized":"2024-04-26T01:42:49.057699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:16.463099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070531+0000","last_change":"2024-04-26T01:42:05.417864+0000","last_active":"2024-04-26T01:42:49.070531+0000","last_peered":"2024-04-26T01:42:49.070531+0000","last_clean":"2024-04-26T01:42:49.070531+0000","last_became_active":"2024-04-26T01:42:05.417414+0000","last_became_peered":"2024-04-26T01:42:05.417414+0000","last_unstale":"2024-04-26T01:42:49.070531+0000","last_undegraded":"2024-04-26T01:42:49.070531+0000","last_fullsized":"2024-04-26T01:42:49.070531+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:23:12.793421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.063769+0000","last_change":"2024-04-26T01:42:49.063769+0000","last_active":"2024-04-26T01:42:49.063769+0000","last_peered":"2024-04-26T01:42:49.063769+0000","last_clean":"2024-04-26T01:42:49.063769+0000","last_became_active":"2024-04-26T01:42:49.057463+0000","last_became_peered":"2024-04-26T01:42:49.057463+0000","last_unstale":"2024-04-26T01:42:49.063769+0000","last_undegraded":"2024-04-26T01:42:49.063769+0000","last_fullsized":"2024-04-26T01:42:49.063769+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:34:54.511304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053377+0000","last_change":"2024-04-26T01:42:05.397980+0000","last_active":"2024-04-26T01:42:49.053377+0000","last_peered":"2024-04-26T01:42:49.053377+0000","last_clean":"2024-04-26T01:42:49.053377+0000","last_became_active":"2024-04-26T01:42:05.397589+0000","last_became_peered":"2024-04-26T01:42:05.397589+0000","last_unstale":"2024-04-26T01:42:49.053377+0000","last_undegraded":"2024-04-26T01:42:49.053377+0000","last_fullsized":"2024-04-26T01:42:49.053377+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:39:55.863566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058561+0000","last_change":"2024-04-26T01:42:02.234313+0000","last_active":"2024-04-26T01:42:49.058561+0000","last_peered":"2024-04-26T01:42:49.058561+0000","last_clean":"2024-04-26T01:42:49.058561+0000","last_became_active":"2024-04-26T01:42:02.233873+0000","last_became_peered":"2024-04-26T01:42:02.233873+0000","last_unstale":"2024-04-26T01:42:49.058561+0000","last_undegraded":"2024-04-26T01:42:49.058561+0000","last_fullsized":"2024-04-26T01:42:49.058561+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:34.092434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053962+0000","last_change":"2024-04-26T01:42:05.387435+0000","last_active":"2024-04-26T01:42:49.053962+0000","last_peered":"2024-04-26T01:42:49.053962+0000","last_clean":"2024-04-26T01:42:49.053962+0000","last_became_active":"2024-04-26T01:42:05.386964+0000","last_became_peered":"2024-04-26T01:42:05.386964+0000","last_unstale":"2024-04-26T01:42:49.053962+0000","last_undegraded":"2024-04-26T01:42:49.053962+0000","last_fullsized":"2024-04-26T01:42:49.053962+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:12:27.905842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050721+0000","last_change":"2024-04-26T01:42:01.326432+0000","last_active":"2024-04-26T01:42:49.050721+0000","last_peered":"2024-04-26T01:42:49.050721+0000","last_clean":"2024-04-26T01:42:49.050721+0000","last_became_active":"2024-04-26T01:42:01.325989+0000","last_became_peered":"2024-04-26T01:42:01.325989+0000","last_unstale":"2024-04-26T01:42:49.050721+0000","last_undegraded":"2024-04-26T01:42:49.050721+0000","last_fullsized":"2024-04-26T01:42:49.050721+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:10.049563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'1","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.053551+0000","last_change":"2024-04-26T01:42:05.412802+0000","last_active":"2024-04-26T01:42:49.053551+0000","last_peered":"2024-04-26T01:42:49.053551+0000","last_clean":"2024-04-26T01:42:49.053551+0000","last_became_active":"2024-04-26T01:42:05.412443+0000","last_became_peered":"2024-04-26T01:42:05.412443+0000","last_unstale":"2024-04-26T01:42:49.053551+0000","last_undegraded":"2024-04-26T01:42:49.053551+0000","last_fullsized":"2024-04-26T01:42:49.053551+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:28:56.360778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.080305+0000","last_change":"2024-04-26T01:42:49.080305+0000","last_active":"2024-04-26T01:42:49.080305+0000","last_peered":"2024-04-26T01:42:49.080305+0000","last_clean":"2024-04-26T01:42:49.080305+0000","last_became_active":"2024-04-26T01:42:49.074476+0000","last_became_peered":"2024-04-26T01:42:49.074476+0000","last_unstale":"2024-04-26T01:42:49.080305+0000","last_undegraded":"2024-04-26T01:42:49.080305+0000","last_fullsized":"2024-04-26T01:42:49.080305+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:58:06.619961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054258+0000","last_change":"2024-04-26T01:42:05.387621+0000","last_active":"2024-04-26T01:42:49.054258+0000","last_peered":"2024-04-26T01:42:49.054258+0000","last_clean":"2024-04-26T01:42:49.054258+0000","last_became_active":"2024-04-26T01:42:05.387165+0000","last_became_peered":"2024-04-26T01:42:05.387165+0000","last_unstale":"2024-04-26T01:42:49.054258+0000","last_undegraded":"2024-04-26T01:42:49.054258+0000","last_fullsized":"2024-04-26T01:42:49.054258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:37.803330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058443+0000","last_change":"2024-04-26T01:42:05.386787+0000","last_active":"2024-04-26T01:42:49.058443+0000","last_peered":"2024-04-26T01:42:49.058443+0000","last_clean":"2024-04-26T01:42:49.058443+0000","last_became_active":"2024-04-26T01:42:05.386410+0000","last_became_peered":"2024-04-26T01:42:05.386410+0000","last_unstale":"2024-04-26T01:42:49.058443+0000","last_undegraded":"2024-04-26T01:42:49.058443+0000","last_fullsized":"2024-04-26T01:42:49.058443+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:42.137191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058087+0000","last_change":"2024-04-26T01:42:05.392142+0000","last_active":"2024-04-26T01:42:49.058087+0000","last_peered":"2024-04-26T01:42:49.058087+0000","last_clean":"2024-04-26T01:42:49.058087+0000","last_became_active":"2024-04-26T01:42:05.391829+0000","last_became_peered":"2024-04-26T01:42:05.391829+0000","last_unstale":"2024-04-26T01:42:49.058087+0000","last_undegraded":"2024-04-26T01:42:49.058087+0000","last_fullsized":"2024-04-26T01:42:49.058087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:00:25.934641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.057545+0000","last_change":"2024-04-26T01:42:05.409184+0000","last_active":"2024-04-26T01:42:49.057545+0000","last_peered":"2024-04-26T01:42:49.057545+0000","last_clean":"2024-04-26T01:42:49.057545+0000","last_became_active":"2024-04-26T01:42:05.408894+0000","last_became_peered":"2024-04-26T01:42:05.408894+0000","last_unstale":"2024-04-26T01:42:49.057545+0000","last_undegraded":"2024-04-26T01:42:49.057545+0000","last_fullsized":"2024-04-26T01:42:49.057545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:53.291844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058733+0000","last_change":"2024-04-26T01:42:05.406940+0000","last_active":"2024-04-26T01:42:49.058733+0000","last_peered":"2024-04-26T01:42:49.058733+0000","last_clean":"2024-04-26T01:42:49.058733+0000","last_became_active":"2024-04-26T01:42:05.406476+0000","last_became_peered":"2024-04-26T01:42:05.406476+0000","last_unstale":"2024-04-26T01:42:49.058733+0000","last_undegraded":"2024-04-26T01:42:49.058733+0000","last_fullsized":"2024-04-26T01:42:49.058733+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:21:37.734633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.068949+0000","last_change":"2024-04-26T01:42:05.413521+0000","last_active":"2024-04-26T01:42:49.068949+0000","last_peered":"2024-04-26T01:42:49.068949+0000","last_clean":"2024-04-26T01:42:49.068949+0000","last_became_active":"2024-04-26T01:42:05.412789+0000","last_became_peered":"2024-04-26T01:42:05.412789+0000","last_unstale":"2024-04-26T01:42:49.068949+0000","last_undegraded":"2024-04-26T01:42:49.068949+0000","last_fullsized":"2024-04-26T01:42:49.068949+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:13.471376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058084+0000","last_change":"2024-04-26T01:42:05.391395+0000","last_active":"2024-04-26T01:42:49.058084+0000","last_peered":"2024-04-26T01:42:49.058084+0000","last_clean":"2024-04-26T01:42:49.058084+0000","last_became_active":"2024-04-26T01:42:05.391048+0000","last_became_peered":"2024-04-26T01:42:05.391048+0000","last_unstale":"2024-04-26T01:42:49.058084+0000","last_undegraded":"2024-04-26T01:42:49.058084+0000","last_fullsized":"2024-04-26T01:42:49.058084+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:45:56.645454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.058824+0000","last_change":"2024-04-26T01:42:05.386868+0000","last_active":"2024-04-26T01:42:49.058824+0000","last_peered":"2024-04-26T01:42:49.058824+0000","last_clean":"2024-04-26T01:42:49.058824+0000","last_became_active":"2024-04-26T01:42:05.386483+0000","last_became_peered":"2024-04-26T01:42:05.386483+0000","last_unstale":"2024-04-26T01:42:49.058824+0000","last_undegraded":"2024-04-26T01:42:49.058824+0000","last_fullsized":"2024-04-26T01:42:49.058824+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:42:44.453076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.072508+0000","last_change":"2024-04-26T01:42:05.406711+0000","last_active":"2024-04-26T01:42:49.072508+0000","last_peered":"2024-04-26T01:42:49.072508+0000","last_clean":"2024-04-26T01:42:49.072508+0000","last_became_active":"2024-04-26T01:42:05.406318+0000","last_became_peered":"2024-04-26T01:42:05.406318+0000","last_unstale":"2024-04-26T01:42:49.072508+0000","last_undegraded":"2024-04-26T01:42:49.072508+0000","last_fullsized":"2024-04-26T01:42:49.072508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:36:16.599678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.050354+0000","last_change":"2024-04-26T01:42:05.406305+0000","last_active":"2024-04-26T01:42:49.050354+0000","last_peered":"2024-04-26T01:42:49.050354+0000","last_clean":"2024-04-26T01:42:49.050354+0000","last_became_active":"2024-04-26T01:42:05.405953+0000","last_became_peered":"2024-04-26T01:42:05.405953+0000","last_unstale":"2024-04-26T01:42:49.050354+0000","last_undegraded":"2024-04-26T01:42:49.050354+0000","last_fullsized":"2024-04-26T01:42:49.050354+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:36.588672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.070537+0000","last_change":"2024-04-26T01:42:05.413757+0000","last_active":"2024-04-26T01:42:49.070537+0000","last_peered":"2024-04-26T01:42:49.070537+0000","last_clean":"2024-04-26T01:42:49.070537+0000","last_became_active":"2024-04-26T01:42:05.413410+0000","last_became_peered":"2024-04-26T01:42:05.413410+0000","last_unstale":"2024-04-26T01:42:49.070537+0000","last_undegraded":"2024-04-26T01:42:49.070537+0000","last_fullsized":"2024-04-26T01:42:49.070537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:30.298443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.054260+0000","last_change":"2024-04-26T01:42:05.412268+0000","last_active":"2024-04-26T01:42:49.054260+0000","last_peered":"2024-04-26T01:42:49.054260+0000","last_clean":"2024-04-26T01:42:49.054260+0000","last_became_active":"2024-04-26T01:42:05.411804+0000","last_became_peered":"2024-04-26T01:42:05.411804+0000","last_unstale":"2024-04-26T01:42:49.054260+0000","last_undegraded":"2024-04-26T01:42:49.054260+0000","last_fullsized":"2024-04-26T01:42:49.054260+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:47.277766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.067407+0000","last_change":"2024-04-26T01:42:05.415563+0000","last_active":"2024-04-26T01:42:49.067407+0000","last_peered":"2024-04-26T01:42:49.067407+0000","last_clean":"2024-04-26T01:42:49.067407+0000","last_became_active":"2024-04-26T01:42:05.414818+0000","last_became_peered":"2024-04-26T01:42:05.414818+0000","last_unstale":"2024-04-26T01:42:49.067407+0000","last_undegraded":"2024-04-26T01:42:49.067407+0000","last_fullsized":"2024-04-26T01:42:49.067407+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:54.713595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T01:42:49.057531+0000","last_change":"2024-04-26T01:42:05.392280+0000","last_active":"2024-04-26T01:42:49.057531+0000","last_peered":"2024-04-26T01:42:49.057531+0000","last_clean":"2024-04-26T01:42:49.057531+0000","last_became_active":"2024-04-26T01:42:05.391943+0000","last_became_peered":"2024-04-26T01:42:05.391943+0000","last_unstale":"2024-04-26T01:42:49.057531+0000","last_undegraded":"2024-04-26T01:42:49.057531+0000","last_fullsized":"2024-04-26T01:42:49.057531+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:38:51.560596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67464,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":245,"num_read_kb":193,"num_write":167,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":202392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142,"ondisk_log_size":142,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285597,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":493526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547233,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41176,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703976,"statfs":{"total":95995035648,"available":95952871424,"internally_reserved":0,"allocated":1720320,"data_stored":1148358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710760,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":493650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972399,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1028096,"data_stored":493458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266739,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1630208,"data_stored":1085891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430264,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":493039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40560,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704592,"statfs":{"total":95995035648,"available":95953502208,"internally_reserved":0,"allocated":1089536,"data_stored":555904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":494310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":557645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":493350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888132,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1028096,"data_stored":493867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691903,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36948,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708204,"statfs":{"total":95995035648,"available":95957200896,"internally_reserved":0,"allocated":1642496,"data_stored":1085184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:43:06.576 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T01:43:06.577 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T01:43:06.577 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T01:43:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:07.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:06 smithi188 ceph-mon[32751]: pgmap v362: 97 pgs: 97 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-26T01:43:07.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:06 smithi188 ceph-mon[32751]: from='client.24638 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:43:07.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:06 smithi029 ceph-mon[32775]: pgmap v362: 97 pgs: 97 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-26T01:43:07.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:06 smithi029 ceph-mon[32775]: from='client.24638 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:43:07.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:06 smithi006 ceph-mon[28419]: pgmap v362: 97 pgs: 97 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-26T01:43:07.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:06 smithi006 ceph-mon[28419]: from='client.24638 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:43:08.931 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:08 smithi006 ceph-mon[28419]: pgmap v363: 97 pgs: 97 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:43:09.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:08 smithi188 ceph-mon[32751]: pgmap v363: 97 pgs: 97 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:43:09.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:08 smithi029 ceph-mon[32775]: pgmap v363: 97 pgs: 97 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:43:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.401+0000 7fd032800700 1 -- 172.21.15.6:0/2142830599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c102ac0 msgr2=0x7fd02c102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.401+0000 7fd032800700 1 --2- 172.21.15.6:0/2142830599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c102ac0 0x7fd02c102ee0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0099c0 tx=0x7fd01c009cd0 comp rx=0 tx=0).stop 2024-04-26T01:43:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.401+0000 7fd032800700 1 -- 172.21.15.6:0/2142830599 shutdown_connections 2024-04-26T01:43:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.401+0000 7fd032800700 1 --2- 172.21.15.6:0/2142830599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd02c104660 0x7fd02c10af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.401+0000 7fd032800700 1 --2- 172.21.15.6:0/2142830599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c102ac0 0x7fd02c102ee0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.401+0000 7fd032800700 1 --2- 172.21.15.6:0/2142830599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd02c103cc0 0x7fd02c104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.401+0000 7fd032800700 1 -- 172.21.15.6:0/2142830599 >> 172.21.15.6:0/2142830599 conn(0x7fd02c0fe060 msgr2=0x7fd02c1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.401+0000 7fd032800700 1 -- 172.21.15.6:0/2142830599 shutdown_connections 2024-04-26T01:43:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.401+0000 7fd032800700 1 -- 172.21.15.6:0/2142830599 wait complete. 2024-04-26T01:43:09.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.403+0000 7fd032800700 1 Processor -- start 2024-04-26T01:43:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.403+0000 7fd032800700 1 -- start start 2024-04-26T01:43:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.404+0000 7fd032800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd02c102ac0 0x7fd02c108850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.404+0000 7fd032800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd02c103cc0 0x7fd02c108d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.404+0000 7fd032800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c104660 0x7fd02c1053d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.404+0000 7fd032800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd02c10e530 con 0x7fd02c103cc0 2024-04-26T01:43:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.404+0000 7fd032800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd02c10e3b0 con 0x7fd02c104660 2024-04-26T01:43:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.404+0000 7fd032800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd02c10e6b0 con 0x7fd02c102ac0 2024-04-26T01:43:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.404+0000 7fd02b400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd02c102ac0 0x7fd02c108850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.404+0000 7fd02aa00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd02c103cc0 0x7fd02c108d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.404+0000 7fd02b400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd02c102ac0 0x7fd02c108850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44570/0 (socket says 172.21.15.6:44570) 2024-04-26T01:43:09.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.404+0000 7fd02b400700 1 -- 172.21.15.6:0/4080122464 learned_addr learned my addr 172.21.15.6:0/4080122464 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:09.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.404+0000 7fd02be00700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c104660 0x7fd02c1053d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:09.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.405+0000 7fd02b400700 1 -- 172.21.15.6:0/4080122464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c104660 msgr2=0x7fd02c1053d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:09.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.405+0000 7fd02b400700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c104660 0x7fd02c1053d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:09.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.405+0000 7fd02b400700 1 -- 172.21.15.6:0/4080122464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd02c103cc0 msgr2=0x7fd02c108d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:09.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.405+0000 7fd02b400700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd02c103cc0 0x7fd02c108d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:09.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.405+0000 7fd02b400700 1 -- 172.21.15.6:0/4080122464 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02c105ca0 con 0x7fd02c102ac0 2024-04-26T01:43:09.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.405+0000 7fd02be00700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c104660 0x7fd02c1053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:43:09.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.405+0000 7fd02b400700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd02c102ac0 0x7fd02c108850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0036f0 tx=0x7fd01c00f710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:09.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.405+0000 7fd023e00700 1 -- 172.21.15.6:0/4080122464 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd01c009e60 con 0x7fd02c102ac0 2024-04-26T01:43:09.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.405+0000 7fd032800700 1 -- 172.21.15.6:0/4080122464 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd02c1a7180 con 0x7fd02c102ac0 2024-04-26T01:43:09.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.405+0000 7fd023e00700 1 -- 172.21.15.6:0/4080122464 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd01c005940 con 0x7fd02c102ac0 2024-04-26T01:43:09.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.405+0000 7fd023e00700 1 -- 172.21.15.6:0/4080122464 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd01c005560 con 0x7fd02c102ac0 2024-04-26T01:43:09.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.406+0000 7fd032800700 1 -- 172.21.15.6:0/4080122464 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd02c1a7570 con 0x7fd02c102ac0 2024-04-26T01:43:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.408+0000 7fd032800700 1 -- 172.21.15.6:0/4080122464 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd02c050f80 con 0x7fd02c102ac0 2024-04-26T01:43:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.409+0000 7fd023e00700 1 -- 172.21.15.6:0/4080122464 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fd01c0038c0 con 0x7fd02c102ac0 2024-04-26T01:43:09.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.411+0000 7fd023e00700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd0100a3e00 0x7fd0100a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.411+0000 7fd023e00700 1 -- 172.21.15.6:0/4080122464 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7fd01c0d1350 con 0x7fd02c102ac0 2024-04-26T01:43:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.411+0000 7fd02aa00700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd0100a3e00 0x7fd0100a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.412+0000 7fd02aa00700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd0100a3e00 0x7fd0100a62c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd018009c60 tx=0x7fd018009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:09.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.416+0000 7fd023e00700 1 -- 172.21.15.6:0/4080122464 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd01c013070 con 0x7fd02c102ac0 2024-04-26T01:43:09.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.742+0000 7fd032800700 1 -- 172.21.15.6:0/4080122464 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fd02c1a7860 con 0x7fd02c102ac0 2024-04-26T01:43:09.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.751+0000 7fd023e00700 1 -- 172.21.15.6:0/4080122464 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fd01c09dbe0 con 0x7fd02c102ac0 2024-04-26T01:43:09.750 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-04-26T01:43:09.750 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCtBitmrNtwLBAANPX+TLrRqUtvvv+oEJ5RjA== 2024-04-26T01:43:09.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.754+0000 7fd032800700 1 -- 172.21.15.6:0/4080122464 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd0100a3e00 msgr2=0x7fd0100a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:09.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.754+0000 7fd032800700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd0100a3e00 0x7fd0100a62c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd018009c60 tx=0x7fd018009260 comp rx=0 tx=0).stop 2024-04-26T01:43:09.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.754+0000 7fd032800700 1 -- 172.21.15.6:0/4080122464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd02c102ac0 msgr2=0x7fd02c108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:09.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.754+0000 7fd032800700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd02c102ac0 0x7fd02c108850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0036f0 tx=0x7fd01c00f710 comp rx=0 tx=0).stop 2024-04-26T01:43:09.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.755+0000 7fd032800700 1 -- 172.21.15.6:0/4080122464 shutdown_connections 2024-04-26T01:43:09.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.755+0000 7fd032800700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c104660 0x7fd02c1053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:09.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.755+0000 7fd032800700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd0100a3e00 0x7fd0100a62c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:09.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.755+0000 7fd032800700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd02c102ac0 0x7fd02c108850 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:09.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.755+0000 7fd032800700 1 --2- 172.21.15.6:0/4080122464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd02c103cc0 0x7fd02c108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:09.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.755+0000 7fd032800700 1 -- 172.21.15.6:0/4080122464 >> 172.21.15.6:0/4080122464 conn(0x7fd02c0fe060 msgr2=0x7fd02c074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:09.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.755+0000 7fd032800700 1 -- 172.21.15.6:0/4080122464 shutdown_connections 2024-04-26T01:43:09.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:09.755+0000 7fd032800700 1 -- 172.21.15.6:0/4080122464 wait complete. 2024-04-26T01:43:10.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:09 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/4080122464' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:43:10.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:09 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:43:10.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:09 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T01:43:10.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:09 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/4080122464' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:43:10.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:09 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:43:10.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:09 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T01:43:10.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:09 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/4080122464' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:43:10.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:09 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:43:10.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:09 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T01:43:10.678 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:43:10.678 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T01:43:10.678 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T01:43:10.819 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T01:43:10.819 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T01:43:10.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T01:43:11.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:10 smithi188 ceph-mon[32751]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:43:11.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:10 smithi029 ceph-mon[32775]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:43:11.315 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:10 smithi006 ceph-mon[28419]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:43:11.321 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:13.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:12 smithi006 ceph-mon[28419]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T01:43:13.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:12 smithi188 ceph-mon[32751]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T01:43:13.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:12 smithi029 ceph-mon[32775]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T01:43:13.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.586+0000 7fab9d000700 1 -- 172.21.15.6:0/3061599894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab98074c70 msgr2=0x7fab98075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.586+0000 7fab9d000700 1 --2- 172.21.15.6:0/3061599894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab98074c70 0x7fab98075050 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fab88009990 tx=0x7fab88009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.586+0000 7fab9d000700 1 -- 172.21.15.6:0/3061599894 shutdown_connections 2024-04-26T01:43:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.586+0000 7fab9d000700 1 --2- 172.21.15.6:0/3061599894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab9810f440 0x7fab98111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.586+0000 7fab9d000700 1 --2- 172.21.15.6:0/3061599894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab98074c70 0x7fab98075050 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.586+0000 7fab9d000700 1 --2- 172.21.15.6:0/3061599894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab98075590 0x7fab9810ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.586+0000 7fab9d000700 1 -- 172.21.15.6:0/3061599894 >> 172.21.15.6:0/3061599894 conn(0x7fab98079ba0 msgr2=0x7fab98079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.586+0000 7fab9d000700 1 -- 172.21.15.6:0/3061599894 shutdown_connections 2024-04-26T01:43:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.586+0000 7fab9d000700 1 -- 172.21.15.6:0/3061599894 wait complete. 2024-04-26T01:43:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.587+0000 7fab9d000700 1 Processor -- start 2024-04-26T01:43:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.588+0000 7fab9d000700 1 -- start start 2024-04-26T01:43:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.588+0000 7fab9d000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab98074c70 0x7fab9819b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.588+0000 7fab9d000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab98075590 0x7fab9819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.588+0000 7fab9d000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9810f440 0x7fab9819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.588+0000 7fab9d000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fab98114e10 con 0x7fab98074c70 2024-04-26T01:43:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.588+0000 7fab9d000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fab98114c90 con 0x7fab9810f440 2024-04-26T01:43:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.588+0000 7fab9d000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fab98114f90 con 0x7fab98075590 2024-04-26T01:43:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.589+0000 7fab8d400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab98075590 0x7fab9819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.589+0000 7fab95e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab98074c70 0x7fab9819b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.589+0000 7fab96800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9810f440 0x7fab9819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.589+0000 7fab8d400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab98075590 0x7fab9819b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44594/0 (socket says 172.21.15.6:44594) 2024-04-26T01:43:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.589+0000 7fab8d400700 1 -- 172.21.15.6:0/2777355636 learned_addr learned my addr 172.21.15.6:0/2777355636 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.589+0000 7fab96800700 1 -- 172.21.15.6:0/2777355636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab98075590 msgr2=0x7fab9819b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.589+0000 7fab96800700 1 --2- 172.21.15.6:0/2777355636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab98075590 0x7fab9819b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.589+0000 7fab96800700 1 -- 172.21.15.6:0/2777355636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab98074c70 msgr2=0x7fab9819b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.589+0000 7fab96800700 1 --2- 172.21.15.6:0/2777355636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab98074c70 0x7fab9819b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.589+0000 7fab96800700 1 -- 172.21.15.6:0/2777355636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab981a0320 con 0x7fab9810f440 2024-04-26T01:43:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.589+0000 7fab96800700 1 --2- 172.21.15.6:0/2777355636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9810f440 0x7fab9819fbe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fab9000e940 tx=0x7fab9000ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.590+0000 7fab8f400700 1 -- 172.21.15.6:0/2777355636 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab90004220 con 0x7fab9810f440 2024-04-26T01:43:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.590+0000 7fab8f400700 1 -- 172.21.15.6:0/2777355636 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab900043c0 con 0x7fab9810f440 2024-04-26T01:43:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.590+0000 7fab9d000700 1 -- 172.21.15.6:0/2777355636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab981a0610 con 0x7fab9810f440 2024-04-26T01:43:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.590+0000 7fab8f400700 1 -- 172.21.15.6:0/2777355636 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab90010700 con 0x7fab9810f440 2024-04-26T01:43:13.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.590+0000 7fab9d000700 1 -- 172.21.15.6:0/2777355636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab981a7e90 con 0x7fab9810f440 2024-04-26T01:43:13.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.591+0000 7fab9d000700 1 -- 172.21.15.6:0/2777355636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab98069730 con 0x7fab9810f440 2024-04-26T01:43:13.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.593+0000 7fab8f400700 1 -- 172.21.15.6:0/2777355636 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fab90004750 con 0x7fab9810f440 2024-04-26T01:43:13.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.594+0000 7fab8f400700 1 --2- 172.21.15.6:0/2777355636 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fab7c0a3d50 0x7fab7c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:13.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.595+0000 7fab8f400700 1 -- 172.21.15.6:0/2777355636 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7fab900d0250 con 0x7fab9810f440 2024-04-26T01:43:13.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.595+0000 7fab95e00700 1 --2- 172.21.15.6:0/2777355636 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fab7c0a3d50 0x7fab7c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:13.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.596+0000 7fab95e00700 1 --2- 172.21.15.6:0/2777355636 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fab7c0a3d50 0x7fab7c0a6210 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fab8800b2e0 tx=0x7fab88005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:13.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.599+0000 7fab8f400700 1 -- 172.21.15.6:0/2777355636 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fab9009ca50 con 0x7fab9810f440 2024-04-26T01:43:13.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.897+0000 7fab9d000700 1 -- 172.21.15.6:0/2777355636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fab981a80f0 con 0x7fab9810f440 2024-04-26T01:43:13.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.903+0000 7fab8f400700 1 -- 172.21.15.6:0/2777355636 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264984 (secure 0 0 0) 0x7fab9001fe10 con 0x7fab9810f440 2024-04-26T01:43:13.906 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:43:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.909+0000 7fab9d000700 1 -- 172.21.15.6:0/2777355636 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fab7c0a3d50 msgr2=0x7fab7c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.909+0000 7fab9d000700 1 --2- 172.21.15.6:0/2777355636 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fab7c0a3d50 0x7fab7c0a6210 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fab8800b2e0 tx=0x7fab88005960 comp rx=0 tx=0).stop 2024-04-26T01:43:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.909+0000 7fab9d000700 1 -- 172.21.15.6:0/2777355636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9810f440 msgr2=0x7fab9819fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.909+0000 7fab9d000700 1 --2- 172.21.15.6:0/2777355636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9810f440 0x7fab9819fbe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fab9000e940 tx=0x7fab9000ec50 comp rx=0 tx=0).stop 2024-04-26T01:43:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.909+0000 7fab9d000700 1 -- 172.21.15.6:0/2777355636 shutdown_connections 2024-04-26T01:43:13.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.909+0000 7fab9d000700 1 --2- 172.21.15.6:0/2777355636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9810f440 0x7fab9819fbe0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.909+0000 7fab9d000700 1 --2- 172.21.15.6:0/2777355636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab98074c70 0x7fab9819b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.909+0000 7fab9d000700 1 --2- 172.21.15.6:0/2777355636 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fab7c0a3d50 0x7fab7c0a6210 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.909+0000 7fab9d000700 1 --2- 172.21.15.6:0/2777355636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab98075590 0x7fab9819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:13.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.909+0000 7fab9d000700 1 -- 172.21.15.6:0/2777355636 >> 172.21.15.6:0/2777355636 conn(0x7fab98079ba0 msgr2=0x7fab98100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:13.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.909+0000 7fab9d000700 1 -- 172.21.15.6:0/2777355636 shutdown_connections 2024-04-26T01:43:13.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:13.909+0000 7fab9d000700 1 -- 172.21.15.6:0/2777355636 wait complete. 2024-04-26T01:43:14.759 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":809793211},{"type":"v1","addr":"172.21.15.6:6801","nonce":809793211}]},"active_addr":"172.21.15.6:6801/809793211","active_change":"2024-04-26T01:33:47.022100+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.6:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":286280546}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":2707582744}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":402969404}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1823104535}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3381009443}]}]} 2024-04-26T01:43:14.764 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T01:43:14.765 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T01:43:14.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:43:15.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:14 smithi006 ceph-mon[28419]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T01:43:15.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:14 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2777355636' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T01:43:15.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:14 smithi188 ceph-mon[32751]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T01:43:15.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:14 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2777355636' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T01:43:15.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:14 smithi029 ceph-mon[32775]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T01:43:15.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:14 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2777355636' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T01:43:15.229 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:17.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:16 smithi188 ceph-mon[32751]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T01:43:17.181 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:16 smithi006 ceph-mon[28419]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T01:43:17.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:16 smithi029 ceph-mon[32775]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T01:43:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.356+0000 7f72dc600700 1 -- 172.21.15.6:0/1134461498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72d4074c70 msgr2=0x7f72d4075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.356+0000 7f72dc600700 1 --2- 172.21.15.6:0/1134461498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72d4074c70 0x7f72d4075050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f72c8009990 tx=0x7f72c8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.356+0000 7f72dc600700 1 -- 172.21.15.6:0/1134461498 shutdown_connections 2024-04-26T01:43:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.356+0000 7f72dc600700 1 --2- 172.21.15.6:0/1134461498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72d410f440 0x7f72d4111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.356+0000 7f72dc600700 1 --2- 172.21.15.6:0/1134461498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72d4074c70 0x7f72d4075050 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.356+0000 7f72dc600700 1 --2- 172.21.15.6:0/1134461498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72d4075590 0x7f72d410ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.356+0000 7f72dc600700 1 -- 172.21.15.6:0/1134461498 >> 172.21.15.6:0/1134461498 conn(0x7f72d4079ba0 msgr2=0x7f72d4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.356+0000 7f72dc600700 1 -- 172.21.15.6:0/1134461498 shutdown_connections 2024-04-26T01:43:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.356+0000 7f72dc600700 1 -- 172.21.15.6:0/1134461498 wait complete. 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.357+0000 7f72dc600700 1 Processor -- start 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.357+0000 7f72dc600700 1 -- start start 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72dc600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72d4074c70 0x7f72d419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72dc600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72d4075590 0x7f72d419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72dc600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72d410f440 0x7f72d419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72dc600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f72d4114df0 con 0x7f72d4074c70 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72dc600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f72d4114c70 con 0x7f72d4075590 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72dc600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f72d4114f70 con 0x7f72d410f440 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72d9c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72d4074c70 0x7f72d419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72d9c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72d4074c70 0x7f72d419b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37226/0 (socket says 172.21.15.6:37226) 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72d9c00700 1 -- 172.21.15.6:0/1914011067 learned_addr learned my addr 172.21.15.6:0/1914011067 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72d9200700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72d4075590 0x7f72d419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72da600700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72d410f440 0x7f72d419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72d9c00700 1 -- 172.21.15.6:0/1914011067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72d410f440 msgr2=0x7f72d419fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72d9c00700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72d410f440 0x7f72d419fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72d9c00700 1 -- 172.21.15.6:0/1914011067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72d4075590 msgr2=0x7f72d419b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72d9c00700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72d4075590 0x7f72d419b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72d9c00700 1 -- 172.21.15.6:0/1914011067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d41a02e0 con 0x7f72d4074c70 2024-04-26T01:43:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72d9200700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72d4075590 0x7f72d419b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.358+0000 7f72da600700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72d410f440 0x7f72d419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.359+0000 7f72d9c00700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72d4074c70 0x7f72d419b320 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f72c80057f0 tx=0x7f72c800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.359+0000 7f72d2000700 1 -- 172.21.15.6:0/1914011067 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8026070 con 0x7f72d4074c70 2024-04-26T01:43:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.359+0000 7f72dc600700 1 -- 172.21.15.6:0/1914011067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d41a0570 con 0x7f72d4074c70 2024-04-26T01:43:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.359+0000 7f72d2000700 1 -- 172.21.15.6:0/1914011067 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72c8021410 con 0x7f72d4074c70 2024-04-26T01:43:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.359+0000 7f72dc600700 1 -- 172.21.15.6:0/1914011067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d41a7e50 con 0x7f72d4074c70 2024-04-26T01:43:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.359+0000 7f72d2000700 1 -- 172.21.15.6:0/1914011067 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c80175f0 con 0x7f72d4074c70 2024-04-26T01:43:17.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.360+0000 7f72dc600700 1 -- 172.21.15.6:0/1914011067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d4069730 con 0x7f72d4074c70 2024-04-26T01:43:17.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.363+0000 7f72d2000700 1 -- 172.21.15.6:0/1914011067 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f72c8029070 con 0x7f72d4074c70 2024-04-26T01:43:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.364+0000 7f72d2000700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f72bc0a3cd0 0x7f72bc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.364+0000 7f72d2000700 1 -- 172.21.15.6:0/1914011067 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7f72c80d1090 con 0x7f72d4074c70 2024-04-26T01:43:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.364+0000 7f72d9200700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f72bc0a3cd0 0x7f72bc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.365+0000 7f72d9200700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f72bc0a3cd0 0x7f72bc0a6190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f72d4111100 tx=0x7f72c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:17.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.367+0000 7f72d2000700 1 -- 172.21.15.6:0/1914011067 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f72c8013070 con 0x7f72d4074c70 2024-04-26T01:43:17.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.607+0000 7f72dc600700 1 -- 172.21.15.6:0/1914011067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f72d402df80 con 0x7f72d4074c70 2024-04-26T01:43:17.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.609+0000 7f72d2000700 1 -- 172.21.15.6:0/1914011067 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+22636 (secure 0 0 0) 0x7f72c809d980 con 0x7f72d4074c70 2024-04-26T01:43:17.608 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:43:17.608 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":93,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:32:29.071544+0000","modified":"2024-04-26T01:42:49.034928+0000","last_up_change":"2024-04-26T01:40:49.915924+0000","last_in_change":"2024-04-26T01:40:29.559647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:36:35.182726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:41:59.612167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:42:04.106467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f164c51d-ce5d-4140-b9e3-a4b84088f5ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6803","nonce":76518486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6805","nonce":76518486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6809","nonce":76518486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6807","nonce":76518486}]},"public_addr":"172.21.15.6:6803/76518486","cluster_addr":"172.21.15.6:6805/76518486","heartbeat_back_addr":"172.21.15.6:6809/76518486","heartbeat_front_addr":"172.21.15.6:6807/76518486","state":["exists","up"]},{"osd":1,"uuid":"57e875dd-ddf1-4ee4-a2a7-9589d7f42818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6801","nonce":400769116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6803","nonce":400769116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6807","nonce":400769116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6805","nonce":400769116}]},"public_addr":"172.21.15.29:6801/400769116","cluster_addr":"172.21.15.29:6803/400769116","heartbeat_back_addr":"172.21.15.29:6807/400769116","heartbeat_front_addr":"172.21.15.29:6805/400769116","state":["exists","up"]},{"osd":2,"uuid":"77e0c294-c48a-4318-938b-4604a533a2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6801","nonce":3508067672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6803","nonce":3508067672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6807","nonce":3508067672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6805","nonce":3508067672}]},"public_addr":"172.21.15.188:6801/3508067672","cluster_addr":"172.21.15.188:6803/3508067672","heartbeat_back_addr":"172.21.15.188:6807/3508067672","heartbeat_front_addr":"172.21.15.188:6805/3508067672","state":["exists","up"]},{"osd":3,"uuid":"6393dac5-996c-47af-9687-0f1393ecc483","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6811","nonce":1987311118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6813","nonce":1987311118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6817","nonce":1987311118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6815","nonce":1987311118}]},"public_addr":"172.21.15.6:6811/1987311118","cluster_addr":"172.21.15.6:6813/1987311118","heartbeat_back_addr":"172.21.15.6:6817/1987311118","heartbeat_front_addr":"172.21.15.6:6815/1987311118","state":["exists","up"]},{"osd":4,"uuid":"f2c1a535-26a0-46c6-a6d0-c2529c786739","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6809","nonce":2125876609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6811","nonce":2125876609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6815","nonce":2125876609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6813","nonce":2125876609}]},"public_addr":"172.21.15.29:6809/2125876609","cluster_addr":"172.21.15.29:6811/2125876609","heartbeat_back_addr":"172.21.15.29:6815/2125876609","heartbeat_front_addr":"172.21.15.29:6813/2125876609","state":["exists","up"]},{"osd":5,"uuid":"9788bb0a-0d79-4743-bce6-80f5342e8f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6809","nonce":2092077007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6811","nonce":2092077007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6815","nonce":2092077007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6813","nonce":2092077007}]},"public_addr":"172.21.15.188:6809/2092077007","cluster_addr":"172.21.15.188:6811/2092077007","heartbeat_back_addr":"172.21.15.188:6815/2092077007","heartbeat_front_addr":"172.21.15.188:6813/2092077007","state":["exists","up"]},{"osd":6,"uuid":"fd8584e1-69ee-4c30-85e4-a637357d1b6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6819","nonce":3256016695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6821","nonce":3256016695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6825","nonce":3256016695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6823","nonce":3256016695}]},"public_addr":"172.21.15.6:6819/3256016695","cluster_addr":"172.21.15.6:6821/3256016695","heartbeat_back_addr":"172.21.15.6:6825/3256016695","heartbeat_front_addr":"172.21.15.6:6823/3256016695","state":["exists","up"]},{"osd":7,"uuid":"6b7e6784-d574-42da-8416-a74a3b6b3755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6817","nonce":3478594037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6819","nonce":3478594037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6823","nonce":3478594037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6821","nonce":3478594037}]},"public_addr":"172.21.15.29:6817/3478594037","cluster_addr":"172.21.15.29:6819/3478594037","heartbeat_back_addr":"172.21.15.29:6823/3478594037","heartbeat_front_addr":"172.21.15.29:6821/3478594037","state":["exists","up"]},{"osd":8,"uuid":"30cd2ea1-2767-4062-8bd9-47214e4b4b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6817","nonce":3290364376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6819","nonce":3290364376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6823","nonce":3290364376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6821","nonce":3290364376}]},"public_addr":"172.21.15.188:6817/3290364376","cluster_addr":"172.21.15.188:6819/3290364376","heartbeat_back_addr":"172.21.15.188:6823/3290364376","heartbeat_front_addr":"172.21.15.188:6821/3290364376","state":["exists","up"]},{"osd":9,"uuid":"f033cdc0-1178-4a7f-a338-69ff32416087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6827","nonce":1918893912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6829","nonce":1918893912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6833","nonce":1918893912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6831","nonce":1918893912}]},"public_addr":"172.21.15.6:6827/1918893912","cluster_addr":"172.21.15.6:6829/1918893912","heartbeat_back_addr":"172.21.15.6:6833/1918893912","heartbeat_front_addr":"172.21.15.6:6831/1918893912","state":["exists","up"]},{"osd":10,"uuid":"04e9b5ef-ac0c-482b-a69e-a7121b36e7da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6825","nonce":1395777729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6827","nonce":1395777729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6831","nonce":1395777729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6829","nonce":1395777729}]},"public_addr":"172.21.15.29:6825/1395777729","cluster_addr":"172.21.15.29:6827/1395777729","heartbeat_back_addr":"172.21.15.29:6831/1395777729","heartbeat_front_addr":"172.21.15.29:6829/1395777729","state":["exists","up"]},{"osd":11,"uuid":"cb1638ce-4189-4241-b63e-b63c607e90ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6825","nonce":3322197880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6827","nonce":3322197880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6831","nonce":3322197880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6829","nonce":3322197880}]},"public_addr":"172.21.15.188:6825/3322197880","cluster_addr":"172.21.15.188:6827/3322197880","heartbeat_back_addr":"172.21.15.188:6831/3322197880","heartbeat_front_addr":"172.21.15.188:6829/3322197880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:41.553455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:07.641467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:31.748891+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:03.743553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:30.065262+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:54.976527+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:29.348206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:56.285928+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:21.246409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:54.468561+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:22.093015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:47.342379+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.19","mappings":[{"from":3,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/3143272062":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1325707596":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6800/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6801/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1359518071":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6801/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/2813598711":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/3140150861":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6800/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/65701414":"2024-04-27T01:33:47.021882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:43:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.613+0000 7f72dc600700 1 -- 172.21.15.6:0/1914011067 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f72bc0a3cd0 msgr2=0x7f72bc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.613+0000 7f72dc600700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f72bc0a3cd0 0x7f72bc0a6190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f72d4111100 tx=0x7f72c0009260 comp rx=0 tx=0).stop 2024-04-26T01:43:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.613+0000 7f72dc600700 1 -- 172.21.15.6:0/1914011067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72d4074c70 msgr2=0x7f72d419b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.613+0000 7f72dc600700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72d4074c70 0x7f72d419b320 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f72c80057f0 tx=0x7f72c800bfd0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.613+0000 7f72dc600700 1 -- 172.21.15.6:0/1914011067 shutdown_connections 2024-04-26T01:43:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.613+0000 7f72dc600700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f72bc0a3cd0 0x7f72bc0a6190 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.613+0000 7f72dc600700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72d410f440 0x7f72d419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.613+0000 7f72dc600700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72d4074c70 0x7f72d419b320 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.613+0000 7f72dc600700 1 --2- 172.21.15.6:0/1914011067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72d4075590 0x7f72d419b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.613+0000 7f72dc600700 1 -- 172.21.15.6:0/1914011067 >> 172.21.15.6:0/1914011067 conn(0x7f72d4079ba0 msgr2=0x7f72d40ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.613+0000 7f72dc600700 1 -- 172.21.15.6:0/1914011067 shutdown_connections 2024-04-26T01:43:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:17.613+0000 7f72dc600700 1 -- 172.21.15.6:0/1914011067 wait complete. 2024-04-26T01:43:18.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:17 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1914011067' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:43:18.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:17 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1914011067' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:43:18.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:17 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1914011067' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:43:18.498 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T01:43:18.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:43:18.965 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:19.138 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:18 smithi006 ceph-mon[28419]: pgmap v368: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T01:43:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:18 smithi188 ceph-mon[32751]: pgmap v368: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T01:43:19.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:18 smithi029 ceph-mon[32775]: pgmap v368: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T01:43:21.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.031+0000 7fddb0a00700 1 -- 172.21.15.6:0/849892330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddac105760 msgr2=0x7fddac1097b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:21.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.031+0000 7fddb0a00700 1 --2- 172.21.15.6:0/849892330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddac105760 0x7fddac1097b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdd94009990 tx=0x7fdd94009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:21.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.031+0000 7fddb0a00700 1 -- 172.21.15.6:0/849892330 shutdown_connections 2024-04-26T01:43:21.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.031+0000 7fddb0a00700 1 --2- 172.21.15.6:0/849892330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddac109e90 0x7fddac111a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:21.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.031+0000 7fddb0a00700 1 --2- 172.21.15.6:0/849892330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddac104db0 0x7fddac105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:21.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.031+0000 7fddb0a00700 1 --2- 172.21.15.6:0/849892330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddac105760 0x7fddac1097b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:21.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.031+0000 7fddb0a00700 1 -- 172.21.15.6:0/849892330 >> 172.21.15.6:0/849892330 conn(0x7fddac100680 msgr2=0x7fddac102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:21.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.031+0000 7fddb0a00700 1 -- 172.21.15.6:0/849892330 shutdown_connections 2024-04-26T01:43:21.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.031+0000 7fddb0a00700 1 -- 172.21.15.6:0/849892330 wait complete. 2024-04-26T01:43:21.030 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:20 smithi006 ceph-mon[28419]: pgmap v369: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-26T01:43:21.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.032+0000 7fddb0a00700 1 Processor -- start 2024-04-26T01:43:21.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.032+0000 7fddb0a00700 1 -- start start 2024-04-26T01:43:21.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.033+0000 7fddb0a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddac104db0 0x7fddac19b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:21.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.033+0000 7fddb0a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddac105760 0x7fddac19b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:21.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.033+0000 7fddb0a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddac109e90 0x7fddac19fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:21.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.033+0000 7fddb0a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fddac114d60 con 0x7fddac104db0 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.033+0000 7fddb0a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fddac114be0 con 0x7fddac109e90 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.033+0000 7fddb0a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fddac114ee0 con 0x7fddac105760 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.033+0000 7fddaa800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddac104db0 0x7fddac19b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.033+0000 7fddaa800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddac104db0 0x7fddac19b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53604/0 (socket says 172.21.15.6:53604) 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.033+0000 7fddaa800700 1 -- 172.21.15.6:0/1192472440 learned_addr learned my addr 172.21.15.6:0/1192472440 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.033+0000 7fdda3e00700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddac105760 0x7fddac19b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.033+0000 7fddab200700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddac109e90 0x7fddac19fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.034+0000 7fddaa800700 1 -- 172.21.15.6:0/1192472440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddac105760 msgr2=0x7fddac19b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.034+0000 7fddaa800700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddac105760 0x7fddac19b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.034+0000 7fddaa800700 1 -- 172.21.15.6:0/1192472440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddac109e90 msgr2=0x7fddac19fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.034+0000 7fddaa800700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddac109e90 0x7fddac19fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.034+0000 7fddaa800700 1 -- 172.21.15.6:0/1192472440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddac1a0270 con 0x7fddac104db0 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.034+0000 7fdda3e00700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddac105760 0x7fddac19b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.034+0000 7fddab200700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddac109e90 0x7fddac19fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.034+0000 7fddaa800700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddac104db0 0x7fddac19b2b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00e9a0 tx=0x7fdd9c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.034+0000 7fdda3400700 1 -- 172.21.15.6:0/1192472440 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c00ca10 con 0x7fddac104db0 2024-04-26T01:43:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.034+0000 7fddb0a00700 1 -- 172.21.15.6:0/1192472440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddac1a0560 con 0x7fddac104db0 2024-04-26T01:43:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.034+0000 7fdda3400700 1 -- 172.21.15.6:0/1192472440 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd9c00cbb0 con 0x7fddac104db0 2024-04-26T01:43:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.034+0000 7fdda3400700 1 -- 172.21.15.6:0/1192472440 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c018680 con 0x7fddac104db0 2024-04-26T01:43:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.035+0000 7fddb0a00700 1 -- 172.21.15.6:0/1192472440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddac1a7de0 con 0x7fddac104db0 2024-04-26T01:43:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.036+0000 7fddb0a00700 1 -- 172.21.15.6:0/1192472440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddac069730 con 0x7fddac104db0 2024-04-26T01:43:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.038+0000 7fdda3400700 1 -- 172.21.15.6:0/1192472440 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fdd9c010430 con 0x7fddac104db0 2024-04-26T01:43:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.039+0000 7fdda3400700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdd900a3cd0 0x7fdd900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.039+0000 7fdda3400700 1 -- 172.21.15.6:0/1192472440 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7fdd9c0d0770 con 0x7fddac104db0 2024-04-26T01:43:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.040+0000 7fdda3e00700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdd900a3cd0 0x7fdd900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:21.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.040+0000 7fdda3e00700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdd900a3cd0 0x7fdd900a6190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fddac103d70 tx=0x7fdd9400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:21.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.044+0000 7fdda3400700 1 -- 172.21.15.6:0/1192472440 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd9c09cf70 con 0x7fddac104db0 2024-04-26T01:43:21.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:20 smithi188 ceph-mon[32751]: pgmap v369: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-26T01:43:21.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:20 smithi029 ceph-mon[32775]: pgmap v369: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-26T01:43:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.277+0000 7fddb0a00700 1 -- 172.21.15.6:0/1192472440 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fddac02df80 con 0x7fddac104db0 2024-04-26T01:43:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.282+0000 7fdda3400700 1 -- 172.21.15.6:0/1192472440 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+22636 (secure 0 0 0) 0x7fdd9c09c870 con 0x7fddac104db0 2024-04-26T01:43:21.280 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:43:21.280 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":93,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:32:29.071544+0000","modified":"2024-04-26T01:42:49.034928+0000","last_up_change":"2024-04-26T01:40:49.915924+0000","last_in_change":"2024-04-26T01:40:29.559647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:36:35.182726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:41:59.612167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:42:04.106467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f164c51d-ce5d-4140-b9e3-a4b84088f5ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6803","nonce":76518486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6805","nonce":76518486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6809","nonce":76518486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6807","nonce":76518486}]},"public_addr":"172.21.15.6:6803/76518486","cluster_addr":"172.21.15.6:6805/76518486","heartbeat_back_addr":"172.21.15.6:6809/76518486","heartbeat_front_addr":"172.21.15.6:6807/76518486","state":["exists","up"]},{"osd":1,"uuid":"57e875dd-ddf1-4ee4-a2a7-9589d7f42818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6801","nonce":400769116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6803","nonce":400769116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6807","nonce":400769116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6805","nonce":400769116}]},"public_addr":"172.21.15.29:6801/400769116","cluster_addr":"172.21.15.29:6803/400769116","heartbeat_back_addr":"172.21.15.29:6807/400769116","heartbeat_front_addr":"172.21.15.29:6805/400769116","state":["exists","up"]},{"osd":2,"uuid":"77e0c294-c48a-4318-938b-4604a533a2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6801","nonce":3508067672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6803","nonce":3508067672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6807","nonce":3508067672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6805","nonce":3508067672}]},"public_addr":"172.21.15.188:6801/3508067672","cluster_addr":"172.21.15.188:6803/3508067672","heartbeat_back_addr":"172.21.15.188:6807/3508067672","heartbeat_front_addr":"172.21.15.188:6805/3508067672","state":["exists","up"]},{"osd":3,"uuid":"6393dac5-996c-47af-9687-0f1393ecc483","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6811","nonce":1987311118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6813","nonce":1987311118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6817","nonce":1987311118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6815","nonce":1987311118}]},"public_addr":"172.21.15.6:6811/1987311118","cluster_addr":"172.21.15.6:6813/1987311118","heartbeat_back_addr":"172.21.15.6:6817/1987311118","heartbeat_front_addr":"172.21.15.6:6815/1987311118","state":["exists","up"]},{"osd":4,"uuid":"f2c1a535-26a0-46c6-a6d0-c2529c786739","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6809","nonce":2125876609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6811","nonce":2125876609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6815","nonce":2125876609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6813","nonce":2125876609}]},"public_addr":"172.21.15.29:6809/2125876609","cluster_addr":"172.21.15.29:6811/2125876609","heartbeat_back_addr":"172.21.15.29:6815/2125876609","heartbeat_front_addr":"172.21.15.29:6813/2125876609","state":["exists","up"]},{"osd":5,"uuid":"9788bb0a-0d79-4743-bce6-80f5342e8f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6809","nonce":2092077007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6811","nonce":2092077007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6815","nonce":2092077007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6813","nonce":2092077007}]},"public_addr":"172.21.15.188:6809/2092077007","cluster_addr":"172.21.15.188:6811/2092077007","heartbeat_back_addr":"172.21.15.188:6815/2092077007","heartbeat_front_addr":"172.21.15.188:6813/2092077007","state":["exists","up"]},{"osd":6,"uuid":"fd8584e1-69ee-4c30-85e4-a637357d1b6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6819","nonce":3256016695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6821","nonce":3256016695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6825","nonce":3256016695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6823","nonce":3256016695}]},"public_addr":"172.21.15.6:6819/3256016695","cluster_addr":"172.21.15.6:6821/3256016695","heartbeat_back_addr":"172.21.15.6:6825/3256016695","heartbeat_front_addr":"172.21.15.6:6823/3256016695","state":["exists","up"]},{"osd":7,"uuid":"6b7e6784-d574-42da-8416-a74a3b6b3755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6817","nonce":3478594037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6819","nonce":3478594037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6823","nonce":3478594037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6821","nonce":3478594037}]},"public_addr":"172.21.15.29:6817/3478594037","cluster_addr":"172.21.15.29:6819/3478594037","heartbeat_back_addr":"172.21.15.29:6823/3478594037","heartbeat_front_addr":"172.21.15.29:6821/3478594037","state":["exists","up"]},{"osd":8,"uuid":"30cd2ea1-2767-4062-8bd9-47214e4b4b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6817","nonce":3290364376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6819","nonce":3290364376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6823","nonce":3290364376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6821","nonce":3290364376}]},"public_addr":"172.21.15.188:6817/3290364376","cluster_addr":"172.21.15.188:6819/3290364376","heartbeat_back_addr":"172.21.15.188:6823/3290364376","heartbeat_front_addr":"172.21.15.188:6821/3290364376","state":["exists","up"]},{"osd":9,"uuid":"f033cdc0-1178-4a7f-a338-69ff32416087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6827","nonce":1918893912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6829","nonce":1918893912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6833","nonce":1918893912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6831","nonce":1918893912}]},"public_addr":"172.21.15.6:6827/1918893912","cluster_addr":"172.21.15.6:6829/1918893912","heartbeat_back_addr":"172.21.15.6:6833/1918893912","heartbeat_front_addr":"172.21.15.6:6831/1918893912","state":["exists","up"]},{"osd":10,"uuid":"04e9b5ef-ac0c-482b-a69e-a7121b36e7da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6825","nonce":1395777729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6827","nonce":1395777729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6831","nonce":1395777729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6829","nonce":1395777729}]},"public_addr":"172.21.15.29:6825/1395777729","cluster_addr":"172.21.15.29:6827/1395777729","heartbeat_back_addr":"172.21.15.29:6831/1395777729","heartbeat_front_addr":"172.21.15.29:6829/1395777729","state":["exists","up"]},{"osd":11,"uuid":"cb1638ce-4189-4241-b63e-b63c607e90ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6825","nonce":3322197880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6827","nonce":3322197880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6831","nonce":3322197880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6829","nonce":3322197880}]},"public_addr":"172.21.15.188:6825/3322197880","cluster_addr":"172.21.15.188:6827/3322197880","heartbeat_back_addr":"172.21.15.188:6831/3322197880","heartbeat_front_addr":"172.21.15.188:6829/3322197880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:41.553455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:07.641467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:31.748891+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:03.743553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:30.065262+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:54.976527+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:29.348206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:56.285928+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:21.246409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:54.468561+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:22.093015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:47.342379+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.19","mappings":[{"from":3,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/3143272062":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1325707596":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6800/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6801/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1359518071":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6801/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/2813598711":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/3140150861":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6800/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/65701414":"2024-04-27T01:33:47.021882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:43:21.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.285+0000 7fddb0a00700 1 -- 172.21.15.6:0/1192472440 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdd900a3cd0 msgr2=0x7fdd900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:21.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.285+0000 7fddb0a00700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdd900a3cd0 0x7fdd900a6190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fddac103d70 tx=0x7fdd9400b260 comp rx=0 tx=0).stop 2024-04-26T01:43:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.285+0000 7fddb0a00700 1 -- 172.21.15.6:0/1192472440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddac104db0 msgr2=0x7fddac19b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.285+0000 7fddb0a00700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddac104db0 0x7fddac19b2b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00e9a0 tx=0x7fdd9c00ecb0 comp rx=0 tx=0).stop 2024-04-26T01:43:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.285+0000 7fddb0a00700 1 -- 172.21.15.6:0/1192472440 shutdown_connections 2024-04-26T01:43:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.285+0000 7fddb0a00700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddac109e90 0x7fddac19fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.285+0000 7fddb0a00700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddac104db0 0x7fddac19b2b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.285+0000 7fddb0a00700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fdd900a3cd0 0x7fdd900a6190 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.285+0000 7fddb0a00700 1 --2- 172.21.15.6:0/1192472440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddac105760 0x7fddac19b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.285+0000 7fddb0a00700 1 -- 172.21.15.6:0/1192472440 >> 172.21.15.6:0/1192472440 conn(0x7fddac100680 msgr2=0x7fddac102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.286+0000 7fddb0a00700 1 -- 172.21.15.6:0/1192472440 shutdown_connections 2024-04-26T01:43:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:21.286+0000 7fddb0a00700 1 -- 172.21.15.6:0/1192472440 wait complete. 2024-04-26T01:43:22.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:21 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1192472440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:43:22.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T01:43:22.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T01:43:22.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T01:43:22.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T01:43:22.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T01:43:22.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T01:43:22.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T01:43:22.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T01:43:22.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-26T01:43:22.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-26T01:43:22.080 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-26T01:43:22.080 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-26T01:43:22.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:21 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1192472440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:43:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:21 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1192472440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:43:23.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:22 smithi029 ceph-mon[32775]: pgmap v370: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:23.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:22 smithi006 ceph-mon[28419]: pgmap v370: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:23.361 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:23.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:22 smithi188 ceph-mon[32751]: pgmap v370: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:24.284 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:24.294 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:24.296 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:24.297 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:24.297 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:24.298 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:24.298 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:24.319 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:24.319 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:24.323 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:24.437 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:25.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:24 smithi006 ceph-mon[28419]: pgmap v371: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:25.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:24 smithi029 ceph-mon[32775]: pgmap v371: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:25.191 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:24 smithi188 ceph-mon[32751]: pgmap v371: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:27.021 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:26 smithi006 ceph-mon[28419]: pgmap v372: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:27.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:26 smithi029 ceph-mon[32775]: pgmap v372: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:27.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:26 smithi188 ceph-mon[32751]: pgmap v372: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:28.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.193+0000 7f22d8e00700 1 -- 172.21.15.6:0/2596022223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d406afe0 msgr2=0x7f22d406b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:28.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.193+0000 7f22d8e00700 1 --2- 172.21.15.6:0/2596022223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d406afe0 0x7f22d406b440 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f22c00098f0 tx=0x7f22c0009c00 comp rx=0 tx=0).stop 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.193+0000 7f22d8e00700 1 -- 172.21.15.6:0/2596022223 shutdown_connections 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.193+0000 7f22d8e00700 1 --2- 172.21.15.6:0/2596022223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d406b980 0x7f22d410f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.193+0000 7f22d8e00700 1 --2- 172.21.15.6:0/2596022223 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d4108a20 0x7f22d4108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.193+0000 7f22d8e00700 1 --2- 172.21.15.6:0/2596022223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d406afe0 0x7f22d406b440 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.193+0000 7f22d8e00700 1 -- 172.21.15.6:0/2596022223 >> 172.21.15.6:0/2596022223 conn(0x7f22d40788b0 msgr2=0x7f22d4078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.193+0000 7f22d8e00700 1 -- 172.21.15.6:0/2596022223 shutdown_connections 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.193+0000 7f22d8e00700 1 -- 172.21.15.6:0/2596022223 wait complete. 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.194+0000 7f22d8e00700 1 Processor -- start 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.194+0000 7f22d8e00700 1 -- start start 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d8e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d406afe0 0x7f22d419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d8e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d406b980 0x7f22d419b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d8e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d4108a20 0x7f22d419fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d8e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f22d4112c10 con 0x7f22d406b980 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d8e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f22d4112a90 con 0x7f22d4108a20 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d8e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f22d4112d90 con 0x7f22d406afe0 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d1400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d406b980 0x7f22d419b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d1400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d406b980 0x7f22d419b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53636/0 (socket says 172.21.15.6:53636) 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d1400700 1 -- 172.21.15.6:0/880482468 learned_addr learned my addr 172.21.15.6:0/880482468 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d1e00700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d406afe0 0x7f22d419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d2800700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d4108a20 0x7f22d419fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d1400700 1 -- 172.21.15.6:0/880482468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d406afe0 msgr2=0x7f22d419b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d1400700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d406afe0 0x7f22d419b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d1400700 1 -- 172.21.15.6:0/880482468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d4108a20 msgr2=0x7f22d419fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d1400700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22d4108a20 0x7f22d419fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.195+0000 7f22d1400700 1 -- 172.21.15.6:0/880482468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d41a03c0 con 0x7f22d406b980 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.196+0000 7f22d1400700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d406b980 0x7f22d419b940 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009230 tx=0x7f22c000fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.196+0000 7f22caa00700 1 -- 172.21.15.6:0/880482468 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0005140 con 0x7f22d406b980 2024-04-26T01:43:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.196+0000 7f22caa00700 1 -- 172.21.15.6:0/880482468 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22c000b560 con 0x7f22d406b980 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.196+0000 7f22d8e00700 1 -- 172.21.15.6:0/880482468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d41a0650 con 0x7f22d406b980 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.196+0000 7f22caa00700 1 -- 172.21.15.6:0/880482468 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0003970 con 0x7f22d406b980 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.196+0000 7f22d8e00700 1 -- 172.21.15.6:0/880482468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d41a7f30 con 0x7f22d406b980 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.197+0000 7f22d8e00700 1 -- 172.21.15.6:0/880482468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f22d4102a00 con 0x7f22d406b980 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.199+0000 7f22caa00700 1 -- 172.21.15.6:0/880482468 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f22c000ba50 con 0x7f22d406b980 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.201+0000 7f22caa00700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f22b80a3cd0 0x7f22b80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.201+0000 7f22caa00700 1 -- 172.21.15.6:0/880482468 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7f22c00d1960 con 0x7f22d406b980 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.201+0000 7f22d1e00700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f22b80a3cd0 0x7f22b80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.201+0000 7f22caa00700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729] conn(0x7f22b80ad7e0 0x7f22b80afc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.201+0000 7f22caa00700 1 -- 172.21.15.6:0/880482468 --> [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22c000bdb0 con 0x7f22b80ad7e0 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.201+0000 7f22caa00700 1 -- 172.21.15.6:0/880482468 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f22c00d1d10 con 0x7f22d406b980 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.202+0000 7f22d2800700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729] conn(0x7f22b80ad7e0 0x7f22b80afc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.202+0000 7f22d1e00700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f22b80a3cd0 0x7f22b80a6190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f22c4005fd0 tx=0x7f22c4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.202+0000 7f22d2800700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729] conn(0x7f22b80ad7e0 0x7f22b80afc80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.203+0000 7f22caa00700 1 -- 172.21.15.6:0/880482468 <== osd.10 v2:172.21.15.29:6824/1395777729 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22c000bdb0 con 0x7f22b80ad7e0 2024-04-26T01:43:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.222+0000 7f22d8e00700 1 -- 172.21.15.6:0/880482468 --> [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f22d4104160 con 0x7f22b80ad7e0 2024-04-26T01:43:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.223+0000 7f22caa00700 1 -- 172.21.15.6:0/880482468 <== osd.10 v2:172.21.15.29:6824/1395777729 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f22d4104160 con 0x7f22b80ad7e0 2024-04-26T01:43:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.223+0000 7f22d8e00700 1 -- 172.21.15.6:0/880482468 >> [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729] conn(0x7f22b80ad7e0 msgr2=0x7f22b80afc80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.223+0000 7f22d8e00700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729] conn(0x7f22b80ad7e0 0x7f22b80afc80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.223+0000 7f22d8e00700 1 -- 172.21.15.6:0/880482468 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f22b80a3cd0 msgr2=0x7f22b80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.223+0000 7f22d8e00700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f22b80a3cd0 0x7f22b80a6190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f22c4005fd0 tx=0x7f22c4009310 comp rx=0 tx=0).stop 2024-04-26T01:43:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.223+0000 7f22d8e00700 1 -- 172.21.15.6:0/880482468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d406b980 msgr2=0x7f22d419b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.223+0000 7f22d8e00700 1 --2- 172.21.15.6:0/880482468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22d406b980 0x7f22d419b940 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009230 tx=0x7f22c000fa70 comp rx=0 tx=0).stop 2024-04-26T01:43:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.224+0000 7f22d2800700 1 -- 172.21.15.6:0/880482468 reap_dead start 2024-04-26T01:43:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.224+0000 7f22d8e00700 1 -- 172.21.15.6:0/880482468 shutdown_connections 2024-04-26T01:43:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.224+0000 7f22d8e00700 1 -- 172.21.15.6:0/880482468 >> 172.21.15.6:0/880482468 conn(0x7f22d40788b0 msgr2=0x7f22d40ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.224+0000 7f22d8e00700 1 -- 172.21.15.6:0/880482468 shutdown_connections 2024-04-26T01:43:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:27.224+0000 7f22d8e00700 1 -- 172.21.15.6:0/880482468 wait complete. 2024-04-26T01:43:29.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:28 smithi029 ceph-mon[32775]: pgmap v373: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:29.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:28 smithi006 ceph-mon[28419]: pgmap v373: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:29.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:28 smithi188 ceph-mon[32751]: pgmap v373: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:29.410 INFO:teuthology.orchestra.run.smithi006.stdout:322122547238 2024-04-26T01:43:29.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-26T01:43:31.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:30 smithi006 ceph-mon[28419]: pgmap v374: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:31.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:30 smithi029 ceph-mon[32775]: pgmap v374: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:31.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:30 smithi188 ceph-mon[32751]: pgmap v374: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:33.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:32 smithi029 ceph-mon[32775]: pgmap v375: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:33.249 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:32 smithi006 ceph-mon[28419]: pgmap v375: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:33.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:32 smithi188 ceph-mon[32751]: pgmap v375: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.605+0000 7fb260200700 1 -- 172.21.15.6:0/1694706145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb25806a750 msgr2=0x7fb25806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.605+0000 7fb260200700 1 --2- 172.21.15.6:0/1694706145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb25806a750 0x7fb25806abb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb244009990 tx=0x7fb244009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.606+0000 7fb260200700 1 -- 172.21.15.6:0/1694706145 shutdown_connections 2024-04-26T01:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.606+0000 7fb260200700 1 --2- 172.21.15.6:0/1694706145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25806b0f0 0x7fb25810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.606+0000 7fb260200700 1 --2- 172.21.15.6:0/1694706145 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb258108a30 0x7fb258108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.606+0000 7fb260200700 1 --2- 172.21.15.6:0/1694706145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb25806a750 0x7fb25806abb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.606+0000 7fb260200700 1 -- 172.21.15.6:0/1694706145 >> 172.21.15.6:0/1694706145 conn(0x7fb2580788f0 msgr2=0x7fb258078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.606+0000 7fb260200700 1 -- 172.21.15.6:0/1694706145 shutdown_connections 2024-04-26T01:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.606+0000 7fb260200700 1 -- 172.21.15.6:0/1694706145 wait complete. 2024-04-26T01:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.607+0000 7fb260200700 1 Processor -- start 2024-04-26T01:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.607+0000 7fb260200700 1 -- start start 2024-04-26T01:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.608+0000 7fb260200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25806a750 0x7fb25819b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.608+0000 7fb260200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb25806b0f0 0x7fb25819b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.608+0000 7fb260200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb258108a30 0x7fb25819fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.608+0000 7fb260200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb258112c60 con 0x7fb258108a30 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.608+0000 7fb260200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb258112ae0 con 0x7fb25806a750 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.608+0000 7fb260200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb258112de0 con 0x7fb25806b0f0 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.608+0000 7fb25d800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25806a750 0x7fb25819b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.608+0000 7fb25d800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25806a750 0x7fb25819b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:60226/0 (socket says 172.21.15.6:60226) 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.608+0000 7fb25e200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb258108a30 0x7fb25819fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.608+0000 7fb25d800700 1 -- 172.21.15.6:0/2019925461 learned_addr learned my addr 172.21.15.6:0/2019925461 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.608+0000 7fb25ce00700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb25806b0f0 0x7fb25819b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.609+0000 7fb25ce00700 1 -- 172.21.15.6:0/2019925461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25806a750 msgr2=0x7fb25819b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.609+0000 7fb25ce00700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25806a750 0x7fb25819b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.609+0000 7fb25ce00700 1 -- 172.21.15.6:0/2019925461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb258108a30 msgr2=0x7fb25819fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.609+0000 7fb25ce00700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb258108a30 0x7fb25819fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.609+0000 7fb25ce00700 1 -- 172.21.15.6:0/2019925461 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2581a0360 con 0x7fb25806b0f0 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.609+0000 7fb25ce00700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb25806b0f0 0x7fb25819b890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb2440098e0 tx=0x7fb2440057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.609+0000 7fb256000700 1 -- 172.21.15.6:0/2019925461 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb244027070 con 0x7fb25806b0f0 2024-04-26T01:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.609+0000 7fb256000700 1 -- 172.21.15.6:0/2019925461 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb244022470 con 0x7fb25806b0f0 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.609+0000 7fb260200700 1 -- 172.21.15.6:0/2019925461 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2581a05f0 con 0x7fb25806b0f0 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.609+0000 7fb256000700 1 -- 172.21.15.6:0/2019925461 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb244017650 con 0x7fb25806b0f0 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.609+0000 7fb260200700 1 -- 172.21.15.6:0/2019925461 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2581a7ed0 con 0x7fb25806b0f0 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.610+0000 7fb260200700 1 -- 172.21.15.6:0/2019925461 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb2580631d0 con 0x7fb25806b0f0 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.613+0000 7fb256000700 1 -- 172.21.15.6:0/2019925461 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fb24402a070 con 0x7fb25806b0f0 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.614+0000 7fb256000700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb2400a3d50 0x7fb2400a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.614+0000 7fb256000700 1 -- 172.21.15.6:0/2019925461 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7fb2440d1750 con 0x7fb25806b0f0 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.614+0000 7fb25d800700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb2400a3d50 0x7fb2400a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.615+0000 7fb256000700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118] conn(0x7fb2400ad990 0x7fb2400afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.615+0000 7fb256000700 1 -- 172.21.15.6:0/2019925461 --> [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2400b0500 con 0x7fb2400ad990 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.615+0000 7fb256000700 1 -- 172.21.15.6:0/2019925461 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fb2440d1b00 con 0x7fb25806b0f0 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.615+0000 7fb25e200700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118] conn(0x7fb2400ad990 0x7fb2400afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.615+0000 7fb25d800700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb2400a3d50 0x7fb2400a6210 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009680 tx=0x7fb24c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.616+0000 7fb25e200700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118] conn(0x7fb2400ad990 0x7fb2400afe30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:34.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.617+0000 7fb256000700 1 -- 172.21.15.6:0/2019925461 <== osd.3 v2:172.21.15.6:6810/1987311118 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb2400b0500 con 0x7fb2400ad990 2024-04-26T01:43:34.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.635+0000 7fb260200700 1 -- 172.21.15.6:0/2019925461 --> [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb2580635d0 con 0x7fb2400ad990 2024-04-26T01:43:34.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.636+0000 7fb256000700 1 -- 172.21.15.6:0/2019925461 <== osd.3 v2:172.21.15.6:6810/1987311118 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb2580635d0 con 0x7fb2400ad990 2024-04-26T01:43:34.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.637+0000 7fb260200700 1 -- 172.21.15.6:0/2019925461 >> [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118] conn(0x7fb2400ad990 msgr2=0x7fb2400afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:34.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.637+0000 7fb260200700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118] conn(0x7fb2400ad990 0x7fb2400afe30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:34.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.637+0000 7fb260200700 1 -- 172.21.15.6:0/2019925461 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb2400a3d50 msgr2=0x7fb2400a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:34.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.637+0000 7fb260200700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb2400a3d50 0x7fb2400a6210 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009680 tx=0x7fb24c009260 comp rx=0 tx=0).stop 2024-04-26T01:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.637+0000 7fb260200700 1 -- 172.21.15.6:0/2019925461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb25806b0f0 msgr2=0x7fb25819b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.637+0000 7fb260200700 1 --2- 172.21.15.6:0/2019925461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb25806b0f0 0x7fb25819b890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb2440098e0 tx=0x7fb2440057c0 comp rx=0 tx=0).stop 2024-04-26T01:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.637+0000 7fb25e200700 1 -- 172.21.15.6:0/2019925461 reap_dead start 2024-04-26T01:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.637+0000 7fb260200700 1 -- 172.21.15.6:0/2019925461 shutdown_connections 2024-04-26T01:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.637+0000 7fb260200700 1 -- 172.21.15.6:0/2019925461 >> 172.21.15.6:0/2019925461 conn(0x7fb2580788f0 msgr2=0x7fb25810df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.637+0000 7fb260200700 1 -- 172.21.15.6:0/2019925461 shutdown_connections 2024-04-26T01:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:34.637+0000 7fb260200700 1 -- 172.21.15.6:0/2019925461 wait complete. 2024-04-26T01:43:35.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:34 smithi029 ceph-mon[32775]: pgmap v376: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:35.271 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:34 smithi006 ceph-mon[28419]: pgmap v376: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:35.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:34 smithi188 ceph-mon[32751]: pgmap v376: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:36.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.754+0000 7fc7f0e00700 1 -- 172.21.15.6:0/2283370367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7ec075590 msgr2=0x7fc7ec10ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.754+0000 7fc7f0e00700 1 --2- 172.21.15.6:0/2283370367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7ec075590 0x7fc7ec10ef00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400b0a0 tx=0x7fc7d400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.754+0000 7fc7f0e00700 1 -- 172.21.15.6:0/2283370367 shutdown_connections 2024-04-26T01:43:36.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.754+0000 7fc7f0e00700 1 --2- 172.21.15.6:0/2283370367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7ec10f440 0x7fc7ec111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.754+0000 7fc7f0e00700 1 --2- 172.21.15.6:0/2283370367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7ec074c70 0x7fc7ec075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.754+0000 7fc7f0e00700 1 --2- 172.21.15.6:0/2283370367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7ec075590 0x7fc7ec10ef00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.754+0000 7fc7f0e00700 1 -- 172.21.15.6:0/2283370367 >> 172.21.15.6:0/2283370367 conn(0x7fc7ec079ba0 msgr2=0x7fc7ec079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:36.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.754+0000 7fc7f0e00700 1 -- 172.21.15.6:0/2283370367 shutdown_connections 2024-04-26T01:43:36.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.755+0000 7fc7f0e00700 1 -- 172.21.15.6:0/2283370367 wait complete. 2024-04-26T01:43:36.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.756+0000 7fc7f0e00700 1 Processor -- start 2024-04-26T01:43:36.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.756+0000 7fc7f0e00700 1 -- start start 2024-04-26T01:43:36.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.756+0000 7fc7f0e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7ec074c70 0x7fc7ec19b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.756+0000 7fc7f0e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7ec075590 0x7fc7ec19b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.756+0000 7fc7f0e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7ec10f440 0x7fc7ec19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.756+0000 7fc7f0e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ec114e00 con 0x7fc7ec10f440 2024-04-26T01:43:36.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.756+0000 7fc7f0e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ec114c80 con 0x7fc7ec074c70 2024-04-26T01:43:36.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.756+0000 7fc7f0e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ec114f80 con 0x7fc7ec075590 2024-04-26T01:43:36.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.757+0000 7fc7ea800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7ec10f440 0x7fc7ec19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.757+0000 7fc7ea800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7ec10f440 0x7fc7ec19fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53588/0 (socket says 172.21.15.6:53588) 2024-04-26T01:43:36.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.757+0000 7fc7ea800700 1 -- 172.21.15.6:0/1199131273 learned_addr learned my addr 172.21.15.6:0/1199131273 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:36.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.757+0000 7fc7e9e00700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7ec074c70 0x7fc7ec19b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.757+0000 7fc7e1400700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7ec075590 0x7fc7ec19b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.757+0000 7fc7e9e00700 1 -- 172.21.15.6:0/1199131273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7ec075590 msgr2=0x7fc7ec19b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.757+0000 7fc7e9e00700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7ec075590 0x7fc7ec19b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.757+0000 7fc7e9e00700 1 -- 172.21.15.6:0/1199131273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7ec10f440 msgr2=0x7fc7ec19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.757+0000 7fc7e9e00700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7ec10f440 0x7fc7ec19fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.757+0000 7fc7e9e00700 1 -- 172.21.15.6:0/1199131273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ec1a0340 con 0x7fc7ec074c70 2024-04-26T01:43:36.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.757+0000 7fc7ea800700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7ec10f440 0x7fc7ec19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:36.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.757+0000 7fc7e1400700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7ec075590 0x7fc7ec19b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:43:36.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.758+0000 7fc7e9e00700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7ec074c70 0x7fc7ec19b330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00e9f0 tx=0x7fc7dc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:36.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.758+0000 7fc7e3400700 1 -- 172.21.15.6:0/1199131273 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc00cbd0 con 0x7fc7ec074c70 2024-04-26T01:43:36.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.758+0000 7fc7e3400700 1 -- 172.21.15.6:0/1199131273 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7dc00cd70 con 0x7fc7ec074c70 2024-04-26T01:43:36.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.758+0000 7fc7f0e00700 1 -- 172.21.15.6:0/1199131273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7ec1a0630 con 0x7fc7ec074c70 2024-04-26T01:43:36.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.758+0000 7fc7e3400700 1 -- 172.21.15.6:0/1199131273 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc018680 con 0x7fc7ec074c70 2024-04-26T01:43:36.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.758+0000 7fc7f0e00700 1 -- 172.21.15.6:0/1199131273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7ec1a7eb0 con 0x7fc7ec074c70 2024-04-26T01:43:36.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.759+0000 7fc7f0e00700 1 -- 172.21.15.6:0/1199131273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc7ec063190 con 0x7fc7ec074c70 2024-04-26T01:43:36.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.761+0000 7fc7e3400700 1 -- 172.21.15.6:0/1199131273 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fc7dc010430 con 0x7fc7ec074c70 2024-04-26T01:43:36.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.763+0000 7fc7e3400700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc7d00a3cd0 0x7fc7d00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.763+0000 7fc7e3400700 1 -- 172.21.15.6:0/1199131273 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7fc7dc0d0f50 con 0x7fc7ec074c70 2024-04-26T01:43:36.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.763+0000 7fc7e1400700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc7d00a3cd0 0x7fc7d00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.763+0000 7fc7e3400700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116] conn(0x7fc7d00ad910 0x7fc7d00afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.763+0000 7fc7e3400700 1 -- 172.21.15.6:0/1199131273 --> [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc7d00b0480 con 0x7fc7d00ad910 2024-04-26T01:43:36.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.763+0000 7fc7e3400700 1 -- 172.21.15.6:0/1199131273 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fc7dc0d1300 con 0x7fc7ec074c70 2024-04-26T01:43:36.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.764+0000 7fc7ea800700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116] conn(0x7fc7d00ad910 0x7fc7d00afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.764+0000 7fc7e1400700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc7d00a3cd0 0x7fc7d00a6190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009fd0 tx=0x7fc7d4009c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:36.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.764+0000 7fc7ea800700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116] conn(0x7fc7d00ad910 0x7fc7d00afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:36.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.765+0000 7fc7e3400700 1 -- 172.21.15.6:0/1199131273 <== osd.1 v2:172.21.15.29:6800/400769116 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc7d00b0480 con 0x7fc7d00ad910 2024-04-26T01:43:36.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.783+0000 7fc7f0e00700 1 -- 172.21.15.6:0/1199131273 --> [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc7ec063610 con 0x7fc7d00ad910 2024-04-26T01:43:36.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.784+0000 7fc7e3400700 1 -- 172.21.15.6:0/1199131273 <== osd.1 v2:172.21.15.29:6800/400769116 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc7ec063610 con 0x7fc7d00ad910 2024-04-26T01:43:36.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.784+0000 7fc7f0e00700 1 -- 172.21.15.6:0/1199131273 >> [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116] conn(0x7fc7d00ad910 msgr2=0x7fc7d00afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.785+0000 7fc7f0e00700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116] conn(0x7fc7d00ad910 0x7fc7d00afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.785+0000 7fc7f0e00700 1 -- 172.21.15.6:0/1199131273 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc7d00a3cd0 msgr2=0x7fc7d00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.785+0000 7fc7f0e00700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc7d00a3cd0 0x7fc7d00a6190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009fd0 tx=0x7fc7d4009c60 comp rx=0 tx=0).stop 2024-04-26T01:43:36.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.785+0000 7fc7f0e00700 1 -- 172.21.15.6:0/1199131273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7ec074c70 msgr2=0x7fc7ec19b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.785+0000 7fc7f0e00700 1 --2- 172.21.15.6:0/1199131273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7ec074c70 0x7fc7ec19b330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00e9f0 tx=0x7fc7dc00ed00 comp rx=0 tx=0).stop 2024-04-26T01:43:36.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.785+0000 7fc7ea800700 1 -- 172.21.15.6:0/1199131273 reap_dead start 2024-04-26T01:43:36.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.785+0000 7fc7f0e00700 1 -- 172.21.15.6:0/1199131273 shutdown_connections 2024-04-26T01:43:36.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.785+0000 7fc7f0e00700 1 -- 172.21.15.6:0/1199131273 >> 172.21.15.6:0/1199131273 conn(0x7fc7ec079ba0 msgr2=0x7fc7ec0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:36.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.785+0000 7fc7f0e00700 1 -- 172.21.15.6:0/1199131273 shutdown_connections 2024-04-26T01:43:36.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:36.785+0000 7fc7f0e00700 1 -- 172.21.15.6:0/1199131273 wait complete. 2024-04-26T01:43:37.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:36 smithi029 ceph-mon[32775]: pgmap v377: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:37.286 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:36 smithi006 ceph-mon[28419]: pgmap v377: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:37.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:36 smithi188 ceph-mon[32751]: pgmap v377: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:39.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:38 smithi006 ceph-mon[28419]: pgmap v378: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:39.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:38 smithi029 ceph-mon[32775]: pgmap v378: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:39.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:38 smithi188 ceph-mon[32751]: pgmap v378: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.873+0000 7f4afc000700 1 -- 172.21.15.6:0/336688301 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af4102a90 msgr2=0x7f4af4102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.873+0000 7f4afc000700 1 --2- 172.21.15.6:0/336688301 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af4102a90 0x7f4af4102f10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0009990 tx=0x7f4ae0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.873+0000 7f4afc000700 1 -- 172.21.15.6:0/336688301 shutdown_connections 2024-04-26T01:43:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.873+0000 7f4afc000700 1 --2- 172.21.15.6:0/336688301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4af4103450 0x7f4af410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.873+0000 7f4afc000700 1 --2- 172.21.15.6:0/336688301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4108a50 0x7f4af4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.873+0000 7f4afc000700 1 --2- 172.21.15.6:0/336688301 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af4102a90 0x7f4af4102f10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.873+0000 7f4afc000700 1 -- 172.21.15.6:0/336688301 >> 172.21.15.6:0/336688301 conn(0x7f4af4079ba0 msgr2=0x7f4af4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:40.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.873+0000 7f4afc000700 1 -- 172.21.15.6:0/336688301 shutdown_connections 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.874+0000 7f4afc000700 1 -- 172.21.15.6:0/336688301 wait complete. 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.875+0000 7f4afc000700 1 Processor -- start 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.875+0000 7f4afc000700 1 -- start start 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.875+0000 7f4afc000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4102a90 0x7f4af4199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.875+0000 7f4afc000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af4103450 0x7f4af41996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.875+0000 7f4afc000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4af4108a50 0x7f4af419da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.875+0000 7f4afc000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4af4112c50 con 0x7f4af4103450 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.875+0000 7f4afc000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4af4112ad0 con 0x7f4af4102a90 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.875+0000 7f4afc000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4af4112dd0 con 0x7f4af4108a50 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.875+0000 7f4af9600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4102a90 0x7f4af4199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.876+0000 7f4af9600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4102a90 0x7f4af4199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47744/0 (socket says 172.21.15.6:47744) 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.876+0000 7f4af9600700 1 -- 172.21.15.6:0/987841742 learned_addr learned my addr 172.21.15.6:0/987841742 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:40.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.876+0000 7f4af8c00700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af4103450 0x7f4af41996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.876+0000 7f4afa000700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4af4108a50 0x7f4af419da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.876+0000 7f4af9600700 1 -- 172.21.15.6:0/987841742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4af4108a50 msgr2=0x7f4af419da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.876+0000 7f4af9600700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4af4108a50 0x7f4af419da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.876+0000 7f4af9600700 1 -- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af4103450 msgr2=0x7f4af41996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.876+0000 7f4af9600700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af4103450 0x7f4af41996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.876+0000 7f4af9600700 1 -- 172.21.15.6:0/987841742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af419e190 con 0x7f4af4102a90 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.876+0000 7f4af9600700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4102a90 0x7f4af4199180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800c370 tx=0x7f4ae800c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.877+0000 7f4af2000700 1 -- 172.21.15.6:0/987841742 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae80092e0 con 0x7f4af4102a90 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.877+0000 7f4afc000700 1 -- 172.21.15.6:0/987841742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af419e420 con 0x7f4af4102a90 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.877+0000 7f4af2000700 1 -- 172.21.15.6:0/987841742 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ae800cbe0 con 0x7f4af4102a90 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.877+0000 7f4af2000700 1 -- 172.21.15.6:0/987841742 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae80046b0 con 0x7f4af4102a90 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.877+0000 7f4afc000700 1 -- 172.21.15.6:0/987841742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af41a5d00 con 0x7f4af4102a90 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.878+0000 7f4afc000700 1 -- 172.21.15.6:0/987841742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4af40631d0 con 0x7f4af4102a90 2024-04-26T01:43:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.880+0000 7f4af2000700 1 -- 172.21.15.6:0/987841742 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f4ae801a020 con 0x7f4af4102a90 2024-04-26T01:43:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.882+0000 7f4af2000700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4adc0a3cd0 0x7f4adc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.882+0000 7f4af2000700 1 -- 172.21.15.6:0/987841742 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7f4ae80d09b0 con 0x7f4af4102a90 2024-04-26T01:43:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.882+0000 7f4af8c00700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4adc0a3cd0 0x7f4adc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.882+0000 7f4af2000700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695] conn(0x7f4adc0ad910 0x7f4adc0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.882+0000 7f4af2000700 1 -- 172.21.15.6:0/987841742 --> [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ae8009c20 con 0x7f4adc0ad910 2024-04-26T01:43:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.882+0000 7f4af2000700 1 -- 172.21.15.6:0/987841742 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f4ae80d0d60 con 0x7f4af4102a90 2024-04-26T01:43:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.882+0000 7f4afa000700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695] conn(0x7f4adc0ad910 0x7f4adc0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.883+0000 7f4af8c00700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4adc0a3cd0 0x7f4adc0a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4af407a2f0 tx=0x7f4ae000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.883+0000 7f4afa000700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695] conn(0x7f4adc0ad910 0x7f4adc0afdb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.884+0000 7f4af2000700 1 -- 172.21.15.6:0/987841742 <== osd.6 v2:172.21.15.6:6818/3256016695 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4ae8009c20 con 0x7f4adc0ad910 2024-04-26T01:43:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.903+0000 7f4afc000700 1 -- 172.21.15.6:0/987841742 --> [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4af40635d0 con 0x7f4adc0ad910 2024-04-26T01:43:40.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.904+0000 7f4af2000700 1 -- 172.21.15.6:0/987841742 <== osd.6 v2:172.21.15.6:6818/3256016695 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4af40635d0 con 0x7f4adc0ad910 2024-04-26T01:43:40.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.904+0000 7f4afc000700 1 -- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695] conn(0x7f4adc0ad910 msgr2=0x7f4adc0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.904+0000 7f4afc000700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695] conn(0x7f4adc0ad910 0x7f4adc0afdb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.905+0000 7f4afc000700 1 -- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4adc0a3cd0 msgr2=0x7f4adc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.905+0000 7f4afc000700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f4adc0a3cd0 0x7f4adc0a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4af407a2f0 tx=0x7f4ae000b260 comp rx=0 tx=0).stop 2024-04-26T01:43:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.905+0000 7f4afc000700 1 -- 172.21.15.6:0/987841742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4102a90 msgr2=0x7f4af4199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.905+0000 7f4afc000700 1 --2- 172.21.15.6:0/987841742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4af4102a90 0x7f4af4199180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800c370 tx=0x7f4ae800c680 comp rx=0 tx=0).stop 2024-04-26T01:43:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.905+0000 7f4afa000700 1 -- 172.21.15.6:0/987841742 reap_dead start 2024-04-26T01:43:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.905+0000 7f4afc000700 1 -- 172.21.15.6:0/987841742 shutdown_connections 2024-04-26T01:43:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.905+0000 7f4afc000700 1 -- 172.21.15.6:0/987841742 >> 172.21.15.6:0/987841742 conn(0x7f4af4079ba0 msgr2=0x7f4af4101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.905+0000 7f4afc000700 1 -- 172.21.15.6:0/987841742 shutdown_connections 2024-04-26T01:43:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:39.905+0000 7f4afc000700 1 -- 172.21.15.6:0/987841742 wait complete. 2024-04-26T01:43:40.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.501+0000 7fa699a00700 1 -- 172.21.15.6:0/3926666628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6941092d0 msgr2=0x7fa6941096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.501+0000 7fa699a00700 1 --2- 172.21.15.6:0/3926666628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6941092d0 0x7fa6941096b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa6840099c0 tx=0x7fa684009cd0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.502+0000 7fa699a00700 1 -- 172.21.15.6:0/3926666628 shutdown_connections 2024-04-26T01:43:40.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.502+0000 7fa699a00700 1 --2- 172.21.15.6:0/3926666628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa69410e3b0 0x7fa694115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.502+0000 7fa699a00700 1 --2- 172.21.15.6:0/3926666628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6941092d0 0x7fa6941096b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.502+0000 7fa699a00700 1 --2- 172.21.15.6:0/3926666628 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa694109c80 0x7fa69410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.502+0000 7fa699a00700 1 -- 172.21.15.6:0/3926666628 >> 172.21.15.6:0/3926666628 conn(0x7fa694076210 msgr2=0x7fa694107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:40.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.502+0000 7fa699a00700 1 -- 172.21.15.6:0/3926666628 shutdown_connections 2024-04-26T01:43:40.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.502+0000 7fa699a00700 1 -- 172.21.15.6:0/3926666628 wait complete. 2024-04-26T01:43:40.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.503+0000 7fa699a00700 1 Processor -- start 2024-04-26T01:43:40.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.503+0000 7fa699a00700 1 -- start start 2024-04-26T01:43:40.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.503+0000 7fa699a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6941092d0 0x7fa69419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.503+0000 7fa699a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa694109c80 0x7fa69419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.503+0000 7fa699a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa69410e3b0 0x7fa6941a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.503+0000 7fa699a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa694119290 con 0x7fa6941092d0 2024-04-26T01:43:40.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.503+0000 7fa699a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa694119110 con 0x7fa69410e3b0 2024-04-26T01:43:40.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.503+0000 7fa699a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa694119410 con 0x7fa694109c80 2024-04-26T01:43:40.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.504+0000 7fa692a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6941092d0 0x7fa69419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.504+0000 7fa692000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa694109c80 0x7fa69419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.504+0000 7fa692a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6941092d0 0x7fa69419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41430/0 (socket says 172.21.15.6:41430) 2024-04-26T01:43:40.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.504+0000 7fa692a00700 1 -- 172.21.15.6:0/2247065602 learned_addr learned my addr 172.21.15.6:0/2247065602 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:40.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.504+0000 7fa693400700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa69410e3b0 0x7fa6941a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.504+0000 7fa692a00700 1 -- 172.21.15.6:0/2247065602 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa694109c80 msgr2=0x7fa69419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.504+0000 7fa692a00700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa694109c80 0x7fa69419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.504+0000 7fa692a00700 1 -- 172.21.15.6:0/2247065602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa69410e3b0 msgr2=0x7fa6941a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.504+0000 7fa692a00700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa69410e3b0 0x7fa6941a4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.504+0000 7fa692a00700 1 -- 172.21.15.6:0/2247065602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6941a4760 con 0x7fa6941092d0 2024-04-26T01:43:40.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.504+0000 7fa692000700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa694109c80 0x7fa69419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:40.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.505+0000 7fa692a00700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6941092d0 0x7fa69419f7a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa684006010 tx=0x7fa6840055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.505+0000 7fa68b400700 1 -- 172.21.15.6:0/2247065602 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa684026070 con 0x7fa6941092d0 2024-04-26T01:43:40.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.505+0000 7fa68b400700 1 -- 172.21.15.6:0/2247065602 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa68400f460 con 0x7fa6941092d0 2024-04-26T01:43:40.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.505+0000 7fa68b400700 1 -- 172.21.15.6:0/2247065602 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa684009e60 con 0x7fa6941092d0 2024-04-26T01:43:40.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.505+0000 7fa699a00700 1 -- 172.21.15.6:0/2247065602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6941a49f0 con 0x7fa6941092d0 2024-04-26T01:43:40.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.505+0000 7fa699a00700 1 -- 172.21.15.6:0/2247065602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6941ac2d0 con 0x7fa6941092d0 2024-04-26T01:43:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.506+0000 7fa699a00700 1 -- 172.21.15.6:0/2247065602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa6941a4e80 con 0x7fa6941092d0 2024-04-26T01:43:40.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.509+0000 7fa68b400700 1 -- 172.21.15.6:0/2247065602 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fa684029070 con 0x7fa6941092d0 2024-04-26T01:43:40.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.510+0000 7fa68b400700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa6780a3d50 0x7fa6780a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.510+0000 7fa68b400700 1 -- 172.21.15.6:0/2247065602 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7fa6840d11e0 con 0x7fa6941092d0 2024-04-26T01:43:40.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.510+0000 7fa692000700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa6780a3d50 0x7fa6780a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.510+0000 7fa68b400700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376] conn(0x7fa6780ad860 0x7fa6780afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.511+0000 7fa68b400700 1 -- 172.21.15.6:0/2247065602 --> [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa684004ea0 con 0x7fa6780ad860 2024-04-26T01:43:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.511+0000 7fa68b400700 1 -- 172.21.15.6:0/2247065602 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fa6840d1590 con 0x7fa6941092d0 2024-04-26T01:43:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.511+0000 7fa692000700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa6780a3d50 0x7fa6780a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa694108910 tx=0x7fa680009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.511+0000 7fa693400700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376] conn(0x7fa6780ad860 0x7fa6780afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.512+0000 7fa693400700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376] conn(0x7fa6780ad860 0x7fa6780afd00 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.513+0000 7fa68b400700 1 -- 172.21.15.6:0/2247065602 <== osd.8 v2:172.21.15.188:6816/3290364376 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa684004ea0 con 0x7fa6780ad860 2024-04-26T01:43:40.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.530+0000 7fa699a00700 1 -- 172.21.15.6:0/2247065602 --> [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa6940634f0 con 0x7fa6780ad860 2024-04-26T01:43:40.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.534+0000 7fa68b400700 1 -- 172.21.15.6:0/2247065602 <== osd.8 v2:172.21.15.188:6816/3290364376 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa6940634f0 con 0x7fa6780ad860 2024-04-26T01:43:40.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.534+0000 7fa699a00700 1 -- 172.21.15.6:0/2247065602 >> [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376] conn(0x7fa6780ad860 msgr2=0x7fa6780afd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.534+0000 7fa699a00700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376] conn(0x7fa6780ad860 0x7fa6780afd00 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.534+0000 7fa699a00700 1 -- 172.21.15.6:0/2247065602 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa6780a3d50 msgr2=0x7fa6780a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.534+0000 7fa699a00700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa6780a3d50 0x7fa6780a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa694108910 tx=0x7fa680009260 comp rx=0 tx=0).stop 2024-04-26T01:43:40.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.534+0000 7fa699a00700 1 -- 172.21.15.6:0/2247065602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6941092d0 msgr2=0x7fa69419f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.534+0000 7fa699a00700 1 --2- 172.21.15.6:0/2247065602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6941092d0 0x7fa69419f7a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa684006010 tx=0x7fa6840055f0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.534+0000 7fa693400700 1 -- 172.21.15.6:0/2247065602 reap_dead start 2024-04-26T01:43:40.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.534+0000 7fa699a00700 1 -- 172.21.15.6:0/2247065602 shutdown_connections 2024-04-26T01:43:40.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.534+0000 7fa699a00700 1 -- 172.21.15.6:0/2247065602 >> 172.21.15.6:0/2247065602 conn(0x7fa694076210 msgr2=0x7fa694077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:40.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.534+0000 7fa699a00700 1 -- 172.21.15.6:0/2247065602 shutdown_connections 2024-04-26T01:43:40.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:40.534+0000 7fa699a00700 1 -- 172.21.15.6:0/2247065602 wait complete. 2024-04-26T01:43:41.013 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:40 smithi006 ceph-mon[28419]: pgmap v379: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:41.090 INFO:teuthology.orchestra.run.smithi006.stdout:55834574941 2024-04-26T01:43:41.090 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T01:43:41.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:40 smithi188 ceph-mon[32751]: pgmap v379: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:41.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:40 smithi029 ceph-mon[32775]: pgmap v379: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:41.584 INFO:teuthology.orchestra.run.smithi006.stdout:266287972408 2024-04-26T01:43:41.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-26T01:43:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.530+0000 7f9bf9a00700 1 -- 172.21.15.6:0/2934432244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bf4076f70 msgr2=0x7f9bf4077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.530+0000 7f9bf9a00700 1 --2- 172.21.15.6:0/2934432244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bf4076f70 0x7f9bf4077350 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009990 tx=0x7f9be4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.531+0000 7f9bf9a00700 1 -- 172.21.15.6:0/2934432244 shutdown_connections 2024-04-26T01:43:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.531+0000 7f9bf9a00700 1 --2- 172.21.15.6:0/2934432244 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf4113920 0x7f9bf4115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.531+0000 7f9bf9a00700 1 --2- 172.21.15.6:0/2934432244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bf4076f70 0x7f9bf4077350 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.531+0000 7f9bf9a00700 1 --2- 172.21.15.6:0/2934432244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf4077890 0x7f9bf41133e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.531+0000 7f9bf9a00700 1 -- 172.21.15.6:0/2934432244 >> 172.21.15.6:0/2934432244 conn(0x7f9bf4100770 msgr2=0x7f9bf4102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.531+0000 7f9bf9a00700 1 -- 172.21.15.6:0/2934432244 shutdown_connections 2024-04-26T01:43:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.531+0000 7f9bf9a00700 1 -- 172.21.15.6:0/2934432244 wait complete. 2024-04-26T01:43:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.532+0000 7f9bf9a00700 1 Processor -- start 2024-04-26T01:43:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.532+0000 7f9bf9a00700 1 -- start start 2024-04-26T01:43:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.532+0000 7f9bf9a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf4076f70 0x7f9bf419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf9a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bf4077890 0x7f9bf419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf9a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf4113920 0x7f9bf41a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf9a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf41192b0 con 0x7f9bf4077890 2024-04-26T01:43:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf9a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf4119130 con 0x7f9bf4113920 2024-04-26T01:43:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf9a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf4119430 con 0x7f9bf4076f70 2024-04-26T01:43:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf2000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bf4077890 0x7f9bf419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf2a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf4076f70 0x7f9bf419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf3400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf4113920 0x7f9bf41a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf2a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf4076f70 0x7f9bf419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59330/0 (socket says 172.21.15.6:59330) 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf2a00700 1 -- 172.21.15.6:0/1831877700 learned_addr learned my addr 172.21.15.6:0/1831877700 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf2000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bf4077890 0x7f9bf419fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41458/0 (socket says 172.21.15.6:41458) 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf3400700 1 -- 172.21.15.6:0/1831877700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf4076f70 msgr2=0x7f9bf419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf3400700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf4076f70 0x7f9bf419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf3400700 1 -- 172.21.15.6:0/1831877700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bf4077890 msgr2=0x7f9bf419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf3400700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bf4077890 0x7f9bf419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf3400700 1 -- 172.21.15.6:0/1831877700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf41a4760 con 0x7f9bf4113920 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.533+0000 7f9bf2a00700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf4076f70 0x7f9bf419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.534+0000 7f9bf3400700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf4113920 0x7f9bf41a4020 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9bf407b260 tx=0x7f9bec00ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.534+0000 7f9beb400700 1 -- 172.21.15.6:0/1831877700 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bec0092e0 con 0x7f9bf4113920 2024-04-26T01:43:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.534+0000 7f9beb400700 1 -- 172.21.15.6:0/1831877700 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bec004bb0 con 0x7f9bf4113920 2024-04-26T01:43:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.534+0000 7f9bf9a00700 1 -- 172.21.15.6:0/1831877700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf41a49f0 con 0x7f9bf4113920 2024-04-26T01:43:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.534+0000 7f9beb400700 1 -- 172.21.15.6:0/1831877700 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bec004020 con 0x7f9bf4113920 2024-04-26T01:43:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.534+0000 7f9bf9a00700 1 -- 172.21.15.6:0/1831877700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf41ac2d0 con 0x7f9bf4113920 2024-04-26T01:43:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.535+0000 7f9bf9a00700 1 -- 172.21.15.6:0/1831877700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9bf40631d0 con 0x7f9bf4113920 2024-04-26T01:43:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.537+0000 7f9beb400700 1 -- 172.21.15.6:0/1831877700 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f9bec01b030 con 0x7f9bf4113920 2024-04-26T01:43:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.539+0000 7f9beb400700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9bd80a3d50 0x7f9bd80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.539+0000 7f9beb400700 1 -- 172.21.15.6:0/1831877700 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7f9bec0d0950 con 0x7f9bf4113920 2024-04-26T01:43:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.539+0000 7f9bf2a00700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9bd80a3d50 0x7f9bd80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.539+0000 7f9beb400700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912] conn(0x7f9bd80ad990 0x7f9bd80afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.539+0000 7f9beb400700 1 -- 172.21.15.6:0/1831877700 --> [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9bec003e70 con 0x7f9bd80ad990 2024-04-26T01:43:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.539+0000 7f9beb400700 1 -- 172.21.15.6:0/1831877700 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f9bec0d0d00 con 0x7f9bf4113920 2024-04-26T01:43:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.540+0000 7f9bf2000700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912] conn(0x7f9bd80ad990 0x7f9bd80afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:42.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.540+0000 7f9bf2a00700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9bd80a3d50 0x7f9bd80a6210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9be4006010 tx=0x7f9be4009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:42.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.540+0000 7f9bf2000700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912] conn(0x7f9bd80ad990 0x7f9bd80afe30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:42.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.541+0000 7f9beb400700 1 -- 172.21.15.6:0/1831877700 <== osd.9 v2:172.21.15.6:6826/1918893912 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9bec003e70 con 0x7f9bd80ad990 2024-04-26T01:43:42.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.562+0000 7f9bf9a00700 1 -- 172.21.15.6:0/1831877700 --> [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9bf4063650 con 0x7f9bd80ad990 2024-04-26T01:43:42.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.563+0000 7f9beb400700 1 -- 172.21.15.6:0/1831877700 <== osd.9 v2:172.21.15.6:6826/1918893912 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9bf4063650 con 0x7f9bd80ad990 2024-04-26T01:43:42.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.564+0000 7f9bf9a00700 1 -- 172.21.15.6:0/1831877700 >> [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912] conn(0x7f9bd80ad990 msgr2=0x7f9bd80afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:42.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.564+0000 7f9bf9a00700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912] conn(0x7f9bd80ad990 0x7f9bd80afe30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:42.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.564+0000 7f9bf9a00700 1 -- 172.21.15.6:0/1831877700 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9bd80a3d50 msgr2=0x7f9bd80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:42.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.564+0000 7f9bf9a00700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9bd80a3d50 0x7f9bd80a6210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9be4006010 tx=0x7f9be4009510 comp rx=0 tx=0).stop 2024-04-26T01:43:42.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.564+0000 7f9bf9a00700 1 -- 172.21.15.6:0/1831877700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf4113920 msgr2=0x7f9bf41a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:42.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.564+0000 7f9bf9a00700 1 --2- 172.21.15.6:0/1831877700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf4113920 0x7f9bf41a4020 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9bf407b260 tx=0x7f9bec00ef90 comp rx=0 tx=0).stop 2024-04-26T01:43:42.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.564+0000 7f9bf3400700 1 -- 172.21.15.6:0/1831877700 reap_dead start 2024-04-26T01:43:42.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.564+0000 7f9bf9a00700 1 -- 172.21.15.6:0/1831877700 shutdown_connections 2024-04-26T01:43:42.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.564+0000 7f9bf9a00700 1 -- 172.21.15.6:0/1831877700 >> 172.21.15.6:0/1831877700 conn(0x7f9bf4100770 msgr2=0x7f9bf4102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:42.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.564+0000 7f9bf9a00700 1 -- 172.21.15.6:0/1831877700 shutdown_connections 2024-04-26T01:43:42.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:42.564+0000 7f9bf9a00700 1 -- 172.21.15.6:0/1831877700 wait complete. 2024-04-26T01:43:43.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:42 smithi006 ceph-mon[28419]: pgmap v380: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:43.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.109+0000 7f9c38e00700 1 -- 172.21.15.6:0/2982510088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c3406afb0 msgr2=0x7f9c3406b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:43.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.109+0000 7f9c38e00700 1 --2- 172.21.15.6:0/2982510088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c3406afb0 0x7f9c3406b430 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009990 tx=0x7f9c20009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:43.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.109+0000 7f9c38e00700 1 -- 172.21.15.6:0/2982510088 shutdown_connections 2024-04-26T01:43:43.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.109+0000 7f9c38e00700 1 --2- 172.21.15.6:0/2982510088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3406b970 0x7f9c3410f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:43.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.109+0000 7f9c38e00700 1 --2- 172.21.15.6:0/2982510088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c34108a20 0x7f9c34108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:43.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.109+0000 7f9c38e00700 1 --2- 172.21.15.6:0/2982510088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c3406afb0 0x7f9c3406b430 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:43.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.109+0000 7f9c38e00700 1 -- 172.21.15.6:0/2982510088 >> 172.21.15.6:0/2982510088 conn(0x7f9c34078780 msgr2=0x7f9c34078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:43.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.109+0000 7f9c38e00700 1 -- 172.21.15.6:0/2982510088 shutdown_connections 2024-04-26T01:43:43.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.109+0000 7f9c38e00700 1 -- 172.21.15.6:0/2982510088 wait complete. 2024-04-26T01:43:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.110+0000 7f9c38e00700 1 Processor -- start 2024-04-26T01:43:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.110+0000 7f9c38e00700 1 -- start start 2024-04-26T01:43:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.110+0000 7f9c38e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3406afb0 0x7f9c34198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.110+0000 7f9c38e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c3406b970 0x7f9c341994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.110+0000 7f9c38e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c34108a20 0x7f9c3419d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.110+0000 7f9c38e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9c34112ba0 con 0x7f9c34108a20 2024-04-26T01:43:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.110+0000 7f9c38e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9c34112a20 con 0x7f9c3406afb0 2024-04-26T01:43:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.110+0000 7f9c38e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9c34112d20 con 0x7f9c3406b970 2024-04-26T01:43:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.111+0000 7f9c32a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c3406b970 0x7f9c341994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.111+0000 7f9c33e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c34108a20 0x7f9c3419d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.111+0000 7f9c33400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3406afb0 0x7f9c34198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.111+0000 7f9c32a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c3406b970 0x7f9c341994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59350/0 (socket says 172.21.15.6:59350) 2024-04-26T01:43:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.111+0000 7f9c33400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3406afb0 0x7f9c34198fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47798/0 (socket says 172.21.15.6:47798) 2024-04-26T01:43:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.111+0000 7f9c32a00700 1 -- 172.21.15.6:0/3134348962 learned_addr learned my addr 172.21.15.6:0/3134348962 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.111+0000 7f9c32a00700 1 -- 172.21.15.6:0/3134348962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3406afb0 msgr2=0x7f9c34198fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.111+0000 7f9c32a00700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3406afb0 0x7f9c34198fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.111+0000 7f9c32a00700 1 -- 172.21.15.6:0/3134348962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c34108a20 msgr2=0x7f9c3419d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.111+0000 7f9c32a00700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c34108a20 0x7f9c3419d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.111+0000 7f9c32a00700 1 -- 172.21.15.6:0/3134348962 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3419dfc0 con 0x7f9c3406b970 2024-04-26T01:43:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.111+0000 7f9c33400700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3406afb0 0x7f9c34198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.112+0000 7f9c32a00700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c3406b970 0x7f9c341994f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9c200057f0 tx=0x7f9c2000bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.112+0000 7f9c2be00700 1 -- 172.21.15.6:0/3134348962 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20026070 con 0x7f9c3406b970 2024-04-26T01:43:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.112+0000 7f9c38e00700 1 -- 172.21.15.6:0/3134348962 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3419e250 con 0x7f9c3406b970 2024-04-26T01:43:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.112+0000 7f9c2be00700 1 -- 172.21.15.6:0/3134348962 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c20021410 con 0x7f9c3406b970 2024-04-26T01:43:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.112+0000 7f9c2be00700 1 -- 172.21.15.6:0/3134348962 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20017650 con 0x7f9c3406b970 2024-04-26T01:43:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.112+0000 7f9c38e00700 1 -- 172.21.15.6:0/3134348962 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c341a5b30 con 0x7f9c3406b970 2024-04-26T01:43:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.113+0000 7f9c38e00700 1 -- 172.21.15.6:0/3134348962 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9c340631d0 con 0x7f9c3406b970 2024-04-26T01:43:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.115+0000 7f9c2be00700 1 -- 172.21.15.6:0/3134348962 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f9c20029070 con 0x7f9c3406b970 2024-04-26T01:43:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.117+0000 7f9c2be00700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9c180a3cd0 0x7f9c180a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.117+0000 7f9c2be00700 1 -- 172.21.15.6:0/3134348962 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7f9c200d1210 con 0x7f9c3406b970 2024-04-26T01:43:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.117+0000 7f9c33400700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9c180a3cd0 0x7f9c180a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.117+0000 7f9c2be00700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007] conn(0x7f9c180ad910 0x7f9c180afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.117+0000 7f9c2be00700 1 -- 172.21.15.6:0/3134348962 --> [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c180b0480 con 0x7f9c180ad910 2024-04-26T01:43:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.117+0000 7f9c2be00700 1 -- 172.21.15.6:0/3134348962 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f9c200d8050 con 0x7f9c3406b970 2024-04-26T01:43:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.117+0000 7f9c33e00700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007] conn(0x7f9c180ad910 0x7f9c180afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.118+0000 7f9c33400700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9c180a3cd0 0x7f9c180a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009fd0 tx=0x7f9c24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:43.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.118+0000 7f9c33e00700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007] conn(0x7f9c180ad910 0x7f9c180afdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:43.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.119+0000 7f9c2be00700 1 -- 172.21.15.6:0/3134348962 <== osd.5 v2:172.21.15.188:6808/2092077007 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9c180b0480 con 0x7f9c180ad910 2024-04-26T01:43:43.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.139+0000 7f9c38e00700 1 -- 172.21.15.6:0/3134348962 --> [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9c34063600 con 0x7f9c180ad910 2024-04-26T01:43:43.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.140+0000 7f9c2be00700 1 -- 172.21.15.6:0/3134348962 <== osd.5 v2:172.21.15.188:6808/2092077007 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9c34063600 con 0x7f9c180ad910 2024-04-26T01:43:43.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.140+0000 7f9c38e00700 1 -- 172.21.15.6:0/3134348962 >> [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007] conn(0x7f9c180ad910 msgr2=0x7f9c180afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:43.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.140+0000 7f9c38e00700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007] conn(0x7f9c180ad910 0x7f9c180afdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:43.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.141+0000 7f9c38e00700 1 -- 172.21.15.6:0/3134348962 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9c180a3cd0 msgr2=0x7f9c180a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:43.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.141+0000 7f9c38e00700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9c180a3cd0 0x7f9c180a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009fd0 tx=0x7f9c24009260 comp rx=0 tx=0).stop 2024-04-26T01:43:43.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.141+0000 7f9c38e00700 1 -- 172.21.15.6:0/3134348962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c3406b970 msgr2=0x7f9c341994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:43.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.141+0000 7f9c38e00700 1 --2- 172.21.15.6:0/3134348962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c3406b970 0x7f9c341994f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9c200057f0 tx=0x7f9c2000bfd0 comp rx=0 tx=0).stop 2024-04-26T01:43:43.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.141+0000 7f9c33e00700 1 -- 172.21.15.6:0/3134348962 reap_dead start 2024-04-26T01:43:43.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.141+0000 7f9c38e00700 1 -- 172.21.15.6:0/3134348962 shutdown_connections 2024-04-26T01:43:43.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.141+0000 7f9c38e00700 1 -- 172.21.15.6:0/3134348962 >> 172.21.15.6:0/3134348962 conn(0x7f9c34078780 msgr2=0x7f9c340ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:43.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.141+0000 7f9c38e00700 1 -- 172.21.15.6:0/3134348962 shutdown_connections 2024-04-26T01:43:43.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.141+0000 7f9c38e00700 1 -- 172.21.15.6:0/3134348962 wait complete. 2024-04-26T01:43:43.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:42 smithi188 ceph-mon[32751]: pgmap v380: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:43.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:42 smithi029 ceph-mon[32775]: pgmap v380: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:44.063 INFO:teuthology.orchestra.run.smithi006.stdout:171798691911 2024-04-26T01:43:44.063 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T01:43:44.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.880+0000 7f812b400700 1 -- 172.21.15.6:0/260624411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8124105870 msgr2=0x7f81241098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:44.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.880+0000 7f812b400700 1 --2- 172.21.15.6:0/260624411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8124105870 0x7f81241098c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8110009990 tx=0x7f8110009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.880+0000 7f812b400700 1 -- 172.21.15.6:0/260624411 shutdown_connections 2024-04-26T01:43:44.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.880+0000 7f812b400700 1 --2- 172.21.15.6:0/260624411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8124109fa0 0x7f8124111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.880+0000 7f812b400700 1 --2- 172.21.15.6:0/260624411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8124104ec0 0x7f81241052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.880+0000 7f812b400700 1 --2- 172.21.15.6:0/260624411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8124105870 0x7f81241098c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.880+0000 7f812b400700 1 -- 172.21.15.6:0/260624411 >> 172.21.15.6:0/260624411 conn(0x7f8124100750 msgr2=0x7f8124102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.880+0000 7f812b400700 1 -- 172.21.15.6:0/260624411 shutdown_connections 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.880+0000 7f812b400700 1 -- 172.21.15.6:0/260624411 wait complete. 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.881+0000 7f812b400700 1 Processor -- start 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.881+0000 7f812b400700 1 -- start start 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f812b400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8124104ec0 0x7f812419b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f812b400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8124105870 0x7f812419b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f812b400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8124109fa0 0x7f812419fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f812b400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8124114e30 con 0x7f8124109fa0 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f812b400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8124114cb0 con 0x7f8124104ec0 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f812b400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8124114fb0 con 0x7f8124105870 2024-04-26T01:43:44.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f8123e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8124105870 0x7f812419b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f8129400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8124109fa0 0x7f812419fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f8128a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8124104ec0 0x7f812419b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f8123e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8124105870 0x7f812419b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59376/0 (socket says 172.21.15.6:59376) 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f8123e00700 1 -- 172.21.15.6:0/4118608500 learned_addr learned my addr 172.21.15.6:0/4118608500 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f8129400700 1 -- 172.21.15.6:0/4118608500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8124105870 msgr2=0x7f812419b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f8129400700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8124105870 0x7f812419b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f8129400700 1 -- 172.21.15.6:0/4118608500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8124104ec0 msgr2=0x7f812419b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f8129400700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8124104ec0 0x7f812419b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f8129400700 1 -- 172.21.15.6:0/4118608500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81241a03d0 con 0x7f8124109fa0 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f8128a00700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8124104ec0 0x7f812419b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.882+0000 7f8123e00700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8124105870 0x7f812419b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.883+0000 7f8129400700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8124109fa0 0x7f812419fc90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f811c007f10 tx=0x7f811c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.883+0000 7f8121600700 1 -- 172.21.15.6:0/4118608500 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c004db0 con 0x7f8124109fa0 2024-04-26T01:43:44.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.883+0000 7f8121600700 1 -- 172.21.15.6:0/4118608500 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f811c003a00 con 0x7f8124109fa0 2024-04-26T01:43:44.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.883+0000 7f812b400700 1 -- 172.21.15.6:0/4118608500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81241a06c0 con 0x7f8124109fa0 2024-04-26T01:43:44.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.883+0000 7f8121600700 1 -- 172.21.15.6:0/4118608500 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c004580 con 0x7f8124109fa0 2024-04-26T01:43:44.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.883+0000 7f812b400700 1 -- 172.21.15.6:0/4118608500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81241a7f40 con 0x7f8124109fa0 2024-04-26T01:43:44.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.884+0000 7f812b400700 1 -- 172.21.15.6:0/4118608500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f81240630d0 con 0x7f8124109fa0 2024-04-26T01:43:44.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.887+0000 7f8121600700 1 -- 172.21.15.6:0/4118608500 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f811c01c020 con 0x7f8124109fa0 2024-04-26T01:43:44.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.889+0000 7f8121600700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f810c0a3cd0 0x7f810c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:44.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.889+0000 7f8121600700 1 -- 172.21.15.6:0/4118608500 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7f811c0d0a00 con 0x7f8124109fa0 2024-04-26T01:43:44.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.889+0000 7f8128a00700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f810c0a3cd0 0x7f810c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.889+0000 7f8121600700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609] conn(0x7f810c0ad910 0x7f810c0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.889+0000 7f8121600700 1 -- 172.21.15.6:0/4118608500 --> [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f811c0020b0 con 0x7f810c0ad910 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.889+0000 7f8121600700 1 -- 172.21.15.6:0/4118608500 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f811c0d0db0 con 0x7f8124109fa0 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.890+0000 7f8123e00700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609] conn(0x7f810c0ad910 0x7f810c0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.890+0000 7f8128a00700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f810c0a3cd0 0x7f810c0a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8118011490 tx=0x7f8118009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.890+0000 7f8123e00700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609] conn(0x7f810c0ad910 0x7f810c0afdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.891+0000 7f8121600700 1 -- 172.21.15.6:0/4118608500 <== osd.4 v2:172.21.15.29:6808/2125876609 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f811c0020b0 con 0x7f810c0ad910 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.910+0000 7f812b400700 1 -- 172.21.15.6:0/4118608500 --> [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8124063550 con 0x7f810c0ad910 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.910+0000 7f8121600700 1 -- 172.21.15.6:0/4118608500 <== osd.4 v2:172.21.15.29:6808/2125876609 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8124063550 con 0x7f810c0ad910 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.911+0000 7f812b400700 1 -- 172.21.15.6:0/4118608500 >> [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609] conn(0x7f810c0ad910 msgr2=0x7f810c0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.911+0000 7f812b400700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609] conn(0x7f810c0ad910 0x7f810c0afdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.911+0000 7f812b400700 1 -- 172.21.15.6:0/4118608500 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f810c0a3cd0 msgr2=0x7f810c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.911+0000 7f812b400700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f810c0a3cd0 0x7f810c0a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8118011490 tx=0x7f8118009260 comp rx=0 tx=0).stop 2024-04-26T01:43:44.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.911+0000 7f812b400700 1 -- 172.21.15.6:0/4118608500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8124109fa0 msgr2=0x7f812419fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:44.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.911+0000 7f812b400700 1 --2- 172.21.15.6:0/4118608500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8124109fa0 0x7f812419fc90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f811c007f10 tx=0x7f811c00d3b0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.911+0000 7f8129400700 1 -- 172.21.15.6:0/4118608500 reap_dead start 2024-04-26T01:43:44.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.911+0000 7f812b400700 1 -- 172.21.15.6:0/4118608500 shutdown_connections 2024-04-26T01:43:44.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.911+0000 7f812b400700 1 -- 172.21.15.6:0/4118608500 >> 172.21.15.6:0/4118608500 conn(0x7f8124100750 msgr2=0x7f8124102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:44.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.911+0000 7f812b400700 1 -- 172.21.15.6:0/4118608500 shutdown_connections 2024-04-26T01:43:44.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:43.911+0000 7f812b400700 1 -- 172.21.15.6:0/4118608500 wait complete. 2024-04-26T01:43:45.027 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:44 smithi006 ceph-mon[28419]: pgmap v381: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:45.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:44 smithi188 ceph-mon[32751]: pgmap v381: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:45.430 INFO:teuthology.orchestra.run.smithi006.stdout:111669149775 2024-04-26T01:43:45.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T01:43:45.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:44 smithi029 ceph-mon[32775]: pgmap v381: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:46.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.161+0000 7f58bb000700 1 -- 172.21.15.6:0/329600425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b4074c70 msgr2=0x7f58b4075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:46.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.161+0000 7f58bb000700 1 --2- 172.21.15.6:0/329600425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b4074c70 0x7f58b4075050 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f58a40098f0 tx=0x7f58a4009c00 comp rx=0 tx=0).stop 2024-04-26T01:43:46.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.161+0000 7f58bb000700 1 -- 172.21.15.6:0/329600425 shutdown_connections 2024-04-26T01:43:46.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.161+0000 7f58bb000700 1 --2- 172.21.15.6:0/329600425 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58b4113c10 0x7f58b4116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.161+0000 7f58bb000700 1 --2- 172.21.15.6:0/329600425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b4074c70 0x7f58b4075050 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.161+0000 7f58bb000700 1 --2- 172.21.15.6:0/329600425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58b4075590 0x7f58b41136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.161+0000 7f58bb000700 1 -- 172.21.15.6:0/329600425 >> 172.21.15.6:0/329600425 conn(0x7f58b4079ba0 msgr2=0x7f58b4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:46.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.162+0000 7f58bb000700 1 -- 172.21.15.6:0/329600425 shutdown_connections 2024-04-26T01:43:46.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.162+0000 7f58bb000700 1 -- 172.21.15.6:0/329600425 wait complete. 2024-04-26T01:43:46.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.163+0000 7f58bb000700 1 Processor -- start 2024-04-26T01:43:46.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.163+0000 7f58bb000700 1 -- start start 2024-04-26T01:43:46.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.164+0000 7f58bb000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58b4074c70 0x7f58b41a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:46.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.164+0000 7f58bb000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58b4075590 0x7f58b41a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:46.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.164+0000 7f58bb000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b4113c10 0x7f58b419f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:46.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.164+0000 7f58bb000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f58b4119240 con 0x7f58b4074c70 2024-04-26T01:43:46.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.164+0000 7f58bb000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f58b41190c0 con 0x7f58b4075590 2024-04-26T01:43:46.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.164+0000 7f58bb000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f58b41193c0 con 0x7f58b4113c10 2024-04-26T01:43:46.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.164+0000 7f58b3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58b4074c70 0x7f58b41a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:46.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.164+0000 7f58b3400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58b4075590 0x7f58b41a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:46.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.164+0000 7f58b3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58b4074c70 0x7f58b41a5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41508/0 (socket says 172.21.15.6:41508) 2024-04-26T01:43:46.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.164+0000 7f58b3e00700 1 -- 172.21.15.6:0/1084212308 learned_addr learned my addr 172.21.15.6:0/1084212308 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:46.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.164+0000 7f58b9000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b4113c10 0x7f58b419f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:46.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.165+0000 7f58b3400700 1 -- 172.21.15.6:0/1084212308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b4113c10 msgr2=0x7f58b419f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:46.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.165+0000 7f58b3400700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58b4113c10 0x7f58b419f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.165+0000 7f58b3400700 1 -- 172.21.15.6:0/1084212308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58b4074c70 msgr2=0x7f58b41a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:46.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.165+0000 7f58b3400700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58b4074c70 0x7f58b41a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.165+0000 7f58b3400700 1 -- 172.21.15.6:0/1084212308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b419fd80 con 0x7f58b4075590 2024-04-26T01:43:46.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.165+0000 7f58b3e00700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58b4074c70 0x7f58b41a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:46.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.165+0000 7f58b3400700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58b4075590 0x7f58b41a59a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f58a000e9f0 tx=0x7f58a000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:46.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.165+0000 7f58b0c00700 1 -- 172.21.15.6:0/1084212308 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a000c9c0 con 0x7f58b4075590 2024-04-26T01:43:46.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.165+0000 7f58b0c00700 1 -- 172.21.15.6:0/1084212308 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58a000cb60 con 0x7f58b4075590 2024-04-26T01:43:46.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.165+0000 7f58bb000700 1 -- 172.21.15.6:0/1084212308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58b41a0070 con 0x7f58b4075590 2024-04-26T01:43:46.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.165+0000 7f58b0c00700 1 -- 172.21.15.6:0/1084212308 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a0018680 con 0x7f58b4075590 2024-04-26T01:43:46.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.165+0000 7f58bb000700 1 -- 172.21.15.6:0/1084212308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58b41ac330 con 0x7f58b4075590 2024-04-26T01:43:46.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.166+0000 7f58bb000700 1 -- 172.21.15.6:0/1084212308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f58b4110110 con 0x7f58b4075590 2024-04-26T01:43:46.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.168+0000 7f58b0c00700 1 -- 172.21.15.6:0/1084212308 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f58a0010430 con 0x7f58b4075590 2024-04-26T01:43:46.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.170+0000 7f58b0c00700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f58980a3d50 0x7f58980a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:46.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.170+0000 7f58b0c00700 1 -- 172.21.15.6:0/1084212308 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7f58a00d07f0 con 0x7f58b4075590 2024-04-26T01:43:46.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.170+0000 7f58b3e00700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f58980a3d50 0x7f58980a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:46.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.170+0000 7f58b0c00700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880] conn(0x7f58980ad990 0x7f58980afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:46.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.170+0000 7f58b0c00700 1 -- 172.21.15.6:0/1084212308 --> [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f58a001ce70 con 0x7f58980ad990 2024-04-26T01:43:46.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.170+0000 7f58b0c00700 1 -- 172.21.15.6:0/1084212308 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f58a00d0ba0 con 0x7f58b4075590 2024-04-26T01:43:46.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.171+0000 7f58b9000700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880] conn(0x7f58980ad990 0x7f58980afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:46.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.171+0000 7f58b3e00700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f58980a3d50 0x7f58980a6210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009fd0 tx=0x7f58a4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:46.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.171+0000 7f58b9000700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880] conn(0x7f58980ad990 0x7f58980afe30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:46.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.172+0000 7f58b0c00700 1 -- 172.21.15.6:0/1084212308 <== osd.11 v2:172.21.15.188:6824/3322197880 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f58a001ce70 con 0x7f58980ad990 2024-04-26T01:43:46.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.191+0000 7f58bb000700 1 -- 172.21.15.6:0/1084212308 --> [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f58b4076360 con 0x7f58980ad990 2024-04-26T01:43:46.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.191+0000 7f58b0c00700 1 -- 172.21.15.6:0/1084212308 <== osd.11 v2:172.21.15.188:6824/3322197880 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f58b4076360 con 0x7f58980ad990 2024-04-26T01:43:46.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.192+0000 7f58bb000700 1 -- 172.21.15.6:0/1084212308 >> [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880] conn(0x7f58980ad990 msgr2=0x7f58980afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:46.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.192+0000 7f58bb000700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880] conn(0x7f58980ad990 0x7f58980afe30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.192+0000 7f58bb000700 1 -- 172.21.15.6:0/1084212308 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f58980a3d50 msgr2=0x7f58980a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:46.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.192+0000 7f58bb000700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f58980a3d50 0x7f58980a6210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009fd0 tx=0x7f58a4019040 comp rx=0 tx=0).stop 2024-04-26T01:43:46.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.192+0000 7f58bb000700 1 -- 172.21.15.6:0/1084212308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58b4075590 msgr2=0x7f58b41a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:46.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.192+0000 7f58bb000700 1 --2- 172.21.15.6:0/1084212308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58b4075590 0x7f58b41a59a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f58a000e9f0 tx=0x7f58a000ed00 comp rx=0 tx=0).stop 2024-04-26T01:43:46.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.193+0000 7f58b9000700 1 -- 172.21.15.6:0/1084212308 reap_dead start 2024-04-26T01:43:46.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.193+0000 7f58bb000700 1 -- 172.21.15.6:0/1084212308 shutdown_connections 2024-04-26T01:43:46.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.193+0000 7f58bb000700 1 -- 172.21.15.6:0/1084212308 >> 172.21.15.6:0/1084212308 conn(0x7f58b4079ba0 msgr2=0x7f58b410f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:46.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.193+0000 7f58bb000700 1 -- 172.21.15.6:0/1084212308 shutdown_connections 2024-04-26T01:43:46.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.193+0000 7f58bb000700 1 -- 172.21.15.6:0/1084212308 wait complete. 2024-04-26T01:43:46.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.240+0000 7f3bd4a00700 1 -- 172.21.15.6:0/136971460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3bd0104ee0 msgr2=0x7f3bd01052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.240+0000 7f3bd4a00700 1 --2- 172.21.15.6:0/136971460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3bd0104ee0 0x7f3bd01052c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009920 tx=0x7f3bc0009c30 comp rx=0 tx=0).stop 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.241+0000 7f3bd4a00700 1 -- 172.21.15.6:0/136971460 shutdown_connections 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.241+0000 7f3bd4a00700 1 --2- 172.21.15.6:0/136971460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd0109fc0 0x7f3bd0111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.241+0000 7f3bd4a00700 1 --2- 172.21.15.6:0/136971460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3bd0104ee0 0x7f3bd01052c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.241+0000 7f3bd4a00700 1 --2- 172.21.15.6:0/136971460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd0105890 0x7f3bd01098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.241+0000 7f3bd4a00700 1 -- 172.21.15.6:0/136971460 >> 172.21.15.6:0/136971460 conn(0x7f3bd0100770 msgr2=0x7f3bd0102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.241+0000 7f3bd4a00700 1 -- 172.21.15.6:0/136971460 shutdown_connections 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.241+0000 7f3bd4a00700 1 -- 172.21.15.6:0/136971460 wait complete. 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.242+0000 7f3bd4a00700 1 Processor -- start 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bd4a00700 1 -- start start 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bd4a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd0104ee0 0x7f3bd019b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bd4a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3bd0105890 0x7f3bd019b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:46.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bd4a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd0109fc0 0x7f3bd019fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bd4a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd0114eb0 con 0x7f3bd0105890 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bd4a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd0114d30 con 0x7f3bd0109fc0 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bd4a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd0115030 con 0x7f3bd0104ee0 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bce800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd0109fc0 0x7f3bd019fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bcde00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd0104ee0 0x7f3bd019b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bcd400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3bd0105890 0x7f3bd019b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bcde00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd0104ee0 0x7f3bd019b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59410/0 (socket says 172.21.15.6:59410) 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bce800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd0109fc0 0x7f3bd019fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47834/0 (socket says 172.21.15.6:47834) 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.243+0000 7f3bcde00700 1 -- 172.21.15.6:0/3026274305 learned_addr learned my addr 172.21.15.6:0/3026274305 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.244+0000 7f3bce800700 1 -- 172.21.15.6:0/3026274305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd0104ee0 msgr2=0x7f3bd019b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.244+0000 7f3bce800700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd0104ee0 0x7f3bd019b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.244+0000 7f3bce800700 1 -- 172.21.15.6:0/3026274305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3bd0105890 msgr2=0x7f3bd019b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.244+0000 7f3bce800700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3bd0105890 0x7f3bd019b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.244+0000 7f3bce800700 1 -- 172.21.15.6:0/3026274305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd01a0470 con 0x7f3bd0109fc0 2024-04-26T01:43:46.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.244+0000 7f3bcde00700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bd0104ee0 0x7f3bd019b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.244+0000 7f3bcd400700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3bd0105890 0x7f3bd019b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.244+0000 7f3bce800700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd0109fc0 0x7f3bd019fd30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800efc0 tx=0x7f3bc800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.245+0000 7f3bc6a00700 1 -- 172.21.15.6:0/3026274305 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc80092e0 con 0x7f3bd0109fc0 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.245+0000 7f3bc6a00700 1 -- 172.21.15.6:0/3026274305 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bc8004bb0 con 0x7f3bd0109fc0 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.245+0000 7f3bc6a00700 1 -- 172.21.15.6:0/3026274305 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8004020 con 0x7f3bd0109fc0 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.245+0000 7f3bd4a00700 1 -- 172.21.15.6:0/3026274305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd01a0700 con 0x7f3bd0109fc0 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.245+0000 7f3bd4a00700 1 -- 172.21.15.6:0/3026274305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd01a7fe0 con 0x7f3bd0109fc0 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.246+0000 7f3bd4a00700 1 -- 172.21.15.6:0/3026274305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3bd0063110 con 0x7f3bd0109fc0 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.248+0000 7f3bc6a00700 1 -- 172.21.15.6:0/3026274305 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f3bc801b030 con 0x7f3bd0109fc0 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.250+0000 7f3bc6a00700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3bb40a3d50 0x7f3bb40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.250+0000 7f3bc6a00700 1 -- 172.21.15.6:0/3026274305 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7f3bc80d0960 con 0x7f3bd0109fc0 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.250+0000 7f3bcde00700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3bb40a3d50 0x7f3bb40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.250+0000 7f3bc6a00700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037] conn(0x7f3bb40ad990 0x7f3bb40afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:46.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.250+0000 7f3bc6a00700 1 -- 172.21.15.6:0/3026274305 --> [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3bb40b0500 con 0x7f3bb40ad990 2024-04-26T01:43:46.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.250+0000 7f3bc6a00700 1 -- 172.21.15.6:0/3026274305 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f3bc80d0d10 con 0x7f3bd0109fc0 2024-04-26T01:43:46.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.251+0000 7f3bcd400700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037] conn(0x7f3bb40ad990 0x7f3bb40afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:46.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.251+0000 7f3bcde00700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3bb40a3d50 0x7f3bb40a6210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009810 tx=0x7f3bc0019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:46.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.251+0000 7f3bcd400700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037] conn(0x7f3bb40ad990 0x7f3bb40afe30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:46.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.252+0000 7f3bc6a00700 1 -- 172.21.15.6:0/3026274305 <== osd.7 v2:172.21.15.29:6816/3478594037 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bb40b0500 con 0x7f3bb40ad990 2024-04-26T01:43:46.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.272+0000 7f3bd4a00700 1 -- 172.21.15.6:0/3026274305 --> [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3bd0063590 con 0x7f3bb40ad990 2024-04-26T01:43:46.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.273+0000 7f3bc6a00700 1 -- 172.21.15.6:0/3026274305 <== osd.7 v2:172.21.15.29:6816/3478594037 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3bd0063590 con 0x7f3bb40ad990 2024-04-26T01:43:46.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.273+0000 7f3bd4a00700 1 -- 172.21.15.6:0/3026274305 >> [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037] conn(0x7f3bb40ad990 msgr2=0x7f3bb40afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:46.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.273+0000 7f3bd4a00700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037] conn(0x7f3bb40ad990 0x7f3bb40afe30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.273+0000 7f3bd4a00700 1 -- 172.21.15.6:0/3026274305 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3bb40a3d50 msgr2=0x7f3bb40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:46.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.273+0000 7f3bd4a00700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3bb40a3d50 0x7f3bb40a6210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009810 tx=0x7f3bc0019040 comp rx=0 tx=0).stop 2024-04-26T01:43:46.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.273+0000 7f3bd4a00700 1 -- 172.21.15.6:0/3026274305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd0109fc0 msgr2=0x7f3bd019fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:46.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.273+0000 7f3bd4a00700 1 --2- 172.21.15.6:0/3026274305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd0109fc0 0x7f3bd019fd30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800efc0 tx=0x7f3bc800c3a0 comp rx=0 tx=0).stop 2024-04-26T01:43:46.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.273+0000 7f3bce800700 1 -- 172.21.15.6:0/3026274305 reap_dead start 2024-04-26T01:43:46.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.273+0000 7f3bd4a00700 1 -- 172.21.15.6:0/3026274305 shutdown_connections 2024-04-26T01:43:46.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.274+0000 7f3bd4a00700 1 -- 172.21.15.6:0/3026274305 >> 172.21.15.6:0/3026274305 conn(0x7f3bd0100770 msgr2=0x7f3bd0101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:46.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.274+0000 7f3bd4a00700 1 -- 172.21.15.6:0/3026274305 shutdown_connections 2024-04-26T01:43:46.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:46.274+0000 7f3bd4a00700 1 -- 172.21.15.6:0/3026274305 wait complete. 2024-04-26T01:43:47.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:46 smithi006 ceph-mon[28419]: pgmap v382: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:47.213 INFO:teuthology.orchestra.run.smithi006.stdout:206158430272 2024-04-26T01:43:47.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T01:43:47.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:46 smithi188 ceph-mon[32751]: pgmap v382: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:47.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:46 smithi029 ceph-mon[32775]: pgmap v382: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:47.543 INFO:teuthology.orchestra.run.smithi006.stdout:146028888141 2024-04-26T01:43:47.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T01:43:48.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.023+0000 7faf5a800700 1 -- 172.21.15.6:0/1205061436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf5410e3b0 msgr2=0x7faf54115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:48.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.023+0000 7faf5a800700 1 --2- 172.21.15.6:0/1205061436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf5410e3b0 0x7faf54115f80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0098f0 tx=0x7faf4c009c00 comp rx=0 tx=0).stop 2024-04-26T01:43:48.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.024+0000 7faf5a800700 1 -- 172.21.15.6:0/1205061436 shutdown_connections 2024-04-26T01:43:48.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.024+0000 7faf5a800700 1 --2- 172.21.15.6:0/1205061436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf5410e3b0 0x7faf54115f80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:48.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.024+0000 7faf5a800700 1 --2- 172.21.15.6:0/1205061436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf541092d0 0x7faf541096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:48.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.024+0000 7faf5a800700 1 --2- 172.21.15.6:0/1205061436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf54109c80 0x7faf5410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:48.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.024+0000 7faf5a800700 1 -- 172.21.15.6:0/1205061436 >> 172.21.15.6:0/1205061436 conn(0x7faf54076210 msgr2=0x7faf54107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:48.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.024+0000 7faf5a800700 1 -- 172.21.15.6:0/1205061436 shutdown_connections 2024-04-26T01:43:48.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.024+0000 7faf5a800700 1 -- 172.21.15.6:0/1205061436 wait complete. 2024-04-26T01:43:48.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.025+0000 7faf5a800700 1 Processor -- start 2024-04-26T01:43:48.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.025+0000 7faf5a800700 1 -- start start 2024-04-26T01:43:48.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.025+0000 7faf5a800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf541092d0 0x7faf5419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:48.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.025+0000 7faf5a800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf54109c80 0x7faf5419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:48.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.025+0000 7faf5a800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5410e3b0 0x7faf541a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:48.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.025+0000 7faf5a800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7faf541192c0 con 0x7faf5410e3b0 2024-04-26T01:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.025+0000 7faf5a800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faf54119140 con 0x7faf54109c80 2024-04-26T01:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.025+0000 7faf5a800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faf54119440 con 0x7faf541092d0 2024-04-26T01:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.026+0000 7faf53e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5410e3b0 0x7faf541a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.026+0000 7faf53e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5410e3b0 0x7faf541a4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42784/0 (socket says 172.21.15.6:42784) 2024-04-26T01:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.026+0000 7faf53e00700 1 -- 172.21.15.6:0/171707394 learned_addr learned my addr 172.21.15.6:0/171707394 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.026+0000 7faf4be00700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf54109c80 0x7faf5419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.026+0000 7faf53400700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf541092d0 0x7faf5419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.026+0000 7faf53e00700 1 -- 172.21.15.6:0/171707394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf541092d0 msgr2=0x7faf5419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.026+0000 7faf53e00700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf541092d0 0x7faf5419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.026+0000 7faf53e00700 1 -- 172.21.15.6:0/171707394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf54109c80 msgr2=0x7faf5419fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.026+0000 7faf53e00700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf54109c80 0x7faf5419fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.026+0000 7faf53e00700 1 -- 172.21.15.6:0/171707394 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf541a4740 con 0x7faf5410e3b0 2024-04-26T01:43:48.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.027+0000 7faf53e00700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5410e3b0 0x7faf541a4000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009810 tx=0x7faf4c00fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:48.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.027+0000 7faf50c00700 1 -- 172.21.15.6:0/171707394 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c027070 con 0x7faf5410e3b0 2024-04-26T01:43:48.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.027+0000 7faf5a800700 1 -- 172.21.15.6:0/171707394 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf541a49d0 con 0x7faf5410e3b0 2024-04-26T01:43:48.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.027+0000 7faf50c00700 1 -- 172.21.15.6:0/171707394 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf4c005570 con 0x7faf5410e3b0 2024-04-26T01:43:48.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.027+0000 7faf50c00700 1 -- 172.21.15.6:0/171707394 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c004930 con 0x7faf5410e3b0 2024-04-26T01:43:48.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.027+0000 7faf5a800700 1 -- 172.21.15.6:0/171707394 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf541ac2b0 con 0x7faf5410e3b0 2024-04-26T01:43:48.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.028+0000 7faf5a800700 1 -- 172.21.15.6:0/171707394 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faf540631d0 con 0x7faf5410e3b0 2024-04-26T01:43:48.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.030+0000 7faf50c00700 1 -- 172.21.15.6:0/171707394 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7faf4c02a070 con 0x7faf5410e3b0 2024-04-26T01:43:48.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.032+0000 7faf50c00700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7faf380a3d50 0x7faf380a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:48.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.032+0000 7faf50c00700 1 -- 172.21.15.6:0/171707394 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8269+0+0 (secure 0 0 0) 0x7faf4c0d1340 con 0x7faf5410e3b0 2024-04-26T01:43:48.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.032+0000 7faf53400700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7faf380a3d50 0x7faf380a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:48.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.032+0000 7faf50c00700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672] conn(0x7faf380ad990 0x7faf380afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:48.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.032+0000 7faf50c00700 1 -- 172.21.15.6:0/171707394 --> [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf380b0500 con 0x7faf380ad990 2024-04-26T01:43:48.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.032+0000 7faf50c00700 1 -- 172.21.15.6:0/171707394 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7faf4c0d16f0 con 0x7faf5410e3b0 2024-04-26T01:43:48.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.033+0000 7faf4be00700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672] conn(0x7faf380ad990 0x7faf380afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:48.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.033+0000 7faf53400700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7faf380a3d50 0x7faf380a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faf4400f540 tx=0x7faf4400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:48.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.033+0000 7faf4be00700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672] conn(0x7faf380ad990 0x7faf380afe30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:48.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.035+0000 7faf50c00700 1 -- 172.21.15.6:0/171707394 <== osd.2 v2:172.21.15.188:6800/3508067672 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf380b0500 con 0x7faf380ad990 2024-04-26T01:43:48.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.055+0000 7faf5a800700 1 -- 172.21.15.6:0/171707394 --> [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faf54063650 con 0x7faf380ad990 2024-04-26T01:43:48.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.056+0000 7faf50c00700 1 -- 172.21.15.6:0/171707394 <== osd.2 v2:172.21.15.188:6800/3508067672 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faf54063650 con 0x7faf380ad990 2024-04-26T01:43:48.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.056+0000 7faf5a800700 1 -- 172.21.15.6:0/171707394 >> [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672] conn(0x7faf380ad990 msgr2=0x7faf380afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:48.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.056+0000 7faf5a800700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672] conn(0x7faf380ad990 0x7faf380afe30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:48.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.057+0000 7faf5a800700 1 -- 172.21.15.6:0/171707394 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7faf380a3d50 msgr2=0x7faf380a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:48.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.057+0000 7faf5a800700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7faf380a3d50 0x7faf380a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faf4400f540 tx=0x7faf4400b040 comp rx=0 tx=0).stop 2024-04-26T01:43:48.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.057+0000 7faf5a800700 1 -- 172.21.15.6:0/171707394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5410e3b0 msgr2=0x7faf541a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:48.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.057+0000 7faf5a800700 1 --2- 172.21.15.6:0/171707394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5410e3b0 0x7faf541a4000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009810 tx=0x7faf4c00fa70 comp rx=0 tx=0).stop 2024-04-26T01:43:48.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.057+0000 7faf53e00700 1 -- 172.21.15.6:0/171707394 reap_dead start 2024-04-26T01:43:48.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.057+0000 7faf5a800700 1 -- 172.21.15.6:0/171707394 shutdown_connections 2024-04-26T01:43:48.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.057+0000 7faf5a800700 1 -- 172.21.15.6:0/171707394 >> 172.21.15.6:0/171707394 conn(0x7faf54076210 msgr2=0x7faf54076790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:48.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.057+0000 7faf5a800700 1 -- 172.21.15.6:0/171707394 shutdown_connections 2024-04-26T01:43:48.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:48.057+0000 7faf5a800700 1 -- 172.21.15.6:0/171707394 wait complete. 2024-04-26T01:43:49.027 INFO:teuthology.orchestra.run.smithi006.stdout:300647710769 2024-04-26T01:43:49.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-26T01:43:49.205 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:48 smithi006 ceph-mon[28419]: pgmap v383: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:49.205 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:48 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-04-26T01:43:49.362 INFO:teuthology.orchestra.run.smithi006.stdout:227633266748 2024-04-26T01:43:49.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T01:43:49.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:48 smithi188 ceph-mon[32751]: pgmap v383: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:49.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:48 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-04-26T01:43:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.387+0000 7f44c2c00700 1 -- 172.21.15.6:0/458546709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc06a750 msgr2=0x7f44bc06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.387+0000 7f44c2c00700 1 --2- 172.21.15.6:0/458546709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc06a750 0x7f44bc06abb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009990 tx=0x7f44a8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.387+0000 7f44c2c00700 1 -- 172.21.15.6:0/458546709 shutdown_connections 2024-04-26T01:43:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.387+0000 7f44c2c00700 1 --2- 172.21.15.6:0/458546709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44bc06b0f0 0x7f44bc10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.387+0000 7f44c2c00700 1 --2- 172.21.15.6:0/458546709 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44bc108a30 0x7f44bc108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.387+0000 7f44c2c00700 1 --2- 172.21.15.6:0/458546709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc06a750 0x7f44bc06abb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.387+0000 7f44c2c00700 1 -- 172.21.15.6:0/458546709 >> 172.21.15.6:0/458546709 conn(0x7f44bc0788f0 msgr2=0x7f44bc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.387+0000 7f44c2c00700 1 -- 172.21.15.6:0/458546709 shutdown_connections 2024-04-26T01:43:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.387+0000 7f44c2c00700 1 -- 172.21.15.6:0/458546709 wait complete. 2024-04-26T01:43:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.389+0000 7f44c2c00700 1 Processor -- start 2024-04-26T01:43:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.389+0000 7f44c2c00700 1 -- start start 2024-04-26T01:43:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.389+0000 7f44c2c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44bc06a750 0x7f44bc1991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.389+0000 7f44c2c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44bc06b0f0 0x7f44bc199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.389+0000 7f44c2c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc108a30 0x7f44bc19dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.389+0000 7f44c2c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc112c20 con 0x7f44bc108a30 2024-04-26T01:43:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.389+0000 7f44c2c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc112aa0 con 0x7f44bc06b0f0 2024-04-26T01:43:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.389+0000 7f44c2c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc112da0 con 0x7f44bc06a750 2024-04-26T01:43:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.389+0000 7f44bbe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44bc06a750 0x7f44bc1991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.389+0000 7f44bbe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44bc06a750 0x7f44bc1991f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34628/0 (socket says 172.21.15.6:34628) 2024-04-26T01:43:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.389+0000 7f44bbe00700 1 -- 172.21.15.6:0/2757812468 learned_addr learned my addr 172.21.15.6:0/2757812468 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:43:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.389+0000 7f44c0c00700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc108a30 0x7f44bc19dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.390+0000 7f44bb400700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44bc06b0f0 0x7f44bc199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.390+0000 7f44c0c00700 1 -- 172.21.15.6:0/2757812468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44bc06a750 msgr2=0x7f44bc1991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.390+0000 7f44c0c00700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44bc06a750 0x7f44bc1991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.390+0000 7f44c0c00700 1 -- 172.21.15.6:0/2757812468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44bc06b0f0 msgr2=0x7f44bc199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.390+0000 7f44c0c00700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44bc06b0f0 0x7f44bc199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.390+0000 7f44c0c00700 1 -- 172.21.15.6:0/2757812468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44bc19e200 con 0x7f44bc108a30 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.390+0000 7f44bbe00700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44bc06a750 0x7f44bc1991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.390+0000 7f44c0c00700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc108a30 0x7f44bc19dac0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f44b400efc0 tx=0x7f44b400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.391+0000 7f44b8c00700 1 -- 172.21.15.6:0/2757812468 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b40092e0 con 0x7f44bc108a30 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.391+0000 7f44b8c00700 1 -- 172.21.15.6:0/2757812468 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b4004bb0 con 0x7f44bc108a30 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.391+0000 7f44c2c00700 1 -- 172.21.15.6:0/2757812468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44bc19e490 con 0x7f44bc108a30 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.391+0000 7f44c2c00700 1 -- 172.21.15.6:0/2757812468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44bc1a5d70 con 0x7f44bc108a30 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.392+0000 7f44b8c00700 1 -- 172.21.15.6:0/2757812468 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b4004020 con 0x7f44bc108a30 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.392+0000 7f44c2c00700 1 -- 172.21.15.6:0/2757812468 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f44bc063150 con 0x7f44bc108a30 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.395+0000 7f44b8c00700 1 -- 172.21.15.6:0/2757812468 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f44b401b030 con 0x7f44bc108a30 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.396+0000 7f44b8c00700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f44a00a3cd0 0x7f44a00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.396+0000 7f44b8c00700 1 -- 172.21.15.6:0/2757812468 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8298+0+0 (secure 0 0 0) 0x7f44b40d0a70 con 0x7f44bc108a30 2024-04-26T01:43:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.397+0000 7f44bbe00700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f44a00a3cd0 0x7f44a00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.397+0000 7f44b8c00700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486] conn(0x7f44a00ad990 0x7f44a00afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.397+0000 7f44b8c00700 1 -- 172.21.15.6:0/2757812468 --> [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44b401de50 con 0x7f44a00ad990 2024-04-26T01:43:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.397+0000 7f44b8c00700 1 -- 172.21.15.6:0/2757812468 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f44b40d0e20 con 0x7f44bc108a30 2024-04-26T01:43:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.397+0000 7f44bb400700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486] conn(0x7f44a00ad990 0x7f44a00afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.398+0000 7f44bb400700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486] conn(0x7f44a00ad990 0x7f44a00afe30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.398+0000 7f44bbe00700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f44a00a3cd0 0x7f44a00a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f44ac009680 tx=0x7f44ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.399+0000 7f44b8c00700 1 -- 172.21.15.6:0/2757812468 <== osd.0 v2:172.21.15.6:6802/76518486 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44b401de50 con 0x7f44a00ad990 2024-04-26T01:43:49.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.424+0000 7f44c2c00700 1 -- 172.21.15.6:0/2757812468 --> [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f44bc063550 con 0x7f44a00ad990 2024-04-26T01:43:49.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.425+0000 7f44b8c00700 1 -- 172.21.15.6:0/2757812468 <== osd.0 v2:172.21.15.6:6802/76518486 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f44bc063550 con 0x7f44a00ad990 2024-04-26T01:43:49.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.425+0000 7f44c2c00700 1 -- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486] conn(0x7f44a00ad990 msgr2=0x7f44a00afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:49.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.425+0000 7f44c2c00700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486] conn(0x7f44a00ad990 0x7f44a00afe30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:49.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.425+0000 7f44c2c00700 1 -- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f44a00a3cd0 msgr2=0x7f44a00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.425+0000 7f44c2c00700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f44a00a3cd0 0x7f44a00a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f44ac009680 tx=0x7f44ac009260 comp rx=0 tx=0).stop 2024-04-26T01:43:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.425+0000 7f44c2c00700 1 -- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc108a30 msgr2=0x7f44bc19dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.425+0000 7f44c2c00700 1 --2- 172.21.15.6:0/2757812468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc108a30 0x7f44bc19dac0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f44b400efc0 tx=0x7f44b400c3a0 comp rx=0 tx=0).stop 2024-04-26T01:43:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.426+0000 7f44c0c00700 1 -- 172.21.15.6:0/2757812468 reap_dead start 2024-04-26T01:43:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.426+0000 7f44c2c00700 1 -- 172.21.15.6:0/2757812468 shutdown_connections 2024-04-26T01:43:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.426+0000 7f44c2c00700 1 -- 172.21.15.6:0/2757812468 >> 172.21.15.6:0/2757812468 conn(0x7f44bc0788f0 msgr2=0x7f44bc0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.426+0000 7f44c2c00700 1 -- 172.21.15.6:0/2757812468 shutdown_connections 2024-04-26T01:43:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:49.426+0000 7f44c2c00700 1 -- 172.21.15.6:0/2757812468 wait complete. 2024-04-26T01:43:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:48 smithi029 ceph-mon[32775]: pgmap v383: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:48 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-04-26T01:43:49.736 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:49.772 INFO:teuthology.orchestra.run.smithi006.stdout:356482285607 2024-04-26T01:43:49.772 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-26T01:43:50.274 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:49 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-04-26T01:43:50.275 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:49 smithi006 ceph-mon[28419]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T01:43:50.277 INFO:teuthology.orchestra.run.smithi006.stdout:77309411417 2024-04-26T01:43:50.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T01:43:50.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:49 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-04-26T01:43:50.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:49 smithi188 ceph-mon[32751]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T01:43:50.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:49 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-04-26T01:43:50.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:49 smithi029 ceph-mon[32775]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T01:43:50.726 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:51.192 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:50 smithi006 ceph-mon[28419]: pgmap v385: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T01:43:51.192 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:50 smithi006 ceph-mon[28419]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T01:43:51.240 INFO:teuthology.orchestra.run.smithi006.stdout:34359738468 2024-04-26T01:43:51.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T01:43:51.273 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:50 smithi188 ceph-mon[32751]: pgmap v385: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T01:43:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:50 smithi188 ceph-mon[32751]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T01:43:51.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:50 smithi029 ceph-mon[32775]: pgmap v385: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T01:43:51.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:50 smithi029 ceph-mon[32775]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T01:43:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:53.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:52 smithi006 ceph-mon[28419]: pgmap v387: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:43:53.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:52 smithi188 ceph-mon[32751]: pgmap v387: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:43:53.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:52 smithi029 ceph-mon[32775]: pgmap v387: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:43:55.256 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:54 smithi006 ceph-mon[28419]: pgmap v388: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:43:55.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:54 smithi188 ceph-mon[32751]: pgmap v388: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:43:55.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:54 smithi029 ceph-mon[32775]: pgmap v388: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:43:57.246 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:56 smithi006 ceph-mon[28419]: pgmap v389: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:43:57.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:56 smithi188 ceph-mon[32751]: pgmap v389: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:43:57.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:56 smithi029 ceph-mon[32775]: pgmap v389: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:43:59.201 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:43:59 smithi006 ceph-mon[28419]: pgmap v390: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:43:59.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:43:59 smithi188 ceph-mon[32751]: pgmap v390: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:43:59.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:43:59 smithi029 ceph-mon[32775]: pgmap v390: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:44:01.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:01 smithi006 ceph-mon[28419]: pgmap v391: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:44:01.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:01 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/4121798900' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T01:44:01.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:01 smithi188 ceph-mon[32751]: pgmap v391: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:44:01.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:01 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/4121798900' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T01:44:01.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:01 smithi029 ceph-mon[32775]: pgmap v391: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:44:01.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:01 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/4121798900' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T01:44:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.912+0000 7efd36e00700 1 -- 172.21.15.6:0/1073075109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd30108a30 msgr2=0x7efd30108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.912+0000 7efd36e00700 1 --2- 172.21.15.6:0/1073075109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd30108a30 0x7efd30108e10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efd20009990 tx=0x7efd20009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.913+0000 7efd36e00700 1 -- 172.21.15.6:0/1073075109 shutdown_connections 2024-04-26T01:44:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.913+0000 7efd36e00700 1 --2- 172.21.15.6:0/1073075109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd3006b0f0 0x7efd3010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.913+0000 7efd36e00700 1 --2- 172.21.15.6:0/1073075109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd30108a30 0x7efd30108e10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.913+0000 7efd36e00700 1 --2- 172.21.15.6:0/1073075109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd3006a750 0x7efd3006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.913+0000 7efd36e00700 1 -- 172.21.15.6:0/1073075109 >> 172.21.15.6:0/1073075109 conn(0x7efd300788f0 msgr2=0x7efd30078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:03.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.913+0000 7efd36e00700 1 -- 172.21.15.6:0/1073075109 shutdown_connections 2024-04-26T01:44:03.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.913+0000 7efd36e00700 1 -- 172.21.15.6:0/1073075109 wait complete. 2024-04-26T01:44:03.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.914+0000 7efd36e00700 1 Processor -- start 2024-04-26T01:44:03.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.915+0000 7efd36e00700 1 -- start start 2024-04-26T01:44:03.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.915+0000 7efd36e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd3006a750 0x7efd30199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:03.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.915+0000 7efd36e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd3006b0f0 0x7efd301996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:03.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.915+0000 7efd36e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd30108a30 0x7efd3019da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:03.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.915+0000 7efd36e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efd30112c20 con 0x7efd3006a750 2024-04-26T01:44:03.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.915+0000 7efd36e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efd30112aa0 con 0x7efd30108a30 2024-04-26T01:44:03.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.915+0000 7efd36e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efd30112da0 con 0x7efd3006b0f0 2024-04-26T01:44:03.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.915+0000 7efd2fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd3006a750 0x7efd30199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.915+0000 7efd2f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd3006b0f0 0x7efd301996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.915+0000 7efd34e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd30108a30 0x7efd3019da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.915+0000 7efd2fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd3006a750 0x7efd30199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46420/0 (socket says 172.21.15.6:46420) 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.915+0000 7efd2fe00700 1 -- 172.21.15.6:0/4121798900 learned_addr learned my addr 172.21.15.6:0/4121798900 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.916+0000 7efd34e00700 1 -- 172.21.15.6:0/4121798900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd3006b0f0 msgr2=0x7efd301996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.916+0000 7efd34e00700 1 --2- 172.21.15.6:0/4121798900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd3006b0f0 0x7efd301996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.916+0000 7efd34e00700 1 -- 172.21.15.6:0/4121798900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd3006a750 msgr2=0x7efd30199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.916+0000 7efd34e00700 1 --2- 172.21.15.6:0/4121798900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd3006a750 0x7efd30199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.916+0000 7efd34e00700 1 -- 172.21.15.6:0/4121798900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3019e180 con 0x7efd30108a30 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.916+0000 7efd34e00700 1 --2- 172.21.15.6:0/4121798900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd30108a30 0x7efd3019da40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efd2800efc0 tx=0x7efd2800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.916+0000 7efd2cc00700 1 -- 172.21.15.6:0/4121798900 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd280092e0 con 0x7efd30108a30 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.916+0000 7efd2cc00700 1 -- 172.21.15.6:0/4121798900 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd28004bb0 con 0x7efd30108a30 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.916+0000 7efd36e00700 1 -- 172.21.15.6:0/4121798900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd3019e410 con 0x7efd30108a30 2024-04-26T01:44:03.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.917+0000 7efd2cc00700 1 -- 172.21.15.6:0/4121798900 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28004020 con 0x7efd30108a30 2024-04-26T01:44:03.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.917+0000 7efd36e00700 1 -- 172.21.15.6:0/4121798900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd30101940 con 0x7efd30108a30 2024-04-26T01:44:03.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.918+0000 7efd36e00700 1 -- 172.21.15.6:0/4121798900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd30104170 con 0x7efd30108a30 2024-04-26T01:44:03.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.920+0000 7efd2cc00700 1 -- 172.21.15.6:0/4121798900 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7efd2801b030 con 0x7efd30108a30 2024-04-26T01:44:03.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.921+0000 7efd2cc00700 1 --2- 172.21.15.6:0/4121798900 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7efd140a3cd0 0x7efd140a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:03.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.922+0000 7efd2cc00700 1 -- 172.21.15.6:0/4121798900 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7efd280d03e0 con 0x7efd30108a30 2024-04-26T01:44:03.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.922+0000 7efd2fe00700 1 --2- 172.21.15.6:0/4121798900 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7efd140a3cd0 0x7efd140a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:03.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.922+0000 7efd2fe00700 1 --2- 172.21.15.6:0/4121798900 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7efd140a3cd0 0x7efd140a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efd20006010 tx=0x7efd2000b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:03.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:43:59.924+0000 7efd2cc00700 1 -- 172.21.15.6:0/4121798900 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd28022020 con 0x7efd30108a30 2024-04-26T01:44:03.081 INFO:teuthology.orchestra.run.smithi006.stdout:300647710772 2024-04-26T01:44:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.167+0000 7efd36e00700 1 -- 172.21.15.6:0/4121798900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7efd3002df80 con 0x7efd30108a30 2024-04-26T01:44:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.168+0000 7efd2cc00700 1 -- 172.21.15.6:0/4121798900 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7efd28010040 con 0x7efd30108a30 2024-04-26T01:44:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.170+0000 7efd36e00700 1 -- 172.21.15.6:0/4121798900 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7efd140a3cd0 msgr2=0x7efd140a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.170+0000 7efd36e00700 1 --2- 172.21.15.6:0/4121798900 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7efd140a3cd0 0x7efd140a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efd20006010 tx=0x7efd2000b280 comp rx=0 tx=0).stop 2024-04-26T01:44:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.170+0000 7efd36e00700 1 -- 172.21.15.6:0/4121798900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd30108a30 msgr2=0x7efd3019da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.171+0000 7efd36e00700 1 --2- 172.21.15.6:0/4121798900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd30108a30 0x7efd3019da40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efd2800efc0 tx=0x7efd2800c3a0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.171+0000 7efd36e00700 1 -- 172.21.15.6:0/4121798900 shutdown_connections 2024-04-26T01:44:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.171+0000 7efd36e00700 1 --2- 172.21.15.6:0/4121798900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd30108a30 0x7efd3019da40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.171+0000 7efd36e00700 1 --2- 172.21.15.6:0/4121798900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd3006a750 0x7efd30199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.171+0000 7efd36e00700 1 --2- 172.21.15.6:0/4121798900 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7efd140a3cd0 0x7efd140a6190 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.171+0000 7efd36e00700 1 --2- 172.21.15.6:0/4121798900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd3006b0f0 0x7efd301996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.171+0000 7efd36e00700 1 -- 172.21.15.6:0/4121798900 >> 172.21.15.6:0/4121798900 conn(0x7efd300788f0 msgr2=0x7efd300ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:03.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.171+0000 7efd36e00700 1 -- 172.21.15.6:0/4121798900 shutdown_connections 2024-04-26T01:44:03.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:00.171+0000 7efd36e00700 1 -- 172.21.15.6:0/4121798900 wait complete. 2024-04-26T01:44:03.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:03 smithi006 ceph-mon[28419]: pgmap v392: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:44:03.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:03 smithi188 ceph-mon[32751]: pgmap v392: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:44:03.402 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:03.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:03 smithi029 ceph-mon[32775]: pgmap v392: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T01:44:03.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.773+0000 7f5d68600700 1 -- 172.21.15.6:0/4252684472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d6010e3b0 msgr2=0x7f5d60115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.773+0000 7f5d68600700 1 --2- 172.21.15.6:0/4252684472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d6010e3b0 0x7f5d60115f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5d580099c0 tx=0x7f5d58009cd0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.773+0000 7f5d68600700 1 -- 172.21.15.6:0/4252684472 shutdown_connections 2024-04-26T01:44:03.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.773+0000 7f5d68600700 1 --2- 172.21.15.6:0/4252684472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d6010e3b0 0x7f5d60115f80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.773+0000 7f5d68600700 1 --2- 172.21.15.6:0/4252684472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d601092d0 0x7f5d601096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.773+0000 7f5d68600700 1 --2- 172.21.15.6:0/4252684472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d60109c80 0x7f5d6010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.773+0000 7f5d68600700 1 -- 172.21.15.6:0/4252684472 >> 172.21.15.6:0/4252684472 conn(0x7f5d60076210 msgr2=0x7f5d60107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:03.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.773+0000 7f5d68600700 1 -- 172.21.15.6:0/4252684472 shutdown_connections 2024-04-26T01:44:03.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.773+0000 7f5d68600700 1 -- 172.21.15.6:0/4252684472 wait complete. 2024-04-26T01:44:03.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.775+0000 7f5d68600700 1 Processor -- start 2024-04-26T01:44:03.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.776+0000 7f5d68600700 1 -- start start 2024-04-26T01:44:03.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.776+0000 7f5d68600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d601092d0 0x7f5d6019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:03.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.776+0000 7f5d68600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d60109c80 0x7f5d6019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.776+0000 7f5d68600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6010e3b0 0x7f5d601a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.776+0000 7f5d68600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5d601192a0 con 0x7f5d60109c80 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.776+0000 7f5d68600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5d60119120 con 0x7f5d6010e3b0 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.776+0000 7f5d68600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5d60119420 con 0x7f5d601092d0 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.776+0000 7f5d65200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d60109c80 0x7f5d6019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.776+0000 7f5d65200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d60109c80 0x7f5d6019fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46446/0 (socket says 172.21.15.6:46446) 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.776+0000 7f5d65200700 1 -- 172.21.15.6:0/794864254 learned_addr learned my addr 172.21.15.6:0/794864254 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.776+0000 7f5d66600700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6010e3b0 0x7f5d601a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.777+0000 7f5d65c00700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d601092d0 0x7f5d6019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.777+0000 7f5d65200700 1 -- 172.21.15.6:0/794864254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d601092d0 msgr2=0x7f5d6019f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.777+0000 7f5d65200700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d601092d0 0x7f5d6019f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.777+0000 7f5d65200700 1 -- 172.21.15.6:0/794864254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6010e3b0 msgr2=0x7f5d601a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.777+0000 7f5d65200700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6010e3b0 0x7f5d601a3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.777+0000 7f5d65200700 1 -- 172.21.15.6:0/794864254 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d601a46e0 con 0x7f5d60109c80 2024-04-26T01:44:03.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.777+0000 7f5d66600700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6010e3b0 0x7f5d601a3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.777+0000 7f5d65200700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d60109c80 0x7f5d6019fc10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00d950 tx=0x7f5d4c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.778+0000 7f5d5e000700 1 -- 172.21.15.6:0/794864254 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c016070 con 0x7f5d60109c80 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.778+0000 7f5d5e000700 1 -- 172.21.15.6:0/794864254 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d4c004830 con 0x7f5d60109c80 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.778+0000 7f5d68600700 1 -- 172.21.15.6:0/794864254 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d601a49d0 con 0x7f5d60109c80 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.778+0000 7f5d5e000700 1 -- 172.21.15.6:0/794864254 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c005520 con 0x7f5d60109c80 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.778+0000 7f5d68600700 1 -- 172.21.15.6:0/794864254 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d601ac250 con 0x7f5d60109c80 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.779+0000 7f5d68600700 1 -- 172.21.15.6:0/794864254 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d60069730 con 0x7f5d60109c80 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.781+0000 7f5d5e000700 1 -- 172.21.15.6:0/794864254 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f5d4c004220 con 0x7f5d60109c80 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.782+0000 7f5d5e000700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5d480a3d50 0x7f5d480a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.783+0000 7f5d5e000700 1 -- 172.21.15.6:0/794864254 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f5d4c0d01e0 con 0x7f5d60109c80 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.783+0000 7f5d65c00700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5d480a3d50 0x7f5d480a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.784+0000 7f5d65c00700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5d480a3d50 0x7f5d480a6210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5d5400f540 tx=0x7f5d5400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:03.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:02.788+0000 7f5d5e000700 1 -- 172.21.15.6:0/794864254 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d4c01f020 con 0x7f5d60109c80 2024-04-26T01:44:03.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.032+0000 7f5d68600700 1 -- 172.21.15.6:0/794864254 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f5d6002df80 con 0x7f5d60109c80 2024-04-26T01:44:03.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.033+0000 7f5d5e000700 1 -- 172.21.15.6:0/794864254 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5d4c09c9c0 con 0x7f5d60109c80 2024-04-26T01:44:03.831 INFO:teuthology.orchestra.run.smithi006.stdout:171798691915 2024-04-26T01:44:03.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.036+0000 7f5d68600700 1 -- 172.21.15.6:0/794864254 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5d480a3d50 msgr2=0x7f5d480a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.036+0000 7f5d68600700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5d480a3d50 0x7f5d480a6210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5d5400f540 tx=0x7f5d5400b040 comp rx=0 tx=0).stop 2024-04-26T01:44:03.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.036+0000 7f5d68600700 1 -- 172.21.15.6:0/794864254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d60109c80 msgr2=0x7f5d6019fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.036+0000 7f5d68600700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d60109c80 0x7f5d6019fc10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00d950 tx=0x7f5d4c00dc60 comp rx=0 tx=0).stop 2024-04-26T01:44:03.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.036+0000 7f5d68600700 1 -- 172.21.15.6:0/794864254 shutdown_connections 2024-04-26T01:44:03.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.036+0000 7f5d68600700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6010e3b0 0x7f5d601a3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.036+0000 7f5d68600700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f5d480a3d50 0x7f5d480a6210 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.036+0000 7f5d68600700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d601092d0 0x7f5d6019f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.036+0000 7f5d68600700 1 --2- 172.21.15.6:0/794864254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d60109c80 0x7f5d6019fc10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.036+0000 7f5d68600700 1 -- 172.21.15.6:0/794864254 >> 172.21.15.6:0/794864254 conn(0x7f5d60076210 msgr2=0x7f5d60076750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:03.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.036+0000 7f5d68600700 1 -- 172.21.15.6:0/794864254 shutdown_connections 2024-04-26T01:44:03.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:03.036+0000 7f5d68600700 1 -- 172.21.15.6:0/794864254 wait complete. 2024-04-26T01:44:04.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:04 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/794864254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T01:44:04.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:04 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/794864254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T01:44:04.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:04 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/794864254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T01:44:04.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.605+0000 7f95ebe00700 1 -- 172.21.15.6:0/3900800294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95e4113c10 msgr2=0x7f95e4116000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:04.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.605+0000 7f95ebe00700 1 --2- 172.21.15.6:0/3900800294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95e4113c10 0x7f95e4116000 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f95dc00b0a0 tx=0x7f95dc00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:44:04.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.605+0000 7f95ebe00700 1 -- 172.21.15.6:0/3900800294 shutdown_connections 2024-04-26T01:44:04.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.605+0000 7f95ebe00700 1 --2- 172.21.15.6:0/3900800294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95e4113c10 0x7f95e4116000 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:04.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.605+0000 7f95ebe00700 1 --2- 172.21.15.6:0/3900800294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e4074c70 0x7f95e4075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:04.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.605+0000 7f95ebe00700 1 --2- 172.21.15.6:0/3900800294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95e4075590 0x7f95e41136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:04.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.605+0000 7f95ebe00700 1 -- 172.21.15.6:0/3900800294 >> 172.21.15.6:0/3900800294 conn(0x7f95e4079ba0 msgr2=0x7f95e4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:04.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.605+0000 7f95ebe00700 1 -- 172.21.15.6:0/3900800294 shutdown_connections 2024-04-26T01:44:04.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.605+0000 7f95ebe00700 1 -- 172.21.15.6:0/3900800294 wait complete. 2024-04-26T01:44:04.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.607+0000 7f95ebe00700 1 Processor -- start 2024-04-26T01:44:04.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.607+0000 7f95ebe00700 1 -- start start 2024-04-26T01:44:04.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.607+0000 7f95ebe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95e4074c70 0x7f95e419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:04.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.607+0000 7f95ebe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e4075590 0x7f95e419fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:04.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.607+0000 7f95ebe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95e4113c10 0x7f95e41a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:04.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.607+0000 7f95ebe00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f95e4119220 con 0x7f95e4075590 2024-04-26T01:44:04.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.607+0000 7f95ebe00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f95e41190a0 con 0x7f95e4113c10 2024-04-26T01:44:04.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.607+0000 7f95ebe00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f95e41193a0 con 0x7f95e4074c70 2024-04-26T01:44:04.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.608+0000 7f95e9400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95e4074c70 0x7f95e419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:04.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.608+0000 7f95e9e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95e4113c10 0x7f95e41a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:04.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.608+0000 7f95e9400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95e4074c70 0x7f95e419f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47476/0 (socket says 172.21.15.6:47476) 2024-04-26T01:44:04.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.608+0000 7f95e9400700 1 -- 172.21.15.6:0/4078257158 learned_addr learned my addr 172.21.15.6:0/4078257158 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:04.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.608+0000 7f95e8a00700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e4075590 0x7f95e419fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.608+0000 7f95e9e00700 1 -- 172.21.15.6:0/4078257158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95e4074c70 msgr2=0x7f95e419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.608+0000 7f95e9e00700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95e4074c70 0x7f95e419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.608+0000 7f95e9e00700 1 -- 172.21.15.6:0/4078257158 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e4075590 msgr2=0x7f95e419fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.608+0000 7f95e9e00700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e4075590 0x7f95e419fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.608+0000 7f95e9e00700 1 -- 172.21.15.6:0/4078257158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e41a4760 con 0x7f95e4113c10 2024-04-26T01:44:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.608+0000 7f95e9400700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95e4074c70 0x7f95e419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.608+0000 7f95e9e00700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95e4113c10 0x7f95e41a4020 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f95dc00b070 tx=0x7f95dc004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.609+0000 7f95e2000700 1 -- 172.21.15.6:0/4078257158 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc021070 con 0x7f95e4113c10 2024-04-26T01:44:04.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.609+0000 7f95e2000700 1 -- 172.21.15.6:0/4078257158 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95dc007e30 con 0x7f95e4113c10 2024-04-26T01:44:04.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.609+0000 7f95ebe00700 1 -- 172.21.15.6:0/4078257158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e41a49f0 con 0x7f95e4113c10 2024-04-26T01:44:04.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.609+0000 7f95e2000700 1 -- 172.21.15.6:0/4078257158 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc0045f0 con 0x7f95e4113c10 2024-04-26T01:44:04.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.609+0000 7f95ebe00700 1 -- 172.21.15.6:0/4078257158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e41ac2d0 con 0x7f95e4113c10 2024-04-26T01:44:04.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.610+0000 7f95ebe00700 1 -- 172.21.15.6:0/4078257158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e4069730 con 0x7f95e4113c10 2024-04-26T01:44:04.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.613+0000 7f95e2000700 1 -- 172.21.15.6:0/4078257158 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f95dc007980 con 0x7f95e4113c10 2024-04-26T01:44:04.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.615+0000 7f95e2000700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f95cc0a3d50 0x7f95cc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:04.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.615+0000 7f95e2000700 1 -- 172.21.15.6:0/4078257158 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f95dc00f020 con 0x7f95e4113c10 2024-04-26T01:44:04.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.615+0000 7f95e9400700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f95cc0a3d50 0x7f95cc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:04.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.616+0000 7f95e9400700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f95cc0a3d50 0x7f95cc0a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f95d8011550 tx=0x7f95d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:04.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.616+0000 7f95e2000700 1 -- 172.21.15.6:0/4078257158 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f95dc02a020 con 0x7f95e4113c10 2024-04-26T01:44:04.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.868+0000 7f95ebe00700 1 -- 172.21.15.6:0/4078257158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f95e41ac4a0 con 0x7f95e4113c10 2024-04-26T01:44:04.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.869+0000 7f95e2000700 1 -- 172.21.15.6:0/4078257158 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f95dc09e500 con 0x7f95e4113c10 2024-04-26T01:44:04.866 INFO:teuthology.orchestra.run.smithi006.stdout:322122547245 2024-04-26T01:44:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.872+0000 7f95ebe00700 1 -- 172.21.15.6:0/4078257158 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f95cc0a3d50 msgr2=0x7f95cc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.872+0000 7f95ebe00700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f95cc0a3d50 0x7f95cc0a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f95d8011550 tx=0x7f95d8009260 comp rx=0 tx=0).stop 2024-04-26T01:44:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.872+0000 7f95ebe00700 1 -- 172.21.15.6:0/4078257158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95e4113c10 msgr2=0x7f95e41a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.872+0000 7f95ebe00700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95e4113c10 0x7f95e41a4020 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f95dc00b070 tx=0x7f95dc004c80 comp rx=0 tx=0).stop 2024-04-26T01:44:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.872+0000 7f95ebe00700 1 -- 172.21.15.6:0/4078257158 shutdown_connections 2024-04-26T01:44:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.872+0000 7f95ebe00700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95e4113c10 0x7f95e41a4020 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.872+0000 7f95ebe00700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f95cc0a3d50 0x7f95cc0a6210 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.872+0000 7f95ebe00700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95e4074c70 0x7f95e419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.872+0000 7f95ebe00700 1 --2- 172.21.15.6:0/4078257158 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e4075590 0x7f95e419fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:04.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.872+0000 7f95ebe00700 1 -- 172.21.15.6:0/4078257158 >> 172.21.15.6:0/4078257158 conn(0x7f95e4079ba0 msgr2=0x7f95e4112a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:04.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.872+0000 7f95ebe00700 1 -- 172.21.15.6:0/4078257158 shutdown_connections 2024-04-26T01:44:04.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.872+0000 7f95ebe00700 1 -- 172.21.15.6:0/4078257158 wait complete. 2024-04-26T01:44:05.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.336+0000 7fbe24800700 1 -- 172.21.15.6:0/1147842909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c074c70 msgr2=0x7fbe1c075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:05.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.336+0000 7fbe24800700 1 --2- 172.21.15.6:0/1147842909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c074c70 0x7fbe1c075050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbe10009990 tx=0x7fbe10009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:05.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.336+0000 7fbe24800700 1 -- 172.21.15.6:0/1147842909 shutdown_connections 2024-04-26T01:44:05.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.336+0000 7fbe24800700 1 --2- 172.21.15.6:0/1147842909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c10f470 0x7fbe1c111860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:05.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.336+0000 7fbe24800700 1 --2- 172.21.15.6:0/1147842909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c074c70 0x7fbe1c075050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.336+0000 7fbe24800700 1 --2- 172.21.15.6:0/1147842909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe1c075590 0x7fbe1c10ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.336+0000 7fbe24800700 1 -- 172.21.15.6:0/1147842909 >> 172.21.15.6:0/1147842909 conn(0x7fbe1c079bf0 msgr2=0x7fbe1c07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.336+0000 7fbe24800700 1 -- 172.21.15.6:0/1147842909 shutdown_connections 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.336+0000 7fbe24800700 1 -- 172.21.15.6:0/1147842909 wait complete. 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.337+0000 7fbe24800700 1 Processor -- start 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.337+0000 7fbe24800700 1 -- start start 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.338+0000 7fbe24800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe1c074c70 0x7fbe1c19b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.338+0000 7fbe24800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c075590 0x7fbe1c19b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.338+0000 7fbe24800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c10f470 0x7fbe1c19fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.338+0000 7fbe24800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1c114e70 con 0x7fbe1c10f470 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.338+0000 7fbe24800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1c114cf0 con 0x7fbe1c075590 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.338+0000 7fbe24800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1c114ff0 con 0x7fbe1c074c70 2024-04-26T01:44:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.338+0000 7fbe21400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c075590 0x7fbe1c19b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.338+0000 7fbe21e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe1c074c70 0x7fbe1c19b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.338+0000 7fbe22800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c10f470 0x7fbe1c19fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.338+0000 7fbe21e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe1c074c70 0x7fbe1c19b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47446/0 (socket says 172.21.15.6:47446) 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.338+0000 7fbe21e00700 1 -- 172.21.15.6:0/1086099937 learned_addr learned my addr 172.21.15.6:0/1086099937 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.338+0000 7fbe21400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c075590 0x7fbe1c19b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:35840/0 (socket says 172.21.15.6:35840) 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.339+0000 7fbe21400700 1 -- 172.21.15.6:0/1086099937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe1c074c70 msgr2=0x7fbe1c19b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.339+0000 7fbe21400700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe1c074c70 0x7fbe1c19b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.339+0000 7fbe21400700 1 -- 172.21.15.6:0/1086099937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c10f470 msgr2=0x7fbe1c19fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.339+0000 7fbe21400700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c10f470 0x7fbe1c19fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.339+0000 7fbe21400700 1 -- 172.21.15.6:0/1086099937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe1c1a02e0 con 0x7fbe1c075590 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.339+0000 7fbe21e00700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe1c074c70 0x7fbe1c19b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.339+0000 7fbe22800700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c10f470 0x7fbe1c19fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.339+0000 7fbe21400700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c075590 0x7fbe1c19b860 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00d740 tx=0x7fbe0c00da50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.340+0000 7fbe16a00700 1 -- 172.21.15.6:0/1086099937 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c016070 con 0x7fbe1c075590 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.340+0000 7fbe16a00700 1 -- 172.21.15.6:0/1086099937 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe0c00f690 con 0x7fbe1c075590 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.340+0000 7fbe16a00700 1 -- 172.21.15.6:0/1086099937 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c003b50 con 0x7fbe1c075590 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.340+0000 7fbe24800700 1 -- 172.21.15.6:0/1086099937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe1c1a05d0 con 0x7fbe1c075590 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.340+0000 7fbe24800700 1 -- 172.21.15.6:0/1086099937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe1c1a7e50 con 0x7fbe1c075590 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.341+0000 7fbe24800700 1 -- 172.21.15.6:0/1086099937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe1c069730 con 0x7fbe1c075590 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.343+0000 7fbe16a00700 1 -- 172.21.15.6:0/1086099937 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fbe0c00f840 con 0x7fbe1c075590 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.344+0000 7fbe16a00700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fbe040a3d50 0x7fbe040a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.344+0000 7fbe16a00700 1 -- 172.21.15.6:0/1086099937 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fbe0c0d0290 con 0x7fbe1c075590 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.345+0000 7fbe21e00700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fbe040a3d50 0x7fbe040a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.345+0000 7fbe21e00700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fbe040a3d50 0x7fbe040a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbe10006010 tx=0x7fbe1000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.349+0000 7fbe16a00700 1 -- 172.21.15.6:0/1086099937 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe0c09ca70 con 0x7fbe1c075590 2024-04-26T01:44:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.589+0000 7fbe24800700 1 -- 172.21.15.6:0/1086099937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbe1c19c630 con 0x7fbe1c075590 2024-04-26T01:44:05.208 INFO:teuthology.orchestra.run.smithi006.stdout:55834574946 2024-04-26T01:44:05.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.590+0000 7fbe16a00700 1 -- 172.21.15.6:0/1086099937 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbe0c09c370 con 0x7fbe1c075590 2024-04-26T01:44:05.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.593+0000 7fbe24800700 1 -- 172.21.15.6:0/1086099937 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fbe040a3d50 msgr2=0x7fbe040a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:05.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.593+0000 7fbe24800700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fbe040a3d50 0x7fbe040a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbe10006010 tx=0x7fbe1000b540 comp rx=0 tx=0).stop 2024-04-26T01:44:05.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.593+0000 7fbe24800700 1 -- 172.21.15.6:0/1086099937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c075590 msgr2=0x7fbe1c19b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:05.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.593+0000 7fbe24800700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c075590 0x7fbe1c19b860 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00d740 tx=0x7fbe0c00da50 comp rx=0 tx=0).stop 2024-04-26T01:44:05.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.594+0000 7fbe24800700 1 -- 172.21.15.6:0/1086099937 shutdown_connections 2024-04-26T01:44:05.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.594+0000 7fbe24800700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe1c10f470 0x7fbe1c19fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:05.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.594+0000 7fbe24800700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fbe040a3d50 0x7fbe040a6210 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:05.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.594+0000 7fbe24800700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe1c074c70 0x7fbe1c19b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:05.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.594+0000 7fbe24800700 1 --2- 172.21.15.6:0/1086099937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c075590 0x7fbe1c19b860 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:05.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.594+0000 7fbe24800700 1 -- 172.21.15.6:0/1086099937 >> 172.21.15.6:0/1086099937 conn(0x7fbe1c079bf0 msgr2=0x7fbe1c0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:05.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.594+0000 7fbe24800700 1 -- 172.21.15.6:0/1086099937 shutdown_connections 2024-04-26T01:44:05.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:04.594+0000 7fbe24800700 1 -- 172.21.15.6:0/1086099937 wait complete. 2024-04-26T01:44:05.209 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:05 smithi006 ceph-mon[28419]: pgmap v393: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:05.209 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:05 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1086099937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T01:44:05.209 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:05 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/4078257158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T01:44:05.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:05 smithi188 ceph-mon[32751]: pgmap v393: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:05.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:05 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1086099937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T01:44:05.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:05 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/4078257158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T01:44:05.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:05 smithi029 ceph-mon[32775]: pgmap v393: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:05.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:05 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1086099937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T01:44:05.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:05 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/4078257158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T01:44:06.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.021+0000 7f330ea00700 1 -- 172.21.15.6:0/2898379467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330806b0f0 msgr2=0x7f330810f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.021+0000 7f330ea00700 1 --2- 172.21.15.6:0/2898379467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330806b0f0 0x7f330810f900 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f330000b0a0 tx=0x7f330000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.021+0000 7f330ea00700 1 -- 172.21.15.6:0/2898379467 shutdown_connections 2024-04-26T01:44:06.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.021+0000 7f330ea00700 1 --2- 172.21.15.6:0/2898379467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330806b0f0 0x7f330810f900 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.021+0000 7f330ea00700 1 --2- 172.21.15.6:0/2898379467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3308108a30 0x7f3308108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.021+0000 7f330ea00700 1 --2- 172.21.15.6:0/2898379467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f330806a750 0x7f330806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.021+0000 7f330ea00700 1 -- 172.21.15.6:0/2898379467 >> 172.21.15.6:0/2898379467 conn(0x7f33080788f0 msgr2=0x7f3308078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.021+0000 7f330ea00700 1 -- 172.21.15.6:0/2898379467 shutdown_connections 2024-04-26T01:44:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.021+0000 7f330ea00700 1 -- 172.21.15.6:0/2898379467 wait complete. 2024-04-26T01:44:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.022+0000 7f330ea00700 1 Processor -- start 2024-04-26T01:44:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.023+0000 7f330ea00700 1 -- start start 2024-04-26T01:44:06.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.023+0000 7f330ea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f330806a750 0x7f330819b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.023+0000 7f330ea00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f330806b0f0 0x7f330819b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.023+0000 7f330ea00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3308108a30 0x7f330819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.023+0000 7f330ea00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3308112c60 con 0x7f330806a750 2024-04-26T01:44:06.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.023+0000 7f330ea00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3308112ae0 con 0x7f330806b0f0 2024-04-26T01:44:06.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.023+0000 7f330ea00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3308112de0 con 0x7f3308108a30 2024-04-26T01:44:06.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.023+0000 7f3307400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f330806b0f0 0x7f330819b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.023+0000 7f3307e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f330806a750 0x7f330819b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.023+0000 7f3307400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f330806b0f0 0x7f330819b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:35886/0 (socket says 172.21.15.6:35886) 2024-04-26T01:44:06.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.023+0000 7f3307400700 1 -- 172.21.15.6:0/966261249 learned_addr learned my addr 172.21.15.6:0/966261249 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:06.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.023+0000 7f330ca00700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3308108a30 0x7f330819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.024+0000 7f3307400700 1 -- 172.21.15.6:0/966261249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3308108a30 msgr2=0x7f330819fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.024+0000 7f3307400700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3308108a30 0x7f330819fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.024+0000 7f3307400700 1 -- 172.21.15.6:0/966261249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f330806a750 msgr2=0x7f330819b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.024+0000 7f3307400700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f330806a750 0x7f330819b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.024+0000 7f3307400700 1 -- 172.21.15.6:0/966261249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33081a0440 con 0x7f330806b0f0 2024-04-26T01:44:06.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.024+0000 7f3307400700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f330806b0f0 0x7f330819b970 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f32f400e9c0 tx=0x7f32f400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:06.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.024+0000 7f3304c00700 1 -- 172.21.15.6:0/966261249 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f400cbd0 con 0x7f330806b0f0 2024-04-26T01:44:06.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.024+0000 7f3304c00700 1 -- 172.21.15.6:0/966261249 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32f400ee60 con 0x7f330806b0f0 2024-04-26T01:44:06.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.024+0000 7f3304c00700 1 -- 172.21.15.6:0/966261249 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f40055c0 con 0x7f330806b0f0 2024-04-26T01:44:06.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.025+0000 7f330ea00700 1 -- 172.21.15.6:0/966261249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33081a0730 con 0x7f330806b0f0 2024-04-26T01:44:06.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.025+0000 7f330ea00700 1 -- 172.21.15.6:0/966261249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33081a7fb0 con 0x7f330806b0f0 2024-04-26T01:44:06.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.026+0000 7f330ea00700 1 -- 172.21.15.6:0/966261249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3308069730 con 0x7f330806b0f0 2024-04-26T01:44:06.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.028+0000 7f3304c00700 1 -- 172.21.15.6:0/966261249 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f32f4004750 con 0x7f330806b0f0 2024-04-26T01:44:06.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.029+0000 7f3304c00700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f32ec0a3d50 0x7f32ec0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.030+0000 7f3304c00700 1 -- 172.21.15.6:0/966261249 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f32f40d0a70 con 0x7f330806b0f0 2024-04-26T01:44:06.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.030+0000 7f3307e00700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f32ec0a3d50 0x7f32ec0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.031+0000 7f3307e00700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f32ec0a3d50 0x7f32ec0a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f32f8009be0 tx=0x7f32f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:06.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.032+0000 7f3304c00700 1 -- 172.21.15.6:0/966261249 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f32f4053ff0 con 0x7f330806b0f0 2024-04-26T01:44:06.253 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710769 got 300647710772 for osd.9 2024-04-26T01:44:06.253 DEBUG:teuthology.parallel:result is None 2024-04-26T01:44:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.284+0000 7f330ea00700 1 -- 172.21.15.6:0/966261249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f330819c790 con 0x7f330806b0f0 2024-04-26T01:44:06.281 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:06 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:44:06.281 INFO:teuthology.orchestra.run.smithi006.stdout:266287972412 2024-04-26T01:44:06.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.284+0000 7f3304c00700 1 -- 172.21.15.6:0/966261249 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f32f409d250 con 0x7f330806b0f0 2024-04-26T01:44:06.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.287+0000 7f330ea00700 1 -- 172.21.15.6:0/966261249 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f32ec0a3d50 msgr2=0x7f32ec0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.287+0000 7f330ea00700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f32ec0a3d50 0x7f32ec0a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f32f8009be0 tx=0x7f32f8009260 comp rx=0 tx=0).stop 2024-04-26T01:44:06.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.287+0000 7f330ea00700 1 -- 172.21.15.6:0/966261249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f330806b0f0 msgr2=0x7f330819b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.287+0000 7f330ea00700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f330806b0f0 0x7f330819b970 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f32f400e9c0 tx=0x7f32f400ecd0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.287+0000 7f330ea00700 1 -- 172.21.15.6:0/966261249 shutdown_connections 2024-04-26T01:44:06.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.287+0000 7f330ea00700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f32ec0a3d50 0x7f32ec0a6210 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.287+0000 7f330ea00700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3308108a30 0x7f330819fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.287+0000 7f330ea00700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f330806a750 0x7f330819b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.287+0000 7f330ea00700 1 --2- 172.21.15.6:0/966261249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f330806b0f0 0x7f330819b970 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.287+0000 7f330ea00700 1 -- 172.21.15.6:0/966261249 >> 172.21.15.6:0/966261249 conn(0x7f33080788f0 msgr2=0x7f330810df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:06.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.287+0000 7f330ea00700 1 -- 172.21.15.6:0/966261249 shutdown_connections 2024-04-26T01:44:06.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.288+0000 7f330ea00700 1 -- 172.21.15.6:0/966261249 wait complete. 2024-04-26T01:44:06.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:06 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:44:06.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:06 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:44:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.557+0000 7f516fc00700 1 -- 172.21.15.6:0/4071323765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f516806b980 msgr2=0x7f516810f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.557+0000 7f516fc00700 1 --2- 172.21.15.6:0/4071323765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f516806b980 0x7f516810f8f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f516000b0a0 tx=0x7f516000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.557+0000 7f516fc00700 1 -- 172.21.15.6:0/4071323765 shutdown_connections 2024-04-26T01:44:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.557+0000 7f516fc00700 1 --2- 172.21.15.6:0/4071323765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f516806b980 0x7f516810f8f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.557+0000 7f516fc00700 1 --2- 172.21.15.6:0/4071323765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5168108a20 0x7f5168108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.557+0000 7f516fc00700 1 --2- 172.21.15.6:0/4071323765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516806afe0 0x7f516806b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.557+0000 7f516fc00700 1 -- 172.21.15.6:0/4071323765 >> 172.21.15.6:0/4071323765 conn(0x7f51680788b0 msgr2=0x7f5168078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.557+0000 7f516fc00700 1 -- 172.21.15.6:0/4071323765 shutdown_connections 2024-04-26T01:44:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.558+0000 7f516fc00700 1 -- 172.21.15.6:0/4071323765 wait complete. 2024-04-26T01:44:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.559+0000 7f516fc00700 1 Processor -- start 2024-04-26T01:44:06.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.559+0000 7f516fc00700 1 -- start start 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.560+0000 7f516fc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f516806afe0 0x7f51681990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.560+0000 7f516fc00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516806b980 0x7f5168199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.560+0000 7f516fc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5168108a20 0x7f516819d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.560+0000 7f516fc00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5168112ba0 con 0x7f5168108a20 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.560+0000 7f516fc00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5168112a20 con 0x7f516806b980 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.560+0000 7f516fc00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5168112d20 con 0x7f516806afe0 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.560+0000 7f516d200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f516806afe0 0x7f51681990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.560+0000 7f516dc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5168108a20 0x7f516819d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.560+0000 7f516d200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f516806afe0 0x7f51681990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47516/0 (socket says 172.21.15.6:47516) 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.560+0000 7f516d200700 1 -- 172.21.15.6:0/313225263 learned_addr learned my addr 172.21.15.6:0/313225263 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.560+0000 7f5167e00700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516806b980 0x7f5168199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.561+0000 7f516d200700 1 -- 172.21.15.6:0/313225263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516806b980 msgr2=0x7f5168199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.561+0000 7f516d200700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516806b980 0x7f5168199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.561+0000 7f516d200700 1 -- 172.21.15.6:0/313225263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5168108a20 msgr2=0x7f516819d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.561+0000 7f516d200700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5168108a20 0x7f516819d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.561+0000 7f516d200700 1 -- 172.21.15.6:0/313225263 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516819e0b0 con 0x7f516806afe0 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.561+0000 7f516dc00700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5168108a20 0x7f516819d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.561+0000 7f516d200700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f516806afe0 0x7f51681990f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f515c00e9a0 tx=0x7f515c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.561+0000 7f5165600700 1 -- 172.21.15.6:0/313225263 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515c00ca10 con 0x7f516806afe0 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.561+0000 7f516fc00700 1 -- 172.21.15.6:0/313225263 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f516819e3a0 con 0x7f516806afe0 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.561+0000 7f5165600700 1 -- 172.21.15.6:0/313225263 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f515c00cbb0 con 0x7f516806afe0 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.561+0000 7f5165600700 1 -- 172.21.15.6:0/313225263 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515c018680 con 0x7f516806afe0 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.561+0000 7f516fc00700 1 -- 172.21.15.6:0/313225263 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5168101930 con 0x7f516806afe0 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.562+0000 7f516fc00700 1 -- 172.21.15.6:0/313225263 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5168069730 con 0x7f516806afe0 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.564+0000 7f5165600700 1 -- 172.21.15.6:0/313225263 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f515c010430 con 0x7f516806afe0 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.566+0000 7f5165600700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f51500a3d50 0x7f51500a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.566+0000 7f5165600700 1 -- 172.21.15.6:0/313225263 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f515c0d06c0 con 0x7f516806afe0 2024-04-26T01:44:06.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.566+0000 7f5167e00700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f51500a3d50 0x7f51500a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.566+0000 7f5167e00700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f51500a3d50 0x7f51500a6210 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5168078f20 tx=0x7f5154009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:06.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.570+0000 7f5165600700 1 -- 172.21.15.6:0/313225263 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f515c09cea0 con 0x7f516806afe0 2024-04-26T01:44:06.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.801+0000 7f516fc00700 1 -- 172.21.15.6:0/313225263 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f516802df80 con 0x7f516806afe0 2024-04-26T01:44:06.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.801+0000 7f5165600700 1 -- 172.21.15.6:0/313225263 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f515c09c7a0 con 0x7f516806afe0 2024-04-26T01:44:06.801 INFO:teuthology.orchestra.run.smithi006.stdout:206158430277 2024-04-26T01:44:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.804+0000 7f516fc00700 1 -- 172.21.15.6:0/313225263 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f51500a3d50 msgr2=0x7f51500a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.804+0000 7f516fc00700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f51500a3d50 0x7f51500a6210 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5168078f20 tx=0x7f5154009260 comp rx=0 tx=0).stop 2024-04-26T01:44:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.804+0000 7f516fc00700 1 -- 172.21.15.6:0/313225263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f516806afe0 msgr2=0x7f51681990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.805+0000 7f516fc00700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f516806afe0 0x7f51681990f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f515c00e9a0 tx=0x7f515c00ecb0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.805+0000 7f516fc00700 1 -- 172.21.15.6:0/313225263 shutdown_connections 2024-04-26T01:44:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.805+0000 7f516fc00700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5168108a20 0x7f516819d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.805+0000 7f516fc00700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f51500a3d50 0x7f51500a6210 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.805+0000 7f516fc00700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f516806afe0 0x7f51681990f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.805+0000 7f516fc00700 1 --2- 172.21.15.6:0/313225263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516806b980 0x7f5168199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.805+0000 7f516fc00700 1 -- 172.21.15.6:0/313225263 >> 172.21.15.6:0/313225263 conn(0x7f51680788b0 msgr2=0x7f51680ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:06.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.805+0000 7f516fc00700 1 -- 172.21.15.6:0/313225263 shutdown_connections 2024-04-26T01:44:06.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:06.805+0000 7f516fc00700 1 -- 172.21.15.6:0/313225263 wait complete. 2024-04-26T01:44:07.265 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:07 smithi006 ceph-mon[28419]: pgmap v394: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:07.265 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:07 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/966261249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T01:44:07.265 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:07 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/313225263' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T01:44:07.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:07 smithi188 ceph-mon[32751]: pgmap v394: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:07.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:07 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/966261249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T01:44:07.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:07 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/313225263' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T01:44:07.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:07 smithi029 ceph-mon[32775]: pgmap v394: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:07.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:07 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/966261249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T01:44:07.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:07 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/313225263' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T01:44:07.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.553+0000 7fc276600700 1 -- 172.21.15.6:0/2971567225 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc270075590 msgr2=0x7fc2701136d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.553+0000 7fc276600700 1 --2- 172.21.15.6:0/2971567225 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc270075590 0x7fc2701136d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009990 tx=0x7fc25c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.553+0000 7fc276600700 1 -- 172.21.15.6:0/2971567225 shutdown_connections 2024-04-26T01:44:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.553+0000 7fc276600700 1 --2- 172.21.15.6:0/2971567225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc270113c10 0x7fc270116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.553+0000 7fc276600700 1 --2- 172.21.15.6:0/2971567225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc270074c70 0x7fc270075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.553+0000 7fc276600700 1 --2- 172.21.15.6:0/2971567225 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc270075590 0x7fc2701136d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.553+0000 7fc276600700 1 -- 172.21.15.6:0/2971567225 >> 172.21.15.6:0/2971567225 conn(0x7fc270079bf0 msgr2=0x7fc27007a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.553+0000 7fc276600700 1 -- 172.21.15.6:0/2971567225 shutdown_connections 2024-04-26T01:44:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.553+0000 7fc276600700 1 -- 172.21.15.6:0/2971567225 wait complete. 2024-04-26T01:44:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.554+0000 7fc276600700 1 Processor -- start 2024-04-26T01:44:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.555+0000 7fc276600700 1 -- start start 2024-04-26T01:44:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.555+0000 7fc276600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc270074c70 0x7fc27019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.555+0000 7fc276600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc270075590 0x7fc27019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.555+0000 7fc276600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc270113c10 0x7fc2701a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.555+0000 7fc276600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc270119200 con 0x7fc270075590 2024-04-26T01:44:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.555+0000 7fc276600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc270119080 con 0x7fc270113c10 2024-04-26T01:44:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.555+0000 7fc276600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc270119380 con 0x7fc270074c70 2024-04-26T01:44:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.555+0000 7fc26f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc270074c70 0x7fc27019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.555+0000 7fc26f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc270074c70 0x7fc27019f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47526/0 (socket says 172.21.15.6:47526) 2024-04-26T01:44:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.555+0000 7fc26f400700 1 -- 172.21.15.6:0/388884693 learned_addr learned my addr 172.21.15.6:0/388884693 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.556+0000 7fc26ea00700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc270075590 0x7fc27019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.556+0000 7fc26fe00700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc270113c10 0x7fc2701a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.556+0000 7fc26f400700 1 -- 172.21.15.6:0/388884693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc270113c10 msgr2=0x7fc2701a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.556+0000 7fc26f400700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc270113c10 0x7fc2701a3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.556+0000 7fc26f400700 1 -- 172.21.15.6:0/388884693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc270075590 msgr2=0x7fc27019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.556+0000 7fc26f400700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc270075590 0x7fc27019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.556+0000 7fc26f400700 1 -- 172.21.15.6:0/388884693 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2701a46e0 con 0x7fc270074c70 2024-04-26T01:44:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.556+0000 7fc26fe00700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc270113c10 0x7fc2701a3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.556+0000 7fc26f400700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc270074c70 0x7fc27019f6d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc26000e9a0 tx=0x7fc26000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.556+0000 7fc267e00700 1 -- 172.21.15.6:0/388884693 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26000cbd0 con 0x7fc270074c70 2024-04-26T01:44:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.556+0000 7fc267e00700 1 -- 172.21.15.6:0/388884693 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc26000cd70 con 0x7fc270074c70 2024-04-26T01:44:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.556+0000 7fc276600700 1 -- 172.21.15.6:0/388884693 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2701a49d0 con 0x7fc270074c70 2024-04-26T01:44:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.557+0000 7fc267e00700 1 -- 172.21.15.6:0/388884693 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc260018680 con 0x7fc270074c70 2024-04-26T01:44:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.557+0000 7fc276600700 1 -- 172.21.15.6:0/388884693 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2701ac250 con 0x7fc270074c70 2024-04-26T01:44:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.558+0000 7fc276600700 1 -- 172.21.15.6:0/388884693 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc270069730 con 0x7fc270074c70 2024-04-26T01:44:07.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.561+0000 7fc267e00700 1 -- 172.21.15.6:0/388884693 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fc260010970 con 0x7fc270074c70 2024-04-26T01:44:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.562+0000 7fc267e00700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc2540a3d50 0x7fc2540a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.562+0000 7fc267e00700 1 -- 172.21.15.6:0/388884693 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fc2600d08f0 con 0x7fc270074c70 2024-04-26T01:44:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.562+0000 7fc26ea00700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc2540a3d50 0x7fc2540a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:07.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.563+0000 7fc26ea00700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc2540a3d50 0x7fc2540a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc25c006010 tx=0x7fc25c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:07.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.571+0000 7fc267e00700 1 -- 172.21.15.6:0/388884693 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc26009d0d0 con 0x7fc270074c70 2024-04-26T01:44:07.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.818+0000 7fc276600700 1 -- 172.21.15.6:0/388884693 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fc2701a0a30 con 0x7fc270074c70 2024-04-26T01:44:07.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.819+0000 7fc267e00700 1 -- 172.21.15.6:0/388884693 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc26009c9d0 con 0x7fc270074c70 2024-04-26T01:44:07.816 INFO:teuthology.orchestra.run.smithi006.stdout:111669149781 2024-04-26T01:44:07.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.822+0000 7fc276600700 1 -- 172.21.15.6:0/388884693 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc2540a3d50 msgr2=0x7fc2540a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:07.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.822+0000 7fc276600700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc2540a3d50 0x7fc2540a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc25c006010 tx=0x7fc25c01a040 comp rx=0 tx=0).stop 2024-04-26T01:44:07.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.822+0000 7fc276600700 1 -- 172.21.15.6:0/388884693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc270074c70 msgr2=0x7fc27019f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:07.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.822+0000 7fc276600700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc270074c70 0x7fc27019f6d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc26000e9a0 tx=0x7fc26000ed60 comp rx=0 tx=0).stop 2024-04-26T01:44:07.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.822+0000 7fc276600700 1 -- 172.21.15.6:0/388884693 shutdown_connections 2024-04-26T01:44:07.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.822+0000 7fc276600700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc270113c10 0x7fc2701a3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.822+0000 7fc276600700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc2540a3d50 0x7fc2540a6210 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.822+0000 7fc276600700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc270074c70 0x7fc27019f6d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.822+0000 7fc276600700 1 --2- 172.21.15.6:0/388884693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc270075590 0x7fc27019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.822+0000 7fc276600700 1 -- 172.21.15.6:0/388884693 >> 172.21.15.6:0/388884693 conn(0x7fc270079bf0 msgr2=0x7fc270112ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:07.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.822+0000 7fc276600700 1 -- 172.21.15.6:0/388884693 shutdown_connections 2024-04-26T01:44:07.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:07.822+0000 7fc276600700 1 -- 172.21.15.6:0/388884693 wait complete. 2024-04-26T01:44:08.550 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:08 smithi006 ceph-mon[28419]: pgmap v395: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:08.550 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.550 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.551 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.551 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.551 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:08 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/388884693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T01:44:08.551 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.551 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:08 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:08 smithi188 ceph-mon[32751]: pgmap v395: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:08 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/388884693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T01:44:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:08 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:08 smithi029 ceph-mon[32775]: pgmap v395: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:08 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/388884693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T01:44:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:08 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:09.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.158+0000 7f8a76000700 1 -- 172.21.15.6:0/2322273548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a70109c80 msgr2=0x7f8a7010dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:09.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.158+0000 7f8a76000700 1 --2- 172.21.15.6:0/2322273548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a70109c80 0x7f8a7010dcd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0099c0 tx=0x7f8a5c009cd0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.158+0000 7f8a76000700 1 -- 172.21.15.6:0/2322273548 shutdown_connections 2024-04-26T01:44:09.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.158+0000 7f8a76000700 1 --2- 172.21.15.6:0/2322273548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a7010e3b0 0x7f8a70115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.158+0000 7f8a76000700 1 --2- 172.21.15.6:0/2322273548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a701092d0 0x7f8a701096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.158+0000 7f8a76000700 1 --2- 172.21.15.6:0/2322273548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a70109c80 0x7f8a7010dcd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.158+0000 7f8a76000700 1 -- 172.21.15.6:0/2322273548 >> 172.21.15.6:0/2322273548 conn(0x7f8a70076210 msgr2=0x7f8a70107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:09.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.158+0000 7f8a76000700 1 -- 172.21.15.6:0/2322273548 shutdown_connections 2024-04-26T01:44:09.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.159+0000 7f8a76000700 1 -- 172.21.15.6:0/2322273548 wait complete. 2024-04-26T01:44:09.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.160+0000 7f8a76000700 1 Processor -- start 2024-04-26T01:44:09.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.160+0000 7f8a76000700 1 -- start start 2024-04-26T01:44:09.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a76000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a701092d0 0x7f8a7019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:09.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a76000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a70109c80 0x7f8a7019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:09.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a76000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a7010e3b0 0x7f8a701a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:09.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a76000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8a70119220 con 0x7f8a70109c80 2024-04-26T01:44:09.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a76000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8a701190a0 con 0x7f8a701092d0 2024-04-26T01:44:09.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a76000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8a701193a0 con 0x7f8a7010e3b0 2024-04-26T01:44:09.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a6ea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a70109c80 0x7f8a7019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:09.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a6f400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a701092d0 0x7f8a7019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:09.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a6f400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a701092d0 0x7f8a7019f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36368/0 (socket says 172.21.15.6:36368) 2024-04-26T01:44:09.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a6f400700 1 -- 172.21.15.6:0/2517507346 learned_addr learned my addr 172.21.15.6:0/2517507346 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:09.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a6fe00700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a7010e3b0 0x7f8a701a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a6f400700 1 -- 172.21.15.6:0/2517507346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a7010e3b0 msgr2=0x7f8a701a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a6f400700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a7010e3b0 0x7f8a701a4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a6f400700 1 -- 172.21.15.6:0/2517507346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a70109c80 msgr2=0x7f8a7019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a6f400700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a70109c80 0x7f8a7019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.161+0000 7f8a6f400700 1 -- 172.21.15.6:0/2517507346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a701a4760 con 0x7f8a701092d0 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.162+0000 7f8a6f400700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a701092d0 0x7f8a7019f750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000eb30 tx=0x7f8a6000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.162+0000 7f8a67e00700 1 -- 172.21.15.6:0/2517507346 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6000cbd0 con 0x7f8a701092d0 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.162+0000 7f8a67e00700 1 -- 172.21.15.6:0/2517507346 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a600044d0 con 0x7f8a701092d0 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.162+0000 7f8a67e00700 1 -- 172.21.15.6:0/2517507346 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60003e30 con 0x7f8a701092d0 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.162+0000 7f8a76000700 1 -- 172.21.15.6:0/2517507346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a701a4a50 con 0x7f8a701092d0 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.162+0000 7f8a76000700 1 -- 172.21.15.6:0/2517507346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a701ac2d0 con 0x7f8a701092d0 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.164+0000 7f8a76000700 1 -- 172.21.15.6:0/2517507346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a70069730 con 0x7f8a701092d0 2024-04-26T01:44:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.166+0000 7f8a67e00700 1 -- 172.21.15.6:0/2517507346 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f8a60004000 con 0x7f8a701092d0 2024-04-26T01:44:09.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.167+0000 7f8a67e00700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f8a540a3d50 0x7f8a540a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:09.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.167+0000 7f8a67e00700 1 -- 172.21.15.6:0/2517507346 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f8a600d0ac0 con 0x7f8a701092d0 2024-04-26T01:44:09.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.167+0000 7f8a6ea00700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f8a540a3d50 0x7f8a540a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:09.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.168+0000 7f8a6ea00700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f8a540a3d50 0x7f8a540a6210 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c000c00 tx=0x7f8a5c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:09.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.169+0000 7f8a67e00700 1 -- 172.21.15.6:0/2517507346 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a600042f0 con 0x7f8a701092d0 2024-04-26T01:44:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.406+0000 7f8a76000700 1 -- 172.21.15.6:0/2517507346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f8a701a0ae0 con 0x7f8a701092d0 2024-04-26T01:44:09.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.406+0000 7f8a67e00700 1 -- 172.21.15.6:0/2517507346 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8a701a0ae0 con 0x7f8a701092d0 2024-04-26T01:44:09.403 INFO:teuthology.orchestra.run.smithi006.stdout:227633266753 2024-04-26T01:44:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.409+0000 7f8a76000700 1 -- 172.21.15.6:0/2517507346 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f8a540a3d50 msgr2=0x7f8a540a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.409+0000 7f8a76000700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f8a540a3d50 0x7f8a540a6210 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c000c00 tx=0x7f8a5c005fb0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.409+0000 7f8a76000700 1 -- 172.21.15.6:0/2517507346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a701092d0 msgr2=0x7f8a7019f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.409+0000 7f8a76000700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a701092d0 0x7f8a7019f750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000eb30 tx=0x7f8a6000ee40 comp rx=0 tx=0).stop 2024-04-26T01:44:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.409+0000 7f8a76000700 1 -- 172.21.15.6:0/2517507346 shutdown_connections 2024-04-26T01:44:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.409+0000 7f8a76000700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f8a540a3d50 0x7f8a540a6210 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.409+0000 7f8a76000700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a7010e3b0 0x7f8a701a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.409+0000 7f8a76000700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a701092d0 0x7f8a7019f750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.409+0000 7f8a76000700 1 --2- 172.21.15.6:0/2517507346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a70109c80 0x7f8a7019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.409+0000 7f8a76000700 1 -- 172.21.15.6:0/2517507346 >> 172.21.15.6:0/2517507346 conn(0x7f8a70076210 msgr2=0x7f8a700768d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.410+0000 7f8a76000700 1 -- 172.21.15.6:0/2517507346 shutdown_connections 2024-04-26T01:44:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.410+0000 7f8a76000700 1 -- 172.21.15.6:0/2517507346 wait complete. 2024-04-26T01:44:09.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.862+0000 7f014ac00700 1 -- 172.21.15.6:0/1641773490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01441016c0 msgr2=0x7f0144101aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:09.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.862+0000 7f014ac00700 1 --2- 172.21.15.6:0/1641773490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01441016c0 0x7f0144101aa0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0138009990 tx=0x7f0138009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.862+0000 7f014ac00700 1 -- 172.21.15.6:0/1641773490 shutdown_connections 2024-04-26T01:44:09.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.862+0000 7f014ac00700 1 --2- 172.21.15.6:0/1641773490 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f014410f270 0x7f0144111660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.862+0000 7f014ac00700 1 --2- 172.21.15.6:0/1641773490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01441016c0 0x7f0144101aa0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.862+0000 7f014ac00700 1 --2- 172.21.15.6:0/1641773490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0144102070 0x7f014410ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.862+0000 7f014ac00700 1 -- 172.21.15.6:0/1641773490 >> 172.21.15.6:0/1641773490 conn(0x7f01440fd360 msgr2=0x7f01440ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:09.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.862+0000 7f014ac00700 1 -- 172.21.15.6:0/1641773490 shutdown_connections 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.862+0000 7f014ac00700 1 -- 172.21.15.6:0/1641773490 wait complete. 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.864+0000 7f014ac00700 1 Processor -- start 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.864+0000 7f014ac00700 1 -- start start 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.864+0000 7f014ac00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01441016c0 0x7f014419b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.864+0000 7f014ac00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0144102070 0x7f014419b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.864+0000 7f014ac00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f014410f270 0x7f014419fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.864+0000 7f014ac00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0144114dc0 con 0x7f01441016c0 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.864+0000 7f014ac00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0144114c40 con 0x7f014410f270 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.864+0000 7f014ac00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0144114f40 con 0x7f0144102070 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.864+0000 7f0149800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01441016c0 0x7f014419b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.865+0000 7f0149800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01441016c0 0x7f014419b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45372/0 (socket says 172.21.15.6:45372) 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.865+0000 7f0149800700 1 -- 172.21.15.6:0/2848982685 learned_addr learned my addr 172.21.15.6:0/2848982685 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.865+0000 7f014a200700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f014410f270 0x7f014419fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.865+0000 7f0148e00700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0144102070 0x7f014419b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:09.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.865+0000 7f0149800700 1 -- 172.21.15.6:0/2848982685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0144102070 msgr2=0x7f014419b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.865+0000 7f0149800700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0144102070 0x7f014419b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.865+0000 7f0149800700 1 -- 172.21.15.6:0/2848982685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f014410f270 msgr2=0x7f014419fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.865+0000 7f0149800700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f014410f270 0x7f014419fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.865+0000 7f0149800700 1 -- 172.21.15.6:0/2848982685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01441a0260 con 0x7f01441016c0 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.865+0000 7f0149800700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01441016c0 0x7f014419b250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0138005910 tx=0x7f013800b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.866+0000 7f0142000700 1 -- 172.21.15.6:0/2848982685 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138027070 con 0x7f01441016c0 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.866+0000 7f0142000700 1 -- 172.21.15.6:0/2848982685 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0138022470 con 0x7f01441016c0 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.866+0000 7f014ac00700 1 -- 172.21.15.6:0/2848982685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01441a04f0 con 0x7f01441016c0 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.866+0000 7f0142000700 1 -- 172.21.15.6:0/2848982685 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138017650 con 0x7f01441016c0 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.866+0000 7f014ac00700 1 -- 172.21.15.6:0/2848982685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01441a7dd0 con 0x7f01441016c0 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.867+0000 7f014ac00700 1 -- 172.21.15.6:0/2848982685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0144069730 con 0x7f01441016c0 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.871+0000 7f0142000700 1 -- 172.21.15.6:0/2848982685 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f013802a070 con 0x7f01441016c0 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.872+0000 7f0142000700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f012c0a3cd0 0x7f012c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.872+0000 7f0142000700 1 -- 172.21.15.6:0/2848982685 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f01380d15e0 con 0x7f01441016c0 2024-04-26T01:44:09.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.872+0000 7f0148e00700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f012c0a3cd0 0x7f012c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:09.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.873+0000 7f0148e00700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f012c0a3cd0 0x7f012c0a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0144103060 tx=0x7f0130009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:09.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.875+0000 7f0142000700 1 -- 172.21.15.6:0/2848982685 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0138013070 con 0x7f01441016c0 2024-04-26T01:44:09.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.972+0000 7f34d8e00700 1 -- 172.21.15.6:0/2049381436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d4101940 msgr2=0x7f34d4101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:09.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.972+0000 7f34d8e00700 1 --2- 172.21.15.6:0/2049381436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d4101940 0x7f34d4101dc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009990 tx=0x7f34c0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.972+0000 7f34d8e00700 1 -- 172.21.15.6:0/2049381436 shutdown_connections 2024-04-26T01:44:09.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.972+0000 7f34d8e00700 1 --2- 172.21.15.6:0/2049381436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34d4102300 0x7f34d410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.972+0000 7f34d8e00700 1 --2- 172.21.15.6:0/2049381436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34d4107990 0x7f34d4107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.972+0000 7f34d8e00700 1 --2- 172.21.15.6:0/2049381436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d4101940 0x7f34d4101dc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.972+0000 7f34d8e00700 1 -- 172.21.15.6:0/2049381436 >> 172.21.15.6:0/2049381436 conn(0x7f34d40788f0 msgr2=0x7f34d4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:09.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.972+0000 7f34d8e00700 1 -- 172.21.15.6:0/2049381436 shutdown_connections 2024-04-26T01:44:09.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.972+0000 7f34d8e00700 1 -- 172.21.15.6:0/2049381436 wait complete. 2024-04-26T01:44:09.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.974+0000 7f34d8e00700 1 Processor -- start 2024-04-26T01:44:09.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.974+0000 7f34d8e00700 1 -- start start 2024-04-26T01:44:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.974+0000 7f34d8e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d4101940 0x7f34d4199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.974+0000 7f34d8e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34d4102300 0x7f34d41996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:09.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.974+0000 7f34d8e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34d4107990 0x7f34d419da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:09.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.974+0000 7f34d8e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f34d4111b30 con 0x7f34d4107990 2024-04-26T01:44:09.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.974+0000 7f34d8e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f34d41119b0 con 0x7f34d4102300 2024-04-26T01:44:09.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.974+0000 7f34d8e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f34d4111cb0 con 0x7f34d4101940 2024-04-26T01:44:09.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.975+0000 7f34d1e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d4101940 0x7f34d4199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:09.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.975+0000 7f34d1e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d4101940 0x7f34d4199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59048/0 (socket says 172.21.15.6:59048) 2024-04-26T01:44:09.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.975+0000 7f34d1e00700 1 -- 172.21.15.6:0/2109868113 learned_addr learned my addr 172.21.15.6:0/2109868113 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:09.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.975+0000 7f34d1400700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34d4102300 0x7f34d41996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:09.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.975+0000 7f34d2800700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34d4107990 0x7f34d419da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.975+0000 7f34d1e00700 1 -- 172.21.15.6:0/2109868113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34d4102300 msgr2=0x7f34d41996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.975+0000 7f34d1e00700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34d4102300 0x7f34d41996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.975+0000 7f34d1e00700 1 -- 172.21.15.6:0/2109868113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34d4107990 msgr2=0x7f34d419da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.975+0000 7f34d1e00700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34d4107990 0x7f34d419da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:09.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.975+0000 7f34d1e00700 1 -- 172.21.15.6:0/2109868113 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d419e140 con 0x7f34d4101940 2024-04-26T01:44:09.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.975+0000 7f34d2800700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34d4107990 0x7f34d419da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:09.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.975+0000 7f34d1400700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34d4102300 0x7f34d41996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:09.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.976+0000 7f34d1e00700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d4101940 0x7f34d4199180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f34c400e9a0 tx=0x7f34c400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:09.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.976+0000 7f34caa00700 1 -- 172.21.15.6:0/2109868113 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c400cbd0 con 0x7f34d4101940 2024-04-26T01:44:09.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.976+0000 7f34caa00700 1 -- 172.21.15.6:0/2109868113 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34c400cd70 con 0x7f34d4101940 2024-04-26T01:44:09.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.976+0000 7f34d8e00700 1 -- 172.21.15.6:0/2109868113 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d419e430 con 0x7f34d4101940 2024-04-26T01:44:09.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.976+0000 7f34caa00700 1 -- 172.21.15.6:0/2109868113 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c4018680 con 0x7f34d4101940 2024-04-26T01:44:09.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.976+0000 7f34d8e00700 1 -- 172.21.15.6:0/2109868113 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d41a5cb0 con 0x7f34d4101940 2024-04-26T01:44:09.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.977+0000 7f34d8e00700 1 -- 172.21.15.6:0/2109868113 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d4069730 con 0x7f34d4101940 2024-04-26T01:44:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.982+0000 7f34caa00700 1 -- 172.21.15.6:0/2109868113 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f34c4010430 con 0x7f34d4101940 2024-04-26T01:44:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.983+0000 7f34caa00700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f34b80a3d50 0x7f34b80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.984+0000 7f34caa00700 1 -- 172.21.15.6:0/2109868113 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f34c40d0c00 con 0x7f34d4101940 2024-04-26T01:44:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.984+0000 7f34d1400700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f34b80a3d50 0x7f34b80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:09.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.984+0000 7f34caa00700 1 -- 172.21.15.6:0/2109868113 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34c40ff070 con 0x7f34d4101940 2024-04-26T01:44:09.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:09.985+0000 7f34d1400700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f34b80a3d50 0x7f34b80a6210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f34d40790a0 tx=0x7f34c000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:10.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.127+0000 7f014ac00700 1 -- 172.21.15.6:0/2848982685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f014402df80 con 0x7f01441016c0 2024-04-26T01:44:10.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.128+0000 7f0142000700 1 -- 172.21.15.6:0/2848982685 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f013809de50 con 0x7f01441016c0 2024-04-26T01:44:10.126 INFO:teuthology.orchestra.run.smithi006.stdout:146028888146 2024-04-26T01:44:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.131+0000 7f014ac00700 1 -- 172.21.15.6:0/2848982685 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f012c0a3cd0 msgr2=0x7f012c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.131+0000 7f014ac00700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f012c0a3cd0 0x7f012c0a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0144103060 tx=0x7f0130009260 comp rx=0 tx=0).stop 2024-04-26T01:44:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.131+0000 7f014ac00700 1 -- 172.21.15.6:0/2848982685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01441016c0 msgr2=0x7f014419b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.131+0000 7f014ac00700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01441016c0 0x7f014419b250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0138005910 tx=0x7f013800b670 comp rx=0 tx=0).stop 2024-04-26T01:44:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.131+0000 7f014ac00700 1 -- 172.21.15.6:0/2848982685 shutdown_connections 2024-04-26T01:44:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.131+0000 7f014ac00700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f014410f270 0x7f014419fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.131+0000 7f014ac00700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01441016c0 0x7f014419b250 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.131+0000 7f014ac00700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f012c0a3cd0 0x7f012c0a6190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.131+0000 7f014ac00700 1 --2- 172.21.15.6:0/2848982685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0144102070 0x7f014419b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.131+0000 7f014ac00700 1 -- 172.21.15.6:0/2848982685 >> 172.21.15.6:0/2848982685 conn(0x7f01440fd360 msgr2=0x7f01440ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:10.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.131+0000 7f014ac00700 1 -- 172.21.15.6:0/2848982685 shutdown_connections 2024-04-26T01:44:10.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.132+0000 7f014ac00700 1 -- 172.21.15.6:0/2848982685 wait complete. 2024-04-26T01:44:10.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.267+0000 7f34d8e00700 1 -- 172.21.15.6:0/2109868113 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f34d419a490 con 0x7f34d4101940 2024-04-26T01:44:10.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.268+0000 7f34caa00700 1 -- 172.21.15.6:0/2109868113 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f34c409d3e0 con 0x7f34d4101940 2024-04-26T01:44:10.266 INFO:teuthology.orchestra.run.smithi006.stdout:77309411421 2024-04-26T01:44:10.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.271+0000 7f34d8e00700 1 -- 172.21.15.6:0/2109868113 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f34b80a3d50 msgr2=0x7f34b80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:10.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.271+0000 7f34d8e00700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f34b80a3d50 0x7f34b80a6210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f34d40790a0 tx=0x7f34c000b260 comp rx=0 tx=0).stop 2024-04-26T01:44:10.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.271+0000 7f34d8e00700 1 -- 172.21.15.6:0/2109868113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d4101940 msgr2=0x7f34d4199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:10.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.271+0000 7f34d8e00700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d4101940 0x7f34d4199180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f34c400e9a0 tx=0x7f34c400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.271+0000 7f34d8e00700 1 -- 172.21.15.6:0/2109868113 shutdown_connections 2024-04-26T01:44:10.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.271+0000 7f34d8e00700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34d4107990 0x7f34d419da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.271+0000 7f34d8e00700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f34b80a3d50 0x7f34b80a6210 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.271+0000 7f34d8e00700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d4101940 0x7f34d4199180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.271+0000 7f34d8e00700 1 --2- 172.21.15.6:0/2109868113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34d4102300 0x7f34d41996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.271+0000 7f34d8e00700 1 -- 172.21.15.6:0/2109868113 >> 172.21.15.6:0/2109868113 conn(0x7f34d40788f0 msgr2=0x7f34d406ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:10.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.271+0000 7f34d8e00700 1 -- 172.21.15.6:0/2109868113 shutdown_connections 2024-04-26T01:44:10.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:10.271+0000 7f34d8e00700 1 -- 172.21.15.6:0/2109868113 wait complete. 2024-04-26T01:44:10.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:10 smithi006 ceph-mon[28419]: pgmap v396: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:10.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:10 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2517507346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T01:44:10.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:10 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2848982685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T01:44:10.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:10 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2109868113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T01:44:10.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:10 smithi188 ceph-mon[32751]: pgmap v396: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:10.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:10 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2517507346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T01:44:10.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:10 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2848982685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T01:44:10.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:10 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2109868113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T01:44:10.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:10 smithi029 ceph-mon[32775]: pgmap v396: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:10.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:10 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2517507346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T01:44:10.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:10 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2848982685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T01:44:10.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:10 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2109868113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T01:44:10.909 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691911 got 171798691915 for osd.5 2024-04-26T01:44:10.910 DEBUG:teuthology.parallel:result is None 2024-04-26T01:44:11.125 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574941 got 55834574946 for osd.1 2024-04-26T01:44:11.125 DEBUG:teuthology.parallel:result is None 2024-04-26T01:44:11.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.538+0000 7fba4f600700 1 -- 172.21.15.6:0/3604232371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba48074c70 msgr2=0x7fba48075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:11.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.538+0000 7fba4f600700 1 --2- 172.21.15.6:0/3604232371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba48074c70 0x7fba48075050 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009990 tx=0x7fba3c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.538+0000 7fba4f600700 1 -- 172.21.15.6:0/3604232371 shutdown_connections 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.538+0000 7fba4f600700 1 --2- 172.21.15.6:0/3604232371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba4810f440 0x7fba48111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.538+0000 7fba4f600700 1 --2- 172.21.15.6:0/3604232371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba48074c70 0x7fba48075050 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.538+0000 7fba4f600700 1 --2- 172.21.15.6:0/3604232371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba48075590 0x7fba4810ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.538+0000 7fba4f600700 1 -- 172.21.15.6:0/3604232371 >> 172.21.15.6:0/3604232371 conn(0x7fba48079ba0 msgr2=0x7fba48079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.538+0000 7fba4f600700 1 -- 172.21.15.6:0/3604232371 shutdown_connections 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.538+0000 7fba4f600700 1 -- 172.21.15.6:0/3604232371 wait complete. 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.540+0000 7fba4f600700 1 Processor -- start 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.540+0000 7fba4f600700 1 -- start start 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.540+0000 7fba4f600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba48074c70 0x7fba4819b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.540+0000 7fba4f600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba48075590 0x7fba4819b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.540+0000 7fba4f600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba4810f440 0x7fba4819fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.540+0000 7fba4f600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fba48114e70 con 0x7fba48075590 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.540+0000 7fba4f600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fba48114cf0 con 0x7fba4810f440 2024-04-26T01:44:11.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.540+0000 7fba4f600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fba48114ff0 con 0x7fba48074c70 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.541+0000 7fba4cc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba48074c70 0x7fba4819b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.541+0000 7fba4cc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba48074c70 0x7fba4819b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59058/0 (socket says 172.21.15.6:59058) 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.541+0000 7fba4cc00700 1 -- 172.21.15.6:0/1430194654 learned_addr learned my addr 172.21.15.6:0/1430194654 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.541+0000 7fba47e00700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba48075590 0x7fba4819b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.541+0000 7fba4d600700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba4810f440 0x7fba4819fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.541+0000 7fba4cc00700 1 -- 172.21.15.6:0/1430194654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba4810f440 msgr2=0x7fba4819fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.541+0000 7fba4cc00700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba4810f440 0x7fba4819fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.541+0000 7fba4cc00700 1 -- 172.21.15.6:0/1430194654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba48075590 msgr2=0x7fba4819b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.541+0000 7fba4cc00700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba48075590 0x7fba4819b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.541+0000 7fba4cc00700 1 -- 172.21.15.6:0/1430194654 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba481a0380 con 0x7fba48074c70 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.542+0000 7fba4cc00700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba48074c70 0x7fba4819b3c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fba3c005910 tx=0x7fba3c00b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.542+0000 7fba45600700 1 -- 172.21.15.6:0/1430194654 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c027070 con 0x7fba48074c70 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.542+0000 7fba45600700 1 -- 172.21.15.6:0/1430194654 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba3c022470 con 0x7fba48074c70 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.542+0000 7fba45600700 1 -- 172.21.15.6:0/1430194654 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c017630 con 0x7fba48074c70 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.542+0000 7fba4f600700 1 -- 172.21.15.6:0/1430194654 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba481a0610 con 0x7fba48074c70 2024-04-26T01:44:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.542+0000 7fba4f600700 1 -- 172.21.15.6:0/1430194654 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba481a7ef0 con 0x7fba48074c70 2024-04-26T01:44:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.544+0000 7fba4f600700 1 -- 172.21.15.6:0/1430194654 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba48076360 con 0x7fba48074c70 2024-04-26T01:44:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.546+0000 7fba45600700 1 -- 172.21.15.6:0/1430194654 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fba3c02a070 con 0x7fba48074c70 2024-04-26T01:44:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.547+0000 7fba45600700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fba300a3d50 0x7fba300a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.547+0000 7fba45600700 1 -- 172.21.15.6:0/1430194654 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fba3c0d1220 con 0x7fba48074c70 2024-04-26T01:44:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.548+0000 7fba47e00700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fba300a3d50 0x7fba300a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.549+0000 7fba47e00700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fba300a3d50 0x7fba300a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fba481111d0 tx=0x7fba34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.551+0000 7fba45600700 1 -- 172.21.15.6:0/1430194654 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba3c013070 con 0x7fba48074c70 2024-04-26T01:44:11.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.785+0000 7fba4f600700 1 -- 172.21.15.6:0/1430194654 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fba4802df80 con 0x7fba48074c70 2024-04-26T01:44:11.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.786+0000 7fba45600700 1 -- 172.21.15.6:0/1430194654 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fba3c09da90 con 0x7fba48074c70 2024-04-26T01:44:11.784 INFO:teuthology.orchestra.run.smithi006.stdout:356482285612 2024-04-26T01:44:11.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.789+0000 7fba4f600700 1 -- 172.21.15.6:0/1430194654 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fba300a3d50 msgr2=0x7fba300a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:11.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.789+0000 7fba4f600700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fba300a3d50 0x7fba300a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fba481111d0 tx=0x7fba34009260 comp rx=0 tx=0).stop 2024-04-26T01:44:11.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.789+0000 7fba4f600700 1 -- 172.21.15.6:0/1430194654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba48074c70 msgr2=0x7fba4819b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:11.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.789+0000 7fba4f600700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba48074c70 0x7fba4819b3c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fba3c005910 tx=0x7fba3c00b670 comp rx=0 tx=0).stop 2024-04-26T01:44:11.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.789+0000 7fba4f600700 1 -- 172.21.15.6:0/1430194654 shutdown_connections 2024-04-26T01:44:11.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.789+0000 7fba4f600700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba4810f440 0x7fba4819fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.789+0000 7fba4f600700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fba300a3d50 0x7fba300a6210 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.789+0000 7fba4f600700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba48074c70 0x7fba4819b3c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.789+0000 7fba4f600700 1 --2- 172.21.15.6:0/1430194654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba48075590 0x7fba4819b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.789+0000 7fba4f600700 1 -- 172.21.15.6:0/1430194654 >> 172.21.15.6:0/1430194654 conn(0x7fba48079ba0 msgr2=0x7fba480ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:11.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.790+0000 7fba4f600700 1 -- 172.21.15.6:0/1430194654 shutdown_connections 2024-04-26T01:44:11.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:11.790+0000 7fba4f600700 1 -- 172.21.15.6:0/1430194654 wait complete. 2024-04-26T01:44:11.844 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972408 got 266287972412 for osd.8 2024-04-26T01:44:11.845 DEBUG:teuthology.parallel:result is None 2024-04-26T01:44:12.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:12 smithi006 ceph-mon[28419]: pgmap v397: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:12.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:12 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1430194654' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T01:44:12.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:12 smithi188 ceph-mon[32751]: pgmap v397: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:12.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:12 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1430194654' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T01:44:12.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:12 smithi029 ceph-mon[32775]: pgmap v397: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:12.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:12 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1430194654' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T01:44:13.201 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430272 got 206158430277 for osd.6 2024-04-26T01:44:13.201 DEBUG:teuthology.parallel:result is None 2024-04-26T01:44:13.319 INFO:tasks.cephadm.ceph_manager.ceph:need seq 322122547238 got 322122547245 for osd.10 2024-04-26T01:44:13.319 DEBUG:teuthology.parallel:result is None 2024-04-26T01:44:13.938 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411417 got 77309411421 for osd.2 2024-04-26T01:44:13.938 DEBUG:teuthology.parallel:result is None 2024-04-26T01:44:14.253 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266748 got 227633266753 for osd.7 2024-04-26T01:44:14.253 DEBUG:teuthology.parallel:result is None 2024-04-26T01:44:14.477 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:14 smithi006 ceph-mon[28419]: pgmap v398: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:14.543 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888141 got 146028888146 for osd.4 2024-04-26T01:44:14.544 DEBUG:teuthology.parallel:result is None 2024-04-26T01:44:14.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:14 smithi188 ceph-mon[32751]: pgmap v398: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:14.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:14 smithi029 ceph-mon[32775]: pgmap v398: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:15.048 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482285607 got 356482285612 for osd.11 2024-04-26T01:44:15.049 DEBUG:teuthology.parallel:result is None 2024-04-26T01:44:15.297 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:15.374 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149775 got 111669149781 for osd.3 2024-04-26T01:44:15.374 DEBUG:teuthology.parallel:result is None 2024-04-26T01:44:16.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:16 smithi006 ceph-mon[28419]: pgmap v399: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:16.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:16 smithi188 ceph-mon[32751]: pgmap v399: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:16.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:16 smithi029 ceph-mon[32775]: pgmap v399: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:17.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.925+0000 7f1c06400700 1 -- 172.21.15.6:0/2835613552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c00074c70 msgr2=0x7f1c00075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.925+0000 7f1c06400700 1 --2- 172.21.15.6:0/2835613552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c00074c70 0x7f1c00075050 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0009990 tx=0x7f1bf0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.925+0000 7f1c06400700 1 -- 172.21.15.6:0/2835613552 shutdown_connections 2024-04-26T01:44:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.925+0000 7f1c06400700 1 --2- 172.21.15.6:0/2835613552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0010f440 0x7f1c00111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.925+0000 7f1c06400700 1 --2- 172.21.15.6:0/2835613552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c00074c70 0x7f1c00075050 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.925+0000 7f1c06400700 1 --2- 172.21.15.6:0/2835613552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c00075590 0x7f1c0010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.925+0000 7f1c06400700 1 -- 172.21.15.6:0/2835613552 >> 172.21.15.6:0/2835613552 conn(0x7f1c00079ba0 msgr2=0x7f1c00079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.926+0000 7f1c06400700 1 -- 172.21.15.6:0/2835613552 shutdown_connections 2024-04-26T01:44:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.926+0000 7f1c06400700 1 -- 172.21.15.6:0/2835613552 wait complete. 2024-04-26T01:44:17.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.928+0000 7f1c06400700 1 Processor -- start 2024-04-26T01:44:17.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.928+0000 7f1c06400700 1 -- start start 2024-04-26T01:44:17.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.928+0000 7f1c06400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c00074c70 0x7f1c0019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:17.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.928+0000 7f1c06400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c00075590 0x7f1c0019b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:17.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1c06400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0010f440 0x7f1c0019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:17.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1c06400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1c00114de0 con 0x7f1c0010f440 2024-04-26T01:44:17.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1c06400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1c00114c60 con 0x7f1c00075590 2024-04-26T01:44:17.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1c06400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1c00114f60 con 0x7f1c00074c70 2024-04-26T01:44:17.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1bff400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c00074c70 0x7f1c0019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:17.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1bffe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0010f440 0x7f1c0019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:17.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1bff400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c00074c70 0x7f1c0019b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:41772/0 (socket says 172.21.15.6:41772) 2024-04-26T01:44:17.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1bff400700 1 -- 172.21.15.6:0/1196451870 learned_addr learned my addr 172.21.15.6:0/1196451870 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:17.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1bfea00700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c00075590 0x7f1c0019b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:17.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1bff400700 1 -- 172.21.15.6:0/1196451870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c00075590 msgr2=0x7f1c0019b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:17.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1bff400700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c00075590 0x7f1c0019b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:17.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1bff400700 1 -- 172.21.15.6:0/1196451870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0010f440 msgr2=0x7f1c0019fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:17.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1bff400700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0010f440 0x7f1c0019fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:17.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.929+0000 7f1bff400700 1 -- 172.21.15.6:0/1196451870 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c001a0320 con 0x7f1c00074c70 2024-04-26T01:44:17.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.930+0000 7f1bff400700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c00074c70 0x7f1c0019b310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0000c00 tx=0x7f1bf0009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:17.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.930+0000 7f1bf7e00700 1 -- 172.21.15.6:0/1196451870 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0027070 con 0x7f1c00074c70 2024-04-26T01:44:17.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.930+0000 7f1c06400700 1 -- 172.21.15.6:0/1196451870 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c001a05b0 con 0x7f1c00074c70 2024-04-26T01:44:17.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.930+0000 7f1bf7e00700 1 -- 172.21.15.6:0/1196451870 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bf0022470 con 0x7f1c00074c70 2024-04-26T01:44:17.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.930+0000 7f1bf7e00700 1 -- 172.21.15.6:0/1196451870 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0017650 con 0x7f1c00074c70 2024-04-26T01:44:17.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.931+0000 7f1c06400700 1 -- 172.21.15.6:0/1196451870 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c001a7e90 con 0x7f1c00074c70 2024-04-26T01:44:17.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.932+0000 7f1c06400700 1 -- 172.21.15.6:0/1196451870 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c00076360 con 0x7f1c00074c70 2024-04-26T01:44:17.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.934+0000 7f1bf7e00700 1 -- 172.21.15.6:0/1196451870 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f1bf002a070 con 0x7f1c00074c70 2024-04-26T01:44:17.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.936+0000 7f1bf7e00700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1be40a3cd0 0x7f1be40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:17.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.936+0000 7f1bf7e00700 1 -- 172.21.15.6:0/1196451870 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f1bf00d1300 con 0x7f1c00074c70 2024-04-26T01:44:17.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.936+0000 7f1bfea00700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1be40a3cd0 0x7f1be40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:17.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.937+0000 7f1bfea00700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1be40a3cd0 0x7f1be40a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1c001025c0 tx=0x7f1bec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:17.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:17.940+0000 7f1bf7e00700 1 -- 172.21.15.6:0/1196451870 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1bf0013070 con 0x7f1c00074c70 2024-04-26T01:44:18.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.191+0000 7f1c06400700 1 -- 172.21.15.6:0/1196451870 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f1c0002df80 con 0x7f1c00074c70 2024-04-26T01:44:18.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.191+0000 7f1bf7e00700 1 -- 172.21.15.6:0/1196451870 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1bf009db70 con 0x7f1c00074c70 2024-04-26T01:44:18.189 INFO:teuthology.orchestra.run.smithi006.stdout:34359738473 2024-04-26T01:44:18.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.194+0000 7f1c06400700 1 -- 172.21.15.6:0/1196451870 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1be40a3cd0 msgr2=0x7f1be40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:18.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.194+0000 7f1c06400700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1be40a3cd0 0x7f1be40a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1c001025c0 tx=0x7f1bec009260 comp rx=0 tx=0).stop 2024-04-26T01:44:18.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.194+0000 7f1c06400700 1 -- 172.21.15.6:0/1196451870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c00074c70 msgr2=0x7f1c0019b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:18.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.194+0000 7f1c06400700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c00074c70 0x7f1c0019b310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0000c00 tx=0x7f1bf0009580 comp rx=0 tx=0).stop 2024-04-26T01:44:18.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.194+0000 7f1c06400700 1 -- 172.21.15.6:0/1196451870 shutdown_connections 2024-04-26T01:44:18.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.194+0000 7f1c06400700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0010f440 0x7f1c0019fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.194+0000 7f1c06400700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f1be40a3cd0 0x7f1be40a6190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.194+0000 7f1c06400700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c00074c70 0x7f1c0019b310 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.194+0000 7f1c06400700 1 --2- 172.21.15.6:0/1196451870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c00075590 0x7f1c0019b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.194+0000 7f1c06400700 1 -- 172.21.15.6:0/1196451870 >> 172.21.15.6:0/1196451870 conn(0x7f1c00079ba0 msgr2=0x7f1c000ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:18.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.194+0000 7f1c06400700 1 -- 172.21.15.6:0/1196451870 shutdown_connections 2024-04-26T01:44:18.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:18.195+0000 7f1c06400700 1 -- 172.21.15.6:0/1196451870 wait complete. 2024-04-26T01:44:18.554 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:18 smithi006 ceph-mon[28419]: pgmap v400: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:18.554 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:18 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1196451870' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T01:44:18.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:18 smithi188 ceph-mon[32751]: pgmap v400: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:18.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:18 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1196451870' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T01:44:18.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:18 smithi029 ceph-mon[32775]: pgmap v400: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:18.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:18 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1196451870' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T01:44:18.928 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738468 got 34359738473 for osd.0 2024-04-26T01:44:18.928 DEBUG:teuthology.parallel:result is None 2024-04-26T01:44:18.929 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T01:44:18.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:44:19.423 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:19.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:19 smithi188 ceph-mon[32751]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-04-26T01:44:19.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:19 smithi029 ceph-mon[32775]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-04-26T01:44:19.734 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:19 smithi006 ceph-mon[28419]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-04-26T01:44:20.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:20 smithi029 ceph-mon[32775]: pgmap v401: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T01:44:20.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:20.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:20 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:20.733 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:20 smithi006 ceph-mon[28419]: pgmap v401: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T01:44:20.733 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:20.733 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:20 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:20.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:20 smithi188 ceph-mon[32751]: pgmap v401: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T01:44:20.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:20.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:20 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:21.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:44:21.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:44:21.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:21 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:21.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.604+0000 7f0ebdc00700 1 -- 172.21.15.6:0/112974302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eb8074e70 msgr2=0x7f0eb8075250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:21.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.604+0000 7f0ebdc00700 1 --2- 172.21.15.6:0/112974302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eb8074e70 0x7f0eb8075250 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009990 tx=0x7f0ea8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:21.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.605+0000 7f0ebdc00700 1 -- 172.21.15.6:0/112974302 shutdown_connections 2024-04-26T01:44:21.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.605+0000 7f0ebdc00700 1 --2- 172.21.15.6:0/112974302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0eb810f1b0 0x7f0eb8111620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:21.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.605+0000 7f0ebdc00700 1 --2- 172.21.15.6:0/112974302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eb8074e70 0x7f0eb8075250 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:21.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.605+0000 7f0ebdc00700 1 --2- 172.21.15.6:0/112974302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0eb8075790 0x7f0eb810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:21.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.605+0000 7f0ebdc00700 1 -- 172.21.15.6:0/112974302 >> 172.21.15.6:0/112974302 conn(0x7f0eb80fe4c0 msgr2=0x7f0eb81008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:21.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.605+0000 7f0ebdc00700 1 -- 172.21.15.6:0/112974302 shutdown_connections 2024-04-26T01:44:21.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.605+0000 7f0ebdc00700 1 -- 172.21.15.6:0/112974302 wait complete. 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.606+0000 7f0ebdc00700 1 Processor -- start 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.606+0000 7f0ebdc00700 1 -- start start 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.606+0000 7f0ebdc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eb8074e70 0x7f0eb819b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.606+0000 7f0ebdc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0eb8075790 0x7f0eb819b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.606+0000 7f0ebdc00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0eb810f1b0 0x7f0eb819fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.606+0000 7f0ebdc00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb8114d40 con 0x7f0eb8074e70 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.606+0000 7f0ebdc00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb8114bc0 con 0x7f0eb810f1b0 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.606+0000 7f0ebdc00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb8114ec0 con 0x7f0eb8075790 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.607+0000 7f0eb7e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eb8074e70 0x7f0eb819b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.607+0000 7f0eb7400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0eb8075790 0x7f0eb819b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.607+0000 7f0eb7e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eb8074e70 0x7f0eb819b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59750/0 (socket says 172.21.15.6:59750) 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.607+0000 7f0eb7400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0eb8075790 0x7f0eb819b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:41786/0 (socket says 172.21.15.6:41786) 2024-04-26T01:44:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.607+0000 7f0eb7e00700 1 -- 172.21.15.6:0/2104969845 learned_addr learned my addr 172.21.15.6:0/2104969845 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:21.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.607+0000 7f0ebd200700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0eb810f1b0 0x7f0eb819fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:21.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.607+0000 7f0eb7e00700 1 -- 172.21.15.6:0/2104969845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0eb8075790 msgr2=0x7f0eb819b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:21.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.607+0000 7f0eb7e00700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0eb8075790 0x7f0eb819b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:21.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.607+0000 7f0eb7e00700 1 -- 172.21.15.6:0/2104969845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0eb810f1b0 msgr2=0x7f0eb819fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:21.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.607+0000 7f0eb7e00700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0eb810f1b0 0x7f0eb819fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:21.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.608+0000 7f0eb7e00700 1 -- 172.21.15.6:0/2104969845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb81a0260 con 0x7f0eb8074e70 2024-04-26T01:44:21.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.608+0000 7f0eb7400700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0eb8075790 0x7f0eb819b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:21.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.608+0000 7f0eb7e00700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eb8074e70 0x7f0eb819b250 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0ea80057f0 tx=0x7f0ea800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:21.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.608+0000 7f0eb4c00700 1 -- 172.21.15.6:0/2104969845 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8026070 con 0x7f0eb8074e70 2024-04-26T01:44:21.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.608+0000 7f0eb4c00700 1 -- 172.21.15.6:0/2104969845 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ea8021410 con 0x7f0eb8074e70 2024-04-26T01:44:21.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.608+0000 7f0eb4c00700 1 -- 172.21.15.6:0/2104969845 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8017650 con 0x7f0eb8074e70 2024-04-26T01:44:21.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.608+0000 7f0ebdc00700 1 -- 172.21.15.6:0/2104969845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eb81a04f0 con 0x7f0eb8074e70 2024-04-26T01:44:21.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.608+0000 7f0ebdc00700 1 -- 172.21.15.6:0/2104969845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eb81a7dd0 con 0x7f0eb8074e70 2024-04-26T01:44:21.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.610+0000 7f0ebdc00700 1 -- 172.21.15.6:0/2104969845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eb8069730 con 0x7f0eb8074e70 2024-04-26T01:44:21.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.612+0000 7f0eb4c00700 1 -- 172.21.15.6:0/2104969845 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f0ea8029070 con 0x7f0eb8074e70 2024-04-26T01:44:21.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.613+0000 7f0eb4c00700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f0e9c0a3cc0 0x7f0e9c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:21.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.613+0000 7f0eb4c00700 1 -- 172.21.15.6:0/2104969845 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f0ea80d1160 con 0x7f0eb8074e70 2024-04-26T01:44:21.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.614+0000 7f0eb7400700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f0e9c0a3cc0 0x7f0e9c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:21.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.614+0000 7f0eb7400700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f0e9c0a3cc0 0x7f0e9c0a6180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0eb81105e0 tx=0x7f0ea4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:21.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.623+0000 7f0eb4c00700 1 -- 172.21.15.6:0/2104969845 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ea805dff0 con 0x7f0eb8074e70 2024-04-26T01:44:21.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:21 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:44:21.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:21 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:44:21.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:21 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:21.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.855+0000 7f0ebdc00700 1 -- 172.21.15.6:0/2104969845 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0eb819c5b0 con 0x7f0e9c0a3cc0 2024-04-26T01:44:21.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.864+0000 7f0eb4c00700 1 -- 172.21.15.6:0/2104969845 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258959 (secure 0 0 0) 0x7f0eb819c5b0 con 0x7f0e9c0a3cc0 2024-04-26T01:44:21.868 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:44:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.869+0000 7f0ebdc00700 1 -- 172.21.15.6:0/2104969845 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f0e9c0a3cc0 msgr2=0x7f0e9c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.869+0000 7f0ebdc00700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f0e9c0a3cc0 0x7f0e9c0a6180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0eb81105e0 tx=0x7f0ea4009260 comp rx=0 tx=0).stop 2024-04-26T01:44:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.869+0000 7f0ebdc00700 1 -- 172.21.15.6:0/2104969845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eb8074e70 msgr2=0x7f0eb819b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.869+0000 7f0ebdc00700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eb8074e70 0x7f0eb819b250 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0ea80057f0 tx=0x7f0ea800bfd0 comp rx=0 tx=0).stop 2024-04-26T01:44:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.870+0000 7f0ebdc00700 1 -- 172.21.15.6:0/2104969845 shutdown_connections 2024-04-26T01:44:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.870+0000 7f0ebdc00700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0eb810f1b0 0x7f0eb819fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.870+0000 7f0ebdc00700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eb8074e70 0x7f0eb819b250 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.870+0000 7f0ebdc00700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f0e9c0a3cc0 0x7f0e9c0a6180 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.870+0000 7f0ebdc00700 1 --2- 172.21.15.6:0/2104969845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0eb8075790 0x7f0eb819b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.870+0000 7f0ebdc00700 1 -- 172.21.15.6:0/2104969845 >> 172.21.15.6:0/2104969845 conn(0x7f0eb80fe4c0 msgr2=0x7f0eb80ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.870+0000 7f0ebdc00700 1 -- 172.21.15.6:0/2104969845 shutdown_connections 2024-04-26T01:44:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:21.870+0000 7f0ebdc00700 1 -- 172.21.15.6:0/2104969845 wait complete. 2024-04-26T01:44:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-26T01:44:21.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:21 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:44:21.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:21 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:44:21.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:21 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:44:22.792 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-04-26T01:44:21.229607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659735,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":839,"num_read_kb":546,"num_write":358,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":746,"ondisk_log_size":746,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392760,"kb_used_data":14472,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549064,"statfs":{"total":1151940427776,"available":1151538241536,"internally_reserved":0,"allocated":14819328,"data_stored":8097444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":37,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004652"},"pg_stats":[{"pgid":"3.3f","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.186201+0000","last_change":"2024-04-26T01:42:05.417304+0000","last_active":"2024-04-26T01:44:17.186201+0000","last_peered":"2024-04-26T01:44:17.186201+0000","last_clean":"2024-04-26T01:44:17.186201+0000","last_became_active":"2024-04-26T01:42:05.416999+0000","last_became_peered":"2024-04-26T01:42:05.416999+0000","last_unstale":"2024-04-26T01:44:17.186201+0000","last_undegraded":"2024-04-26T01:44:17.186201+0000","last_fullsized":"2024-04-26T01:44:17.186201+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:33:30.958943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"95'4","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.176097+0000","last_change":"2024-04-26T01:42:05.413089+0000","last_active":"2024-04-26T01:44:17.176097+0000","last_peered":"2024-04-26T01:44:17.176097+0000","last_clean":"2024-04-26T01:44:17.176097+0000","last_became_active":"2024-04-26T01:42:05.412776+0000","last_became_peered":"2024-04-26T01:42:05.412776+0000","last_unstale":"2024-04-26T01:44:17.176097+0000","last_undegraded":"2024-04-26T01:44:17.176097+0000","last_fullsized":"2024-04-26T01:44:17.176097+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:47.861340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"95'7","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.185817+0000","last_change":"2024-04-26T01:42:05.404864+0000","last_active":"2024-04-26T01:44:17.185817+0000","last_peered":"2024-04-26T01:44:17.185817+0000","last_clean":"2024-04-26T01:44:17.185817+0000","last_became_active":"2024-04-26T01:42:05.404529+0000","last_became_peered":"2024-04-26T01:42:05.404529+0000","last_unstale":"2024-04-26T01:44:17.185817+0000","last_undegraded":"2024-04-26T01:44:17.185817+0000","last_fullsized":"2024-04-26T01:44:17.185817+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:04:18.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.155446+0000","last_change":"2024-04-26T01:42:05.393582+0000","last_active":"2024-04-26T01:44:17.155446+0000","last_peered":"2024-04-26T01:44:17.155446+0000","last_clean":"2024-04-26T01:44:17.155446+0000","last_became_active":"2024-04-26T01:42:05.393193+0000","last_became_peered":"2024-04-26T01:42:05.393193+0000","last_unstale":"2024-04-26T01:44:17.155446+0000","last_undegraded":"2024-04-26T01:44:17.155446+0000","last_fullsized":"2024-04-26T01:44:17.155446+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:20:29.420459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"95'4","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.187247+0000","last_change":"2024-04-26T01:42:05.412979+0000","last_active":"2024-04-26T01:44:17.187247+0000","last_peered":"2024-04-26T01:44:17.187247+0000","last_clean":"2024-04-26T01:44:17.187247+0000","last_became_active":"2024-04-26T01:42:05.412679+0000","last_became_peered":"2024-04-26T01:42:05.412679+0000","last_unstale":"2024-04-26T01:44:17.187247+0000","last_undegraded":"2024-04-26T01:44:17.187247+0000","last_fullsized":"2024-04-26T01:44:17.187247+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:22.774960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"95'6","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.180795+0000","last_change":"2024-04-26T01:42:49.062507+0000","last_active":"2024-04-26T01:44:17.180795+0000","last_peered":"2024-04-26T01:44:17.180795+0000","last_clean":"2024-04-26T01:44:17.180795+0000","last_became_active":"2024-04-26T01:42:49.062095+0000","last_became_peered":"2024-04-26T01:42:49.062095+0000","last_unstale":"2024-04-26T01:44:17.180795+0000","last_undegraded":"2024-04-26T01:44:17.180795+0000","last_fullsized":"2024-04-26T01:44:17.180795+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:47.846697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"95'5","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.178020+0000","last_change":"2024-04-26T01:42:05.385221+0000","last_active":"2024-04-26T01:44:17.178020+0000","last_peered":"2024-04-26T01:44:17.178020+0000","last_clean":"2024-04-26T01:44:17.178020+0000","last_became_active":"2024-04-26T01:42:05.384856+0000","last_became_peered":"2024-04-26T01:42:05.384856+0000","last_unstale":"2024-04-26T01:44:17.178020+0000","last_undegraded":"2024-04-26T01:44:17.178020+0000","last_fullsized":"2024-04-26T01:44:17.178020+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:07:23.461324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"95'9","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.177700+0000","last_change":"2024-04-26T01:42:05.404754+0000","last_active":"2024-04-26T01:44:17.177700+0000","last_peered":"2024-04-26T01:44:17.177700+0000","last_clean":"2024-04-26T01:44:17.177700+0000","last_became_active":"2024-04-26T01:42:05.404220+0000","last_became_peered":"2024-04-26T01:42:05.404220+0000","last_unstale":"2024-04-26T01:44:17.177700+0000","last_undegraded":"2024-04-26T01:44:17.177700+0000","last_fullsized":"2024-04-26T01:44:17.177700+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:50.243552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"95'11","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.169628+0000","last_change":"2024-04-26T01:42:05.390972+0000","last_active":"2024-04-26T01:44:17.169628+0000","last_peered":"2024-04-26T01:44:17.169628+0000","last_clean":"2024-04-26T01:44:17.169628+0000","last_became_active":"2024-04-26T01:42:05.390599+0000","last_became_peered":"2024-04-26T01:42:05.390599+0000","last_unstale":"2024-04-26T01:44:17.169628+0000","last_undegraded":"2024-04-26T01:44:17.169628+0000","last_fullsized":"2024-04-26T01:44:17.169628+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:19:46.328640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"95'16","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.167601+0000","last_change":"2024-04-26T01:42:05.392320+0000","last_active":"2024-04-26T01:44:17.167601+0000","last_peered":"2024-04-26T01:44:17.167601+0000","last_clean":"2024-04-26T01:44:17.167601+0000","last_became_active":"2024-04-26T01:42:05.391986+0000","last_became_peered":"2024-04-26T01:42:05.391986+0000","last_unstale":"2024-04-26T01:44:17.167601+0000","last_undegraded":"2024-04-26T01:44:17.167601+0000","last_fullsized":"2024-04-26T01:44:17.167601+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:17:27.302358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"95'6","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.167895+0000","last_change":"2024-04-26T01:42:05.388077+0000","last_active":"2024-04-26T01:44:17.167895+0000","last_peered":"2024-04-26T01:44:17.167895+0000","last_clean":"2024-04-26T01:44:17.167895+0000","last_became_active":"2024-04-26T01:42:05.387545+0000","last_became_peered":"2024-04-26T01:42:05.387545+0000","last_unstale":"2024-04-26T01:44:17.167895+0000","last_undegraded":"2024-04-26T01:44:17.167895+0000","last_fullsized":"2024-04-26T01:44:17.167895+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:51.288177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.310710+0000","last_change":"2024-04-26T01:42:49.067449+0000","last_active":"2024-04-26T01:43:50.310710+0000","last_peered":"2024-04-26T01:43:50.310710+0000","last_clean":"2024-04-26T01:43:50.310710+0000","last_became_active":"2024-04-26T01:42:49.065991+0000","last_became_peered":"2024-04-26T01:42:49.065991+0000","last_unstale":"2024-04-26T01:43:50.310710+0000","last_undegraded":"2024-04-26T01:43:50.310710+0000","last_fullsized":"2024-04-26T01:43:50.310710+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:13:36.565257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"95'10","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.193148+0000","last_change":"2024-04-26T01:42:05.402847+0000","last_active":"2024-04-26T01:44:17.193148+0000","last_peered":"2024-04-26T01:44:17.193148+0000","last_clean":"2024-04-26T01:44:17.193148+0000","last_became_active":"2024-04-26T01:42:05.402503+0000","last_became_peered":"2024-04-26T01:42:05.402503+0000","last_unstale":"2024-04-26T01:44:17.193148+0000","last_undegraded":"2024-04-26T01:44:17.193148+0000","last_fullsized":"2024-04-26T01:44:17.193148+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:06.935060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"95'7","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.185541+0000","last_change":"2024-04-26T01:42:05.405257+0000","last_active":"2024-04-26T01:44:17.185541+0000","last_peered":"2024-04-26T01:44:17.185541+0000","last_clean":"2024-04-26T01:44:17.185541+0000","last_became_active":"2024-04-26T01:42:05.404551+0000","last_became_peered":"2024-04-26T01:42:05.404551+0000","last_unstale":"2024-04-26T01:44:17.185541+0000","last_undegraded":"2024-04-26T01:44:17.185541+0000","last_fullsized":"2024-04-26T01:44:17.185541+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:20.754088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"95'7","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.184893+0000","last_change":"2024-04-26T01:42:05.408847+0000","last_active":"2024-04-26T01:44:17.184893+0000","last_peered":"2024-04-26T01:44:17.184893+0000","last_clean":"2024-04-26T01:44:17.184893+0000","last_became_active":"2024-04-26T01:42:05.406189+0000","last_became_peered":"2024-04-26T01:42:05.406189+0000","last_unstale":"2024-04-26T01:44:17.184893+0000","last_undegraded":"2024-04-26T01:44:17.184893+0000","last_fullsized":"2024-04-26T01:44:17.184893+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:59:58.633630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"95'9","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.179233+0000","last_change":"2024-04-26T01:42:05.404647+0000","last_active":"2024-04-26T01:44:17.179233+0000","last_peered":"2024-04-26T01:44:17.179233+0000","last_clean":"2024-04-26T01:44:17.179233+0000","last_became_active":"2024-04-26T01:42:05.404265+0000","last_became_peered":"2024-04-26T01:42:05.404265+0000","last_unstale":"2024-04-26T01:44:17.179233+0000","last_undegraded":"2024-04-26T01:44:17.179233+0000","last_fullsized":"2024-04-26T01:44:17.179233+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:00:43.472541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"95'8","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.186395+0000","last_change":"2024-04-26T01:42:05.417602+0000","last_active":"2024-04-26T01:44:17.186395+0000","last_peered":"2024-04-26T01:44:17.186395+0000","last_clean":"2024-04-26T01:44:17.186395+0000","last_became_active":"2024-04-26T01:42:05.417144+0000","last_became_peered":"2024-04-26T01:42:05.417144+0000","last_unstale":"2024-04-26T01:44:17.186395+0000","last_undegraded":"2024-04-26T01:44:17.186395+0000","last_fullsized":"2024-04-26T01:44:17.186395+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:34.043597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.311101+0000","last_change":"2024-04-26T01:42:05.392948+0000","last_active":"2024-04-26T01:43:50.311101+0000","last_peered":"2024-04-26T01:43:50.311101+0000","last_clean":"2024-04-26T01:43:50.311101+0000","last_became_active":"2024-04-26T01:42:05.392040+0000","last_became_peered":"2024-04-26T01:42:05.392040+0000","last_unstale":"2024-04-26T01:43:50.311101+0000","last_undegraded":"2024-04-26T01:43:50.311101+0000","last_fullsized":"2024-04-26T01:43:50.311101+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:17.307979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"95'7","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.189177+0000","last_change":"2024-04-26T01:42:05.406588+0000","last_active":"2024-04-26T01:44:17.189177+0000","last_peered":"2024-04-26T01:44:17.189177+0000","last_clean":"2024-04-26T01:44:17.189177+0000","last_became_active":"2024-04-26T01:42:05.406200+0000","last_became_peered":"2024-04-26T01:42:05.406200+0000","last_unstale":"2024-04-26T01:44:17.189177+0000","last_undegraded":"2024-04-26T01:44:17.189177+0000","last_fullsized":"2024-04-26T01:44:17.189177+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:14.095383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.311220+0000","last_change":"2024-04-26T01:42:01.339289+0000","last_active":"2024-04-26T01:43:50.311220+0000","last_peered":"2024-04-26T01:43:50.311220+0000","last_clean":"2024-04-26T01:43:50.311220+0000","last_became_active":"2024-04-26T01:42:01.335402+0000","last_became_peered":"2024-04-26T01:42:01.335402+0000","last_unstale":"2024-04-26T01:43:50.311220+0000","last_undegraded":"2024-04-26T01:43:50.311220+0000","last_fullsized":"2024-04-26T01:43:50.311220+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:51.411524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"95'5","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.172713+0000","last_change":"2024-04-26T01:42:05.409569+0000","last_active":"2024-04-26T01:44:17.172713+0000","last_peered":"2024-04-26T01:44:17.172713+0000","last_clean":"2024-04-26T01:44:17.172713+0000","last_became_active":"2024-04-26T01:42:05.407441+0000","last_became_peered":"2024-04-26T01:42:05.407441+0000","last_unstale":"2024-04-26T01:44:17.172713+0000","last_undegraded":"2024-04-26T01:44:17.172713+0000","last_fullsized":"2024-04-26T01:44:17.172713+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:36:58.043359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.990349+0000","last_change":"2024-04-26T01:42:02.231871+0000","last_active":"2024-04-26T01:43:49.990349+0000","last_peered":"2024-04-26T01:43:49.990349+0000","last_clean":"2024-04-26T01:43:49.990349+0000","last_became_active":"2024-04-26T01:42:02.230813+0000","last_became_peered":"2024-04-26T01:42:02.230813+0000","last_unstale":"2024-04-26T01:43:49.990349+0000","last_undegraded":"2024-04-26T01:43:49.990349+0000","last_fullsized":"2024-04-26T01:43:49.990349+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:53.117266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"95'6","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.184410+0000","last_change":"2024-04-26T01:42:05.412414+0000","last_active":"2024-04-26T01:44:17.184410+0000","last_peered":"2024-04-26T01:44:17.184410+0000","last_clean":"2024-04-26T01:44:17.184410+0000","last_became_active":"2024-04-26T01:42:05.412057+0000","last_became_peered":"2024-04-26T01:42:05.412057+0000","last_unstale":"2024-04-26T01:44:17.184410+0000","last_undegraded":"2024-04-26T01:44:17.184410+0000","last_fullsized":"2024-04-26T01:44:17.184410+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:51:46.331246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.985428+0000","last_change":"2024-04-26T01:42:01.329957+0000","last_active":"2024-04-26T01:43:49.985428+0000","last_peered":"2024-04-26T01:43:49.985428+0000","last_clean":"2024-04-26T01:43:49.985428+0000","last_became_active":"2024-04-26T01:42:01.329289+0000","last_became_peered":"2024-04-26T01:42:01.329289+0000","last_unstale":"2024-04-26T01:43:49.985428+0000","last_undegraded":"2024-04-26T01:43:49.985428+0000","last_fullsized":"2024-04-26T01:43:49.985428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:46:38.983138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"95'9","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.190195+0000","last_change":"2024-04-26T01:42:05.399438+0000","last_active":"2024-04-26T01:44:17.190195+0000","last_peered":"2024-04-26T01:44:17.190195+0000","last_clean":"2024-04-26T01:44:17.190195+0000","last_became_active":"2024-04-26T01:42:05.399082+0000","last_became_peered":"2024-04-26T01:42:05.399082+0000","last_unstale":"2024-04-26T01:44:17.190195+0000","last_undegraded":"2024-04-26T01:44:17.190195+0000","last_fullsized":"2024-04-26T01:44:17.190195+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:13.619893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"91'2","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.993619+0000","last_change":"2024-04-26T01:43:49.993134+0000","last_active":"2024-04-26T01:43:49.993619+0000","last_peered":"2024-04-26T01:43:49.993619+0000","last_clean":"2024-04-26T01:43:49.993619+0000","last_became_active":"2024-04-26T01:43:49.990337+0000","last_became_peered":"2024-04-26T01:43:49.990337+0000","last_unstale":"2024-04-26T01:43:49.993619+0000","last_undegraded":"2024-04-26T01:43:49.993619+0000","last_fullsized":"2024-04-26T01:43:49.993619+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:50.502668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.310845+0000","last_change":"2024-04-26T01:42:49.066057+0000","last_active":"2024-04-26T01:43:50.310845+0000","last_peered":"2024-04-26T01:43:50.310845+0000","last_clean":"2024-04-26T01:43:50.310845+0000","last_became_active":"2024-04-26T01:42:49.064684+0000","last_became_peered":"2024-04-26T01:42:49.064684+0000","last_unstale":"2024-04-26T01:43:50.310845+0000","last_undegraded":"2024-04-26T01:43:50.310845+0000","last_fullsized":"2024-04-26T01:43:50.310845+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:39.159528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"95'7","reported_seq":153,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.751256+0000","last_change":"2024-04-26T01:42:01.332623+0000","last_active":"2024-04-26T01:44:17.751256+0000","last_peered":"2024-04-26T01:44:17.751256+0000","last_clean":"2024-04-26T01:44:17.751256+0000","last_became_active":"2024-04-26T01:42:01.332227+0000","last_became_peered":"2024-04-26T01:42:01.332227+0000","last_unstale":"2024-04-26T01:44:17.751256+0000","last_undegraded":"2024-04-26T01:44:17.751256+0000","last_fullsized":"2024-04-26T01:44:17.751256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:58.611464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"95'7","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.179190+0000","last_change":"2024-04-26T01:42:05.396133+0000","last_active":"2024-04-26T01:44:17.179190+0000","last_peered":"2024-04-26T01:44:17.179190+0000","last_clean":"2024-04-26T01:44:17.179190+0000","last_became_active":"2024-04-26T01:42:05.395553+0000","last_became_peered":"2024-04-26T01:42:05.395553+0000","last_unstale":"2024-04-26T01:44:17.179190+0000","last_undegraded":"2024-04-26T01:44:17.179190+0000","last_fullsized":"2024-04-26T01:44:17.179190+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:36:01.363909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.146452+0000","last_change":"2024-04-26T01:42:01.340624+0000","last_active":"2024-04-26T01:43:50.146452+0000","last_peered":"2024-04-26T01:43:50.146452+0000","last_clean":"2024-04-26T01:43:50.146452+0000","last_became_active":"2024-04-26T01:42:01.340158+0000","last_became_peered":"2024-04-26T01:42:01.340158+0000","last_unstale":"2024-04-26T01:43:50.146452+0000","last_undegraded":"2024-04-26T01:43:50.146452+0000","last_fullsized":"2024-04-26T01:43:50.146452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:47:05.529922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"95'8","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.169120+0000","last_change":"2024-04-26T01:42:05.406354+0000","last_active":"2024-04-26T01:44:17.169120+0000","last_peered":"2024-04-26T01:44:17.169120+0000","last_clean":"2024-04-26T01:44:17.169120+0000","last_became_active":"2024-04-26T01:42:05.405989+0000","last_became_peered":"2024-04-26T01:42:05.405989+0000","last_unstale":"2024-04-26T01:44:17.169120+0000","last_undegraded":"2024-04-26T01:44:17.169120+0000","last_fullsized":"2024-04-26T01:44:17.169120+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:43.115225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.309897+0000","last_change":"2024-04-26T01:42:01.340549+0000","last_active":"2024-04-26T01:43:50.309897+0000","last_peered":"2024-04-26T01:43:50.309897+0000","last_clean":"2024-04-26T01:43:50.309897+0000","last_became_active":"2024-04-26T01:42:01.339861+0000","last_became_peered":"2024-04-26T01:42:01.339861+0000","last_unstale":"2024-04-26T01:43:50.309897+0000","last_undegraded":"2024-04-26T01:43:50.309897+0000","last_fullsized":"2024-04-26T01:43:50.309897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:02:53.154367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"95'8","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.191565+0000","last_change":"2024-04-26T01:42:05.402664+0000","last_active":"2024-04-26T01:44:17.191565+0000","last_peered":"2024-04-26T01:44:17.191565+0000","last_clean":"2024-04-26T01:44:17.191565+0000","last_became_active":"2024-04-26T01:42:05.400785+0000","last_became_peered":"2024-04-26T01:42:05.400785+0000","last_unstale":"2024-04-26T01:44:17.191565+0000","last_undegraded":"2024-04-26T01:44:17.191565+0000","last_fullsized":"2024-04-26T01:44:17.191565+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:40:32.137752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.988999+0000","last_change":"2024-04-26T01:42:01.328600+0000","last_active":"2024-04-26T01:43:49.988999+0000","last_peered":"2024-04-26T01:43:49.988999+0000","last_clean":"2024-04-26T01:43:49.988999+0000","last_became_active":"2024-04-26T01:42:01.328149+0000","last_became_peered":"2024-04-26T01:42:01.328149+0000","last_unstale":"2024-04-26T01:43:49.988999+0000","last_undegraded":"2024-04-26T01:43:49.988999+0000","last_fullsized":"2024-04-26T01:43:49.988999+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:45:42.811088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.310560+0000","last_change":"2024-04-26T01:42:49.065824+0000","last_active":"2024-04-26T01:43:50.310560+0000","last_peered":"2024-04-26T01:43:50.310560+0000","last_clean":"2024-04-26T01:43:50.310560+0000","last_became_active":"2024-04-26T01:42:49.064670+0000","last_became_peered":"2024-04-26T01:42:49.064670+0000","last_unstale":"2024-04-26T01:43:50.310560+0000","last_undegraded":"2024-04-26T01:43:50.310560+0000","last_fullsized":"2024-04-26T01:43:50.310560+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:44.790184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"95'4","reported_seq":146,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.748772+0000","last_change":"2024-04-26T01:42:01.335438+0000","last_active":"2024-04-26T01:44:17.748772+0000","last_peered":"2024-04-26T01:44:17.748772+0000","last_clean":"2024-04-26T01:44:17.748772+0000","last_became_active":"2024-04-26T01:42:01.334962+0000","last_became_peered":"2024-04-26T01:42:01.334962+0000","last_unstale":"2024-04-26T01:44:17.748772+0000","last_undegraded":"2024-04-26T01:44:17.748772+0000","last_fullsized":"2024-04-26T01:44:17.748772+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:35:08.252274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"95'10","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.182898+0000","last_change":"2024-04-26T01:42:05.393294+0000","last_active":"2024-04-26T01:44:17.182898+0000","last_peered":"2024-04-26T01:44:17.182898+0000","last_clean":"2024-04-26T01:44:17.182898+0000","last_became_active":"2024-04-26T01:42:05.392974+0000","last_became_peered":"2024-04-26T01:42:05.392974+0000","last_unstale":"2024-04-26T01:44:17.182898+0000","last_undegraded":"2024-04-26T01:44:17.182898+0000","last_fullsized":"2024-04-26T01:44:17.182898+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:58.188706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.109030+0000","last_change":"2024-04-26T01:42:01.333505+0000","last_active":"2024-04-26T01:43:50.109030+0000","last_peered":"2024-04-26T01:43:50.109030+0000","last_clean":"2024-04-26T01:43:50.109030+0000","last_became_active":"2024-04-26T01:42:01.333062+0000","last_became_peered":"2024-04-26T01:42:01.333062+0000","last_unstale":"2024-04-26T01:43:50.109030+0000","last_undegraded":"2024-04-26T01:43:50.109030+0000","last_fullsized":"2024-04-26T01:43:50.109030+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:50:48.804443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"95'9","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.183747+0000","last_change":"2024-04-26T01:42:49.063948+0000","last_active":"2024-04-26T01:44:17.183747+0000","last_peered":"2024-04-26T01:44:17.183747+0000","last_clean":"2024-04-26T01:44:17.183747+0000","last_became_active":"2024-04-26T01:42:49.060852+0000","last_became_peered":"2024-04-26T01:42:49.060852+0000","last_unstale":"2024-04-26T01:44:17.183747+0000","last_undegraded":"2024-04-26T01:44:17.183747+0000","last_fullsized":"2024-04-26T01:44:17.183747+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.370431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.989442+0000","last_change":"2024-04-26T01:42:02.232308+0000","last_active":"2024-04-26T01:43:49.989442+0000","last_peered":"2024-04-26T01:43:49.989442+0000","last_clean":"2024-04-26T01:43:49.989442+0000","last_became_active":"2024-04-26T01:42:02.231897+0000","last_became_peered":"2024-04-26T01:42:02.231897+0000","last_unstale":"2024-04-26T01:43:49.989442+0000","last_undegraded":"2024-04-26T01:43:49.989442+0000","last_fullsized":"2024-04-26T01:43:49.989442+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:36:16.847087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"95'11","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.190211+0000","last_change":"2024-04-26T01:42:49.070798+0000","last_active":"2024-04-26T01:44:17.190211+0000","last_peered":"2024-04-26T01:44:17.190211+0000","last_clean":"2024-04-26T01:44:17.190211+0000","last_became_active":"2024-04-26T01:42:49.070470+0000","last_became_peered":"2024-04-26T01:42:49.070470+0000","last_unstale":"2024-04-26T01:44:17.190211+0000","last_undegraded":"2024-04-26T01:44:17.190211+0000","last_fullsized":"2024-04-26T01:44:17.190211+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:04:22.103704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"95'34","reported_seq":299,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:18.696011+0000","last_change":"2024-04-26T01:42:01.330822+0000","last_active":"2024-04-26T01:44:18.696011+0000","last_peered":"2024-04-26T01:44:18.696011+0000","last_clean":"2024-04-26T01:44:18.696011+0000","last_became_active":"2024-04-26T01:42:01.330337+0000","last_became_peered":"2024-04-26T01:42:01.330337+0000","last_unstale":"2024-04-26T01:44:18.696011+0000","last_undegraded":"2024-04-26T01:44:18.696011+0000","last_fullsized":"2024-04-26T01:44:18.696011+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:04.024645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64459,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":235,"num_read_kb":181,"num_write":35,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"95'6","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.165940+0000","last_change":"2024-04-26T01:42:05.390601+0000","last_active":"2024-04-26T01:44:17.165940+0000","last_peered":"2024-04-26T01:44:17.165940+0000","last_clean":"2024-04-26T01:44:17.165940+0000","last_became_active":"2024-04-26T01:42:05.390279+0000","last_became_peered":"2024-04-26T01:42:05.390279+0000","last_unstale":"2024-04-26T01:44:17.165940+0000","last_undegraded":"2024-04-26T01:44:17.165940+0000","last_fullsized":"2024-04-26T01:44:17.165940+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:20.171769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.308520+0000","last_change":"2024-04-26T01:42:01.337200+0000","last_active":"2024-04-26T01:43:50.308520+0000","last_peered":"2024-04-26T01:43:50.308520+0000","last_clean":"2024-04-26T01:43:50.308520+0000","last_became_active":"2024-04-26T01:42:01.336919+0000","last_became_peered":"2024-04-26T01:42:01.336919+0000","last_unstale":"2024-04-26T01:43:50.308520+0000","last_undegraded":"2024-04-26T01:43:50.308520+0000","last_fullsized":"2024-04-26T01:43:50.308520+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:29.799903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"95'9","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.187810+0000","last_change":"2024-04-26T01:42:05.410081+0000","last_active":"2024-04-26T01:44:17.187810+0000","last_peered":"2024-04-26T01:44:17.187810+0000","last_clean":"2024-04-26T01:44:17.187810+0000","last_became_active":"2024-04-26T01:42:05.409341+0000","last_became_peered":"2024-04-26T01:42:05.409341+0000","last_unstale":"2024-04-26T01:44:17.187810+0000","last_undegraded":"2024-04-26T01:44:17.187810+0000","last_fullsized":"2024-04-26T01:44:17.187810+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:25:31.925296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.989169+0000","last_change":"2024-04-26T01:42:01.331702+0000","last_active":"2024-04-26T01:43:49.989169+0000","last_peered":"2024-04-26T01:43:49.989169+0000","last_clean":"2024-04-26T01:43:49.989169+0000","last_became_active":"2024-04-26T01:42:01.331311+0000","last_became_peered":"2024-04-26T01:42:01.331311+0000","last_unstale":"2024-04-26T01:43:49.989169+0000","last_undegraded":"2024-04-26T01:43:49.989169+0000","last_fullsized":"2024-04-26T01:43:49.989169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:54:49.513627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"95'15","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.193275+0000","last_change":"2024-04-26T01:42:05.403957+0000","last_active":"2024-04-26T01:44:17.193275+0000","last_peered":"2024-04-26T01:44:17.193275+0000","last_clean":"2024-04-26T01:44:17.193275+0000","last_became_active":"2024-04-26T01:42:05.403583+0000","last_became_peered":"2024-04-26T01:42:05.403583+0000","last_unstale":"2024-04-26T01:44:17.193275+0000","last_undegraded":"2024-04-26T01:44:17.193275+0000","last_fullsized":"2024-04-26T01:44:17.193275+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:12:20.095407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"95'5","reported_seq":148,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:16.716891+0000","last_change":"2024-04-26T01:42:01.332456+0000","last_active":"2024-04-26T01:44:16.716891+0000","last_peered":"2024-04-26T01:44:16.716891+0000","last_clean":"2024-04-26T01:44:16.716891+0000","last_became_active":"2024-04-26T01:42:01.331910+0000","last_became_peered":"2024-04-26T01:42:01.331910+0000","last_unstale":"2024-04-26T01:44:16.716891+0000","last_undegraded":"2024-04-26T01:44:16.716891+0000","last_fullsized":"2024-04-26T01:44:16.716891+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:13:25.368205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"95'6","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.175614+0000","last_change":"2024-04-26T01:42:49.062130+0000","last_active":"2024-04-26T01:44:17.175614+0000","last_peered":"2024-04-26T01:44:17.175614+0000","last_clean":"2024-04-26T01:44:17.175614+0000","last_became_active":"2024-04-26T01:42:49.061627+0000","last_became_peered":"2024-04-26T01:42:49.061627+0000","last_unstale":"2024-04-26T01:44:17.175614+0000","last_undegraded":"2024-04-26T01:44:17.175614+0000","last_fullsized":"2024-04-26T01:44:17.175614+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:16.101940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.986820+0000","last_change":"2024-04-26T01:42:49.082618+0000","last_active":"2024-04-26T01:43:49.986820+0000","last_peered":"2024-04-26T01:43:49.986820+0000","last_clean":"2024-04-26T01:43:49.986820+0000","last_became_active":"2024-04-26T01:42:49.064654+0000","last_became_peered":"2024-04-26T01:42:49.064654+0000","last_unstale":"2024-04-26T01:43:49.986820+0000","last_undegraded":"2024-04-26T01:43:49.986820+0000","last_fullsized":"2024-04-26T01:43:49.986820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:13:23.937059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"95'6","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.155496+0000","last_change":"2024-04-26T01:42:05.391392+0000","last_active":"2024-04-26T01:44:17.155496+0000","last_peered":"2024-04-26T01:44:17.155496+0000","last_clean":"2024-04-26T01:44:17.155496+0000","last_became_active":"2024-04-26T01:42:05.390939+0000","last_became_peered":"2024-04-26T01:42:05.390939+0000","last_unstale":"2024-04-26T01:44:17.155496+0000","last_undegraded":"2024-04-26T01:44:17.155496+0000","last_fullsized":"2024-04-26T01:44:17.155496+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:36.141915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.985486+0000","last_change":"2024-04-26T01:42:01.332932+0000","last_active":"2024-04-26T01:43:49.985486+0000","last_peered":"2024-04-26T01:43:49.985486+0000","last_clean":"2024-04-26T01:43:49.985486+0000","last_became_active":"2024-04-26T01:42:01.332626+0000","last_became_peered":"2024-04-26T01:42:01.332626+0000","last_unstale":"2024-04-26T01:43:49.985486+0000","last_undegraded":"2024-04-26T01:43:49.985486+0000","last_fullsized":"2024-04-26T01:43:49.985486+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:20.593730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"95'5","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.162333+0000","last_change":"2024-04-26T01:42:05.394925+0000","last_active":"2024-04-26T01:44:17.162333+0000","last_peered":"2024-04-26T01:44:17.162333+0000","last_clean":"2024-04-26T01:44:17.162333+0000","last_became_active":"2024-04-26T01:42:05.394531+0000","last_became_peered":"2024-04-26T01:42:05.394531+0000","last_unstale":"2024-04-26T01:44:17.162333+0000","last_undegraded":"2024-04-26T01:44:17.162333+0000","last_fullsized":"2024-04-26T01:44:17.162333+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:33.177780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.310879+0000","last_change":"2024-04-26T01:42:02.237482+0000","last_active":"2024-04-26T01:43:50.310879+0000","last_peered":"2024-04-26T01:43:50.310879+0000","last_clean":"2024-04-26T01:43:50.310879+0000","last_became_active":"2024-04-26T01:42:02.236752+0000","last_became_peered":"2024-04-26T01:42:02.236752+0000","last_unstale":"2024-04-26T01:43:50.310879+0000","last_undegraded":"2024-04-26T01:43:50.310879+0000","last_fullsized":"2024-04-26T01:43:50.310879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:27.211245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"95'5","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.187385+0000","last_change":"2024-04-26T01:42:05.402157+0000","last_active":"2024-04-26T01:44:17.187385+0000","last_peered":"2024-04-26T01:44:17.187385+0000","last_clean":"2024-04-26T01:44:17.187385+0000","last_became_active":"2024-04-26T01:42:05.401889+0000","last_became_peered":"2024-04-26T01:42:05.401889+0000","last_unstale":"2024-04-26T01:44:17.187385+0000","last_undegraded":"2024-04-26T01:44:17.187385+0000","last_fullsized":"2024-04-26T01:44:17.187385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:33.854901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.108839+0000","last_change":"2024-04-26T01:42:01.339675+0000","last_active":"2024-04-26T01:43:50.108839+0000","last_peered":"2024-04-26T01:43:50.108839+0000","last_clean":"2024-04-26T01:43:50.108839+0000","last_became_active":"2024-04-26T01:42:01.339247+0000","last_became_peered":"2024-04-26T01:42:01.339247+0000","last_unstale":"2024-04-26T01:43:50.108839+0000","last_undegraded":"2024-04-26T01:43:50.108839+0000","last_fullsized":"2024-04-26T01:43:50.108839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:32.724830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"95'7","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.185556+0000","last_change":"2024-04-26T01:42:05.412913+0000","last_active":"2024-04-26T01:44:17.185556+0000","last_peered":"2024-04-26T01:44:17.185556+0000","last_clean":"2024-04-26T01:44:17.185556+0000","last_became_active":"2024-04-26T01:42:05.412556+0000","last_became_peered":"2024-04-26T01:42:05.412556+0000","last_unstale":"2024-04-26T01:44:17.185556+0000","last_undegraded":"2024-04-26T01:44:17.185556+0000","last_fullsized":"2024-04-26T01:44:17.185556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:00:52.798069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.156940+0000","last_change":"2024-04-26T01:42:05.391235+0000","last_active":"2024-04-26T01:44:17.156940+0000","last_peered":"2024-04-26T01:44:17.156940+0000","last_clean":"2024-04-26T01:44:17.156940+0000","last_became_active":"2024-04-26T01:42:05.390717+0000","last_became_peered":"2024-04-26T01:42:05.390717+0000","last_unstale":"2024-04-26T01:44:17.156940+0000","last_undegraded":"2024-04-26T01:44:17.156940+0000","last_fullsized":"2024-04-26T01:44:17.156940+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:07.541593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"95'5","reported_seq":150,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:19.724175+0000","last_change":"2024-04-26T01:42:02.232823+0000","last_active":"2024-04-26T01:44:19.724175+0000","last_peered":"2024-04-26T01:44:19.724175+0000","last_clean":"2024-04-26T01:44:19.724175+0000","last_became_active":"2024-04-26T01:42:02.231899+0000","last_became_peered":"2024-04-26T01:42:02.231899+0000","last_unstale":"2024-04-26T01:44:19.724175+0000","last_undegraded":"2024-04-26T01:44:19.724175+0000","last_fullsized":"2024-04-26T01:44:19.724175+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:23.287305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.177388+0000","last_change":"2024-04-26T01:42:05.405690+0000","last_active":"2024-04-26T01:44:17.177388+0000","last_peered":"2024-04-26T01:44:17.177388+0000","last_clean":"2024-04-26T01:44:17.177388+0000","last_became_active":"2024-04-26T01:42:05.405345+0000","last_became_peered":"2024-04-26T01:42:05.405345+0000","last_unstale":"2024-04-26T01:44:17.177388+0000","last_undegraded":"2024-04-26T01:44:17.177388+0000","last_fullsized":"2024-04-26T01:44:17.177388+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:48.377916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.985226+0000","last_change":"2024-04-26T01:42:01.335721+0000","last_active":"2024-04-26T01:43:49.985226+0000","last_peered":"2024-04-26T01:43:49.985226+0000","last_clean":"2024-04-26T01:43:49.985226+0000","last_became_active":"2024-04-26T01:42:01.335149+0000","last_became_peered":"2024-04-26T01:42:01.335149+0000","last_unstale":"2024-04-26T01:43:49.985226+0000","last_undegraded":"2024-04-26T01:43:49.985226+0000","last_fullsized":"2024-04-26T01:43:49.985226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:36.372033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":366,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.108654+0000","last_change":"2024-04-26T01:40:28.086333+0000","last_active":"2024-04-26T01:43:50.108654+0000","last_peered":"2024-04-26T01:43:50.108654+0000","last_clean":"2024-04-26T01:43:50.108654+0000","last_became_active":"2024-04-26T01:40:28.085902+0000","last_became_peered":"2024-04-26T01:40:28.085902+0000","last_unstale":"2024-04-26T01:43:50.108654+0000","last_undegraded":"2024-04-26T01:43:50.108654+0000","last_fullsized":"2024-04-26T01:43:50.108654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_clean_scrub_stamp":"2024-04-26T01:36:35.357698+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:43.290117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"95'5","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.174305+0000","last_change":"2024-04-26T01:42:05.401896+0000","last_active":"2024-04-26T01:44:17.174305+0000","last_peered":"2024-04-26T01:44:17.174305+0000","last_clean":"2024-04-26T01:44:17.174305+0000","last_became_active":"2024-04-26T01:42:05.401496+0000","last_became_peered":"2024-04-26T01:42:05.401496+0000","last_unstale":"2024-04-26T01:44:17.174305+0000","last_undegraded":"2024-04-26T01:44:17.174305+0000","last_fullsized":"2024-04-26T01:44:17.174305+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:49.758444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.108813+0000","last_change":"2024-04-26T01:42:01.329705+0000","last_active":"2024-04-26T01:43:50.108813+0000","last_peered":"2024-04-26T01:43:50.108813+0000","last_clean":"2024-04-26T01:43:50.108813+0000","last_became_active":"2024-04-26T01:42:01.329153+0000","last_became_peered":"2024-04-26T01:42:01.329153+0000","last_unstale":"2024-04-26T01:43:50.108813+0000","last_undegraded":"2024-04-26T01:43:50.108813+0000","last_fullsized":"2024-04-26T01:43:50.108813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:06.747378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.985397+0000","last_change":"2024-04-26T01:42:02.232148+0000","last_active":"2024-04-26T01:43:49.985397+0000","last_peered":"2024-04-26T01:43:49.985397+0000","last_clean":"2024-04-26T01:43:49.985397+0000","last_became_active":"2024-04-26T01:42:02.231739+0000","last_became_peered":"2024-04-26T01:42:02.231739+0000","last_unstale":"2024-04-26T01:43:49.985397+0000","last_undegraded":"2024-04-26T01:43:49.985397+0000","last_fullsized":"2024-04-26T01:43:49.985397+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:09.752433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"95'9","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.182401+0000","last_change":"2024-04-26T01:42:05.399938+0000","last_active":"2024-04-26T01:44:17.182401+0000","last_peered":"2024-04-26T01:44:17.182401+0000","last_clean":"2024-04-26T01:44:17.182401+0000","last_became_active":"2024-04-26T01:42:05.399550+0000","last_became_peered":"2024-04-26T01:42:05.399550+0000","last_unstale":"2024-04-26T01:44:17.182401+0000","last_undegraded":"2024-04-26T01:44:17.182401+0000","last_fullsized":"2024-04-26T01:44:17.182401+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:44.142284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"93'12","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.311223+0000","last_change":"2024-04-26T01:42:01.337437+0000","last_active":"2024-04-26T01:43:50.311223+0000","last_peered":"2024-04-26T01:43:50.311223+0000","last_clean":"2024-04-26T01:43:50.311223+0000","last_became_active":"2024-04-26T01:42:01.336894+0000","last_became_peered":"2024-04-26T01:42:01.336894+0000","last_unstale":"2024-04-26T01:43:50.311223+0000","last_undegraded":"2024-04-26T01:43:50.311223+0000","last_fullsized":"2024-04-26T01:43:50.311223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:04:31.093273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"95'5","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.173990+0000","last_change":"2024-04-26T01:42:05.411694+0000","last_active":"2024-04-26T01:44:17.173990+0000","last_peered":"2024-04-26T01:44:17.173990+0000","last_clean":"2024-04-26T01:44:17.173990+0000","last_became_active":"2024-04-26T01:42:05.411300+0000","last_became_peered":"2024-04-26T01:42:05.411300+0000","last_unstale":"2024-04-26T01:44:17.173990+0000","last_undegraded":"2024-04-26T01:44:17.173990+0000","last_fullsized":"2024-04-26T01:44:17.173990+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:42:08.116145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.985445+0000","last_change":"2024-04-26T01:42:01.427501+0000","last_active":"2024-04-26T01:43:49.985445+0000","last_peered":"2024-04-26T01:43:49.985445+0000","last_clean":"2024-04-26T01:43:49.985445+0000","last_became_active":"2024-04-26T01:42:01.426970+0000","last_became_peered":"2024-04-26T01:42:01.426970+0000","last_unstale":"2024-04-26T01:43:49.985445+0000","last_undegraded":"2024-04-26T01:43:49.985445+0000","last_fullsized":"2024-04-26T01:43:49.985445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:53:53.118296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"95'7","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.186200+0000","last_change":"2024-04-26T01:42:05.414790+0000","last_active":"2024-04-26T01:44:17.186200+0000","last_peered":"2024-04-26T01:44:17.186200+0000","last_clean":"2024-04-26T01:44:17.186200+0000","last_became_active":"2024-04-26T01:42:05.411947+0000","last_became_peered":"2024-04-26T01:42:05.411947+0000","last_unstale":"2024-04-26T01:44:17.186200+0000","last_undegraded":"2024-04-26T01:44:17.186200+0000","last_fullsized":"2024-04-26T01:44:17.186200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:11:34.450815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.989082+0000","last_change":"2024-04-26T01:42:02.230539+0000","last_active":"2024-04-26T01:43:49.989082+0000","last_peered":"2024-04-26T01:43:49.989082+0000","last_clean":"2024-04-26T01:43:49.989082+0000","last_became_active":"2024-04-26T01:42:02.230150+0000","last_became_peered":"2024-04-26T01:42:02.230150+0000","last_unstale":"2024-04-26T01:43:49.989082+0000","last_undegraded":"2024-04-26T01:43:49.989082+0000","last_fullsized":"2024-04-26T01:43:49.989082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:08:03.073069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"95'2","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.150118+0000","last_change":"2024-04-26T01:42:05.388877+0000","last_active":"2024-04-26T01:44:17.150118+0000","last_peered":"2024-04-26T01:44:17.150118+0000","last_clean":"2024-04-26T01:44:17.150118+0000","last_became_active":"2024-04-26T01:42:05.388457+0000","last_became_peered":"2024-04-26T01:42:05.388457+0000","last_unstale":"2024-04-26T01:44:17.150118+0000","last_undegraded":"2024-04-26T01:44:17.150118+0000","last_fullsized":"2024-04-26T01:44:17.150118+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:54.436859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.989575+0000","last_change":"2024-04-26T01:42:01.343461+0000","last_active":"2024-04-26T01:43:49.989575+0000","last_peered":"2024-04-26T01:43:49.989575+0000","last_clean":"2024-04-26T01:43:49.989575+0000","last_became_active":"2024-04-26T01:42:01.342713+0000","last_became_peered":"2024-04-26T01:42:01.342713+0000","last_unstale":"2024-04-26T01:43:49.989575+0000","last_undegraded":"2024-04-26T01:43:49.989575+0000","last_fullsized":"2024-04-26T01:43:49.989575+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:38.471858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"95'9","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.178839+0000","last_change":"2024-04-26T01:42:05.415101+0000","last_active":"2024-04-26T01:44:17.178839+0000","last_peered":"2024-04-26T01:44:17.178839+0000","last_clean":"2024-04-26T01:44:17.178839+0000","last_became_active":"2024-04-26T01:42:05.414745+0000","last_became_peered":"2024-04-26T01:42:05.414745+0000","last_unstale":"2024-04-26T01:44:17.178839+0000","last_undegraded":"2024-04-26T01:44:17.178839+0000","last_fullsized":"2024-04-26T01:44:17.178839+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:13:08.573547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.308613+0000","last_change":"2024-04-26T01:42:01.339501+0000","last_active":"2024-04-26T01:43:50.308613+0000","last_peered":"2024-04-26T01:43:50.308613+0000","last_clean":"2024-04-26T01:43:50.308613+0000","last_became_active":"2024-04-26T01:42:01.339132+0000","last_became_peered":"2024-04-26T01:42:01.339132+0000","last_unstale":"2024-04-26T01:43:50.308613+0000","last_undegraded":"2024-04-26T01:43:50.308613+0000","last_fullsized":"2024-04-26T01:43:50.308613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:16.463099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.187446+0000","last_change":"2024-04-26T01:42:05.417864+0000","last_active":"2024-04-26T01:44:17.187446+0000","last_peered":"2024-04-26T01:44:17.187446+0000","last_clean":"2024-04-26T01:44:17.187446+0000","last_became_active":"2024-04-26T01:42:05.417414+0000","last_became_peered":"2024-04-26T01:42:05.417414+0000","last_unstale":"2024-04-26T01:44:17.187446+0000","last_undegraded":"2024-04-26T01:44:17.187446+0000","last_fullsized":"2024-04-26T01:44:17.187446+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:23:12.793421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.985155+0000","last_change":"2024-04-26T01:42:49.063769+0000","last_active":"2024-04-26T01:43:49.985155+0000","last_peered":"2024-04-26T01:43:49.985155+0000","last_clean":"2024-04-26T01:43:49.985155+0000","last_became_active":"2024-04-26T01:42:49.057463+0000","last_became_peered":"2024-04-26T01:42:49.057463+0000","last_unstale":"2024-04-26T01:43:49.985155+0000","last_undegraded":"2024-04-26T01:43:49.985155+0000","last_fullsized":"2024-04-26T01:43:49.985155+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:34:54.511304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"95'5","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.191252+0000","last_change":"2024-04-26T01:42:05.397980+0000","last_active":"2024-04-26T01:44:17.191252+0000","last_peered":"2024-04-26T01:44:17.191252+0000","last_clean":"2024-04-26T01:44:17.191252+0000","last_became_active":"2024-04-26T01:42:05.397589+0000","last_became_peered":"2024-04-26T01:42:05.397589+0000","last_unstale":"2024-04-26T01:44:17.191252+0000","last_undegraded":"2024-04-26T01:44:17.191252+0000","last_fullsized":"2024-04-26T01:44:17.191252+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:39:55.863566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.308969+0000","last_change":"2024-04-26T01:42:02.234313+0000","last_active":"2024-04-26T01:43:50.308969+0000","last_peered":"2024-04-26T01:43:50.308969+0000","last_clean":"2024-04-26T01:43:50.308969+0000","last_became_active":"2024-04-26T01:42:02.233873+0000","last_became_peered":"2024-04-26T01:42:02.233873+0000","last_unstale":"2024-04-26T01:43:50.308969+0000","last_undegraded":"2024-04-26T01:43:50.308969+0000","last_fullsized":"2024-04-26T01:43:50.308969+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:34.092434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"95'7","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.168261+0000","last_change":"2024-04-26T01:42:05.387435+0000","last_active":"2024-04-26T01:44:17.168261+0000","last_peered":"2024-04-26T01:44:17.168261+0000","last_clean":"2024-04-26T01:44:17.168261+0000","last_became_active":"2024-04-26T01:42:05.386964+0000","last_became_peered":"2024-04-26T01:42:05.386964+0000","last_unstale":"2024-04-26T01:44:17.168261+0000","last_undegraded":"2024-04-26T01:44:17.168261+0000","last_fullsized":"2024-04-26T01:44:17.168261+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:12:27.905842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.989474+0000","last_change":"2024-04-26T01:42:01.326432+0000","last_active":"2024-04-26T01:43:49.989474+0000","last_peered":"2024-04-26T01:43:49.989474+0000","last_clean":"2024-04-26T01:43:49.989474+0000","last_became_active":"2024-04-26T01:42:01.325989+0000","last_became_peered":"2024-04-26T01:42:01.325989+0000","last_unstale":"2024-04-26T01:43:49.989474+0000","last_undegraded":"2024-04-26T01:43:49.989474+0000","last_fullsized":"2024-04-26T01:43:49.989474+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:10.049563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"95'7","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.180957+0000","last_change":"2024-04-26T01:42:05.412802+0000","last_active":"2024-04-26T01:44:17.180957+0000","last_peered":"2024-04-26T01:44:17.180957+0000","last_clean":"2024-04-26T01:44:17.180957+0000","last_became_active":"2024-04-26T01:42:05.412443+0000","last_became_peered":"2024-04-26T01:42:05.412443+0000","last_unstale":"2024-04-26T01:44:17.180957+0000","last_undegraded":"2024-04-26T01:44:17.180957+0000","last_fullsized":"2024-04-26T01:44:17.180957+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:28:56.360778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'2","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.311619+0000","last_change":"2024-04-26T01:42:49.080305+0000","last_active":"2024-04-26T01:43:50.311619+0000","last_peered":"2024-04-26T01:43:50.311619+0000","last_clean":"2024-04-26T01:43:50.311619+0000","last_became_active":"2024-04-26T01:42:49.074476+0000","last_became_peered":"2024-04-26T01:42:49.074476+0000","last_unstale":"2024-04-26T01:43:50.311619+0000","last_undegraded":"2024-04-26T01:43:50.311619+0000","last_fullsized":"2024-04-26T01:43:50.311619+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:58:06.619961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"95'9","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.162972+0000","last_change":"2024-04-26T01:42:05.387621+0000","last_active":"2024-04-26T01:44:17.162972+0000","last_peered":"2024-04-26T01:44:17.162972+0000","last_clean":"2024-04-26T01:44:17.162972+0000","last_became_active":"2024-04-26T01:42:05.387165+0000","last_became_peered":"2024-04-26T01:42:05.387165+0000","last_unstale":"2024-04-26T01:44:17.162972+0000","last_undegraded":"2024-04-26T01:44:17.162972+0000","last_fullsized":"2024-04-26T01:44:17.162972+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:37.803330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"95'9","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.162500+0000","last_change":"2024-04-26T01:42:05.386787+0000","last_active":"2024-04-26T01:44:17.162500+0000","last_peered":"2024-04-26T01:44:17.162500+0000","last_clean":"2024-04-26T01:44:17.162500+0000","last_became_active":"2024-04-26T01:42:05.386410+0000","last_became_peered":"2024-04-26T01:42:05.386410+0000","last_unstale":"2024-04-26T01:44:17.162500+0000","last_undegraded":"2024-04-26T01:44:17.162500+0000","last_fullsized":"2024-04-26T01:44:17.162500+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:42.137191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"95'12","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.161732+0000","last_change":"2024-04-26T01:42:05.392142+0000","last_active":"2024-04-26T01:44:17.161732+0000","last_peered":"2024-04-26T01:44:17.161732+0000","last_clean":"2024-04-26T01:44:17.161732+0000","last_became_active":"2024-04-26T01:42:05.391829+0000","last_became_peered":"2024-04-26T01:42:05.391829+0000","last_unstale":"2024-04-26T01:44:17.161732+0000","last_undegraded":"2024-04-26T01:44:17.161732+0000","last_fullsized":"2024-04-26T01:44:17.161732+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:00:25.934641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.180591+0000","last_change":"2024-04-26T01:42:05.409184+0000","last_active":"2024-04-26T01:44:17.180591+0000","last_peered":"2024-04-26T01:44:17.180591+0000","last_clean":"2024-04-26T01:44:17.180591+0000","last_became_active":"2024-04-26T01:42:05.408894+0000","last_became_peered":"2024-04-26T01:42:05.408894+0000","last_unstale":"2024-04-26T01:44:17.180591+0000","last_undegraded":"2024-04-26T01:44:17.180591+0000","last_fullsized":"2024-04-26T01:44:17.180591+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:53.291844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"95'7","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.191169+0000","last_change":"2024-04-26T01:42:05.406940+0000","last_active":"2024-04-26T01:44:17.191169+0000","last_peered":"2024-04-26T01:44:17.191169+0000","last_clean":"2024-04-26T01:44:17.191169+0000","last_became_active":"2024-04-26T01:42:05.406476+0000","last_became_peered":"2024-04-26T01:42:05.406476+0000","last_unstale":"2024-04-26T01:44:17.191169+0000","last_undegraded":"2024-04-26T01:44:17.191169+0000","last_fullsized":"2024-04-26T01:44:17.191169+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:21:37.734633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"95'4","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.189162+0000","last_change":"2024-04-26T01:42:05.413521+0000","last_active":"2024-04-26T01:44:17.189162+0000","last_peered":"2024-04-26T01:44:17.189162+0000","last_clean":"2024-04-26T01:44:17.189162+0000","last_became_active":"2024-04-26T01:42:05.412789+0000","last_became_peered":"2024-04-26T01:42:05.412789+0000","last_unstale":"2024-04-26T01:44:17.189162+0000","last_undegraded":"2024-04-26T01:44:17.189162+0000","last_fullsized":"2024-04-26T01:44:17.189162+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:13.471376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"95'6","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.161927+0000","last_change":"2024-04-26T01:42:05.391395+0000","last_active":"2024-04-26T01:44:17.161927+0000","last_peered":"2024-04-26T01:44:17.161927+0000","last_clean":"2024-04-26T01:44:17.161927+0000","last_became_active":"2024-04-26T01:42:05.391048+0000","last_became_peered":"2024-04-26T01:42:05.391048+0000","last_unstale":"2024-04-26T01:44:17.161927+0000","last_undegraded":"2024-04-26T01:44:17.161927+0000","last_fullsized":"2024-04-26T01:44:17.161927+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:45:56.645454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"95'15","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.168013+0000","last_change":"2024-04-26T01:42:05.386868+0000","last_active":"2024-04-26T01:44:17.168013+0000","last_peered":"2024-04-26T01:44:17.168013+0000","last_clean":"2024-04-26T01:44:17.168013+0000","last_became_active":"2024-04-26T01:42:05.386483+0000","last_became_peered":"2024-04-26T01:42:05.386483+0000","last_unstale":"2024-04-26T01:44:17.168013+0000","last_undegraded":"2024-04-26T01:44:17.168013+0000","last_fullsized":"2024-04-26T01:44:17.168013+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:42:44.453076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"95'2","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.173530+0000","last_change":"2024-04-26T01:42:05.406711+0000","last_active":"2024-04-26T01:44:17.173530+0000","last_peered":"2024-04-26T01:44:17.173530+0000","last_clean":"2024-04-26T01:44:17.173530+0000","last_became_active":"2024-04-26T01:42:05.406318+0000","last_became_peered":"2024-04-26T01:42:05.406318+0000","last_unstale":"2024-04-26T01:44:17.173530+0000","last_undegraded":"2024-04-26T01:44:17.173530+0000","last_fullsized":"2024-04-26T01:44:17.173530+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:36:16.599678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"95'6","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.176898+0000","last_change":"2024-04-26T01:42:05.406305+0000","last_active":"2024-04-26T01:44:17.176898+0000","last_peered":"2024-04-26T01:44:17.176898+0000","last_clean":"2024-04-26T01:44:17.176898+0000","last_became_active":"2024-04-26T01:42:05.405953+0000","last_became_peered":"2024-04-26T01:42:05.405953+0000","last_unstale":"2024-04-26T01:44:17.176898+0000","last_undegraded":"2024-04-26T01:44:17.176898+0000","last_fullsized":"2024-04-26T01:44:17.176898+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:36.588672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.174739+0000","last_change":"2024-04-26T01:42:05.413757+0000","last_active":"2024-04-26T01:44:17.174739+0000","last_peered":"2024-04-26T01:44:17.174739+0000","last_clean":"2024-04-26T01:44:17.174739+0000","last_became_active":"2024-04-26T01:42:05.413410+0000","last_became_peered":"2024-04-26T01:42:05.413410+0000","last_unstale":"2024-04-26T01:44:17.174739+0000","last_undegraded":"2024-04-26T01:44:17.174739+0000","last_fullsized":"2024-04-26T01:44:17.174739+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:30.298443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"95'10","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.180834+0000","last_change":"2024-04-26T01:42:05.412268+0000","last_active":"2024-04-26T01:44:17.180834+0000","last_peered":"2024-04-26T01:44:17.180834+0000","last_clean":"2024-04-26T01:44:17.180834+0000","last_became_active":"2024-04-26T01:42:05.411804+0000","last_became_peered":"2024-04-26T01:42:05.411804+0000","last_unstale":"2024-04-26T01:44:17.180834+0000","last_undegraded":"2024-04-26T01:44:17.180834+0000","last_fullsized":"2024-04-26T01:44:17.180834+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:47.277766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"95'11","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.193062+0000","last_change":"2024-04-26T01:42:05.415563+0000","last_active":"2024-04-26T01:44:17.193062+0000","last_peered":"2024-04-26T01:44:17.193062+0000","last_clean":"2024-04-26T01:44:17.193062+0000","last_became_active":"2024-04-26T01:42:05.414818+0000","last_became_peered":"2024-04-26T01:42:05.414818+0000","last_unstale":"2024-04-26T01:44:17.193062+0000","last_undegraded":"2024-04-26T01:44:17.193062+0000","last_fullsized":"2024-04-26T01:44:17.193062+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:54.713595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.162110+0000","last_change":"2024-04-26T01:42:05.392280+0000","last_active":"2024-04-26T01:44:17.162110+0000","last_peered":"2024-04-26T01:44:17.162110+0000","last_clean":"2024-04-26T01:44:17.162110+0000","last_became_active":"2024-04-26T01:42:05.391943+0000","last_became_peered":"2024-04-26T01:42:05.391943+0000","last_unstale":"2024-04-26T01:44:17.162110+0000","last_undegraded":"2024-04-26T01:44:17.162110+0000","last_fullsized":"2024-04-26T01:44:17.162110+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:38:51.560596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":459,"ondisk_log_size":459,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69133,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":703,"num_read_kb":427,"num_write":175,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":207399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285614,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":510559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547249,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41208,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703944,"statfs":{"total":95995035648,"available":95952838656,"internally_reserved":0,"allocated":1753088,"data_stored":1166919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710776,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":510683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972416,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1060864,"data_stored":510491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266755,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28640,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716512,"statfs":{"total":95995035648,"available":95965708288,"internally_reserved":0,"allocated":1662976,"data_stored":1103206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430281,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":510072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40592,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704560,"statfs":{"total":95995035648,"available":95953469440,"internally_reserved":0,"allocated":1122304,"data_stored":574324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":511343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1146880,"data_stored":576206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":510383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888148,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1060864,"data_stored":510900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691919,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1675264,"data_stored":1102358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":66341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:44:22.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:44:23.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:22 smithi188 ceph-mon[32751]: pgmap v402: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T01:44:23.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:22 smithi188 ceph-mon[32751]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:44:23.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:22 smithi029 ceph-mon[32775]: pgmap v402: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T01:44:23.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:22 smithi029 ceph-mon[32775]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:44:23.235 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:23.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:22 smithi006 ceph-mon[28419]: pgmap v402: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T01:44:23.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:22 smithi006 ceph-mon[28419]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:44:25.108 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:24 smithi006 ceph-mon[28419]: pgmap v403: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T01:44:25.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:24 smithi188 ceph-mon[32751]: pgmap v403: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T01:44:25.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:24 smithi029 ceph-mon[32775]: pgmap v403: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T01:44:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.614+0000 7f07fea00700 1 -- 172.21.15.6:0/617115185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07f8102a90 msgr2=0x7f07f8102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.614+0000 7f07fea00700 1 --2- 172.21.15.6:0/617115185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07f8102a90 0x7f07f8102f10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f07e4009990 tx=0x7f07e4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.614+0000 7f07fea00700 1 -- 172.21.15.6:0/617115185 shutdown_connections 2024-04-26T01:44:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.614+0000 7f07fea00700 1 --2- 172.21.15.6:0/617115185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07f8103450 0x7f07f810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.614+0000 7f07fea00700 1 --2- 172.21.15.6:0/617115185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f8108a50 0x7f07f8108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.614+0000 7f07fea00700 1 --2- 172.21.15.6:0/617115185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07f8102a90 0x7f07f8102f10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.614+0000 7f07fea00700 1 -- 172.21.15.6:0/617115185 >> 172.21.15.6:0/617115185 conn(0x7f07f8079ba0 msgr2=0x7f07f8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.614+0000 7f07fea00700 1 -- 172.21.15.6:0/617115185 shutdown_connections 2024-04-26T01:44:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.614+0000 7f07fea00700 1 -- 172.21.15.6:0/617115185 wait complete. 2024-04-26T01:44:25.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.615+0000 7f07fea00700 1 Processor -- start 2024-04-26T01:44:25.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.615+0000 7f07fea00700 1 -- start start 2024-04-26T01:44:25.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.615+0000 7f07fea00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07f8102a90 0x7f07f81991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:25.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.615+0000 7f07fea00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f8103450 0x7f07f81996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:25.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.615+0000 7f07fea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07f8108a50 0x7f07f819da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:25.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.615+0000 7f07fea00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f07f8112be0 con 0x7f07f8108a50 2024-04-26T01:44:25.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.615+0000 7f07fea00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f07f8112a60 con 0x7f07f8103450 2024-04-26T01:44:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.616+0000 7f07fea00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f07f8112d60 con 0x7f07f8102a90 2024-04-26T01:44:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.616+0000 7f07fca00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07f8108a50 0x7f07f819da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.616+0000 7f07f7e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07f8102a90 0x7f07f81991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.616+0000 7f07f7e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07f8102a90 0x7f07f81991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:41800/0 (socket says 172.21.15.6:41800) 2024-04-26T01:44:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.616+0000 7f07f7e00700 1 -- 172.21.15.6:0/1416782167 learned_addr learned my addr 172.21.15.6:0/1416782167 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.616+0000 7f07f7400700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f8103450 0x7f07f81996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.616+0000 7f07fca00700 1 -- 172.21.15.6:0/1416782167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07f8102a90 msgr2=0x7f07f81991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.616+0000 7f07fca00700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07f8102a90 0x7f07f81991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.616+0000 7f07fca00700 1 -- 172.21.15.6:0/1416782167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f8103450 msgr2=0x7f07f81996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.616+0000 7f07fca00700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f8103450 0x7f07f81996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.616+0000 7f07fca00700 1 -- 172.21.15.6:0/1416782167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f819e1c0 con 0x7f07f8108a50 2024-04-26T01:44:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.616+0000 7f07f7400700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f8103450 0x7f07f81996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:25.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.617+0000 7f07fca00700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07f8108a50 0x7f07f819da80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f07f0007f10 tx=0x7f07f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:25.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.617+0000 7f07f4c00700 1 -- 172.21.15.6:0/1416782167 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f0004db0 con 0x7f07f8108a50 2024-04-26T01:44:25.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.617+0000 7f07f4c00700 1 -- 172.21.15.6:0/1416782167 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07f0003a00 con 0x7f07f8108a50 2024-04-26T01:44:25.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.617+0000 7f07f4c00700 1 -- 172.21.15.6:0/1416782167 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f0004580 con 0x7f07f8108a50 2024-04-26T01:44:25.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.617+0000 7f07fea00700 1 -- 172.21.15.6:0/1416782167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f819e4b0 con 0x7f07f8108a50 2024-04-26T01:44:25.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.617+0000 7f07fea00700 1 -- 172.21.15.6:0/1416782167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f81a5d30 con 0x7f07f8108a50 2024-04-26T01:44:25.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.619+0000 7f07fea00700 1 -- 172.21.15.6:0/1416782167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f8069730 con 0x7f07f8108a50 2024-04-26T01:44:25.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.621+0000 7f07f4c00700 1 -- 172.21.15.6:0/1416782167 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f07f001c020 con 0x7f07f8108a50 2024-04-26T01:44:25.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.622+0000 7f07f4c00700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07dc0a3c40 0x7f07dc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:25.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.622+0000 7f07f4c00700 1 -- 172.21.15.6:0/1416782167 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f07f00cfbf0 con 0x7f07f8108a50 2024-04-26T01:44:25.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.622+0000 7f07f7e00700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07dc0a3c40 0x7f07dc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:25.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.623+0000 7f07f7e00700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07dc0a3c40 0x7f07dc0a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009ca0 tx=0x7f07e8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:25.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.627+0000 7f07f4c00700 1 -- 172.21.15.6:0/1416782167 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f07f0010070 con 0x7f07f8108a50 2024-04-26T01:44:25.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.856+0000 7f07fea00700 1 -- 172.21.15.6:0/1416782167 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f07f819a510 con 0x7f07dc0a3c40 2024-04-26T01:44:25.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.866+0000 7f07f4c00700 1 -- 172.21.15.6:0/1416782167 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258924 (secure 0 0 0) 0x7f07f819a510 con 0x7f07dc0a3c40 2024-04-26T01:44:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:44:25.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.874+0000 7f07fea00700 1 -- 172.21.15.6:0/1416782167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07dc0a3c40 msgr2=0x7f07dc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:25.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.874+0000 7f07fea00700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07dc0a3c40 0x7f07dc0a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009ca0 tx=0x7f07e8009260 comp rx=0 tx=0).stop 2024-04-26T01:44:25.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.874+0000 7f07fea00700 1 -- 172.21.15.6:0/1416782167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07f8108a50 msgr2=0x7f07f819da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:25.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.874+0000 7f07fea00700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07f8108a50 0x7f07f819da80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f07f0007f10 tx=0x7f07f000d3b0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.874+0000 7f07fea00700 1 -- 172.21.15.6:0/1416782167 shutdown_connections 2024-04-26T01:44:25.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.874+0000 7f07fea00700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07f8108a50 0x7f07f819da80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.874+0000 7f07fea00700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f07dc0a3c40 0x7f07dc0a6100 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.874+0000 7f07fea00700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07f8102a90 0x7f07f81991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.874+0000 7f07fea00700 1 --2- 172.21.15.6:0/1416782167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f8103450 0x7f07f81996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.874+0000 7f07fea00700 1 -- 172.21.15.6:0/1416782167 >> 172.21.15.6:0/1416782167 conn(0x7f07f8079ba0 msgr2=0x7f07f8101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.874+0000 7f07fea00700 1 -- 172.21.15.6:0/1416782167 shutdown_connections 2024-04-26T01:44:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:25.874+0000 7f07fea00700 1 -- 172.21.15.6:0/1416782167 wait complete. 2024-04-26T01:44:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-26T01:44:26.675 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-04-26T01:44:25.231169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659735,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":865,"num_read_kb":557,"num_write":360,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392744,"kb_used_data":14472,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549080,"statfs":{"total":1151940427776,"available":1151538257920,"internally_reserved":0,"allocated":14819328,"data_stored":8097585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":43,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004505"},"pg_stats":[{"pgid":"3.3f","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.186201+0000","last_change":"2024-04-26T01:42:05.417304+0000","last_active":"2024-04-26T01:44:17.186201+0000","last_peered":"2024-04-26T01:44:17.186201+0000","last_clean":"2024-04-26T01:44:17.186201+0000","last_became_active":"2024-04-26T01:42:05.416999+0000","last_became_peered":"2024-04-26T01:42:05.416999+0000","last_unstale":"2024-04-26T01:44:17.186201+0000","last_undegraded":"2024-04-26T01:44:17.186201+0000","last_fullsized":"2024-04-26T01:44:17.186201+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:33:30.958943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"95'4","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.176097+0000","last_change":"2024-04-26T01:42:05.413089+0000","last_active":"2024-04-26T01:44:17.176097+0000","last_peered":"2024-04-26T01:44:17.176097+0000","last_clean":"2024-04-26T01:44:17.176097+0000","last_became_active":"2024-04-26T01:42:05.412776+0000","last_became_peered":"2024-04-26T01:42:05.412776+0000","last_unstale":"2024-04-26T01:44:17.176097+0000","last_undegraded":"2024-04-26T01:44:17.176097+0000","last_fullsized":"2024-04-26T01:44:17.176097+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:47.861340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"95'7","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.185817+0000","last_change":"2024-04-26T01:42:05.404864+0000","last_active":"2024-04-26T01:44:17.185817+0000","last_peered":"2024-04-26T01:44:17.185817+0000","last_clean":"2024-04-26T01:44:17.185817+0000","last_became_active":"2024-04-26T01:42:05.404529+0000","last_became_peered":"2024-04-26T01:42:05.404529+0000","last_unstale":"2024-04-26T01:44:17.185817+0000","last_undegraded":"2024-04-26T01:44:17.185817+0000","last_fullsized":"2024-04-26T01:44:17.185817+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:04:18.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.155446+0000","last_change":"2024-04-26T01:42:05.393582+0000","last_active":"2024-04-26T01:44:17.155446+0000","last_peered":"2024-04-26T01:44:17.155446+0000","last_clean":"2024-04-26T01:44:17.155446+0000","last_became_active":"2024-04-26T01:42:05.393193+0000","last_became_peered":"2024-04-26T01:42:05.393193+0000","last_unstale":"2024-04-26T01:44:17.155446+0000","last_undegraded":"2024-04-26T01:44:17.155446+0000","last_fullsized":"2024-04-26T01:44:17.155446+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:20:29.420459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"95'4","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.187247+0000","last_change":"2024-04-26T01:42:05.412979+0000","last_active":"2024-04-26T01:44:17.187247+0000","last_peered":"2024-04-26T01:44:17.187247+0000","last_clean":"2024-04-26T01:44:17.187247+0000","last_became_active":"2024-04-26T01:42:05.412679+0000","last_became_peered":"2024-04-26T01:42:05.412679+0000","last_unstale":"2024-04-26T01:44:17.187247+0000","last_undegraded":"2024-04-26T01:44:17.187247+0000","last_fullsized":"2024-04-26T01:44:17.187247+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:22.774960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"95'6","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.180795+0000","last_change":"2024-04-26T01:42:49.062507+0000","last_active":"2024-04-26T01:44:17.180795+0000","last_peered":"2024-04-26T01:44:17.180795+0000","last_clean":"2024-04-26T01:44:17.180795+0000","last_became_active":"2024-04-26T01:42:49.062095+0000","last_became_peered":"2024-04-26T01:42:49.062095+0000","last_unstale":"2024-04-26T01:44:17.180795+0000","last_undegraded":"2024-04-26T01:44:17.180795+0000","last_fullsized":"2024-04-26T01:44:17.180795+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:47.846697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"95'5","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.178020+0000","last_change":"2024-04-26T01:42:05.385221+0000","last_active":"2024-04-26T01:44:17.178020+0000","last_peered":"2024-04-26T01:44:17.178020+0000","last_clean":"2024-04-26T01:44:17.178020+0000","last_became_active":"2024-04-26T01:42:05.384856+0000","last_became_peered":"2024-04-26T01:42:05.384856+0000","last_unstale":"2024-04-26T01:44:17.178020+0000","last_undegraded":"2024-04-26T01:44:17.178020+0000","last_fullsized":"2024-04-26T01:44:17.178020+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:07:23.461324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"95'9","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.177700+0000","last_change":"2024-04-26T01:42:05.404754+0000","last_active":"2024-04-26T01:44:17.177700+0000","last_peered":"2024-04-26T01:44:17.177700+0000","last_clean":"2024-04-26T01:44:17.177700+0000","last_became_active":"2024-04-26T01:42:05.404220+0000","last_became_peered":"2024-04-26T01:42:05.404220+0000","last_unstale":"2024-04-26T01:44:17.177700+0000","last_undegraded":"2024-04-26T01:44:17.177700+0000","last_fullsized":"2024-04-26T01:44:17.177700+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:50.243552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"95'11","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.169628+0000","last_change":"2024-04-26T01:42:05.390972+0000","last_active":"2024-04-26T01:44:17.169628+0000","last_peered":"2024-04-26T01:44:17.169628+0000","last_clean":"2024-04-26T01:44:17.169628+0000","last_became_active":"2024-04-26T01:42:05.390599+0000","last_became_peered":"2024-04-26T01:42:05.390599+0000","last_unstale":"2024-04-26T01:44:17.169628+0000","last_undegraded":"2024-04-26T01:44:17.169628+0000","last_fullsized":"2024-04-26T01:44:17.169628+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:19:46.328640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"95'16","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.167601+0000","last_change":"2024-04-26T01:42:05.392320+0000","last_active":"2024-04-26T01:44:17.167601+0000","last_peered":"2024-04-26T01:44:17.167601+0000","last_clean":"2024-04-26T01:44:17.167601+0000","last_became_active":"2024-04-26T01:42:05.391986+0000","last_became_peered":"2024-04-26T01:42:05.391986+0000","last_unstale":"2024-04-26T01:44:17.167601+0000","last_undegraded":"2024-04-26T01:44:17.167601+0000","last_fullsized":"2024-04-26T01:44:17.167601+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:17:27.302358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"95'6","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.167895+0000","last_change":"2024-04-26T01:42:05.388077+0000","last_active":"2024-04-26T01:44:17.167895+0000","last_peered":"2024-04-26T01:44:17.167895+0000","last_clean":"2024-04-26T01:44:17.167895+0000","last_became_active":"2024-04-26T01:42:05.387545+0000","last_became_peered":"2024-04-26T01:42:05.387545+0000","last_unstale":"2024-04-26T01:44:17.167895+0000","last_undegraded":"2024-04-26T01:44:17.167895+0000","last_fullsized":"2024-04-26T01:44:17.167895+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:51.288177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"95'7","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.170949+0000","last_change":"2024-04-26T01:42:49.067449+0000","last_active":"2024-04-26T01:44:17.170949+0000","last_peered":"2024-04-26T01:44:17.170949+0000","last_clean":"2024-04-26T01:44:17.170949+0000","last_became_active":"2024-04-26T01:42:49.065991+0000","last_became_peered":"2024-04-26T01:42:49.065991+0000","last_unstale":"2024-04-26T01:44:17.170949+0000","last_undegraded":"2024-04-26T01:44:17.170949+0000","last_fullsized":"2024-04-26T01:44:17.170949+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:13:36.565257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"95'10","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.193148+0000","last_change":"2024-04-26T01:42:05.402847+0000","last_active":"2024-04-26T01:44:17.193148+0000","last_peered":"2024-04-26T01:44:17.193148+0000","last_clean":"2024-04-26T01:44:17.193148+0000","last_became_active":"2024-04-26T01:42:05.402503+0000","last_became_peered":"2024-04-26T01:42:05.402503+0000","last_unstale":"2024-04-26T01:44:17.193148+0000","last_undegraded":"2024-04-26T01:44:17.193148+0000","last_fullsized":"2024-04-26T01:44:17.193148+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:06.935060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"95'7","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.185541+0000","last_change":"2024-04-26T01:42:05.405257+0000","last_active":"2024-04-26T01:44:17.185541+0000","last_peered":"2024-04-26T01:44:17.185541+0000","last_clean":"2024-04-26T01:44:17.185541+0000","last_became_active":"2024-04-26T01:42:05.404551+0000","last_became_peered":"2024-04-26T01:42:05.404551+0000","last_unstale":"2024-04-26T01:44:17.185541+0000","last_undegraded":"2024-04-26T01:44:17.185541+0000","last_fullsized":"2024-04-26T01:44:17.185541+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:20.754088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"95'7","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.184893+0000","last_change":"2024-04-26T01:42:05.408847+0000","last_active":"2024-04-26T01:44:17.184893+0000","last_peered":"2024-04-26T01:44:17.184893+0000","last_clean":"2024-04-26T01:44:17.184893+0000","last_became_active":"2024-04-26T01:42:05.406189+0000","last_became_peered":"2024-04-26T01:42:05.406189+0000","last_unstale":"2024-04-26T01:44:17.184893+0000","last_undegraded":"2024-04-26T01:44:17.184893+0000","last_fullsized":"2024-04-26T01:44:17.184893+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:59:58.633630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"95'9","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.179233+0000","last_change":"2024-04-26T01:42:05.404647+0000","last_active":"2024-04-26T01:44:17.179233+0000","last_peered":"2024-04-26T01:44:17.179233+0000","last_clean":"2024-04-26T01:44:17.179233+0000","last_became_active":"2024-04-26T01:42:05.404265+0000","last_became_peered":"2024-04-26T01:42:05.404265+0000","last_unstale":"2024-04-26T01:44:17.179233+0000","last_undegraded":"2024-04-26T01:44:17.179233+0000","last_fullsized":"2024-04-26T01:44:17.179233+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:00:43.472541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"95'8","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.186395+0000","last_change":"2024-04-26T01:42:05.417602+0000","last_active":"2024-04-26T01:44:17.186395+0000","last_peered":"2024-04-26T01:44:17.186395+0000","last_clean":"2024-04-26T01:44:17.186395+0000","last_became_active":"2024-04-26T01:42:05.417144+0000","last_became_peered":"2024-04-26T01:42:05.417144+0000","last_unstale":"2024-04-26T01:44:17.186395+0000","last_undegraded":"2024-04-26T01:44:17.186395+0000","last_fullsized":"2024-04-26T01:44:17.186395+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:34.043597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"95'10","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.183054+0000","last_change":"2024-04-26T01:42:05.392948+0000","last_active":"2024-04-26T01:44:17.183054+0000","last_peered":"2024-04-26T01:44:17.183054+0000","last_clean":"2024-04-26T01:44:17.183054+0000","last_became_active":"2024-04-26T01:42:05.392040+0000","last_became_peered":"2024-04-26T01:42:05.392040+0000","last_unstale":"2024-04-26T01:44:17.183054+0000","last_undegraded":"2024-04-26T01:44:17.183054+0000","last_fullsized":"2024-04-26T01:44:17.183054+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:17.307979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"95'7","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.189280+0000","last_change":"2024-04-26T01:42:05.406588+0000","last_active":"2024-04-26T01:44:17.189280+0000","last_peered":"2024-04-26T01:44:17.189280+0000","last_clean":"2024-04-26T01:44:17.189280+0000","last_became_active":"2024-04-26T01:42:05.406200+0000","last_became_peered":"2024-04-26T01:42:05.406200+0000","last_unstale":"2024-04-26T01:44:17.189280+0000","last_undegraded":"2024-04-26T01:44:17.189280+0000","last_fullsized":"2024-04-26T01:44:17.189280+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:14.095383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.311220+0000","last_change":"2024-04-26T01:42:01.339289+0000","last_active":"2024-04-26T01:43:50.311220+0000","last_peered":"2024-04-26T01:43:50.311220+0000","last_clean":"2024-04-26T01:43:50.311220+0000","last_became_active":"2024-04-26T01:42:01.335402+0000","last_became_peered":"2024-04-26T01:42:01.335402+0000","last_unstale":"2024-04-26T01:43:50.311220+0000","last_undegraded":"2024-04-26T01:43:50.311220+0000","last_fullsized":"2024-04-26T01:43:50.311220+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:51.411524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"95'5","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.172713+0000","last_change":"2024-04-26T01:42:05.409569+0000","last_active":"2024-04-26T01:44:17.172713+0000","last_peered":"2024-04-26T01:44:17.172713+0000","last_clean":"2024-04-26T01:44:17.172713+0000","last_became_active":"2024-04-26T01:42:05.407441+0000","last_became_peered":"2024-04-26T01:42:05.407441+0000","last_unstale":"2024-04-26T01:44:17.172713+0000","last_undegraded":"2024-04-26T01:44:17.172713+0000","last_fullsized":"2024-04-26T01:44:17.172713+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:36:58.043359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.990349+0000","last_change":"2024-04-26T01:42:02.231871+0000","last_active":"2024-04-26T01:43:49.990349+0000","last_peered":"2024-04-26T01:43:49.990349+0000","last_clean":"2024-04-26T01:43:49.990349+0000","last_became_active":"2024-04-26T01:42:02.230813+0000","last_became_peered":"2024-04-26T01:42:02.230813+0000","last_unstale":"2024-04-26T01:43:49.990349+0000","last_undegraded":"2024-04-26T01:43:49.990349+0000","last_fullsized":"2024-04-26T01:43:49.990349+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:53.117266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"95'6","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.184410+0000","last_change":"2024-04-26T01:42:05.412414+0000","last_active":"2024-04-26T01:44:17.184410+0000","last_peered":"2024-04-26T01:44:17.184410+0000","last_clean":"2024-04-26T01:44:17.184410+0000","last_became_active":"2024-04-26T01:42:05.412057+0000","last_became_peered":"2024-04-26T01:42:05.412057+0000","last_unstale":"2024-04-26T01:44:17.184410+0000","last_undegraded":"2024-04-26T01:44:17.184410+0000","last_fullsized":"2024-04-26T01:44:17.184410+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:51:46.331246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.985428+0000","last_change":"2024-04-26T01:42:01.329957+0000","last_active":"2024-04-26T01:43:49.985428+0000","last_peered":"2024-04-26T01:43:49.985428+0000","last_clean":"2024-04-26T01:43:49.985428+0000","last_became_active":"2024-04-26T01:42:01.329289+0000","last_became_peered":"2024-04-26T01:42:01.329289+0000","last_unstale":"2024-04-26T01:43:49.985428+0000","last_undegraded":"2024-04-26T01:43:49.985428+0000","last_fullsized":"2024-04-26T01:43:49.985428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:46:38.983138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"95'9","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.190195+0000","last_change":"2024-04-26T01:42:05.399438+0000","last_active":"2024-04-26T01:44:17.190195+0000","last_peered":"2024-04-26T01:44:17.190195+0000","last_clean":"2024-04-26T01:44:17.190195+0000","last_became_active":"2024-04-26T01:42:05.399082+0000","last_became_peered":"2024-04-26T01:42:05.399082+0000","last_unstale":"2024-04-26T01:44:17.190195+0000","last_undegraded":"2024-04-26T01:44:17.190195+0000","last_fullsized":"2024-04-26T01:44:17.190195+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:13.619893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"91'2","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.993619+0000","last_change":"2024-04-26T01:43:49.993134+0000","last_active":"2024-04-26T01:43:49.993619+0000","last_peered":"2024-04-26T01:43:49.993619+0000","last_clean":"2024-04-26T01:43:49.993619+0000","last_became_active":"2024-04-26T01:43:49.990337+0000","last_became_peered":"2024-04-26T01:43:49.990337+0000","last_unstale":"2024-04-26T01:43:49.993619+0000","last_undegraded":"2024-04-26T01:43:49.993619+0000","last_fullsized":"2024-04-26T01:43:49.993619+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:50.502668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"95'15","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.184152+0000","last_change":"2024-04-26T01:42:49.066057+0000","last_active":"2024-04-26T01:44:17.184152+0000","last_peered":"2024-04-26T01:44:17.184152+0000","last_clean":"2024-04-26T01:44:17.184152+0000","last_became_active":"2024-04-26T01:42:49.064684+0000","last_became_peered":"2024-04-26T01:42:49.064684+0000","last_unstale":"2024-04-26T01:44:17.184152+0000","last_undegraded":"2024-04-26T01:44:17.184152+0000","last_fullsized":"2024-04-26T01:44:17.184152+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:39.159528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"95'7","reported_seq":158,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:22.761958+0000","last_change":"2024-04-26T01:42:01.332623+0000","last_active":"2024-04-26T01:44:22.761958+0000","last_peered":"2024-04-26T01:44:22.761958+0000","last_clean":"2024-04-26T01:44:22.761958+0000","last_became_active":"2024-04-26T01:42:01.332227+0000","last_became_peered":"2024-04-26T01:42:01.332227+0000","last_unstale":"2024-04-26T01:44:22.761958+0000","last_undegraded":"2024-04-26T01:44:22.761958+0000","last_fullsized":"2024-04-26T01:44:22.761958+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:58.611464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"95'7","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.179190+0000","last_change":"2024-04-26T01:42:05.396133+0000","last_active":"2024-04-26T01:44:17.179190+0000","last_peered":"2024-04-26T01:44:17.179190+0000","last_clean":"2024-04-26T01:44:17.179190+0000","last_became_active":"2024-04-26T01:42:05.395553+0000","last_became_peered":"2024-04-26T01:42:05.395553+0000","last_unstale":"2024-04-26T01:44:17.179190+0000","last_undegraded":"2024-04-26T01:44:17.179190+0000","last_fullsized":"2024-04-26T01:44:17.179190+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:36:01.363909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.146452+0000","last_change":"2024-04-26T01:42:01.340624+0000","last_active":"2024-04-26T01:43:50.146452+0000","last_peered":"2024-04-26T01:43:50.146452+0000","last_clean":"2024-04-26T01:43:50.146452+0000","last_became_active":"2024-04-26T01:42:01.340158+0000","last_became_peered":"2024-04-26T01:42:01.340158+0000","last_unstale":"2024-04-26T01:43:50.146452+0000","last_undegraded":"2024-04-26T01:43:50.146452+0000","last_fullsized":"2024-04-26T01:43:50.146452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:47:05.529922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"95'8","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.169120+0000","last_change":"2024-04-26T01:42:05.406354+0000","last_active":"2024-04-26T01:44:17.169120+0000","last_peered":"2024-04-26T01:44:17.169120+0000","last_clean":"2024-04-26T01:44:17.169120+0000","last_became_active":"2024-04-26T01:42:05.405989+0000","last_became_peered":"2024-04-26T01:42:05.405989+0000","last_unstale":"2024-04-26T01:44:17.169120+0000","last_undegraded":"2024-04-26T01:44:17.169120+0000","last_fullsized":"2024-04-26T01:44:17.169120+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:43.115225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.309897+0000","last_change":"2024-04-26T01:42:01.340549+0000","last_active":"2024-04-26T01:43:50.309897+0000","last_peered":"2024-04-26T01:43:50.309897+0000","last_clean":"2024-04-26T01:43:50.309897+0000","last_became_active":"2024-04-26T01:42:01.339861+0000","last_became_peered":"2024-04-26T01:42:01.339861+0000","last_unstale":"2024-04-26T01:43:50.309897+0000","last_undegraded":"2024-04-26T01:43:50.309897+0000","last_fullsized":"2024-04-26T01:43:50.309897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:02:53.154367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"95'8","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.191565+0000","last_change":"2024-04-26T01:42:05.402664+0000","last_active":"2024-04-26T01:44:17.191565+0000","last_peered":"2024-04-26T01:44:17.191565+0000","last_clean":"2024-04-26T01:44:17.191565+0000","last_became_active":"2024-04-26T01:42:05.400785+0000","last_became_peered":"2024-04-26T01:42:05.400785+0000","last_unstale":"2024-04-26T01:44:17.191565+0000","last_undegraded":"2024-04-26T01:44:17.191565+0000","last_fullsized":"2024-04-26T01:44:17.191565+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:40:32.137752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.988999+0000","last_change":"2024-04-26T01:42:01.328600+0000","last_active":"2024-04-26T01:43:49.988999+0000","last_peered":"2024-04-26T01:43:49.988999+0000","last_clean":"2024-04-26T01:43:49.988999+0000","last_became_active":"2024-04-26T01:42:01.328149+0000","last_became_peered":"2024-04-26T01:42:01.328149+0000","last_unstale":"2024-04-26T01:43:49.988999+0000","last_undegraded":"2024-04-26T01:43:49.988999+0000","last_fullsized":"2024-04-26T01:43:49.988999+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:45:42.811088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"95'11","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.182002+0000","last_change":"2024-04-26T01:42:49.065824+0000","last_active":"2024-04-26T01:44:17.182002+0000","last_peered":"2024-04-26T01:44:17.182002+0000","last_clean":"2024-04-26T01:44:17.182002+0000","last_became_active":"2024-04-26T01:42:49.064670+0000","last_became_peered":"2024-04-26T01:42:49.064670+0000","last_unstale":"2024-04-26T01:44:17.182002+0000","last_undegraded":"2024-04-26T01:44:17.182002+0000","last_fullsized":"2024-04-26T01:44:17.182002+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:44.790184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"95'4","reported_seq":151,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:22.761124+0000","last_change":"2024-04-26T01:42:01.335438+0000","last_active":"2024-04-26T01:44:22.761124+0000","last_peered":"2024-04-26T01:44:22.761124+0000","last_clean":"2024-04-26T01:44:22.761124+0000","last_became_active":"2024-04-26T01:42:01.334962+0000","last_became_peered":"2024-04-26T01:42:01.334962+0000","last_unstale":"2024-04-26T01:44:22.761124+0000","last_undegraded":"2024-04-26T01:44:22.761124+0000","last_fullsized":"2024-04-26T01:44:22.761124+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:35:08.252274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"95'10","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.182898+0000","last_change":"2024-04-26T01:42:05.393294+0000","last_active":"2024-04-26T01:44:17.182898+0000","last_peered":"2024-04-26T01:44:17.182898+0000","last_clean":"2024-04-26T01:44:17.182898+0000","last_became_active":"2024-04-26T01:42:05.392974+0000","last_became_peered":"2024-04-26T01:42:05.392974+0000","last_unstale":"2024-04-26T01:44:17.182898+0000","last_undegraded":"2024-04-26T01:44:17.182898+0000","last_fullsized":"2024-04-26T01:44:17.182898+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:58.188706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.109030+0000","last_change":"2024-04-26T01:42:01.333505+0000","last_active":"2024-04-26T01:43:50.109030+0000","last_peered":"2024-04-26T01:43:50.109030+0000","last_clean":"2024-04-26T01:43:50.109030+0000","last_became_active":"2024-04-26T01:42:01.333062+0000","last_became_peered":"2024-04-26T01:42:01.333062+0000","last_unstale":"2024-04-26T01:43:50.109030+0000","last_undegraded":"2024-04-26T01:43:50.109030+0000","last_fullsized":"2024-04-26T01:43:50.109030+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:50:48.804443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"95'9","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.183747+0000","last_change":"2024-04-26T01:42:49.063948+0000","last_active":"2024-04-26T01:44:17.183747+0000","last_peered":"2024-04-26T01:44:17.183747+0000","last_clean":"2024-04-26T01:44:17.183747+0000","last_became_active":"2024-04-26T01:42:49.060852+0000","last_became_peered":"2024-04-26T01:42:49.060852+0000","last_unstale":"2024-04-26T01:44:17.183747+0000","last_undegraded":"2024-04-26T01:44:17.183747+0000","last_fullsized":"2024-04-26T01:44:17.183747+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.370431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.989442+0000","last_change":"2024-04-26T01:42:02.232308+0000","last_active":"2024-04-26T01:43:49.989442+0000","last_peered":"2024-04-26T01:43:49.989442+0000","last_clean":"2024-04-26T01:43:49.989442+0000","last_became_active":"2024-04-26T01:42:02.231897+0000","last_became_peered":"2024-04-26T01:42:02.231897+0000","last_unstale":"2024-04-26T01:43:49.989442+0000","last_undegraded":"2024-04-26T01:43:49.989442+0000","last_fullsized":"2024-04-26T01:43:49.989442+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:36:16.847087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"95'11","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.190211+0000","last_change":"2024-04-26T01:42:49.070798+0000","last_active":"2024-04-26T01:44:17.190211+0000","last_peered":"2024-04-26T01:44:17.190211+0000","last_clean":"2024-04-26T01:44:17.190211+0000","last_became_active":"2024-04-26T01:42:49.070470+0000","last_became_peered":"2024-04-26T01:42:49.070470+0000","last_unstale":"2024-04-26T01:44:17.190211+0000","last_undegraded":"2024-04-26T01:44:17.190211+0000","last_fullsized":"2024-04-26T01:44:17.190211+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:04:22.103704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"95'34","reported_seq":310,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:23.705622+0000","last_change":"2024-04-26T01:42:01.330822+0000","last_active":"2024-04-26T01:44:23.705622+0000","last_peered":"2024-04-26T01:44:23.705622+0000","last_clean":"2024-04-26T01:44:23.705622+0000","last_became_active":"2024-04-26T01:42:01.330337+0000","last_became_peered":"2024-04-26T01:42:01.330337+0000","last_unstale":"2024-04-26T01:44:23.705622+0000","last_undegraded":"2024-04-26T01:44:23.705622+0000","last_fullsized":"2024-04-26T01:44:23.705622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:04.024645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64459,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":245,"num_read_kb":186,"num_write":35,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"95'6","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.165940+0000","last_change":"2024-04-26T01:42:05.390601+0000","last_active":"2024-04-26T01:44:17.165940+0000","last_peered":"2024-04-26T01:44:17.165940+0000","last_clean":"2024-04-26T01:44:17.165940+0000","last_became_active":"2024-04-26T01:42:05.390279+0000","last_became_peered":"2024-04-26T01:42:05.390279+0000","last_unstale":"2024-04-26T01:44:17.165940+0000","last_undegraded":"2024-04-26T01:44:17.165940+0000","last_fullsized":"2024-04-26T01:44:17.165940+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:20.171769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.308838+0000","last_change":"2024-04-26T01:42:01.337200+0000","last_active":"2024-04-26T01:43:50.308838+0000","last_peered":"2024-04-26T01:43:50.308838+0000","last_clean":"2024-04-26T01:43:50.308838+0000","last_became_active":"2024-04-26T01:42:01.336919+0000","last_became_peered":"2024-04-26T01:42:01.336919+0000","last_unstale":"2024-04-26T01:43:50.308838+0000","last_undegraded":"2024-04-26T01:43:50.308838+0000","last_fullsized":"2024-04-26T01:43:50.308838+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:29.799903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"95'9","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.187810+0000","last_change":"2024-04-26T01:42:05.410081+0000","last_active":"2024-04-26T01:44:17.187810+0000","last_peered":"2024-04-26T01:44:17.187810+0000","last_clean":"2024-04-26T01:44:17.187810+0000","last_became_active":"2024-04-26T01:42:05.409341+0000","last_became_peered":"2024-04-26T01:42:05.409341+0000","last_unstale":"2024-04-26T01:44:17.187810+0000","last_undegraded":"2024-04-26T01:44:17.187810+0000","last_fullsized":"2024-04-26T01:44:17.187810+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:25:31.925296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.989295+0000","last_change":"2024-04-26T01:42:01.331702+0000","last_active":"2024-04-26T01:43:49.989295+0000","last_peered":"2024-04-26T01:43:49.989295+0000","last_clean":"2024-04-26T01:43:49.989295+0000","last_became_active":"2024-04-26T01:42:01.331311+0000","last_became_peered":"2024-04-26T01:42:01.331311+0000","last_unstale":"2024-04-26T01:43:49.989295+0000","last_undegraded":"2024-04-26T01:43:49.989295+0000","last_fullsized":"2024-04-26T01:43:49.989295+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:54:49.513627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"95'15","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.193275+0000","last_change":"2024-04-26T01:42:05.403957+0000","last_active":"2024-04-26T01:44:17.193275+0000","last_peered":"2024-04-26T01:44:17.193275+0000","last_clean":"2024-04-26T01:44:17.193275+0000","last_became_active":"2024-04-26T01:42:05.403583+0000","last_became_peered":"2024-04-26T01:42:05.403583+0000","last_unstale":"2024-04-26T01:44:17.193275+0000","last_undegraded":"2024-04-26T01:44:17.193275+0000","last_fullsized":"2024-04-26T01:44:17.193275+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:12:20.095407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"95'5","reported_seq":155,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:21.729565+0000","last_change":"2024-04-26T01:42:01.332456+0000","last_active":"2024-04-26T01:44:21.729565+0000","last_peered":"2024-04-26T01:44:21.729565+0000","last_clean":"2024-04-26T01:44:21.729565+0000","last_became_active":"2024-04-26T01:42:01.331910+0000","last_became_peered":"2024-04-26T01:42:01.331910+0000","last_unstale":"2024-04-26T01:44:21.729565+0000","last_undegraded":"2024-04-26T01:44:21.729565+0000","last_fullsized":"2024-04-26T01:44:21.729565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:13:25.368205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"95'6","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.175614+0000","last_change":"2024-04-26T01:42:49.062130+0000","last_active":"2024-04-26T01:44:17.175614+0000","last_peered":"2024-04-26T01:44:17.175614+0000","last_clean":"2024-04-26T01:44:17.175614+0000","last_became_active":"2024-04-26T01:42:49.061627+0000","last_became_peered":"2024-04-26T01:42:49.061627+0000","last_unstale":"2024-04-26T01:44:17.175614+0000","last_undegraded":"2024-04-26T01:44:17.175614+0000","last_fullsized":"2024-04-26T01:44:17.175614+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:16.101940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.986820+0000","last_change":"2024-04-26T01:42:49.082618+0000","last_active":"2024-04-26T01:43:49.986820+0000","last_peered":"2024-04-26T01:43:49.986820+0000","last_clean":"2024-04-26T01:43:49.986820+0000","last_became_active":"2024-04-26T01:42:49.064654+0000","last_became_peered":"2024-04-26T01:42:49.064654+0000","last_unstale":"2024-04-26T01:43:49.986820+0000","last_undegraded":"2024-04-26T01:43:49.986820+0000","last_fullsized":"2024-04-26T01:43:49.986820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:13:23.937059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"95'6","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.155496+0000","last_change":"2024-04-26T01:42:05.391392+0000","last_active":"2024-04-26T01:44:17.155496+0000","last_peered":"2024-04-26T01:44:17.155496+0000","last_clean":"2024-04-26T01:44:17.155496+0000","last_became_active":"2024-04-26T01:42:05.390939+0000","last_became_peered":"2024-04-26T01:42:05.390939+0000","last_unstale":"2024-04-26T01:44:17.155496+0000","last_undegraded":"2024-04-26T01:44:17.155496+0000","last_fullsized":"2024-04-26T01:44:17.155496+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:36.141915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.985486+0000","last_change":"2024-04-26T01:42:01.332932+0000","last_active":"2024-04-26T01:43:49.985486+0000","last_peered":"2024-04-26T01:43:49.985486+0000","last_clean":"2024-04-26T01:43:49.985486+0000","last_became_active":"2024-04-26T01:42:01.332626+0000","last_became_peered":"2024-04-26T01:42:01.332626+0000","last_unstale":"2024-04-26T01:43:49.985486+0000","last_undegraded":"2024-04-26T01:43:49.985486+0000","last_fullsized":"2024-04-26T01:43:49.985486+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:20.593730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"95'5","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.162333+0000","last_change":"2024-04-26T01:42:05.394925+0000","last_active":"2024-04-26T01:44:17.162333+0000","last_peered":"2024-04-26T01:44:17.162333+0000","last_clean":"2024-04-26T01:44:17.162333+0000","last_became_active":"2024-04-26T01:42:05.394531+0000","last_became_peered":"2024-04-26T01:42:05.394531+0000","last_unstale":"2024-04-26T01:44:17.162333+0000","last_undegraded":"2024-04-26T01:44:17.162333+0000","last_fullsized":"2024-04-26T01:44:17.162333+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:33.177780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.310879+0000","last_change":"2024-04-26T01:42:02.237482+0000","last_active":"2024-04-26T01:43:50.310879+0000","last_peered":"2024-04-26T01:43:50.310879+0000","last_clean":"2024-04-26T01:43:50.310879+0000","last_became_active":"2024-04-26T01:42:02.236752+0000","last_became_peered":"2024-04-26T01:42:02.236752+0000","last_unstale":"2024-04-26T01:43:50.310879+0000","last_undegraded":"2024-04-26T01:43:50.310879+0000","last_fullsized":"2024-04-26T01:43:50.310879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:27.211245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"95'5","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.187385+0000","last_change":"2024-04-26T01:42:05.402157+0000","last_active":"2024-04-26T01:44:17.187385+0000","last_peered":"2024-04-26T01:44:17.187385+0000","last_clean":"2024-04-26T01:44:17.187385+0000","last_became_active":"2024-04-26T01:42:05.401889+0000","last_became_peered":"2024-04-26T01:42:05.401889+0000","last_unstale":"2024-04-26T01:44:17.187385+0000","last_undegraded":"2024-04-26T01:44:17.187385+0000","last_fullsized":"2024-04-26T01:44:17.187385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:33.854901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.108839+0000","last_change":"2024-04-26T01:42:01.339675+0000","last_active":"2024-04-26T01:43:50.108839+0000","last_peered":"2024-04-26T01:43:50.108839+0000","last_clean":"2024-04-26T01:43:50.108839+0000","last_became_active":"2024-04-26T01:42:01.339247+0000","last_became_peered":"2024-04-26T01:42:01.339247+0000","last_unstale":"2024-04-26T01:43:50.108839+0000","last_undegraded":"2024-04-26T01:43:50.108839+0000","last_fullsized":"2024-04-26T01:43:50.108839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:32.724830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"95'7","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.185556+0000","last_change":"2024-04-26T01:42:05.412913+0000","last_active":"2024-04-26T01:44:17.185556+0000","last_peered":"2024-04-26T01:44:17.185556+0000","last_clean":"2024-04-26T01:44:17.185556+0000","last_became_active":"2024-04-26T01:42:05.412556+0000","last_became_peered":"2024-04-26T01:42:05.412556+0000","last_unstale":"2024-04-26T01:44:17.185556+0000","last_undegraded":"2024-04-26T01:44:17.185556+0000","last_fullsized":"2024-04-26T01:44:17.185556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:00:52.798069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.156940+0000","last_change":"2024-04-26T01:42:05.391235+0000","last_active":"2024-04-26T01:44:17.156940+0000","last_peered":"2024-04-26T01:44:17.156940+0000","last_clean":"2024-04-26T01:44:17.156940+0000","last_became_active":"2024-04-26T01:42:05.390717+0000","last_became_peered":"2024-04-26T01:42:05.390717+0000","last_unstale":"2024-04-26T01:44:17.156940+0000","last_undegraded":"2024-04-26T01:44:17.156940+0000","last_fullsized":"2024-04-26T01:44:17.156940+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:07.541593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"95'5","reported_seq":150,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:19.724175+0000","last_change":"2024-04-26T01:42:02.232823+0000","last_active":"2024-04-26T01:44:19.724175+0000","last_peered":"2024-04-26T01:44:19.724175+0000","last_clean":"2024-04-26T01:44:19.724175+0000","last_became_active":"2024-04-26T01:42:02.231899+0000","last_became_peered":"2024-04-26T01:42:02.231899+0000","last_unstale":"2024-04-26T01:44:19.724175+0000","last_undegraded":"2024-04-26T01:44:19.724175+0000","last_fullsized":"2024-04-26T01:44:19.724175+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:23.287305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.177388+0000","last_change":"2024-04-26T01:42:05.405690+0000","last_active":"2024-04-26T01:44:17.177388+0000","last_peered":"2024-04-26T01:44:17.177388+0000","last_clean":"2024-04-26T01:44:17.177388+0000","last_became_active":"2024-04-26T01:42:05.405345+0000","last_became_peered":"2024-04-26T01:42:05.405345+0000","last_unstale":"2024-04-26T01:44:17.177388+0000","last_undegraded":"2024-04-26T01:44:17.177388+0000","last_fullsized":"2024-04-26T01:44:17.177388+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:48.377916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.985226+0000","last_change":"2024-04-26T01:42:01.335721+0000","last_active":"2024-04-26T01:43:49.985226+0000","last_peered":"2024-04-26T01:43:49.985226+0000","last_clean":"2024-04-26T01:43:49.985226+0000","last_became_active":"2024-04-26T01:42:01.335149+0000","last_became_peered":"2024-04-26T01:42:01.335149+0000","last_unstale":"2024-04-26T01:43:49.985226+0000","last_undegraded":"2024-04-26T01:43:49.985226+0000","last_fullsized":"2024-04-26T01:43:49.985226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:36.372033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":366,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.108654+0000","last_change":"2024-04-26T01:40:28.086333+0000","last_active":"2024-04-26T01:43:50.108654+0000","last_peered":"2024-04-26T01:43:50.108654+0000","last_clean":"2024-04-26T01:43:50.108654+0000","last_became_active":"2024-04-26T01:40:28.085902+0000","last_became_peered":"2024-04-26T01:40:28.085902+0000","last_unstale":"2024-04-26T01:43:50.108654+0000","last_undegraded":"2024-04-26T01:43:50.108654+0000","last_fullsized":"2024-04-26T01:43:50.108654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_clean_scrub_stamp":"2024-04-26T01:36:35.357698+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:43.290117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"95'5","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.174305+0000","last_change":"2024-04-26T01:42:05.401896+0000","last_active":"2024-04-26T01:44:17.174305+0000","last_peered":"2024-04-26T01:44:17.174305+0000","last_clean":"2024-04-26T01:44:17.174305+0000","last_became_active":"2024-04-26T01:42:05.401496+0000","last_became_peered":"2024-04-26T01:42:05.401496+0000","last_unstale":"2024-04-26T01:44:17.174305+0000","last_undegraded":"2024-04-26T01:44:17.174305+0000","last_fullsized":"2024-04-26T01:44:17.174305+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:49.758444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.108813+0000","last_change":"2024-04-26T01:42:01.329705+0000","last_active":"2024-04-26T01:43:50.108813+0000","last_peered":"2024-04-26T01:43:50.108813+0000","last_clean":"2024-04-26T01:43:50.108813+0000","last_became_active":"2024-04-26T01:42:01.329153+0000","last_became_peered":"2024-04-26T01:42:01.329153+0000","last_unstale":"2024-04-26T01:43:50.108813+0000","last_undegraded":"2024-04-26T01:43:50.108813+0000","last_fullsized":"2024-04-26T01:43:50.108813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:06.747378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.985397+0000","last_change":"2024-04-26T01:42:02.232148+0000","last_active":"2024-04-26T01:43:49.985397+0000","last_peered":"2024-04-26T01:43:49.985397+0000","last_clean":"2024-04-26T01:43:49.985397+0000","last_became_active":"2024-04-26T01:42:02.231739+0000","last_became_peered":"2024-04-26T01:42:02.231739+0000","last_unstale":"2024-04-26T01:43:49.985397+0000","last_undegraded":"2024-04-26T01:43:49.985397+0000","last_fullsized":"2024-04-26T01:43:49.985397+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:09.752433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"95'9","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.182401+0000","last_change":"2024-04-26T01:42:05.399938+0000","last_active":"2024-04-26T01:44:17.182401+0000","last_peered":"2024-04-26T01:44:17.182401+0000","last_clean":"2024-04-26T01:44:17.182401+0000","last_became_active":"2024-04-26T01:42:05.399550+0000","last_became_peered":"2024-04-26T01:42:05.399550+0000","last_unstale":"2024-04-26T01:44:17.182401+0000","last_undegraded":"2024-04-26T01:44:17.182401+0000","last_fullsized":"2024-04-26T01:44:17.182401+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:44.142284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"95'13","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:12.013918+0000","last_change":"2024-04-26T01:42:01.337437+0000","last_active":"2024-04-26T01:44:12.013918+0000","last_peered":"2024-04-26T01:44:12.013918+0000","last_clean":"2024-04-26T01:44:12.013918+0000","last_became_active":"2024-04-26T01:42:01.336894+0000","last_became_peered":"2024-04-26T01:42:01.336894+0000","last_unstale":"2024-04-26T01:44:12.013918+0000","last_undegraded":"2024-04-26T01:44:12.013918+0000","last_fullsized":"2024-04-26T01:44:12.013918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:04:31.093273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"95'5","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.173990+0000","last_change":"2024-04-26T01:42:05.411694+0000","last_active":"2024-04-26T01:44:17.173990+0000","last_peered":"2024-04-26T01:44:17.173990+0000","last_clean":"2024-04-26T01:44:17.173990+0000","last_became_active":"2024-04-26T01:42:05.411300+0000","last_became_peered":"2024-04-26T01:42:05.411300+0000","last_unstale":"2024-04-26T01:44:17.173990+0000","last_undegraded":"2024-04-26T01:44:17.173990+0000","last_fullsized":"2024-04-26T01:44:17.173990+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:42:08.116145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.985445+0000","last_change":"2024-04-26T01:42:01.427501+0000","last_active":"2024-04-26T01:43:49.985445+0000","last_peered":"2024-04-26T01:43:49.985445+0000","last_clean":"2024-04-26T01:43:49.985445+0000","last_became_active":"2024-04-26T01:42:01.426970+0000","last_became_peered":"2024-04-26T01:42:01.426970+0000","last_unstale":"2024-04-26T01:43:49.985445+0000","last_undegraded":"2024-04-26T01:43:49.985445+0000","last_fullsized":"2024-04-26T01:43:49.985445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:53:53.118296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"95'7","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.186200+0000","last_change":"2024-04-26T01:42:05.414790+0000","last_active":"2024-04-26T01:44:17.186200+0000","last_peered":"2024-04-26T01:44:17.186200+0000","last_clean":"2024-04-26T01:44:17.186200+0000","last_became_active":"2024-04-26T01:42:05.411947+0000","last_became_peered":"2024-04-26T01:42:05.411947+0000","last_unstale":"2024-04-26T01:44:17.186200+0000","last_undegraded":"2024-04-26T01:44:17.186200+0000","last_fullsized":"2024-04-26T01:44:17.186200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:11:34.450815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.989082+0000","last_change":"2024-04-26T01:42:02.230539+0000","last_active":"2024-04-26T01:43:49.989082+0000","last_peered":"2024-04-26T01:43:49.989082+0000","last_clean":"2024-04-26T01:43:49.989082+0000","last_became_active":"2024-04-26T01:42:02.230150+0000","last_became_peered":"2024-04-26T01:42:02.230150+0000","last_unstale":"2024-04-26T01:43:49.989082+0000","last_undegraded":"2024-04-26T01:43:49.989082+0000","last_fullsized":"2024-04-26T01:43:49.989082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:08:03.073069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"95'2","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.150118+0000","last_change":"2024-04-26T01:42:05.388877+0000","last_active":"2024-04-26T01:44:17.150118+0000","last_peered":"2024-04-26T01:44:17.150118+0000","last_clean":"2024-04-26T01:44:17.150118+0000","last_became_active":"2024-04-26T01:42:05.388457+0000","last_became_peered":"2024-04-26T01:42:05.388457+0000","last_unstale":"2024-04-26T01:44:17.150118+0000","last_undegraded":"2024-04-26T01:44:17.150118+0000","last_fullsized":"2024-04-26T01:44:17.150118+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:54.436859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.989575+0000","last_change":"2024-04-26T01:42:01.343461+0000","last_active":"2024-04-26T01:43:49.989575+0000","last_peered":"2024-04-26T01:43:49.989575+0000","last_clean":"2024-04-26T01:43:49.989575+0000","last_became_active":"2024-04-26T01:42:01.342713+0000","last_became_peered":"2024-04-26T01:42:01.342713+0000","last_unstale":"2024-04-26T01:43:49.989575+0000","last_undegraded":"2024-04-26T01:43:49.989575+0000","last_fullsized":"2024-04-26T01:43:49.989575+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:38.471858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"95'9","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.178839+0000","last_change":"2024-04-26T01:42:05.415101+0000","last_active":"2024-04-26T01:44:17.178839+0000","last_peered":"2024-04-26T01:44:17.178839+0000","last_clean":"2024-04-26T01:44:17.178839+0000","last_became_active":"2024-04-26T01:42:05.414745+0000","last_became_peered":"2024-04-26T01:42:05.414745+0000","last_unstale":"2024-04-26T01:44:17.178839+0000","last_undegraded":"2024-04-26T01:44:17.178839+0000","last_fullsized":"2024-04-26T01:44:17.178839+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:13:08.573547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.308613+0000","last_change":"2024-04-26T01:42:01.339501+0000","last_active":"2024-04-26T01:43:50.308613+0000","last_peered":"2024-04-26T01:43:50.308613+0000","last_clean":"2024-04-26T01:43:50.308613+0000","last_became_active":"2024-04-26T01:42:01.339132+0000","last_became_peered":"2024-04-26T01:42:01.339132+0000","last_unstale":"2024-04-26T01:43:50.308613+0000","last_undegraded":"2024-04-26T01:43:50.308613+0000","last_fullsized":"2024-04-26T01:43:50.308613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:16.463099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.187446+0000","last_change":"2024-04-26T01:42:05.417864+0000","last_active":"2024-04-26T01:44:17.187446+0000","last_peered":"2024-04-26T01:44:17.187446+0000","last_clean":"2024-04-26T01:44:17.187446+0000","last_became_active":"2024-04-26T01:42:05.417414+0000","last_became_peered":"2024-04-26T01:42:05.417414+0000","last_unstale":"2024-04-26T01:44:17.187446+0000","last_undegraded":"2024-04-26T01:44:17.187446+0000","last_fullsized":"2024-04-26T01:44:17.187446+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:23:12.793421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.985155+0000","last_change":"2024-04-26T01:42:49.063769+0000","last_active":"2024-04-26T01:43:49.985155+0000","last_peered":"2024-04-26T01:43:49.985155+0000","last_clean":"2024-04-26T01:43:49.985155+0000","last_became_active":"2024-04-26T01:42:49.057463+0000","last_became_peered":"2024-04-26T01:42:49.057463+0000","last_unstale":"2024-04-26T01:43:49.985155+0000","last_undegraded":"2024-04-26T01:43:49.985155+0000","last_fullsized":"2024-04-26T01:43:49.985155+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:34:54.511304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"95'5","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.191252+0000","last_change":"2024-04-26T01:42:05.397980+0000","last_active":"2024-04-26T01:44:17.191252+0000","last_peered":"2024-04-26T01:44:17.191252+0000","last_clean":"2024-04-26T01:44:17.191252+0000","last_became_active":"2024-04-26T01:42:05.397589+0000","last_became_peered":"2024-04-26T01:42:05.397589+0000","last_unstale":"2024-04-26T01:44:17.191252+0000","last_undegraded":"2024-04-26T01:44:17.191252+0000","last_fullsized":"2024-04-26T01:44:17.191252+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:39:55.863566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.308969+0000","last_change":"2024-04-26T01:42:02.234313+0000","last_active":"2024-04-26T01:43:50.308969+0000","last_peered":"2024-04-26T01:43:50.308969+0000","last_clean":"2024-04-26T01:43:50.308969+0000","last_became_active":"2024-04-26T01:42:02.233873+0000","last_became_peered":"2024-04-26T01:42:02.233873+0000","last_unstale":"2024-04-26T01:43:50.308969+0000","last_undegraded":"2024-04-26T01:43:50.308969+0000","last_fullsized":"2024-04-26T01:43:50.308969+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:34.092434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"95'7","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.168380+0000","last_change":"2024-04-26T01:42:05.387435+0000","last_active":"2024-04-26T01:44:17.168380+0000","last_peered":"2024-04-26T01:44:17.168380+0000","last_clean":"2024-04-26T01:44:17.168380+0000","last_became_active":"2024-04-26T01:42:05.386964+0000","last_became_peered":"2024-04-26T01:42:05.386964+0000","last_unstale":"2024-04-26T01:44:17.168380+0000","last_undegraded":"2024-04-26T01:44:17.168380+0000","last_fullsized":"2024-04-26T01:44:17.168380+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:12:27.905842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:49.989474+0000","last_change":"2024-04-26T01:42:01.326432+0000","last_active":"2024-04-26T01:43:49.989474+0000","last_peered":"2024-04-26T01:43:49.989474+0000","last_clean":"2024-04-26T01:43:49.989474+0000","last_became_active":"2024-04-26T01:42:01.325989+0000","last_became_peered":"2024-04-26T01:42:01.325989+0000","last_unstale":"2024-04-26T01:43:49.989474+0000","last_undegraded":"2024-04-26T01:43:49.989474+0000","last_fullsized":"2024-04-26T01:43:49.989474+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:10.049563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"95'7","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.181068+0000","last_change":"2024-04-26T01:42:05.412802+0000","last_active":"2024-04-26T01:44:17.181068+0000","last_peered":"2024-04-26T01:44:17.181068+0000","last_clean":"2024-04-26T01:44:17.181068+0000","last_became_active":"2024-04-26T01:42:05.412443+0000","last_became_peered":"2024-04-26T01:42:05.412443+0000","last_unstale":"2024-04-26T01:44:17.181068+0000","last_undegraded":"2024-04-26T01:44:17.181068+0000","last_fullsized":"2024-04-26T01:44:17.181068+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:28:56.360778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'2","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:43:50.311619+0000","last_change":"2024-04-26T01:42:49.080305+0000","last_active":"2024-04-26T01:43:50.311619+0000","last_peered":"2024-04-26T01:43:50.311619+0000","last_clean":"2024-04-26T01:43:50.311619+0000","last_became_active":"2024-04-26T01:42:49.074476+0000","last_became_peered":"2024-04-26T01:42:49.074476+0000","last_unstale":"2024-04-26T01:43:50.311619+0000","last_undegraded":"2024-04-26T01:43:50.311619+0000","last_fullsized":"2024-04-26T01:43:50.311619+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:58:06.619961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"95'9","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.163094+0000","last_change":"2024-04-26T01:42:05.387621+0000","last_active":"2024-04-26T01:44:17.163094+0000","last_peered":"2024-04-26T01:44:17.163094+0000","last_clean":"2024-04-26T01:44:17.163094+0000","last_became_active":"2024-04-26T01:42:05.387165+0000","last_became_peered":"2024-04-26T01:42:05.387165+0000","last_unstale":"2024-04-26T01:44:17.163094+0000","last_undegraded":"2024-04-26T01:44:17.163094+0000","last_fullsized":"2024-04-26T01:44:17.163094+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:37.803330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"95'9","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.162500+0000","last_change":"2024-04-26T01:42:05.386787+0000","last_active":"2024-04-26T01:44:17.162500+0000","last_peered":"2024-04-26T01:44:17.162500+0000","last_clean":"2024-04-26T01:44:17.162500+0000","last_became_active":"2024-04-26T01:42:05.386410+0000","last_became_peered":"2024-04-26T01:42:05.386410+0000","last_unstale":"2024-04-26T01:44:17.162500+0000","last_undegraded":"2024-04-26T01:44:17.162500+0000","last_fullsized":"2024-04-26T01:44:17.162500+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:42.137191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"95'12","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.161732+0000","last_change":"2024-04-26T01:42:05.392142+0000","last_active":"2024-04-26T01:44:17.161732+0000","last_peered":"2024-04-26T01:44:17.161732+0000","last_clean":"2024-04-26T01:44:17.161732+0000","last_became_active":"2024-04-26T01:42:05.391829+0000","last_became_peered":"2024-04-26T01:42:05.391829+0000","last_unstale":"2024-04-26T01:44:17.161732+0000","last_undegraded":"2024-04-26T01:44:17.161732+0000","last_fullsized":"2024-04-26T01:44:17.161732+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:00:25.934641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.180591+0000","last_change":"2024-04-26T01:42:05.409184+0000","last_active":"2024-04-26T01:44:17.180591+0000","last_peered":"2024-04-26T01:44:17.180591+0000","last_clean":"2024-04-26T01:44:17.180591+0000","last_became_active":"2024-04-26T01:42:05.408894+0000","last_became_peered":"2024-04-26T01:42:05.408894+0000","last_unstale":"2024-04-26T01:44:17.180591+0000","last_undegraded":"2024-04-26T01:44:17.180591+0000","last_fullsized":"2024-04-26T01:44:17.180591+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:53.291844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"95'7","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.191169+0000","last_change":"2024-04-26T01:42:05.406940+0000","last_active":"2024-04-26T01:44:17.191169+0000","last_peered":"2024-04-26T01:44:17.191169+0000","last_clean":"2024-04-26T01:44:17.191169+0000","last_became_active":"2024-04-26T01:42:05.406476+0000","last_became_peered":"2024-04-26T01:42:05.406476+0000","last_unstale":"2024-04-26T01:44:17.191169+0000","last_undegraded":"2024-04-26T01:44:17.191169+0000","last_fullsized":"2024-04-26T01:44:17.191169+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:21:37.734633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"95'4","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.189162+0000","last_change":"2024-04-26T01:42:05.413521+0000","last_active":"2024-04-26T01:44:17.189162+0000","last_peered":"2024-04-26T01:44:17.189162+0000","last_clean":"2024-04-26T01:44:17.189162+0000","last_became_active":"2024-04-26T01:42:05.412789+0000","last_became_peered":"2024-04-26T01:42:05.412789+0000","last_unstale":"2024-04-26T01:44:17.189162+0000","last_undegraded":"2024-04-26T01:44:17.189162+0000","last_fullsized":"2024-04-26T01:44:17.189162+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:13.471376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"95'6","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.161927+0000","last_change":"2024-04-26T01:42:05.391395+0000","last_active":"2024-04-26T01:44:17.161927+0000","last_peered":"2024-04-26T01:44:17.161927+0000","last_clean":"2024-04-26T01:44:17.161927+0000","last_became_active":"2024-04-26T01:42:05.391048+0000","last_became_peered":"2024-04-26T01:42:05.391048+0000","last_unstale":"2024-04-26T01:44:17.161927+0000","last_undegraded":"2024-04-26T01:44:17.161927+0000","last_fullsized":"2024-04-26T01:44:17.161927+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:45:56.645454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"95'15","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.168013+0000","last_change":"2024-04-26T01:42:05.386868+0000","last_active":"2024-04-26T01:44:17.168013+0000","last_peered":"2024-04-26T01:44:17.168013+0000","last_clean":"2024-04-26T01:44:17.168013+0000","last_became_active":"2024-04-26T01:42:05.386483+0000","last_became_peered":"2024-04-26T01:42:05.386483+0000","last_unstale":"2024-04-26T01:44:17.168013+0000","last_undegraded":"2024-04-26T01:44:17.168013+0000","last_fullsized":"2024-04-26T01:44:17.168013+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:42:44.453076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"95'2","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.173530+0000","last_change":"2024-04-26T01:42:05.406711+0000","last_active":"2024-04-26T01:44:17.173530+0000","last_peered":"2024-04-26T01:44:17.173530+0000","last_clean":"2024-04-26T01:44:17.173530+0000","last_became_active":"2024-04-26T01:42:05.406318+0000","last_became_peered":"2024-04-26T01:42:05.406318+0000","last_unstale":"2024-04-26T01:44:17.173530+0000","last_undegraded":"2024-04-26T01:44:17.173530+0000","last_fullsized":"2024-04-26T01:44:17.173530+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:36:16.599678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"95'6","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.176898+0000","last_change":"2024-04-26T01:42:05.406305+0000","last_active":"2024-04-26T01:44:17.176898+0000","last_peered":"2024-04-26T01:44:17.176898+0000","last_clean":"2024-04-26T01:44:17.176898+0000","last_became_active":"2024-04-26T01:42:05.405953+0000","last_became_peered":"2024-04-26T01:42:05.405953+0000","last_unstale":"2024-04-26T01:44:17.176898+0000","last_undegraded":"2024-04-26T01:44:17.176898+0000","last_fullsized":"2024-04-26T01:44:17.176898+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:36.588672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.174739+0000","last_change":"2024-04-26T01:42:05.413757+0000","last_active":"2024-04-26T01:44:17.174739+0000","last_peered":"2024-04-26T01:44:17.174739+0000","last_clean":"2024-04-26T01:44:17.174739+0000","last_became_active":"2024-04-26T01:42:05.413410+0000","last_became_peered":"2024-04-26T01:42:05.413410+0000","last_unstale":"2024-04-26T01:44:17.174739+0000","last_undegraded":"2024-04-26T01:44:17.174739+0000","last_fullsized":"2024-04-26T01:44:17.174739+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:30.298443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"95'10","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.180834+0000","last_change":"2024-04-26T01:42:05.412268+0000","last_active":"2024-04-26T01:44:17.180834+0000","last_peered":"2024-04-26T01:44:17.180834+0000","last_clean":"2024-04-26T01:44:17.180834+0000","last_became_active":"2024-04-26T01:42:05.411804+0000","last_became_peered":"2024-04-26T01:42:05.411804+0000","last_unstale":"2024-04-26T01:44:17.180834+0000","last_undegraded":"2024-04-26T01:44:17.180834+0000","last_fullsized":"2024-04-26T01:44:17.180834+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:47.277766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"95'11","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.193062+0000","last_change":"2024-04-26T01:42:05.415563+0000","last_active":"2024-04-26T01:44:17.193062+0000","last_peered":"2024-04-26T01:44:17.193062+0000","last_clean":"2024-04-26T01:44:17.193062+0000","last_became_active":"2024-04-26T01:42:05.414818+0000","last_became_peered":"2024-04-26T01:42:05.414818+0000","last_unstale":"2024-04-26T01:44:17.193062+0000","last_undegraded":"2024-04-26T01:44:17.193062+0000","last_fullsized":"2024-04-26T01:44:17.193062+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:54.713595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"95'8","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:17.162110+0000","last_change":"2024-04-26T01:42:05.392280+0000","last_active":"2024-04-26T01:44:17.162110+0000","last_peered":"2024-04-26T01:44:17.162110+0000","last_clean":"2024-04-26T01:44:17.162110+0000","last_became_active":"2024-04-26T01:42:05.391943+0000","last_became_peered":"2024-04-26T01:42:05.391943+0000","last_unstale":"2024-04-26T01:44:17.162110+0000","last_undegraded":"2024-04-26T01:44:17.162110+0000","last_fullsized":"2024-04-26T01:44:17.162110+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:38:51.560596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69133,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":729,"num_read_kb":438,"num_write":177,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":207399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285614,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":510559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547250,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41208,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703944,"statfs":{"total":95995035648,"available":95952838656,"internally_reserved":0,"allocated":1753088,"data_stored":1166919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710777,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":510683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972416,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1060864,"data_stored":510491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266756,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28640,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716512,"statfs":{"total":95995035648,"available":95965708288,"internally_reserved":0,"allocated":1662976,"data_stored":1103206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430281,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":510072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40592,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704560,"statfs":{"total":95995035648,"available":95953469440,"internally_reserved":0,"allocated":1122304,"data_stored":574324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":511343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1146880,"data_stored":576206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":510383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888149,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1060864,"data_stored":511041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691920,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1675264,"data_stored":1102358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":66341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:44:26.679 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T01:44:26.679 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T01:44:26.679 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T01:44:26.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T01:44:27.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:26 smithi006 ceph-mon[28419]: pgmap v404: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-26T01:44:27.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:26 smithi006 ceph-mon[28419]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:44:27.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:26 smithi188 ceph-mon[32751]: pgmap v404: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-26T01:44:27.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:26 smithi188 ceph-mon[32751]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:44:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:26 smithi029 ceph-mon[32775]: pgmap v404: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-26T01:44:27.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:26 smithi029 ceph-mon[32775]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:44:27.198 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:29.112 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:28 smithi006 ceph-mon[28419]: pgmap v405: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T01:44:29.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:28 smithi188 ceph-mon[32751]: pgmap v405: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T01:44:29.189 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:28 smithi029 ceph-mon[32775]: pgmap v405: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T01:44:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.479+0000 7fead0c00700 1 -- 172.21.15.6:0/2173464990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feacc104ec0 msgr2=0x7feacc1052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.479+0000 7fead0c00700 1 --2- 172.21.15.6:0/2173464990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feacc104ec0 0x7feacc1052a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7feabc0099c0 tx=0x7feabc009cd0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.479+0000 7fead0c00700 1 -- 172.21.15.6:0/2173464990 shutdown_connections 2024-04-26T01:44:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.479+0000 7fead0c00700 1 --2- 172.21.15.6:0/2173464990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feacc109fa0 0x7feacc111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.479+0000 7fead0c00700 1 --2- 172.21.15.6:0/2173464990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feacc104ec0 0x7feacc1052a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.479+0000 7fead0c00700 1 --2- 172.21.15.6:0/2173464990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc105870 0x7feacc1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.479+0000 7fead0c00700 1 -- 172.21.15.6:0/2173464990 >> 172.21.15.6:0/2173464990 conn(0x7feacc100750 msgr2=0x7feacc102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.479+0000 7fead0c00700 1 -- 172.21.15.6:0/2173464990 shutdown_connections 2024-04-26T01:44:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.479+0000 7fead0c00700 1 -- 172.21.15.6:0/2173464990 wait complete. 2024-04-26T01:44:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.480+0000 7fead0c00700 1 Processor -- start 2024-04-26T01:44:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.481+0000 7fead0c00700 1 -- start start 2024-04-26T01:44:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.481+0000 7fead0c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feacc104ec0 0x7feacc19b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.481+0000 7fead0c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc105870 0x7feacc19b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.481+0000 7fead0c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feacc109fa0 0x7feacc19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.481+0000 7fead0c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feacc114e30 con 0x7feacc104ec0 2024-04-26T01:44:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.481+0000 7fead0c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feacc114cb0 con 0x7feacc105870 2024-04-26T01:44:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.481+0000 7fead0c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7feacc114fb0 con 0x7feacc109fa0 2024-04-26T01:44:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.482+0000 7feac9e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feacc104ec0 0x7feacc19b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.482+0000 7feac9e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feacc104ec0 0x7feacc19b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47294/0 (socket says 172.21.15.6:47294) 2024-04-26T01:44:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.482+0000 7feac9e00700 1 -- 172.21.15.6:0/1009532414 learned_addr learned my addr 172.21.15.6:0/1009532414 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.482+0000 7feaca800700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feacc109fa0 0x7feacc19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:29.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.482+0000 7feac9400700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc105870 0x7feacc19b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.482+0000 7feaca800700 1 -- 172.21.15.6:0/1009532414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc105870 msgr2=0x7feacc19b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.482+0000 7feaca800700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc105870 0x7feacc19b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.482+0000 7feaca800700 1 -- 172.21.15.6:0/1009532414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feacc104ec0 msgr2=0x7feacc19b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.482+0000 7feaca800700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feacc104ec0 0x7feacc19b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.482+0000 7feaca800700 1 -- 172.21.15.6:0/1009532414 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feacc1a0350 con 0x7feacc109fa0 2024-04-26T01:44:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.482+0000 7feac9400700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc105870 0x7feacc19b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.483+0000 7feaca800700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feacc109fa0 0x7feacc19fc10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feac400efb0 tx=0x7feac400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.483+0000 7feac2a00700 1 -- 172.21.15.6:0/1009532414 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac4003a00 con 0x7feacc109fa0 2024-04-26T01:44:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.483+0000 7fead0c00700 1 -- 172.21.15.6:0/1009532414 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feacc1a0640 con 0x7feacc109fa0 2024-04-26T01:44:29.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.483+0000 7feac2a00700 1 -- 172.21.15.6:0/1009532414 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feac400ce50 con 0x7feacc109fa0 2024-04-26T01:44:29.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.483+0000 7feac2a00700 1 -- 172.21.15.6:0/1009532414 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac40047c0 con 0x7feacc109fa0 2024-04-26T01:44:29.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.483+0000 7fead0c00700 1 -- 172.21.15.6:0/1009532414 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feacc1a7ec0 con 0x7feacc109fa0 2024-04-26T01:44:29.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.484+0000 7fead0c00700 1 -- 172.21.15.6:0/1009532414 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feacc069730 con 0x7feacc109fa0 2024-04-26T01:44:29.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.487+0000 7feac2a00700 1 -- 172.21.15.6:0/1009532414 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7feac4012050 con 0x7feacc109fa0 2024-04-26T01:44:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.488+0000 7feac2a00700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7feab00a3cc0 0x7feab00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.489+0000 7feac2a00700 1 -- 172.21.15.6:0/1009532414 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7feac40cfc00 con 0x7feacc109fa0 2024-04-26T01:44:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.489+0000 7feac9e00700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7feab00a3cc0 0x7feab00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:29.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.489+0000 7feac9e00700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7feab00a3cc0 0x7feab00a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7feabc009990 tx=0x7feabc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:29.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.493+0000 7feac2a00700 1 -- 172.21.15.6:0/1009532414 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feac4022060 con 0x7feacc109fa0 2024-04-26T01:44:29.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.777+0000 7fead0c00700 1 -- 172.21.15.6:0/1009532414 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7feacc02e140 con 0x7feacc109fa0 2024-04-26T01:44:29.775 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:44:29.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.778+0000 7feac2a00700 1 -- 172.21.15.6:0/1009532414 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7feac4010040 con 0x7feacc109fa0 2024-04-26T01:44:29.776 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T01:44:29.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.782+0000 7fead0c00700 1 -- 172.21.15.6:0/1009532414 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7feab00a3cc0 msgr2=0x7feab00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:29.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.782+0000 7fead0c00700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7feab00a3cc0 0x7feab00a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7feabc009990 tx=0x7feabc005960 comp rx=0 tx=0).stop 2024-04-26T01:44:29.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.782+0000 7fead0c00700 1 -- 172.21.15.6:0/1009532414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feacc109fa0 msgr2=0x7feacc19fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:29.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.782+0000 7fead0c00700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feacc109fa0 0x7feacc19fc10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feac400efb0 tx=0x7feac400c3a0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.782+0000 7fead0c00700 1 -- 172.21.15.6:0/1009532414 shutdown_connections 2024-04-26T01:44:29.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.782+0000 7fead0c00700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7feab00a3cc0 0x7feab00a6180 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.782+0000 7fead0c00700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feacc109fa0 0x7feacc19fc10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.782+0000 7fead0c00700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feacc104ec0 0x7feacc19b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.782+0000 7fead0c00700 1 --2- 172.21.15.6:0/1009532414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc105870 0x7feacc19b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.782+0000 7fead0c00700 1 -- 172.21.15.6:0/1009532414 >> 172.21.15.6:0/1009532414 conn(0x7feacc100750 msgr2=0x7feacc102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.782+0000 7fead0c00700 1 -- 172.21.15.6:0/1009532414 shutdown_connections 2024-04-26T01:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:29.782+0000 7fead0c00700 1 -- 172.21.15.6:0/1009532414 wait complete. 2024-04-26T01:44:30.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:29 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1009532414' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T01:44:30.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:29 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1009532414' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T01:44:30.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:29 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1009532414' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T01:44:30.654 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T01:44:30.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:44:31.137 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:31.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:30 smithi188 ceph-mon[32751]: pgmap v406: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-26T01:44:31.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:30 smithi029 ceph-mon[32775]: pgmap v406: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-26T01:44:31.287 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:30 smithi006 ceph-mon[28419]: pgmap v406: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-26T01:44:33.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:32 smithi188 ceph-mon[32751]: pgmap v407: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:44:33.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:32 smithi029 ceph-mon[32775]: pgmap v407: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:44:33.201 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:32 smithi006 ceph-mon[28419]: pgmap v407: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:44:33.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.377+0000 7f7d72400700 1 -- 172.21.15.6:0/3200757281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c10f470 msgr2=0x7f7d6c111860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.377+0000 7f7d72400700 1 --2- 172.21.15.6:0/3200757281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c10f470 0x7f7d6c111860 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7d6400b0a0 tx=0x7f7d6400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.377+0000 7f7d72400700 1 -- 172.21.15.6:0/3200757281 shutdown_connections 2024-04-26T01:44:33.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.377+0000 7f7d72400700 1 --2- 172.21.15.6:0/3200757281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c10f470 0x7f7d6c111860 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.377+0000 7f7d72400700 1 --2- 172.21.15.6:0/3200757281 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d6c074c70 0x7f7d6c075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.377+0000 7f7d72400700 1 --2- 172.21.15.6:0/3200757281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d6c075590 0x7f7d6c10ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.377+0000 7f7d72400700 1 -- 172.21.15.6:0/3200757281 >> 172.21.15.6:0/3200757281 conn(0x7f7d6c079bf0 msgr2=0x7f7d6c07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:33.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.377+0000 7f7d72400700 1 -- 172.21.15.6:0/3200757281 shutdown_connections 2024-04-26T01:44:33.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.377+0000 7f7d72400700 1 -- 172.21.15.6:0/3200757281 wait complete. 2024-04-26T01:44:33.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.378+0000 7f7d72400700 1 Processor -- start 2024-04-26T01:44:33.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.379+0000 7f7d72400700 1 -- start start 2024-04-26T01:44:33.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.379+0000 7f7d72400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c074c70 0x7f7d6c102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.379+0000 7f7d72400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d6c075590 0x7f7d6c1030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.379+0000 7f7d72400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d6c10f470 0x7f7d6c103610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.379+0000 7f7d72400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c114e60 con 0x7f7d6c10f470 2024-04-26T01:44:33.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.379+0000 7f7d72400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c114ce0 con 0x7f7d6c074c70 2024-04-26T01:44:33.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.379+0000 7f7d72400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c114fe0 con 0x7f7d6c075590 2024-04-26T01:44:33.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.379+0000 7f7d6be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d6c10f470 0x7f7d6c103610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.379+0000 7f7d6b400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c074c70 0x7f7d6c102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.379+0000 7f7d6be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d6c10f470 0x7f7d6c103610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47316/0 (socket says 172.21.15.6:47316) 2024-04-26T01:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.379+0000 7f7d6aa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d6c075590 0x7f7d6c1030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.379+0000 7f7d6be00700 1 -- 172.21.15.6:0/1097019783 learned_addr learned my addr 172.21.15.6:0/1097019783 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.380+0000 7f7d6be00700 1 -- 172.21.15.6:0/1097019783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d6c075590 msgr2=0x7f7d6c1030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.380+0000 7f7d6be00700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d6c075590 0x7f7d6c1030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.380+0000 7f7d6be00700 1 -- 172.21.15.6:0/1097019783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c074c70 msgr2=0x7f7d6c102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.380+0000 7f7d6be00700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c074c70 0x7f7d6c102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.380+0000 7f7d6be00700 1 -- 172.21.15.6:0/1097019783 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c1ab9c0 con 0x7f7d6c10f470 2024-04-26T01:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.380+0000 7f7d6b400700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c074c70 0x7f7d6c102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.380+0000 7f7d6aa00700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d6c075590 0x7f7d6c1030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.380+0000 7f7d6be00700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d6c10f470 0x7f7d6c103610 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d6400b070 tx=0x7f7d64004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:33.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.381+0000 7f7d63e00700 1 -- 172.21.15.6:0/1097019783 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d64021070 con 0x7f7d6c10f470 2024-04-26T01:44:33.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.381+0000 7f7d72400700 1 -- 172.21.15.6:0/1097019783 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d6c1abc50 con 0x7f7d6c10f470 2024-04-26T01:44:33.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.381+0000 7f7d63e00700 1 -- 172.21.15.6:0/1097019783 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d64007e30 con 0x7f7d6c10f470 2024-04-26T01:44:33.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.381+0000 7f7d63e00700 1 -- 172.21.15.6:0/1097019783 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d64004690 con 0x7f7d6c10f470 2024-04-26T01:44:33.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.381+0000 7f7d72400700 1 -- 172.21.15.6:0/1097019783 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d6c1ac190 con 0x7f7d6c10f470 2024-04-26T01:44:33.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.382+0000 7f7d72400700 1 -- 172.21.15.6:0/1097019783 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d6c0763b0 con 0x7f7d6c10f470 2024-04-26T01:44:33.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.384+0000 7f7d63e00700 1 -- 172.21.15.6:0/1097019783 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f7d64007980 con 0x7f7d6c10f470 2024-04-26T01:44:33.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.386+0000 7f7d63e00700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7d500a3cf0 0x7f7d500a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.386+0000 7f7d63e00700 1 -- 172.21.15.6:0/1097019783 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f7d640d1220 con 0x7f7d6c10f470 2024-04-26T01:44:33.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.386+0000 7f7d6b400700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7d500a3cf0 0x7f7d500a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.387+0000 7f7d6b400700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7d500a3cf0 0x7f7d500a61b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c011490 tx=0x7f7d5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:33.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.392+0000 7f7d63e00700 1 -- 172.21.15.6:0/1097019783 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d6409dbf0 con 0x7f7d6c10f470 2024-04-26T01:44:33.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.673+0000 7f7d72400700 1 -- 172.21.15.6:0/1097019783 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d6c1ac3f0 con 0x7f7d6c10f470 2024-04-26T01:44:33.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.675+0000 7f7d63e00700 1 -- 172.21.15.6:0/1097019783 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11850 (secure 0 0 0) 0x7f7d6409d4f0 con 0x7f7d6c10f470 2024-04-26T01:44:33.673 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:44:33.674 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:42:31.294242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:44:33.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.678+0000 7f7d72400700 1 -- 172.21.15.6:0/1097019783 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7d500a3cf0 msgr2=0x7f7d500a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.678+0000 7f7d72400700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7d500a3cf0 0x7f7d500a61b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c011490 tx=0x7f7d5c009260 comp rx=0 tx=0).stop 2024-04-26T01:44:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.678+0000 7f7d72400700 1 -- 172.21.15.6:0/1097019783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d6c10f470 msgr2=0x7f7d6c103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.678+0000 7f7d72400700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d6c10f470 0x7f7d6c103610 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d6400b070 tx=0x7f7d64004d00 comp rx=0 tx=0).stop 2024-04-26T01:44:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.678+0000 7f7d72400700 1 -- 172.21.15.6:0/1097019783 shutdown_connections 2024-04-26T01:44:33.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.678+0000 7f7d72400700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d6c10f470 0x7f7d6c103610 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.678+0000 7f7d72400700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c074c70 0x7f7d6c102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.678+0000 7f7d72400700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f7d500a3cf0 0x7f7d500a61b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.678+0000 7f7d72400700 1 --2- 172.21.15.6:0/1097019783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d6c075590 0x7f7d6c1030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.678+0000 7f7d72400700 1 -- 172.21.15.6:0/1097019783 >> 172.21.15.6:0/1097019783 conn(0x7f7d6c079bf0 msgr2=0x7f7d6c0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:33.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.679+0000 7f7d72400700 1 -- 172.21.15.6:0/1097019783 shutdown_connections 2024-04-26T01:44:33.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:33.679+0000 7f7d72400700 1 -- 172.21.15.6:0/1097019783 wait complete. 2024-04-26T01:44:33.678 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-26T01:44:34.058 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:33 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1097019783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:34.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:33 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1097019783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:34.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:33 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1097019783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:34.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:44:34.993 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:35.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:34 smithi188 ceph-mon[32751]: pgmap v408: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T01:44:35.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:34 smithi006 ceph-mon[28419]: pgmap v408: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T01:44:35.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:34 smithi029 ceph-mon[32775]: pgmap v408: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T01:44:37.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:36 smithi006 ceph-mon[28419]: pgmap v409: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:37.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:36 smithi029 ceph-mon[32775]: pgmap v409: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:37.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.242+0000 7fc95ac00700 1 -- 172.21.15.6:0/566183376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc95410e3b0 msgr2=0x7fc954115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:37.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.242+0000 7fc95ac00700 1 --2- 172.21.15.6:0/566183376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc95410e3b0 0x7fc954115f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00b0a0 tx=0x7fc94c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.242+0000 7fc95ac00700 1 -- 172.21.15.6:0/566183376 shutdown_connections 2024-04-26T01:44:37.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.242+0000 7fc95ac00700 1 --2- 172.21.15.6:0/566183376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc95410e3b0 0x7fc954115f80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.242+0000 7fc95ac00700 1 --2- 172.21.15.6:0/566183376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9541092d0 0x7fc9541096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.242+0000 7fc95ac00700 1 --2- 172.21.15.6:0/566183376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc954109c80 0x7fc95410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.242+0000 7fc95ac00700 1 -- 172.21.15.6:0/566183376 >> 172.21.15.6:0/566183376 conn(0x7fc954076210 msgr2=0x7fc954107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:37.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.242+0000 7fc95ac00700 1 -- 172.21.15.6:0/566183376 shutdown_connections 2024-04-26T01:44:37.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.242+0000 7fc95ac00700 1 -- 172.21.15.6:0/566183376 wait complete. 2024-04-26T01:44:37.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.243+0000 7fc95ac00700 1 Processor -- start 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.243+0000 7fc95ac00700 1 -- start start 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.244+0000 7fc95ac00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9541092d0 0x7fc95419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.244+0000 7fc95ac00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc954109c80 0x7fc95419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.244+0000 7fc95ac00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95410e3b0 0x7fc9541a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.244+0000 7fc95ac00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541192a0 con 0x7fc9541092d0 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.244+0000 7fc95ac00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc954119120 con 0x7fc954109c80 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.244+0000 7fc95ac00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc954119420 con 0x7fc95410e3b0 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.244+0000 7fc953e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9541092d0 0x7fc95419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.244+0000 7fc953400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc954109c80 0x7fc95419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.244+0000 7fc953400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc954109c80 0x7fc95419fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:49298/0 (socket says 172.21.15.6:49298) 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.244+0000 7fc953e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9541092d0 0x7fc95419f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47342/0 (socket says 172.21.15.6:47342) 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.244+0000 7fc953400700 1 -- 172.21.15.6:0/904267687 learned_addr learned my addr 172.21.15.6:0/904267687 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.244+0000 7fc958c00700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95410e3b0 0x7fc9541a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.245+0000 7fc953400700 1 -- 172.21.15.6:0/904267687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95410e3b0 msgr2=0x7fc9541a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:37.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.245+0000 7fc953400700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95410e3b0 0x7fc9541a4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.245+0000 7fc953400700 1 -- 172.21.15.6:0/904267687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9541092d0 msgr2=0x7fc95419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:37.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.245+0000 7fc953400700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9541092d0 0x7fc95419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.245+0000 7fc953400700 1 -- 172.21.15.6:0/904267687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9541a4740 con 0x7fc954109c80 2024-04-26T01:44:37.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.245+0000 7fc958c00700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95410e3b0 0x7fc9541a4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:37.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.245+0000 7fc953400700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc954109c80 0x7fc95419fc70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc94000e970 tx=0x7fc94000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:37.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.245+0000 7fc950c00700 1 -- 172.21.15.6:0/904267687 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94000cbd0 con 0x7fc954109c80 2024-04-26T01:44:37.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.245+0000 7fc95ac00700 1 -- 172.21.15.6:0/904267687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9541a4a30 con 0x7fc954109c80 2024-04-26T01:44:37.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.245+0000 7fc950c00700 1 -- 172.21.15.6:0/904267687 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc94000cd70 con 0x7fc954109c80 2024-04-26T01:44:37.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.245+0000 7fc950c00700 1 -- 172.21.15.6:0/904267687 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940005680 con 0x7fc954109c80 2024-04-26T01:44:37.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.246+0000 7fc95ac00700 1 -- 172.21.15.6:0/904267687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9541ac2b0 con 0x7fc954109c80 2024-04-26T01:44:37.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.246+0000 7fc95ac00700 1 -- 172.21.15.6:0/904267687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc954069730 con 0x7fc954109c80 2024-04-26T01:44:37.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.249+0000 7fc950c00700 1 -- 172.21.15.6:0/904267687 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fc940004750 con 0x7fc954109c80 2024-04-26T01:44:37.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.250+0000 7fc950c00700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc9380a3cc0 0x7fc9380a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:37.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.250+0000 7fc950c00700 1 -- 172.21.15.6:0/904267687 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fc9400d05c0 con 0x7fc954109c80 2024-04-26T01:44:37.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.250+0000 7fc953e00700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc9380a3cc0 0x7fc9380a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:37.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.251+0000 7fc953e00700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc9380a3cc0 0x7fc9380a6180 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc944005950 tx=0x7fc94400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:37.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.254+0000 7fc950c00700 1 -- 172.21.15.6:0/904267687 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc94009cf00 con 0x7fc954109c80 2024-04-26T01:44:37.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:36 smithi188 ceph-mon[32751]: pgmap v409: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:37.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.547+0000 7fc95ac00700 1 -- 172.21.15.6:0/904267687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc95402df80 con 0x7fc954109c80 2024-04-26T01:44:37.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.548+0000 7fc950c00700 1 -- 172.21.15.6:0/904267687 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11850 (secure 0 0 0) 0x7fc94009c800 con 0x7fc954109c80 2024-04-26T01:44:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:44:37.547 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:42:31.294242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:44:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.553+0000 7fc95ac00700 1 -- 172.21.15.6:0/904267687 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc9380a3cc0 msgr2=0x7fc9380a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.553+0000 7fc95ac00700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc9380a3cc0 0x7fc9380a6180 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc944005950 tx=0x7fc94400b410 comp rx=0 tx=0).stop 2024-04-26T01:44:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.553+0000 7fc95ac00700 1 -- 172.21.15.6:0/904267687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc954109c80 msgr2=0x7fc95419fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.553+0000 7fc95ac00700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc954109c80 0x7fc95419fc70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc94000e970 tx=0x7fc94000ec80 comp rx=0 tx=0).stop 2024-04-26T01:44:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.553+0000 7fc95ac00700 1 -- 172.21.15.6:0/904267687 shutdown_connections 2024-04-26T01:44:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.553+0000 7fc95ac00700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc9380a3cc0 0x7fc9380a6180 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.553+0000 7fc95ac00700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95410e3b0 0x7fc9541a4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.553+0000 7fc95ac00700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9541092d0 0x7fc95419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.553+0000 7fc95ac00700 1 --2- 172.21.15.6:0/904267687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc954109c80 0x7fc95419fc70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.553+0000 7fc95ac00700 1 -- 172.21.15.6:0/904267687 >> 172.21.15.6:0/904267687 conn(0x7fc954076210 msgr2=0x7fc954076790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.553+0000 7fc95ac00700 1 -- 172.21.15.6:0/904267687 shutdown_connections 2024-04-26T01:44:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:37.553+0000 7fc95ac00700 1 -- 172.21.15.6:0/904267687 wait complete. 2024-04-26T01:44:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-26T01:44:38.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:37 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/904267687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:38.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:37 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/904267687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:38.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:37 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/904267687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:38.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:44:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:39.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:38 smithi029 ceph-mon[32775]: pgmap v410: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:39.199 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:38 smithi006 ceph-mon[28419]: pgmap v410: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:39.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:38 smithi188 ceph-mon[32751]: pgmap v410: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:41.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:40 smithi006 ceph-mon[28419]: pgmap v411: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:41.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:40 smithi029 ceph-mon[32775]: pgmap v411: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.288+0000 7f608fc00700 1 -- 172.21.15.6:0/2836723104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6088103450 msgr2=0x7f608810f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.288+0000 7f608fc00700 1 --2- 172.21.15.6:0/2836723104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6088103450 0x7f608810f8b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f608000b0a0 tx=0x7f608000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.289+0000 7f608fc00700 1 -- 172.21.15.6:0/2836723104 shutdown_connections 2024-04-26T01:44:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.289+0000 7f608fc00700 1 --2- 172.21.15.6:0/2836723104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6088103450 0x7f608810f8b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.289+0000 7f608fc00700 1 --2- 172.21.15.6:0/2836723104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6088108a50 0x7f6088108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.289+0000 7f608fc00700 1 --2- 172.21.15.6:0/2836723104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6088102a90 0x7f6088102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.289+0000 7f608fc00700 1 -- 172.21.15.6:0/2836723104 >> 172.21.15.6:0/2836723104 conn(0x7f6088079ba0 msgr2=0x7f6088079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.289+0000 7f608fc00700 1 -- 172.21.15.6:0/2836723104 shutdown_connections 2024-04-26T01:44:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.289+0000 7f608fc00700 1 -- 172.21.15.6:0/2836723104 wait complete. 2024-04-26T01:44:41.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.290+0000 7f608fc00700 1 Processor -- start 2024-04-26T01:44:41.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.290+0000 7f608fc00700 1 -- start start 2024-04-26T01:44:41.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.290+0000 7f608fc00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6088102a90 0x7f60881991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.290+0000 7f608fc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6088103450 0x7f6088199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.290+0000 7f608fc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6088108a50 0x7f608819dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.290+0000 7f608fc00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6088112c50 con 0x7f6088108a50 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.290+0000 7f608fc00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6088112ad0 con 0x7f6088102a90 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.290+0000 7f608fc00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6088112dd0 con 0x7f6088103450 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.291+0000 7f608d200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6088102a90 0x7f60881991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.291+0000 7f608d200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6088102a90 0x7f60881991e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55100/0 (socket says 172.21.15.6:55100) 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.291+0000 7f608d200700 1 -- 172.21.15.6:0/853711167 learned_addr learned my addr 172.21.15.6:0/853711167 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.291+0000 7f6087e00700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6088103450 0x7f6088199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.291+0000 7f608dc00700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6088108a50 0x7f608819dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.291+0000 7f6087e00700 1 -- 172.21.15.6:0/853711167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6088102a90 msgr2=0x7f60881991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.291+0000 7f6087e00700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6088102a90 0x7f60881991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.291+0000 7f6087e00700 1 -- 172.21.15.6:0/853711167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6088108a50 msgr2=0x7f608819dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.291+0000 7f6087e00700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6088108a50 0x7f608819dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.291+0000 7f6087e00700 1 -- 172.21.15.6:0/853711167 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608819e1f0 con 0x7f6088103450 2024-04-26T01:44:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.292+0000 7f6087e00700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6088103450 0x7f6088199720 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f607400e9f0 tx=0x7f607400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.292+0000 7f6085600700 1 -- 172.21.15.6:0/853711167 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607400c9c0 con 0x7f6088103450 2024-04-26T01:44:41.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.292+0000 7f6085600700 1 -- 172.21.15.6:0/853711167 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607400cb60 con 0x7f6088103450 2024-04-26T01:44:41.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.292+0000 7f6085600700 1 -- 172.21.15.6:0/853711167 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074018680 con 0x7f6088103450 2024-04-26T01:44:41.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.292+0000 7f608fc00700 1 -- 172.21.15.6:0/853711167 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f608819e4e0 con 0x7f6088103450 2024-04-26T01:44:41.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.292+0000 7f608fc00700 1 -- 172.21.15.6:0/853711167 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60881a5d60 con 0x7f6088103450 2024-04-26T01:44:41.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.293+0000 7f608fc00700 1 -- 172.21.15.6:0/853711167 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6088069730 con 0x7f6088103450 2024-04-26T01:44:41.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.295+0000 7f6085600700 1 -- 172.21.15.6:0/853711167 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f6074010430 con 0x7f6088103450 2024-04-26T01:44:41.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.297+0000 7f6085600700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f60700a3cc0 0x7f60700a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.297+0000 7f6085600700 1 -- 172.21.15.6:0/853711167 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f60740d08a0 con 0x7f6088103450 2024-04-26T01:44:41.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.297+0000 7f608d200700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f60700a3cc0 0x7f60700a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.298+0000 7f608d200700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f60700a3cc0 0x7f60700a6180 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f607c009ca0 tx=0x7f607c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:41.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.299+0000 7f6085600700 1 -- 172.21.15.6:0/853711167 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f607409d1e0 con 0x7f6088103450 2024-04-26T01:44:41.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:40 smithi188 ceph-mon[32751]: pgmap v411: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:41.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.533+0000 7f608fc00700 1 -- 172.21.15.6:0/853711167 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f608802df80 con 0x7f6088103450 2024-04-26T01:44:41.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.535+0000 7f6085600700 1 -- 172.21.15.6:0/853711167 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+22684 (secure 0 0 0) 0x7f607409cae0 con 0x7f6088103450 2024-04-26T01:44:41.534 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:44:41.534 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":95,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:32:29.071544+0000","modified":"2024-04-26T01:43:49.972785+0000","last_up_change":"2024-04-26T01:40:49.915924+0000","last_in_change":"2024-04-26T01:40:29.559647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:36:35.182726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:41:59.612167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:42:04.106467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f164c51d-ce5d-4140-b9e3-a4b84088f5ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6803","nonce":76518486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6805","nonce":76518486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6809","nonce":76518486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6807","nonce":76518486}]},"public_addr":"172.21.15.6:6803/76518486","cluster_addr":"172.21.15.6:6805/76518486","heartbeat_back_addr":"172.21.15.6:6809/76518486","heartbeat_front_addr":"172.21.15.6:6807/76518486","state":["exists","up"]},{"osd":1,"uuid":"57e875dd-ddf1-4ee4-a2a7-9589d7f42818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6801","nonce":400769116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6803","nonce":400769116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6807","nonce":400769116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6805","nonce":400769116}]},"public_addr":"172.21.15.29:6801/400769116","cluster_addr":"172.21.15.29:6803/400769116","heartbeat_back_addr":"172.21.15.29:6807/400769116","heartbeat_front_addr":"172.21.15.29:6805/400769116","state":["exists","up"]},{"osd":2,"uuid":"77e0c294-c48a-4318-938b-4604a533a2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6801","nonce":3508067672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6803","nonce":3508067672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6807","nonce":3508067672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6805","nonce":3508067672}]},"public_addr":"172.21.15.188:6801/3508067672","cluster_addr":"172.21.15.188:6803/3508067672","heartbeat_back_addr":"172.21.15.188:6807/3508067672","heartbeat_front_addr":"172.21.15.188:6805/3508067672","state":["exists","up"]},{"osd":3,"uuid":"6393dac5-996c-47af-9687-0f1393ecc483","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6811","nonce":1987311118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6813","nonce":1987311118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6817","nonce":1987311118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6815","nonce":1987311118}]},"public_addr":"172.21.15.6:6811/1987311118","cluster_addr":"172.21.15.6:6813/1987311118","heartbeat_back_addr":"172.21.15.6:6817/1987311118","heartbeat_front_addr":"172.21.15.6:6815/1987311118","state":["exists","up"]},{"osd":4,"uuid":"f2c1a535-26a0-46c6-a6d0-c2529c786739","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6809","nonce":2125876609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6811","nonce":2125876609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6815","nonce":2125876609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6813","nonce":2125876609}]},"public_addr":"172.21.15.29:6809/2125876609","cluster_addr":"172.21.15.29:6811/2125876609","heartbeat_back_addr":"172.21.15.29:6815/2125876609","heartbeat_front_addr":"172.21.15.29:6813/2125876609","state":["exists","up"]},{"osd":5,"uuid":"9788bb0a-0d79-4743-bce6-80f5342e8f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6809","nonce":2092077007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6811","nonce":2092077007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6815","nonce":2092077007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6813","nonce":2092077007}]},"public_addr":"172.21.15.188:6809/2092077007","cluster_addr":"172.21.15.188:6811/2092077007","heartbeat_back_addr":"172.21.15.188:6815/2092077007","heartbeat_front_addr":"172.21.15.188:6813/2092077007","state":["exists","up"]},{"osd":6,"uuid":"fd8584e1-69ee-4c30-85e4-a637357d1b6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6819","nonce":3256016695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6821","nonce":3256016695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6825","nonce":3256016695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6823","nonce":3256016695}]},"public_addr":"172.21.15.6:6819/3256016695","cluster_addr":"172.21.15.6:6821/3256016695","heartbeat_back_addr":"172.21.15.6:6825/3256016695","heartbeat_front_addr":"172.21.15.6:6823/3256016695","state":["exists","up"]},{"osd":7,"uuid":"6b7e6784-d574-42da-8416-a74a3b6b3755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6817","nonce":3478594037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6819","nonce":3478594037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6823","nonce":3478594037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6821","nonce":3478594037}]},"public_addr":"172.21.15.29:6817/3478594037","cluster_addr":"172.21.15.29:6819/3478594037","heartbeat_back_addr":"172.21.15.29:6823/3478594037","heartbeat_front_addr":"172.21.15.29:6821/3478594037","state":["exists","up"]},{"osd":8,"uuid":"30cd2ea1-2767-4062-8bd9-47214e4b4b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6817","nonce":3290364376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6819","nonce":3290364376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6823","nonce":3290364376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6821","nonce":3290364376}]},"public_addr":"172.21.15.188:6817/3290364376","cluster_addr":"172.21.15.188:6819/3290364376","heartbeat_back_addr":"172.21.15.188:6823/3290364376","heartbeat_front_addr":"172.21.15.188:6821/3290364376","state":["exists","up"]},{"osd":9,"uuid":"f033cdc0-1178-4a7f-a338-69ff32416087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6827","nonce":1918893912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6829","nonce":1918893912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6833","nonce":1918893912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6831","nonce":1918893912}]},"public_addr":"172.21.15.6:6827/1918893912","cluster_addr":"172.21.15.6:6829/1918893912","heartbeat_back_addr":"172.21.15.6:6833/1918893912","heartbeat_front_addr":"172.21.15.6:6831/1918893912","state":["exists","up"]},{"osd":10,"uuid":"04e9b5ef-ac0c-482b-a69e-a7121b36e7da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6825","nonce":1395777729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6827","nonce":1395777729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6831","nonce":1395777729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6829","nonce":1395777729}]},"public_addr":"172.21.15.29:6825/1395777729","cluster_addr":"172.21.15.29:6827/1395777729","heartbeat_back_addr":"172.21.15.29:6831/1395777729","heartbeat_front_addr":"172.21.15.29:6829/1395777729","state":["exists","up"]},{"osd":11,"uuid":"cb1638ce-4189-4241-b63e-b63c607e90ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6825","nonce":3322197880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6827","nonce":3322197880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6831","nonce":3322197880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6829","nonce":3322197880}]},"public_addr":"172.21.15.188:6825/3322197880","cluster_addr":"172.21.15.188:6827/3322197880","heartbeat_back_addr":"172.21.15.188:6831/3322197880","heartbeat_front_addr":"172.21.15.188:6829/3322197880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:41.553455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:07.641467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:31.748891+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:03.743553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:30.065262+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:54.976527+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:29.348206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:56.285928+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:21.246409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:54.468561+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:22.093015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:47.342379+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.19","mappings":[{"from":3,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/3143272062":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1325707596":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6800/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6801/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1359518071":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6801/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/2813598711":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/3140150861":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6800/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/65701414":"2024-04-27T01:33:47.021882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:44:41.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.539+0000 7f608fc00700 1 -- 172.21.15.6:0/853711167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f60700a3cc0 msgr2=0x7f60700a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.539+0000 7f608fc00700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f60700a3cc0 0x7f60700a6180 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f607c009ca0 tx=0x7f607c009260 comp rx=0 tx=0).stop 2024-04-26T01:44:41.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.539+0000 7f608fc00700 1 -- 172.21.15.6:0/853711167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6088103450 msgr2=0x7f6088199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.539+0000 7f608fc00700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6088103450 0x7f6088199720 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f607400e9f0 tx=0x7f607400ed00 comp rx=0 tx=0).stop 2024-04-26T01:44:41.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.539+0000 7f608fc00700 1 -- 172.21.15.6:0/853711167 shutdown_connections 2024-04-26T01:44:41.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.539+0000 7f608fc00700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6088108a50 0x7f608819dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.539+0000 7f608fc00700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6088102a90 0x7f60881991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.539+0000 7f608fc00700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f60700a3cc0 0x7f60700a6180 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.539+0000 7f608fc00700 1 --2- 172.21.15.6:0/853711167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6088103450 0x7f6088199720 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.539+0000 7f608fc00700 1 -- 172.21.15.6:0/853711167 >> 172.21.15.6:0/853711167 conn(0x7f6088079ba0 msgr2=0x7f6088101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:41.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.540+0000 7f608fc00700 1 -- 172.21.15.6:0/853711167 shutdown_connections 2024-04-26T01:44:41.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:41.540+0000 7f608fc00700 1 -- 172.21.15.6:0/853711167 wait complete. 2024-04-26T01:44:42.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:41 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/853711167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:44:42.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:41 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/853711167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:44:42.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:41 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/853711167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:44:42.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:44:42.936 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:43.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:42 smithi029 ceph-mon[32775]: pgmap v412: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:43.245 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:42 smithi006 ceph-mon[28419]: pgmap v412: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:43.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:42 smithi188 ceph-mon[32751]: pgmap v412: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:45.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:44 smithi006 ceph-mon[28419]: pgmap v413: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:45.190 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:44 smithi029 ceph-mon[32775]: pgmap v413: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.216+0000 7fe76e200700 1 -- 172.21.15.6:0/2681421045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe76810e3b0 msgr2=0x7fe768115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.216+0000 7fe76e200700 1 --2- 172.21.15.6:0/2681421045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe76810e3b0 0x7fe768115f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe76000b0a0 tx=0x7fe76000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:44:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.216+0000 7fe76e200700 1 -- 172.21.15.6:0/2681421045 shutdown_connections 2024-04-26T01:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.216+0000 7fe76e200700 1 --2- 172.21.15.6:0/2681421045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe76810e3b0 0x7fe768115f80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.216+0000 7fe76e200700 1 --2- 172.21.15.6:0/2681421045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7681092d0 0x7fe7681096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.216+0000 7fe76e200700 1 --2- 172.21.15.6:0/2681421045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe768109c80 0x7fe76810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.216+0000 7fe76e200700 1 -- 172.21.15.6:0/2681421045 >> 172.21.15.6:0/2681421045 conn(0x7fe768076210 msgr2=0x7fe768107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.216+0000 7fe76e200700 1 -- 172.21.15.6:0/2681421045 shutdown_connections 2024-04-26T01:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.216+0000 7fe76e200700 1 -- 172.21.15.6:0/2681421045 wait complete. 2024-04-26T01:44:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.218+0000 7fe76e200700 1 Processor -- start 2024-04-26T01:44:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.218+0000 7fe76e200700 1 -- start start 2024-04-26T01:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.218+0000 7fe76e200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7681092d0 0x7fe76819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.218+0000 7fe76e200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe768109c80 0x7fe76819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.218+0000 7fe76e200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe76810e3b0 0x7fe7681a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.218+0000 7fe76e200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe7681192c0 con 0x7fe768109c80 2024-04-26T01:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.218+0000 7fe76e200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe768119140 con 0x7fe7681092d0 2024-04-26T01:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.218+0000 7fe76e200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe768119440 con 0x7fe76810e3b0 2024-04-26T01:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.219+0000 7fe767400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7681092d0 0x7fe76819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.219+0000 7fe767400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7681092d0 0x7fe76819f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55118/0 (socket says 172.21.15.6:55118) 2024-04-26T01:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.219+0000 7fe766a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe768109c80 0x7fe76819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.219+0000 7fe767400700 1 -- 172.21.15.6:0/148805978 learned_addr learned my addr 172.21.15.6:0/148805978 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.219+0000 7fe767e00700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe76810e3b0 0x7fe7681a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.219+0000 7fe767400700 1 -- 172.21.15.6:0/148805978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe76810e3b0 msgr2=0x7fe7681a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.219+0000 7fe767400700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe76810e3b0 0x7fe7681a4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.219+0000 7fe767400700 1 -- 172.21.15.6:0/148805978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe768109c80 msgr2=0x7fe76819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.219+0000 7fe767400700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe768109c80 0x7fe76819fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.219+0000 7fe767400700 1 -- 172.21.15.6:0/148805978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7681a4760 con 0x7fe7681092d0 2024-04-26T01:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.219+0000 7fe767e00700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe76810e3b0 0x7fe7681a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.219+0000 7fe767400700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7681092d0 0x7fe76819f750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe75800e9c0 tx=0x7fe75800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.220+0000 7fe75fe00700 1 -- 172.21.15.6:0/148805978 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75800cbd0 con 0x7fe7681092d0 2024-04-26T01:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.220+0000 7fe75fe00700 1 -- 172.21.15.6:0/148805978 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe75800cd70 con 0x7fe7681092d0 2024-04-26T01:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.220+0000 7fe76e200700 1 -- 172.21.15.6:0/148805978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7681a4a50 con 0x7fe7681092d0 2024-04-26T01:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.220+0000 7fe75fe00700 1 -- 172.21.15.6:0/148805978 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe758005610 con 0x7fe7681092d0 2024-04-26T01:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.220+0000 7fe76e200700 1 -- 172.21.15.6:0/148805978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7681ac2d0 con 0x7fe7681092d0 2024-04-26T01:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.221+0000 7fe76e200700 1 -- 172.21.15.6:0/148805978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe768069730 con 0x7fe7681092d0 2024-04-26T01:44:45.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.223+0000 7fe75fe00700 1 -- 172.21.15.6:0/148805978 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fe758004750 con 0x7fe7681092d0 2024-04-26T01:44:45.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.224+0000 7fe75fe00700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe74c0a3cc0 0x7fe74c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:45.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.224+0000 7fe75fe00700 1 -- 172.21.15.6:0/148805978 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fe7580d0750 con 0x7fe7681092d0 2024-04-26T01:44:45.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.225+0000 7fe766a00700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe74c0a3cc0 0x7fe74c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:45.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.225+0000 7fe766a00700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe74c0a3cc0 0x7fe74c0a6180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe754009680 tx=0x7fe754009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.228+0000 7fe75fe00700 1 -- 172.21.15.6:0/148805978 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe75809d090 con 0x7fe7681092d0 2024-04-26T01:44:45.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:44 smithi188 ceph-mon[32751]: pgmap v413: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.527+0000 7fe76e200700 1 -- 172.21.15.6:0/148805978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7681a0ab0 con 0x7fe7681092d0 2024-04-26T01:44:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.528+0000 7fe75fe00700 1 -- 172.21.15.6:0/148805978 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11850 (secure 0 0 0) 0x7fe75809c990 con 0x7fe7681092d0 2024-04-26T01:44:45.526 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:44:45.526 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:42:31.294242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:44:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.531+0000 7fe76e200700 1 -- 172.21.15.6:0/148805978 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe74c0a3cc0 msgr2=0x7fe74c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.531+0000 7fe76e200700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe74c0a3cc0 0x7fe74c0a6180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe754009680 tx=0x7fe754009260 comp rx=0 tx=0).stop 2024-04-26T01:44:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.531+0000 7fe76e200700 1 -- 172.21.15.6:0/148805978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7681092d0 msgr2=0x7fe76819f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.531+0000 7fe76e200700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7681092d0 0x7fe76819f750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe75800e9c0 tx=0x7fe75800ecd0 comp rx=0 tx=0).stop 2024-04-26T01:44:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.531+0000 7fe76e200700 1 -- 172.21.15.6:0/148805978 shutdown_connections 2024-04-26T01:44:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.531+0000 7fe76e200700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe74c0a3cc0 0x7fe74c0a6180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.531+0000 7fe76e200700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe76810e3b0 0x7fe7681a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.531+0000 7fe76e200700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7681092d0 0x7fe76819f750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:45.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.531+0000 7fe76e200700 1 --2- 172.21.15.6:0/148805978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe768109c80 0x7fe76819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:45.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.531+0000 7fe76e200700 1 -- 172.21.15.6:0/148805978 >> 172.21.15.6:0/148805978 conn(0x7fe768076210 msgr2=0x7fe768076790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:45.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.531+0000 7fe76e200700 1 -- 172.21.15.6:0/148805978 shutdown_connections 2024-04-26T01:44:45.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:45.532+0000 7fe76e200700 1 -- 172.21.15.6:0/148805978 wait complete. 2024-04-26T01:44:45.531 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-26T01:44:46.270 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:45 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/148805978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:46.352 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T01:42:08.299563+0000', 'modified': '2024-04-26T01:42:31.294242+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24523, 'mds_1': 24512, 'mds_2': 24505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6837/402171153', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6836', 'nonce': 402171153}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 402171153}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'g', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.6:6839/3936238112', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6838', 'nonce': 3936238112}, {'type': 'v1', 'addr': '172.21.15.6:6839', 'nonce': 3936238112}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6835/1255666766', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6834', 'nonce': 1255666766}, {'type': 'v1', 'addr': '172.21.15.188:6835', 'nonce': 1255666766}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.188:6837/489767141', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6836', 'nonce': 489767141}, {'type': 'v1', 'addr': '172.21.15.188:6837', 'nonce': 489767141}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24512': {'gid': 24512, 'name': 'b', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.29:6839/2325582359', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6838', 'nonce': 2325582359}, {'type': 'v1', 'addr': '172.21.15.29:6839', 'nonce': 2325582359}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.188:6839/674623421', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6838', 'nonce': 674623421}, {'type': 'v1', 'addr': '172.21.15.188:6839', 'nonce': 674623421}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24523, 'qdb_cluster': [24523, 24505, 24512]} 2024-04-26T01:44:46.352 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T01:44:46.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell mds.c status 2024-04-26T01:44:46.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:45 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/148805978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:46.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:45 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/148805978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:47.243 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:46 smithi006 ceph-mon[28419]: pgmap v414: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:47.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:46 smithi188 ceph-mon[32751]: pgmap v414: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:47.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:46 smithi029 ceph-mon[32775]: pgmap v414: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:48.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.932+0000 7f817a400700 1 -- 172.21.15.6:0/3889558285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8174077890 msgr2=0x7f8174113380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.932+0000 7f817a400700 1 --2- 172.21.15.6:0/3889558285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8174077890 0x7f8174113380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8160009990 tx=0x7f8160009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.933+0000 7f817a400700 1 -- 172.21.15.6:0/3889558285 shutdown_connections 2024-04-26T01:44:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.933+0000 7f817a400700 1 --2- 172.21.15.6:0/3889558285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81741138c0 0x7f8174115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.933+0000 7f817a400700 1 --2- 172.21.15.6:0/3889558285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8174076f70 0x7f8174077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.933+0000 7f817a400700 1 --2- 172.21.15.6:0/3889558285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8174077890 0x7f8174113380 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.933+0000 7f817a400700 1 -- 172.21.15.6:0/3889558285 >> 172.21.15.6:0/3889558285 conn(0x7f81741006f0 msgr2=0x7f8174102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.933+0000 7f817a400700 1 -- 172.21.15.6:0/3889558285 shutdown_connections 2024-04-26T01:44:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.933+0000 7f817a400700 1 -- 172.21.15.6:0/3889558285 wait complete. 2024-04-26T01:44:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.934+0000 7f817a400700 1 Processor -- start 2024-04-26T01:44:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.934+0000 7f817a400700 1 -- start start 2024-04-26T01:44:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.935+0000 7f817a400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8174076f70 0x7f81741a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.935+0000 7f817a400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8174077890 0x7f81741a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.935+0000 7f817a400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81741138c0 0x7f817419f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.935+0000 7f817a400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8174119240 con 0x7f81741138c0 2024-04-26T01:44:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.935+0000 7f817a400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f81741190c0 con 0x7f8174077890 2024-04-26T01:44:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.935+0000 7f817a400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f81741193c0 con 0x7f8174076f70 2024-04-26T01:44:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.935+0000 7f8173400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8174076f70 0x7f81741a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.935+0000 7f8173400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8174076f70 0x7f81741a5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44902/0 (socket says 172.21.15.6:44902) 2024-04-26T01:44:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.935+0000 7f8173400700 1 -- 172.21.15.6:0/3961138010 learned_addr learned my addr 172.21.15.6:0/3961138010 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.935+0000 7f8172a00700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8174077890 0x7f81741a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.935+0000 7f8173e00700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81741138c0 0x7f817419f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.936+0000 7f8173400700 1 -- 172.21.15.6:0/3961138010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8174077890 msgr2=0x7f81741a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.936+0000 7f8173400700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8174077890 0x7f81741a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.936+0000 7f8173400700 1 -- 172.21.15.6:0/3961138010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81741138c0 msgr2=0x7f817419f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.936+0000 7f8173400700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81741138c0 0x7f817419f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.936+0000 7f8173400700 1 -- 172.21.15.6:0/3961138010 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817419fd80 con 0x7f8174076f70 2024-04-26T01:44:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.936+0000 7f8173e00700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81741138c0 0x7f817419f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.936+0000 7f8173400700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8174076f70 0x7f81741a5460 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f816400e9a0 tx=0x7f816400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.936+0000 7f816be00700 1 -- 172.21.15.6:0/3961138010 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816400cbd0 con 0x7f8174076f70 2024-04-26T01:44:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.936+0000 7f817a400700 1 -- 172.21.15.6:0/3961138010 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81741a0070 con 0x7f8174076f70 2024-04-26T01:44:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.936+0000 7f816be00700 1 -- 172.21.15.6:0/3961138010 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f816400cd70 con 0x7f8174076f70 2024-04-26T01:44:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.936+0000 7f816be00700 1 -- 172.21.15.6:0/3961138010 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8164018680 con 0x7f8174076f70 2024-04-26T01:44:48.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.936+0000 7f817a400700 1 -- 172.21.15.6:0/3961138010 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81741ac330 con 0x7f8174076f70 2024-04-26T01:44:48.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.940+0000 7f816be00700 1 -- 172.21.15.6:0/3961138010 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f8164010430 con 0x7f8174076f70 2024-04-26T01:44:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.941+0000 7f816be00700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f81580a3cc0 0x7f81580a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.941+0000 7f8184e0ab80 1 Processor -- start 2024-04-26T01:44:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.941+0000 7f816be00700 1 -- 172.21.15.6:0/3961138010 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f81640d06b0 con 0x7f8174076f70 2024-04-26T01:44:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.941+0000 7f8184e0ab80 1 -- start start 2024-04-26T01:44:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.941+0000 7f8172a00700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f81580a3cc0 0x7f81580a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.942+0000 7f8184e0ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca842f0 0x55a2cca846d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.942+0000 7f8184e0ab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca84c10 0x55a2cca8f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.942+0000 7f8184e0ab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca8fab0 0x55a2cca91ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.942+0000 7f8184e0ab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a2cc9d7e40 con 0x55a2cca8fab0 2024-04-26T01:44:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.942+0000 7f8184e0ab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a2cc97e6e0 con 0x55a2cca84c10 2024-04-26T01:44:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.942+0000 7f8184e0ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a2cc9d3e00 con 0x55a2cca842f0 2024-04-26T01:44:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.942+0000 7f8173e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca8fab0 0x55a2cca91ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.942+0000 7f8172a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca84c10 0x55a2cca8f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.942+0000 7f8173400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca842f0 0x55a2cca846d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.942+0000 7f8173400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca842f0 0x55a2cca846d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44912/0 (socket says 172.21.15.6:44912) 2024-04-26T01:44:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.942+0000 7f8173400700 1 -- 172.21.15.6:0/1017120089 learned_addr learned my addr 172.21.15.6:0/1017120089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.943+0000 7f8172a00700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f81580a3cc0 0x7f81580a6180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f81600098e0 tx=0x7f8160005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:48.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.943+0000 7f8172a00700 1 -- 172.21.15.6:0/1017120089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca842f0 msgr2=0x55a2cca846d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.943+0000 7f8172a00700 1 --2- 172.21.15.6:0/1017120089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca842f0 0x55a2cca846d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.943+0000 7f8172a00700 1 -- 172.21.15.6:0/1017120089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca8fab0 msgr2=0x55a2cca91ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.943+0000 7f8172a00700 1 --2- 172.21.15.6:0/1017120089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca8fab0 0x55a2cca91ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.943+0000 7f8172a00700 1 -- 172.21.15.6:0/1017120089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2cc974630 con 0x55a2cca84c10 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.943+0000 7f8173e00700 1 --2- 172.21.15.6:0/1017120089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca8fab0 0x55a2cca91ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.943+0000 7f8172a00700 1 --2- 172.21.15.6:0/1017120089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca84c10 0x55a2cca8f570 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8160006010 tx=0x7f8160024000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74e6c17a5fb75d11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.943+0000 7f8168c00700 1 -- 172.21.15.6:0/1017120089 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160033070 con 0x55a2cca84c10 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.943+0000 7f8168c00700 1 -- 172.21.15.6:0/1017120089 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f816000be60 con 0x55a2cca84c10 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.944+0000 7f8168c00700 1 -- 172.21.15.6:0/1017120089 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816002a040 con 0x55a2cca84c10 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.944+0000 7f8184e0ab80 1 -- 172.21.15.6:0/1017120089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca84c10 msgr2=0x55a2cca8f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.944+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/1017120089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca84c10 0x55a2cca8f570 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8160006010 tx=0x7f8160024000 comp rx=0 tx=0).stop 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.945+0000 7f8184e0ab80 1 -- 172.21.15.6:0/1017120089 shutdown_connections 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.945+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/1017120089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca8fab0 0x55a2cca91ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.945+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/1017120089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca842f0 0x55a2cca846d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.945+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/1017120089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca84c10 0x55a2cca8f570 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.945+0000 7f8184e0ab80 1 -- 172.21.15.6:0/1017120089 >> 172.21.15.6:0/1017120089 conn(0x55a2cc9b3550 msgr2=0x55a2cca80790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.945+0000 7f8184e0ab80 1 -- 172.21.15.6:0/1017120089 shutdown_connections 2024-04-26T01:44:48.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.945+0000 7f8184e0ab80 1 -- 172.21.15.6:0/1017120089 wait complete. 2024-04-26T01:44:48.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.945+0000 7f8184e0ab80 1 -- start start 2024-04-26T01:44:48.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.946+0000 7f8184e0ab80 1 Processor -- start 2024-04-26T01:44:48.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.946+0000 7f8184e0ab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 0x55a2cca8f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.946+0000 7f8184e0ab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca84c10 0x55a2cca8c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.946+0000 7f8184e0ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca8fab0 0x55a2cca8ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.946+0000 7f8184e0ab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a2cca92540 con 0x55a2cca84c10 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.946+0000 7f8184e0ab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a2cc991810 con 0x55a2cca842f0 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.946+0000 7f8184e0ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a2cca926c0 con 0x55a2cca8fab0 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.946+0000 7f8184e0ab80 10 client.0 _ll_register_callbacks cb 0x55a2cc9706d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.946+0000 7f8173400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 0x55a2cca8f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f8172a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca84c10 0x55a2cca8c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f8173400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 0x55a2cca8f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:60254/0 (socket says 172.21.15.6:60254) 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f8173400700 1 -- 172.21.15.6:0/4255048507 learned_addr learned my addr 172.21.15.6:0/4255048507 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f8173e00700 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca8fab0 0x55a2cca8ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f8173400700 1 -- 172.21.15.6:0/4255048507 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca8fab0 msgr2=0x55a2cca8ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f8173400700 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca8fab0 0x55a2cca8ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f8173400700 1 -- 172.21.15.6:0/4255048507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca84c10 msgr2=0x55a2cca8c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f8173400700 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca84c10 0x55a2cca8c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f8173400700 1 -- 172.21.15.6:0/4255048507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2cc925880 con 0x55a2cca842f0 2024-04-26T01:44:48.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f8173e00700 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca8fab0 0x55a2cca8ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:48.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f8173400700 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 0x55a2cca8f220 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f816400c5b0 tx=0x7f816409ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b29666bc160eaab8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:48.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f815d600700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:44:48.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f815d600700 1 -- 172.21.15.6:0/4255048507 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81640a52d0 con 0x55a2cca842f0 2024-04-26T01:44:48.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.947+0000 7f815d600700 1 -- 172.21.15.6:0/4255048507 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f8164010720 con 0x55a2cca842f0 2024-04-26T01:44:48.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.948+0000 7f8184e0ab80 1 -- 172.21.15.6:0/4255048507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f816000be60 con 0x55a2cca842f0 2024-04-26T01:44:48.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.948+0000 7f815d600700 1 -- 172.21.15.6:0/4255048507 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f816409c8f0 con 0x55a2cca842f0 2024-04-26T01:44:48.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.948+0000 7f8184e0ab80 10 client.24860 fetch_fsmap learned FSMap version 25 2024-04-26T01:44:48.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.948+0000 7f8184e0ab80 1 -- 172.21.15.6:0/4255048507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a2cc96eaa0 con 0x55a2cca842f0 2024-04-26T01:44:48.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.948+0000 7f815d600700 1 -- 172.21.15.6:0/4255048507 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f816000be60 con 0x55a2cca842f0 2024-04-26T01:44:48.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.948+0000 7f815d600700 1 -- 172.21.15.6:0/4255048507 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f81640987d0 con 0x55a2cca842f0 2024-04-26T01:44:48.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.948+0000 7f8184e0ab80 10 client.24860 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:44:48.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.948+0000 7f8184e0ab80 10 client.24860 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T01:44:48.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.949+0000 7f8184e0ab80 20 client.24860 populate_metadata read hostname 'smithi006' 2024-04-26T01:44:48.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.949+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55a2cca9f340 0x55a2ccaa07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.949+0000 7f8184e0ab80 4 client.24860 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:48.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.949+0000 7f8184e0ab80 1 -- 172.21.15.6:0/4255048507 --> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2cc9e5630 con 0x55a2cca9f340 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.949+0000 7f8172a00700 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55a2cca9f340 0x55a2ccaa07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.950+0000 7f8172a00700 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55a2cca9f340 0x55a2ccaa07b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2392589b2ba59597 server_cookie=407bfe1916df86f8 in_seq=0 out_seq=0 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.950+0000 7f815d600700 10 client.24860 ms_handle_connect on v2:172.21.15.188:6836/489767141 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f815d600700 1 -- 172.21.15.6:0/4255048507 <== mds.2 v2:172.21.15.188:6836/489767141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a2cc9e5630 con 0x55a2cca9f340 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f815d600700 10 client.24860 handle_command_reply: tid=0 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f8184e0ab80 1 client.24860 shutdown 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f8184e0ab80 1 -- 172.21.15.6:0/4255048507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 msgr2=0x55a2cca8f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 0x55a2cca8f220 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f816400c5b0 tx=0x7f816409ce30 comp rx=0 tx=0).stop 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f8184e0ab80 1 -- 172.21.15.6:0/4255048507 shutdown_connections 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca8fab0 0x55a2cca8ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 0x55a2cca8f220 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55a2cca9f340 0x55a2ccaa07b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/4255048507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca84c10 0x55a2cca8c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f8184e0ab80 1 -- 172.21.15.6:0/4255048507 >> 172.21.15.6:0/4255048507 conn(0x55a2cc9b3550 msgr2=0x55a2cca86c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:48.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f815d600700 0 client.24860 ms_handle_reset on v2:172.21.15.188:6836/489767141 2024-04-26T01:44:48.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f8184e0ab80 1 -- 172.21.15.6:0/4255048507 shutdown_connections 2024-04-26T01:44:48.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f8184e0ab80 1 -- 172.21.15.6:0/4255048507 wait complete. 2024-04-26T01:44:48.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.951+0000 7f8184e0ab80 20 client.24860 trim_cache size 0 max 16384 2024-04-26T01:44:48.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.975+0000 7f8184e0ab80 1 Processor -- start 2024-04-26T01:44:48.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.975+0000 7f8184e0ab80 1 -- start start 2024-04-26T01:44:48.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.975+0000 7f8184e0ab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca842f0 0x55a2cca80920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.975+0000 7f8184e0ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca84c10 0x55a2ccaa9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.975+0000 7f8184e0ab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca8fab0 0x55a2ccaaa150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.975+0000 7f8184e0ab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a2cca9eaf0 con 0x55a2cca842f0 2024-04-26T01:44:48.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.975+0000 7f8184e0ab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a2cca9f1c0 con 0x55a2cca8fab0 2024-04-26T01:44:48.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.975+0000 7f8184e0ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a2cca9f890 con 0x55a2cca84c10 2024-04-26T01:44:48.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.975+0000 7f8173400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca842f0 0x55a2cca80920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.976+0000 7f8173400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca842f0 0x55a2cca80920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39250/0 (socket says 172.21.15.6:39250) 2024-04-26T01:44:48.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.976+0000 7f8173400700 1 -- 172.21.15.6:0/42877154 learned_addr learned my addr 172.21.15.6:0/42877154 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:48.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.976+0000 7f8172a00700 1 --2- 172.21.15.6:0/42877154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca84c10 0x55a2ccaa9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.976+0000 7f8173e00700 1 --2- 172.21.15.6:0/42877154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca8fab0 0x55a2ccaaa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.976+0000 7f8173400700 1 -- 172.21.15.6:0/42877154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca84c10 msgr2=0x55a2ccaa9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.976+0000 7f8173400700 1 --2- 172.21.15.6:0/42877154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca84c10 0x55a2ccaa9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.976+0000 7f8173400700 1 -- 172.21.15.6:0/42877154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca8fab0 msgr2=0x55a2ccaaa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.976+0000 7f8173400700 1 --2- 172.21.15.6:0/42877154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca8fab0 0x55a2ccaaa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.976+0000 7f8173400700 1 -- 172.21.15.6:0/42877154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2cca9cd30 con 0x55a2cca842f0 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.976+0000 7f8173400700 1 --2- 172.21.15.6:0/42877154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca842f0 0x55a2cca80920 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8164009860 tx=0x7f816409ae00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd34e89c047bf53 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.977+0000 7f815fe00700 1 -- 172.21.15.6:0/42877154 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816409c250 con 0x55a2cca842f0 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.977+0000 7f815fe00700 1 -- 172.21.15.6:0/42877154 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81640a5560 con 0x55a2cca842f0 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.977+0000 7f815fe00700 1 -- 172.21.15.6:0/42877154 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816409bfa0 con 0x55a2cca842f0 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.977+0000 7f8184e0ab80 1 -- 172.21.15.6:0/42877154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca842f0 msgr2=0x55a2cca80920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.977+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/42877154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca842f0 0x55a2cca80920 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8164009860 tx=0x7f816409ae00 comp rx=0 tx=0).stop 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.978+0000 7f8184e0ab80 1 -- 172.21.15.6:0/42877154 shutdown_connections 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.978+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/42877154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca8fab0 0x55a2ccaaa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.978+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/42877154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca842f0 0x55a2cca80920 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.978+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/42877154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca84c10 0x55a2ccaa9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.978+0000 7f8184e0ab80 1 -- 172.21.15.6:0/42877154 >> 172.21.15.6:0/42877154 conn(0x55a2cc9b3550 msgr2=0x55a2cca80210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:48.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.978+0000 7f8184e0ab80 1 -- 172.21.15.6:0/42877154 shutdown_connections 2024-04-26T01:44:48.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.978+0000 7f8184e0ab80 1 -- 172.21.15.6:0/42877154 wait complete. 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.978+0000 7f8184e0ab80 1 -- start start 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.979+0000 7f8184e0ab80 1 Processor -- start 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.979+0000 7f8184e0ab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 0x55a2ccaa3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.979+0000 7f8184e0ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca84c10 0x55a2ccaa3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8184e0ab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca8fab0 0x55a2ccaa40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8184e0ab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a2ccaa0630 con 0x55a2cca8fab0 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8184e0ab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a2cca9ff60 con 0x55a2cca842f0 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8184e0ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a2ccaa0d00 con 0x55a2cca84c10 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8184e0ab80 10 client.0 _ll_register_callbacks cb 0x55a2cc9706d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8173e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca8fab0 0x55a2ccaa40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8173400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 0x55a2ccaa3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8173e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca8fab0 0x55a2ccaa40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39256/0 (socket says 172.21.15.6:39256) 2024-04-26T01:44:48.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8173e00700 1 -- 172.21.15.6:0/2688145757 learned_addr learned my addr 172.21.15.6:0/2688145757 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8172a00700 1 --2- 172.21.15.6:0/2688145757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca84c10 0x55a2ccaa3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8173400700 1 -- 172.21.15.6:0/2688145757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca84c10 msgr2=0x55a2ccaa3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8173400700 1 --2- 172.21.15.6:0/2688145757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca84c10 0x55a2ccaa3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8173400700 1 -- 172.21.15.6:0/2688145757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca8fab0 msgr2=0x55a2ccaa40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8173400700 1 --2- 172.21.15.6:0/2688145757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca8fab0 0x55a2ccaa40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.980+0000 7f8173400700 1 -- 172.21.15.6:0/2688145757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2ccaa7ed0 con 0x55a2cca842f0 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.981+0000 7f8173400700 1 --2- 172.21.15.6:0/2688145757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 0x55a2ccaa3640 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f816409aba0 tx=0x7f816409ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1d98a5fbab3e35f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.981+0000 7f815d600700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.981+0000 7f815d600700 1 -- 172.21.15.6:0/2688145757 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816409c250 con 0x55a2cca842f0 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.981+0000 7f815d600700 1 -- 172.21.15.6:0/2688145757 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f816409bd40 con 0x55a2cca842f0 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.981+0000 7f8184e0ab80 1 -- 172.21.15.6:0/2688145757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81640a5560 con 0x55a2cca842f0 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.981+0000 7f815d600700 1 -- 172.21.15.6:0/2688145757 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f816409bf80 con 0x55a2cca842f0 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.981+0000 7f8184e0ab80 10 client.24866 fetch_fsmap learned FSMap version 25 2024-04-26T01:44:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.981+0000 7f8184e0ab80 1 -- 172.21.15.6:0/2688145757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a2ccb06830 con 0x55a2cca842f0 2024-04-26T01:44:48.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.981+0000 7f815d600700 1 -- 172.21.15.6:0/2688145757 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81640a5560 con 0x55a2cca842f0 2024-04-26T01:44:48.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.981+0000 7f815d600700 1 -- 172.21.15.6:0/2688145757 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f8164083020 con 0x55a2cca842f0 2024-04-26T01:44:48.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.982+0000 7f8184e0ab80 10 client.24866 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:44:48.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.982+0000 7f8184e0ab80 10 client.24866 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.982+0000 7f8184e0ab80 20 client.24866 populate_metadata read hostname 'smithi006' 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.982+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/2688145757 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55a2ccb0af20 0x55a2ccb09c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.982+0000 7f8184e0ab80 4 client.24866 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.982+0000 7f8184e0ab80 1 -- 172.21.15.6:0/2688145757 --> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] -- command(tid 0: {"prefix": "status"}) -- 0x55a2cc9df6d0 con 0x55a2ccb0af20 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.982+0000 7f8172a00700 1 --2- 172.21.15.6:0/2688145757 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55a2ccb0af20 0x55a2ccb09c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.982+0000 7f8172a00700 1 --2- 172.21.15.6:0/2688145757 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55a2ccb0af20 0x55a2ccb09c30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8a8d8a167a95c7ab server_cookie=eddb0fc95a6a72dd in_seq=0 out_seq=0 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.982+0000 7f815d600700 10 client.24866 ms_handle_connect on v2:172.21.15.188:6836/489767141 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f815d600700 1 -- 172.21.15.6:0/2688145757 <== mds.2 v2:172.21.15.188:6836/489767141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55a2cc9df6d0 con 0x55a2ccb0af20 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f815d600700 10 client.24866 handle_command_reply: tid=0 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f8184e0ab80 1 client.24866 shutdown 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f8184e0ab80 1 -- 172.21.15.6:0/2688145757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 msgr2=0x55a2ccaa3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/2688145757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 0x55a2ccaa3640 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f816409aba0 tx=0x7f816409ce30 comp rx=0 tx=0).stop 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f8184e0ab80 1 -- 172.21.15.6:0/2688145757 shutdown_connections 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/2688145757 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55a2ccb0af20 0x55a2ccb09c30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/2688145757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a2cca8fab0 0x55a2ccaa40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/2688145757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cca842f0 0x55a2ccaa3640 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f8184e0ab80 1 --2- 172.21.15.6:0/2688145757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a2cca84c10 0x55a2ccaa3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f815d600700 0 client.24866 ms_handle_reset on v2:172.21.15.188:6836/489767141 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f8184e0ab80 1 -- 172.21.15.6:0/2688145757 >> 172.21.15.6:0/2688145757 conn(0x55a2cc9b3550 msgr2=0x55a2cca8c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f8184e0ab80 1 -- 172.21.15.6:0/2688145757 shutdown_connections 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f8184e0ab80 1 -- 172.21.15.6:0/2688145757 wait complete. 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.983+0000 7f8184e0ab80 20 client.24866 trim_cache size 0 max 16384 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 2, 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 24505, 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-26T01:44:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-26T01:44:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-26T01:44:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 144.51328237999999, 2024-04-26T01:44:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 25, 2024-04-26T01:44:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 95, 2024-04-26T01:44:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T01:44:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 200.01447668 2024-04-26T01:44:48.985 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:44:48.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.984+0000 7f817a400700 1 -- 172.21.15.6:0/3961138010 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f81580a3cc0 msgr2=0x7f81580a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.984+0000 7f817a400700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f81580a3cc0 0x7f81580a6180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f81600098e0 tx=0x7f8160005960 comp rx=0 tx=0).stop 2024-04-26T01:44:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.984+0000 7f817a400700 1 -- 172.21.15.6:0/3961138010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8174076f70 msgr2=0x7f81741a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.984+0000 7f817a400700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8174076f70 0x7f81741a5460 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f816400e9a0 tx=0x7f816400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.984+0000 7f817a400700 1 -- 172.21.15.6:0/3961138010 shutdown_connections 2024-04-26T01:44:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.984+0000 7f817a400700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81741138c0 0x7f817419f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.984+0000 7f817a400700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f81580a3cc0 0x7f81580a6180 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.984+0000 7f817a400700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8174076f70 0x7f81741a5460 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.984+0000 7f817a400700 1 --2- 172.21.15.6:0/3961138010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8174077890 0x7f81741a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.984+0000 7f817a400700 1 -- 172.21.15.6:0/3961138010 >> 172.21.15.6:0/3961138010 conn(0x7f81741006f0 msgr2=0x7f8174114530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.984+0000 7f817a400700 1 -- 172.21.15.6:0/3961138010 shutdown_connections 2024-04-26T01:44:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:48.985+0000 7f817a400700 1 -- 172.21.15.6:0/3961138010 wait complete. 2024-04-26T01:44:49.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:48 smithi006 ceph-mon[28419]: pgmap v415: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:49.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:48 smithi188 ceph-mon[32751]: pgmap v415: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:49.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:48 smithi029 ceph-mon[32775]: pgmap v415: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:49.971 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T01:44:50.453 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:51.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:50 smithi006 ceph-mon[28419]: pgmap v416: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:51.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:50 smithi188 ceph-mon[32751]: pgmap v416: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:51.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:50 smithi029 ceph-mon[32775]: pgmap v416: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:52.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.707+0000 7f7028c00700 1 -- 172.21.15.6:0/2880763217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70241092d0 msgr2=0x7f70241096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:52.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.707+0000 7f7028c00700 1 --2- 172.21.15.6:0/2880763217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70241092d0 0x7f70241096b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f70140099c0 tx=0x7f7014009cd0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.707+0000 7f7028c00700 1 -- 172.21.15.6:0/2880763217 shutdown_connections 2024-04-26T01:44:52.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.707+0000 7f7028c00700 1 --2- 172.21.15.6:0/2880763217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f702410e3b0 0x7f7024115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.707+0000 7f7028c00700 1 --2- 172.21.15.6:0/2880763217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70241092d0 0x7f70241096b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.707+0000 7f7028c00700 1 --2- 172.21.15.6:0/2880763217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7024109c80 0x7f702410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.707+0000 7f7028c00700 1 -- 172.21.15.6:0/2880763217 >> 172.21.15.6:0/2880763217 conn(0x7f7024076210 msgr2=0x7f7024107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:52.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.707+0000 7f7028c00700 1 -- 172.21.15.6:0/2880763217 shutdown_connections 2024-04-26T01:44:52.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.707+0000 7f7028c00700 1 -- 172.21.15.6:0/2880763217 wait complete. 2024-04-26T01:44:52.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.708+0000 7f7028c00700 1 Processor -- start 2024-04-26T01:44:52.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.708+0000 7f7028c00700 1 -- start start 2024-04-26T01:44:52.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7028c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70241092d0 0x7f702419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7028c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7024109c80 0x7f702419fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7028c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f702410e3b0 0x7f70241a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7028c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7024119270 con 0x7f702410e3b0 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7028c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70241190f0 con 0x7f70241092d0 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7028c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f70241193f0 con 0x7f7024109c80 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7021400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7024109c80 0x7f702419fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7022800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f702410e3b0 0x7f70241a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7021e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70241092d0 0x7f702419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7022800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f702410e3b0 0x7f70241a3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39272/0 (socket says 172.21.15.6:39272) 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7022800700 1 -- 172.21.15.6:0/890788223 learned_addr learned my addr 172.21.15.6:0/890788223 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7021400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7024109c80 0x7f702419fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44942/0 (socket says 172.21.15.6:44942) 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7022800700 1 -- 172.21.15.6:0/890788223 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7024109c80 msgr2=0x7f702419fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7022800700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7024109c80 0x7f702419fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7022800700 1 -- 172.21.15.6:0/890788223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70241092d0 msgr2=0x7f702419f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:52.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7022800700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70241092d0 0x7f702419f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7022800700 1 -- 172.21.15.6:0/890788223 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70241a4700 con 0x7f702410e3b0 2024-04-26T01:44:52.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.709+0000 7f7021400700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7024109c80 0x7f702419fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:52.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.710+0000 7f7022800700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f702410e3b0 0x7f70241a3fc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f701c00ef30 tx=0x7f701c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:52.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.710+0000 7f701aa00700 1 -- 172.21.15.6:0/890788223 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c0092e0 con 0x7f702410e3b0 2024-04-26T01:44:52.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.710+0000 7f701aa00700 1 -- 172.21.15.6:0/890788223 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f701c004bb0 con 0x7f702410e3b0 2024-04-26T01:44:52.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.710+0000 7f7028c00700 1 -- 172.21.15.6:0/890788223 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70241a4990 con 0x7f702410e3b0 2024-04-26T01:44:52.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.710+0000 7f701aa00700 1 -- 172.21.15.6:0/890788223 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c004280 con 0x7f702410e3b0 2024-04-26T01:44:52.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.710+0000 7f7028c00700 1 -- 172.21.15.6:0/890788223 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70241ac270 con 0x7f702410e3b0 2024-04-26T01:44:52.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.714+0000 7f701aa00700 1 -- 172.21.15.6:0/890788223 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f701c01b030 con 0x7f702410e3b0 2024-04-26T01:44:52.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.715+0000 7f703363db80 1 Processor -- start 2024-04-26T01:44:52.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.715+0000 7f701aa00700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f70080a3cc0 0x7f70080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.715+0000 7f703363db80 1 -- start start 2024-04-26T01:44:52.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.715+0000 7f701aa00700 1 -- 172.21.15.6:0/890788223 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f701c0cfec0 con 0x7f702410e3b0 2024-04-26T01:44:52.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.715+0000 7f7021400700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f70080a3cc0 0x7f70080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.715+0000 7f703363db80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d208102f0 0x564d208106d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.715+0000 7f703363db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d20810c10 0x564d2081b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.715+0000 7f703363db80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d2081bab0 0x564d2081dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f703363db80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564d20763e40 con 0x564d208102f0 2024-04-26T01:44:52.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f703363db80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564d2070a6e0 con 0x564d2081bab0 2024-04-26T01:44:52.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f703363db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564d2075fe00 con 0x564d20810c10 2024-04-26T01:44:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f7021e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d208102f0 0x564d208106d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f7021e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d208102f0 0x564d208106d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39288/0 (socket says 172.21.15.6:39288) 2024-04-26T01:44:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f7021e00700 1 -- 172.21.15.6:0/4076216025 learned_addr learned my addr 172.21.15.6:0/4076216025 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f7021400700 1 --2- 172.21.15.6:0/4076216025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d20810c10 0x564d2081b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f7022800700 1 --2- 172.21.15.6:0/4076216025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d2081bab0 0x564d2081dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f7021e00700 1 -- 172.21.15.6:0/4076216025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d20810c10 msgr2=0x564d2081b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f7021e00700 1 --2- 172.21.15.6:0/4076216025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d20810c10 0x564d2081b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f7021e00700 1 -- 172.21.15.6:0/4076216025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d2081bab0 msgr2=0x564d2081dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f7021e00700 1 --2- 172.21.15.6:0/4076216025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d2081bab0 0x564d2081dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f7021e00700 1 -- 172.21.15.6:0/4076216025 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d20700630 con 0x564d208102f0 2024-04-26T01:44:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.716+0000 7f7021400700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f70080a3cc0 0x7f70080a6180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f70240770a0 tx=0x7f7010011830 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.717+0000 7f7021e00700 1 --2- 172.21.15.6:0/4076216025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d208102f0 0x564d208106d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7014006010 tx=0x7f7014005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d061b2b23b3c1b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:52.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.717+0000 7f700f400700 1 -- 172.21.15.6:0/4076216025 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7014027070 con 0x564d208102f0 2024-04-26T01:44:52.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.717+0000 7f700f400700 1 -- 172.21.15.6:0/4076216025 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7014009590 con 0x564d208102f0 2024-04-26T01:44:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.717+0000 7f700f400700 1 -- 172.21.15.6:0/4076216025 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701400b9f0 con 0x564d208102f0 2024-04-26T01:44:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.718+0000 7f703363db80 1 -- 172.21.15.6:0/4076216025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d208102f0 msgr2=0x564d208106d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.718+0000 7f703363db80 1 --2- 172.21.15.6:0/4076216025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d208102f0 0x564d208106d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7014006010 tx=0x7f7014005c80 comp rx=0 tx=0).stop 2024-04-26T01:44:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.718+0000 7f703363db80 1 -- 172.21.15.6:0/4076216025 shutdown_connections 2024-04-26T01:44:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.718+0000 7f703363db80 1 --2- 172.21.15.6:0/4076216025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d2081bab0 0x564d2081dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.718+0000 7f703363db80 1 --2- 172.21.15.6:0/4076216025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d208102f0 0x564d208106d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.718+0000 7f703363db80 1 --2- 172.21.15.6:0/4076216025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d20810c10 0x564d2081b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.718+0000 7f703363db80 1 -- 172.21.15.6:0/4076216025 >> 172.21.15.6:0/4076216025 conn(0x564d2073f550 msgr2=0x564d2080c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.718+0000 7f703363db80 1 -- 172.21.15.6:0/4076216025 shutdown_connections 2024-04-26T01:44:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.718+0000 7f703363db80 1 -- 172.21.15.6:0/4076216025 wait complete. 2024-04-26T01:44:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.718+0000 7f703363db80 1 -- start start 2024-04-26T01:44:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.719+0000 7f703363db80 1 Processor -- start 2024-04-26T01:44:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.719+0000 7f703363db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d2081b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.719+0000 7f703363db80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d20810c10 0x564d20818560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.719+0000 7f703363db80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d2081bab0 0x564d20818aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.719+0000 7f703363db80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564d2081e540 con 0x564d2081bab0 2024-04-26T01:44:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.719+0000 7f703363db80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564d2071d810 con 0x564d20810c10 2024-04-26T01:44:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.719+0000 7f703363db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564d2081e6c0 con 0x564d208102f0 2024-04-26T01:44:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.719+0000 7f703363db80 10 client.0 _ll_register_callbacks cb 0x564d206fc6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.719+0000 7f7021e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d2081b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.720+0000 7f7021e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d2081b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44964/0 (socket says 172.21.15.6:44964) 2024-04-26T01:44:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.720+0000 7f7021e00700 1 -- 172.21.15.6:0/3998271048 learned_addr learned my addr 172.21.15.6:0/3998271048 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.720+0000 7f7022800700 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d2081bab0 0x564d20818aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.720+0000 7f7021400700 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d20810c10 0x564d20818560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.720+0000 7f7021e00700 1 -- 172.21.15.6:0/3998271048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d20810c10 msgr2=0x564d20818560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.720+0000 7f7021e00700 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d20810c10 0x564d20818560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.720+0000 7f7021e00700 1 -- 172.21.15.6:0/3998271048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d2081bab0 msgr2=0x564d20818aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.720+0000 7f7021e00700 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d2081bab0 0x564d20818aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.720+0000 7f7021e00700 1 -- 172.21.15.6:0/3998271048 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d206b1880 con 0x564d208102f0 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.720+0000 7f7022800700 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d2081bab0 0x564d20818aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.720+0000 7f7021e00700 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d2081b260 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7014003d40 tx=0x7f7014004eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59d3a1de4303088e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.720+0000 7f7003e00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.721+0000 7f7003e00700 1 -- 172.21.15.6:0/3998271048 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7014027040 con 0x564d208102f0 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.721+0000 7f7003e00700 1 -- 172.21.15.6:0/3998271048 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f70140050d0 con 0x564d208102f0 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.721+0000 7f703363db80 1 -- 172.21.15.6:0/3998271048 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7014009590 con 0x564d208102f0 2024-04-26T01:44:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.721+0000 7f7003e00700 1 -- 172.21.15.6:0/3998271048 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f70140052d0 con 0x564d208102f0 2024-04-26T01:44:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.721+0000 7f703363db80 10 client.24916 fetch_fsmap learned FSMap version 25 2024-04-26T01:44:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.721+0000 7f703363db80 1 -- 172.21.15.6:0/3998271048 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564d206faaa0 con 0x564d208102f0 2024-04-26T01:44:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.721+0000 7f7003e00700 1 -- 172.21.15.6:0/3998271048 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7014009590 con 0x564d208102f0 2024-04-26T01:44:52.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.722+0000 7f7003e00700 1 -- 172.21.15.6:0/3998271048 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f701402a070 con 0x564d208102f0 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.722+0000 7f703363db80 10 client.24916 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.722+0000 7f703363db80 10 client.24916 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.722+0000 7f703363db80 20 client.24916 populate_metadata read hostname 'smithi006' 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.722+0000 7f703363db80 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x564d20829fa0 0x564d2082c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.722+0000 7f703363db80 4 client.24916 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.722+0000 7f703363db80 1 -- 172.21.15.6:0/3998271048 --> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d20771630 con 0x564d20829fa0 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.722+0000 7f7021400700 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x564d20829fa0 0x564d2082c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.723+0000 7f7021400700 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x564d20829fa0 0x564d2082c0e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=482d2cbd7af537e9 server_cookie=10089bf90a4dd313 in_seq=0 out_seq=0 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.723+0000 7f7003e00700 10 client.24916 ms_handle_connect on v2:172.21.15.29:6838/2325582359 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f7003e00700 1 -- 172.21.15.6:0/3998271048 <== mds.1 v2:172.21.15.29:6838/2325582359 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d20771630 con 0x564d20829fa0 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f7003e00700 10 client.24916 handle_command_reply: tid=0 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f703363db80 1 client.24916 shutdown 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f703363db80 1 -- 172.21.15.6:0/3998271048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 msgr2=0x564d2081b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:52.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f703363db80 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d2081b260 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7014003d40 tx=0x7f7014004eb0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f703363db80 1 -- 172.21.15.6:0/3998271048 shutdown_connections 2024-04-26T01:44:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f703363db80 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d2081bab0 0x564d20818aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f703363db80 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d2081b260 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f703363db80 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x564d20829fa0 0x564d2082c0e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f703363db80 1 --2- 172.21.15.6:0/3998271048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d20810c10 0x564d20818560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f7003e00700 0 client.24916 ms_handle_reset on v2:172.21.15.29:6838/2325582359 2024-04-26T01:44:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f703363db80 1 -- 172.21.15.6:0/3998271048 >> 172.21.15.6:0/3998271048 conn(0x564d2073f550 msgr2=0x564d20812c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.724+0000 7f703363db80 1 -- 172.21.15.6:0/3998271048 shutdown_connections 2024-04-26T01:44:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.725+0000 7f703363db80 1 -- 172.21.15.6:0/3998271048 wait complete. 2024-04-26T01:44:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.725+0000 7f703363db80 20 client.24916 trim_cache size 0 max 16384 2024-04-26T01:44:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.746+0000 7f703363db80 1 Processor -- start 2024-04-26T01:44:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.746+0000 7f703363db80 1 -- start start 2024-04-26T01:44:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.746+0000 7f703363db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d20818a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.746+0000 7f703363db80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d20810c10 0x564d208157e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f703363db80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d2081bab0 0x564d20815d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f703363db80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564d2082a420 con 0x564d20810c10 2024-04-26T01:44:52.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f703363db80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564d2082aaf0 con 0x564d2081bab0 2024-04-26T01:44:52.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f703363db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564d2082b1c0 con 0x564d208102f0 2024-04-26T01:44:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f7021400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d20810c10 0x564d208157e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f7021400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d20810c10 0x564d208157e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39306/0 (socket says 172.21.15.6:39306) 2024-04-26T01:44:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f7021400700 1 -- 172.21.15.6:0/815720155 learned_addr learned my addr 172.21.15.6:0/815720155 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f7022800700 1 --2- 172.21.15.6:0/815720155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d2081bab0 0x564d20815d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f7021e00700 1 --2- 172.21.15.6:0/815720155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d20818a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f7021400700 1 -- 172.21.15.6:0/815720155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 msgr2=0x564d20818a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f7021400700 1 --2- 172.21.15.6:0/815720155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d20818a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f7021400700 1 -- 172.21.15.6:0/815720155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d2081bab0 msgr2=0x564d20815d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f7021400700 1 --2- 172.21.15.6:0/815720155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d2081bab0 0x564d20815d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f7021400700 1 -- 172.21.15.6:0/815720155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d20811b60 con 0x564d20810c10 2024-04-26T01:44:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f7021e00700 1 --2- 172.21.15.6:0/815720155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d20818a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.747+0000 7f7022800700 1 --2- 172.21.15.6:0/815720155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d2081bab0 0x564d20815d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.748+0000 7f7021400700 1 --2- 172.21.15.6:0/815720155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d20810c10 0x564d208157e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x564d20819b30 tx=0x7f701001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b0c7e083625675 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.748+0000 7f700ea00700 1 -- 172.21.15.6:0/815720155 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7010017040 con 0x564d20810c10 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.748+0000 7f700ea00700 1 -- 172.21.15.6:0/815720155 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7010019050 con 0x564d20810c10 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.748+0000 7f700ea00700 1 -- 172.21.15.6:0/815720155 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701001d860 con 0x564d20810c10 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.749+0000 7f703363db80 1 -- 172.21.15.6:0/815720155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d20810c10 msgr2=0x564d208157e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.749+0000 7f703363db80 1 --2- 172.21.15.6:0/815720155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d20810c10 0x564d208157e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x564d20819b30 tx=0x7f701001c000 comp rx=0 tx=0).stop 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.749+0000 7f703363db80 1 -- 172.21.15.6:0/815720155 shutdown_connections 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.749+0000 7f703363db80 1 --2- 172.21.15.6:0/815720155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d2081bab0 0x564d20815d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.749+0000 7f703363db80 1 --2- 172.21.15.6:0/815720155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d20818a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.749+0000 7f703363db80 1 --2- 172.21.15.6:0/815720155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d20810c10 0x564d208157e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.749+0000 7f703363db80 1 -- 172.21.15.6:0/815720155 >> 172.21.15.6:0/815720155 conn(0x564d2073f550 msgr2=0x564d20818340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:52.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.749+0000 7f703363db80 1 -- 172.21.15.6:0/815720155 shutdown_connections 2024-04-26T01:44:52.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.749+0000 7f703363db80 1 -- 172.21.15.6:0/815720155 wait complete. 2024-04-26T01:44:52.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.749+0000 7f703363db80 1 -- start start 2024-04-26T01:44:52.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.749+0000 7f703363db80 1 Processor -- start 2024-04-26T01:44:52.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f703363db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d2081cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f703363db80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d20810c10 0x564d2081d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f703363db80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d2081bab0 0x564d20831b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f703363db80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564d2082bf60 con 0x564d2081bab0 2024-04-26T01:44:52.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f703363db80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564d2082b890 con 0x564d20810c10 2024-04-26T01:44:52.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f703363db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564d2082c630 con 0x564d208102f0 2024-04-26T01:44:52.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f703363db80 10 client.0 _ll_register_callbacks cb 0x564d206fc6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:52.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f7021e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d2081cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f7021400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d20810c10 0x564d2081d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f7021e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d2081cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44990/0 (socket says 172.21.15.6:44990) 2024-04-26T01:44:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f7021e00700 1 -- 172.21.15.6:0/2996111328 learned_addr learned my addr 172.21.15.6:0/2996111328 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f7022800700 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d2081bab0 0x564d20831b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f7021400700 1 -- 172.21.15.6:0/2996111328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 msgr2=0x564d2081cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f7021400700 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d2081cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.750+0000 7f7021400700 1 -- 172.21.15.6:0/2996111328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d2081bab0 msgr2=0x564d20831b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.751+0000 7f7021400700 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d2081bab0 0x564d20831b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.751+0000 7f7021400700 1 -- 172.21.15.6:0/2996111328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d20832080 con 0x564d20810c10 2024-04-26T01:44:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.751+0000 7f7022800700 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d2081bab0 0x564d20831b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.751+0000 7f7021e00700 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d2081cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.751+0000 7f7021400700 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d20810c10 0x564d2081d490 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7010005920 tx=0x7f7010003e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ebc166f61da4e61 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.751+0000 7f7003e00700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.751+0000 7f7003e00700 1 -- 172.21.15.6:0/2996111328 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701003b060 con 0x564d20810c10 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.751+0000 7f703363db80 1 -- 172.21.15.6:0/2996111328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7010019050 con 0x564d20810c10 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.751+0000 7f7003e00700 1 -- 172.21.15.6:0/2996111328 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f70100053c0 con 0x564d20810c10 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.752+0000 7f7003e00700 1 -- 172.21.15.6:0/2996111328 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7010017040 con 0x564d20810c10 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.752+0000 7f703363db80 10 client.24884 fetch_fsmap learned FSMap version 25 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.752+0000 7f703363db80 1 -- 172.21.15.6:0/2996111328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564d20892520 con 0x564d20810c10 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.752+0000 7f7003e00700 1 -- 172.21.15.6:0/2996111328 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7010019050 con 0x564d20810c10 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.752+0000 7f7003e00700 1 -- 172.21.15.6:0/2996111328 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f7010004730 con 0x564d20810c10 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.752+0000 7f703363db80 10 client.24884 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.752+0000 7f703363db80 10 client.24884 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.752+0000 7f703363db80 20 client.24884 populate_metadata read hostname 'smithi006' 2024-04-26T01:44:52.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.752+0000 7f703363db80 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x564d20896d50 0x564d20895a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.752+0000 7f703363db80 4 client.24884 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.752+0000 7f703363db80 1 -- 172.21.15.6:0/2996111328 --> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] -- command(tid 0: {"prefix": "status"}) -- 0x564d2076b6d0 con 0x564d20896d50 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.753+0000 7f7021e00700 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x564d20896d50 0x564d20895a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.753+0000 7f7021e00700 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x564d20896d50 0x564d20895a60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8f51629792fc64d8 server_cookie=3b8fcf6cceb77ce7 in_seq=0 out_seq=0 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.753+0000 7f7003e00700 10 client.24884 ms_handle_connect on v2:172.21.15.29:6838/2325582359 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.753+0000 7f7003e00700 1 -- 172.21.15.6:0/2996111328 <== mds.1 v2:172.21.15.29:6838/2325582359 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564d2076b6d0 con 0x564d20896d50 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.753+0000 7f7003e00700 10 client.24884 handle_command_reply: tid=0 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.753+0000 7f703363db80 1 client.24884 shutdown 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.753+0000 7f703363db80 1 -- 172.21.15.6:0/2996111328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d20810c10 msgr2=0x564d2081d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.753+0000 7f703363db80 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d20810c10 0x564d2081d490 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7010005920 tx=0x7f7010003e50 comp rx=0 tx=0).stop 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.754+0000 7f703363db80 1 -- 172.21.15.6:0/2996111328 shutdown_connections 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.754+0000 7f703363db80 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564d2081bab0 0x564d20831b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.754+0000 7f703363db80 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564d208102f0 0x564d2081cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.754+0000 7f703363db80 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x564d20896d50 0x564d20895a60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.754+0000 7f703363db80 1 --2- 172.21.15.6:0/2996111328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d20810c10 0x564d2081d490 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.754+0000 7f7003e00700 0 client.24884 ms_handle_reset on v2:172.21.15.29:6838/2325582359 2024-04-26T01:44:52.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.754+0000 7f703363db80 1 -- 172.21.15.6:0/2996111328 >> 172.21.15.6:0/2996111328 conn(0x564d2073f550 msgr2=0x564d20818a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:52.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.754+0000 7f703363db80 1 -- 172.21.15.6:0/2996111328 shutdown_connections 2024-04-26T01:44:52.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.754+0000 7f703363db80 1 -- 172.21.15.6:0/2996111328 wait complete. 2024-04-26T01:44:52.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.754+0000 7f703363db80 20 client.24884 trim_cache size 0 max 16384 2024-04-26T01:44:52.756 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:44:52.756 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:44:52.756 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 1, 2024-04-26T01:44:52.757 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 24512, 2024-04-26T01:44:52.757 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-26T01:44:52.757 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-26T01:44:52.757 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-26T01:44:52.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 149.29456999199999, 2024-04-26T01:44:52.757 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 25, 2024-04-26T01:44:52.757 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 95, 2024-04-26T01:44:52.757 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T01:44:52.757 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 183.86405484299999 2024-04-26T01:44:52.757 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:44:52.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.755+0000 7f7028c00700 1 -- 172.21.15.6:0/890788223 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f70080a3cc0 msgr2=0x7f70080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:52.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.755+0000 7f7028c00700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f70080a3cc0 0x7f70080a6180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f70240770a0 tx=0x7f7010011830 comp rx=0 tx=0).stop 2024-04-26T01:44:52.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.755+0000 7f7028c00700 1 -- 172.21.15.6:0/890788223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f702410e3b0 msgr2=0x7f70241a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:52.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.755+0000 7f7028c00700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f702410e3b0 0x7f70241a3fc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f701c00ef30 tx=0x7f701c00c3a0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.755+0000 7f7028c00700 1 -- 172.21.15.6:0/890788223 shutdown_connections 2024-04-26T01:44:52.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.755+0000 7f7028c00700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f702410e3b0 0x7f70241a3fc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.755+0000 7f7028c00700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70241092d0 0x7f702419f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.755+0000 7f7028c00700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f70080a3cc0 0x7f70080a6180 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.755+0000 7f7028c00700 1 --2- 172.21.15.6:0/890788223 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7024109c80 0x7f702419fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:52.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.755+0000 7f7028c00700 1 -- 172.21.15.6:0/890788223 >> 172.21.15.6:0/890788223 conn(0x7f7024076210 msgr2=0x7f7024077e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.755+0000 7f7028c00700 1 -- 172.21.15.6:0/890788223 shutdown_connections 2024-04-26T01:44:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:52.755+0000 7f7028c00700 1 -- 172.21.15.6:0/890788223 wait complete. 2024-04-26T01:44:53.249 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:52 smithi006 ceph-mon[28419]: pgmap v417: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:53.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:52 smithi188 ceph-mon[32751]: pgmap v417: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:53.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:52 smithi029 ceph-mon[32775]: pgmap v417: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:53.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T01:44:54.239 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:55.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:54 smithi006 ceph-mon[28419]: pgmap v418: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:55.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:54 smithi188 ceph-mon[32751]: pgmap v418: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:55.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:54 smithi029 ceph-mon[32775]: pgmap v418: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:56.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.409+0000 7f0867400700 1 -- 172.21.15.6:0/1895955495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0860105890 msgr2=0x7f08601098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:56.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.409+0000 7f0867400700 1 --2- 172.21.15.6:0/1895955495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0860105890 0x7f08601098e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f084c0099c0 tx=0x7f084c009cd0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.409+0000 7f0867400700 1 -- 172.21.15.6:0/1895955495 shutdown_connections 2024-04-26T01:44:56.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.409+0000 7f0867400700 1 --2- 172.21.15.6:0/1895955495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0860109fc0 0x7f0860111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.409+0000 7f0867400700 1 --2- 172.21.15.6:0/1895955495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0860104ee0 0x7f08601052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.409+0000 7f0867400700 1 --2- 172.21.15.6:0/1895955495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0860105890 0x7f08601098e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.409+0000 7f0867400700 1 -- 172.21.15.6:0/1895955495 >> 172.21.15.6:0/1895955495 conn(0x7f0860100770 msgr2=0x7f0860102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:56.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.409+0000 7f0867400700 1 -- 172.21.15.6:0/1895955495 shutdown_connections 2024-04-26T01:44:56.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.410+0000 7f0867400700 1 -- 172.21.15.6:0/1895955495 wait complete. 2024-04-26T01:44:56.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.410+0000 7f0867400700 1 Processor -- start 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.411+0000 7f0867400700 1 -- start start 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.411+0000 7f0867400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0860104ee0 0x7f086019b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.411+0000 7f0867400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0860105890 0x7f086019b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.411+0000 7f0867400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0860109fc0 0x7f086019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.411+0000 7f0867400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0860114ea0 con 0x7f0860105890 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.411+0000 7f0867400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0860114d20 con 0x7f0860109fc0 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.411+0000 7f0867400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0860115020 con 0x7f0860104ee0 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.411+0000 7f085fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0860105890 0x7f086019b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.411+0000 7f0864a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0860104ee0 0x7f086019b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.412+0000 7f0864a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0860104ee0 0x7f086019b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44998/0 (socket says 172.21.15.6:44998) 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.412+0000 7f0864a00700 1 -- 172.21.15.6:0/2910399055 learned_addr learned my addr 172.21.15.6:0/2910399055 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.412+0000 7f0865400700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0860109fc0 0x7f086019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.412+0000 7f0864a00700 1 -- 172.21.15.6:0/2910399055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0860109fc0 msgr2=0x7f086019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.412+0000 7f0864a00700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0860109fc0 0x7f086019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.412+0000 7f0864a00700 1 -- 172.21.15.6:0/2910399055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0860105890 msgr2=0x7f086019b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:56.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.412+0000 7f0864a00700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0860105890 0x7f086019b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.412+0000 7f0864a00700 1 -- 172.21.15.6:0/2910399055 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08601a03d0 con 0x7f0860104ee0 2024-04-26T01:44:56.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.412+0000 7f085fe00700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0860105890 0x7f086019b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:56.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.412+0000 7f0864a00700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0860104ee0 0x7f086019b410 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f085400d980 tx=0x7f085400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:56.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.413+0000 7f085d600700 1 -- 172.21.15.6:0/2910399055 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854016070 con 0x7f0860104ee0 2024-04-26T01:44:56.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.413+0000 7f085d600700 1 -- 172.21.15.6:0/2910399055 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0854004830 con 0x7f0860104ee0 2024-04-26T01:44:56.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.413+0000 7f085d600700 1 -- 172.21.15.6:0/2910399055 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854005540 con 0x7f0860104ee0 2024-04-26T01:44:56.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.413+0000 7f0867400700 1 -- 172.21.15.6:0/2910399055 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08601a06c0 con 0x7f0860104ee0 2024-04-26T01:44:56.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.413+0000 7f0867400700 1 -- 172.21.15.6:0/2910399055 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08601a7f40 con 0x7f0860104ee0 2024-04-26T01:44:56.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.418+0000 7f085d600700 1 -- 172.21.15.6:0/2910399055 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f0854004220 con 0x7f0860104ee0 2024-04-26T01:44:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.419+0000 7f0871e2ab80 1 Processor -- start 2024-04-26T01:44:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.419+0000 7f0871e2ab80 1 -- start start 2024-04-26T01:44:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.419+0000 7f085d600700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f08480a3c40 0x7f08480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.419+0000 7f0871e2ab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc72f0 0x5572e2cc76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0871e2ab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc7c10 0x5572e2cd2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0871e2ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cd2ab0 0x5572e2cd4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0871e2ab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5572e2c1ae40 con 0x5572e2cc7c10 2024-04-26T01:44:56.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0871e2ab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5572e2bc16e0 con 0x5572e2cc72f0 2024-04-26T01:44:56.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0871e2ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5572e2c16e00 con 0x5572e2cd2ab0 2024-04-26T01:44:56.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f085d600700 1 -- 172.21.15.6:0/2910399055 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f08540d0240 con 0x7f0860104ee0 2024-04-26T01:44:56.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f085fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc7c10 0x5572e2cd2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f085fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc7c10 0x5572e2cd2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39342/0 (socket says 172.21.15.6:39342) 2024-04-26T01:44:56.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f085fe00700 1 -- 172.21.15.6:0/363004265 learned_addr learned my addr 172.21.15.6:0/363004265 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:56.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0865400700 1 --2- 172.21.15.6:0/363004265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cd2ab0 0x5572e2cd4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0864a00700 1 --2- 172.21.15.6:0/363004265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc72f0 0x5572e2cc76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0865400700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f08480a3c40 0x7f08480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0864a00700 1 -- 172.21.15.6:0/363004265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cd2ab0 msgr2=0x5572e2cd4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:56.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0864a00700 1 --2- 172.21.15.6:0/363004265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cd2ab0 0x5572e2cd4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0864a00700 1 -- 172.21.15.6:0/363004265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc7c10 msgr2=0x5572e2cd2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:56.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0864a00700 1 --2- 172.21.15.6:0/363004265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc7c10 0x5572e2cd2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.420+0000 7f0864a00700 1 -- 172.21.15.6:0/363004265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572e2bb7630 con 0x5572e2cc72f0 2024-04-26T01:44:56.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.421+0000 7f0864a00700 1 --2- 172.21.15.6:0/363004265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc72f0 0x5572e2cc76d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f08540cdca0 tx=0x7f085409bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faa8a7ccc2a50db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.421+0000 7f0865400700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f08480a3c40 0x7f08480a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x5572e2cc8d70 tx=0x7f085800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.421+0000 7f0852000700 1 -- 172.21.15.6:0/363004265 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085409c480 con 0x5572e2cc72f0 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.421+0000 7f0852000700 1 -- 172.21.15.6:0/363004265 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08540a0830 con 0x5572e2cc72f0 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.421+0000 7f0852000700 1 -- 172.21.15.6:0/363004265 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085409a5b0 con 0x5572e2cc72f0 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.422+0000 7f0871e2ab80 1 -- 172.21.15.6:0/363004265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc72f0 msgr2=0x5572e2cc76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.422+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/363004265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc72f0 0x5572e2cc76d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f08540cdca0 tx=0x7f085409bf10 comp rx=0 tx=0).stop 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.422+0000 7f0871e2ab80 1 -- 172.21.15.6:0/363004265 shutdown_connections 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.422+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/363004265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cd2ab0 0x5572e2cd4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.422+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/363004265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc72f0 0x5572e2cc76d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.422+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/363004265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc7c10 0x5572e2cd2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.422+0000 7f0871e2ab80 1 -- 172.21.15.6:0/363004265 >> 172.21.15.6:0/363004265 conn(0x5572e2bf6550 msgr2=0x5572e2cc3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.422+0000 7f0871e2ab80 1 -- 172.21.15.6:0/363004265 shutdown_connections 2024-04-26T01:44:56.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.422+0000 7f0871e2ab80 1 -- 172.21.15.6:0/363004265 wait complete. 2024-04-26T01:44:56.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.423+0000 7f0871e2ab80 1 -- start start 2024-04-26T01:44:56.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.423+0000 7f0871e2ab80 1 Processor -- start 2024-04-26T01:44:56.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.423+0000 7f0871e2ab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 0x5572e2cd2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.423+0000 7f0871e2ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cc7c10 0x5572e2ccf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.423+0000 7f0871e2ab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cd2ab0 0x5572e2ccfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.423+0000 7f0871e2ab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5572e2cd5540 con 0x5572e2cc72f0 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.423+0000 7f0871e2ab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5572e2bd4810 con 0x5572e2cd2ab0 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.423+0000 7f0871e2ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5572e2cd56c0 con 0x5572e2cc7c10 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.424+0000 7f0871e2ab80 10 client.0 _ll_register_callbacks cb 0x5572e2bb36d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.424+0000 7f0865400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cd2ab0 0x5572e2ccfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.424+0000 7f0865400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cd2ab0 0x5572e2ccfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:60400/0 (socket says 172.21.15.6:60400) 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.424+0000 7f0865400700 1 -- 172.21.15.6:0/1794157340 learned_addr learned my addr 172.21.15.6:0/1794157340 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.424+0000 7f0864a00700 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 0x5572e2cd2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.424+0000 7f085fe00700 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cc7c10 0x5572e2ccf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.424+0000 7f0865400700 1 -- 172.21.15.6:0/1794157340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cc7c10 msgr2=0x5572e2ccf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.424+0000 7f0865400700 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cc7c10 0x5572e2ccf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.424+0000 7f0865400700 1 -- 172.21.15.6:0/1794157340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 msgr2=0x5572e2cd2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.424+0000 7f0865400700 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 0x5572e2cd2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.424+0000 7f0865400700 1 -- 172.21.15.6:0/1794157340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572e2b68880 con 0x5572e2cd2ab0 2024-04-26T01:44:56.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.424+0000 7f0864a00700 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 0x5572e2cd2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:56.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.425+0000 7f0865400700 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cd2ab0 0x5572e2ccfa70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f085801a7a0 tx=0x7f0858004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c261a119a5abf5bf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:56.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.425+0000 7f0846a00700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:44:56.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.425+0000 7f0846a00700 1 -- 172.21.15.6:0/1794157340 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0858008260 con 0x5572e2cd2ab0 2024-04-26T01:44:56.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.425+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1794157340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08540a0830 con 0x5572e2cd2ab0 2024-04-26T01:44:56.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.425+0000 7f0846a00700 1 -- 172.21.15.6:0/1794157340 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f0858012040 con 0x5572e2cd2ab0 2024-04-26T01:44:56.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.425+0000 7f0846a00700 1 -- 172.21.15.6:0/1794157340 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0858004040 con 0x5572e2cd2ab0 2024-04-26T01:44:56.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.425+0000 7f0871e2ab80 10 client.24896 fetch_fsmap learned FSMap version 25 2024-04-26T01:44:56.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.425+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1794157340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5572e2bb1aa0 con 0x5572e2cd2ab0 2024-04-26T01:44:56.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.426+0000 7f0846a00700 1 -- 172.21.15.6:0/1794157340 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08540a0830 con 0x5572e2cd2ab0 2024-04-26T01:44:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.426+0000 7f0846a00700 1 -- 172.21.15.6:0/1794157340 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f08580077b0 con 0x5572e2cd2ab0 2024-04-26T01:44:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.426+0000 7f0871e2ab80 10 client.24896 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:44:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.426+0000 7f0871e2ab80 10 client.24896 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:44:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.426+0000 7f0871e2ab80 20 client.24896 populate_metadata read hostname 'smithi006' 2024-04-26T01:44:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.426+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5572e2ce2270 0x5572e2ce36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.426+0000 7f0871e2ab80 4 client.24896 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.426+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1794157340 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572e2c28630 con 0x5572e2ce2270 2024-04-26T01:44:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.427+0000 7f0864a00700 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5572e2ce2270 0x5572e2ce36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.427+0000 7f0864a00700 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5572e2ce2270 0x5572e2ce36e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e003d848b8ea0f0 server_cookie=9e696920bb46b493 in_seq=0 out_seq=0 2024-04-26T01:44:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.427+0000 7f0846a00700 10 client.24896 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:44:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.428+0000 7f0846a00700 1 -- 172.21.15.6:0/1794157340 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5572e2c28630 con 0x5572e2ce2270 2024-04-26T01:44:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.428+0000 7f0846a00700 10 client.24896 handle_command_reply: tid=0 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.428+0000 7f0871e2ab80 1 client.24896 shutdown 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.428+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1794157340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cd2ab0 msgr2=0x5572e2ccfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.428+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cd2ab0 0x5572e2ccfa70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f085801a7a0 tx=0x7f0858004b40 comp rx=0 tx=0).stop 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.428+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1794157340 shutdown_connections 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.428+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cd2ab0 0x5572e2ccfa70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.428+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 0x5572e2cd2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.428+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5572e2ce2270 0x5572e2ce36e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.428+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/1794157340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cc7c10 0x5572e2ccf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.428+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1794157340 >> 172.21.15.6:0/1794157340 conn(0x5572e2bf6550 msgr2=0x5572e2cc2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.428+0000 7f0846a00700 0 client.24896 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.429+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1794157340 shutdown_connections 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.429+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1794157340 wait complete. 2024-04-26T01:44:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.429+0000 7f0871e2ab80 20 client.24896 trim_cache size 0 max 16384 2024-04-26T01:44:56.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.454+0000 7f0871e2ab80 1 Processor -- start 2024-04-26T01:44:56.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.454+0000 7f0871e2ab80 1 -- start start 2024-04-26T01:44:56.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.454+0000 7f0871e2ab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc72f0 0x5572e2ccfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.454+0000 7f0871e2ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cc7c10 0x5572e2cc30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.454+0000 7f0871e2ab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cd2ab0 0x5572e2cc3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.454+0000 7f0871e2ab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5572e2ce1a20 con 0x5572e2cd2ab0 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.454+0000 7f0871e2ab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5572e2ce20f0 con 0x5572e2cc72f0 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.454+0000 7f0871e2ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5572e2ce27c0 con 0x5572e2cc7c10 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.454+0000 7f0865400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cd2ab0 0x5572e2cc3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.455+0000 7f0865400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cd2ab0 0x5572e2cc3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39370/0 (socket says 172.21.15.6:39370) 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.455+0000 7f0865400700 1 -- 172.21.15.6:0/3013414682 learned_addr learned my addr 172.21.15.6:0/3013414682 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.455+0000 7f0864a00700 1 --2- 172.21.15.6:0/3013414682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc72f0 0x5572e2ccfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.455+0000 7f085fe00700 1 --2- 172.21.15.6:0/3013414682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cc7c10 0x5572e2cc30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.455+0000 7f0865400700 1 -- 172.21.15.6:0/3013414682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cc7c10 msgr2=0x5572e2cc30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.455+0000 7f0865400700 1 --2- 172.21.15.6:0/3013414682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cc7c10 0x5572e2cc30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.455+0000 7f0865400700 1 -- 172.21.15.6:0/3013414682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc72f0 msgr2=0x5572e2ccfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.455+0000 7f0865400700 1 --2- 172.21.15.6:0/3013414682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc72f0 0x5572e2ccfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.455+0000 7f0865400700 1 -- 172.21.15.6:0/3013414682 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572e2ccdaa0 con 0x5572e2cd2ab0 2024-04-26T01:44:56.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.455+0000 7f0864a00700 1 --2- 172.21.15.6:0/3013414682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc72f0 0x5572e2ccfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.455+0000 7f0865400700 1 --2- 172.21.15.6:0/3013414682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cd2ab0 0x5572e2cc3610 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0858007fa0 tx=0x7f0858007fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c09ce203f64e80 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.456+0000 7f0851600700 1 -- 172.21.15.6:0/3013414682 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0858008260 con 0x5572e2cd2ab0 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.456+0000 7f0851600700 1 -- 172.21.15.6:0/3013414682 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0858008400 con 0x5572e2cd2ab0 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.456+0000 7f0851600700 1 -- 172.21.15.6:0/3013414682 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0858004480 con 0x5572e2cd2ab0 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.456+0000 7f0871e2ab80 1 -- 172.21.15.6:0/3013414682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cd2ab0 msgr2=0x5572e2cc3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.456+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/3013414682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cd2ab0 0x5572e2cc3610 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0858007fa0 tx=0x7f0858007fd0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.456+0000 7f0871e2ab80 1 -- 172.21.15.6:0/3013414682 shutdown_connections 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.456+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/3013414682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cd2ab0 0x5572e2cc3610 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.456+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/3013414682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc72f0 0x5572e2ccfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.457+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/3013414682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cc7c10 0x5572e2cc30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.457+0000 7f0871e2ab80 1 -- 172.21.15.6:0/3013414682 >> 172.21.15.6:0/3013414682 conn(0x5572e2bf6550 msgr2=0x5572e2ccf310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.457+0000 7f0871e2ab80 1 -- 172.21.15.6:0/3013414682 shutdown_connections 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.457+0000 7f0871e2ab80 1 -- 172.21.15.6:0/3013414682 wait complete. 2024-04-26T01:44:56.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.457+0000 7f0871e2ab80 1 -- start start 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.457+0000 7f0871e2ab80 1 Processor -- start 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.458+0000 7f0871e2ab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 0x5572e2ceccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.458+0000 7f0871e2ab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc7c10 0x5572e2cd3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.458+0000 7f0871e2ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cd2ab0 0x5572e2cd4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.458+0000 7f0871e2ab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5572e2ce3560 con 0x5572e2cc72f0 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.458+0000 7f0871e2ab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5572e2ce2e90 con 0x5572e2cc7c10 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.458+0000 7f0871e2ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5572e2ce3c30 con 0x5572e2cd2ab0 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.458+0000 7f0871e2ab80 10 client.0 _ll_register_callbacks cb 0x5572e2bb36d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.458+0000 7f0864a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 0x5572e2ceccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.458+0000 7f0864a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 0x5572e2ceccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39376/0 (socket says 172.21.15.6:39376) 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.458+0000 7f0864a00700 1 -- 172.21.15.6:0/1183613806 learned_addr learned my addr 172.21.15.6:0/1183613806 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.458+0000 7f0865400700 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cd2ab0 0x5572e2cd4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.458+0000 7f085fe00700 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc7c10 0x5572e2cd3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.459+0000 7f0864a00700 1 -- 172.21.15.6:0/1183613806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cd2ab0 msgr2=0x5572e2cd4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:56.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.459+0000 7f0864a00700 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cd2ab0 0x5572e2cd4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.459+0000 7f0864a00700 1 -- 172.21.15.6:0/1183613806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc7c10 msgr2=0x5572e2cd3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:56.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.459+0000 7f0864a00700 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc7c10 0x5572e2cd3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.459+0000 7f0864a00700 1 -- 172.21.15.6:0/1183613806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572e2cd4600 con 0x5572e2cc72f0 2024-04-26T01:44:56.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.459+0000 7f0864a00700 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 0x5572e2ceccd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0854007e80 tx=0x7f0854004fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c792074269f07dec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:56.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.459+0000 7f0846a00700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:44:56.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.459+0000 7f0846a00700 1 -- 172.21.15.6:0/1183613806 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08540b9070 con 0x5572e2cc72f0 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.459+0000 7f0846a00700 1 -- 172.21.15.6:0/1183613806 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f08540c41b0 con 0x5572e2cc72f0 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.459+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1183613806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0858008400 con 0x5572e2cc72f0 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.460+0000 7f0846a00700 1 -- 172.21.15.6:0/1183613806 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f08540a0830 con 0x5572e2cc72f0 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.460+0000 7f0871e2ab80 10 client.14799 fetch_fsmap learned FSMap version 25 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.460+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1183613806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5572e2d493a0 con 0x5572e2cc72f0 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.460+0000 7f0846a00700 1 -- 172.21.15.6:0/1183613806 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0858008400 con 0x5572e2cc72f0 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.460+0000 7f0846a00700 1 -- 172.21.15.6:0/1183613806 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f08540bb1f0 con 0x5572e2cc72f0 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.460+0000 7f0871e2ab80 10 client.14799 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.460+0000 7f0871e2ab80 10 client.14799 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.460+0000 7f0871e2ab80 20 client.14799 populate_metadata read hostname 'smithi006' 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.460+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5572e2d4db10 0x5572e2d4c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.460+0000 7f0871e2ab80 4 client.14799 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.460+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1183613806 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "status"}) -- 0x5572e2c226d0 con 0x5572e2d4db10 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.461+0000 7f085fe00700 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5572e2d4db10 0x5572e2d4c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:56.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.461+0000 7f085fe00700 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5572e2d4db10 0x5572e2d4c820 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83e11124cad6f4de server_cookie=2ac88569fa3c7dd3 in_seq=0 out_seq=0 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.461+0000 7f0846a00700 10 client.14799 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0846a00700 1 -- 172.21.15.6:0/1183613806 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5572e2c226d0 con 0x5572e2d4db10 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0846a00700 10 client.14799 handle_command_reply: tid=0 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0871e2ab80 1 client.14799 shutdown 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1183613806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 msgr2=0x5572e2ceccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 0x5572e2ceccd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0854007e80 tx=0x7f0854004fb0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1183613806 shutdown_connections 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5572e2d4db10 0x5572e2d4c820 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5572e2cd2ab0 0x5572e2cd4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5572e2cc72f0 0x5572e2ceccd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0871e2ab80 1 --2- 172.21.15.6:0/1183613806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572e2cc7c10 0x5572e2cd3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1183613806 >> 172.21.15.6:0/1183613806 conn(0x5572e2bf6550 msgr2=0x5572e2cd0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:56.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0846a00700 0 client.14799 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:44:56.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1183613806 shutdown_connections 2024-04-26T01:44:56.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0871e2ab80 1 -- 172.21.15.6:0/1183613806 wait complete. 2024-04-26T01:44:56.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.462+0000 7f0871e2ab80 20 client.14799 trim_cache size 0 max 16384 2024-04-26T01:44:56.462 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:44:56.462 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:44:56.463 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-04-26T01:44:56.463 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 24523, 2024-04-26T01:44:56.463 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-26T01:44:56.463 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-26T01:44:56.463 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-26T01:44:56.463 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 168.05767308599999, 2024-04-26T01:44:56.463 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 25, 2024-04-26T01:44:56.463 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 95, 2024-04-26T01:44:56.463 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T01:44:56.463 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 202.03969011300001 2024-04-26T01:44:56.463 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:44:56.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.463+0000 7f0867400700 1 -- 172.21.15.6:0/2910399055 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f08480a3c40 msgr2=0x7f08480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:56.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.463+0000 7f0867400700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f08480a3c40 0x7f08480a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x5572e2cc8d70 tx=0x7f085800b040 comp rx=0 tx=0).stop 2024-04-26T01:44:56.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.463+0000 7f0867400700 1 -- 172.21.15.6:0/2910399055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0860104ee0 msgr2=0x7f086019b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:56.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.463+0000 7f0867400700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0860104ee0 0x7f086019b410 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f085400d980 tx=0x7f085400dc90 comp rx=0 tx=0).stop 2024-04-26T01:44:56.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.463+0000 7f0867400700 1 -- 172.21.15.6:0/2910399055 shutdown_connections 2024-04-26T01:44:56.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.463+0000 7f0867400700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0860109fc0 0x7f086019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.463+0000 7f0867400700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f08480a3c40 0x7f08480a6100 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.463+0000 7f0867400700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0860104ee0 0x7f086019b410 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.463+0000 7f0867400700 1 --2- 172.21.15.6:0/2910399055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0860105890 0x7f086019b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:56.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.463+0000 7f0867400700 1 -- 172.21.15.6:0/2910399055 >> 172.21.15.6:0/2910399055 conn(0x7f0860100770 msgr2=0x7f0860102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:56.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.463+0000 7f0867400700 1 -- 172.21.15.6:0/2910399055 shutdown_connections 2024-04-26T01:44:56.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:44:56.463+0000 7f0867400700 1 -- 172.21.15.6:0/2910399055 wait complete. 2024-04-26T01:44:57.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:56 smithi006 ceph-mon[28419]: pgmap v419: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:57.311 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T01:44:57.311 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T01:44:57.322 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi006.front.sepia.ceph.com 2024-04-26T01:44:57.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-26T01:44:57.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:56 smithi188 ceph-mon[32751]: pgmap v419: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:57.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:56 smithi029 ceph-mon[32775]: pgmap v419: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:57.791 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:59.220 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:44:58 smithi006 ceph-mon[28419]: pgmap v420: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:59.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:44:58 smithi188 ceph-mon[32751]: pgmap v420: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:59.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:44:58 smithi029 ceph-mon[32775]: pgmap v420: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:00.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.055+0000 7f997de00700 1 -- 172.21.15.6:0/4173706474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f997810a8a0 msgr2=0x7f997810cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:00.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.055+0000 7f997de00700 1 --2- 172.21.15.6:0/4173706474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f997810a8a0 0x7f997810cd30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f997000b0a0 tx=0x7f997000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.055+0000 7f997de00700 1 -- 172.21.15.6:0/4173706474 shutdown_connections 2024-04-26T01:45:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.055+0000 7f997de00700 1 --2- 172.21.15.6:0/4173706474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f997810a8a0 0x7f997810cd30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.055+0000 7f997de00700 1 --2- 172.21.15.6:0/4173706474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9978076a70 0x7f9978074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.055+0000 7f997de00700 1 --2- 172.21.15.6:0/4173706474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99780754a0 0x7f9978075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.055+0000 7f997de00700 1 -- 172.21.15.6:0/4173706474 >> 172.21.15.6:0/4173706474 conn(0x7f99780fdf70 msgr2=0x7f99781003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.055+0000 7f997de00700 1 -- 172.21.15.6:0/4173706474 shutdown_connections 2024-04-26T01:45:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.055+0000 7f997de00700 1 -- 172.21.15.6:0/4173706474 wait complete. 2024-04-26T01:45:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.056+0000 7f997de00700 1 Processor -- start 2024-04-26T01:45:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.057+0000 7f997de00700 1 -- start start 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.057+0000 7f997de00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780754a0 0x7f997819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.057+0000 7f997de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9978076a70 0x7f997819b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.057+0000 7f997de00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997810a8a0 0x7f99781a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.057+0000 7f997de00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9978110680 con 0x7f997810a8a0 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.057+0000 7f997de00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9978110500 con 0x7f99780754a0 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.057+0000 7f997de00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9978110800 con 0x7f9978076a70 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.057+0000 7f997ca00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780754a0 0x7f997819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.057+0000 7f997ca00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780754a0 0x7f997819ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56908/0 (socket says 172.21.15.6:56908) 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.057+0000 7f997ca00700 1 -- 172.21.15.6:0/1136145207 learned_addr learned my addr 172.21.15.6:0/1136145207 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.058+0000 7f996fe00700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9978076a70 0x7f997819b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.058+0000 7f997d400700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997810a8a0 0x7f99781a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.058+0000 7f997ca00700 1 -- 172.21.15.6:0/1136145207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9978076a70 msgr2=0x7f997819b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:00.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.058+0000 7f997ca00700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9978076a70 0x7f997819b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.058+0000 7f997ca00700 1 -- 172.21.15.6:0/1136145207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997810a8a0 msgr2=0x7f99781a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:00.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.058+0000 7f997ca00700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997810a8a0 0x7f99781a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.058+0000 7f997ca00700 1 -- 172.21.15.6:0/1136145207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99781a28f0 con 0x7f99780754a0 2024-04-26T01:45:00.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.058+0000 7f997ca00700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780754a0 0x7f997819ac70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f996800ea30 tx=0x7f996800ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:00.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.058+0000 7f9976000700 1 -- 172.21.15.6:0/1136145207 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996800cbd0 con 0x7f99780754a0 2024-04-26T01:45:00.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.059+0000 7f9976000700 1 -- 172.21.15.6:0/1136145207 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99680044d0 con 0x7f99780754a0 2024-04-26T01:45:00.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.059+0000 7f997de00700 1 -- 172.21.15.6:0/1136145207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99781a2be0 con 0x7f99780754a0 2024-04-26T01:45:00.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.059+0000 7f9976000700 1 -- 172.21.15.6:0/1136145207 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9968003dc0 con 0x7f99780754a0 2024-04-26T01:45:00.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.059+0000 7f997de00700 1 -- 172.21.15.6:0/1136145207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99781a3120 con 0x7f99780754a0 2024-04-26T01:45:00.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.060+0000 7f997de00700 1 -- 172.21.15.6:0/1136145207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9978069730 con 0x7f99780754a0 2024-04-26T01:45:00.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.062+0000 7f9976000700 1 -- 172.21.15.6:0/1136145207 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f9968004000 con 0x7f99780754a0 2024-04-26T01:45:00.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.063+0000 7f9976000700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f995c0a3c40 0x7f995c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:00.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.063+0000 7f9976000700 1 -- 172.21.15.6:0/1136145207 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f99680d0580 con 0x7f99780754a0 2024-04-26T01:45:00.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.063+0000 7f996fe00700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f995c0a3c40 0x7f995c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:00.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.064+0000 7f996fe00700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f995c0a3c40 0x7f995c0a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f997819c180 tx=0x7f9960009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:00.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.068+0000 7f9976000700 1 -- 172.21.15.6:0/1136145207 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99680042f0 con 0x7f99780754a0 2024-04-26T01:45:00.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.313+0000 7f997de00700 1 -- 172.21.15.6:0/1136145207 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f9978062e80 con 0x7f995c0a3c40 2024-04-26T01:45:00.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.315+0000 7f9976000700 1 -- 172.21.15.6:0/1136145207 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f9978062e80 con 0x7f995c0a3c40 2024-04-26T01:45:00.313 INFO:teuthology.orchestra.run.smithi006.stdout:Backend: cephadm 2024-04-26T01:45:00.313 INFO:teuthology.orchestra.run.smithi006.stdout:Available: Yes 2024-04-26T01:45:00.313 INFO:teuthology.orchestra.run.smithi006.stdout:Paused: No 2024-04-26T01:45:00.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.318+0000 7f997de00700 1 -- 172.21.15.6:0/1136145207 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f995c0a3c40 msgr2=0x7f995c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:00.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.318+0000 7f997de00700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f995c0a3c40 0x7f995c0a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f997819c180 tx=0x7f9960009260 comp rx=0 tx=0).stop 2024-04-26T01:45:00.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.318+0000 7f997de00700 1 -- 172.21.15.6:0/1136145207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780754a0 msgr2=0x7f997819ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:00.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.318+0000 7f997de00700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780754a0 0x7f997819ac70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f996800ea30 tx=0x7f996800ed40 comp rx=0 tx=0).stop 2024-04-26T01:45:00.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.318+0000 7f997de00700 1 -- 172.21.15.6:0/1136145207 shutdown_connections 2024-04-26T01:45:00.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.318+0000 7f997de00700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997810a8a0 0x7f99781a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.318+0000 7f997de00700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780754a0 0x7f997819ac70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.318+0000 7f997de00700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f995c0a3c40 0x7f995c0a6100 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.318+0000 7f997de00700 1 --2- 172.21.15.6:0/1136145207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9978076a70 0x7f997819b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.318+0000 7f997de00700 1 -- 172.21.15.6:0/1136145207 >> 172.21.15.6:0/1136145207 conn(0x7f99780fdf70 msgr2=0x7f99781067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:00.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.318+0000 7f997de00700 1 -- 172.21.15.6:0/1136145207 shutdown_connections 2024-04-26T01:45:00.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:00.318+0000 7f997de00700 1 -- 172.21.15.6:0/1136145207 wait complete. 2024-04-26T01:45:01.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:00 smithi006 ceph-mon[28419]: pgmap v421: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:01.080 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-26T01:45:01.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:00 smithi188 ceph-mon[32751]: pgmap v421: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:01.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:00 smithi029 ceph-mon[32775]: pgmap v421: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:01.543 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:02.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:01 smithi006 ceph-mon[28419]: from='client.24908 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:02.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:01 smithi188 ceph-mon[32751]: from='client.24908 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:02.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:01 smithi029 ceph-mon[32775]: from='client.24908 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:03.255 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:02 smithi006 ceph-mon[28419]: pgmap v422: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:03.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:02 smithi188 ceph-mon[32751]: pgmap v422: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:03.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:02 smithi029 ceph-mon[32775]: pgmap v422: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:03.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.575+0000 7f9b9e600700 1 -- 172.21.15.6:0/3046033494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b98103530 msgr2=0x7f9b98105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:03.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.575+0000 7f9b9e600700 1 --2- 172.21.15.6:0/3046033494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b98103530 0x7f9b98105920 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800b0a0 tx=0x7f9b8800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.575+0000 7f9b9e600700 1 -- 172.21.15.6:0/3046033494 shutdown_connections 2024-04-26T01:45:03.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.575+0000 7f9b9e600700 1 --2- 172.21.15.6:0/3046033494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b98108790 0x7f9b9810ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.575+0000 7f9b9e600700 1 --2- 172.21.15.6:0/3046033494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b98103530 0x7f9b98105920 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.575+0000 7f9b9e600700 1 --2- 172.21.15.6:0/3046033494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98105e60 0x7f9b98108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.575+0000 7f9b9e600700 1 -- 172.21.15.6:0/3046033494 >> 172.21.15.6:0/3046033494 conn(0x7f9b980fcf40 msgr2=0x7f9b980ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:03.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.575+0000 7f9b9e600700 1 -- 172.21.15.6:0/3046033494 shutdown_connections 2024-04-26T01:45:03.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.575+0000 7f9b9e600700 1 -- 172.21.15.6:0/3046033494 wait complete. 2024-04-26T01:45:03.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.577+0000 7f9b9e600700 1 Processor -- start 2024-04-26T01:45:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.577+0000 7f9b9e600700 1 -- start start 2024-04-26T01:45:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b9e600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98103530 0x7f9b98102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b9e600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b98105e60 0x7f9b981014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b9e600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b98108790 0x7f9b98101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b9e600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9810e5a0 con 0x7f9b98103530 2024-04-26T01:45:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b9e600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9810e420 con 0x7f9b98108790 2024-04-26T01:45:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b9e600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9810e720 con 0x7f9b98105e60 2024-04-26T01:45:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b97400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98103530 0x7f9b98102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b8ea00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b98105e60 0x7f9b981014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b97400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98103530 0x7f9b98102e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58722/0 (socket says 172.21.15.6:58722) 2024-04-26T01:45:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b97400700 1 -- 172.21.15.6:0/776305808 learned_addr learned my addr 172.21.15.6:0/776305808 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b97e00700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b98108790 0x7f9b98101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:03.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b97400700 1 -- 172.21.15.6:0/776305808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b98105e60 msgr2=0x7f9b981014e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:03.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b97400700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b98105e60 0x7f9b981014e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b97400700 1 -- 172.21.15.6:0/776305808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b98108790 msgr2=0x7f9b98101a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:03.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b97400700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b98108790 0x7f9b98101a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.578+0000 7f9b97400700 1 -- 172.21.15.6:0/776305808 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b981022d0 con 0x7f9b98103530 2024-04-26T01:45:03.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.579+0000 7f9b97400700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98103530 0x7f9b98102e90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800b070 tx=0x7f9b88004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:03.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.579+0000 7f9b94c00700 1 -- 172.21.15.6:0/776305808 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b88021070 con 0x7f9b98103530 2024-04-26T01:45:03.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.579+0000 7f9b9e600700 1 -- 172.21.15.6:0/776305808 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b981a2d80 con 0x7f9b98103530 2024-04-26T01:45:03.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.579+0000 7f9b94c00700 1 -- 172.21.15.6:0/776305808 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b88007e30 con 0x7f9b98103530 2024-04-26T01:45:03.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.579+0000 7f9b94c00700 1 -- 172.21.15.6:0/776305808 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b880047e0 con 0x7f9b98103530 2024-04-26T01:45:03.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.579+0000 7f9b9e600700 1 -- 172.21.15.6:0/776305808 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b981a3180 con 0x7f9b98103530 2024-04-26T01:45:03.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.580+0000 7f9b9e600700 1 -- 172.21.15.6:0/776305808 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b9802df80 con 0x7f9b98103530 2024-04-26T01:45:03.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.582+0000 7f9b94c00700 1 -- 172.21.15.6:0/776305808 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f9b88003c00 con 0x7f9b98103530 2024-04-26T01:45:03.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.584+0000 7f9b94c00700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9b7c0a3cc0 0x7f9b7c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:03.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.584+0000 7f9b94c00700 1 -- 172.21.15.6:0/776305808 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f9b8800f020 con 0x7f9b98103530 2024-04-26T01:45:03.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.584+0000 7f9b8ea00700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9b7c0a3cc0 0x7f9b7c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:03.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.585+0000 7f9b8ea00700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9b7c0a3cc0 0x7f9b7c0a6180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009c60 tx=0x7f9b80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:03.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.587+0000 7f9b94c00700 1 -- 172.21.15.6:0/776305808 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9b8802a020 con 0x7f9b98103530 2024-04-26T01:45:03.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.826+0000 7f9b9e600700 1 -- 172.21.15.6:0/776305808 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f9b98062e80 con 0x7f9b7c0a3cc0 2024-04-26T01:45:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.868+0000 7f9b94c00700 1 -- 172.21.15.6:0/776305808 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f9b98062e80 con 0x7f9b7c0a3cc0 2024-04-26T01:45:03.866 INFO:teuthology.orchestra.run.smithi006.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-26T01:45:03.866 INFO:teuthology.orchestra.run.smithi006.stdout:mds.a smithi006 running (3m) 44s ago 3m 12.8M - 19.0.0-3156-gd69f13d8 da4b004c09e8 359947f8e343 2024-04-26T01:45:03.866 INFO:teuthology.orchestra.run.smithi006.stdout:mds.b smithi029 running (3m) 56s ago 3m 22.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 e0537abc3b1f 2024-04-26T01:45:03.866 INFO:teuthology.orchestra.run.smithi006.stdout:mds.c smithi188 running (3m) 56s ago 3m 20.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 9a2bf011e344 2024-04-26T01:45:03.866 INFO:teuthology.orchestra.run.smithi006.stdout:mds.d smithi006 running (3m) 44s ago 3m 13.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 f7c3779515b9 2024-04-26T01:45:03.866 INFO:teuthology.orchestra.run.smithi006.stdout:mds.e smithi029 running (3m) 56s ago 3m 16.7M - 19.0.0-3156-gd69f13d8 da4b004c09e8 2a76fddeb2ef 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mds.f smithi188 running (3m) 56s ago 3m 19.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 146c93119142 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mds.g smithi006 running (3m) 44s ago 3m 17.8M - 19.0.0-3156-gd69f13d8 da4b004c09e8 c3d95a031c7f 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mds.h smithi029 running (3m) 56s ago 3m 19.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 cbe1eaab51c0 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mds.i smithi188 running (3m) 56s ago 3m 26.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 70a79c79ae13 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mds.j smithi006 running (3m) 44s ago 3m 13.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 1eef28c3762a 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mds.k smithi029 running (3m) 56s ago 3m 16.7M - 19.0.0-3156-gd69f13d8 da4b004c09e8 a4fa73014541 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mds.l smithi188 running (3m) 56s ago 3m 15.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 109c8f8d7b6c 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.x smithi006 *:9283,8765 running (12m) 44s ago 12m 515M - 19.0.0-3156-gd69f13d8 da4b004c09e8 879fdfd52bda 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.y smithi029 *:8443,8765 running (9m) 56s ago 9m 453M - 19.0.0-3156-gd69f13d8 da4b004c09e8 c9b317ef545a 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.z smithi188 *:8443,8765 running (9m) 56s ago 9m 453M - 19.0.0-3156-gd69f13d8 da4b004c09e8 8a425eb7f7ea 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mon.a smithi006 running (12m) 44s ago 12m 48.4M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 3d4cd5294d32 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mon.b smithi029 running (10m) 56s ago 10m 44.1M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 3724a8af3fb8 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:mon.c smithi188 running (10m) 56s ago 10m 40.8M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 344aa2641bc9 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0 smithi006 running (9m) 44s ago 9m 47.8M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 675a86c10535 2024-04-26T01:45:03.867 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 smithi029 running (9m) 56s ago 9m 59.8M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 4e4681024de1 2024-04-26T01:45:03.868 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2 smithi188 running (8m) 56s ago 8m 52.0M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 17999d9593c4 2024-04-26T01:45:03.868 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3 smithi006 running (8m) 44s ago 8m 48.2M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 2ee7646876b8 2024-04-26T01:45:03.868 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4 smithi029 running (7m) 56s ago 7m 54.5M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 71819c45edc1 2024-04-26T01:45:03.868 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5 smithi188 running (7m) 56s ago 7m 55.6M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 e2c65132c774 2024-04-26T01:45:03.868 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6 smithi006 running (6m) 44s ago 6m 44.0M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 6655478b65eb 2024-04-26T01:45:03.868 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7 smithi029 running (6m) 56s ago 6m 47.6M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 1e3e25e66831 2024-04-26T01:45:03.868 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8 smithi188 running (5m) 56s ago 5m 54.4M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 ae578cb99da6 2024-04-26T01:45:03.868 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9 smithi006 running (5m) 44s ago 5m 44.6M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 6e4b86ec1598 2024-04-26T01:45:03.868 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10 smithi029 running (4m) 56s ago 4m 55.9M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 2fcaace8a976 2024-04-26T01:45:03.868 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11 smithi188 running (4m) 56s ago 4m 52.5M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 d8a773005a8d 2024-04-26T01:45:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.871+0000 7f9b9e600700 1 -- 172.21.15.6:0/776305808 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9b7c0a3cc0 msgr2=0x7f9b7c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.871+0000 7f9b9e600700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9b7c0a3cc0 0x7f9b7c0a6180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009c60 tx=0x7f9b80009260 comp rx=0 tx=0).stop 2024-04-26T01:45:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.871+0000 7f9b9e600700 1 -- 172.21.15.6:0/776305808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98103530 msgr2=0x7f9b98102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.871+0000 7f9b9e600700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98103530 0x7f9b98102e90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800b070 tx=0x7f9b88004d70 comp rx=0 tx=0).stop 2024-04-26T01:45:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.871+0000 7f9b9e600700 1 -- 172.21.15.6:0/776305808 shutdown_connections 2024-04-26T01:45:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.871+0000 7f9b9e600700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b98108790 0x7f9b98101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.871+0000 7f9b9e600700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98103530 0x7f9b98102e90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.871+0000 7f9b9e600700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f9b7c0a3cc0 0x7f9b7c0a6180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.871+0000 7f9b9e600700 1 --2- 172.21.15.6:0/776305808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b98105e60 0x7f9b981014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.871+0000 7f9b9e600700 1 -- 172.21.15.6:0/776305808 >> 172.21.15.6:0/776305808 conn(0x7f9b980fcf40 msgr2=0x7f9b98103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.871+0000 7f9b9e600700 1 -- 172.21.15.6:0/776305808 shutdown_connections 2024-04-26T01:45:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:03.872+0000 7f9b9e600700 1 -- 172.21.15.6:0/776305808 wait complete. 2024-04-26T01:45:04.673 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-26T01:45:05.150 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:05.306 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:05 smithi006 ceph-mon[28419]: pgmap v423: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:05.306 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:05 smithi006 ceph-mon[28419]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:05.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:05 smithi188 ceph-mon[32751]: pgmap v423: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:05.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:05 smithi188 ceph-mon[32751]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:05.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:05 smithi029 ceph-mon[32775]: pgmap v423: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:05.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:05 smithi029 ceph-mon[32775]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:07.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.276+0000 7f411e400700 1 -- 172.21.15.6:0/2376494493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4118103c90 msgr2=0x7f4118106120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.276+0000 7f411e400700 1 --2- 172.21.15.6:0/2376494493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4118103c90 0x7f4118106120 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4104009990 tx=0x7f4104009ca0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.277+0000 7f411e400700 1 -- 172.21.15.6:0/2376494493 shutdown_connections 2024-04-26T01:45:07.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.277+0000 7f411e400700 1 --2- 172.21.15.6:0/2376494493 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4118106660 0x7f4118106ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.277+0000 7f411e400700 1 --2- 172.21.15.6:0/2376494493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4118101320 0x7f4118103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.277+0000 7f411e400700 1 --2- 172.21.15.6:0/2376494493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4118103c90 0x7f4118106120 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.277+0000 7f411e400700 1 -- 172.21.15.6:0/2376494493 >> 172.21.15.6:0/2376494493 conn(0x7f41180fcf40 msgr2=0x7f41180ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:07.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.277+0000 7f411e400700 1 -- 172.21.15.6:0/2376494493 shutdown_connections 2024-04-26T01:45:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.277+0000 7f411e400700 1 -- 172.21.15.6:0/2376494493 wait complete. 2024-04-26T01:45:07.275 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:07 smithi006 ceph-mon[28419]: pgmap v424: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:45:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f411e400700 1 Processor -- start 2024-04-26T01:45:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f411e400700 1 -- start start 2024-04-26T01:45:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f411e400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4118101320 0x7f411819f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f411e400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4118103c90 0x7f411819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f411e400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4118106660 0x7f41181a6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f411e400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4118114a90 con 0x7f4118103c90 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f411e400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4118114910 con 0x7f4118106660 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f411e400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4118114c10 con 0x7f4118101320 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f4116a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4118103c90 0x7f411819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f4116a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4118103c90 0x7f411819f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58744/0 (socket says 172.21.15.6:58744) 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f4116a00700 1 -- 172.21.15.6:0/4269846002 learned_addr learned my addr 172.21.15.6:0/4269846002 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f4117400700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4118101320 0x7f411819f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f4117e00700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4118106660 0x7f41181a6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f4116a00700 1 -- 172.21.15.6:0/4269846002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4118101320 msgr2=0x7f411819f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f4116a00700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4118101320 0x7f411819f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f4116a00700 1 -- 172.21.15.6:0/4269846002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4118106660 msgr2=0x7f41181a6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f4116a00700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4118106660 0x7f41181a6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f4116a00700 1 -- 172.21.15.6:0/4269846002 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41181a6dc0 con 0x7f4118103c90 2024-04-26T01:45:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.278+0000 7f4117e00700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4118106660 0x7f41181a6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.280+0000 7f4116a00700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4118103c90 0x7f411819f6a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4104000c00 tx=0x7f4104009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.280+0000 7f410fe00700 1 -- 172.21.15.6:0/4269846002 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104026070 con 0x7f4118103c90 2024-04-26T01:45:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.280+0000 7f410fe00700 1 -- 172.21.15.6:0/4269846002 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4104021410 con 0x7f4118103c90 2024-04-26T01:45:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.280+0000 7f411e400700 1 -- 172.21.15.6:0/4269846002 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41181a7050 con 0x7f4118103c90 2024-04-26T01:45:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.280+0000 7f410fe00700 1 -- 172.21.15.6:0/4269846002 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104017650 con 0x7f4118103c90 2024-04-26T01:45:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.280+0000 7f411e400700 1 -- 172.21.15.6:0/4269846002 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41181a75b0 con 0x7f4118103c90 2024-04-26T01:45:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.282+0000 7f411e400700 1 -- 172.21.15.6:0/4269846002 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4118069730 con 0x7f4118103c90 2024-04-26T01:45:07.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.284+0000 7f410fe00700 1 -- 172.21.15.6:0/4269846002 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f4104029070 con 0x7f4118103c90 2024-04-26T01:45:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.285+0000 7f410fe00700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f40fc0a3cc0 0x7f40fc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.285+0000 7f410fe00700 1 -- 172.21.15.6:0/4269846002 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f41040d1340 con 0x7f4118103c90 2024-04-26T01:45:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.285+0000 7f4117400700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f40fc0a3cc0 0x7f40fc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:07.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.286+0000 7f4117400700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f40fc0a3cc0 0x7f40fc0a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f410800f540 tx=0x7f410800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:07.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.290+0000 7f410fe00700 1 -- 172.21.15.6:0/4269846002 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f410405dff0 con 0x7f4118103c90 2024-04-26T01:45:07.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:07 smithi188 ceph-mon[32751]: pgmap v424: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:45:07.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:07 smithi029 ceph-mon[32775]: pgmap v424: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:45:07.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.529+0000 7f411e400700 1 -- 172.21.15.6:0/4269846002 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f4118062e80 con 0x7f40fc0a3cc0 2024-04-26T01:45:07.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.553+0000 7f410fe00700 1 -- 172.21.15.6:0/4269846002 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f4118062e80 con 0x7f40fc0a3cc0 2024-04-26T01:45:07.550 INFO:teuthology.orchestra.run.smithi006.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-26T01:45:07.551 INFO:teuthology.orchestra.run.smithi006.stdout:mds.all 12/12 60s ago 4m smithi006=a;smithi006=d;smithi006=g;smithi006=j;smithi029=b;smithi029=e;smithi029=h;smithi029=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2024-04-26T01:45:07.551 INFO:teuthology.orchestra.run.smithi006.stdout:mgr 3/3 60s ago 10m smithi006=x;smithi029=y;smithi188=z;count:3 2024-04-26T01:45:07.551 INFO:teuthology.orchestra.run.smithi006.stdout:mon 3/3 60s ago 10m smithi006:172.21.15.6=a;smithi029:172.21.15.29=b;smithi188:172.21.15.188=c;count:3 2024-04-26T01:45:07.551 INFO:teuthology.orchestra.run.smithi006.stdout:osd 12 60s ago - 2024-04-26T01:45:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.556+0000 7f411e400700 1 -- 172.21.15.6:0/4269846002 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f40fc0a3cc0 msgr2=0x7f40fc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.556+0000 7f411e400700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f40fc0a3cc0 0x7f40fc0a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f410800f540 tx=0x7f410800b040 comp rx=0 tx=0).stop 2024-04-26T01:45:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.556+0000 7f411e400700 1 -- 172.21.15.6:0/4269846002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4118103c90 msgr2=0x7f411819f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.556+0000 7f411e400700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4118103c90 0x7f411819f6a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4104000c00 tx=0x7f4104009580 comp rx=0 tx=0).stop 2024-04-26T01:45:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.556+0000 7f411e400700 1 -- 172.21.15.6:0/4269846002 shutdown_connections 2024-04-26T01:45:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.556+0000 7f411e400700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4118106660 0x7f41181a6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.556+0000 7f411e400700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f40fc0a3cc0 0x7f40fc0a6180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.556+0000 7f411e400700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4118101320 0x7f411819f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.556+0000 7f411e400700 1 --2- 172.21.15.6:0/4269846002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4118103c90 0x7f411819f6a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.556+0000 7f411e400700 1 -- 172.21.15.6:0/4269846002 >> 172.21.15.6:0/4269846002 conn(0x7f41180fcf40 msgr2=0x7f4118101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.557+0000 7f411e400700 1 -- 172.21.15.6:0/4269846002 shutdown_connections 2024-04-26T01:45:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:07.557+0000 7f411e400700 1 -- 172.21.15.6:0/4269846002 wait complete. 2024-04-26T01:45:08.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-26T01:45:08.776 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:09.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:09 smithi006 ceph-mon[28419]: pgmap v425: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:09.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:09 smithi006 ceph-mon[28419]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:09.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:09 smithi188 ceph-mon[32751]: pgmap v425: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:09.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:09 smithi188 ceph-mon[32751]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:09.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:09 smithi029 ceph-mon[32775]: pgmap v425: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:09.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:09 smithi029 ceph-mon[32775]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:11.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.212+0000 7f01d6400700 1 -- 172.21.15.6:0/1849790111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d0078640 msgr2=0x7f01d0078ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:11.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.212+0000 7f01d6400700 1 --2- 172.21.15.6:0/1849790111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d0078640 0x7f01d0078ac0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009990 tx=0x7f01b8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:45:11.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.212+0000 7f01d6400700 1 -- 172.21.15.6:0/1849790111 shutdown_connections 2024-04-26T01:45:11.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.212+0000 7f01d6400700 1 --2- 172.21.15.6:0/1849790111 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d0079000 0x7f01d0079880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:11.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.212+0000 7f01d6400700 1 --2- 172.21.15.6:0/1849790111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01d00773f0 0x7f01d0077810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:11.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.212+0000 7f01d6400700 1 --2- 172.21.15.6:0/1849790111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d0078640 0x7f01d0078ac0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:11.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.212+0000 7f01d6400700 1 -- 172.21.15.6:0/1849790111 >> 172.21.15.6:0/1849790111 conn(0x7f01d0100290 msgr2=0x7f01d01026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:11.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.212+0000 7f01d6400700 1 -- 172.21.15.6:0/1849790111 shutdown_connections 2024-04-26T01:45:11.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.212+0000 7f01d6400700 1 -- 172.21.15.6:0/1849790111 wait complete. 2024-04-26T01:45:11.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.213+0000 7f01d6400700 1 Processor -- start 2024-04-26T01:45:11.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.214+0000 7f01d6400700 1 -- start start 2024-04-26T01:45:11.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.214+0000 7f01d6400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d00773f0 0x7f01d019f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:11.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.214+0000 7f01d6400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d0078640 0x7f01d019f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:11.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.214+0000 7f01d6400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01d0079000 0x7f01d01a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:11.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.214+0000 7f01d6400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f01d0114ba0 con 0x7f01d00773f0 2024-04-26T01:45:11.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.214+0000 7f01d6400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f01d0114a20 con 0x7f01d0078640 2024-04-26T01:45:11.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.214+0000 7f01d6400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f01d0114d20 con 0x7f01d0079000 2024-04-26T01:45:11.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.214+0000 7f01cf400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d00773f0 0x7f01d019f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:11.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.214+0000 7f01cf400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d00773f0 0x7f01d019f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40008/0 (socket says 172.21.15.6:40008) 2024-04-26T01:45:11.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.214+0000 7f01cf400700 1 -- 172.21.15.6:0/71331755 learned_addr learned my addr 172.21.15.6:0/71331755 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:11.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.214+0000 7f01c7e00700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d0078640 0x7f01d019f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:11.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.215+0000 7f01cfe00700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01d0079000 0x7f01d01a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:11.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.215+0000 7f01c7e00700 1 -- 172.21.15.6:0/71331755 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01d0079000 msgr2=0x7f01d01a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:11.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.215+0000 7f01c7e00700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01d0079000 0x7f01d01a6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:11.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.215+0000 7f01c7e00700 1 -- 172.21.15.6:0/71331755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d00773f0 msgr2=0x7f01d019f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:11.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.215+0000 7f01c7e00700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d00773f0 0x7f01d019f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:11.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.215+0000 7f01c7e00700 1 -- 172.21.15.6:0/71331755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d01a6e20 con 0x7f01d0078640 2024-04-26T01:45:11.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.215+0000 7f01cf400700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d00773f0 0x7f01d019f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:11.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.215+0000 7f01c7e00700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d0078640 0x7f01d019f700 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f01b8000c00 tx=0x7f01b80057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:11.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.216+0000 7f01ccc00700 1 -- 172.21.15.6:0/71331755 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b8026070 con 0x7f01d0078640 2024-04-26T01:45:11.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.216+0000 7f01d6400700 1 -- 172.21.15.6:0/71331755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d01a70b0 con 0x7f01d0078640 2024-04-26T01:45:11.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.216+0000 7f01ccc00700 1 -- 172.21.15.6:0/71331755 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01b8021410 con 0x7f01d0078640 2024-04-26T01:45:11.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.216+0000 7f01ccc00700 1 -- 172.21.15.6:0/71331755 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b8017650 con 0x7f01d0078640 2024-04-26T01:45:11.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.216+0000 7f01d6400700 1 -- 172.21.15.6:0/71331755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d01a7560 con 0x7f01d0078640 2024-04-26T01:45:11.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.217+0000 7f01d6400700 1 -- 172.21.15.6:0/71331755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d0069730 con 0x7f01d0078640 2024-04-26T01:45:11.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.219+0000 7f01ccc00700 1 -- 172.21.15.6:0/71331755 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f01b8029070 con 0x7f01d0078640 2024-04-26T01:45:11.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.220+0000 7f01ccc00700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f01b40a3c40 0x7f01b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:11.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.220+0000 7f01ccc00700 1 -- 172.21.15.6:0/71331755 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f01b80d1160 con 0x7f01d0078640 2024-04-26T01:45:11.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.221+0000 7f01cf400700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f01b40a3c40 0x7f01b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:11.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.221+0000 7f01cf400700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f01b40a3c40 0x7f01b40a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f01c0011550 tx=0x7f01c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:11.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.225+0000 7f01ccc00700 1 -- 172.21.15.6:0/71331755 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f01b8013070 con 0x7f01d0078640 2024-04-26T01:45:11.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:11 smithi006 ceph-mon[28419]: pgmap v426: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:11.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:11 smithi188 ceph-mon[32751]: pgmap v426: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:11.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:11 smithi029 ceph-mon[32775]: pgmap v426: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:11.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.466+0000 7f01d6400700 1 -- 172.21.15.6:0/71331755 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f01d0062e80 con 0x7f01b40a3c40 2024-04-26T01:45:11.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.468+0000 7f01ccc00700 1 -- 172.21.15.6:0/71331755 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f01d0062e80 con 0x7f01b40a3c40 2024-04-26T01:45:11.465 INFO:teuthology.orchestra.run.smithi006.stdout:HOST ADDR LABELS STATUS 2024-04-26T01:45:11.465 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 172.21.15.6 2024-04-26T01:45:11.465 INFO:teuthology.orchestra.run.smithi006.stdout:smithi029 172.21.15.29 2024-04-26T01:45:11.465 INFO:teuthology.orchestra.run.smithi006.stdout:smithi188 172.21.15.188 2024-04-26T01:45:11.465 INFO:teuthology.orchestra.run.smithi006.stdout:3 hosts in cluster 2024-04-26T01:45:11.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.471+0000 7f01d6400700 1 -- 172.21.15.6:0/71331755 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f01b40a3c40 msgr2=0x7f01b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:11.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.471+0000 7f01d6400700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f01b40a3c40 0x7f01b40a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f01c0011550 tx=0x7f01c0009260 comp rx=0 tx=0).stop 2024-04-26T01:45:11.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.471+0000 7f01d6400700 1 -- 172.21.15.6:0/71331755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d0078640 msgr2=0x7f01d019f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:11.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.471+0000 7f01d6400700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d0078640 0x7f01d019f700 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f01b8000c00 tx=0x7f01b80057a0 comp rx=0 tx=0).stop 2024-04-26T01:45:11.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.471+0000 7f01d6400700 1 -- 172.21.15.6:0/71331755 shutdown_connections 2024-04-26T01:45:11.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.471+0000 7f01d6400700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f01b40a3c40 0x7f01b40a6100 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:11.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.471+0000 7f01d6400700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01d0079000 0x7f01d01a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:11.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.471+0000 7f01d6400700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d00773f0 0x7f01d019f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:11.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.471+0000 7f01d6400700 1 --2- 172.21.15.6:0/71331755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01d0078640 0x7f01d019f700 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:11.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.471+0000 7f01d6400700 1 -- 172.21.15.6:0/71331755 >> 172.21.15.6:0/71331755 conn(0x7f01d0100290 msgr2=0x7f01d0101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:11.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.471+0000 7f01d6400700 1 -- 172.21.15.6:0/71331755 shutdown_connections 2024-04-26T01:45:11.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:11.471+0000 7f01d6400700 1 -- 172.21.15.6:0/71331755 wait complete. 2024-04-26T01:45:12.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-26T01:45:12.638 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:13.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:13 smithi006 ceph-mon[28419]: pgmap v427: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:13.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:13 smithi006 ceph-mon[28419]: from='client.24926 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:13.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:13 smithi188 ceph-mon[32751]: pgmap v427: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:13.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:13 smithi188 ceph-mon[32751]: from='client.24926 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:13.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:13 smithi029 ceph-mon[32775]: pgmap v427: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:13.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:13 smithi029 ceph-mon[32775]: from='client.24926 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:14.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.709+0000 7ff9e5200700 1 -- 172.21.15.6:0/2907278573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9e010ac80 msgr2=0x7ff9e0111530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:14.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.709+0000 7ff9e5200700 1 --2- 172.21.15.6:0/2907278573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9e010ac80 0x7ff9e0111530 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800b0d0 tx=0x7ff9d800b3e0 comp rx=0 tx=0).stop 2024-04-26T01:45:14.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.709+0000 7ff9e5200700 1 -- 172.21.15.6:0/2907278573 shutdown_connections 2024-04-26T01:45:14.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.709+0000 7ff9e5200700 1 --2- 172.21.15.6:0/2907278573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9e010ac80 0x7ff9e0111530 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:14.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.709+0000 7ff9e5200700 1 --2- 172.21.15.6:0/2907278573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9e01090e0 0x7ff9e0109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:14.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.709+0000 7ff9e5200700 1 --2- 172.21.15.6:0/2907278573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e010a2e0 0x7ff9e010a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:14.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.709+0000 7ff9e5200700 1 -- 172.21.15.6:0/2907278573 >> 172.21.15.6:0/2907278573 conn(0x7ff9e00780b0 msgr2=0x7ff9e007a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:14.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.709+0000 7ff9e5200700 1 -- 172.21.15.6:0/2907278573 shutdown_connections 2024-04-26T01:45:14.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.709+0000 7ff9e5200700 1 -- 172.21.15.6:0/2907278573 wait complete. 2024-04-26T01:45:14.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.710+0000 7ff9e5200700 1 Processor -- start 2024-04-26T01:45:14.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.711+0000 7ff9e5200700 1 -- start start 2024-04-26T01:45:14.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.711+0000 7ff9e5200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e01090e0 0x7ff9e019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:14.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.711+0000 7ff9e5200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9e010a2e0 0x7ff9e019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:14.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.711+0000 7ff9e5200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9e010ac80 0x7ff9e01a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:14.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.711+0000 7ff9e5200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e0114bb0 con 0x7ff9e010a2e0 2024-04-26T01:45:14.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.711+0000 7ff9e5200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e0114a30 con 0x7ff9e010ac80 2024-04-26T01:45:14.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.711+0000 7ff9e5200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e0114d30 con 0x7ff9e01090e0 2024-04-26T01:45:14.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.711+0000 7ff9dd400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9e010a2e0 0x7ff9e019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:14.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.711+0000 7ff9dde00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e01090e0 0x7ff9e019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:14.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.711+0000 7ff9dd400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9e010a2e0 0x7ff9e019f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40016/0 (socket says 172.21.15.6:40016) 2024-04-26T01:45:14.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.711+0000 7ff9dd400700 1 -- 172.21.15.6:0/609685472 learned_addr learned my addr 172.21.15.6:0/609685472 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:14.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.711+0000 7ff9de800700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9e010ac80 0x7ff9e01a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:14.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.712+0000 7ff9dde00700 1 -- 172.21.15.6:0/609685472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9e010ac80 msgr2=0x7ff9e01a6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:14.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.712+0000 7ff9dde00700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9e010ac80 0x7ff9e01a6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:14.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.712+0000 7ff9dde00700 1 -- 172.21.15.6:0/609685472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9e010a2e0 msgr2=0x7ff9e019f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:14.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.712+0000 7ff9dde00700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9e010a2e0 0x7ff9e019f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:14.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.712+0000 7ff9dde00700 1 -- 172.21.15.6:0/609685472 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e01a6da0 con 0x7ff9e01090e0 2024-04-26T01:45:14.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.712+0000 7ff9dd400700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9e010a2e0 0x7ff9e019f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:14.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.712+0000 7ff9dde00700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e01090e0 0x7ff9e019f140 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000d770 tx=0x7ff9d000da80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:14.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.713+0000 7ff9d6a00700 1 -- 172.21.15.6:0/609685472 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0016070 con 0x7ff9e01090e0 2024-04-26T01:45:14.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.713+0000 7ff9d6a00700 1 -- 172.21.15.6:0/609685472 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9d000f690 con 0x7ff9e01090e0 2024-04-26T01:45:14.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.713+0000 7ff9e5200700 1 -- 172.21.15.6:0/609685472 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9e01a7090 con 0x7ff9e01090e0 2024-04-26T01:45:14.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.713+0000 7ff9d6a00700 1 -- 172.21.15.6:0/609685472 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0003b50 con 0x7ff9e01090e0 2024-04-26T01:45:14.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.713+0000 7ff9e5200700 1 -- 172.21.15.6:0/609685472 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9e01a7620 con 0x7ff9e01090e0 2024-04-26T01:45:14.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.714+0000 7ff9e5200700 1 -- 172.21.15.6:0/609685472 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9e0069730 con 0x7ff9e01090e0 2024-04-26T01:45:14.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.716+0000 7ff9d6a00700 1 -- 172.21.15.6:0/609685472 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7ff9d000f840 con 0x7ff9e01090e0 2024-04-26T01:45:14.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.717+0000 7ff9d6a00700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff9c40a3cc0 0x7ff9c40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:14.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.718+0000 7ff9d6a00700 1 -- 172.21.15.6:0/609685472 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7ff9d00d0240 con 0x7ff9e01090e0 2024-04-26T01:45:14.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.718+0000 7ff9dd400700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff9c40a3cc0 0x7ff9c40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:14.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.718+0000 7ff9dd400700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff9c40a3cc0 0x7ff9c40a6180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc005950 tx=0x7ff9cc00b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:14.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.720+0000 7ff9d6a00700 1 -- 172.21.15.6:0/609685472 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff9d009cb80 con 0x7ff9e01090e0 2024-04-26T01:45:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.960+0000 7ff9e5200700 1 -- 172.21.15.6:0/609685472 --> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7ff9e0062e80 con 0x7ff9c40a3cc0 2024-04-26T01:45:14.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.964+0000 7ff9d6a00700 1 -- 172.21.15.6:0/609685472 <== mgr.14152 v2:172.21.15.6:6800/809793211 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7ff9e0062e80 con 0x7ff9c40a3cc0 2024-04-26T01:45:14.961 INFO:teuthology.orchestra.run.smithi006.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-26T01:45:14.962 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330000H400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-04-26T01:45:14.962 INFO:teuthology.orchestra.run.smithi006.stdout:smithi029 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009B480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-04-26T01:45:14.962 INFO:teuthology.orchestra.run.smithi006.stdout:smithi188 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500MP400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-26T01:45:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.967+0000 7ff9e5200700 1 -- 172.21.15.6:0/609685472 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff9c40a3cc0 msgr2=0x7ff9c40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.967+0000 7ff9e5200700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff9c40a3cc0 0x7ff9c40a6180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc005950 tx=0x7ff9cc00b500 comp rx=0 tx=0).stop 2024-04-26T01:45:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.967+0000 7ff9e5200700 1 -- 172.21.15.6:0/609685472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e01090e0 msgr2=0x7ff9e019f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.967+0000 7ff9e5200700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e01090e0 0x7ff9e019f140 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000d770 tx=0x7ff9d000da80 comp rx=0 tx=0).stop 2024-04-26T01:45:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.968+0000 7ff9e5200700 1 -- 172.21.15.6:0/609685472 shutdown_connections 2024-04-26T01:45:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.968+0000 7ff9e5200700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9e010ac80 0x7ff9e01a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.968+0000 7ff9e5200700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7ff9c40a3cc0 0x7ff9c40a6180 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.968+0000 7ff9e5200700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e01090e0 0x7ff9e019f140 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.968+0000 7ff9e5200700 1 --2- 172.21.15.6:0/609685472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9e010a2e0 0x7ff9e019f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.968+0000 7ff9e5200700 1 -- 172.21.15.6:0/609685472 >> 172.21.15.6:0/609685472 conn(0x7ff9e00780b0 msgr2=0x7ff9e010d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.968+0000 7ff9e5200700 1 -- 172.21.15.6:0/609685472 shutdown_connections 2024-04-26T01:45:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:14.968+0000 7ff9e5200700 1 -- 172.21.15.6:0/609685472 wait complete. 2024-04-26T01:45:15.277 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:15 smithi006 ceph-mon[28419]: pgmap v428: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:15.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:15 smithi188 ceph-mon[32751]: pgmap v428: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:15.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:15 smithi029 ceph-mon[32775]: pgmap v428: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:15.681 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T01:45:15.690 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi006.front.sepia.ceph.com 2024-04-26T01:45:15.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-26T01:45:16.163 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:16.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:16 smithi006 ceph-mon[28419]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:16.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:16 smithi188 ceph-mon[32751]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:16.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:16 smithi029 ceph-mon[32775]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:17.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:17 smithi006 ceph-mon[28419]: pgmap v429: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:45:17.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:17 smithi188 ceph-mon[32751]: pgmap v429: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:45:17.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:17 smithi029 ceph-mon[32775]: pgmap v429: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:45:18.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.366+0000 7fbc88a00700 1 -- 172.21.15.6:0/1902267393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc8410a2e0 msgr2=0x7fbc8410a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:18.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.366+0000 7fbc88a00700 1 --2- 172.21.15.6:0/1902267393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc8410a2e0 0x7fbc8410a740 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbc700098f0 tx=0x7fbc70009c00 comp rx=0 tx=0).stop 2024-04-26T01:45:18.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.366+0000 7fbc88a00700 1 -- 172.21.15.6:0/1902267393 shutdown_connections 2024-04-26T01:45:18.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.366+0000 7fbc88a00700 1 --2- 172.21.15.6:0/1902267393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc8410ac80 0x7fbc84111530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:18.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.366+0000 7fbc88a00700 1 --2- 172.21.15.6:0/1902267393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc841090e0 0x7fbc84109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:18.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.366+0000 7fbc88a00700 1 --2- 172.21.15.6:0/1902267393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc8410a2e0 0x7fbc8410a740 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:18.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.366+0000 7fbc88a00700 1 -- 172.21.15.6:0/1902267393 >> 172.21.15.6:0/1902267393 conn(0x7fbc840780b0 msgr2=0x7fbc8407a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:18.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.366+0000 7fbc88a00700 1 -- 172.21.15.6:0/1902267393 shutdown_connections 2024-04-26T01:45:18.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.366+0000 7fbc88a00700 1 -- 172.21.15.6:0/1902267393 wait complete. 2024-04-26T01:45:18.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.367+0000 7fbc88a00700 1 Processor -- start 2024-04-26T01:45:18.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.368+0000 7fbc88a00700 1 -- start start 2024-04-26T01:45:18.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.368+0000 7fbc88a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc841090e0 0x7fbc8419f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:18.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.368+0000 7fbc88a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc8410a2e0 0x7fbc8419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:18.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.368+0000 7fbc88a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8410ac80 0x7fbc841a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:18.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.368+0000 7fbc88a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbc84114b50 con 0x7fbc8410a2e0 2024-04-26T01:45:18.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.368+0000 7fbc88a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbc841149d0 con 0x7fbc8410ac80 2024-04-26T01:45:18.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.368+0000 7fbc88a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbc84114cd0 con 0x7fbc841090e0 2024-04-26T01:45:18.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.368+0000 7fbc81e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc841090e0 0x7fbc8419f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:18.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.369+0000 7fbc81400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc8410a2e0 0x7fbc8419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:18.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.369+0000 7fbc81e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc841090e0 0x7fbc8419f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34140/0 (socket says 172.21.15.6:34140) 2024-04-26T01:45:18.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.369+0000 7fbc82800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8410ac80 0x7fbc841a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:18.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.369+0000 7fbc81e00700 1 -- 172.21.15.6:0/262110967 learned_addr learned my addr 172.21.15.6:0/262110967 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:18.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.369+0000 7fbc81e00700 1 -- 172.21.15.6:0/262110967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8410ac80 msgr2=0x7fbc841a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:18.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.369+0000 7fbc81e00700 1 --2- 172.21.15.6:0/262110967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8410ac80 0x7fbc841a67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:18.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.369+0000 7fbc81e00700 1 -- 172.21.15.6:0/262110967 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc8410a2e0 msgr2=0x7fbc8419f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:18.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.369+0000 7fbc81e00700 1 --2- 172.21.15.6:0/262110967 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc8410a2e0 0x7fbc8419f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:18.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.369+0000 7fbc81e00700 1 -- 172.21.15.6:0/262110967 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc841a6e80 con 0x7fbc841090e0 2024-04-26T01:45:18.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.369+0000 7fbc81e00700 1 --2- 172.21.15.6:0/262110967 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc841090e0 0x7fbc8419f220 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400e9a0 tx=0x7fbc7400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:18.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.370+0000 7fbc7aa00700 1 -- 172.21.15.6:0/262110967 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74004220 con 0x7fbc841090e0 2024-04-26T01:45:18.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.370+0000 7fbc88a00700 1 -- 172.21.15.6:0/262110967 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc841a7170 con 0x7fbc841090e0 2024-04-26T01:45:18.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.370+0000 7fbc7aa00700 1 -- 172.21.15.6:0/262110967 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc740043c0 con 0x7fbc841090e0 2024-04-26T01:45:18.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.370+0000 7fbc7aa00700 1 -- 172.21.15.6:0/262110967 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74003cc0 con 0x7fbc841090e0 2024-04-26T01:45:18.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.370+0000 7fbc88a00700 1 -- 172.21.15.6:0/262110967 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc841a7700 con 0x7fbc841090e0 2024-04-26T01:45:18.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.371+0000 7fbc88a00700 1 -- 172.21.15.6:0/262110967 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc84069730 con 0x7fbc841090e0 2024-04-26T01:45:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.373+0000 7fbc7aa00700 1 -- 172.21.15.6:0/262110967 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fbc7400c810 con 0x7fbc841090e0 2024-04-26T01:45:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.375+0000 7fbc7aa00700 1 --2- 172.21.15.6:0/262110967 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fbc680a3cc0 0x7fbc680a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.375+0000 7fbc7aa00700 1 -- 172.21.15.6:0/262110967 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fbc740d06a0 con 0x7fbc841090e0 2024-04-26T01:45:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.375+0000 7fbc81400700 1 --2- 172.21.15.6:0/262110967 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fbc680a3cc0 0x7fbc680a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.376+0000 7fbc81400700 1 --2- 172.21.15.6:0/262110967 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fbc680a3cc0 0x7fbc680a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000c010 tx=0x7fbc70019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.377+0000 7fbc7aa00700 1 -- 172.21.15.6:0/262110967 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc7409d070 con 0x7fbc841090e0 2024-04-26T01:45:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.659+0000 7fbc88a00700 1 -- 172.21.15.6:0/262110967 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fbc8402e040 con 0x7fbc841090e0 2024-04-26T01:45:18.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.660+0000 7fbc7aa00700 1 -- 172.21.15.6:0/262110967 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) ==== 76+0+3037 (secure 0 0 0) 0x7fbc7409c970 con 0x7fbc841090e0 2024-04-26T01:45:18.658 INFO:teuthology.orchestra.run.smithi006.stdout:e25 2024-04-26T01:45:18.658 INFO:teuthology.orchestra.run.smithi006.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:legacy client fscid: 1 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:Filesystem 'cephfs' (1) 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:fs_name cephfs 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 25 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-04-26T01:42:08.299563+0000 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:modified 2024-04-26T01:42:31.294242+0000 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:tableserver 0 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:root 0 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:session_timeout 300 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:session_autoclose 300 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:max_file_size 1099511627776 2024-04-26T01:45:18.659 INFO:teuthology.orchestra.run.smithi006.stdout:max_xattr_size 65536 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:required_client_features {} 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:last_failure 0 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:last_failure_osd_epoch 0 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:max_mds 3 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:in 0,1,2 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:up {0=24523,1=24512,2=24505} 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:failed 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:damaged 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:stopped 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:data_pools [3] 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:metadata_pool 2 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:inline_data disabled 2024-04-26T01:45:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:balancer 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:bal_rank_mask -1 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:standby_count_wanted 0 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:qdb_cluster leader: 24523 members: 24523,24505,24512 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.i{0:24523} state up:active seq 10 addr [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.f{0:24499} state up:standby-replay seq 1 addr [v2:172.21.15.188:6834/1255666766,v1:172.21.15.188:6835/1255666766] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b{1:24512} state up:active seq 10 addr [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.g{1:24494} state up:standby-replay seq 1 addr [v2:172.21.15.6:6838/3936238112,v1:172.21.15.6:6839/3936238112] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c{2:24505} state up:active seq 15 addr [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.h{2:24479} state up:standby-replay seq 1 addr [v2:172.21.15.29:6836/402171153,v1:172.21.15.29:6837/402171153] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:Standby daemons: 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.j{-1:14361} state up:standby seq 1 addr [v2:172.21.15.6:6834/3962248183,v1:172.21.15.6:6835/3962248183] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:45:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a{-1:14409} state up:standby seq 1 addr [v2:172.21.15.6:6840/1062358442,v1:172.21.15.6:6841/1062358442] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:45:18.662 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.k{-1:24428} state up:standby seq 1 addr [v2:172.21.15.29:6832/3500196993,v1:172.21.15.29:6833/3500196993] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:45:18.662 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.l{-1:24451} state up:standby seq 1 addr [v2:172.21.15.188:6832/3510817210,v1:172.21.15.188:6833/3510817210] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:45:18.662 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.e{-1:24458} state up:standby seq 1 addr [v2:172.21.15.29:6834/2373773979,v1:172.21.15.29:6835/2373773979] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:45:18.662 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d{-1:24478} state up:standby seq 1 addr [v2:172.21.15.6:6836/680270961,v1:172.21.15.6:6837/680270961] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:45:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.665+0000 7fbc88a00700 1 -- 172.21.15.6:0/262110967 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fbc680a3cc0 msgr2=0x7fbc680a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.665+0000 7fbc88a00700 1 --2- 172.21.15.6:0/262110967 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fbc680a3cc0 0x7fbc680a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000c010 tx=0x7fbc70019040 comp rx=0 tx=0).stop 2024-04-26T01:45:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.665+0000 7fbc88a00700 1 -- 172.21.15.6:0/262110967 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc841090e0 msgr2=0x7fbc8419f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.665+0000 7fbc88a00700 1 --2- 172.21.15.6:0/262110967 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc841090e0 0x7fbc8419f220 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400e9a0 tx=0x7fbc7400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:45:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.665+0000 7fbc88a00700 1 -- 172.21.15.6:0/262110967 shutdown_connections 2024-04-26T01:45:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.665+0000 7fbc88a00700 1 --2- 172.21.15.6:0/262110967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8410ac80 0x7fbc841a67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.665+0000 7fbc88a00700 1 --2- 172.21.15.6:0/262110967 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fbc680a3cc0 0x7fbc680a6180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.666+0000 7fbc88a00700 1 --2- 172.21.15.6:0/262110967 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc841090e0 0x7fbc8419f220 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.666+0000 7fbc88a00700 1 --2- 172.21.15.6:0/262110967 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc8410a2e0 0x7fbc8419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.666+0000 7fbc88a00700 1 -- 172.21.15.6:0/262110967 >> 172.21.15.6:0/262110967 conn(0x7fbc840780b0 msgr2=0x7fbc8410d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.666+0000 7fbc88a00700 1 -- 172.21.15.6:0/262110967 shutdown_connections 2024-04-26T01:45:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:18.666+0000 7fbc88a00700 1 -- 172.21.15.6:0/262110967 wait complete. 2024-04-26T01:45:18.664 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-26T01:45:19.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:19 smithi006 ceph-mon[28419]: pgmap v430: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:19.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:19 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/262110967' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T01:45:19.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:19 smithi188 ceph-mon[32751]: pgmap v430: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:19.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:19 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/262110967' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T01:45:19.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-26T01:45:19.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:19 smithi029 ceph-mon[32775]: pgmap v430: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:19.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:19 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/262110967' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T01:45:19.907 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:21.288 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:21 smithi006 ceph-mon[28419]: pgmap v431: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:21.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:21 smithi188 ceph-mon[32751]: pgmap v431: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:21.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:21 smithi029 ceph-mon[32775]: pgmap v431: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:22.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.087+0000 7fe668600700 1 -- 172.21.15.6:0/233966758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe660103cb0 msgr2=0x7fe660104110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:22.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.087+0000 7fe668600700 1 --2- 172.21.15.6:0/233966758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe660103cb0 0x7fe660104110 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0099c0 tx=0x7fe64c009cd0 comp rx=0 tx=0).stop 2024-04-26T01:45:22.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.087+0000 7fe668600700 1 -- 172.21.15.6:0/233966758 shutdown_connections 2024-04-26T01:45:22.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.087+0000 7fe668600700 1 --2- 172.21.15.6:0/233966758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe660104650 0x7fe66010af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:22.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.087+0000 7fe668600700 1 --2- 172.21.15.6:0/233966758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660102ab0 0x7fe660102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:22.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.087+0000 7fe668600700 1 --2- 172.21.15.6:0/233966758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe660103cb0 0x7fe660104110 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:22.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.087+0000 7fe668600700 1 -- 172.21.15.6:0/233966758 >> 172.21.15.6:0/233966758 conn(0x7fe6600fe010 msgr2=0x7fe660100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:22.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.087+0000 7fe668600700 1 -- 172.21.15.6:0/233966758 shutdown_connections 2024-04-26T01:45:22.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.088+0000 7fe668600700 1 -- 172.21.15.6:0/233966758 wait complete. 2024-04-26T01:45:22.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.089+0000 7fe668600700 1 Processor -- start 2024-04-26T01:45:22.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.089+0000 7fe668600700 1 -- start start 2024-04-26T01:45:22.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe668600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660102ab0 0x7fe66019ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:22.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe668600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe660103cb0 0x7fe66019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:22.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe668600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe660104650 0x7fe6601a23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:22.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe668600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe66010e570 con 0x7fe660103cb0 2024-04-26T01:45:22.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe668600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe66010e3f0 con 0x7fe660104650 2024-04-26T01:45:22.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe668600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe66010e6f0 con 0x7fe660102ab0 2024-04-26T01:45:22.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe665c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660102ab0 0x7fe66019ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:22.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe665c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660102ab0 0x7fe66019ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34160/0 (socket says 172.21.15.6:34160) 2024-04-26T01:45:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe665c00700 1 -- 172.21.15.6:0/1682533603 learned_addr learned my addr 172.21.15.6:0/1682533603 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe666600700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe660104650 0x7fe6601a23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe665200700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe660103cb0 0x7fe66019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe665c00700 1 -- 172.21.15.6:0/1682533603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe660104650 msgr2=0x7fe6601a23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe665c00700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe660104650 0x7fe6601a23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe665c00700 1 -- 172.21.15.6:0/1682533603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe660103cb0 msgr2=0x7fe66019b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe665c00700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe660103cb0 0x7fe66019b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe665c00700 1 -- 172.21.15.6:0/1682533603 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6601a2a80 con 0x7fe660102ab0 2024-04-26T01:45:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.090+0000 7fe665200700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe660103cb0 0x7fe66019b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.091+0000 7fe665c00700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660102ab0 0x7fe66019ae20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe65400ea80 tx=0x7fe65400ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.091+0000 7fe65e000700 1 -- 172.21.15.6:0/1682533603 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65400cbd0 con 0x7fe660102ab0 2024-04-26T01:45:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.091+0000 7fe65e000700 1 -- 172.21.15.6:0/1682533603 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe65400cd70 con 0x7fe660102ab0 2024-04-26T01:45:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.091+0000 7fe65e000700 1 -- 172.21.15.6:0/1682533603 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe654018750 con 0x7fe660102ab0 2024-04-26T01:45:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.091+0000 7fe668600700 1 -- 172.21.15.6:0/1682533603 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6601a2d70 con 0x7fe660102ab0 2024-04-26T01:45:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.091+0000 7fe668600700 1 -- 172.21.15.6:0/1682533603 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6601a31f0 con 0x7fe660102ab0 2024-04-26T01:45:22.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.093+0000 7fe668600700 1 -- 172.21.15.6:0/1682533603 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe660069760 con 0x7fe660102ab0 2024-04-26T01:45:22.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.094+0000 7fe65e000700 1 -- 172.21.15.6:0/1682533603 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fe65401f020 con 0x7fe660102ab0 2024-04-26T01:45:22.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.096+0000 7fe65e000700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe6480a3c40 0x7fe6480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:22.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.096+0000 7fe65e000700 1 -- 172.21.15.6:0/1682533603 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fe6540d05f0 con 0x7fe660102ab0 2024-04-26T01:45:22.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.096+0000 7fe665200700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe6480a3c40 0x7fe6480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:22.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.097+0000 7fe665200700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe6480a3c40 0x7fe6480a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe64c000c00 tx=0x7fe64c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.104+0000 7fe65e000700 1 -- 172.21.15.6:0/1682533603 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe654053ff0 con 0x7fe660102ab0 2024-04-26T01:45:22.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:22.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:22 smithi006 ceph-mon[28419]: pgmap v432: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:22.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:22.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:22.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:22.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:22 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:22.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.344+0000 7fe668600700 1 -- 172.21.15.6:0/1682533603 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fe6601a3c60 con 0x7fe660102ab0 2024-04-26T01:45:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.345+0000 7fe65e000700 1 -- 172.21.15.6:0/1682533603 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v95) ==== 56+0+5316 (secure 0 0 0) 0x7fe660069760 con 0x7fe660102ab0 2024-04-26T01:45:22.342 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 95 2024-04-26T01:45:22.342 INFO:teuthology.orchestra.run.smithi006.stdout:fsid a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T01:45:22.342 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-04-26T01:32:29.071544+0000 2024-04-26T01:45:22.342 INFO:teuthology.orchestra.run.smithi006.stdout:modified 2024-04-26T01:43:49.972785+0000 2024-04-26T01:45:22.342 INFO:teuthology.orchestra.run.smithi006.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:crush_version 26 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:full_ratio 0.95 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:backfillfull_ratio 0.9 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:nearfull_ratio 0.85 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:require_min_compat_client luminous 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:min_compat_client luminous 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:require_osd_release squid 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:stretch_mode_enabled false 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 91 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 91 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:max_osd 12 2024-04-26T01:45:22.343 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0 up in weight 1 up_from 8 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6802/76518486,v1:172.21.15.6:6803/76518486] [v2:172.21.15.6:6804/76518486,v1:172.21.15.6:6805/76518486] exists,up f164c51d-ce5d-4140-b9e3-a4b84088f5ae 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 up in weight 1 up_from 13 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6800/400769116,v1:172.21.15.29:6801/400769116] [v2:172.21.15.29:6802/400769116,v1:172.21.15.29:6803/400769116] exists,up 57e875dd-ddf1-4ee4-a2a7-9589d7f42818 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2 up in weight 1 up_from 18 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6800/3508067672,v1:172.21.15.188:6801/3508067672] [v2:172.21.15.188:6802/3508067672,v1:172.21.15.188:6803/3508067672] exists,up 77e0c294-c48a-4318-938b-4604a533a2d6 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3 up in weight 1 up_from 26 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6810/1987311118,v1:172.21.15.6:6811/1987311118] [v2:172.21.15.6:6812/1987311118,v1:172.21.15.6:6813/1987311118] exists,up 6393dac5-996c-47af-9687-0f1393ecc483 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4 up in weight 1 up_from 34 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6808/2125876609,v1:172.21.15.29:6809/2125876609] [v2:172.21.15.29:6810/2125876609,v1:172.21.15.29:6811/2125876609] exists,up f2c1a535-26a0-46c6-a6d0-c2529c786739 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5 up in weight 1 up_from 40 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6808/2092077007,v1:172.21.15.188:6809/2092077007] [v2:172.21.15.188:6810/2092077007,v1:172.21.15.188:6811/2092077007] exists,up 9788bb0a-0d79-4743-bce6-80f5342e8f4a 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6 up in weight 1 up_from 48 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6818/3256016695,v1:172.21.15.6:6819/3256016695] [v2:172.21.15.6:6820/3256016695,v1:172.21.15.6:6821/3256016695] exists,up fd8584e1-69ee-4c30-85e4-a637357d1b6d 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7 up in weight 1 up_from 53 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6816/3478594037,v1:172.21.15.29:6817/3478594037] [v2:172.21.15.29:6818/3478594037,v1:172.21.15.29:6819/3478594037] exists,up 6b7e6784-d574-42da-8416-a74a3b6b3755 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8 up in weight 1 up_from 62 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6816/3290364376,v1:172.21.15.188:6817/3290364376] [v2:172.21.15.188:6818/3290364376,v1:172.21.15.188:6819/3290364376] exists,up 30cd2ea1-2767-4062-8bd9-47214e4b4b9e 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9 up in weight 1 up_from 70 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6826/1918893912,v1:172.21.15.6:6827/1918893912] [v2:172.21.15.6:6828/1918893912,v1:172.21.15.6:6829/1918893912] exists,up f033cdc0-1178-4a7f-a338-69ff32416087 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10 up in weight 1 up_from 75 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6824/1395777729,v1:172.21.15.29:6825/1395777729] [v2:172.21.15.29:6826/1395777729,v1:172.21.15.29:6827/1395777729] exists,up 04e9b5ef-ac0c-482b-a69e-a7121b36e7da 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11 up in weight 1 up_from 83 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6824/3322197880,v1:172.21.15.188:6825/3322197880] [v2:172.21.15.188:6826/3322197880,v1:172.21.15.188:6827/3322197880] exists,up cb1638ce-4189-4241-b63e-b63c607e90ff 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 2.b [2,1] 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 2.e [2,4] 2024-04-26T01:45:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 2.10 [8,11] 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 2.1c [8,11] 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 3.10 [5,9] 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 3.14 [4,6] 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 3.17 [4,7] 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 3.19 [3,8] 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 3.1d [5,9] 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 3.34 [4,9] 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 3.3a [4,8] 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/3143272062 expires 2024-04-27T01:33:47.021882+0000 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/1325707596 expires 2024-04-27T01:33:47.021882+0000 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:6800/635066163 expires 2024-04-27T01:33:47.021882+0000 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:6801/635066163 expires 2024-04-27T01:33:47.021882+0000 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/1359518071 expires 2024-04-27T01:33:11.534520+0000 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:6801/801052348 expires 2024-04-27T01:33:11.534520+0000 2024-04-26T01:45:22.345 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/2813598711 expires 2024-04-27T01:33:11.534520+0000 2024-04-26T01:45:22.346 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/3140150861 expires 2024-04-27T01:33:11.534520+0000 2024-04-26T01:45:22.346 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:6800/801052348 expires 2024-04-27T01:33:11.534520+0000 2024-04-26T01:45:22.346 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/65701414 expires 2024-04-27T01:33:47.021882+0000 2024-04-26T01:45:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.348+0000 7fe668600700 1 -- 172.21.15.6:0/1682533603 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe6480a3c40 msgr2=0x7fe6480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.349+0000 7fe668600700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe6480a3c40 0x7fe6480a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe64c000c00 tx=0x7fe64c005fd0 comp rx=0 tx=0).stop 2024-04-26T01:45:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.349+0000 7fe668600700 1 -- 172.21.15.6:0/1682533603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660102ab0 msgr2=0x7fe66019ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.349+0000 7fe668600700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660102ab0 0x7fe66019ae20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe65400ea80 tx=0x7fe65400ed90 comp rx=0 tx=0).stop 2024-04-26T01:45:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.349+0000 7fe668600700 1 -- 172.21.15.6:0/1682533603 shutdown_connections 2024-04-26T01:45:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.349+0000 7fe668600700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe660104650 0x7fe6601a23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.349+0000 7fe668600700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fe6480a3c40 0x7fe6480a6100 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.349+0000 7fe668600700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660102ab0 0x7fe66019ae20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.349+0000 7fe668600700 1 --2- 172.21.15.6:0/1682533603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe660103cb0 0x7fe66019b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.349+0000 7fe668600700 1 -- 172.21.15.6:0/1682533603 >> 172.21.15.6:0/1682533603 conn(0x7fe6600fe010 msgr2=0x7fe660109480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.349+0000 7fe668600700 1 -- 172.21.15.6:0/1682533603 shutdown_connections 2024-04-26T01:45:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:22.349+0000 7fe668600700 1 -- 172.21.15.6:0/1682533603 wait complete. 2024-04-26T01:45:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:22 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:22 smithi188 ceph-mon[32751]: pgmap v432: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:22 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:22 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:22 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:22.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:22 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:22.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:22 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:22.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:22 smithi029 ceph-mon[32775]: pgmap v432: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:22.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:22 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:22.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:22 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:22.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:22 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:22.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:22 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:23.150 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-26T01:45:23.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:45:23.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:23 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1682533603' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T01:45:23.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:23 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:45:23.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:23 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:45:23.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:23 smithi006 ceph-mon[28419]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:23.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:23 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1682533603' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T01:45:23.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:23 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:45:23.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:23 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:45:23.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:23 smithi188 ceph-mon[32751]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:23.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:23 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1682533603' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T01:45:23.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:23 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:45:23.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:23 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:45:23.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:23 smithi029 ceph-mon[32775]: from='mgr.14152 172.21.15.6:0/3503229404' entity='mgr.x' 2024-04-26T01:45:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:24.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:24 smithi188 ceph-mon[32751]: pgmap v433: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:24.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:24 smithi029 ceph-mon[32775]: pgmap v433: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:24 smithi006 ceph-mon[28419]: pgmap v433: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.763+0000 7f399f200700 1 -- 172.21.15.6:0/278788875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39a0103b80 msgr2=0x7f39a0103f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.763+0000 7f399f200700 1 --2- 172.21.15.6:0/278788875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39a0103b80 0x7f39a0103f60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3990009990 tx=0x7f3990009ca0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.764+0000 7f399f200700 1 -- 172.21.15.6:0/278788875 shutdown_connections 2024-04-26T01:45:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.764+0000 7f399f200700 1 --2- 172.21.15.6:0/278788875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39a010cfb0 0x7f39a010f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.764+0000 7f399f200700 1 --2- 172.21.15.6:0/278788875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39a0103b80 0x7f39a0103f60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.764+0000 7f399f200700 1 --2- 172.21.15.6:0/278788875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a01044a0 0x7f39a010c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.764+0000 7f399f200700 1 -- 172.21.15.6:0/278788875 >> 172.21.15.6:0/278788875 conn(0x7f39a00fd3d0 msgr2=0x7f39a00ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.764+0000 7f399f200700 1 -- 172.21.15.6:0/278788875 shutdown_connections 2024-04-26T01:45:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.764+0000 7f399f200700 1 -- 172.21.15.6:0/278788875 wait complete. 2024-04-26T01:45:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.765+0000 7f399f200700 1 Processor -- start 2024-04-26T01:45:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.765+0000 7f399f200700 1 -- start start 2024-04-26T01:45:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.765+0000 7f399f200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a0103b80 0x7f39a019b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.765+0000 7f399f200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39a01044a0 0x7f39a019b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399f200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39a010cfb0 0x7f39a019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399f200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f39a0112bc0 con 0x7f39a01044a0 2024-04-26T01:45:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399f200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f39a0112a40 con 0x7f39a0103b80 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399f200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f39a0112d40 con 0x7f39a010cfb0 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399d400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39a01044a0 0x7f39a019b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399d400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39a01044a0 0x7f39a019b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41738/0 (socket says 172.21.15.6:41738) 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399de00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a0103b80 0x7f39a019b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399d400700 1 -- 172.21.15.6:0/874256271 learned_addr learned my addr 172.21.15.6:0/874256271 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399e800700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39a010cfb0 0x7f39a019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399de00700 1 -- 172.21.15.6:0/874256271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39a010cfb0 msgr2=0x7f39a019fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399de00700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39a010cfb0 0x7f39a019fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399de00700 1 -- 172.21.15.6:0/874256271 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39a01044a0 msgr2=0x7f39a019b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399de00700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39a01044a0 0x7f39a019b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.766+0000 7f399de00700 1 -- 172.21.15.6:0/874256271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a01a0360 con 0x7f39a0103b80 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.767+0000 7f399de00700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a0103b80 0x7f39a019b350 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3990006010 tx=0x7f39900057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.767+0000 7f3996a00700 1 -- 172.21.15.6:0/874256271 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3990027070 con 0x7f39a0103b80 2024-04-26T01:45:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.767+0000 7f3996a00700 1 -- 172.21.15.6:0/874256271 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3990022470 con 0x7f39a0103b80 2024-04-26T01:45:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.767+0000 7f399f200700 1 -- 172.21.15.6:0/874256271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a01a05f0 con 0x7f39a0103b80 2024-04-26T01:45:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.767+0000 7f399f200700 1 -- 172.21.15.6:0/874256271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a01a7ed0 con 0x7f39a0103b80 2024-04-26T01:45:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.767+0000 7f3996a00700 1 -- 172.21.15.6:0/874256271 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3990017650 con 0x7f39a0103b80 2024-04-26T01:45:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.768+0000 7f399f200700 1 -- 172.21.15.6:0/874256271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39a0069730 con 0x7f39a0103b80 2024-04-26T01:45:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.770+0000 7f3996a00700 1 -- 172.21.15.6:0/874256271 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f399002a070 con 0x7f39a0103b80 2024-04-26T01:45:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.772+0000 7f3996a00700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f39840a3c40 0x7f39840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.772+0000 7f3996a00700 1 -- 172.21.15.6:0/874256271 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f39900d10d0 con 0x7f39a0103b80 2024-04-26T01:45:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.772+0000 7f399d400700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f39840a3c40 0x7f39840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.773+0000 7f399d400700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f39840a3c40 0x7f39840a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f39a010e810 tx=0x7f398c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:25.774+0000 7f3996a00700 1 -- 172.21.15.6:0/874256271 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3990013070 con 0x7f39a0103b80 2024-04-26T01:45:26.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.058+0000 7f399f200700 1 -- 172.21.15.6:0/874256271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39a01a80a0 con 0x7f39a0103b80 2024-04-26T01:45:26.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.059+0000 7f3996a00700 1 -- 172.21.15.6:0/874256271 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11850 (secure 0 0 0) 0x7f399009dbc0 con 0x7f39a0103b80 2024-04-26T01:45:26.056 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:45:26.056 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:42:31.294242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:45:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.062+0000 7f399f200700 1 -- 172.21.15.6:0/874256271 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f39840a3c40 msgr2=0x7f39840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.062+0000 7f399f200700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f39840a3c40 0x7f39840a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f39a010e810 tx=0x7f398c009260 comp rx=0 tx=0).stop 2024-04-26T01:45:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.062+0000 7f399f200700 1 -- 172.21.15.6:0/874256271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a0103b80 msgr2=0x7f39a019b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.062+0000 7f399f200700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a0103b80 0x7f39a019b350 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3990006010 tx=0x7f39900057c0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.062+0000 7f399f200700 1 -- 172.21.15.6:0/874256271 shutdown_connections 2024-04-26T01:45:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.062+0000 7f399f200700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f39840a3c40 0x7f39840a6100 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.062+0000 7f399f200700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39a010cfb0 0x7f39a019fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.062+0000 7f399f200700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a0103b80 0x7f39a019b350 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.062+0000 7f399f200700 1 --2- 172.21.15.6:0/874256271 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39a01044a0 0x7f39a019b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.062+0000 7f399f200700 1 -- 172.21.15.6:0/874256271 >> 172.21.15.6:0/874256271 conn(0x7f39a00fd3d0 msgr2=0x7f39a00ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.062+0000 7f399f200700 1 -- 172.21.15.6:0/874256271 shutdown_connections 2024-04-26T01:45:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:26.062+0000 7f399f200700 1 -- 172.21.15.6:0/874256271 wait complete. 2024-04-26T01:45:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-26T01:45:26.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:26 smithi006 ceph-mon[28419]: pgmap v434: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:45:26.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:26 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/874256271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:26.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:26 smithi188 ceph-mon[32751]: pgmap v434: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:45:26.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:26 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/874256271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:26 smithi029 ceph-mon[32775]: pgmap v434: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:45:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:26 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/874256271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:26.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:45:27.258 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:28.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:28 smithi188 ceph-mon[32751]: pgmap v435: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:28.669 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:28 smithi006 ceph-mon[28419]: pgmap v435: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:28.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:28 smithi029 ceph-mon[32775]: pgmap v435: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:29.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.478+0000 7fc8c9400700 1 -- 172.21.15.6:0/712334768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8c4102d60 msgr2=0x7fc8c41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.478+0000 7fc8c9400700 1 --2- 172.21.15.6:0/712334768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8c4102d60 0x7fc8c41031e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00098f0 tx=0x7fc8b0009c00 comp rx=0 tx=0).stop 2024-04-26T01:45:29.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.478+0000 7fc8c9400700 1 -- 172.21.15.6:0/712334768 shutdown_connections 2024-04-26T01:45:29.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.478+0000 7fc8c9400700 1 --2- 172.21.15.6:0/712334768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c4103720 0x7fc8c410f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.478+0000 7fc8c9400700 1 --2- 172.21.15.6:0/712334768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8c4108bc0 0x7fc8c4108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.478+0000 7fc8c9400700 1 --2- 172.21.15.6:0/712334768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8c4102d60 0x7fc8c41031e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.478+0000 7fc8c9400700 1 -- 172.21.15.6:0/712334768 >> 172.21.15.6:0/712334768 conn(0x7fc8c407a9d0 msgr2=0x7fc8c407ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:29.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.478+0000 7fc8c9400700 1 -- 172.21.15.6:0/712334768 shutdown_connections 2024-04-26T01:45:29.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.478+0000 7fc8c9400700 1 -- 172.21.15.6:0/712334768 wait complete. 2024-04-26T01:45:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.480+0000 7fc8c9400700 1 Processor -- start 2024-04-26T01:45:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.480+0000 7fc8c9400700 1 -- start start 2024-04-26T01:45:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.480+0000 7fc8c9400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8c4102d60 0x7fc8c41992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.480+0000 7fc8c9400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8c4103720 0x7fc8c4199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.480+0000 7fc8c9400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c4108bc0 0x7fc8c419db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.480+0000 7fc8c9400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c4112d10 con 0x7fc8c4108bc0 2024-04-26T01:45:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.480+0000 7fc8c9400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c4112b90 con 0x7fc8c4102d60 2024-04-26T01:45:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.480+0000 7fc8c9400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c4112e90 con 0x7fc8c4103720 2024-04-26T01:45:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.481+0000 7fc8c1e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8c4103720 0x7fc8c4199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.481+0000 7fc8c1e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8c4103720 0x7fc8c4199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47704/0 (socket says 172.21.15.6:47704) 2024-04-26T01:45:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.481+0000 7fc8c1e00700 1 -- 172.21.15.6:0/847105010 learned_addr learned my addr 172.21.15.6:0/847105010 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.481+0000 7fc8c2800700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8c4102d60 0x7fc8c41992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.481+0000 7fc8c3200700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c4108bc0 0x7fc8c419db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.481+0000 7fc8c1e00700 1 -- 172.21.15.6:0/847105010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8c4102d60 msgr2=0x7fc8c41992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.481+0000 7fc8c1e00700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8c4102d60 0x7fc8c41992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.481+0000 7fc8c1e00700 1 -- 172.21.15.6:0/847105010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c4108bc0 msgr2=0x7fc8c419db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.481+0000 7fc8c1e00700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c4108bc0 0x7fc8c419db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.481+0000 7fc8c1e00700 1 -- 172.21.15.6:0/847105010 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c419e2d0 con 0x7fc8c4103720 2024-04-26T01:45:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.481+0000 7fc8c1e00700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8c4103720 0x7fc8c4199800 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0009230 tx=0x7fc8b000fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.482+0000 7fc8bb400700 1 -- 172.21.15.6:0/847105010 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b00050c0 con 0x7fc8c4103720 2024-04-26T01:45:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.482+0000 7fc8bb400700 1 -- 172.21.15.6:0/847105010 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8b000b560 con 0x7fc8c4103720 2024-04-26T01:45:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.482+0000 7fc8c9400700 1 -- 172.21.15.6:0/847105010 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8c419e560 con 0x7fc8c4103720 2024-04-26T01:45:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.482+0000 7fc8bb400700 1 -- 172.21.15.6:0/847105010 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0003a00 con 0x7fc8c4103720 2024-04-26T01:45:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.482+0000 7fc8c9400700 1 -- 172.21.15.6:0/847105010 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8c41a5e40 con 0x7fc8c4103720 2024-04-26T01:45:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.483+0000 7fc8c9400700 1 -- 172.21.15.6:0/847105010 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8c4069730 con 0x7fc8c4103720 2024-04-26T01:45:29.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.485+0000 7fc8bb400700 1 -- 172.21.15.6:0/847105010 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fc8b000b710 con 0x7fc8c4103720 2024-04-26T01:45:29.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.486+0000 7fc8bb400700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc8a80a3cc0 0x7fc8a80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.486+0000 7fc8bb400700 1 -- 172.21.15.6:0/847105010 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fc8b00d12e0 con 0x7fc8c4103720 2024-04-26T01:45:29.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.486+0000 7fc8c2800700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc8a80a3cc0 0x7fc8a80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.487+0000 7fc8c2800700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc8a80a3cc0 0x7fc8a80a6180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4103e80 tx=0x7fc8b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:29.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.490+0000 7fc8bb400700 1 -- 172.21.15.6:0/847105010 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc8b0013030 con 0x7fc8c4103720 2024-04-26T01:45:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.772+0000 7fc8c9400700 1 -- 172.21.15.6:0/847105010 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8c419a620 con 0x7fc8c4103720 2024-04-26T01:45:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.773+0000 7fc8bb400700 1 -- 172.21.15.6:0/847105010 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11850 (secure 0 0 0) 0x7fc8b0021380 con 0x7fc8c4103720 2024-04-26T01:45:29.770 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:45:29.770 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:42:31.294242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:45:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.776+0000 7fc8c9400700 1 -- 172.21.15.6:0/847105010 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc8a80a3cc0 msgr2=0x7fc8a80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.776+0000 7fc8c9400700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc8a80a3cc0 0x7fc8a80a6180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4103e80 tx=0x7fc8b4009260 comp rx=0 tx=0).stop 2024-04-26T01:45:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.776+0000 7fc8c9400700 1 -- 172.21.15.6:0/847105010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8c4103720 msgr2=0x7fc8c4199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.776+0000 7fc8c9400700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8c4103720 0x7fc8c4199800 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0009230 tx=0x7fc8b000fa70 comp rx=0 tx=0).stop 2024-04-26T01:45:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.776+0000 7fc8c9400700 1 -- 172.21.15.6:0/847105010 shutdown_connections 2024-04-26T01:45:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.776+0000 7fc8c9400700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c4108bc0 0x7fc8c419db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.776+0000 7fc8c9400700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8c4102d60 0x7fc8c41992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.776+0000 7fc8c9400700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fc8a80a3cc0 0x7fc8a80a6180 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.776+0000 7fc8c9400700 1 --2- 172.21.15.6:0/847105010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8c4103720 0x7fc8c4199800 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.776+0000 7fc8c9400700 1 -- 172.21.15.6:0/847105010 >> 172.21.15.6:0/847105010 conn(0x7fc8c407a9d0 msgr2=0x7fc8c4101b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.777+0000 7fc8c9400700 1 -- 172.21.15.6:0/847105010 shutdown_connections 2024-04-26T01:45:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:29.777+0000 7fc8c9400700 1 -- 172.21.15.6:0/847105010 wait complete. 2024-04-26T01:45:29.775 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-26T01:45:30.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:45:30.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:30 smithi006 ceph-mon[28419]: pgmap v436: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:30.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:30 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/847105010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:30.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:30 smithi188 ceph-mon[32751]: pgmap v436: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:30.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:30 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/847105010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:30.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:30 smithi029 ceph-mon[32775]: pgmap v436: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:30.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:30 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/847105010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:30.972 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:32.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:32 smithi006 ceph-mon[28419]: pgmap v437: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:32.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:32 smithi188 ceph-mon[32751]: pgmap v437: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:32.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:32 smithi029 ceph-mon[32775]: pgmap v437: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:33.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.098+0000 7f3b3b200700 1 -- 172.21.15.6:0/2786944950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b3406b0f0 msgr2=0x7f3b3410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.098+0000 7f3b3b200700 1 --2- 172.21.15.6:0/2786944950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b3406b0f0 0x7f3b3410f900 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c00b0a0 tx=0x7f3b2c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:45:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.098+0000 7f3b3b200700 1 -- 172.21.15.6:0/2786944950 shutdown_connections 2024-04-26T01:45:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.098+0000 7f3b3b200700 1 --2- 172.21.15.6:0/2786944950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b3406b0f0 0x7f3b3410f900 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.098+0000 7f3b3b200700 1 --2- 172.21.15.6:0/2786944950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b34108a30 0x7f3b34108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.098+0000 7f3b3b200700 1 --2- 172.21.15.6:0/2786944950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b3406a750 0x7f3b3406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.098+0000 7f3b3b200700 1 -- 172.21.15.6:0/2786944950 >> 172.21.15.6:0/2786944950 conn(0x7f3b340788f0 msgr2=0x7f3b34078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.098+0000 7f3b3b200700 1 -- 172.21.15.6:0/2786944950 shutdown_connections 2024-04-26T01:45:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.098+0000 7f3b3b200700 1 -- 172.21.15.6:0/2786944950 wait complete. 2024-04-26T01:45:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.099+0000 7f3b3b200700 1 Processor -- start 2024-04-26T01:45:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.099+0000 7f3b3b200700 1 -- start start 2024-04-26T01:45:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.100+0000 7f3b3b200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b3406a750 0x7f3b34199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.100+0000 7f3b3b200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b3406b0f0 0x7f3b341995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.100+0000 7f3b3b200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b34108a30 0x7f3b3419d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.100+0000 7f3b3b200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3b34112bd0 con 0x7f3b34108a30 2024-04-26T01:45:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.100+0000 7f3b3b200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3b34112a50 con 0x7f3b3406b0f0 2024-04-26T01:45:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.100+0000 7f3b3b200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3b34112d50 con 0x7f3b3406a750 2024-04-26T01:45:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.100+0000 7f3b33400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b3406b0f0 0x7f3b341995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.100+0000 7f3b33400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b3406b0f0 0x7f3b341995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:58754/0 (socket says 172.21.15.6:58754) 2024-04-26T01:45:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.100+0000 7f3b33400700 1 -- 172.21.15.6:0/2852969711 learned_addr learned my addr 172.21.15.6:0/2852969711 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.100+0000 7f3b33e00700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b3406a750 0x7f3b34199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.101+0000 7f3b39200700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b34108a30 0x7f3b3419d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.101+0000 7f3b33400700 1 -- 172.21.15.6:0/2852969711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b3406a750 msgr2=0x7f3b34199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.101+0000 7f3b33400700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b3406a750 0x7f3b34199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.101+0000 7f3b33400700 1 -- 172.21.15.6:0/2852969711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b34108a30 msgr2=0x7f3b3419d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.101+0000 7f3b33400700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b34108a30 0x7f3b3419d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.101+0000 7f3b33400700 1 -- 172.21.15.6:0/2852969711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3419e070 con 0x7f3b3406b0f0 2024-04-26T01:45:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.101+0000 7f3b39200700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b34108a30 0x7f3b3419d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.101+0000 7f3b33400700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b3406b0f0 0x7f3b341995a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000e9a0 tx=0x7f3b2000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.101+0000 7f3b30c00700 1 -- 172.21.15.6:0/2852969711 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2000cbd0 con 0x7f3b3406b0f0 2024-04-26T01:45:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.102+0000 7f3b3b200700 1 -- 172.21.15.6:0/2852969711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b3419e360 con 0x7f3b3406b0f0 2024-04-26T01:45:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.102+0000 7f3b30c00700 1 -- 172.21.15.6:0/2852969711 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b2000cd70 con 0x7f3b3406b0f0 2024-04-26T01:45:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.102+0000 7f3b30c00700 1 -- 172.21.15.6:0/2852969711 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b20018680 con 0x7f3b3406b0f0 2024-04-26T01:45:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.102+0000 7f3b3b200700 1 -- 172.21.15.6:0/2852969711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b341a5be0 con 0x7f3b3406b0f0 2024-04-26T01:45:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.103+0000 7f3b3b200700 1 -- 172.21.15.6:0/2852969711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b34069730 con 0x7f3b3406b0f0 2024-04-26T01:45:33.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.105+0000 7f3b30c00700 1 -- 172.21.15.6:0/2852969711 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f3b20010430 con 0x7f3b3406b0f0 2024-04-26T01:45:33.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.106+0000 7f3b30c00700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3b180a3c40 0x7f3b180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:33.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.107+0000 7f3b30c00700 1 -- 172.21.15.6:0/2852969711 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f3b200d0930 con 0x7f3b3406b0f0 2024-04-26T01:45:33.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.107+0000 7f3b33e00700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3b180a3c40 0x7f3b180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:33.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.107+0000 7f3b33e00700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3b180a3c40 0x7f3b180a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009be0 tx=0x7f3b24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:33.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.109+0000 7f3b30c00700 1 -- 172.21.15.6:0/2852969711 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b2009d270 con 0x7f3b3406b0f0 2024-04-26T01:45:33.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.338+0000 7f3b3b200700 1 -- 172.21.15.6:0/2852969711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b3402df80 con 0x7f3b3406b0f0 2024-04-26T01:45:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.341+0000 7f3b30c00700 1 -- 172.21.15.6:0/2852969711 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+22684 (secure 0 0 0) 0x7f3b2009cb70 con 0x7f3b3406b0f0 2024-04-26T01:45:33.342 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:45:33.342 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":95,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:32:29.071544+0000","modified":"2024-04-26T01:43:49.972785+0000","last_up_change":"2024-04-26T01:40:49.915924+0000","last_in_change":"2024-04-26T01:40:29.559647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:36:35.182726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:41:59.612167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:42:04.106467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f164c51d-ce5d-4140-b9e3-a4b84088f5ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6803","nonce":76518486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6805","nonce":76518486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6809","nonce":76518486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6807","nonce":76518486}]},"public_addr":"172.21.15.6:6803/76518486","cluster_addr":"172.21.15.6:6805/76518486","heartbeat_back_addr":"172.21.15.6:6809/76518486","heartbeat_front_addr":"172.21.15.6:6807/76518486","state":["exists","up"]},{"osd":1,"uuid":"57e875dd-ddf1-4ee4-a2a7-9589d7f42818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6801","nonce":400769116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6803","nonce":400769116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6807","nonce":400769116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6805","nonce":400769116}]},"public_addr":"172.21.15.29:6801/400769116","cluster_addr":"172.21.15.29:6803/400769116","heartbeat_back_addr":"172.21.15.29:6807/400769116","heartbeat_front_addr":"172.21.15.29:6805/400769116","state":["exists","up"]},{"osd":2,"uuid":"77e0c294-c48a-4318-938b-4604a533a2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6801","nonce":3508067672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6803","nonce":3508067672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6807","nonce":3508067672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6805","nonce":3508067672}]},"public_addr":"172.21.15.188:6801/3508067672","cluster_addr":"172.21.15.188:6803/3508067672","heartbeat_back_addr":"172.21.15.188:6807/3508067672","heartbeat_front_addr":"172.21.15.188:6805/3508067672","state":["exists","up"]},{"osd":3,"uuid":"6393dac5-996c-47af-9687-0f1393ecc483","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6811","nonce":1987311118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6813","nonce":1987311118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6817","nonce":1987311118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6815","nonce":1987311118}]},"public_addr":"172.21.15.6:6811/1987311118","cluster_addr":"172.21.15.6:6813/1987311118","heartbeat_back_addr":"172.21.15.6:6817/1987311118","heartbeat_front_addr":"172.21.15.6:6815/1987311118","state":["exists","up"]},{"osd":4,"uuid":"f2c1a535-26a0-46c6-a6d0-c2529c786739","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6809","nonce":2125876609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6811","nonce":2125876609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6815","nonce":2125876609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6813","nonce":2125876609}]},"public_addr":"172.21.15.29:6809/2125876609","cluster_addr":"172.21.15.29:6811/2125876609","heartbeat_back_addr":"172.21.15.29:6815/2125876609","heartbeat_front_addr":"172.21.15.29:6813/2125876609","state":["exists","up"]},{"osd":5,"uuid":"9788bb0a-0d79-4743-bce6-80f5342e8f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6809","nonce":2092077007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6811","nonce":2092077007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6815","nonce":2092077007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6813","nonce":2092077007}]},"public_addr":"172.21.15.188:6809/2092077007","cluster_addr":"172.21.15.188:6811/2092077007","heartbeat_back_addr":"172.21.15.188:6815/2092077007","heartbeat_front_addr":"172.21.15.188:6813/2092077007","state":["exists","up"]},{"osd":6,"uuid":"fd8584e1-69ee-4c30-85e4-a637357d1b6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6819","nonce":3256016695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6821","nonce":3256016695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6825","nonce":3256016695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6823","nonce":3256016695}]},"public_addr":"172.21.15.6:6819/3256016695","cluster_addr":"172.21.15.6:6821/3256016695","heartbeat_back_addr":"172.21.15.6:6825/3256016695","heartbeat_front_addr":"172.21.15.6:6823/3256016695","state":["exists","up"]},{"osd":7,"uuid":"6b7e6784-d574-42da-8416-a74a3b6b3755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6817","nonce":3478594037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6819","nonce":3478594037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6823","nonce":3478594037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6821","nonce":3478594037}]},"public_addr":"172.21.15.29:6817/3478594037","cluster_addr":"172.21.15.29:6819/3478594037","heartbeat_back_addr":"172.21.15.29:6823/3478594037","heartbeat_front_addr":"172.21.15.29:6821/3478594037","state":["exists","up"]},{"osd":8,"uuid":"30cd2ea1-2767-4062-8bd9-47214e4b4b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6817","nonce":3290364376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6819","nonce":3290364376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6823","nonce":3290364376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6821","nonce":3290364376}]},"public_addr":"172.21.15.188:6817/3290364376","cluster_addr":"172.21.15.188:6819/3290364376","heartbeat_back_addr":"172.21.15.188:6823/3290364376","heartbeat_front_addr":"172.21.15.188:6821/3290364376","state":["exists","up"]},{"osd":9,"uuid":"f033cdc0-1178-4a7f-a338-69ff32416087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6827","nonce":1918893912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6829","nonce":1918893912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6833","nonce":1918893912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6831","nonce":1918893912}]},"public_addr":"172.21.15.6:6827/1918893912","cluster_addr":"172.21.15.6:6829/1918893912","heartbeat_back_addr":"172.21.15.6:6833/1918893912","heartbeat_front_addr":"172.21.15.6:6831/1918893912","state":["exists","up"]},{"osd":10,"uuid":"04e9b5ef-ac0c-482b-a69e-a7121b36e7da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6825","nonce":1395777729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6827","nonce":1395777729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6831","nonce":1395777729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6829","nonce":1395777729}]},"public_addr":"172.21.15.29:6825/1395777729","cluster_addr":"172.21.15.29:6827/1395777729","heartbeat_back_addr":"172.21.15.29:6831/1395777729","heartbeat_front_addr":"172.21.15.29:6829/1395777729","state":["exists","up"]},{"osd":11,"uuid":"cb1638ce-4189-4241-b63e-b63c607e90ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6825","nonce":3322197880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6827","nonce":3322197880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6831","nonce":3322197880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6829","nonce":3322197880}]},"public_addr":"172.21.15.188:6825/3322197880","cluster_addr":"172.21.15.188:6827/3322197880","heartbeat_back_addr":"172.21.15.188:6831/3322197880","heartbeat_front_addr":"172.21.15.188:6829/3322197880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:41.553455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:07.641467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:31.748891+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:03.743553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:30.065262+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:54.976527+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:29.348206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:56.285928+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:21.246409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:54.468561+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:22.093015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:47.342379+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.19","mappings":[{"from":3,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/3143272062":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1325707596":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6800/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6801/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1359518071":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6801/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/2813598711":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/3140150861":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6800/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/65701414":"2024-04-27T01:33:47.021882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:45:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.344+0000 7f3b3b200700 1 -- 172.21.15.6:0/2852969711 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3b180a3c40 msgr2=0x7f3b180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.344+0000 7f3b3b200700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3b180a3c40 0x7f3b180a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009be0 tx=0x7f3b24009260 comp rx=0 tx=0).stop 2024-04-26T01:45:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.344+0000 7f3b3b200700 1 -- 172.21.15.6:0/2852969711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b3406b0f0 msgr2=0x7f3b341995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.344+0000 7f3b3b200700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b3406b0f0 0x7f3b341995a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000e9a0 tx=0x7f3b2000ecb0 comp rx=0 tx=0).stop 2024-04-26T01:45:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.344+0000 7f3b3b200700 1 -- 172.21.15.6:0/2852969711 shutdown_connections 2024-04-26T01:45:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.344+0000 7f3b3b200700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b34108a30 0x7f3b3419d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.344+0000 7f3b3b200700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3b180a3c40 0x7f3b180a6100 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.344+0000 7f3b3b200700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b3406a750 0x7f3b34199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.344+0000 7f3b3b200700 1 --2- 172.21.15.6:0/2852969711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b3406b0f0 0x7f3b341995a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.344+0000 7f3b3b200700 1 -- 172.21.15.6:0/2852969711 >> 172.21.15.6:0/2852969711 conn(0x7f3b340788f0 msgr2=0x7f3b340ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.345+0000 7f3b3b200700 1 -- 172.21.15.6:0/2852969711 shutdown_connections 2024-04-26T01:45:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:33.345+0000 7f3b3b200700 1 -- 172.21.15.6:0/2852969711 wait complete. 2024-04-26T01:45:34.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T01:42:08.299563+0000', 'modified': '2024-04-26T01:42:31.294242+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24523, 'mds_1': 24512, 'mds_2': 24505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6837/402171153', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6836', 'nonce': 402171153}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 402171153}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'g', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.6:6839/3936238112', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6838', 'nonce': 3936238112}, {'type': 'v1', 'addr': '172.21.15.6:6839', 'nonce': 3936238112}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6835/1255666766', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6834', 'nonce': 1255666766}, {'type': 'v1', 'addr': '172.21.15.188:6835', 'nonce': 1255666766}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.188:6837/489767141', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6836', 'nonce': 489767141}, {'type': 'v1', 'addr': '172.21.15.188:6837', 'nonce': 489767141}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24512': {'gid': 24512, 'name': 'b', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.29:6839/2325582359', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6838', 'nonce': 2325582359}, {'type': 'v1', 'addr': '172.21.15.29:6839', 'nonce': 2325582359}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.188:6839/674623421', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6838', 'nonce': 674623421}, {'type': 'v1', 'addr': '172.21.15.188:6839', 'nonce': 674623421}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24523, 'qdb_cluster': [24523, 24505, 24512]} 2024-04-26T01:45:34.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T01:45:34.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell mds.c status 2024-04-26T01:45:34.503 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:34 smithi188 ceph-mon[32751]: pgmap v438: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:34 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2852969711' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:45:34.657 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:34 smithi006 ceph-mon[28419]: pgmap v438: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:34.657 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:34 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2852969711' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:45:34.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:34 smithi029 ceph-mon[32775]: pgmap v438: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:34.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:34 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2852969711' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:45:36.460 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:36 smithi006 ceph-mon[28419]: pgmap v439: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:45:36.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.639+0000 7f353ec00700 1 -- 172.21.15.6:0/2534132933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f353810f440 msgr2=0x7f3538111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.639+0000 7f353ec00700 1 --2- 172.21.15.6:0/2534132933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f353810f440 0x7f3538111830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3530009990 tx=0x7f3530009ca0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.639+0000 7f353ec00700 1 -- 172.21.15.6:0/2534132933 shutdown_connections 2024-04-26T01:45:36.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.639+0000 7f353ec00700 1 --2- 172.21.15.6:0/2534132933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f353810f440 0x7f3538111830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.639+0000 7f353ec00700 1 --2- 172.21.15.6:0/2534132933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3538074c70 0x7f3538075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.639+0000 7f353ec00700 1 --2- 172.21.15.6:0/2534132933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3538075590 0x7f353810ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.639+0000 7f353ec00700 1 -- 172.21.15.6:0/2534132933 >> 172.21.15.6:0/2534132933 conn(0x7f3538079ba0 msgr2=0x7f3538079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:36.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.639+0000 7f353ec00700 1 -- 172.21.15.6:0/2534132933 shutdown_connections 2024-04-26T01:45:36.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.639+0000 7f353ec00700 1 -- 172.21.15.6:0/2534132933 wait complete. 2024-04-26T01:45:36.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.640+0000 7f353ec00700 1 Processor -- start 2024-04-26T01:45:36.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.641+0000 7f353ec00700 1 -- start start 2024-04-26T01:45:36.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.641+0000 7f353ec00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538074c70 0x7f353819b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.641+0000 7f353ec00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3538075590 0x7f353819b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.641+0000 7f353ec00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f353810f440 0x7f353819fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.641+0000 7f353ec00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3538114e40 con 0x7f3538074c70 2024-04-26T01:45:36.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.641+0000 7f353ec00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3538114cc0 con 0x7f3538075590 2024-04-26T01:45:36.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.641+0000 7f353ec00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3538114fc0 con 0x7f353810f440 2024-04-26T01:45:36.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:36 smithi188 ceph-mon[32751]: pgmap v439: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:45:36.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.641+0000 7f3537e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538074c70 0x7f353819b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.641+0000 7f3537e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538074c70 0x7f353819b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42584/0 (socket says 172.21.15.6:42584) 2024-04-26T01:45:36.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.641+0000 7f3537e00700 1 -- 172.21.15.6:0/1698396760 learned_addr learned my addr 172.21.15.6:0/1698396760 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:36.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.642+0000 7f352f400700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3538075590 0x7f353819b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.642+0000 7f353cc00700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f353810f440 0x7f353819fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.642+0000 7f3537e00700 1 -- 172.21.15.6:0/1698396760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f353810f440 msgr2=0x7f353819fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.642+0000 7f3537e00700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f353810f440 0x7f353819fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.642+0000 7f3537e00700 1 -- 172.21.15.6:0/1698396760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3538075590 msgr2=0x7f353819b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.642+0000 7f3537e00700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3538075590 0x7f353819b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.642+0000 7f3537e00700 1 -- 172.21.15.6:0/1698396760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35381a0360 con 0x7f3538074c70 2024-04-26T01:45:36.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.642+0000 7f352f400700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3538075590 0x7f353819b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:36.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.642+0000 7f3537e00700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538074c70 0x7f353819b350 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f352800ef30 tx=0x7f352800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:36.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.643+0000 7f3535600700 1 -- 172.21.15.6:0/1698396760 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35280092e0 con 0x7f3538074c70 2024-04-26T01:45:36.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.643+0000 7f3535600700 1 -- 172.21.15.6:0/1698396760 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3528004bb0 con 0x7f3538074c70 2024-04-26T01:45:36.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.643+0000 7f3535600700 1 -- 172.21.15.6:0/1698396760 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528004040 con 0x7f3538074c70 2024-04-26T01:45:36.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.643+0000 7f353ec00700 1 -- 172.21.15.6:0/1698396760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35381a05f0 con 0x7f3538074c70 2024-04-26T01:45:36.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.643+0000 7f353ec00700 1 -- 172.21.15.6:0/1698396760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35381a7ed0 con 0x7f3538074c70 2024-04-26T01:45:36.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.646+0000 7f3535600700 1 -- 172.21.15.6:0/1698396760 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f352801b030 con 0x7f3538074c70 2024-04-26T01:45:36.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.648+0000 7f3535600700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f351c0a3cc0 0x7f351c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.648+0000 7f354951eb80 1 Processor -- start 2024-04-26T01:45:36.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.648+0000 7f3535600700 1 -- 172.21.15.6:0/1698396760 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f35280cff50 con 0x7f3538074c70 2024-04-26T01:45:36.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.648+0000 7f354951eb80 1 -- start start 2024-04-26T01:45:36.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.648+0000 7f352f400700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f351c0a3cc0 0x7f351c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.648+0000 7f354951eb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c8296d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.648+0000 7f354951eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c834570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f354951eb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c834ab0 0x55991c836ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f354951eb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55991c77ce40 con 0x55991c8292f0 2024-04-26T01:45:36.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f354951eb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55991c7236e0 con 0x55991c834ab0 2024-04-26T01:45:36.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f354951eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55991c778e00 con 0x55991c829c10 2024-04-26T01:45:36.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f3537e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c8296d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f3537e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c8296d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42588/0 (socket says 172.21.15.6:42588) 2024-04-26T01:45:36.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f3537e00700 1 -- 172.21.15.6:0/2416428427 learned_addr learned my addr 172.21.15.6:0/2416428427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:36.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f352f400700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f351c0a3cc0 0x7f351c0a6180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3520009680 tx=0x7f3520009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:36.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f352f400700 1 --2- 172.21.15.6:0/2416428427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c834570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f353cc00700 1 --2- 172.21.15.6:0/2416428427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c834ab0 0x55991c836ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f3537e00700 1 -- 172.21.15.6:0/2416428427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 msgr2=0x55991c834570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f3537e00700 1 --2- 172.21.15.6:0/2416428427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c834570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f3537e00700 1 -- 172.21.15.6:0/2416428427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c834ab0 msgr2=0x55991c836ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f3537e00700 1 --2- 172.21.15.6:0/2416428427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c834ab0 0x55991c836ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.649+0000 7f3537e00700 1 -- 172.21.15.6:0/2416428427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55991c719630 con 0x55991c8292f0 2024-04-26T01:45:36.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.650+0000 7f3537e00700 1 --2- 172.21.15.6:0/2416428427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c8296d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3528098680 tx=0x7f352800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f15a08d6d6b2fb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:36.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.650+0000 7f352d600700 1 -- 172.21.15.6:0/2416428427 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352809c920 con 0x55991c8292f0 2024-04-26T01:45:36.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.650+0000 7f352d600700 1 -- 172.21.15.6:0/2416428427 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f352809c220 con 0x55991c8292f0 2024-04-26T01:45:36.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.650+0000 7f352d600700 1 -- 172.21.15.6:0/2416428427 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35280a0380 con 0x55991c8292f0 2024-04-26T01:45:36.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.651+0000 7f354951eb80 1 -- 172.21.15.6:0/2416428427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 msgr2=0x55991c8296d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.651+0000 7f354951eb80 1 --2- 172.21.15.6:0/2416428427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c8296d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3528098680 tx=0x7f352800ca40 comp rx=0 tx=0).stop 2024-04-26T01:45:36.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.651+0000 7f354951eb80 1 -- 172.21.15.6:0/2416428427 shutdown_connections 2024-04-26T01:45:36.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.651+0000 7f354951eb80 1 --2- 172.21.15.6:0/2416428427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c834ab0 0x55991c836ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.651+0000 7f354951eb80 1 --2- 172.21.15.6:0/2416428427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c8296d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.651+0000 7f354951eb80 1 --2- 172.21.15.6:0/2416428427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c834570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.651+0000 7f354951eb80 1 -- 172.21.15.6:0/2416428427 >> 172.21.15.6:0/2416428427 conn(0x55991c758550 msgr2=0x55991c825790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:36.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.651+0000 7f354951eb80 1 -- 172.21.15.6:0/2416428427 shutdown_connections 2024-04-26T01:45:36.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.651+0000 7f354951eb80 1 -- 172.21.15.6:0/2416428427 wait complete. 2024-04-26T01:45:36.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.652+0000 7f354951eb80 1 -- start start 2024-04-26T01:45:36.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.652+0000 7f354951eb80 1 Processor -- start 2024-04-26T01:45:36.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f354951eb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c8341a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f354951eb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c829c10 0x55991c8314a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f354951eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c834ab0 0x55991c8319e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f354951eb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55991c8375a0 con 0x55991c8292f0 2024-04-26T01:45:36.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f354951eb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55991c736810 con 0x55991c829c10 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f354951eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55991c837720 con 0x55991c834ab0 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f354951eb80 10 client.0 _ll_register_callbacks cb 0x55991c7156d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f3537e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c8341a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f3537e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c8341a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42592/0 (socket says 172.21.15.6:42592) 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f3537e00700 1 -- 172.21.15.6:0/4032451088 learned_addr learned my addr 172.21.15.6:0/4032451088 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f353cc00700 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c834ab0 0x55991c8319e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f352f400700 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c829c10 0x55991c8314a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f3537e00700 1 -- 172.21.15.6:0/4032451088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c834ab0 msgr2=0x55991c8319e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f3537e00700 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c834ab0 0x55991c8319e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f3537e00700 1 -- 172.21.15.6:0/4032451088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c829c10 msgr2=0x55991c8314a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f3537e00700 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c829c10 0x55991c8314a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.653+0000 7f3537e00700 1 -- 172.21.15.6:0/4032451088 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55991c6ca880 con 0x55991c8292f0 2024-04-26T01:45:36.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.654+0000 7f3537e00700 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c8341a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f35280090a0 tx=0x7f352800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f1f89b8c2a2701 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:36.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.654+0000 7f3526000700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:45:36.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.654+0000 7f3526000700 1 -- 172.21.15.6:0/4032451088 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352809c920 con 0x55991c8292f0 2024-04-26T01:45:36.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.654+0000 7f3526000700 1 -- 172.21.15.6:0/4032451088 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f3528004d60 con 0x55991c8292f0 2024-04-26T01:45:36.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.654+0000 7f354951eb80 1 -- 172.21.15.6:0/4032451088 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f352809c220 con 0x55991c8292f0 2024-04-26T01:45:36.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.654+0000 7f3526000700 1 -- 172.21.15.6:0/4032451088 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f352807f070 con 0x55991c8292f0 2024-04-26T01:45:36.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.654+0000 7f354951eb80 10 client.14871 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:36.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.654+0000 7f354951eb80 1 -- 172.21.15.6:0/4032451088 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55991c713aa0 con 0x55991c8292f0 2024-04-26T01:45:36.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.655+0000 7f3526000700 1 -- 172.21.15.6:0/4032451088 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f352809c220 con 0x55991c8292f0 2024-04-26T01:45:36.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.655+0000 7f3526000700 1 -- 172.21.15.6:0/4032451088 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f35280980e0 con 0x55991c8292f0 2024-04-26T01:45:36.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.655+0000 7f354951eb80 10 client.14871 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:36.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.655+0000 7f354951eb80 10 client.14871 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T01:45:36.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.655+0000 7f354951eb80 20 client.14871 populate_metadata read hostname 'smithi006' 2024-04-26T01:45:36.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.655+0000 7f354951eb80 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55991c842f70 0x55991c845160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.655+0000 7f354951eb80 4 client.14871 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:36.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.655+0000 7f354951eb80 1 -- 172.21.15.6:0/4032451088 --> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55991c78a630 con 0x55991c842f70 2024-04-26T01:45:36.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.655+0000 7f352f400700 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55991c842f70 0x55991c845160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.656+0000 7f352f400700 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55991c842f70 0x55991c845160 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3216a69211c992e1 server_cookie=6e0216b25e39a02f in_seq=0 out_seq=0 2024-04-26T01:45:36.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.656+0000 7f3526000700 10 client.14871 ms_handle_connect on v2:172.21.15.188:6836/489767141 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.657+0000 7f3526000700 1 -- 172.21.15.6:0/4032451088 <== mds.2 v2:172.21.15.188:6836/489767141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55991c78a630 con 0x55991c842f70 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.657+0000 7f3526000700 10 client.14871 handle_command_reply: tid=0 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.657+0000 7f354951eb80 1 client.14871 shutdown 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.657+0000 7f354951eb80 1 -- 172.21.15.6:0/4032451088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 msgr2=0x55991c8341a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.657+0000 7f354951eb80 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c8341a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f35280090a0 tx=0x7f352800c890 comp rx=0 tx=0).stop 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.657+0000 7f354951eb80 1 -- 172.21.15.6:0/4032451088 shutdown_connections 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.657+0000 7f354951eb80 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c834ab0 0x55991c8319e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.657+0000 7f354951eb80 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55991c842f70 0x55991c845160 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.657+0000 7f354951eb80 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c8341a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.657+0000 7f354951eb80 1 --2- 172.21.15.6:0/4032451088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c829c10 0x55991c8314a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.657+0000 7f354951eb80 1 -- 172.21.15.6:0/4032451088 >> 172.21.15.6:0/4032451088 conn(0x55991c758550 msgr2=0x55991c82bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.657+0000 7f3526000700 0 client.14871 ms_handle_reset on v2:172.21.15.188:6836/489767141 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.658+0000 7f354951eb80 1 -- 172.21.15.6:0/4032451088 shutdown_connections 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.658+0000 7f354951eb80 1 -- 172.21.15.6:0/4032451088 wait complete. 2024-04-26T01:45:36.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.658+0000 7f354951eb80 20 client.14871 trim_cache size 0 max 16384 2024-04-26T01:45:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.682+0000 7f354951eb80 1 Processor -- start 2024-04-26T01:45:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.682+0000 7f354951eb80 1 -- start start 2024-04-26T01:45:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.682+0000 7f354951eb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c8292f0 0x55991c843620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.682+0000 7f354951eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c8304a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.682+0000 7f354951eb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c834ab0 0x55991c8250d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.682+0000 7f354951eb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55991c8434a0 con 0x55991c834ab0 2024-04-26T01:45:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.682+0000 7f354951eb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55991c843b70 con 0x55991c8292f0 2024-04-26T01:45:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.682+0000 7f354951eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55991c844240 con 0x55991c829c10 2024-04-26T01:45:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.682+0000 7f353cc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c834ab0 0x55991c8250d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.683+0000 7f352f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c8304a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.683+0000 7f3537e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c8292f0 0x55991c843620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.683+0000 7f352f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c8304a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47772/0 (socket says 172.21.15.6:47772) 2024-04-26T01:45:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.683+0000 7f352f400700 1 -- 172.21.15.6:0/1157729290 learned_addr learned my addr 172.21.15.6:0/1157729290 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.683+0000 7f353cc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c834ab0 0x55991c8250d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42602/0 (socket says 172.21.15.6:42602) 2024-04-26T01:45:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.683+0000 7f3537e00700 1 -- 172.21.15.6:0/1157729290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 msgr2=0x55991c8304a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.683+0000 7f3537e00700 1 --2- 172.21.15.6:0/1157729290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c8304a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.683+0000 7f3537e00700 1 -- 172.21.15.6:0/1157729290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c834ab0 msgr2=0x55991c8250d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.683+0000 7f3537e00700 1 --2- 172.21.15.6:0/1157729290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c834ab0 0x55991c8250d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.683+0000 7f3537e00700 1 -- 172.21.15.6:0/1157729290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55991c826ad0 con 0x55991c8292f0 2024-04-26T01:45:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.683+0000 7f3537e00700 1 --2- 172.21.15.6:0/1157729290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c8292f0 0x55991c843620 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f352809b320 tx=0x7f352809a730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9792280d1ac0a1c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:36.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.683+0000 7f352cc00700 1 -- 172.21.15.6:0/1157729290 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528003a00 con 0x55991c8292f0 2024-04-26T01:45:36.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.684+0000 7f352cc00700 1 -- 172.21.15.6:0/1157729290 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f352809c920 con 0x55991c8292f0 2024-04-26T01:45:36.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.684+0000 7f352cc00700 1 -- 172.21.15.6:0/1157729290 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528003bf0 con 0x55991c8292f0 2024-04-26T01:45:36.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.684+0000 7f354951eb80 1 -- 172.21.15.6:0/1157729290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c8292f0 msgr2=0x55991c843620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.684+0000 7f354951eb80 1 --2- 172.21.15.6:0/1157729290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c8292f0 0x55991c843620 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f352809b320 tx=0x7f352809a730 comp rx=0 tx=0).stop 2024-04-26T01:45:36.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.684+0000 7f354951eb80 1 -- 172.21.15.6:0/1157729290 shutdown_connections 2024-04-26T01:45:36.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.684+0000 7f354951eb80 1 --2- 172.21.15.6:0/1157729290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c834ab0 0x55991c8250d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.684+0000 7f354951eb80 1 --2- 172.21.15.6:0/1157729290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c8292f0 0x55991c843620 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.684+0000 7f354951eb80 1 --2- 172.21.15.6:0/1157729290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c8304a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.684+0000 7f354951eb80 1 -- 172.21.15.6:0/1157729290 >> 172.21.15.6:0/1157729290 conn(0x55991c758550 msgr2=0x55991c82fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:36.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.684+0000 7f354951eb80 1 -- 172.21.15.6:0/1157729290 shutdown_connections 2024-04-26T01:45:36.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.684+0000 7f354951eb80 1 -- 172.21.15.6:0/1157729290 wait complete. 2024-04-26T01:45:36.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.685+0000 7f354951eb80 1 -- start start 2024-04-26T01:45:36.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.685+0000 7f354951eb80 1 Processor -- start 2024-04-26T01:45:36.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f354951eb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c84e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f354951eb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c84eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f354951eb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c834ab0 0x55991c847580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f354951eb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55991c844fe0 con 0x55991c8292f0 2024-04-26T01:45:36.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f354951eb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55991c844910 con 0x55991c834ab0 2024-04-26T01:45:36.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f354951eb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55991c8456b0 con 0x55991c829c10 2024-04-26T01:45:36.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f354951eb80 10 client.0 _ll_register_callbacks cb 0x55991c7156d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:36.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f3537e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c84e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f353cc00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c834ab0 0x55991c847580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f3537e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c84e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42606/0 (socket says 172.21.15.6:42606) 2024-04-26T01:45:36.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f3537e00700 1 -- 172.21.15.6:0/1513799808 learned_addr learned my addr 172.21.15.6:0/1513799808 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:36.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f352f400700 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c84eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f3537e00700 1 -- 172.21.15.6:0/1513799808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 msgr2=0x55991c84eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f3537e00700 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c84eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f3537e00700 1 -- 172.21.15.6:0/1513799808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c834ab0 msgr2=0x55991c847580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.686+0000 7f3537e00700 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c834ab0 0x55991c847580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.687+0000 7f3537e00700 1 -- 172.21.15.6:0/1513799808 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55991c847d80 con 0x55991c8292f0 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.687+0000 7f353cc00700 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c834ab0 0x55991c847580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.687+0000 7f3537e00700 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c84e5e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f352809b1b0 tx=0x7f352809bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe0708fd07c050a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.687+0000 7f3526000700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.687+0000 7f3526000700 1 -- 172.21.15.6:0/1513799808 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35280a4e00 con 0x55991c8292f0 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.687+0000 7f3526000700 1 -- 172.21.15.6:0/1513799808 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f35280a4970 con 0x55991c8292f0 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.687+0000 7f354951eb80 1 -- 172.21.15.6:0/1513799808 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f352809c920 con 0x55991c8292f0 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.687+0000 7f3526000700 1 -- 172.21.15.6:0/1513799808 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f352809c220 con 0x55991c8292f0 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.687+0000 7f354951eb80 10 client.14877 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.687+0000 7f354951eb80 1 -- 172.21.15.6:0/1513799808 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55991c847d80 con 0x55991c8292f0 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.688+0000 7f3526000700 1 -- 172.21.15.6:0/1513799808 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f352809c920 con 0x55991c8292f0 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.688+0000 7f3526000700 1 -- 172.21.15.6:0/1513799808 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f3528003a00 con 0x55991c8292f0 2024-04-26T01:45:36.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.688+0000 7f354951eb80 10 client.14877 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.688+0000 7f354951eb80 10 client.14877 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.688+0000 7f354951eb80 20 client.14877 populate_metadata read hostname 'smithi006' 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.688+0000 7f354951eb80 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55991c8afd20 0x55991c8aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.688+0000 7f354951eb80 4 client.14877 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.688+0000 7f354951eb80 1 -- 172.21.15.6:0/1513799808 --> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] -- command(tid 0: {"prefix": "status"}) -- 0x55991c7846d0 con 0x55991c8afd20 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.688+0000 7f352f400700 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55991c8afd20 0x55991c8aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f352f400700 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55991c8afd20 0x55991c8aea30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b731b2a1e2c951ee server_cookie=958947fa798662d7 in_seq=0 out_seq=0 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f3526000700 10 client.14877 ms_handle_connect on v2:172.21.15.188:6836/489767141 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f3526000700 1 -- 172.21.15.6:0/1513799808 <== mds.2 v2:172.21.15.188:6836/489767141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55991c7846d0 con 0x55991c8afd20 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f3526000700 10 client.14877 handle_command_reply: tid=0 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f354951eb80 1 client.14877 shutdown 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f354951eb80 1 -- 172.21.15.6:0/1513799808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 msgr2=0x55991c84e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f354951eb80 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c84e5e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f352809b1b0 tx=0x7f352809bff0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f354951eb80 1 -- 172.21.15.6:0/1513799808 shutdown_connections 2024-04-26T01:45:36.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f354951eb80 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55991c834ab0 0x55991c847580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f354951eb80 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55991c8afd20 0x55991c8aea30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f354951eb80 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55991c8292f0 0x55991c84e5e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f354951eb80 1 --2- 172.21.15.6:0/1513799808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55991c829c10 0x55991c84eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f3526000700 0 client.14877 ms_handle_reset on v2:172.21.15.188:6836/489767141 2024-04-26T01:45:36.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.689+0000 7f354951eb80 1 -- 172.21.15.6:0/1513799808 >> 172.21.15.6:0/1513799808 conn(0x55991c758550 msgr2=0x55991c8304a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:36.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.690+0000 7f354951eb80 1 -- 172.21.15.6:0/1513799808 shutdown_connections 2024-04-26T01:45:36.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.690+0000 7f354951eb80 1 -- 172.21.15.6:0/1513799808 wait complete. 2024-04-26T01:45:36.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.690+0000 7f354951eb80 20 client.14877 trim_cache size 0 max 16384 2024-04-26T01:45:36.690 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:45:36.690 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:45:36.691 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 2, 2024-04-26T01:45:36.691 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 24505, 2024-04-26T01:45:36.691 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-26T01:45:36.691 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-26T01:45:36.691 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-26T01:45:36.691 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 192.21875508799999, 2024-04-26T01:45:36.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 25, 2024-04-26T01:45:36.691 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 95, 2024-04-26T01:45:36.691 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T01:45:36.691 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 247.719949593 2024-04-26T01:45:36.691 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:45:36.692 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:36 smithi029 ceph-mon[32775]: pgmap v439: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:45:36.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.690+0000 7f353ec00700 1 -- 172.21.15.6:0/1698396760 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f351c0a3cc0 msgr2=0x7f351c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.690+0000 7f353ec00700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f351c0a3cc0 0x7f351c0a6180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3520009680 tx=0x7f3520009260 comp rx=0 tx=0).stop 2024-04-26T01:45:36.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.690+0000 7f353ec00700 1 -- 172.21.15.6:0/1698396760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538074c70 msgr2=0x7f353819b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.690+0000 7f353ec00700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538074c70 0x7f353819b350 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f352800ef30 tx=0x7f352800c3a0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.691+0000 7f353ec00700 1 -- 172.21.15.6:0/1698396760 shutdown_connections 2024-04-26T01:45:36.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.691+0000 7f353ec00700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f351c0a3cc0 0x7f351c0a6180 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.691+0000 7f353ec00700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f353810f440 0x7f353819fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.691+0000 7f353ec00700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538074c70 0x7f353819b350 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.691+0000 7f353ec00700 1 --2- 172.21.15.6:0/1698396760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3538075590 0x7f353819b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.691+0000 7f353ec00700 1 -- 172.21.15.6:0/1698396760 >> 172.21.15.6:0/1698396760 conn(0x7f3538079ba0 msgr2=0x7f35380ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:36.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.691+0000 7f353ec00700 1 -- 172.21.15.6:0/1698396760 shutdown_connections 2024-04-26T01:45:36.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:36.691+0000 7f353ec00700 1 -- 172.21.15.6:0/1698396760 wait complete. 2024-04-26T01:45:37.527 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T01:45:37.982 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:38.444 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:38 smithi006 ceph-mon[28419]: pgmap v440: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:38.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:38 smithi188 ceph-mon[32751]: pgmap v440: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:38.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:38 smithi029 ceph-mon[32775]: pgmap v440: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.032+0000 7fd723000700 1 -- 172.21.15.6:0/3185012642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd71c101940 msgr2=0x7fd71c101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.032+0000 7fd723000700 1 --2- 172.21.15.6:0/3185012642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd71c101940 0x7fd71c101dc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd708009990 tx=0x7fd708009ca0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.032+0000 7fd723000700 1 -- 172.21.15.6:0/3185012642 shutdown_connections 2024-04-26T01:45:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.032+0000 7fd723000700 1 --2- 172.21.15.6:0/3185012642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd71c102300 0x7fd71c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.032+0000 7fd723000700 1 --2- 172.21.15.6:0/3185012642 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd71c107990 0x7fd71c107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.032+0000 7fd723000700 1 --2- 172.21.15.6:0/3185012642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd71c101940 0x7fd71c101dc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.032+0000 7fd723000700 1 -- 172.21.15.6:0/3185012642 >> 172.21.15.6:0/3185012642 conn(0x7fd71c0788f0 msgr2=0x7fd71c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.033+0000 7fd723000700 1 -- 172.21.15.6:0/3185012642 shutdown_connections 2024-04-26T01:45:40.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.033+0000 7fd723000700 1 -- 172.21.15.6:0/3185012642 wait complete. 2024-04-26T01:45:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.034+0000 7fd723000700 1 Processor -- start 2024-04-26T01:45:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.034+0000 7fd723000700 1 -- start start 2024-04-26T01:45:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.034+0000 7fd723000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd71c101940 0x7fd71c199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.034+0000 7fd723000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd71c102300 0x7fd71c199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.034+0000 7fd723000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd71c107990 0x7fd71c19dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.034+0000 7fd723000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c111b50 con 0x7fd71c102300 2024-04-26T01:45:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.034+0000 7fd723000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c1119d0 con 0x7fd71c107990 2024-04-26T01:45:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.034+0000 7fd723000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c111cd0 con 0x7fd71c101940 2024-04-26T01:45:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.035+0000 7fd71b400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd71c102300 0x7fd71c199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.035+0000 7fd71be00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd71c101940 0x7fd71c199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.035+0000 7fd71b400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd71c102300 0x7fd71c199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46808/0 (socket says 172.21.15.6:46808) 2024-04-26T01:45:40.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.035+0000 7fd71b400700 1 -- 172.21.15.6:0/4030608924 learned_addr learned my addr 172.21.15.6:0/4030608924 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:40.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.035+0000 7fd721000700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd71c107990 0x7fd71c19dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.035+0000 7fd71be00700 1 -- 172.21.15.6:0/4030608924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd71c107990 msgr2=0x7fd71c19dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.035+0000 7fd71be00700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd71c107990 0x7fd71c19dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.035+0000 7fd71be00700 1 -- 172.21.15.6:0/4030608924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd71c102300 msgr2=0x7fd71c199770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.035+0000 7fd71be00700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd71c102300 0x7fd71c199770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.035+0000 7fd71be00700 1 -- 172.21.15.6:0/4030608924 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71c19e1f0 con 0x7fd71c101940 2024-04-26T01:45:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.035+0000 7fd71b400700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd71c102300 0x7fd71c199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.036+0000 7fd71be00700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd71c101940 0x7fd71c199230 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00e9f0 tx=0x7fd70c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.036+0000 7fd718c00700 1 -- 172.21.15.6:0/4030608924 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c004220 con 0x7fd71c101940 2024-04-26T01:45:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.036+0000 7fd718c00700 1 -- 172.21.15.6:0/4030608924 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd70c0043c0 con 0x7fd71c101940 2024-04-26T01:45:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.036+0000 7fd723000700 1 -- 172.21.15.6:0/4030608924 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71c19e4e0 con 0x7fd71c101940 2024-04-26T01:45:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.036+0000 7fd718c00700 1 -- 172.21.15.6:0/4030608924 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c005020 con 0x7fd71c101940 2024-04-26T01:45:40.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.036+0000 7fd723000700 1 -- 172.21.15.6:0/4030608924 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd71c1a5d60 con 0x7fd71c101940 2024-04-26T01:45:40.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.040+0000 7fd718c00700 1 -- 172.21.15.6:0/4030608924 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fd70c004750 con 0x7fd71c101940 2024-04-26T01:45:40.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.041+0000 7fd72d953b80 1 Processor -- start 2024-04-26T01:45:40.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.041+0000 7fd72d953b80 1 -- start start 2024-04-26T01:45:40.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.041+0000 7fd718c00700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd7000a3cc0 0x7fd7000a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.041+0000 7fd718c00700 1 -- 172.21.15.6:0/4030608924 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fd70c0d0300 con 0x7fd71c101940 2024-04-26T01:45:40.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.041+0000 7fd72d953b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e52f0 0x5569cc1e56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.041+0000 7fd72d953b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 0x5569cc1f4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.041+0000 7fd721000700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd7000a3cc0 0x7fd7000a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.041+0000 7fd72d953b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.041+0000 7fd72d953b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5569cc138e40 con 0x5569cc1e5c10 2024-04-26T01:45:40.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.041+0000 7fd72d953b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5569cc0df6e0 con 0x5569cc1f4ec0 2024-04-26T01:45:40.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.041+0000 7fd72d953b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5569cc134e00 con 0x5569cc1e52f0 2024-04-26T01:45:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.042+0000 7fd71be00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e52f0 0x5569cc1e56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.042+0000 7fd71b400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 0x5569cc1f4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.042+0000 7fd71be00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e52f0 0x5569cc1e56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:60908/0 (socket says 172.21.15.6:60908) 2024-04-26T01:45:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.042+0000 7fd71be00700 1 -- 172.21.15.6:0/1139196953 learned_addr learned my addr 172.21.15.6:0/1139196953 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.042+0000 7fd721000700 1 --2- 172.21.15.6:0/1139196953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.042+0000 7fd71b400700 1 -- 172.21.15.6:0/1139196953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e52f0 msgr2=0x5569cc1e56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.042+0000 7fd71b400700 1 --2- 172.21.15.6:0/1139196953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e52f0 0x5569cc1e56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.042+0000 7fd71b400700 1 -- 172.21.15.6:0/1139196953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 msgr2=0x5569cc1f72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.042+0000 7fd71b400700 1 --2- 172.21.15.6:0/1139196953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.042+0000 7fd71b400700 1 -- 172.21.15.6:0/1139196953 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569cc0d5630 con 0x5569cc1e5c10 2024-04-26T01:45:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.042+0000 7fd721000700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd7000a3cc0 0x7fd7000a6180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd714009ec0 tx=0x7fd71400fc40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.043+0000 7fd71b400700 1 --2- 172.21.15.6:0/1139196953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 0x5569cc1f4980 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd708006010 tx=0x7fd7080096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553186cdf84dbf88 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.043+0000 7fd711600700 1 -- 172.21.15.6:0/1139196953 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd708027070 con 0x5569cc1e5c10 2024-04-26T01:45:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.043+0000 7fd711600700 1 -- 172.21.15.6:0/1139196953 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd708003680 con 0x5569cc1e5c10 2024-04-26T01:45:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.043+0000 7fd711600700 1 -- 172.21.15.6:0/1139196953 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70800f6c0 con 0x5569cc1e5c10 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.044+0000 7fd72d953b80 1 -- 172.21.15.6:0/1139196953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 msgr2=0x5569cc1f4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.044+0000 7fd72d953b80 1 --2- 172.21.15.6:0/1139196953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 0x5569cc1f4980 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd708006010 tx=0x7fd7080096a0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.044+0000 7fd72d953b80 1 -- 172.21.15.6:0/1139196953 shutdown_connections 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.044+0000 7fd72d953b80 1 --2- 172.21.15.6:0/1139196953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.044+0000 7fd72d953b80 1 --2- 172.21.15.6:0/1139196953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e52f0 0x5569cc1e56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.044+0000 7fd72d953b80 1 --2- 172.21.15.6:0/1139196953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 0x5569cc1f4980 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.044+0000 7fd72d953b80 1 -- 172.21.15.6:0/1139196953 >> 172.21.15.6:0/1139196953 conn(0x5569cc114550 msgr2=0x5569cc1e1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.044+0000 7fd72d953b80 1 -- 172.21.15.6:0/1139196953 shutdown_connections 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.044+0000 7fd72d953b80 1 -- 172.21.15.6:0/1139196953 wait complete. 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.045+0000 7fd72d953b80 1 -- start start 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.045+0000 7fd72d953b80 1 Processor -- start 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.045+0000 7fd72d953b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e52f0 0x5569cc1f4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.045+0000 7fd72d953b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 0x5569cc1f1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.045+0000 7fd72d953b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd72d953b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5569cc1f7950 con 0x5569cc1e5c10 2024-04-26T01:45:40.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd72d953b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5569cc0f2810 con 0x5569cc1f4ec0 2024-04-26T01:45:40.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd72d953b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5569cc1f7ad0 con 0x5569cc1e52f0 2024-04-26T01:45:40.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd72d953b80 10 client.0 _ll_register_callbacks cb 0x5569cc0d16d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:40.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd71b400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 0x5569cc1f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd71b400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 0x5569cc1f1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46822/0 (socket says 172.21.15.6:46822) 2024-04-26T01:45:40.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd71be00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e52f0 0x5569cc1f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd71b400700 1 -- 172.21.15.6:0/4111823935 learned_addr learned my addr 172.21.15.6:0/4111823935 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd721000700 1 --2- 172.21.15.6:0/4111823935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd71b400700 1 -- 172.21.15.6:0/4111823935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e52f0 msgr2=0x5569cc1f4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd71b400700 1 --2- 172.21.15.6:0/4111823935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e52f0 0x5569cc1f4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd71b400700 1 -- 172.21.15.6:0/4111823935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 msgr2=0x5569cc1f1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd71b400700 1 --2- 172.21.15.6:0/4111823935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.046+0000 7fd71b400700 1 -- 172.21.15.6:0/4111823935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569cc086880 con 0x5569cc1e5c10 2024-04-26T01:45:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.047+0000 7fd71b400700 1 --2- 172.21.15.6:0/4111823935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 0x5569cc1f1930 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd708003f40 tx=0x7fd70800bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b705d7e7e52ee9fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.047+0000 7fd706000700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:45:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.047+0000 7fd706000700 1 -- 172.21.15.6:0/4111823935 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd708027030 con 0x5569cc1e5c10 2024-04-26T01:45:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.047+0000 7fd706000700 1 -- 172.21.15.6:0/4111823935 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fd708005480 con 0x5569cc1e5c10 2024-04-26T01:45:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.047+0000 7fd72d953b80 1 -- 172.21.15.6:0/4111823935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd708003680 con 0x5569cc1e5c10 2024-04-26T01:45:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.047+0000 7fd706000700 1 -- 172.21.15.6:0/4111823935 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd708005680 con 0x5569cc1e5c10 2024-04-26T01:45:40.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.047+0000 7fd72d953b80 10 client.14889 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:40.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.047+0000 7fd72d953b80 1 -- 172.21.15.6:0/4111823935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5569cc0cfaa0 con 0x5569cc1e5c10 2024-04-26T01:45:40.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.048+0000 7fd706000700 1 -- 172.21.15.6:0/4111823935 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd708003680 con 0x5569cc1e5c10 2024-04-26T01:45:40.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.048+0000 7fd706000700 1 -- 172.21.15.6:0/4111823935 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fd708021030 con 0x5569cc1e5c10 2024-04-26T01:45:40.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.048+0000 7fd72d953b80 10 client.14889 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:40.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.048+0000 7fd72d953b80 10 client.14889 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:45:40.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.048+0000 7fd72d953b80 20 client.14889 populate_metadata read hostname 'smithi006' 2024-04-26T01:45:40.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.048+0000 7fd72d953b80 1 --2- 172.21.15.6:0/4111823935 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x5569cc2033b0 0x5569cc205550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.048+0000 7fd72d953b80 4 client.14889 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:40.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.048+0000 7fd72d953b80 1 -- 172.21.15.6:0/4111823935 --> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569cc146630 con 0x5569cc2033b0 2024-04-26T01:45:40.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.049+0000 7fd71be00700 1 --2- 172.21.15.6:0/4111823935 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x5569cc2033b0 0x5569cc205550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.049+0000 7fd71be00700 1 --2- 172.21.15.6:0/4111823935 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x5569cc2033b0 0x5569cc205550 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=65d50600f1a7effd server_cookie=ea595c97f83a4e75 in_seq=0 out_seq=0 2024-04-26T01:45:40.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.049+0000 7fd706000700 10 client.14889 ms_handle_connect on v2:172.21.15.29:6838/2325582359 2024-04-26T01:45:40.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.050+0000 7fd706000700 1 -- 172.21.15.6:0/4111823935 <== mds.1 v2:172.21.15.29:6838/2325582359 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5569cc146630 con 0x5569cc2033b0 2024-04-26T01:45:40.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.050+0000 7fd706000700 10 client.14889 handle_command_reply: tid=0 2024-04-26T01:45:40.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.050+0000 7fd72d953b80 1 client.14889 shutdown 2024-04-26T01:45:40.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.050+0000 7fd72d953b80 1 -- 172.21.15.6:0/4111823935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 msgr2=0x5569cc1f1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.050+0000 7fd72d953b80 1 --2- 172.21.15.6:0/4111823935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 0x5569cc1f1930 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd708003f40 tx=0x7fd70800bfb0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.050+0000 7fd72d953b80 1 -- 172.21.15.6:0/4111823935 shutdown_connections 2024-04-26T01:45:40.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.050+0000 7fd72d953b80 1 --2- 172.21.15.6:0/4111823935 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x5569cc2033b0 0x5569cc205550 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.050+0000 7fd72d953b80 1 --2- 172.21.15.6:0/4111823935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.050+0000 7fd72d953b80 1 --2- 172.21.15.6:0/4111823935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e52f0 0x5569cc1f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.050+0000 7fd72d953b80 1 --2- 172.21.15.6:0/4111823935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e5c10 0x5569cc1f1930 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.050+0000 7fd72d953b80 1 -- 172.21.15.6:0/4111823935 >> 172.21.15.6:0/4111823935 conn(0x5569cc114550 msgr2=0x5569cc1e0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:40.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.050+0000 7fd706000700 0 client.14889 ms_handle_reset on v2:172.21.15.29:6838/2325582359 2024-04-26T01:45:40.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.051+0000 7fd72d953b80 1 -- 172.21.15.6:0/4111823935 shutdown_connections 2024-04-26T01:45:40.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.051+0000 7fd72d953b80 1 -- 172.21.15.6:0/4111823935 wait complete. 2024-04-26T01:45:40.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.051+0000 7fd72d953b80 20 client.14889 trim_cache size 0 max 16384 2024-04-26T01:45:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.073+0000 7fd72d953b80 1 Processor -- start 2024-04-26T01:45:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.074+0000 7fd72d953b80 1 -- start start 2024-04-26T01:45:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.074+0000 7fd72d953b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e52f0 0x5569cc1ed1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.074+0000 7fd72d953b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e5c10 0x5569cc1f4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.074+0000 7fd72d953b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.074+0000 7fd72d953b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5569cc203890 con 0x5569cc1e52f0 2024-04-26T01:45:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.074+0000 7fd72d953b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5569cc203f60 con 0x5569cc1f4ec0 2024-04-26T01:45:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.074+0000 7fd72d953b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5569cc204630 con 0x5569cc1e5c10 2024-04-26T01:45:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.074+0000 7fd721000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.074+0000 7fd71b400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e5c10 0x5569cc1f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.074+0000 7fd721000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52076/0 (socket says 172.21.15.6:52076) 2024-04-26T01:45:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.074+0000 7fd721000700 1 -- 172.21.15.6:0/1987897937 learned_addr learned my addr 172.21.15.6:0/1987897937 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.074+0000 7fd71be00700 1 --2- 172.21.15.6:0/1987897937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e52f0 0x5569cc1ed1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.075+0000 7fd721000700 1 -- 172.21.15.6:0/1987897937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e5c10 msgr2=0x5569cc1f4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.075+0000 7fd721000700 1 --2- 172.21.15.6:0/1987897937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e5c10 0x5569cc1f4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.075+0000 7fd721000700 1 -- 172.21.15.6:0/1987897937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e52f0 msgr2=0x5569cc1ed1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.075+0000 7fd721000700 1 --2- 172.21.15.6:0/1987897937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e52f0 0x5569cc1ed1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.075+0000 7fd721000700 1 -- 172.21.15.6:0/1987897937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569cc1d9020 con 0x5569cc1f4ec0 2024-04-26T01:45:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.075+0000 7fd721000700 1 --2- 172.21.15.6:0/1987897937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f1e20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd714009040 tx=0x7fd71401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f24b89c4b16daee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.076+0000 7fd710c00700 1 -- 172.21.15.6:0/1987897937 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714003a00 con 0x5569cc1f4ec0 2024-04-26T01:45:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.076+0000 7fd710c00700 1 -- 172.21.15.6:0/1987897937 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd714028070 con 0x5569cc1f4ec0 2024-04-26T01:45:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.076+0000 7fd710c00700 1 -- 172.21.15.6:0/1987897937 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714008510 con 0x5569cc1f4ec0 2024-04-26T01:45:40.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.076+0000 7fd72d953b80 1 -- 172.21.15.6:0/1987897937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 msgr2=0x5569cc1f1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.076+0000 7fd72d953b80 1 --2- 172.21.15.6:0/1987897937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f1e20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd714009040 tx=0x7fd71401f000 comp rx=0 tx=0).stop 2024-04-26T01:45:40.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.077+0000 7fd72d953b80 1 -- 172.21.15.6:0/1987897937 shutdown_connections 2024-04-26T01:45:40.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.077+0000 7fd72d953b80 1 --2- 172.21.15.6:0/1987897937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1f4ec0 0x5569cc1f1e20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.077+0000 7fd72d953b80 1 --2- 172.21.15.6:0/1987897937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e52f0 0x5569cc1ed1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.077+0000 7fd72d953b80 1 --2- 172.21.15.6:0/1987897937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1e5c10 0x5569cc1f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.077+0000 7fd72d953b80 1 -- 172.21.15.6:0/1987897937 >> 172.21.15.6:0/1987897937 conn(0x5569cc114550 msgr2=0x5569cc1f1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.077+0000 7fd72d953b80 1 -- 172.21.15.6:0/1987897937 shutdown_connections 2024-04-26T01:45:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.077+0000 7fd72d953b80 1 -- 172.21.15.6:0/1987897937 wait complete. 2024-04-26T01:45:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.077+0000 7fd72d953b80 1 -- start start 2024-04-26T01:45:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.077+0000 7fd72d953b80 1 Processor -- start 2024-04-26T01:45:40.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.078+0000 7fd72d953b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e52f0 0x5569cc1ec180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.078+0000 7fd72d953b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1e5c10 0x5569cc1f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.078+0000 7fd72d953b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1f4ec0 0x5569cc20e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.078+0000 7fd72d953b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5569cc2053d0 con 0x5569cc1e52f0 2024-04-26T01:45:40.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.078+0000 7fd72d953b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5569cc204d00 con 0x5569cc1e5c10 2024-04-26T01:45:40.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.078+0000 7fd72d953b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5569cc205aa0 con 0x5569cc1f4ec0 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.078+0000 7fd72d953b80 10 client.0 _ll_register_callbacks cb 0x5569cc0d16d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.078+0000 7fd71be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e52f0 0x5569cc1ec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.078+0000 7fd721000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1f4ec0 0x5569cc20e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.078+0000 7fd71be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e52f0 0x5569cc1ec180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46828/0 (socket says 172.21.15.6:46828) 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.078+0000 7fd71be00700 1 -- 172.21.15.6:0/3434368283 learned_addr learned my addr 172.21.15.6:0/3434368283 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.078+0000 7fd71b400700 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1e5c10 0x5569cc1f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.079+0000 7fd71be00700 1 -- 172.21.15.6:0/3434368283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1f4ec0 msgr2=0x5569cc20e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.079+0000 7fd71be00700 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1f4ec0 0x5569cc20e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.079+0000 7fd71be00700 1 -- 172.21.15.6:0/3434368283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1e5c10 msgr2=0x5569cc1f62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.079+0000 7fd71be00700 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1e5c10 0x5569cc1f62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.079+0000 7fd71be00700 1 -- 172.21.15.6:0/3434368283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569cc20f1f0 con 0x5569cc1e52f0 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.079+0000 7fd71b400700 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1e5c10 0x5569cc1f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.079+0000 7fd721000700 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1f4ec0 0x5569cc20e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:40.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.079+0000 7fd71be00700 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e52f0 0x5569cc1ec180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00c910 tx=0x7fd70c09bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5a4f72f4c58c2c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:40.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.079+0000 7fd706000700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:45:40.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.079+0000 7fd706000700 1 -- 172.21.15.6:0/3434368283 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c01c040 con 0x5569cc1e52f0 2024-04-26T01:45:40.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.079+0000 7fd706000700 1 -- 172.21.15.6:0/3434368283 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fd70c09a840 con 0x5569cc1e52f0 2024-04-26T01:45:40.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.079+0000 7fd72d953b80 1 -- 172.21.15.6:0/3434368283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd714028070 con 0x5569cc1e52f0 2024-04-26T01:45:40.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.080+0000 7fd706000700 1 -- 172.21.15.6:0/3434368283 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd70c0a08f0 con 0x5569cc1e52f0 2024-04-26T01:45:40.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.080+0000 7fd72d953b80 10 client.14895 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:40.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.080+0000 7fd72d953b80 1 -- 172.21.15.6:0/3434368283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5569cc26ff40 con 0x5569cc1e52f0 2024-04-26T01:45:40.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.080+0000 7fd706000700 1 -- 172.21.15.6:0/3434368283 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd714028070 con 0x5569cc1e52f0 2024-04-26T01:45:40.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.080+0000 7fd706000700 1 -- 172.21.15.6:0/3434368283 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fd70c0a4d20 con 0x5569cc1e52f0 2024-04-26T01:45:40.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.080+0000 7fd72d953b80 10 client.14895 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:40.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.080+0000 7fd72d953b80 10 client.14895 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:45:40.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.080+0000 7fd72d953b80 20 client.14895 populate_metadata read hostname 'smithi006' 2024-04-26T01:45:40.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.080+0000 7fd72d953b80 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x5569cc274b80 0x5569cc273890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.080+0000 7fd72d953b80 4 client.14895 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:45:40.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.080+0000 7fd72d953b80 1 -- 172.21.15.6:0/3434368283 --> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] -- command(tid 0: {"prefix": "status"}) -- 0x5569cc1406d0 con 0x5569cc274b80 2024-04-26T01:45:40.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.081+0000 7fd71b400700 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x5569cc274b80 0x5569cc273890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.081+0000 7fd71b400700 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x5569cc274b80 0x5569cc273890 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f6bef3f20a0b2930 server_cookie=17df5f3eb44f1f5a in_seq=0 out_seq=0 2024-04-26T01:45:40.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.081+0000 7fd706000700 10 client.14895 ms_handle_connect on v2:172.21.15.29:6838/2325582359 2024-04-26T01:45:40.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.081+0000 7fd706000700 1 -- 172.21.15.6:0/3434368283 <== mds.1 v2:172.21.15.29:6838/2325582359 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5569cc1406d0 con 0x5569cc274b80 2024-04-26T01:45:40.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.081+0000 7fd706000700 10 client.14895 handle_command_reply: tid=0 2024-04-26T01:45:40.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd72d953b80 1 client.14895 shutdown 2024-04-26T01:45:40.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd72d953b80 1 -- 172.21.15.6:0/3434368283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e52f0 msgr2=0x5569cc1ec180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd72d953b80 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e52f0 0x5569cc1ec180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00c910 tx=0x7fd70c09bef0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd72d953b80 1 -- 172.21.15.6:0/3434368283 shutdown_connections 2024-04-26T01:45:40.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd72d953b80 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5569cc1f4ec0 0x5569cc20e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd72d953b80 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5569cc1e52f0 0x5569cc1ec180 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd72d953b80 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x5569cc274b80 0x5569cc273890 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd72d953b80 1 --2- 172.21.15.6:0/3434368283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569cc1e5c10 0x5569cc1f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd72d953b80 1 -- 172.21.15.6:0/3434368283 >> 172.21.15.6:0/3434368283 conn(0x5569cc114550 msgr2=0x5569cc1e7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:40.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd706000700 0 client.14895 ms_handle_reset on v2:172.21.15.29:6838/2325582359 2024-04-26T01:45:40.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd72d953b80 1 -- 172.21.15.6:0/3434368283 shutdown_connections 2024-04-26T01:45:40.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd72d953b80 1 -- 172.21.15.6:0/3434368283 wait complete. 2024-04-26T01:45:40.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.082+0000 7fd72d953b80 20 client.14895 trim_cache size 0 max 16384 2024-04-26T01:45:40.082 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:45:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:45:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 1, 2024-04-26T01:45:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 24512, 2024-04-26T01:45:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-26T01:45:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-26T01:45:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-26T01:45:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 196.621732141, 2024-04-26T01:45:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 25, 2024-04-26T01:45:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 95, 2024-04-26T01:45:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T01:45:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 231.191216341 2024-04-26T01:45:40.083 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:45:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.083+0000 7fd723000700 1 -- 172.21.15.6:0/4030608924 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd7000a3cc0 msgr2=0x7fd7000a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.083+0000 7fd723000700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd7000a3cc0 0x7fd7000a6180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd714009ec0 tx=0x7fd71400fc40 comp rx=0 tx=0).stop 2024-04-26T01:45:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.083+0000 7fd723000700 1 -- 172.21.15.6:0/4030608924 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd71c101940 msgr2=0x7fd71c199230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.083+0000 7fd723000700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd71c101940 0x7fd71c199230 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00e9f0 tx=0x7fd70c00ed00 comp rx=0 tx=0).stop 2024-04-26T01:45:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.083+0000 7fd723000700 1 -- 172.21.15.6:0/4030608924 shutdown_connections 2024-04-26T01:45:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.083+0000 7fd723000700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd71c107990 0x7fd71c19dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.083+0000 7fd723000700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fd7000a3cc0 0x7fd7000a6180 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.083+0000 7fd723000700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd71c101940 0x7fd71c199230 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.083+0000 7fd723000700 1 --2- 172.21.15.6:0/4030608924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd71c102300 0x7fd71c199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.083+0000 7fd723000700 1 -- 172.21.15.6:0/4030608924 >> 172.21.15.6:0/4030608924 conn(0x7fd71c0788f0 msgr2=0x7fd71c06aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:40.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.083+0000 7fd723000700 1 -- 172.21.15.6:0/4030608924 shutdown_connections 2024-04-26T01:45:40.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:40.083+0000 7fd723000700 1 -- 172.21.15.6:0/4030608924 wait complete. 2024-04-26T01:45:40.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:40 smithi188 ceph-mon[32751]: pgmap v441: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:40.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:40 smithi029 ceph-mon[32775]: pgmap v441: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:40.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:40 smithi006 ceph-mon[28419]: pgmap v441: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:45:40.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T01:45:41.404 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:42.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:42 smithi188 ceph-mon[32751]: pgmap v442: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:42.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:42 smithi029 ceph-mon[32775]: pgmap v442: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:42.739 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:42 smithi006 ceph-mon[28419]: pgmap v442: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:45:43.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.587+0000 7f4018200700 1 -- 172.21.15.6:0/3953233630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010102a90 msgr2=0x7f4010102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:43.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.587+0000 7f4018200700 1 --2- 172.21.15.6:0/3953233630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010102a90 0x7f4010102f10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009990 tx=0x7f3ffc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.587+0000 7f4018200700 1 -- 172.21.15.6:0/3953233630 shutdown_connections 2024-04-26T01:45:43.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.587+0000 7f4018200700 1 --2- 172.21.15.6:0/3953233630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4010103450 0x7f401010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.587+0000 7f4018200700 1 --2- 172.21.15.6:0/3953233630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4010108a50 0x7f4010108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.587+0000 7f4018200700 1 --2- 172.21.15.6:0/3953233630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010102a90 0x7f4010102f10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.587+0000 7f4018200700 1 -- 172.21.15.6:0/3953233630 >> 172.21.15.6:0/3953233630 conn(0x7f4010079ba0 msgr2=0x7f4010079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:43.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.587+0000 7f4018200700 1 -- 172.21.15.6:0/3953233630 shutdown_connections 2024-04-26T01:45:43.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.587+0000 7f4018200700 1 -- 172.21.15.6:0/3953233630 wait complete. 2024-04-26T01:45:43.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.588+0000 7f4018200700 1 Processor -- start 2024-04-26T01:45:43.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.588+0000 7f4018200700 1 -- start start 2024-04-26T01:45:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.589+0000 7f4018200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4010102a90 0x7f4010199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.589+0000 7f4018200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010103450 0x7f40101996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.589+0000 7f4018200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4010108a50 0x7f401019da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.589+0000 7f4018200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4010112c50 con 0x7f4010102a90 2024-04-26T01:45:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.589+0000 7f4018200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4010112ad0 con 0x7f4010103450 2024-04-26T01:45:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.589+0000 7f4018200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4010112dd0 con 0x7f4010108a50 2024-04-26T01:45:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.589+0000 7f4015800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4010102a90 0x7f4010199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.589+0000 7f4016200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4010108a50 0x7f401019da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.589+0000 7f4015800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4010102a90 0x7f4010199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46842/0 (socket says 172.21.15.6:46842) 2024-04-26T01:45:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.589+0000 7f4014e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010103450 0x7f40101996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.589+0000 7f4015800700 1 -- 172.21.15.6:0/851975987 learned_addr learned my addr 172.21.15.6:0/851975987 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.590+0000 7f4016200700 1 -- 172.21.15.6:0/851975987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010103450 msgr2=0x7f40101996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.590+0000 7f4016200700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010103450 0x7f40101996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.590+0000 7f4016200700 1 -- 172.21.15.6:0/851975987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4010102a90 msgr2=0x7f4010199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.590+0000 7f4016200700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4010102a90 0x7f4010199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.590+0000 7f4016200700 1 -- 172.21.15.6:0/851975987 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401019e190 con 0x7f4010108a50 2024-04-26T01:45:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.590+0000 7f4015800700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4010102a90 0x7f4010199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.590+0000 7f4016200700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4010108a50 0x7f401019da50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f400800efc0 tx=0x7f400800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.590+0000 7f400e000700 1 -- 172.21.15.6:0/851975987 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40080092e0 con 0x7f4010108a50 2024-04-26T01:45:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.590+0000 7f400e000700 1 -- 172.21.15.6:0/851975987 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4008004bb0 con 0x7f4010108a50 2024-04-26T01:45:43.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.590+0000 7f400e000700 1 -- 172.21.15.6:0/851975987 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008004020 con 0x7f4010108a50 2024-04-26T01:45:43.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.590+0000 7f4018200700 1 -- 172.21.15.6:0/851975987 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f401019e420 con 0x7f4010108a50 2024-04-26T01:45:43.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.590+0000 7f4018200700 1 -- 172.21.15.6:0/851975987 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40101a5d00 con 0x7f4010108a50 2024-04-26T01:45:43.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.595+0000 7f400e000700 1 -- 172.21.15.6:0/851975987 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f400801b030 con 0x7f4010108a50 2024-04-26T01:45:43.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.596+0000 7f4022c7db80 1 Processor -- start 2024-04-26T01:45:43.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.597+0000 7f4022c7db80 1 -- start start 2024-04-26T01:45:43.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.597+0000 7f400e000700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3ff80a3cc0 0x7f3ff80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.597+0000 7f400e000700 1 -- 172.21.15.6:0/851975987 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f40080cfea0 con 0x7f4010108a50 2024-04-26T01:45:43.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.597+0000 7f4014e00700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3ff80a3cc0 0x7f3ff80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.597+0000 7f4022c7db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55716138eb20 0x55716138ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.597+0000 7f4022c7db80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x557161388dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.597+0000 7f4022c7db80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557161389300 0x557161389720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.597+0000 7f4022c7db80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5571612d6e40 con 0x557161389300 2024-04-26T01:45:43.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.597+0000 7f4022c7db80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55716127d6e0 con 0x55716138f440 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.597+0000 7f4022c7db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5571612d2e00 con 0x55716138eb20 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.598+0000 7f4014e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x557161388dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.598+0000 7f4014e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x557161388dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52102/0 (socket says 172.21.15.6:52102) 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.598+0000 7f4014e00700 1 -- 172.21.15.6:0/1611718787 learned_addr learned my addr 172.21.15.6:0/1611718787 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.598+0000 7f4015800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55716138eb20 0x55716138ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.598+0000 7f4016200700 1 --2- 172.21.15.6:0/1611718787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557161389300 0x557161389720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.598+0000 7f4015800700 1 -- 172.21.15.6:0/1611718787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 msgr2=0x557161388dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.598+0000 7f4015800700 1 --2- 172.21.15.6:0/1611718787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x557161388dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.598+0000 7f4015800700 1 -- 172.21.15.6:0/1611718787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557161389300 msgr2=0x557161389720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.598+0000 7f4015800700 1 --2- 172.21.15.6:0/1611718787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557161389300 0x557161389720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.598+0000 7f4015800700 1 -- 172.21.15.6:0/1611718787 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557161273630 con 0x55716138eb20 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.598+0000 7f4014e00700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3ff80a3cc0 0x7f3ff80a6180 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009990 tx=0x7f3ffc0051c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.599+0000 7f4015800700 1 --2- 172.21.15.6:0/1611718787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55716138eb20 0x55716138ef00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f40040114f0 tx=0x7f4004011800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a3917a2cb9be99 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.599+0000 7f4002a00700 1 -- 172.21.15.6:0/1611718787 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004004230 con 0x55716138eb20 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.599+0000 7f4002a00700 1 -- 172.21.15.6:0/1611718787 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40040043d0 con 0x55716138eb20 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.599+0000 7f4002a00700 1 -- 172.21.15.6:0/1611718787 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400400e8f0 con 0x55716138eb20 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.600+0000 7f4022c7db80 1 -- 172.21.15.6:0/1611718787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55716138eb20 msgr2=0x55716138ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.600+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1611718787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55716138eb20 0x55716138ef00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f40040114f0 tx=0x7f4004011800 comp rx=0 tx=0).stop 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.600+0000 7f4022c7db80 1 -- 172.21.15.6:0/1611718787 shutdown_connections 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.600+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1611718787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557161389300 0x557161389720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.600+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1611718787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55716138eb20 0x55716138ef00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.600+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1611718787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x557161388dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.600+0000 7f4022c7db80 1 -- 172.21.15.6:0/1611718787 >> 172.21.15.6:0/1611718787 conn(0x5571612b2f00 msgr2=0x55716137f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.600+0000 7f4022c7db80 1 -- 172.21.15.6:0/1611718787 shutdown_connections 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.600+0000 7f4022c7db80 1 -- 172.21.15.6:0/1611718787 wait complete. 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.600+0000 7f4022c7db80 1 -- start start 2024-04-26T01:45:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.601+0000 7f4022c7db80 1 Processor -- start 2024-04-26T01:45:43.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.601+0000 7f4022c7db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557161389300 0x5571613950b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.601+0000 7f4022c7db80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613923b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.601+0000 7f4022c7db80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x5571613928f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.601+0000 7f4022c7db80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x557161395d40 con 0x55716138eb20 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.601+0000 7f4022c7db80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557161290810 con 0x55716138f440 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.601+0000 7f4022c7db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557161395ec0 con 0x557161389300 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.601+0000 7f4022c7db80 10 client.0 _ll_register_callbacks cb 0x55716126f6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.601+0000 7f4014e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613923b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.602+0000 7f4015800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557161389300 0x5571613950b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.602+0000 7f4014e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613923b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46862/0 (socket says 172.21.15.6:46862) 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.602+0000 7f4014e00700 1 -- 172.21.15.6:0/1729349691 learned_addr learned my addr 172.21.15.6:0/1729349691 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.602+0000 7f4016200700 1 --2- 172.21.15.6:0/1729349691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x5571613928f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.602+0000 7f4014e00700 1 -- 172.21.15.6:0/1729349691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557161389300 msgr2=0x5571613950b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.602+0000 7f4014e00700 1 --2- 172.21.15.6:0/1729349691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557161389300 0x5571613950b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.602+0000 7f4014e00700 1 -- 172.21.15.6:0/1729349691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 msgr2=0x5571613928f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.602+0000 7f4014e00700 1 --2- 172.21.15.6:0/1729349691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x5571613928f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.602+0000 7f4014e00700 1 -- 172.21.15.6:0/1729349691 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557161224880 con 0x55716138eb20 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.602+0000 7f4014e00700 1 --2- 172.21.15.6:0/1729349691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613923b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc000c00 tx=0x7f3ffc009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa4016fc0262e5c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.602+0000 7f3ff7400700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.603+0000 7f3ff7400700 1 -- 172.21.15.6:0/1729349691 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc036060 con 0x55716138eb20 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.603+0000 7f3ff7400700 1 -- 172.21.15.6:0/1729349691 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f3ffc003880 con 0x55716138eb20 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.603+0000 7f4022c7db80 1 -- 172.21.15.6:0/1729349691 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40040043d0 con 0x55716138eb20 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.603+0000 7f3ff7400700 1 -- 172.21.15.6:0/1729349691 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3ffc021040 con 0x55716138eb20 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.603+0000 7f4022c7db80 10 client.14907 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.603+0000 7f4022c7db80 1 -- 172.21.15.6:0/1729349691 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55716126daa0 con 0x55716138eb20 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.603+0000 7f3ff7400700 1 -- 172.21.15.6:0/1729349691 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40040043d0 con 0x55716138eb20 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.604+0000 7f3ff7400700 1 -- 172.21.15.6:0/1729349691 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f3ffc031430 con 0x55716138eb20 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.604+0000 7f4022c7db80 10 client.14907 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.604+0000 7f4022c7db80 10 client.14907 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:45:43.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.604+0000 7f4022c7db80 20 client.14907 populate_metadata read hostname 'smithi006' 2024-04-26T01:45:43.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.604+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1729349691 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5571613a2a60 0x5571613a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.604+0000 7f4022c7db80 4 client.14907 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:43.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.604+0000 7f4022c7db80 1 -- 172.21.15.6:0/1729349691 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571612e4630 con 0x5571613a2a60 2024-04-26T01:45:43.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.604+0000 7f4015800700 1 --2- 172.21.15.6:0/1729349691 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5571613a2a60 0x5571613a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.605+0000 7f4015800700 1 --2- 172.21.15.6:0/1729349691 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5571613a2a60 0x5571613a3ed0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0e6d6f9a2be82db server_cookie=dd6dc4312d237e6e in_seq=0 out_seq=0 2024-04-26T01:45:43.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.605+0000 7f3ff7400700 10 client.14907 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:45:43.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f3ff7400700 1 -- 172.21.15.6:0/1729349691 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5571612e4630 con 0x5571613a2a60 2024-04-26T01:45:43.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f3ff7400700 10 client.14907 handle_command_reply: tid=0 2024-04-26T01:45:43.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f4022c7db80 1 client.14907 shutdown 2024-04-26T01:45:43.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f4022c7db80 1 -- 172.21.15.6:0/1729349691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 msgr2=0x5571613923b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1729349691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613923b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc000c00 tx=0x7f3ffc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f4022c7db80 1 -- 172.21.15.6:0/1729349691 shutdown_connections 2024-04-26T01:45:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1729349691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x5571613928f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1729349691 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5571613a2a60 0x5571613a3ed0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1729349691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557161389300 0x5571613950b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1729349691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613923b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f4022c7db80 1 -- 172.21.15.6:0/1729349691 >> 172.21.15.6:0/1729349691 conn(0x5571612b2f00 msgr2=0x55716137ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f3ff7400700 0 client.14907 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:45:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.606+0000 7f4022c7db80 1 -- 172.21.15.6:0/1729349691 shutdown_connections 2024-04-26T01:45:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.607+0000 7f4022c7db80 1 -- 172.21.15.6:0/1729349691 wait complete. 2024-04-26T01:45:43.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.607+0000 7f4022c7db80 20 client.14907 trim_cache size 0 max 16384 2024-04-26T01:45:43.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.630+0000 7f4022c7db80 1 Processor -- start 2024-04-26T01:45:43.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.630+0000 7f4022c7db80 1 -- start start 2024-04-26T01:45:43.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.630+0000 7f4022c7db80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557161389300 0x55716137f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.630+0000 7f4022c7db80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613ad330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.630+0000 7f4022c7db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55716138f440 0x5571613ad870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.630+0000 7f4022c7db80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5571613a2210 con 0x55716138eb20 2024-04-26T01:45:43.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.630+0000 7f4022c7db80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5571613a28e0 con 0x557161389300 2024-04-26T01:45:43.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.630+0000 7f4022c7db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5571613a2fb0 con 0x55716138f440 2024-04-26T01:45:43.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.631+0000 7f4014e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613ad330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.631+0000 7f4014e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613ad330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46870/0 (socket says 172.21.15.6:46870) 2024-04-26T01:45:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.631+0000 7f4014e00700 1 -- 172.21.15.6:0/1884334543 learned_addr learned my addr 172.21.15.6:0/1884334543 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.631+0000 7f4015800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557161389300 0x55716137f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.631+0000 7f4016200700 1 --2- 172.21.15.6:0/1884334543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55716138f440 0x5571613ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.631+0000 7f4014e00700 1 -- 172.21.15.6:0/1884334543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55716138f440 msgr2=0x5571613ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.631+0000 7f4014e00700 1 --2- 172.21.15.6:0/1884334543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55716138f440 0x5571613ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.631+0000 7f4014e00700 1 -- 172.21.15.6:0/1884334543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557161389300 msgr2=0x55716137f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.631+0000 7f4014e00700 1 --2- 172.21.15.6:0/1884334543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557161389300 0x55716137f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.631+0000 7f4014e00700 1 -- 172.21.15.6:0/1884334543 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571613a34a0 con 0x55716138eb20 2024-04-26T01:45:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.631+0000 7f4015800700 1 --2- 172.21.15.6:0/1884334543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557161389300 0x55716137f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.632+0000 7f4014e00700 1 --2- 172.21.15.6:0/1884334543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613ad330 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc0318d0 tx=0x7f3ffc0316c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d161cb5091525e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.632+0000 7f4002000700 1 -- 172.21.15.6:0/1884334543 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc036040 con 0x55716138eb20 2024-04-26T01:45:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.632+0000 7f4002000700 1 -- 172.21.15.6:0/1884334543 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ffc02d040 con 0x55716138eb20 2024-04-26T01:45:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.632+0000 7f4002000700 1 -- 172.21.15.6:0/1884334543 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc00f6c0 con 0x55716138eb20 2024-04-26T01:45:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.633+0000 7f4022c7db80 1 -- 172.21.15.6:0/1884334543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 msgr2=0x5571613ad330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.633+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1884334543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613ad330 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc0318d0 tx=0x7f3ffc0316c0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.633+0000 7f4022c7db80 1 -- 172.21.15.6:0/1884334543 shutdown_connections 2024-04-26T01:45:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.633+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1884334543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55716138f440 0x5571613ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.633+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1884334543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557161389300 0x55716137f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.633+0000 7f4022c7db80 1 --2- 172.21.15.6:0/1884334543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613ad330 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.633+0000 7f4022c7db80 1 -- 172.21.15.6:0/1884334543 >> 172.21.15.6:0/1884334543 conn(0x5571612b2f00 msgr2=0x557161390cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.633+0000 7f4022c7db80 1 -- 172.21.15.6:0/1884334543 shutdown_connections 2024-04-26T01:45:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.633+0000 7f4022c7db80 1 -- 172.21.15.6:0/1884334543 wait complete. 2024-04-26T01:45:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.633+0000 7f4022c7db80 1 -- start start 2024-04-26T01:45:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.634+0000 7f4022c7db80 1 Processor -- start 2024-04-26T01:45:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.634+0000 7f4022c7db80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557161389300 0x5571613ad530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.634+0000 7f4022c7db80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613a8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.634+0000 7f4022c7db80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x5571613a8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.634+0000 7f4022c7db80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5571613a3d50 con 0x55716138eb20 2024-04-26T01:45:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.634+0000 7f4022c7db80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5571613a3680 con 0x55716138f440 2024-04-26T01:45:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.634+0000 7f4022c7db80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5571613a4420 con 0x557161389300 2024-04-26T01:45:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.634+0000 7f4022c7db80 10 client.0 _ll_register_callbacks cb 0x55716126f6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.635+0000 7f4014e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613a8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.635+0000 7f4016200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x5571613a8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.635+0000 7f4016200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x5571613a8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52138/0 (socket says 172.21.15.6:52138) 2024-04-26T01:45:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.635+0000 7f4015800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557161389300 0x5571613ad530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.635+0000 7f4016200700 1 -- 172.21.15.6:0/3957648802 learned_addr learned my addr 172.21.15.6:0/3957648802 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.635+0000 7f4016200700 1 -- 172.21.15.6:0/3957648802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557161389300 msgr2=0x5571613ad530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.635+0000 7f4016200700 1 --2- 172.21.15.6:0/3957648802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557161389300 0x5571613ad530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.635+0000 7f4016200700 1 -- 172.21.15.6:0/3957648802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 msgr2=0x5571613a8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.635+0000 7f4016200700 1 --2- 172.21.15.6:0/3957648802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613a8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.635+0000 7f4016200700 1 -- 172.21.15.6:0/3957648802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571613a8e80 con 0x55716138f440 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.635+0000 7f4016200700 1 --2- 172.21.15.6:0/3957648802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x5571613a8840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f400800ef90 tx=0x7f40080a03a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79e75f1d228f5c51 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.635+0000 7f3ff7400700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.636+0000 7f3ff7400700 1 -- 172.21.15.6:0/3957648802 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40080a4ac0 con 0x55716138f440 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.636+0000 7f4022c7db80 1 -- 172.21.15.6:0/3957648802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ffc02d040 con 0x55716138f440 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.636+0000 7f3ff7400700 1 -- 172.21.15.6:0/3957648802 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f40080b0b80 con 0x55716138f440 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.636+0000 7f3ff7400700 1 -- 172.21.15.6:0/3957648802 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f400809c870 con 0x55716138f440 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.636+0000 7f4022c7db80 10 client.25016 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.636+0000 7f4022c7db80 1 -- 172.21.15.6:0/3957648802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55716140aab0 con 0x55716138f440 2024-04-26T01:45:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.636+0000 7f3ff7400700 1 -- 172.21.15.6:0/3957648802 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ffc02d040 con 0x55716138f440 2024-04-26T01:45:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.636+0000 7f3ff7400700 1 -- 172.21.15.6:0/3957648802 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f4008097ec0 con 0x55716138f440 2024-04-26T01:45:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.636+0000 7f4022c7db80 10 client.25016 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.637+0000 7f4022c7db80 10 client.25016 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:45:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.637+0000 7f4022c7db80 20 client.25016 populate_metadata read hostname 'smithi006' 2024-04-26T01:45:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.637+0000 7f4022c7db80 1 --2- 172.21.15.6:0/3957648802 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55716140f600 0x55716140e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.637+0000 7f4022c7db80 4 client.25016 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:45:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.637+0000 7f4022c7db80 1 -- 172.21.15.6:0/3957648802 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "status"}) -- 0x5571612de6d0 con 0x55716140f600 2024-04-26T01:45:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.637+0000 7f4015800700 1 --2- 172.21.15.6:0/3957648802 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55716140f600 0x55716140e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.637+0000 7f4015800700 1 --2- 172.21.15.6:0/3957648802 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55716140f600 0x55716140e310 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6d018e9b4ea6ee3 server_cookie=8c258149b402570b in_seq=0 out_seq=0 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.637+0000 7f3ff7400700 10 client.25016 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f3ff7400700 1 -- 172.21.15.6:0/3957648802 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5571612de6d0 con 0x55716140f600 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f3ff7400700 10 client.25016 handle_command_reply: tid=0 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f4022c7db80 1 client.25016 shutdown 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f4022c7db80 1 -- 172.21.15.6:0/3957648802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 msgr2=0x5571613a8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f4022c7db80 1 --2- 172.21.15.6:0/3957648802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x5571613a8840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f400800ef90 tx=0x7f40080a03a0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f4022c7db80 1 -- 172.21.15.6:0/3957648802 shutdown_connections 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f4022c7db80 1 --2- 172.21.15.6:0/3957648802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55716138f440 0x5571613a8840 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f4022c7db80 1 --2- 172.21.15.6:0/3957648802 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55716140f600 0x55716140e310 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f4022c7db80 1 --2- 172.21.15.6:0/3957648802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557161389300 0x5571613ad530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f4022c7db80 1 --2- 172.21.15.6:0/3957648802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55716138eb20 0x5571613a8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f3ff7400700 0 client.25016 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:45:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f4022c7db80 1 -- 172.21.15.6:0/3957648802 >> 172.21.15.6:0/3957648802 conn(0x5571612b2f00 msgr2=0x55716138f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f4022c7db80 1 -- 172.21.15.6:0/3957648802 shutdown_connections 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f4022c7db80 1 -- 172.21.15.6:0/3957648802 wait complete. 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.638+0000 7f4022c7db80 20 client.25016 trim_cache size 0 max 16384 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 24523, 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 215.23374632100001, 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 25, 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 95, 2024-04-26T01:45:43.639 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T01:45:43.640 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 249.215763173 2024-04-26T01:45:43.640 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:45:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.639+0000 7f4018200700 1 -- 172.21.15.6:0/851975987 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3ff80a3cc0 msgr2=0x7f3ff80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.639+0000 7f4018200700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3ff80a3cc0 0x7f3ff80a6180 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009990 tx=0x7f3ffc0051c0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.639+0000 7f4018200700 1 -- 172.21.15.6:0/851975987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4010108a50 msgr2=0x7f401019da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.639+0000 7f4018200700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4010108a50 0x7f401019da50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f400800efc0 tx=0x7f400800c3a0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.639+0000 7f4018200700 1 -- 172.21.15.6:0/851975987 shutdown_connections 2024-04-26T01:45:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.639+0000 7f4018200700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3ff80a3cc0 0x7f3ff80a6180 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.639+0000 7f4018200700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4010108a50 0x7f401019da50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.639+0000 7f4018200700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4010102a90 0x7f4010199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.639+0000 7f4018200700 1 --2- 172.21.15.6:0/851975987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010103450 0x7f40101996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.639+0000 7f4018200700 1 -- 172.21.15.6:0/851975987 >> 172.21.15.6:0/851975987 conn(0x7f4010079ba0 msgr2=0x7f4010101990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.640+0000 7f4018200700 1 -- 172.21.15.6:0/851975987 shutdown_connections 2024-04-26T01:45:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:43.640+0000 7f4018200700 1 -- 172.21.15.6:0/851975987 wait complete. 2024-04-26T01:45:44.440 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:44 smithi006 ceph-mon[28419]: pgmap v443: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:44.522 INFO:teuthology.run_tasks:Running task sequential... 2024-04-26T01:45:44.534 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-26T01:45:44.534 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-26T01:45:44.534 INFO:teuthology.task.print:Enabling mgr modules 2024-04-26T01:45:44.534 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-26T01:45:44.535 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T01:45:44.535 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi006.front.sepia.ceph.com 2024-04-26T01:45:44.535 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-26T01:45:44.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:44 smithi188 ceph-mon[32751]: pgmap v443: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:44.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:44 smithi029 ceph-mon[32775]: pgmap v443: 97 pgs: 97 active+clean; 644 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.755+0000 7fa11fc00640 1 -- 172.21.15.6:0/1988483215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa118108650 msgr2=0x7fa118108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.755+0000 7fa11fc00640 1 --2- 172.21.15.6:0/1988483215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa118108650 0x7fa118108a70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa108009be0 tx=0x7fa10802a6d0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.756+0000 7fa11fc00640 1 -- 172.21.15.6:0/1988483215 shutdown_connections 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.756+0000 7fa11fc00640 1 --2- 172.21.15.6:0/1988483215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1181097f0 0x7fa118109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.756+0000 7fa11fc00640 1 --2- 172.21.15.6:0/1988483215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa11810a1b0 0x7fa118110a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.756+0000 7fa11fc00640 1 --2- 172.21.15.6:0/1988483215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa118108650 0x7fa118108a70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.756+0000 7fa11fc00640 1 -- 172.21.15.6:0/1988483215 >> 172.21.15.6:0/1988483215 conn(0x7fa1180775a0 msgr2=0x7fa1180799e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.756+0000 7fa11fc00640 1 -- 172.21.15.6:0/1988483215 shutdown_connections 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.756+0000 7fa11fc00640 1 -- 172.21.15.6:0/1988483215 wait complete. 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.757+0000 7fa11fc00640 1 Processor -- start 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.757+0000 7fa11fc00640 1 -- start start 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.757+0000 7fa11fc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa118108650 0x7fa11819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.757+0000 7fa11fc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1181097f0 0x7fa11819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.757+0000 7fa11fc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa11810a1b0 0x7fa1181a5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.757+0000 7fa11fc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa118114120 con 0x7fa118108650 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.757+0000 7fa11fc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa118113fa0 con 0x7fa1181097f0 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.757+0000 7fa11fc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181142a0 con 0x7fa11810a1b0 2024-04-26T01:45:44.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.757+0000 7fa11d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa118108650 0x7fa11819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.757+0000 7fa10fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1181097f0 0x7fa11819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.757+0000 7fa11d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa118108650 0x7fa11819e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46890/0 (socket says 172.21.15.6:46890) 2024-04-26T01:45:44.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.758+0000 7fa11d200640 1 -- 172.21.15.6:0/81243570 learned_addr learned my addr 172.21.15.6:0/81243570 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:44.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.758+0000 7fa10fe00640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1181097f0 0x7fa11819ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52160/0 (socket says 172.21.15.6:52160) 2024-04-26T01:45:44.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.758+0000 7fa11dc00640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa11810a1b0 0x7fa1181a5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.758+0000 7fa10fe00640 1 -- 172.21.15.6:0/81243570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa11810a1b0 msgr2=0x7fa1181a5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.758+0000 7fa10fe00640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa11810a1b0 0x7fa1181a5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.758+0000 7fa10fe00640 1 -- 172.21.15.6:0/81243570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa118108650 msgr2=0x7fa11819e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.758+0000 7fa10fe00640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa118108650 0x7fa11819e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.758+0000 7fa10fe00640 1 -- 172.21.15.6:0/81243570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1181a6440 con 0x7fa1181097f0 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.758+0000 7fa11d200640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa118108650 0x7fa11819e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.758+0000 7fa10fe00640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1181097f0 0x7fa11819ed60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa100005fd0 tx=0x7fa10000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.759+0000 7fa116000640 1 -- 172.21.15.6:0/81243570 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10000cd00 con 0x7fa1181097f0 2024-04-26T01:45:44.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.759+0000 7fa116000640 1 -- 172.21.15.6:0/81243570 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1000035f0 con 0x7fa1181097f0 2024-04-26T01:45:44.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.759+0000 7fa116000640 1 -- 172.21.15.6:0/81243570 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa100010690 con 0x7fa1181097f0 2024-04-26T01:45:44.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.759+0000 7fa11fc00640 1 -- 172.21.15.6:0/81243570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1181a6730 con 0x7fa1181097f0 2024-04-26T01:45:44.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.759+0000 7fa11fc00640 1 -- 172.21.15.6:0/81243570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1181a6cd0 con 0x7fa1181097f0 2024-04-26T01:45:44.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.760+0000 7fa11fc00640 1 -- 172.21.15.6:0/81243570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa11810f9f0 con 0x7fa1181097f0 2024-04-26T01:45:44.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.762+0000 7fa116000640 1 -- 172.21.15.6:0/81243570 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fa100003a60 con 0x7fa1181097f0 2024-04-26T01:45:44.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.763+0000 7fa116000640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa0fc0a3c20 0x7fa0fc0a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.764+0000 7fa116000640 1 -- 172.21.15.6:0/81243570 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fa1000d05f0 con 0x7fa1181097f0 2024-04-26T01:45:44.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.764+0000 7fa11d200640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa0fc0a3c20 0x7fa0fc0a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.765+0000 7fa11d200640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa0fc0a3c20 0x7fa0fc0a60e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa108004af0 tx=0x7fa108004a80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:44.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:44.768+0000 7fa116000640 1 -- 172.21.15.6:0/81243570 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa100009510 con 0x7fa1181097f0 2024-04-26T01:45:45.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.033+0000 7fa11fc00640 1 -- 172.21.15.6:0/81243570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fa118062b50 con 0x7fa1181097f0 2024-04-26T01:45:45.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.382+0000 7fa116000640 1 -- 172.21.15.6:0/81243570 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 19) ==== 138662+0+0 (secure 0 0 0) 0x7fa100098ea0 con 0x7fa1181097f0 2024-04-26T01:45:45.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.383+0000 7fa116000640 1 -- 172.21.15.6:0/81243570 <== mon.2 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fa10009cf30 con 0x7fa1181097f0 2024-04-26T01:45:45.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.387+0000 7fa11fc00640 1 -- 172.21.15.6:0/81243570 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa0fc0a3c20 msgr2=0x7fa0fc0a60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:45.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.387+0000 7fa11fc00640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa0fc0a3c20 0x7fa0fc0a60e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa108004af0 tx=0x7fa108004a80 comp rx=0 tx=0).stop 2024-04-26T01:45:45.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.387+0000 7fa11fc00640 1 -- 172.21.15.6:0/81243570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1181097f0 msgr2=0x7fa11819ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:45.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.387+0000 7fa11fc00640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1181097f0 0x7fa11819ed60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa100005fd0 tx=0x7fa10000ee50 comp rx=0 tx=0).stop 2024-04-26T01:45:45.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.387+0000 7fa11fc00640 1 -- 172.21.15.6:0/81243570 shutdown_connections 2024-04-26T01:45:45.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.387+0000 7fa11fc00640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa11810a1b0 0x7fa1181a5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.387+0000 7fa11fc00640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1181097f0 0x7fa11819ed60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.387+0000 7fa11fc00640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fa0fc0a3c20 0x7fa0fc0a60e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.387+0000 7fa11fc00640 1 --2- 172.21.15.6:0/81243570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa118108650 0x7fa11819e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.387+0000 7fa11fc00640 1 -- 172.21.15.6:0/81243570 >> 172.21.15.6:0/81243570 conn(0x7fa1180775a0 msgr2=0x7fa11810cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:45.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.388+0000 7fa11fc00640 1 -- 172.21.15.6:0/81243570 shutdown_connections 2024-04-26T01:45:45.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.388+0000 7fa11fc00640 1 -- 172.21.15.6:0/81243570 wait complete. 2024-04-26T01:45:45.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-26T01:45:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.587+0000 7f3c42600640 1 -- 172.21.15.6:0/2524254017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c3c105db0 msgr2=0x7f3c3c10c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.587+0000 7f3c42600640 1 --2- 172.21.15.6:0/2524254017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c3c105db0 0x7f3c3c10c640 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009ca0 tx=0x7f3c3402a8c0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.587+0000 7f3c42600640 1 -- 172.21.15.6:0/2524254017 shutdown_connections 2024-04-26T01:45:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.587+0000 7f3c42600640 1 --2- 172.21.15.6:0/2524254017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c3c1053f0 0x7f3c3c105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.587+0000 7f3c42600640 1 --2- 172.21.15.6:0/2524254017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c3c105db0 0x7f3c3c10c640 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.587+0000 7f3c42600640 1 --2- 172.21.15.6:0/2524254017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c3c104200 0x7f3c3c104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.587+0000 7f3c42600640 1 -- 172.21.15.6:0/2524254017 >> 172.21.15.6:0/2524254017 conn(0x7f3c3c0ff770 msgr2=0x7f3c3c101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:45.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.587+0000 7f3c42600640 1 -- 172.21.15.6:0/2524254017 shutdown_connections 2024-04-26T01:45:45.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.587+0000 7f3c42600640 1 -- 172.21.15.6:0/2524254017 wait complete. 2024-04-26T01:45:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.588+0000 7f3c42600640 1 Processor -- start 2024-04-26T01:45:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.588+0000 7f3c42600640 1 -- start start 2024-04-26T01:45:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.588+0000 7f3c42600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c3c104200 0x7f3c3c19a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.588+0000 7f3c42600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c3c1053f0 0x7f3c3c19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.588+0000 7f3c42600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c3c105db0 0x7f3c3c1a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.588+0000 7f3c42600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3c10fd60 con 0x7f3c3c105db0 2024-04-26T01:45:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.588+0000 7f3c42600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3c10fbe0 con 0x7f3c3c104200 2024-04-26T01:45:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.588+0000 7f3c42600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3c10fee0 con 0x7f3c3c1053f0 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.589+0000 7f3c3b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c3c104200 0x7f3c3c19a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.589+0000 7f3c3b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c3c104200 0x7f3c3c19a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52184/0 (socket says 172.21.15.6:52184) 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.589+0000 7f3c3b400640 1 -- 172.21.15.6:0/657519200 learned_addr learned my addr 172.21.15.6:0/657519200 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.589+0000 7f3c32a00640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c3c1053f0 0x7f3c3c19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.589+0000 7f3c3be00640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c3c105db0 0x7f3c3c1a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.589+0000 7f3c3b400640 1 -- 172.21.15.6:0/657519200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c3c1053f0 msgr2=0x7f3c3c19a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.589+0000 7f3c3b400640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c3c1053f0 0x7f3c3c19a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.589+0000 7f3c3b400640 1 -- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c3c105db0 msgr2=0x7f3c3c1a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.589+0000 7f3c3b400640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c3c105db0 0x7f3c3c1a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.589+0000 7f3c3b400640 1 -- 172.21.15.6:0/657519200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c3c1a1ff0 con 0x7f3c3c104200 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.589+0000 7f3c32a00640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c3c1053f0 0x7f3c3c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.589+0000 7f3c3be00640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c3c105db0 0x7f3c3c1a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.590+0000 7f3c3b400640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c3c104200 0x7f3c3c19a3d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00e9c0 tx=0x7f3c2c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:45.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.590+0000 7f3c38c00640 1 -- 172.21.15.6:0/657519200 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c00cd80 con 0x7f3c3c104200 2024-04-26T01:45:45.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.590+0000 7f3c38c00640 1 -- 172.21.15.6:0/657519200 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c2c0035f0 con 0x7f3c3c104200 2024-04-26T01:45:45.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.590+0000 7f3c42600640 1 -- 172.21.15.6:0/657519200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c3c1a22e0 con 0x7f3c3c104200 2024-04-26T01:45:45.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.590+0000 7f3c38c00640 1 -- 172.21.15.6:0/657519200 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c010690 con 0x7f3c3c104200 2024-04-26T01:45:45.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.590+0000 7f3c42600640 1 -- 172.21.15.6:0/657519200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c3c1a2880 con 0x7f3c3c104200 2024-04-26T01:45:45.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.591+0000 7f3c42600640 1 -- 172.21.15.6:0/657519200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c3c10b5a0 con 0x7f3c3c104200 2024-04-26T01:45:45.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.595+0000 7f3c38c00640 1 -- 172.21.15.6:0/657519200 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 138662+0+0 (secure 0 0 0) 0x7f3c2c0037a0 con 0x7f3c3c104200 2024-04-26T01:45:45.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.596+0000 7f3c38c00640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3c200a3c70 0x7f3c200a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:45.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.597+0000 7f3c32a00640 1 -- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3c200a3c70 msgr2=0x7f3c200a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/809793211 2024-04-26T01:45:45.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.597+0000 7f3c38c00640 1 -- 172.21.15.6:0/657519200 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f3c2c0d0890 con 0x7f3c3c104200 2024-04-26T01:45:45.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.597+0000 7f3c32a00640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3c200a3c70 0x7f3c200a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:45:45.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.601+0000 7f3c38c00640 1 -- 172.21.15.6:0/657519200 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c2c09d1d0 con 0x7f3c3c104200 2024-04-26T01:45:45.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:45 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/81243570' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T01:45:45.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:45 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T01:45:45.640 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: ignoring --setuser ceph since I am not root 2024-04-26T01:45:45.640 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: ignoring --setgroup ceph since I am not root 2024-04-26T01:45:45.640 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:45 smithi188 ceph-mgr[34775]: -- 172.21.15.188:0/3934424063 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c1419f45a0 con 0x55c1419c9000 2024-04-26T01:45:45.640 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:45.551+0000 7f9eb7c65200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:45:45.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:45 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/81243570' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T01:45:45.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:45 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T01:45:45.690 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: ignoring --setuser ceph since I am not root 2024-04-26T01:45:45.690 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: ignoring --setgroup ceph since I am not root 2024-04-26T01:45:45.690 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:45 smithi029 ceph-mgr[34169]: -- 172.21.15.29:0/4080471602 <== mon.2 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557901bc85a0 con 0x557901b9cc00 2024-04-26T01:45:45.690 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:45.559+0000 7f436f1a2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:45:45.696 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: ignoring --setuser ceph since I am not root 2024-04-26T01:45:45.696 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: ignoring --setgroup ceph since I am not root 2024-04-26T01:45:45.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:45 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/81243570' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T01:45:45.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:45 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T01:45:45.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.798+0000 7f3c32a00640 1 -- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3c200a3c70 msgr2=0x7f3c200a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/809793211 2024-04-26T01:45:45.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.798+0000 7f3c32a00640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3c200a3c70 0x7f3c200a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:45:45.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.808+0000 7f3c42600640 1 -- 172.21.15.6:0/657519200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f3c3c02d5c0 con 0x7f3c3c104200 2024-04-26T01:45:45.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.814+0000 7f3c38c00640 1 -- 172.21.15.6:0/657519200 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f3c2c09cad0 con 0x7f3c3c104200 2024-04-26T01:45:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.817+0000 7f3c42600640 1 -- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3c200a3c70 msgr2=0x7f3c200a6130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:45:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.817+0000 7f3c42600640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3c200a3c70 0x7f3c200a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.817+0000 7f3c42600640 1 -- 172.21.15.6:0/657519200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c3c104200 msgr2=0x7f3c3c19a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.817+0000 7f3c42600640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c3c104200 0x7f3c3c19a3d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00e9c0 tx=0x7f3c2c00ee80 comp rx=0 tx=0).stop 2024-04-26T01:45:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.817+0000 7f3c42600640 1 -- 172.21.15.6:0/657519200 shutdown_connections 2024-04-26T01:45:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.817+0000 7f3c42600640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f3c200a3c70 0x7f3c200a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.817+0000 7f3c42600640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c3c105db0 0x7f3c3c1a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.817+0000 7f3c42600640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c3c1053f0 0x7f3c3c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.817+0000 7f3c42600640 1 --2- 172.21.15.6:0/657519200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c3c104200 0x7f3c3c19a3d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:45.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.817+0000 7f3c42600640 1 -- 172.21.15.6:0/657519200 >> 172.21.15.6:0/657519200 conn(0x7f3c3c0ff770 msgr2=0x7f3c3c108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:45.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.817+0000 7f3c42600640 1 -- 172.21.15.6:0/657519200 shutdown_connections 2024-04-26T01:45:45.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:45.817+0000 7f3c42600640 1 -- 172.21.15.6:0/657519200 wait complete. 2024-04-26T01:45:45.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-26T01:45:45.948 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:45.697+0000 7f0198b4a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:45:46.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.024+0000 7f61b5600640 1 -- 172.21.15.6:0/1536731940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b0109e80 msgr2=0x7f61b010c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:46.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.024+0000 7f61b5600640 1 --2- 172.21.15.6:0/1536731940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b0109e80 0x7f61b010c310 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009900 tx=0x7f61a002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:45:46.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.025+0000 7f61b5600640 1 -- 172.21.15.6:0/1536731940 shutdown_connections 2024-04-26T01:45:46.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.025+0000 7f61b5600640 1 --2- 172.21.15.6:0/1536731940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b0109e80 0x7f61b010c310 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:46.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.025+0000 7f61b5600640 1 --2- 172.21.15.6:0/1536731940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61b00749b0 0x7f61b0074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:46.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.025+0000 7f61b5600640 1 --2- 172.21.15.6:0/1536731940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b0075f90 0x7f61b00743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:46.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.025+0000 7f61b5600640 1 -- 172.21.15.6:0/1536731940 >> 172.21.15.6:0/1536731940 conn(0x7f61b00fd570 msgr2=0x7f61b00ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:46.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.025+0000 7f61b5600640 1 -- 172.21.15.6:0/1536731940 shutdown_connections 2024-04-26T01:45:46.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.025+0000 7f61b5600640 1 -- 172.21.15.6:0/1536731940 wait complete. 2024-04-26T01:45:46.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.025+0000 7f61b5600640 1 Processor -- start 2024-04-26T01:45:46.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61b5600640 1 -- start start 2024-04-26T01:45:46.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61b5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b00749b0 0x7f61b019a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:46.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61b5600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61b0075f90 0x7f61b019a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61b5600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b0109e80 0x7f61b01a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61b5600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f61b010fd10 con 0x7f61b00749b0 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61b5600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f61b010fb90 con 0x7f61b0109e80 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61b5600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f61b010fe90 con 0x7f61b0075f90 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61ae000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61b0075f90 0x7f61b019a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61aea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b00749b0 0x7f61b019a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61af400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b0109e80 0x7f61b01a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61ae000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61b0075f90 0x7f61b019a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:32812/0 (socket says 172.21.15.6:32812) 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61ae000640 1 -- 172.21.15.6:0/728941699 learned_addr learned my addr 172.21.15.6:0/728941699 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.026+0000 7f61af400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b0109e80 0x7f61b01a1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52196/0 (socket says 172.21.15.6:52196) 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.027+0000 7f61ae000640 1 -- 172.21.15.6:0/728941699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b0109e80 msgr2=0x7f61b01a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.027+0000 7f61ae000640 1 --2- 172.21.15.6:0/728941699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b0109e80 0x7f61b01a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:46.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.027+0000 7f61ae000640 1 -- 172.21.15.6:0/728941699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b00749b0 msgr2=0x7f61b019a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.027+0000 7f61ae000640 1 --2- 172.21.15.6:0/728941699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b00749b0 0x7f61b019a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.027+0000 7f61ae000640 1 -- 172.21.15.6:0/728941699 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b01a1fc0 con 0x7f61b0075f90 2024-04-26T01:45:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.027+0000 7f61ae000640 1 --2- 172.21.15.6:0/728941699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61b0075f90 0x7f61b019a8e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f619800e9f0 tx=0x7f619800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.027+0000 7f61a7400640 1 -- 172.21.15.6:0/728941699 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619800cd80 con 0x7f61b0075f90 2024-04-26T01:45:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.027+0000 7f61a7400640 1 -- 172.21.15.6:0/728941699 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61980108e0 con 0x7f61b0075f90 2024-04-26T01:45:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.027+0000 7f61b5600640 1 -- 172.21.15.6:0/728941699 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b01a22b0 con 0x7f61b0075f90 2024-04-26T01:45:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.027+0000 7f61a7400640 1 -- 172.21.15.6:0/728941699 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198018980 con 0x7f61b0075f90 2024-04-26T01:45:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.027+0000 7f61b5600640 1 -- 172.21.15.6:0/728941699 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b01a2800 con 0x7f61b0075f90 2024-04-26T01:45:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.028+0000 7f61b5600640 1 -- 172.21.15.6:0/728941699 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b0108bb0 con 0x7f61b0075f90 2024-04-26T01:45:46.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.030+0000 7f61a7400640 1 -- 172.21.15.6:0/728941699 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 19) ==== 138662+0+0 (secure 0 0 0) 0x7f6198010430 con 0x7f61b0075f90 2024-04-26T01:45:46.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.032+0000 7f61a7400640 1 --2- 172.21.15.6:0/728941699 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f61940a3bf0 0x7f61940a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:46.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.032+0000 7f61aea00640 1 -- 172.21.15.6:0/728941699 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f61940a3bf0 msgr2=0x7f61940a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/809793211 2024-04-26T01:45:46.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.032+0000 7f61a7400640 1 -- 172.21.15.6:0/728941699 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f61980d07d0 con 0x7f61b0075f90 2024-04-26T01:45:46.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.032+0000 7f61aea00640 1 --2- 172.21.15.6:0/728941699 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f61940a3bf0 0x7f61940a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:45:46.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.036+0000 7f61a7400640 1 -- 172.21.15.6:0/728941699 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f619809cac0 con 0x7f61b0075f90 2024-04-26T01:45:46.139 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:45.736+0000 7f9eb7c65200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:45:46.139 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:45.838+0000 7f9eb7c65200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:45:46.177 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:45.751+0000 7f436f1a2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:45:46.177 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:45.857+0000 7f436f1a2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:45:46.177 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:46 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:46.178+0000 7f436f1a2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:45:46.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.233+0000 7f61aea00640 1 -- 172.21.15.6:0/728941699 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f61940a3bf0 msgr2=0x7f61940a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/809793211 2024-04-26T01:45:46.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.233+0000 7f61aea00640 1 --2- 172.21.15.6:0/728941699 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f61940a3bf0 0x7f61940a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:45:46.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.242+0000 7f61b5600640 1 -- 172.21.15.6:0/728941699 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f61b0062b50 con 0x7f61b0075f90 2024-04-26T01:45:46.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.251+0000 7f61a7400640 1 -- 172.21.15.6:0/728941699 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f61980a0dc0 con 0x7f61b0075f90 2024-04-26T01:45:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.253+0000 7f61b5600640 1 -- 172.21.15.6:0/728941699 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f61940a3bf0 msgr2=0x7f61940a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:45:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.253+0000 7f61b5600640 1 --2- 172.21.15.6:0/728941699 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f61940a3bf0 0x7f61940a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.253+0000 7f61b5600640 1 -- 172.21.15.6:0/728941699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61b0075f90 msgr2=0x7f61b019a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.253+0000 7f61b5600640 1 --2- 172.21.15.6:0/728941699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61b0075f90 0x7f61b019a8e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f619800e9f0 tx=0x7f619800eeb0 comp rx=0 tx=0).stop 2024-04-26T01:45:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.253+0000 7f61b5600640 1 -- 172.21.15.6:0/728941699 shutdown_connections 2024-04-26T01:45:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.253+0000 7f61b5600640 1 --2- 172.21.15.6:0/728941699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b0109e80 0x7f61b01a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.253+0000 7f61b5600640 1 --2- 172.21.15.6:0/728941699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61b0075f90 0x7f61b019a8e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.253+0000 7f61b5600640 1 --2- 172.21.15.6:0/728941699 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f61940a3bf0 0x7f61940a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.253+0000 7f61b5600640 1 --2- 172.21.15.6:0/728941699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b00749b0 0x7f61b019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.253+0000 7f61b5600640 1 -- 172.21.15.6:0/728941699 >> 172.21.15.6:0/728941699 conn(0x7f61b00fd570 msgr2=0x7f61b0105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.254+0000 7f61b5600640 1 -- 172.21.15.6:0/728941699 shutdown_connections 2024-04-26T01:45:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:46.254+0000 7f61b5600640 1 -- 172.21.15.6:0/728941699 wait complete. 2024-04-26T01:45:46.266 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:45.949+0000 7f0198b4a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:45:46.266 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:46 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:46.090+0000 7f0198b4a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:45:46.268 INFO:teuthology.run_tasks:Running task kclient... 2024-04-26T01:45:46.294 INFO:tasks.kclient:Mounting kernel clients... 2024-04-26T01:45:46.294 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-26T01:45:46.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:45:46.295 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-04-26T01:45:46.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:45:46.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link delete ceph-brx 2024-04-26T01:45:46.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:46 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T01:45:46.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:46 smithi029 ceph-mon[32775]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-04-26T01:45:46.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:46 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' 2024-04-26T01:45:46.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:46 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' 2024-04-26T01:45:46.440 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:46 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:46.267+0000 7f436f1a2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:45:46.463 INFO:teuthology.orchestra.run.smithi006.stderr:Cannot find device "ceph-brx" 2024-04-26T01:45:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:45:46.468 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-26T01:45:46.469 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T01:45:46.469 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T01:45:46.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T01:45:46.531 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:46 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T01:45:46.531 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:46 smithi006 ceph-mon[28419]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-04-26T01:45:46.532 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:46 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' 2024-04-26T01:45:46.532 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:46 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' 2024-04-26T01:45:46.532 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:46 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:46.504+0000 7f0198b4a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:45:46.639 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:46 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:46.147+0000 7f9eb7c65200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:45:46.639 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:46 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:46.231+0000 7f9eb7c65200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:45:46.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:46 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T01:45:46.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:46 smithi188 ceph-mon[32751]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-04-26T01:45:46.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:46 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' 2024-04-26T01:45:46.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:46 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' 2024-04-26T01:45:46.816 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:46 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:46.624+0000 7f0198b4a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:45:46.894 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:47.750 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:47 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:47.411+0000 7f9eb7c65200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:45:47.750 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:47 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:47.582+0000 7f9eb7c65200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:45:47.750 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:47 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:47.753+0000 7f9eb7c65200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:45:47.860 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:47 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:47.508+0000 7f436f1a2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:45:47.860 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:47 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:47.685+0000 7f436f1a2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:45:47.860 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:47 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:47.861+0000 7f436f1a2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:45:48.336 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:48.042+0000 7f9eb7c65200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:45:48.336 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:48.134+0000 7f9eb7c65200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:45:48.336 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:48.338+0000 7f9eb7c65200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:45:48.440 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:48.162+0000 7f436f1a2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:45:48.440 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:48.258+0000 7f436f1a2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:45:48.566 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:48 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:48.302+0000 7f0198b4a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:45:48.566 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:48 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:48.539+0000 7f0198b4a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:45:48.639 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:48.533+0000 7f9eb7c65200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:45:48.940 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:48.468+0000 7f436f1a2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:45:48.940 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:48.668+0000 7f436f1a2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:45:49.044 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:48 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:48.763+0000 7f0198b4a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:45:49.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.210+0000 7fb46be00700 1 -- 172.21.15.6:0/293019603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb464108a30 msgr2=0x7fb464108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:49.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.210+0000 7fb46be00700 1 --2- 172.21.15.6:0/293019603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb464108a30 0x7fb464108e10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb4580098f0 tx=0x7fb458009c00 comp rx=0 tx=0).stop 2024-04-26T01:45:49.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.210+0000 7fb46be00700 1 -- 172.21.15.6:0/293019603 shutdown_connections 2024-04-26T01:45:49.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.210+0000 7fb46be00700 1 --2- 172.21.15.6:0/293019603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb46406b0f0 0x7fb46410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:49.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.210+0000 7fb46be00700 1 --2- 172.21.15.6:0/293019603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb464108a30 0x7fb464108e10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:49.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.210+0000 7fb46be00700 1 --2- 172.21.15.6:0/293019603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb46406a750 0x7fb46406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:49.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.210+0000 7fb46be00700 1 -- 172.21.15.6:0/293019603 >> 172.21.15.6:0/293019603 conn(0x7fb4640788f0 msgr2=0x7fb464078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:49.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.211+0000 7fb46be00700 1 -- 172.21.15.6:0/293019603 shutdown_connections 2024-04-26T01:45:49.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.211+0000 7fb46be00700 1 -- 172.21.15.6:0/293019603 wait complete. 2024-04-26T01:45:49.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.212+0000 7fb46be00700 1 Processor -- start 2024-04-26T01:45:49.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.212+0000 7fb46be00700 1 -- start start 2024-04-26T01:45:49.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb46be00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb46406a750 0x7fb4641991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:49.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb46be00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb46406b0f0 0x7fb4641996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:49.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb46be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb464108a30 0x7fb46419da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:49.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb46be00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb464112c00 con 0x7fb464108a30 2024-04-26T01:45:49.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb46be00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb464112a80 con 0x7fb46406a750 2024-04-26T01:45:49.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb46be00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb464112d80 con 0x7fb46406b0f0 2024-04-26T01:45:49.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb468a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb46406b0f0 0x7fb4641996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:49.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb468a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb46406b0f0 0x7fb4641996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50636/0 (socket says 172.21.15.6:50636) 2024-04-26T01:45:49.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb468a00700 1 -- 172.21.15.6:0/1624032611 learned_addr learned my addr 172.21.15.6:0/1624032611 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:49.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb469400700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb46406a750 0x7fb4641991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:49.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb469e00700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb464108a30 0x7fb46419da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:49.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb468a00700 1 -- 172.21.15.6:0/1624032611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb46406a750 msgr2=0x7fb4641991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:49.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb468a00700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb46406a750 0x7fb4641991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:49.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb468a00700 1 -- 172.21.15.6:0/1624032611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb464108a30 msgr2=0x7fb46419da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:49.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb468a00700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb464108a30 0x7fb46419da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:49.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.213+0000 7fb468a00700 1 -- 172.21.15.6:0/1624032611 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb46419e1c0 con 0x7fb46406b0f0 2024-04-26T01:45:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.214+0000 7fb468a00700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb46406b0f0 0x7fb4641996f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb45000ec10 tx=0x7fb45000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.214+0000 7fb462000700 1 -- 172.21.15.6:0/1624032611 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45000cd40 con 0x7fb46406b0f0 2024-04-26T01:45:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.214+0000 7fb462000700 1 -- 172.21.15.6:0/1624032611 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4500044d0 con 0x7fb46406b0f0 2024-04-26T01:45:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.214+0000 7fb462000700 1 -- 172.21.15.6:0/1624032611 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb450010690 con 0x7fb46406b0f0 2024-04-26T01:45:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.214+0000 7fb46be00700 1 -- 172.21.15.6:0/1624032611 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb46419e4b0 con 0x7fb46406b0f0 2024-04-26T01:45:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.214+0000 7fb46be00700 1 -- 172.21.15.6:0/1624032611 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4641a5d30 con 0x7fb46406b0f0 2024-04-26T01:45:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.215+0000 7fb46be00700 1 -- 172.21.15.6:0/1624032611 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb464069730 con 0x7fb46406b0f0 2024-04-26T01:45:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.217+0000 7fb462000700 1 -- 172.21.15.6:0/1624032611 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 19) ==== 138662+0+0 (secure 0 0 0) 0x7fb450003e50 con 0x7fb46406b0f0 2024-04-26T01:45:49.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.218+0000 7fb462000700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb44c0a3d10 0x7fb44c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:49.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.218+0000 7fb469400700 1 -- 172.21.15.6:0/1624032611 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb44c0a3d10 msgr2=0x7fb44c0a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/809793211 2024-04-26T01:45:49.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.218+0000 7fb462000700 1 -- 172.21.15.6:0/1624032611 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7fb4500d07b0 con 0x7fb46406b0f0 2024-04-26T01:45:49.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.219+0000 7fb469400700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb44c0a3d10 0x7fb44c0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:45:49.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.222+0000 7fb462000700 1 -- 172.21.15.6:0/1624032611 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb45009d0f0 con 0x7fb46406b0f0 2024-04-26T01:45:49.297 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:49 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:49.167+0000 7f0198b4a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:45:49.297 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:49 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:49.298+0000 7f0198b4a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:45:49.389 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:49.134+0000 7f9eb7c65200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:45:49.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.420+0000 7fb469400700 1 -- 172.21.15.6:0/1624032611 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb44c0a3d10 msgr2=0x7fb44c0a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/809793211 2024-04-26T01:45:49.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.420+0000 7fb469400700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb44c0a3d10 0x7fb44c0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:45:49.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.512+0000 7fb46be00700 1 -- 172.21.15.6:0/1624032611 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb46402df80 con 0x7fb46406b0f0 2024-04-26T01:45:49.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.512+0000 7fb462000700 1 -- 172.21.15.6:0/1624032611 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7fb45009c9f0 con 0x7fb46406b0f0 2024-04-26T01:45:49.511 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T01:45:49.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.515+0000 7fb46be00700 1 -- 172.21.15.6:0/1624032611 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb44c0a3d10 msgr2=0x7fb44c0a61d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:45:49.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.515+0000 7fb46be00700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb44c0a3d10 0x7fb44c0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:49.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.515+0000 7fb46be00700 1 -- 172.21.15.6:0/1624032611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb46406b0f0 msgr2=0x7fb4641996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.515+0000 7fb46be00700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb46406b0f0 0x7fb4641996f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb45000ec10 tx=0x7fb45000ef20 comp rx=0 tx=0).stop 2024-04-26T01:45:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.515+0000 7fb46be00700 1 -- 172.21.15.6:0/1624032611 shutdown_connections 2024-04-26T01:45:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.515+0000 7fb46be00700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb464108a30 0x7fb46419da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.515+0000 7fb46be00700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb46406a750 0x7fb4641991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.515+0000 7fb46be00700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7fb44c0a3d10 0x7fb44c0a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.515+0000 7fb46be00700 1 --2- 172.21.15.6:0/1624032611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb46406b0f0 0x7fb4641996f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.515+0000 7fb46be00700 1 -- 172.21.15.6:0/1624032611 >> 172.21.15.6:0/1624032611 conn(0x7fb4640788f0 msgr2=0x7fb4640ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.516+0000 7fb46be00700 1 -- 172.21.15.6:0/1624032611 shutdown_connections 2024-04-26T01:45:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:49.516+0000 7fb46be00700 1 -- 172.21.15.6:0/1624032611 wait complete. 2024-04-26T01:45:49.562 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:49 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:49.291+0000 7f436f1a2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:45:49.816 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:49 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:49.573+0000 7f0198b4a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:45:49.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:49 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1624032611' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T01:45:49.830 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:49 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1624032611' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T01:45:49.830 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:49 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:49.731+0000 7f436f1a2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:45:49.858 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:49 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1624032611' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T01:45:49.858 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:49.562+0000 7f9eb7c65200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:45:49.858 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:49.659+0000 7f9eb7c65200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:45:50.139 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:49.861+0000 7f9eb7c65200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:45:50.190 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:49 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:49.832+0000 7f436f1a2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:45:50.190 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:50 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:50.041+0000 7f436f1a2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:45:50.297 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:49 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:49.837+0000 7f0198b4a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:45:50.367 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T01:45:50.367 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T01:45:50.367 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-04-26T01:45:50.367 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T01:45:50.367 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T01:45:50.367 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T01:45:50.367 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T01:45:50.367 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T01:45:50.368 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T01:45:50.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:45:50.368 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-04-26T01:45:50.420 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-04-26T01:45:50.421 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T01:45:50.422 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 41833sec preferred_lft 41833sec 2024-04-26T01:45:50.422 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-04-26T01:45:50.422 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-04-26T01:45:50.422 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T01:45:50.422 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-04-26T01:45:50.422 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-04-26T01:45:50.422 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T01:45:50.422 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:45:50.423 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-26T01:45:50.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add name ceph-brx type bridge 2024-04-26T01:45:50.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr flush dev ceph-brx 2024-04-26T01:45:50.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set ceph-brx up 2024-04-26T01:45:50.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T01:45:50.423 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-26T01:45:50.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:45:50.639 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:50.388+0000 7f9eb7c65200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:45:50.639 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:50.480+0000 7f9eb7c65200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:45:50.816 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:50 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:50.667+0000 7f0198b4a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:45:50.929 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:50 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:50.584+0000 7f436f1a2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:45:50.930 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:50 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:50.679+0000 7f436f1a2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:45:50.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:45:50.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:45:50.942 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T01:45:51.011 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-04-26T01:45:51.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:45:51.015 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-04-26T01:45:51.063 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:50.723+0000 7f9eb7c65200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:45:51.063 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:50.855+0000 7f9eb7c65200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:45:51.063 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:50.950+0000 7f9eb7c65200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:45:51.063 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:51 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:51.066+0000 7f9eb7c65200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:45:51.071 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-04-26T01:45:51.071 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T01:45:51.071 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T01:45:51.071 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T01:45:51.071 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T01:45:51.072 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:45:51.072 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-26T01:45:51.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T01:45:51.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T01:45:51.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T01:45:51.072 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-26T01:45:51.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:45:51.190 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:50 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:50.931+0000 7f436f1a2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:45:51.190 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:51 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:51.069+0000 7f436f1a2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:45:51.190 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:51 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:51.167+0000 7f436f1a2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:45:51.446 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:51 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:51.263+0000 7f0198b4a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:45:51.446 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:51 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:51.401+0000 7f0198b4a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:45:51.596 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:51 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:51.287+0000 7f436f1a2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:45:51.596 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:51 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:51.598+0000 7f436f1a2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:45:51.639 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:51 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:51.372+0000 7f9eb7c65200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:45:51.798 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:51 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:51.688+0000 7f0198b4a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:45:51.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:45:51.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:45:51.809 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-04-26T01:45:51.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:45:51.854 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list-id 2024-04-26T01:45:51.953 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:45:51.953 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-26T01:45:51.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T01:45:51.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-26T01:45:51.954 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-26T01:45:52.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:45:52.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:45:52.120 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-26T01:45:52.121 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:45:52.121 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-26T01:45:52.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-26T01:45:52.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T01:45:52.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-26T01:45:52.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-26T01:45:52.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-26T01:45:52.121 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-26T01:45:52.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:45:52.256 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:51 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:51.972+0000 7f9eb7c65200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:45:52.256 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 01:45:52 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z[34771]: 2024-04-26T01:45:52.188+0000 7f9eb7c65200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:45:52.329 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:52 smithi006 ceph-mon[28419]: Standby manager daemon z restarted 2024-04-26T01:45:52.330 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:52 smithi006 ceph-mon[28419]: Standby manager daemon z started 2024-04-26T01:45:52.330 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:52 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.188:0/518348383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:45:52.330 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:52 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.188:0/518348383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:45:52.330 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:52 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.188:0/518348383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:45:52.330 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:52 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.188:0/518348383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:45:52.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:45:52.410 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:45:52.410 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-26T01:45:52.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-04-26T01:45:52.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T01:45:52.410 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-26T01:45:52.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:45:52.584 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:52 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:52.401+0000 7f0198b4a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:45:52.584 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:52 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:52.539+0000 7f0198b4a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:45:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:45:52.602 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T01:45:52.602 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T01:45:52.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:52 smithi188 ceph-mon[32751]: Standby manager daemon z restarted 2024-04-26T01:45:52.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:52 smithi188 ceph-mon[32751]: Standby manager daemon z started 2024-04-26T01:45:52.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:52 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.188:0/518348383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:45:52.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:52 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.188:0/518348383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:45:52.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:52 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.188:0/518348383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:45:52.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:52 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.188:0/518348383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:45:52.641 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T01:45:52.642 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T01:45:52.642 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T01:45:52.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:52 smithi029 ceph-mon[32775]: Standby manager daemon z restarted 2024-04-26T01:45:52.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:52 smithi029 ceph-mon[32775]: Standby manager daemon z started 2024-04-26T01:45:52.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:52 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.188:0/518348383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:45:52.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:52 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.188:0/518348383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:45:52.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:52 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.188:0/518348383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:45:52.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:52 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.188:0/518348383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:45:52.690 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:52 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:52.225+0000 7f436f1a2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:45:52.691 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 01:45:52 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y[34165]: 2024-04-26T01:45:52.453+0000 7f436f1a2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:45:52.731 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-26T01:45:52.731 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-26T01:45:52.731 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/29257dce-2e4e-437a-81bf-7b5b8d792c03 2024-04-26T01:45:52.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:45:52.732 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/29257dce-2e4e-437a-81bf-7b5b8d792c03 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-26T01:45:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-26T01:45:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-04-26T01:45:52.852 INFO:teuthology.orchestra.run.smithi006.stdout:invalid new device string format 2024-04-26T01:45:53.208 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:52 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:52.882+0000 7f0198b4a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:45:53.208 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:53.074+0000 7f0198b4a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:45:53.208 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:53.209+0000 7f0198b4a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:45:53.481 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:53.374+0000 7f0198b4a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:53 smithi006 ceph-mon[28419]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-04-26T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:53 smithi006 ceph-mon[28419]: Standby manager daemon y restarted 2024-04-26T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:53 smithi006 ceph-mon[28419]: Standby manager daemon y started 2024-04-26T01:45:53.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:53 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.29:0/2443936817' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:45:53.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:53 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.29:0/2443936817' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:45:53.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:53 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.29:0/2443936817' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:45:53.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:53 smithi006 ceph-mon[28419]: from='mgr.? 172.21.15.29:0/2443936817' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:45:53.512 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-26T01:45:53.512 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-04-26T01:45:53.512 INFO:teuthology.orchestra.run.smithi006.stdout:invalid new device string format 2024-04-26T01:45:53.512 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.29:3300,172.21.15.188:3300" 2024-04-26T01:45:53.512 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.6:3300,172.21.15.29:3300,172.21.15.188:3300:/volumes/qa/sv_1/29257dce-2e4e-437a-81bf-7b5b8d792c03 2024-04-26T01:45:53.513 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,key=0,fsid=a5b72bee-036c-11ef-bc93-c7b262605968" will pass to kernel 2024-04-26T01:45:53.518 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T01:45:53.518 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-04-26T01:45:53.518 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T01:45:53.518 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T01:45:53.518 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T01:45:53.523 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T01:45:53.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:45:53.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T01:45:53.607 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T01:45:53.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:53 smithi188 ceph-mon[32751]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-04-26T01:45:53.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:53 smithi188 ceph-mon[32751]: Standby manager daemon y restarted 2024-04-26T01:45:53.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:53 smithi188 ceph-mon[32751]: Standby manager daemon y started 2024-04-26T01:45:53.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:53 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.29:0/2443936817' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:45:53.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:53 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.29:0/2443936817' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:45:53.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:53 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.29:0/2443936817' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:45:53.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:53 smithi188 ceph-mon[32751]: from='mgr.? 172.21.15.29:0/2443936817' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:45:53.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:53 smithi029 ceph-mon[32775]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-04-26T01:45:53.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:53 smithi029 ceph-mon[32775]: Standby manager daemon y restarted 2024-04-26T01:45:53.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:53 smithi029 ceph-mon[32775]: Standby manager daemon y started 2024-04-26T01:45:53.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:53 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.29:0/2443936817' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:45:53.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:53 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.29:0/2443936817' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:45:53.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:53 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.29:0/2443936817' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:45:53.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:53 smithi029 ceph-mon[32775]: from='mgr.? 172.21.15.29:0/2443936817' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:45:53.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T01:45:53.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:45:53.729 INFO:teuthology.orchestra.run.smithi006.stdout:client25058 2024-04-26T01:45:53.729 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:45:53.729 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T01:45:53.779 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T01:45:53.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T01:45:53.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:45:53.898 INFO:teuthology.orchestra.run.smithi006.stdout:client25058 2024-04-26T01:45:53.899 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T01:45:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T01:45:54.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:45:54.009 INFO:teuthology.orchestra.run.smithi006.stdout:client25058 2024-04-26T01:45:54.010 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/a5b72bee-036c-11ef-bc93-c7b262605968.client25058/status') 2024-04-26T01:45:54.066 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:53.772+0000 7f0198b4a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:45:54.139 INFO:teuthology.orchestra.run.smithi006.stdout:instance: client.25058 (3)172.21.15.6:0/392983730 2024-04-26T01:45:54.139 INFO:teuthology.orchestra.run.smithi006.stdout:blocklisted: false 2024-04-26T01:45:54.139 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records in 2024-04-26T01:45:54.139 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records out 2024-04-26T01:45:54.140 INFO:teuthology.orchestra.run.smithi006.stderr:69 bytes copied, 7.5516e-05 s, 914 kB/s 2024-04-26T01:45:54.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:45:54.151 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T01:45:54.162 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T01:45:54.162 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi006.front.sepia.ceph.com 2024-04-26T01:45:54.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-26T01:45:54.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.355+0000 7f6aa8a00640 1 -- 172.21.15.6:0/2887905863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6aa4103920 msgr2=0x7f6aa410a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:54.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.355+0000 7f6aa8a00640 1 --2- 172.21.15.6:0/2887905863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6aa4103920 0x7f6aa410a240 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009ca0 tx=0x7f6a9c02a8c0 comp rx=0 tx=0).stop 2024-04-26T01:45:54.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.355+0000 7f6aa8a00640 1 -- 172.21.15.6:0/2887905863 shutdown_connections 2024-04-26T01:45:54.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.355+0000 7f6aa8a00640 1 --2- 172.21.15.6:0/2887905863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6aa4103920 0x7f6aa410a240 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:54.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.355+0000 7f6aa8a00640 1 --2- 172.21.15.6:0/2887905863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aa4102f60 0x7f6aa41033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:54.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.355+0000 7f6aa8a00640 1 --2- 172.21.15.6:0/2887905863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa4101e10 0x7f6aa4102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:54.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.355+0000 7f6aa8a00640 1 -- 172.21.15.6:0/2887905863 >> 172.21.15.6:0/2887905863 conn(0x7f6aa40fd380 msgr2=0x7f6aa40ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:54.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.355+0000 7f6aa8a00640 1 -- 172.21.15.6:0/2887905863 shutdown_connections 2024-04-26T01:45:54.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.355+0000 7f6aa8a00640 1 -- 172.21.15.6:0/2887905863 wait complete. 2024-04-26T01:45:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.356+0000 7f6aa8a00640 1 Processor -- start 2024-04-26T01:45:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.356+0000 7f6aa8a00640 1 -- start start 2024-04-26T01:45:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.356+0000 7f6aa8a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa4101e10 0x7f6aa419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.356+0000 7f6aa8a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aa4102f60 0x7f6aa419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.356+0000 7f6aa8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6aa4103920 0x7f6aa41a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.356+0000 7f6aa8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa410d970 con 0x7f6aa4103920 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.356+0000 7f6aa8a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa410d7f0 con 0x7f6aa4101e10 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.356+0000 7f6aa8a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa410daf0 con 0x7f6aa4102f60 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.357+0000 7f6aa3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6aa4103920 0x7f6aa41a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.357+0000 7f6aa1e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aa4102f60 0x7f6aa419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.357+0000 7f6aa3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6aa4103920 0x7f6aa41a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48558/0 (socket says 172.21.15.6:48558) 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.357+0000 7f6aa2800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa4101e10 0x7f6aa419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.357+0000 7f6aa3200640 1 -- 172.21.15.6:0/3896017491 learned_addr learned my addr 172.21.15.6:0/3896017491 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.357+0000 7f6aa1e00640 1 -- 172.21.15.6:0/3896017491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa4101e10 msgr2=0x7f6aa419a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.357+0000 7f6aa1e00640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa4101e10 0x7f6aa419a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.357+0000 7f6aa1e00640 1 -- 172.21.15.6:0/3896017491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6aa4103920 msgr2=0x7f6aa41a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.357+0000 7f6aa1e00640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6aa4103920 0x7f6aa41a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.357+0000 7f6aa1e00640 1 -- 172.21.15.6:0/3896017491 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa41a1e10 con 0x7f6aa4102f60 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.357+0000 7f6aa2800640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa4101e10 0x7f6aa419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.358+0000 7f6aa1e00640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aa4102f60 0x7f6aa419a730 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000e9c0 tx=0x7f6a9000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:54.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.358+0000 7f6a9b400640 1 -- 172.21.15.6:0/3896017491 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9000ccc0 con 0x7f6aa4102f60 2024-04-26T01:45:54.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.358+0000 7f6a9b400640 1 -- 172.21.15.6:0/3896017491 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a900035f0 con 0x7f6aa4102f60 2024-04-26T01:45:54.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.358+0000 7f6a9b400640 1 -- 172.21.15.6:0/3896017491 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9000ce60 con 0x7f6aa4102f60 2024-04-26T01:45:54.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.358+0000 7f6aa8a00640 1 -- 172.21.15.6:0/3896017491 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa41a2100 con 0x7f6aa4102f60 2024-04-26T01:45:54.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.358+0000 7f6aa8a00640 1 -- 172.21.15.6:0/3896017491 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa41a2600 con 0x7f6aa4102f60 2024-04-26T01:45:54.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.360+0000 7f6aa8a00640 1 -- 172.21.15.6:0/3896017491 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aa41a28c0 con 0x7f6aa4102f60 2024-04-26T01:45:54.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.362+0000 7f6a9b400640 1 -- 172.21.15.6:0/3896017491 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 21) ==== 138662+0+0 (secure 0 0 0) 0x7f6a90004000 con 0x7f6aa4102f60 2024-04-26T01:45:54.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.363+0000 7f6a9b400640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6a880a3be0 0x7f6a880a60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:54.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.363+0000 7f6aa2800640 1 -- 172.21.15.6:0/3896017491 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6a880a3be0 msgr2=0x7f6a880a60a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/809793211 2024-04-26T01:45:54.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.363+0000 7f6a9b400640 1 -- 172.21.15.6:0/3896017491 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8298+0+0 (secure 0 0 0) 0x7f6a900d0870 con 0x7f6aa4102f60 2024-04-26T01:45:54.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.363+0000 7f6aa2800640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6a880a3be0 0x7f6a880a60a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:45:54.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.368+0000 7f6a9b400640 1 -- 172.21.15.6:0/3896017491 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a9009d1b0 con 0x7f6aa4102f60 2024-04-26T01:45:54.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.564+0000 7f6aa2800640 1 -- 172.21.15.6:0/3896017491 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6a880a3be0 msgr2=0x7f6a880a60a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/809793211 2024-04-26T01:45:54.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.564+0000 7f6aa2800640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6a880a3be0 0x7f6a880a60a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:45:54.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:54 smithi006 ceph-mon[28419]: mgrmap e21: x(active, since 12m), standbys: y, z 2024-04-26T01:45:54.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.641+0000 7f6aa8a00640 1 -- 172.21.15.6:0/3896017491 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f6aa41a2f30 con 0x7f6aa4102f60 2024-04-26T01:45:54.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:54 smithi188 ceph-mon[32751]: mgrmap e21: x(active, since 12m), standbys: y, z 2024-04-26T01:45:54.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:54 smithi029 ceph-mon[32775]: mgrmap e21: x(active, since 12m), standbys: y, z 2024-04-26T01:45:54.913 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:54 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:54.616+0000 7f0198b4a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:45:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.941+0000 7f6a9b400640 1 -- 172.21.15.6:0/3896017491 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7f6a90099120 con 0x7f6aa4102f60 2024-04-26T01:45:54.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.943+0000 7f6a9b400640 1 -- 172.21.15.6:0/3896017491 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6a880a3be0 msgr2=0x7f6a880a60a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:45:54.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:54.943+0000 7f6a9b400640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6a880a3be0 0x7f6a880a60a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.243 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:54 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:54.913+0000 7f0198b4a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:45:55.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.284+0000 7f6a9b400640 1 -- 172.21.15.6:0/3896017491 <== mon.1 v2:172.21.15.188:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v26) ==== 117+0+0 (secure 0 0 0) 0x7f6a900586b0 con 0x7f6aa4102f60 2024-04-26T01:45:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.287+0000 7f6aa8a00640 1 -- 172.21.15.6:0/3896017491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aa4102f60 msgr2=0x7f6aa419a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.287+0000 7f6aa8a00640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aa4102f60 0x7f6aa419a730 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000e9c0 tx=0x7f6a9000ee80 comp rx=0 tx=0).stop 2024-04-26T01:45:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.287+0000 7f6aa8a00640 1 -- 172.21.15.6:0/3896017491 shutdown_connections 2024-04-26T01:45:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.287+0000 7f6aa8a00640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.6:6800/809793211,v1:172.21.15.6:6801/809793211] conn(0x7f6a880a3be0 0x7f6a880a60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.287+0000 7f6aa8a00640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6aa4103920 0x7f6aa41a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.287+0000 7f6aa8a00640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aa4102f60 0x7f6aa419a730 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.287+0000 7f6aa8a00640 1 --2- 172.21.15.6:0/3896017491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa4101e10 0x7f6aa419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.287+0000 7f6aa8a00640 1 -- 172.21.15.6:0/3896017491 >> 172.21.15.6:0/3896017491 conn(0x7f6aa40fd380 msgr2=0x7f6aa4074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.287+0000 7f6aa8a00640 1 -- 172.21.15.6:0/3896017491 shutdown_connections 2024-04-26T01:45:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.287+0000 7f6aa8a00640 1 -- 172.21.15.6:0/3896017491 wait complete. 2024-04-26T01:45:55.303 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-26T01:45:55.315 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T01:45:55.325 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T01:45:55.326 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi006.front.sepia.ceph.com 2024-04-26T01:45:55.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-26T01:45:55.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3896017491' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:45:55.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:45:55.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: Active manager daemon x restarted 2024-04-26T01:45:55.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: Activating manager daemon x 2024-04-26T01:45:55.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T01:45:55.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: mgrmap e22: x(active, starting, since 0.0249069s), standbys: y, z 2024-04-26T01:45:55.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:45:55.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:45:55.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:45:55.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:45:55.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:45:55.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:45:55.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:45:55.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:45:55.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: Manager daemon x is now available 2024-04-26T01:45:55.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:55.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:45:55.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:45:55.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:45:55.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:55 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:45:55.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3896017491' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:45:55.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:45:55.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: Active manager daemon x restarted 2024-04-26T01:45:55.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: Activating manager daemon x 2024-04-26T01:45:55.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T01:45:55.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: mgrmap e22: x(active, starting, since 0.0249069s), standbys: y, z 2024-04-26T01:45:55.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:45:55.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:45:55.442 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:45:55.442 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:45:55.442 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:45:55.442 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:45:55.442 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:45:55.442 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:45:55.442 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:45:55.442 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:45:55.442 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:45:55.442 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:45:55.442 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:45:55.443 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:45:55.443 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:45:55.443 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:45:55.443 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:45:55.443 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:45:55.443 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:45:55.443 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:45:55.443 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:45:55.443 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:45:55.443 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:45:55.443 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:45:55.444 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:45:55.444 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:45:55.444 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:45:55.444 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:45:55.444 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:45:55.444 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:45:55.444 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:45:55.444 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:45:55.444 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:45:55.444 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: Manager daemon x is now available 2024-04-26T01:45:55.445 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:55.445 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:45:55.445 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:45:55.445 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:45:55.445 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:55 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:45:55.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.510+0000 7f2ecfe00640 1 -- 172.21.15.6:0/739387100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8074930 msgr2=0x7f2ec8074d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.510+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/739387100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8074930 0x7f2ec8074d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4009900 tx=0x7f2eb402a490 comp rx=0 tx=0).stop 2024-04-26T01:45:55.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.510+0000 7f2ecfe00640 1 -- 172.21.15.6:0/739387100 shutdown_connections 2024-04-26T01:45:55.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.510+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/739387100 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ec8109e90 0x7f2ec810c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.510+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/739387100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8074930 0x7f2ec8074d90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.510+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/739387100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ec8075f10 0x7f2ec8074360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.510+0000 7f2ecfe00640 1 -- 172.21.15.6:0/739387100 >> 172.21.15.6:0/739387100 conn(0x7f2ec80fd570 msgr2=0x7f2ec80ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:55.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.510+0000 7f2ecfe00640 1 -- 172.21.15.6:0/739387100 shutdown_connections 2024-04-26T01:45:55.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.511+0000 7f2ecfe00640 1 -- 172.21.15.6:0/739387100 wait complete. 2024-04-26T01:45:55.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecfe00640 1 Processor -- start 2024-04-26T01:45:55.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecfe00640 1 -- start start 2024-04-26T01:45:55.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8074930 0x7f2ec819a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ec8075f10 0x7f2ec819a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ec8109e90 0x7f2ec81a1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecfe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec810fd80 con 0x7f2ec8109e90 2024-04-26T01:45:55.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecfe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec810fc00 con 0x7f2ec8074930 2024-04-26T01:45:55.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecfe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec810ff00 con 0x7f2ec8075f10 2024-04-26T01:45:55.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecca00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ec8075f10 0x7f2ec819a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecd400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8074930 0x7f2ec819a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecca00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ec8075f10 0x7f2ec819a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50818/0 (socket says 172.21.15.6:50818) 2024-04-26T01:45:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecca00640 1 -- 172.21.15.6:0/2409269109 learned_addr learned my addr 172.21.15.6:0/2409269109 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.512+0000 7f2ecd400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8074930 0x7f2ec819a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41706/0 (socket says 172.21.15.6:41706) 2024-04-26T01:45:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.513+0000 7f2ecde00640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ec8109e90 0x7f2ec81a1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.513+0000 7f2ecd400640 1 -- 172.21.15.6:0/2409269109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ec8075f10 msgr2=0x7f2ec819a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.513+0000 7f2ecd400640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ec8075f10 0x7f2ec819a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.513+0000 7f2ecd400640 1 -- 172.21.15.6:0/2409269109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ec8109e90 msgr2=0x7f2ec81a1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.513+0000 7f2ecd400640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ec8109e90 0x7f2ec81a1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.513+0000 7f2ecd400640 1 -- 172.21.15.6:0/2409269109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec81a20a0 con 0x7f2ec8074930 2024-04-26T01:45:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.513+0000 7f2ecde00640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ec8109e90 0x7f2ec81a1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.513+0000 7f2ecd400640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8074930 0x7f2ec819a480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00e9f0 tx=0x7f2ebc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.513+0000 7f2ec6000640 1 -- 172.21.15.6:0/2409269109 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc00ccc0 con 0x7f2ec8074930 2024-04-26T01:45:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.514+0000 7f2ecfe00640 1 -- 172.21.15.6:0/2409269109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec81a2390 con 0x7f2ec8074930 2024-04-26T01:45:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.514+0000 7f2ec6000640 1 -- 172.21.15.6:0/2409269109 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ebc005020 con 0x7f2ec8074930 2024-04-26T01:45:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.514+0000 7f2ecfe00640 1 -- 172.21.15.6:0/2409269109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec81a28e0 con 0x7f2ec8074930 2024-04-26T01:45:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.514+0000 7f2ec6000640 1 -- 172.21.15.6:0/2409269109 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc004a00 con 0x7f2ec8074930 2024-04-26T01:45:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.515+0000 7f2ecfe00640 1 -- 172.21.15.6:0/2409269109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea80052c0 con 0x7f2ec8074930 2024-04-26T01:45:55.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.517+0000 7f2ec6000640 1 -- 172.21.15.6:0/2409269109 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7f2ebc0051d0 con 0x7f2ec8074930 2024-04-26T01:45:55.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.519+0000 7f2ec6000640 1 -- 172.21.15.6:0/2409269109 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f2ebc0cff60 con 0x7f2ec8074930 2024-04-26T01:45:55.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.523+0000 7f2ec6000640 1 -- 172.21.15.6:0/2409269109 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ebc09cba0 con 0x7f2ec8074930 2024-04-26T01:45:55.566 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:55.243+0000 7f00ea000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:55.566 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:55.243+0000 7f00ea000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:55.566 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:55.243+0000 7f00ea000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:55.567 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:55.243+0000 7f00ea000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:55.567 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:55.243+0000 7f00ea000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:55.567 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:55.244+0000 7f00ef000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:55.567 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:55.244+0000 7f00ef000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:55.567 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:55.244+0000 7f00ef000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:55.567 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:55.244+0000 7f00ef000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:55.567 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:45:55.244+0000 7f00ef000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:55.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3896017491' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:45:55.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:45:55.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: Active manager daemon x restarted 2024-04-26T01:45:55.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: Activating manager daemon x 2024-04-26T01:45:55.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T01:45:55.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: mgrmap e22: x(active, starting, since 0.0249069s), standbys: y, z 2024-04-26T01:45:55.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:45:55.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:45:55.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:45:55.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:45:55.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:45:55.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:45:55.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:45:55.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:45:55.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:45:55.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:45:55.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:45:55.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:45:55.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:45:55.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:45:55.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:45:55.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:45:55.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:45:55.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:45:55.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:45:55.570 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:45:55.570 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:45:55.570 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:45:55.570 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:45:55.570 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:45:55.570 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:45:55.570 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:45:55.570 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:45:55.570 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:45:55.570 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:45:55.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:45:55.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:45:55.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:45:55.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:45:55.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: Manager daemon x is now available 2024-04-26T01:45:55.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:55.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:45:55.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:45:55.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:45:55.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:55 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:45:55.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.948+0000 7f2ec6000640 1 -- 172.21.15.6:0/2409269109 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f2ebc0981f0 con 0x7f2ec8074930 2024-04-26T01:45:55.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.949+0000 7f2ec6000640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2eb00aaaa0 0x7f2eb00ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.949+0000 7f2ec6000640 1 -- 172.21.15.6:0/2409269109 --> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f2eb001dc80 con 0x7f2eb00aaaa0 2024-04-26T01:45:55.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.958+0000 7f2ecca00640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2eb00aaaa0 0x7f2eb00ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:55.962+0000 7f2ecca00640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2eb00aaaa0 0x7f2eb00ace90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2ec819b990 tx=0x7f2eb4005a20 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:56.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.060+0000 7f2ec6000640 1 -- 172.21.15.6:0/2409269109 <== mgr.25037 v2:172.21.15.6:6800/3353570381 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2eb001dc80 con 0x7f2eb00aaaa0 2024-04-26T01:45:56.058 INFO:teuthology.orchestra.run.smithi006.stdout:Schedule set for path / 2024-04-26T01:45:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.063+0000 7f2ecfe00640 1 -- 172.21.15.6:0/2409269109 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2eb00aaaa0 msgr2=0x7f2eb00ace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.063+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2eb00aaaa0 0x7f2eb00ace90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2ec819b990 tx=0x7f2eb4005a20 comp rx=0 tx=0).stop 2024-04-26T01:45:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.063+0000 7f2ecfe00640 1 -- 172.21.15.6:0/2409269109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8074930 msgr2=0x7f2ec819a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.063+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8074930 0x7f2ec819a480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00e9f0 tx=0x7f2ebc00eeb0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.063+0000 7f2ecfe00640 1 -- 172.21.15.6:0/2409269109 shutdown_connections 2024-04-26T01:45:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.063+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2eb00aaaa0 0x7f2eb00ace90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.063+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ec8109e90 0x7f2ec81a1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.063+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ec8075f10 0x7f2ec819a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.063+0000 7f2ecfe00640 1 --2- 172.21.15.6:0/2409269109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8074930 0x7f2ec819a480 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.063+0000 7f2ecfe00640 1 -- 172.21.15.6:0/2409269109 >> 172.21.15.6:0/2409269109 conn(0x7f2ec80fd570 msgr2=0x7f2ec8105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:56.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.063+0000 7f2ecfe00640 1 -- 172.21.15.6:0/2409269109 shutdown_connections 2024-04-26T01:45:56.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.063+0000 7f2ecfe00640 1 -- 172.21.15.6:0/2409269109 wait complete. 2024-04-26T01:45:56.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-26T01:45:56.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.262+0000 7f36be800640 1 -- 172.21.15.6:0/4243711533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b8107af0 msgr2=0x7f36b8109f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:56.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.262+0000 7f36be800640 1 --2- 172.21.15.6:0/4243711533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b8107af0 0x7f36b8109f40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009ca0 tx=0x7f36b002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.262+0000 7f36be800640 1 -- 172.21.15.6:0/4243711533 shutdown_connections 2024-04-26T01:45:56.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.262+0000 7f36be800640 1 --2- 172.21.15.6:0/4243711533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b8107af0 0x7f36b8109f40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.262+0000 7f36be800640 1 --2- 172.21.15.6:0/4243711533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b81051c0 0x7f36b81075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.262+0000 7f36be800640 1 --2- 172.21.15.6:0/4243711533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36b8102890 0x7f36b8104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.262+0000 7f36be800640 1 -- 172.21.15.6:0/4243711533 >> 172.21.15.6:0/4243711533 conn(0x7f36b80fc280 msgr2=0x7f36b80fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:56.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.262+0000 7f36be800640 1 -- 172.21.15.6:0/4243711533 shutdown_connections 2024-04-26T01:45:56.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.262+0000 7f36be800640 1 -- 172.21.15.6:0/4243711533 wait complete. 2024-04-26T01:45:56.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.263+0000 7f36be800640 1 Processor -- start 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.263+0000 7f36be800640 1 -- start start 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.264+0000 7f36be800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b8102890 0x7f36b8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.264+0000 7f36be800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b81051c0 0x7f36b8198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.264+0000 7f36be800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36b8107af0 0x7f36b819f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.264+0000 7f36be800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f36b810d990 con 0x7f36b8107af0 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.264+0000 7f36be800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36b810d810 con 0x7f36b8102890 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.264+0000 7f36be800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f36b810db10 con 0x7f36b81051c0 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.264+0000 7f36bd400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b8102890 0x7f36b8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.264+0000 7f36bd400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b8102890 0x7f36b8197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41716/0 (socket says 172.21.15.6:41716) 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.264+0000 7f36bd400640 1 -- 172.21.15.6:0/4093116999 learned_addr learned my addr 172.21.15.6:0/4093116999 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.264+0000 7f36bca00640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b81051c0 0x7f36b8198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.264+0000 7f36bde00640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36b8107af0 0x7f36b819f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.265+0000 7f36bd400640 1 -- 172.21.15.6:0/4093116999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b81051c0 msgr2=0x7f36b8198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:56.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.265+0000 7f36bd400640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b81051c0 0x7f36b8198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.265+0000 7f36bd400640 1 -- 172.21.15.6:0/4093116999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36b8107af0 msgr2=0x7f36b819f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:56.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.265+0000 7f36bd400640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36b8107af0 0x7f36b819f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.265+0000 7f36bd400640 1 -- 172.21.15.6:0/4093116999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b819fb60 con 0x7f36b8102890 2024-04-26T01:45:56.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.265+0000 7f36bd400640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b8102890 0x7f36b8197f40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00e9f0 tx=0x7f36ac00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:56.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.265+0000 7f36b6000640 1 -- 172.21.15.6:0/4093116999 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac00ccc0 con 0x7f36b8102890 2024-04-26T01:45:56.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.265+0000 7f36b6000640 1 -- 172.21.15.6:0/4093116999 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36ac0108e0 con 0x7f36b8102890 2024-04-26T01:45:56.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.265+0000 7f36be800640 1 -- 172.21.15.6:0/4093116999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b819fe50 con 0x7f36b8102890 2024-04-26T01:45:56.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.265+0000 7f36b6000640 1 -- 172.21.15.6:0/4093116999 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac018880 con 0x7f36b8102890 2024-04-26T01:45:56.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.265+0000 7f36be800640 1 -- 172.21.15.6:0/4093116999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b81a02b0 con 0x7f36b8102890 2024-04-26T01:45:56.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.267+0000 7f36be800640 1 -- 172.21.15.6:0/4093116999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b80fdd70 con 0x7f36b8102890 2024-04-26T01:45:56.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.268+0000 7f36b6000640 1 -- 172.21.15.6:0/4093116999 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f36ac010430 con 0x7f36b8102890 2024-04-26T01:45:56.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.270+0000 7f36b6000640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36a00a3990 0x7f36a00a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:56.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.270+0000 7f36b6000640 1 -- 172.21.15.6:0/4093116999 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f36ac0d0520 con 0x7f36b8102890 2024-04-26T01:45:56.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.270+0000 7f36bca00640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36a00a3990 0x7f36a00a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:56.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.271+0000 7f36bca00640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36a00a3990 0x7f36a00a5e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f36b8199450 tx=0x7f36a400a2d0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:56.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.274+0000 7f36b6000640 1 -- 172.21.15.6:0/4093116999 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36ac09d070 con 0x7f36b8102890 2024-04-26T01:45:56.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.518+0000 7f36be800640 1 -- 172.21.15.6:0/4093116999 --> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f36b8062430 con 0x7f36a00a3990 2024-04-26T01:45:56.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:56 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T01:45:56.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:56 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:56.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:56 smithi006 ceph-mon[28419]: mgrmap e23: x(active, since 1.04032s), standbys: y, z 2024-04-26T01:45:56.607 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:56 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T01:45:56.607 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:56 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:56.608 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:56 smithi029 ceph-mon[32775]: mgrmap e23: x(active, since 1.04032s), standbys: y, z 2024-04-26T01:45:56.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:56 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T01:45:56.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:56 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:56.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:56 smithi188 ceph-mon[32751]: mgrmap e23: x(active, since 1.04032s), standbys: y, z 2024-04-26T01:45:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.663+0000 7f36b6000640 1 -- 172.21.15.6:0/4093116999 <== mgr.25037 v2:172.21.15.6:6800/3353570381 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f36b8062430 con 0x7f36a00a3990 2024-04-26T01:45:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:Retention added to path / 2024-04-26T01:45:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.665+0000 7f36be800640 1 -- 172.21.15.6:0/4093116999 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36a00a3990 msgr2=0x7f36a00a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.665+0000 7f36be800640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36a00a3990 0x7f36a00a5e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f36b8199450 tx=0x7f36a400a2d0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.666+0000 7f36be800640 1 -- 172.21.15.6:0/4093116999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b8102890 msgr2=0x7f36b8197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.666+0000 7f36be800640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b8102890 0x7f36b8197f40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00e9f0 tx=0x7f36ac00edf0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.666+0000 7f36be800640 1 -- 172.21.15.6:0/4093116999 shutdown_connections 2024-04-26T01:45:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.666+0000 7f36be800640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36a00a3990 0x7f36a00a5e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.666+0000 7f36be800640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36b8107af0 0x7f36b819f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.666+0000 7f36be800640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b81051c0 0x7f36b8198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.666+0000 7f36be800640 1 --2- 172.21.15.6:0/4093116999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b8102890 0x7f36b8197f40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.666+0000 7f36be800640 1 -- 172.21.15.6:0/4093116999 >> 172.21.15.6:0/4093116999 conn(0x7f36b80fc280 msgr2=0x7f36b81031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.666+0000 7f36be800640 1 -- 172.21.15.6:0/4093116999 shutdown_connections 2024-04-26T01:45:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.666+0000 7f36be800640 1 -- 172.21.15.6:0/4093116999 wait complete. 2024-04-26T01:45:56.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-26T01:45:56.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.861+0000 7fa97fc00640 1 -- 172.21.15.6:0/825322946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9780784f0 msgr2=0x7fa978078d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:56.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.861+0000 7fa97fc00640 1 --2- 172.21.15.6:0/825322946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9780784f0 0x7fa978078d90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa970009ca0 tx=0x7fa97002a8b0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.861+0000 7fa97fc00640 1 -- 172.21.15.6:0/825322946 shutdown_connections 2024-04-26T01:45:56.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.861+0000 7fa97fc00640 1 --2- 172.21.15.6:0/825322946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9780784f0 0x7fa978078d90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.861+0000 7fa97fc00640 1 --2- 172.21.15.6:0/825322946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa978077b50 0x7fa978077fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.861+0000 7fa97fc00640 1 --2- 172.21.15.6:0/825322946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa978076910 0x7fa978076d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.861+0000 7fa97fc00640 1 -- 172.21.15.6:0/825322946 >> 172.21.15.6:0/825322946 conn(0x7fa9780ff770 msgr2=0x7fa978101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:56.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.861+0000 7fa97fc00640 1 -- 172.21.15.6:0/825322946 shutdown_connections 2024-04-26T01:45:56.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.861+0000 7fa97fc00640 1 -- 172.21.15.6:0/825322946 wait complete. 2024-04-26T01:45:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.862+0000 7fa97fc00640 1 Processor -- start 2024-04-26T01:45:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.862+0000 7fa97fc00640 1 -- start start 2024-04-26T01:45:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.862+0000 7fa97fc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa978076910 0x7fa97819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.862+0000 7fa97fc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa978077b50 0x7fa97819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.862+0000 7fa97fc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9780784f0 0x7fa9781a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.862+0000 7fa97fc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa978114160 con 0x7fa978077b50 2024-04-26T01:45:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.862+0000 7fa97fc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa978113fe0 con 0x7fa978076910 2024-04-26T01:45:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.862+0000 7fa97fc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa9781142e0 con 0x7fa9780784f0 2024-04-26T01:45:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.863+0000 7fa97d200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa978076910 0x7fa97819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.863+0000 7fa977e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa978077b50 0x7fa97819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.863+0000 7fa97d200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa978076910 0x7fa97819e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41736/0 (socket says 172.21.15.6:41736) 2024-04-26T01:45:56.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.863+0000 7fa97dc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9780784f0 0x7fa9781a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:56.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.863+0000 7fa97d200640 1 -- 172.21.15.6:0/3756501665 learned_addr learned my addr 172.21.15.6:0/3756501665 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:56.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.863+0000 7fa97d200640 1 -- 172.21.15.6:0/3756501665 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9780784f0 msgr2=0x7fa9781a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:56.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.863+0000 7fa97d200640 1 --2- 172.21.15.6:0/3756501665 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9780784f0 0x7fa9781a5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.863+0000 7fa97d200640 1 -- 172.21.15.6:0/3756501665 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa978077b50 msgr2=0x7fa97819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:56.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.863+0000 7fa97d200640 1 --2- 172.21.15.6:0/3756501665 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa978077b50 0x7fa97819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:56.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.863+0000 7fa97d200640 1 -- 172.21.15.6:0/3756501665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9781a63d0 con 0x7fa978076910 2024-04-26T01:45:56.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.863+0000 7fa97d200640 1 --2- 172.21.15.6:0/3756501665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa978076910 0x7fa97819e7b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa96c00d950 tx=0x7fa96c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:56.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.864+0000 7fa975600640 1 -- 172.21.15.6:0/3756501665 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96c017030 con 0x7fa978076910 2024-04-26T01:45:56.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.864+0000 7fa975600640 1 -- 172.21.15.6:0/3756501665 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa96c004020 con 0x7fa978076910 2024-04-26T01:45:56.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.864+0000 7fa97fc00640 1 -- 172.21.15.6:0/3756501665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9781a66c0 con 0x7fa978076910 2024-04-26T01:45:56.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.864+0000 7fa975600640 1 -- 172.21.15.6:0/3756501665 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96c0047a0 con 0x7fa978076910 2024-04-26T01:45:56.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.864+0000 7fa97fc00640 1 -- 172.21.15.6:0/3756501665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9781a6c10 con 0x7fa978076910 2024-04-26T01:45:56.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.865+0000 7fa97fc00640 1 -- 172.21.15.6:0/3756501665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa978110ce0 con 0x7fa978076910 2024-04-26T01:45:56.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.867+0000 7fa975600640 1 -- 172.21.15.6:0/3756501665 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7fa96c009a30 con 0x7fa978076910 2024-04-26T01:45:56.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.868+0000 7fa975600640 1 --2- 172.21.15.6:0/3756501665 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa9600a3990 0x7fa9600a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:56.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.868+0000 7fa975600640 1 -- 172.21.15.6:0/3756501665 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7fa96c0cfb90 con 0x7fa978076910 2024-04-26T01:45:56.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.868+0000 7fa977e00640 1 --2- 172.21.15.6:0/3756501665 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa9600a3990 0x7fa9600a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:56.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.869+0000 7fa977e00640 1 --2- 172.21.15.6:0/3756501665 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa9600a3990 0x7fa9600a5e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa97819fcc0 tx=0x7fa964005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:56.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:56.873+0000 7fa975600640 1 -- 172.21.15.6:0/3756501665 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa96c09c6e0 con 0x7fa978076910 2024-04-26T01:45:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.098+0000 7fa97fc00640 1 -- 172.21.15.6:0/3756501665 --> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fa978062390 con 0x7fa9600a3990 2024-04-26T01:45:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.115+0000 7fa975600640 1 -- 172.21.15.6:0/3756501665 <== mgr.25037 v2:172.21.15.6:6800/3353570381 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fa978062390 con 0x7fa9600a3990 2024-04-26T01:45:57.113 INFO:teuthology.orchestra.run.smithi006.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-26T00:00:00", "created": "2024-04-26T01:45:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-26T01:45:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.118+0000 7fa97fc00640 1 -- 172.21.15.6:0/3756501665 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa9600a3990 msgr2=0x7fa9600a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.118+0000 7fa97fc00640 1 --2- 172.21.15.6:0/3756501665 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa9600a3990 0x7fa9600a5e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa97819fcc0 tx=0x7fa964005f90 comp rx=0 tx=0).stop 2024-04-26T01:45:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.118+0000 7fa97fc00640 1 -- 172.21.15.6:0/3756501665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa978076910 msgr2=0x7fa97819e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.118+0000 7fa97fc00640 1 --2- 172.21.15.6:0/3756501665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa978076910 0x7fa97819e7b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa96c00d950 tx=0x7fa96c00de10 comp rx=0 tx=0).stop 2024-04-26T01:45:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.118+0000 7fa97fc00640 1 -- 172.21.15.6:0/3756501665 shutdown_connections 2024-04-26T01:45:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.118+0000 7fa97fc00640 1 --2- 172.21.15.6:0/3756501665 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa9600a3990 0x7fa9600a5e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.118+0000 7fa97fc00640 1 --2- 172.21.15.6:0/3756501665 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa978077b50 0x7fa97819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.118+0000 7fa97fc00640 1 --2- 172.21.15.6:0/3756501665 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9780784f0 0x7fa9781a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.118+0000 7fa97fc00640 1 --2- 172.21.15.6:0/3756501665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa978076910 0x7fa97819e7b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.118+0000 7fa97fc00640 1 -- 172.21.15.6:0/3756501665 >> 172.21.15.6:0/3756501665 conn(0x7fa9780ff770 msgr2=0x7fa978100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.118+0000 7fa97fc00640 1 -- 172.21.15.6:0/3756501665 shutdown_connections 2024-04-26T01:45:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.118+0000 7fa97fc00640 1 -- 172.21.15.6:0/3756501665 wait complete. 2024-04-26T01:45:57.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-26T01:45:57.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.343+0000 7f8c02a00640 1 -- 172.21.15.6:0/3813406471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc101e10 msgr2=0x7f8bfc102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:57.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.343+0000 7f8c02a00640 1 --2- 172.21.15.6:0/3813406471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc101e10 0x7f8bfc102230 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0009be0 tx=0x7f8bf002a6c0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.344+0000 7f8c02a00640 1 -- 172.21.15.6:0/3813406471 shutdown_connections 2024-04-26T01:45:57.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.344+0000 7f8c02a00640 1 --2- 172.21.15.6:0/3813406471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bfc103920 0x7f8bfc10a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.344+0000 7f8c02a00640 1 --2- 172.21.15.6:0/3813406471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bfc102f60 0x7f8bfc1033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.344+0000 7f8c02a00640 1 --2- 172.21.15.6:0/3813406471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc101e10 0x7f8bfc102230 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.344+0000 7f8c02a00640 1 -- 172.21.15.6:0/3813406471 >> 172.21.15.6:0/3813406471 conn(0x7f8bfc0fd380 msgr2=0x7f8bfc0ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:57.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.344+0000 7f8c02a00640 1 -- 172.21.15.6:0/3813406471 shutdown_connections 2024-04-26T01:45:57.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.344+0000 7f8c02a00640 1 -- 172.21.15.6:0/3813406471 wait complete. 2024-04-26T01:45:57.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c02a00640 1 Processor -- start 2024-04-26T01:45:57.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c02a00640 1 -- start start 2024-04-26T01:45:57.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c02a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bfc101e10 0x7f8bfc19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c02a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bfc102f60 0x7f8bfc19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c02a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc103920 0x7f8bfc1a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c02a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8bfc10d940 con 0x7f8bfc103920 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c02a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8bfc10d7c0 con 0x7f8bfc101e10 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c02a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8bfc10dac0 con 0x7f8bfc102f60 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c01600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bfc101e10 0x7f8bfc19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c00c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bfc102f60 0x7f8bfc19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c01600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bfc101e10 0x7f8bfc19a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41768/0 (socket says 172.21.15.6:41768) 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c01600640 1 -- 172.21.15.6:0/1962292532 learned_addr learned my addr 172.21.15.6:0/1962292532 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.345+0000 7f8c02000640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc103920 0x7f8bfc1a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.346+0000 7f8c00c00640 1 -- 172.21.15.6:0/1962292532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bfc101e10 msgr2=0x7f8bfc19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.346+0000 7f8c00c00640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bfc101e10 0x7f8bfc19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.346+0000 7f8c00c00640 1 -- 172.21.15.6:0/1962292532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc103920 msgr2=0x7f8bfc1a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:57.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.346+0000 7f8c00c00640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc103920 0x7f8bfc1a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.346+0000 7f8c00c00640 1 -- 172.21.15.6:0/1962292532 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bfc1a1e40 con 0x7f8bfc102f60 2024-04-26T01:45:57.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.346+0000 7f8c01600640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bfc101e10 0x7f8bfc19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:57.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.346+0000 7f8c00c00640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bfc102f60 0x7f8bfc19a760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8be8005fd0 tx=0x7f8be800ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:57.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.346+0000 7f8bfa000640 1 -- 172.21.15.6:0/1962292532 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be800ccc0 con 0x7f8bfc102f60 2024-04-26T01:45:57.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.346+0000 7f8bfa000640 1 -- 172.21.15.6:0/1962292532 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8be800ee70 con 0x7f8bfc102f60 2024-04-26T01:45:57.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.346+0000 7f8c02a00640 1 -- 172.21.15.6:0/1962292532 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bfc1a2130 con 0x7f8bfc102f60 2024-04-26T01:45:57.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.346+0000 7f8bfa000640 1 -- 172.21.15.6:0/1962292532 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be8003e10 con 0x7f8bfc102f60 2024-04-26T01:45:57.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.347+0000 7f8c02a00640 1 -- 172.21.15.6:0/1962292532 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bfc1a2680 con 0x7f8bfc102f60 2024-04-26T01:45:57.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.348+0000 7f8c02a00640 1 -- 172.21.15.6:0/1962292532 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bfc1a2940 con 0x7f8bfc102f60 2024-04-26T01:45:57.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.350+0000 7f8bfa000640 1 -- 172.21.15.6:0/1962292532 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 24) ==== 138765+0+0 (secure 0 0 0) 0x7f8be8003fb0 con 0x7f8bfc102f60 2024-04-26T01:45:57.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.352+0000 7f8bfa000640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8be40a3d00 0x7f8be40a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:57.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.352+0000 7f8bfa000640 1 -- 172.21.15.6:0/1962292532 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f8be80d0a40 con 0x7f8bfc102f60 2024-04-26T01:45:57.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.352+0000 7f8c01600640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8be40a3d00 0x7f8be40a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:57.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.353+0000 7f8c01600640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8be40a3d00 0x7f8be40a61c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0004af0 tx=0x7f8bf0004a80 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:57.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.356+0000 7f8bfa000640 1 -- 172.21.15.6:0/1962292532 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8be809d060 con 0x7f8bfc102f60 2024-04-26T01:45:57.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:57 smithi006 ceph-mon[28419]: [26/Apr/2024:01:45:55] ENGINE Bus STARTING 2024-04-26T01:45:57.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:57 smithi006 ceph-mon[28419]: [26/Apr/2024:01:45:55] ENGINE Serving on http://172.21.15.6:8765 2024-04-26T01:45:57.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:57 smithi006 ceph-mon[28419]: [26/Apr/2024:01:45:55] ENGINE Serving on https://172.21.15.6:7150 2024-04-26T01:45:57.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:57 smithi006 ceph-mon[28419]: [26/Apr/2024:01:45:55] ENGINE Bus STARTED 2024-04-26T01:45:57.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:57 smithi006 ceph-mon[28419]: from='client.25106 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:57.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:57 smithi006 ceph-mon[28419]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:57.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:57 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:57.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:57 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:57.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.603+0000 7f8c02a00640 1 -- 172.21.15.6:0/1962292532 --> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f8bfc0758a0 con 0x7f8be40a3d00 2024-04-26T01:45:57.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.613+0000 7f8bfa000640 1 -- 172.21.15.6:0/1962292532 <== mgr.25037 v2:172.21.15.6:6800/3353570381 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f8bfc0758a0 con 0x7f8be40a3d00 2024-04-26T01:45:57.611 INFO:teuthology.orchestra.run.smithi006.stdout:/ 1m 6m3h 2024-04-26T01:45:57.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.615+0000 7f8c02a00640 1 -- 172.21.15.6:0/1962292532 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8be40a3d00 msgr2=0x7f8be40a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:57.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.615+0000 7f8c02a00640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8be40a3d00 0x7f8be40a61c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0004af0 tx=0x7f8bf0004a80 comp rx=0 tx=0).stop 2024-04-26T01:45:57.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.615+0000 7f8c02a00640 1 -- 172.21.15.6:0/1962292532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bfc102f60 msgr2=0x7f8bfc19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:57.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.615+0000 7f8c02a00640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bfc102f60 0x7f8bfc19a760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8be8005fd0 tx=0x7f8be800ec20 comp rx=0 tx=0).stop 2024-04-26T01:45:57.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.615+0000 7f8c02a00640 1 -- 172.21.15.6:0/1962292532 shutdown_connections 2024-04-26T01:45:57.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.615+0000 7f8c02a00640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8be40a3d00 0x7f8be40a61c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.615+0000 7f8c02a00640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc103920 0x7f8bfc1a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.615+0000 7f8c02a00640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bfc102f60 0x7f8bfc19a760 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.615+0000 7f8c02a00640 1 --2- 172.21.15.6:0/1962292532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bfc101e10 0x7f8bfc19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:57.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.615+0000 7f8c02a00640 1 -- 172.21.15.6:0/1962292532 >> 172.21.15.6:0/1962292532 conn(0x7f8bfc0fd380 msgr2=0x7f8bfc074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:57.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.616+0000 7f8c02a00640 1 -- 172.21.15.6:0/1962292532 shutdown_connections 2024-04-26T01:45:57.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:45:57.616+0000 7f8c02a00640 1 -- 172.21.15.6:0/1962292532 wait complete. 2024-04-26T01:45:57.631 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-26T01:45:57.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:57 smithi188 ceph-mon[32751]: [26/Apr/2024:01:45:55] ENGINE Bus STARTING 2024-04-26T01:45:57.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:57 smithi188 ceph-mon[32751]: [26/Apr/2024:01:45:55] ENGINE Serving on http://172.21.15.6:8765 2024-04-26T01:45:57.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:57 smithi188 ceph-mon[32751]: [26/Apr/2024:01:45:55] ENGINE Serving on https://172.21.15.6:7150 2024-04-26T01:45:57.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:57 smithi188 ceph-mon[32751]: [26/Apr/2024:01:45:55] ENGINE Bus STARTED 2024-04-26T01:45:57.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:57 smithi188 ceph-mon[32751]: from='client.25106 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:57.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:57 smithi188 ceph-mon[32751]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:57.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:57 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:57.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:57 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:57.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:57 smithi029 ceph-mon[32775]: [26/Apr/2024:01:45:55] ENGINE Bus STARTING 2024-04-26T01:45:57.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:57 smithi029 ceph-mon[32775]: [26/Apr/2024:01:45:55] ENGINE Serving on http://172.21.15.6:8765 2024-04-26T01:45:57.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:57 smithi029 ceph-mon[32775]: [26/Apr/2024:01:45:55] ENGINE Serving on https://172.21.15.6:7150 2024-04-26T01:45:57.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:57 smithi029 ceph-mon[32775]: [26/Apr/2024:01:45:55] ENGINE Bus STARTED 2024-04-26T01:45:57.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:57 smithi029 ceph-mon[32775]: from='client.25106 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:57.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:57 smithi029 ceph-mon[32775]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:57.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:57 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:57.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:57 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:57.698 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-26T01:45:57.708 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-26T01:45:57.718 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi006.front.sepia.ceph.com 2024-04-26T01:45:57.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-26T01:45:57.722 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-26T01:45:57.734 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T01:45:57.734 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:45:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:58.554 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:58 smithi188 ceph-mon[32751]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:58.554 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:58 smithi188 ceph-mon[32751]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:58.554 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:58 smithi188 ceph-mon[32751]: from='client.25115 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:58.554 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:58 smithi188 ceph-mon[32751]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-26T01:45:58.554 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:58 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:58.554 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:58 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:58.555 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:58 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:58.555 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:58 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:58.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:58 smithi006 ceph-mon[28419]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:58.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:58 smithi006 ceph-mon[28419]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:58.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:58 smithi006 ceph-mon[28419]: from='client.25115 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:58.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:58 smithi006 ceph-mon[28419]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-26T01:45:58.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:58 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:58.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:58 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:58.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:58 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:58.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:58 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:58.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:58 smithi029 ceph-mon[32775]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:58.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:58 smithi029 ceph-mon[32775]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:58.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:58 smithi029 ceph-mon[32775]: from='client.25115 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:58.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:58 smithi029 ceph-mon[32775]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-26T01:45:58.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:58 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:58.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:58 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:58.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:58 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:58.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:58 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='client.25162 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:59.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: Unable to set osd_memory_target on smithi188 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-04-26T01:45:59.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:45:59 smithi006 ceph-mon[28419]: Unable to set osd_memory_target on smithi029 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-26T01:45:59.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='client.25162 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:59.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: Unable to set osd_memory_target on smithi188 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-04-26T01:45:59.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:45:59 smithi188 ceph-mon[32751]: Unable to set osd_memory_target on smithi029 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-26T01:45:59.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='client.25162 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:59.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: Unable to set osd_memory_target on smithi188 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.692 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.692 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.692 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.692 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.692 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:59.692 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-04-26T01:45:59.692 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:45:59 smithi029 ceph-mon[32775]: Unable to set osd_memory_target on smithi029 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-26T01:46:00.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.449+0000 7fa9f4c00700 1 -- 172.21.15.6:0/459198160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9f0108a50 msgr2=0x7fa9f0108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:00.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.449+0000 7fa9f4c00700 1 --2- 172.21.15.6:0/459198160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9f0108a50 0x7fa9f0108e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009990 tx=0x7fa9e0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:46:00.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.449+0000 7fa9f4c00700 1 -- 172.21.15.6:0/459198160 shutdown_connections 2024-04-26T01:46:00.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.449+0000 7fa9f4c00700 1 --2- 172.21.15.6:0/459198160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f0103450 0x7fa9f010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:00.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.449+0000 7fa9f4c00700 1 --2- 172.21.15.6:0/459198160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9f0108a50 0x7fa9f0108e30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:00.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.449+0000 7fa9f4c00700 1 --2- 172.21.15.6:0/459198160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f0102a90 0x7fa9f0102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:00.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.449+0000 7fa9f4c00700 1 -- 172.21.15.6:0/459198160 >> 172.21.15.6:0/459198160 conn(0x7fa9f0079ba0 msgr2=0x7fa9f0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:00.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.449+0000 7fa9f4c00700 1 -- 172.21.15.6:0/459198160 shutdown_connections 2024-04-26T01:46:00.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.449+0000 7fa9f4c00700 1 -- 172.21.15.6:0/459198160 wait complete. 2024-04-26T01:46:00.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.450+0000 7fa9f4c00700 1 Processor -- start 2024-04-26T01:46:00.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.450+0000 7fa9f4c00700 1 -- start start 2024-04-26T01:46:00.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.451+0000 7fa9f4c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9f0102a90 0x7fa9f0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:00.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.451+0000 7fa9f4c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f0103450 0x7fa9f0075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:00.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.451+0000 7fa9f4c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f0108a50 0x7fa9f00757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:00.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.451+0000 7fa9f4c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f0112c30 con 0x7fa9f0108a50 2024-04-26T01:46:00.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.451+0000 7fa9f4c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f0112ab0 con 0x7fa9f0103450 2024-04-26T01:46:00.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.451+0000 7fa9f4c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f0112db0 con 0x7fa9f0102a90 2024-04-26T01:46:00.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.451+0000 7fa9ede00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9f0102a90 0x7fa9f0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:00.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.452+0000 7fa9ede00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9f0102a90 0x7fa9f0074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:53988/0 (socket says 172.21.15.6:53988) 2024-04-26T01:46:00.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.452+0000 7fa9ede00700 1 -- 172.21.15.6:0/3994619826 learned_addr learned my addr 172.21.15.6:0/3994619826 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:00.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.452+0000 7fa9e7e00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f0103450 0x7fa9f0075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:00.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.452+0000 7fa9ee800700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f0108a50 0x7fa9f00757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:00.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.452+0000 7fa9ede00700 1 -- 172.21.15.6:0/3994619826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f0103450 msgr2=0x7fa9f0075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:00.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.452+0000 7fa9ede00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f0103450 0x7fa9f0075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:00.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.452+0000 7fa9ede00700 1 -- 172.21.15.6:0/3994619826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f0108a50 msgr2=0x7fa9f00757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:00.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.452+0000 7fa9ede00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f0108a50 0x7fa9f00757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:00.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.452+0000 7fa9ede00700 1 -- 172.21.15.6:0/3994619826 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f01a76d0 con 0x7fa9f0102a90 2024-04-26T01:46:00.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.452+0000 7fa9ee800700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f0108a50 0x7fa9f00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:00.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.452+0000 7fa9ede00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9f0102a90 0x7fa9f0074d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0000c00 tx=0x7fa9e00057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:00.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.453+0000 7fa9e6a00700 1 -- 172.21.15.6:0/3994619826 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0027070 con 0x7fa9f0102a90 2024-04-26T01:46:00.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.453+0000 7fa9e6a00700 1 -- 172.21.15.6:0/3994619826 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9e0022470 con 0x7fa9f0102a90 2024-04-26T01:46:00.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.453+0000 7fa9f4c00700 1 -- 172.21.15.6:0/3994619826 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9f01a7960 con 0x7fa9f0102a90 2024-04-26T01:46:00.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.453+0000 7fa9e6a00700 1 -- 172.21.15.6:0/3994619826 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0017650 con 0x7fa9f0102a90 2024-04-26T01:46:00.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.453+0000 7fa9f4c00700 1 -- 172.21.15.6:0/3994619826 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9f01a7e40 con 0x7fa9f0102a90 2024-04-26T01:46:00.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.454+0000 7fa9f4c00700 1 -- 172.21.15.6:0/3994619826 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9f0069730 con 0x7fa9f0102a90 2024-04-26T01:46:00.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.457+0000 7fa9e6a00700 1 -- 172.21.15.6:0/3994619826 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 25) ==== 138821+0+0 (secure 0 0 0) 0x7fa9e002a070 con 0x7fa9f0102a90 2024-04-26T01:46:00.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.459+0000 7fa9e6a00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa9d40a3e40 0x7fa9d40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:00.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.459+0000 7fa9e6a00700 1 -- 172.21.15.6:0/3994619826 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7fa9e00d17e0 con 0x7fa9f0102a90 2024-04-26T01:46:00.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.459+0000 7fa9e7e00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa9d40a3e40 0x7fa9d40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:00.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.460+0000 7fa9e7e00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa9d40a3e40 0x7fa9d40a6300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa9f007a210 tx=0x7fa9d8009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:00.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.460+0000 7fa9e6a00700 1 -- 172.21.15.6:0/3994619826 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa9e0013070 con 0x7fa9f0102a90 2024-04-26T01:46:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:00 smithi006 ceph-mon[28419]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:46:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:00 smithi006 ceph-mon[28419]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-26T01:46:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.709+0000 7fa9f4c00700 1 -- 172.21.15.6:0/3994619826 --> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa9f0062e80 con 0x7fa9d40a3e40 2024-04-26T01:46:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.722+0000 7fa9e6a00700 1 -- 172.21.15.6:0/3994619826 <== mgr.25037 v2:172.21.15.6:6800/3353570381 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257773 (secure 0 0 0) 0x7fa9f0062e80 con 0x7fa9d40a3e40 2024-04-26T01:46:00.721 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:46:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.729+0000 7fa9f4c00700 1 -- 172.21.15.6:0/3994619826 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa9d40a3e40 msgr2=0x7fa9d40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.729+0000 7fa9f4c00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa9d40a3e40 0x7fa9d40a6300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa9f007a210 tx=0x7fa9d8009260 comp rx=0 tx=0).stop 2024-04-26T01:46:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.729+0000 7fa9f4c00700 1 -- 172.21.15.6:0/3994619826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9f0102a90 msgr2=0x7fa9f0074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.729+0000 7fa9f4c00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9f0102a90 0x7fa9f0074d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0000c00 tx=0x7fa9e00057c0 comp rx=0 tx=0).stop 2024-04-26T01:46:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.730+0000 7fa9f4c00700 1 -- 172.21.15.6:0/3994619826 shutdown_connections 2024-04-26T01:46:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.730+0000 7fa9f4c00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa9d40a3e40 0x7fa9d40a6300 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.730+0000 7fa9f4c00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f0108a50 0x7fa9f00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.730+0000 7fa9f4c00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9f0102a90 0x7fa9f0074d40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.730+0000 7fa9f4c00700 1 --2- 172.21.15.6:0/3994619826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f0103450 0x7fa9f0075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.730+0000 7fa9f4c00700 1 -- 172.21.15.6:0/3994619826 >> 172.21.15.6:0/3994619826 conn(0x7fa9f0079ba0 msgr2=0x7fa9f01017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.730+0000 7fa9f4c00700 1 -- 172.21.15.6:0/3994619826 shutdown_connections 2024-04-26T01:46:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:00.730+0000 7fa9f4c00700 1 -- 172.21.15.6:0/3994619826 wait complete. 2024-04-26T01:46:00.730 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-26T01:46:00.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:00 smithi188 ceph-mon[32751]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:46:00.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:00 smithi188 ceph-mon[32751]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-26T01:46:00.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:00 smithi029 ceph-mon[32775]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:46:00.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:00 smithi029 ceph-mon[32775]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-26T01:46:01.006 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:46:00 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:46:00.678+0000 7f00c4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:46:01.007 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:46:00 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:46:00.678+0000 7f00c4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:46:01.007 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:46:00 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:46:00.678+0000 7f00c4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:46:01.007 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:46:00 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:46:00.678+0000 7f00c4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:46:01.007 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 01:46:00 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T01:46:00.678+0000 7f00c4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:46:01.667 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-26T01:45:58.956220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1192108,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1526,"num_read_kb":932,"num_write":486,"num_write_kb":4387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":876,"ondisk_log_size":876,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394412,"kb_used_data":16100,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124547412,"statfs":{"total":1151940427776,"available":1151536549888,"internally_reserved":0,"allocated":16486400,"data_stored":9670099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.978570"},"pg_stats":[{"pgid":"3.3f","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962627+0000","last_change":"2024-04-26T01:42:05.417304+0000","last_active":"2024-04-26T01:45:54.962627+0000","last_peered":"2024-04-26T01:45:54.962627+0000","last_clean":"2024-04-26T01:45:54.962627+0000","last_became_active":"2024-04-26T01:42:05.416999+0000","last_became_peered":"2024-04-26T01:42:05.416999+0000","last_unstale":"2024-04-26T01:45:54.962627+0000","last_undegraded":"2024-04-26T01:45:54.962627+0000","last_fullsized":"2024-04-26T01:45:54.962627+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:33:30.958943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"95'4","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.184448+0000","last_change":"2024-04-26T01:42:05.413089+0000","last_active":"2024-04-26T01:45:55.184448+0000","last_peered":"2024-04-26T01:45:55.184448+0000","last_clean":"2024-04-26T01:45:55.184448+0000","last_became_active":"2024-04-26T01:42:05.412776+0000","last_became_peered":"2024-04-26T01:42:05.412776+0000","last_unstale":"2024-04-26T01:45:55.184448+0000","last_undegraded":"2024-04-26T01:45:55.184448+0000","last_fullsized":"2024-04-26T01:45:55.184448+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:47.861340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.184102+0000","last_change":"2024-04-26T01:42:05.404864+0000","last_active":"2024-04-26T01:45:55.184102+0000","last_peered":"2024-04-26T01:45:55.184102+0000","last_clean":"2024-04-26T01:45:55.184102+0000","last_became_active":"2024-04-26T01:42:05.404529+0000","last_became_peered":"2024-04-26T01:42:05.404529+0000","last_unstale":"2024-04-26T01:45:55.184102+0000","last_undegraded":"2024-04-26T01:45:55.184102+0000","last_fullsized":"2024-04-26T01:45:55.184102+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:04:18.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.947186+0000","last_change":"2024-04-26T01:42:05.393582+0000","last_active":"2024-04-26T01:45:54.947186+0000","last_peered":"2024-04-26T01:45:54.947186+0000","last_clean":"2024-04-26T01:45:54.947186+0000","last_became_active":"2024-04-26T01:42:05.393193+0000","last_became_peered":"2024-04-26T01:42:05.393193+0000","last_unstale":"2024-04-26T01:45:54.947186+0000","last_undegraded":"2024-04-26T01:45:54.947186+0000","last_fullsized":"2024-04-26T01:45:54.947186+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:20:29.420459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"95'4","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962087+0000","last_change":"2024-04-26T01:42:05.412979+0000","last_active":"2024-04-26T01:45:54.962087+0000","last_peered":"2024-04-26T01:45:54.962087+0000","last_clean":"2024-04-26T01:45:54.962087+0000","last_became_active":"2024-04-26T01:42:05.412679+0000","last_became_peered":"2024-04-26T01:42:05.412679+0000","last_unstale":"2024-04-26T01:45:54.962087+0000","last_undegraded":"2024-04-26T01:45:54.962087+0000","last_fullsized":"2024-04-26T01:45:54.962087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:22.774960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"95'6","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931536+0000","last_change":"2024-04-26T01:42:49.062507+0000","last_active":"2024-04-26T01:45:54.931536+0000","last_peered":"2024-04-26T01:45:54.931536+0000","last_clean":"2024-04-26T01:45:54.931536+0000","last_became_active":"2024-04-26T01:42:49.062095+0000","last_became_peered":"2024-04-26T01:42:49.062095+0000","last_unstale":"2024-04-26T01:45:54.931536+0000","last_undegraded":"2024-04-26T01:45:54.931536+0000","last_fullsized":"2024-04-26T01:45:54.931536+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:47.846697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962044+0000","last_change":"2024-04-26T01:42:05.385221+0000","last_active":"2024-04-26T01:45:54.962044+0000","last_peered":"2024-04-26T01:45:54.962044+0000","last_clean":"2024-04-26T01:45:54.962044+0000","last_became_active":"2024-04-26T01:42:05.384856+0000","last_became_peered":"2024-04-26T01:42:05.384856+0000","last_unstale":"2024-04-26T01:45:54.962044+0000","last_undegraded":"2024-04-26T01:45:54.962044+0000","last_fullsized":"2024-04-26T01:45:54.962044+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:07:23.461324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949107+0000","last_change":"2024-04-26T01:42:05.404754+0000","last_active":"2024-04-26T01:45:54.949107+0000","last_peered":"2024-04-26T01:45:54.949107+0000","last_clean":"2024-04-26T01:45:54.949107+0000","last_became_active":"2024-04-26T01:42:05.404220+0000","last_became_peered":"2024-04-26T01:42:05.404220+0000","last_unstale":"2024-04-26T01:45:54.949107+0000","last_undegraded":"2024-04-26T01:45:54.949107+0000","last_fullsized":"2024-04-26T01:45:54.949107+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:50.243552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"95'11","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961852+0000","last_change":"2024-04-26T01:42:05.390972+0000","last_active":"2024-04-26T01:45:54.961852+0000","last_peered":"2024-04-26T01:45:54.961852+0000","last_clean":"2024-04-26T01:45:54.961852+0000","last_became_active":"2024-04-26T01:42:05.390599+0000","last_became_peered":"2024-04-26T01:42:05.390599+0000","last_unstale":"2024-04-26T01:45:54.961852+0000","last_undegraded":"2024-04-26T01:45:54.961852+0000","last_fullsized":"2024-04-26T01:45:54.961852+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:19:46.328640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"95'16","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932040+0000","last_change":"2024-04-26T01:42:05.392320+0000","last_active":"2024-04-26T01:45:54.932040+0000","last_peered":"2024-04-26T01:45:54.932040+0000","last_clean":"2024-04-26T01:45:54.932040+0000","last_became_active":"2024-04-26T01:42:05.391986+0000","last_became_peered":"2024-04-26T01:42:05.391986+0000","last_unstale":"2024-04-26T01:45:54.932040+0000","last_undegraded":"2024-04-26T01:45:54.932040+0000","last_fullsized":"2024-04-26T01:45:54.932040+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:17:27.302358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962508+0000","last_change":"2024-04-26T01:42:05.388077+0000","last_active":"2024-04-26T01:45:54.962508+0000","last_peered":"2024-04-26T01:45:54.962508+0000","last_clean":"2024-04-26T01:45:54.962508+0000","last_became_active":"2024-04-26T01:42:05.387545+0000","last_became_peered":"2024-04-26T01:42:05.387545+0000","last_unstale":"2024-04-26T01:45:54.962508+0000","last_undegraded":"2024-04-26T01:45:54.962508+0000","last_fullsized":"2024-04-26T01:45:54.962508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:51.288177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"95'7","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932538+0000","last_change":"2024-04-26T01:42:49.067449+0000","last_active":"2024-04-26T01:45:54.932538+0000","last_peered":"2024-04-26T01:45:54.932538+0000","last_clean":"2024-04-26T01:45:54.932538+0000","last_became_active":"2024-04-26T01:42:49.065991+0000","last_became_peered":"2024-04-26T01:42:49.065991+0000","last_unstale":"2024-04-26T01:45:54.932538+0000","last_undegraded":"2024-04-26T01:45:54.932538+0000","last_fullsized":"2024-04-26T01:45:54.932538+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:13:36.565257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"95'10","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944606+0000","last_change":"2024-04-26T01:42:05.402847+0000","last_active":"2024-04-26T01:45:54.944606+0000","last_peered":"2024-04-26T01:45:54.944606+0000","last_clean":"2024-04-26T01:45:54.944606+0000","last_became_active":"2024-04-26T01:42:05.402503+0000","last_became_peered":"2024-04-26T01:42:05.402503+0000","last_unstale":"2024-04-26T01:45:54.944606+0000","last_undegraded":"2024-04-26T01:45:54.944606+0000","last_fullsized":"2024-04-26T01:45:54.944606+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:06.935060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.183651+0000","last_change":"2024-04-26T01:42:05.405257+0000","last_active":"2024-04-26T01:45:55.183651+0000","last_peered":"2024-04-26T01:45:55.183651+0000","last_clean":"2024-04-26T01:45:55.183651+0000","last_became_active":"2024-04-26T01:42:05.404551+0000","last_became_peered":"2024-04-26T01:42:05.404551+0000","last_unstale":"2024-04-26T01:45:55.183651+0000","last_undegraded":"2024-04-26T01:45:55.183651+0000","last_fullsized":"2024-04-26T01:45:55.183651+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:20.754088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.183628+0000","last_change":"2024-04-26T01:42:05.408847+0000","last_active":"2024-04-26T01:45:55.183628+0000","last_peered":"2024-04-26T01:45:55.183628+0000","last_clean":"2024-04-26T01:45:55.183628+0000","last_became_active":"2024-04-26T01:42:05.406189+0000","last_became_peered":"2024-04-26T01:42:05.406189+0000","last_unstale":"2024-04-26T01:45:55.183628+0000","last_undegraded":"2024-04-26T01:45:55.183628+0000","last_fullsized":"2024-04-26T01:45:55.183628+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:59:58.633630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961111+0000","last_change":"2024-04-26T01:42:05.404647+0000","last_active":"2024-04-26T01:45:54.961111+0000","last_peered":"2024-04-26T01:45:54.961111+0000","last_clean":"2024-04-26T01:45:54.961111+0000","last_became_active":"2024-04-26T01:42:05.404265+0000","last_became_peered":"2024-04-26T01:42:05.404265+0000","last_unstale":"2024-04-26T01:45:54.961111+0000","last_undegraded":"2024-04-26T01:45:54.961111+0000","last_fullsized":"2024-04-26T01:45:54.961111+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:00:43.472541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962928+0000","last_change":"2024-04-26T01:42:05.417602+0000","last_active":"2024-04-26T01:45:54.962928+0000","last_peered":"2024-04-26T01:45:54.962928+0000","last_clean":"2024-04-26T01:45:54.962928+0000","last_became_active":"2024-04-26T01:42:05.417144+0000","last_became_peered":"2024-04-26T01:42:05.417144+0000","last_unstale":"2024-04-26T01:45:54.962928+0000","last_undegraded":"2024-04-26T01:45:54.962928+0000","last_fullsized":"2024-04-26T01:45:54.962928+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:34.043597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"95'10","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.934166+0000","last_change":"2024-04-26T01:42:05.392948+0000","last_active":"2024-04-26T01:45:54.934166+0000","last_peered":"2024-04-26T01:45:54.934166+0000","last_clean":"2024-04-26T01:45:54.934166+0000","last_became_active":"2024-04-26T01:42:05.392040+0000","last_became_peered":"2024-04-26T01:42:05.392040+0000","last_unstale":"2024-04-26T01:45:54.934166+0000","last_undegraded":"2024-04-26T01:45:54.934166+0000","last_fullsized":"2024-04-26T01:45:54.934166+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:17.307979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931776+0000","last_change":"2024-04-26T01:42:05.406588+0000","last_active":"2024-04-26T01:45:54.931776+0000","last_peered":"2024-04-26T01:45:54.931776+0000","last_clean":"2024-04-26T01:45:54.931776+0000","last_became_active":"2024-04-26T01:42:05.406200+0000","last_became_peered":"2024-04-26T01:42:05.406200+0000","last_unstale":"2024-04-26T01:45:54.931776+0000","last_undegraded":"2024-04-26T01:45:54.931776+0000","last_fullsized":"2024-04-26T01:45:54.931776+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:14.095383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932704+0000","last_change":"2024-04-26T01:42:01.339289+0000","last_active":"2024-04-26T01:45:54.932704+0000","last_peered":"2024-04-26T01:45:54.932704+0000","last_clean":"2024-04-26T01:45:54.932704+0000","last_became_active":"2024-04-26T01:42:01.335402+0000","last_became_peered":"2024-04-26T01:42:01.335402+0000","last_unstale":"2024-04-26T01:45:54.932704+0000","last_undegraded":"2024-04-26T01:45:54.932704+0000","last_fullsized":"2024-04-26T01:45:54.932704+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:51.411524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.950320+0000","last_change":"2024-04-26T01:42:05.409569+0000","last_active":"2024-04-26T01:45:54.950320+0000","last_peered":"2024-04-26T01:45:54.950320+0000","last_clean":"2024-04-26T01:45:54.950320+0000","last_became_active":"2024-04-26T01:42:05.407441+0000","last_became_peered":"2024-04-26T01:42:05.407441+0000","last_unstale":"2024-04-26T01:45:54.950320+0000","last_undegraded":"2024-04-26T01:45:54.950320+0000","last_fullsized":"2024-04-26T01:45:54.950320+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:36:58.043359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.951626+0000","last_change":"2024-04-26T01:42:02.231871+0000","last_active":"2024-04-26T01:45:54.951626+0000","last_peered":"2024-04-26T01:45:54.951626+0000","last_clean":"2024-04-26T01:45:54.951626+0000","last_became_active":"2024-04-26T01:42:02.230813+0000","last_became_peered":"2024-04-26T01:42:02.230813+0000","last_unstale":"2024-04-26T01:45:54.951626+0000","last_undegraded":"2024-04-26T01:45:54.951626+0000","last_fullsized":"2024-04-26T01:45:54.951626+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:53.117266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.183893+0000","last_change":"2024-04-26T01:42:05.412414+0000","last_active":"2024-04-26T01:45:55.183893+0000","last_peered":"2024-04-26T01:45:55.183893+0000","last_clean":"2024-04-26T01:45:55.183893+0000","last_became_active":"2024-04-26T01:42:05.412057+0000","last_became_peered":"2024-04-26T01:42:05.412057+0000","last_unstale":"2024-04-26T01:45:55.183893+0000","last_undegraded":"2024-04-26T01:45:55.183893+0000","last_fullsized":"2024-04-26T01:45:55.183893+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:51:46.331246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.963097+0000","last_change":"2024-04-26T01:42:01.329957+0000","last_active":"2024-04-26T01:45:54.963097+0000","last_peered":"2024-04-26T01:45:54.963097+0000","last_clean":"2024-04-26T01:45:54.963097+0000","last_became_active":"2024-04-26T01:42:01.329289+0000","last_became_peered":"2024-04-26T01:42:01.329289+0000","last_unstale":"2024-04-26T01:45:54.963097+0000","last_undegraded":"2024-04-26T01:45:54.963097+0000","last_fullsized":"2024-04-26T01:45:54.963097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:46:38.983138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961476+0000","last_change":"2024-04-26T01:42:05.399438+0000","last_active":"2024-04-26T01:45:54.961476+0000","last_peered":"2024-04-26T01:45:54.961476+0000","last_clean":"2024-04-26T01:45:54.961476+0000","last_became_active":"2024-04-26T01:42:05.399082+0000","last_became_peered":"2024-04-26T01:42:05.399082+0000","last_unstale":"2024-04-26T01:45:54.961476+0000","last_undegraded":"2024-04-26T01:45:54.961476+0000","last_fullsized":"2024-04-26T01:45:54.961476+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:13.619893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"91'2","reported_seq":17,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961916+0000","last_change":"2024-04-26T01:43:49.993134+0000","last_active":"2024-04-26T01:45:54.961916+0000","last_peered":"2024-04-26T01:45:54.961916+0000","last_clean":"2024-04-26T01:45:54.961916+0000","last_became_active":"2024-04-26T01:43:49.990337+0000","last_became_peered":"2024-04-26T01:43:49.990337+0000","last_unstale":"2024-04-26T01:45:54.961916+0000","last_undegraded":"2024-04-26T01:45:54.961916+0000","last_fullsized":"2024-04-26T01:45:54.961916+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:50.502668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"95'15","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.933121+0000","last_change":"2024-04-26T01:42:49.066057+0000","last_active":"2024-04-26T01:45:54.933121+0000","last_peered":"2024-04-26T01:45:54.933121+0000","last_clean":"2024-04-26T01:45:54.933121+0000","last_became_active":"2024-04-26T01:42:49.064684+0000","last_became_peered":"2024-04-26T01:42:49.064684+0000","last_unstale":"2024-04-26T01:45:54.933121+0000","last_undegraded":"2024-04-26T01:45:54.933121+0000","last_fullsized":"2024-04-26T01:45:54.933121+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:39.159528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"96'9","reported_seq":255,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.963271+0000","last_change":"2024-04-26T01:42:01.332623+0000","last_active":"2024-04-26T01:45:54.963271+0000","last_peered":"2024-04-26T01:45:54.963271+0000","last_clean":"2024-04-26T01:45:54.963271+0000","last_became_active":"2024-04-26T01:42:01.332227+0000","last_became_peered":"2024-04-26T01:42:01.332227+0000","last_unstale":"2024-04-26T01:45:54.963271+0000","last_undegraded":"2024-04-26T01:45:54.963271+0000","last_fullsized":"2024-04-26T01:45:54.963271+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:58.611464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1855,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":213,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961832+0000","last_change":"2024-04-26T01:42:05.396133+0000","last_active":"2024-04-26T01:45:54.961832+0000","last_peered":"2024-04-26T01:45:54.961832+0000","last_clean":"2024-04-26T01:45:54.961832+0000","last_became_active":"2024-04-26T01:42:05.395553+0000","last_became_peered":"2024-04-26T01:42:05.395553+0000","last_unstale":"2024-04-26T01:45:54.961832+0000","last_undegraded":"2024-04-26T01:45:54.961832+0000","last_fullsized":"2024-04-26T01:45:54.961832+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:36:01.363909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931431+0000","last_change":"2024-04-26T01:42:01.340624+0000","last_active":"2024-04-26T01:45:54.931431+0000","last_peered":"2024-04-26T01:45:54.931431+0000","last_clean":"2024-04-26T01:45:54.931431+0000","last_became_active":"2024-04-26T01:42:01.340158+0000","last_became_peered":"2024-04-26T01:42:01.340158+0000","last_unstale":"2024-04-26T01:45:54.931431+0000","last_undegraded":"2024-04-26T01:45:54.931431+0000","last_fullsized":"2024-04-26T01:45:54.931431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:47:05.529922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.963143+0000","last_change":"2024-04-26T01:42:05.406354+0000","last_active":"2024-04-26T01:45:54.963143+0000","last_peered":"2024-04-26T01:45:54.963143+0000","last_clean":"2024-04-26T01:45:54.963143+0000","last_became_active":"2024-04-26T01:42:05.405989+0000","last_became_peered":"2024-04-26T01:42:05.405989+0000","last_unstale":"2024-04-26T01:45:54.963143+0000","last_undegraded":"2024-04-26T01:45:54.963143+0000","last_fullsized":"2024-04-26T01:45:54.963143+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:43.115225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944206+0000","last_change":"2024-04-26T01:42:01.340549+0000","last_active":"2024-04-26T01:45:54.944206+0000","last_peered":"2024-04-26T01:45:54.944206+0000","last_clean":"2024-04-26T01:45:54.944206+0000","last_became_active":"2024-04-26T01:42:01.339861+0000","last_became_peered":"2024-04-26T01:42:01.339861+0000","last_unstale":"2024-04-26T01:45:54.944206+0000","last_undegraded":"2024-04-26T01:45:54.944206+0000","last_fullsized":"2024-04-26T01:45:54.944206+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:02:53.154367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949325+0000","last_change":"2024-04-26T01:42:05.402664+0000","last_active":"2024-04-26T01:45:54.949325+0000","last_peered":"2024-04-26T01:45:54.949325+0000","last_clean":"2024-04-26T01:45:54.949325+0000","last_became_active":"2024-04-26T01:42:05.400785+0000","last_became_peered":"2024-04-26T01:42:05.400785+0000","last_unstale":"2024-04-26T01:45:54.949325+0000","last_undegraded":"2024-04-26T01:45:54.949325+0000","last_fullsized":"2024-04-26T01:45:54.949325+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:40:32.137752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961622+0000","last_change":"2024-04-26T01:42:01.328600+0000","last_active":"2024-04-26T01:45:54.961622+0000","last_peered":"2024-04-26T01:45:54.961622+0000","last_clean":"2024-04-26T01:45:54.961622+0000","last_became_active":"2024-04-26T01:42:01.328149+0000","last_became_peered":"2024-04-26T01:42:01.328149+0000","last_unstale":"2024-04-26T01:45:54.961622+0000","last_undegraded":"2024-04-26T01:45:54.961622+0000","last_fullsized":"2024-04-26T01:45:54.961622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:45:42.811088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"95'11","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932233+0000","last_change":"2024-04-26T01:42:49.065824+0000","last_active":"2024-04-26T01:45:54.932233+0000","last_peered":"2024-04-26T01:45:54.932233+0000","last_clean":"2024-04-26T01:45:54.932233+0000","last_became_active":"2024-04-26T01:42:49.064670+0000","last_became_peered":"2024-04-26T01:42:49.064670+0000","last_unstale":"2024-04-26T01:45:54.932233+0000","last_undegraded":"2024-04-26T01:45:54.932233+0000","last_fullsized":"2024-04-26T01:45:54.932233+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:44.790184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"95'5","reported_seq":247,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.959285+0000","last_change":"2024-04-26T01:42:01.335438+0000","last_active":"2024-04-26T01:45:54.959285+0000","last_peered":"2024-04-26T01:45:54.959285+0000","last_clean":"2024-04-26T01:45:54.959285+0000","last_became_active":"2024-04-26T01:42:01.334962+0000","last_became_peered":"2024-04-26T01:42:01.334962+0000","last_unstale":"2024-04-26T01:45:54.959285+0000","last_undegraded":"2024-04-26T01:45:54.959285+0000","last_fullsized":"2024-04-26T01:45:54.959285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:35:08.252274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":209,"num_read_kb":209,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"95'10","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961591+0000","last_change":"2024-04-26T01:42:05.393294+0000","last_active":"2024-04-26T01:45:54.961591+0000","last_peered":"2024-04-26T01:45:54.961591+0000","last_clean":"2024-04-26T01:45:54.961591+0000","last_became_active":"2024-04-26T01:42:05.392974+0000","last_became_peered":"2024-04-26T01:42:05.392974+0000","last_unstale":"2024-04-26T01:45:54.961591+0000","last_undegraded":"2024-04-26T01:45:54.961591+0000","last_fullsized":"2024-04-26T01:45:54.961591+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:58.188706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962548+0000","last_change":"2024-04-26T01:42:01.333505+0000","last_active":"2024-04-26T01:45:54.962548+0000","last_peered":"2024-04-26T01:45:54.962548+0000","last_clean":"2024-04-26T01:45:54.962548+0000","last_became_active":"2024-04-26T01:42:01.333062+0000","last_became_peered":"2024-04-26T01:42:01.333062+0000","last_unstale":"2024-04-26T01:45:54.962548+0000","last_undegraded":"2024-04-26T01:45:54.962548+0000","last_fullsized":"2024-04-26T01:45:54.962548+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:50:48.804443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"95'9","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.184062+0000","last_change":"2024-04-26T01:42:49.063948+0000","last_active":"2024-04-26T01:45:55.184062+0000","last_peered":"2024-04-26T01:45:55.184062+0000","last_clean":"2024-04-26T01:45:55.184062+0000","last_became_active":"2024-04-26T01:42:49.060852+0000","last_became_peered":"2024-04-26T01:42:49.060852+0000","last_unstale":"2024-04-26T01:45:55.184062+0000","last_undegraded":"2024-04-26T01:45:55.184062+0000","last_fullsized":"2024-04-26T01:45:55.184062+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.370431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.963018+0000","last_change":"2024-04-26T01:42:02.232308+0000","last_active":"2024-04-26T01:45:54.963018+0000","last_peered":"2024-04-26T01:45:54.963018+0000","last_clean":"2024-04-26T01:45:54.963018+0000","last_became_active":"2024-04-26T01:42:02.231897+0000","last_became_peered":"2024-04-26T01:42:02.231897+0000","last_unstale":"2024-04-26T01:45:54.963018+0000","last_undegraded":"2024-04-26T01:45:54.963018+0000","last_fullsized":"2024-04-26T01:45:54.963018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:36:16.847087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"95'11","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961415+0000","last_change":"2024-04-26T01:42:49.070798+0000","last_active":"2024-04-26T01:45:54.961415+0000","last_peered":"2024-04-26T01:45:54.961415+0000","last_clean":"2024-04-26T01:45:54.961415+0000","last_became_active":"2024-04-26T01:42:49.070470+0000","last_became_peered":"2024-04-26T01:42:49.070470+0000","last_unstale":"2024-04-26T01:45:54.961415+0000","last_undegraded":"2024-04-26T01:45:54.961415+0000","last_fullsized":"2024-04-26T01:45:54.961415+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:04:22.103704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"96'39","reported_seq":504,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.852793+0000","last_change":"2024-04-26T01:42:01.330822+0000","last_active":"2024-04-26T01:45:55.852793+0000","last_peered":"2024-04-26T01:45:55.852793+0000","last_clean":"2024-04-26T01:45:55.852793+0000","last_became_active":"2024-04-26T01:42:01.330337+0000","last_became_peered":"2024-04-26T01:42:01.330337+0000","last_unstale":"2024-04-26T01:45:55.852793+0000","last_undegraded":"2024-04-26T01:45:55.852793+0000","last_fullsized":"2024-04-26T01:45:55.852793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:04.024645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70924,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":432,"num_read_kb":286,"num_write":40,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962763+0000","last_change":"2024-04-26T01:42:05.390601+0000","last_active":"2024-04-26T01:45:54.962763+0000","last_peered":"2024-04-26T01:45:54.962763+0000","last_clean":"2024-04-26T01:45:54.962763+0000","last_became_active":"2024-04-26T01:42:05.390279+0000","last_became_peered":"2024-04-26T01:42:05.390279+0000","last_unstale":"2024-04-26T01:45:54.962763+0000","last_undegraded":"2024-04-26T01:45:54.962763+0000","last_fullsized":"2024-04-26T01:45:54.962763+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:20.171769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931520+0000","last_change":"2024-04-26T01:42:01.337200+0000","last_active":"2024-04-26T01:45:54.931520+0000","last_peered":"2024-04-26T01:45:54.931520+0000","last_clean":"2024-04-26T01:45:54.931520+0000","last_became_active":"2024-04-26T01:42:01.336919+0000","last_became_peered":"2024-04-26T01:42:01.336919+0000","last_unstale":"2024-04-26T01:45:54.931520+0000","last_undegraded":"2024-04-26T01:45:54.931520+0000","last_fullsized":"2024-04-26T01:45:54.931520+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:29.799903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944349+0000","last_change":"2024-04-26T01:42:05.410081+0000","last_active":"2024-04-26T01:45:54.944349+0000","last_peered":"2024-04-26T01:45:54.944349+0000","last_clean":"2024-04-26T01:45:54.944349+0000","last_became_active":"2024-04-26T01:42:05.409341+0000","last_became_peered":"2024-04-26T01:42:05.409341+0000","last_unstale":"2024-04-26T01:45:54.944349+0000","last_undegraded":"2024-04-26T01:45:54.944349+0000","last_fullsized":"2024-04-26T01:45:54.944349+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:25:31.925296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962585+0000","last_change":"2024-04-26T01:42:01.331702+0000","last_active":"2024-04-26T01:45:54.962585+0000","last_peered":"2024-04-26T01:45:54.962585+0000","last_clean":"2024-04-26T01:45:54.962585+0000","last_became_active":"2024-04-26T01:42:01.331311+0000","last_became_peered":"2024-04-26T01:42:01.331311+0000","last_unstale":"2024-04-26T01:45:54.962585+0000","last_undegraded":"2024-04-26T01:45:54.962585+0000","last_fullsized":"2024-04-26T01:45:54.962585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:54:49.513627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"95'15","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962407+0000","last_change":"2024-04-26T01:42:05.403957+0000","last_active":"2024-04-26T01:45:54.962407+0000","last_peered":"2024-04-26T01:45:54.962407+0000","last_clean":"2024-04-26T01:45:54.962407+0000","last_became_active":"2024-04-26T01:42:05.403583+0000","last_became_peered":"2024-04-26T01:42:05.403583+0000","last_unstale":"2024-04-26T01:45:54.962407+0000","last_undegraded":"2024-04-26T01:45:54.962407+0000","last_fullsized":"2024-04-26T01:45:54.962407+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:12:20.095407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"96'6","reported_seq":253,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.956378+0000","last_change":"2024-04-26T01:42:01.332456+0000","last_active":"2024-04-26T01:45:55.956378+0000","last_peered":"2024-04-26T01:45:55.956378+0000","last_clean":"2024-04-26T01:45:55.956378+0000","last_became_active":"2024-04-26T01:42:01.331910+0000","last_became_peered":"2024-04-26T01:42:01.331910+0000","last_unstale":"2024-04-26T01:45:55.956378+0000","last_undegraded":"2024-04-26T01:45:55.956378+0000","last_fullsized":"2024-04-26T01:45:55.956378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:13:25.368205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":215,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"95'6","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961737+0000","last_change":"2024-04-26T01:42:49.062130+0000","last_active":"2024-04-26T01:45:54.961737+0000","last_peered":"2024-04-26T01:45:54.961737+0000","last_clean":"2024-04-26T01:45:54.961737+0000","last_became_active":"2024-04-26T01:42:49.061627+0000","last_became_peered":"2024-04-26T01:42:49.061627+0000","last_unstale":"2024-04-26T01:45:54.961737+0000","last_undegraded":"2024-04-26T01:45:54.961737+0000","last_fullsized":"2024-04-26T01:45:54.961737+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:16.101940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961990+0000","last_change":"2024-04-26T01:42:49.082618+0000","last_active":"2024-04-26T01:45:54.961990+0000","last_peered":"2024-04-26T01:45:54.961990+0000","last_clean":"2024-04-26T01:45:54.961990+0000","last_became_active":"2024-04-26T01:42:49.064654+0000","last_became_peered":"2024-04-26T01:42:49.064654+0000","last_unstale":"2024-04-26T01:45:54.961990+0000","last_undegraded":"2024-04-26T01:45:54.961990+0000","last_fullsized":"2024-04-26T01:45:54.961990+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:13:23.937059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962550+0000","last_change":"2024-04-26T01:42:05.391392+0000","last_active":"2024-04-26T01:45:54.962550+0000","last_peered":"2024-04-26T01:45:54.962550+0000","last_clean":"2024-04-26T01:45:54.962550+0000","last_became_active":"2024-04-26T01:42:05.390939+0000","last_became_peered":"2024-04-26T01:42:05.390939+0000","last_unstale":"2024-04-26T01:45:54.962550+0000","last_undegraded":"2024-04-26T01:45:54.962550+0000","last_fullsized":"2024-04-26T01:45:54.962550+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:36.141915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962306+0000","last_change":"2024-04-26T01:42:01.332932+0000","last_active":"2024-04-26T01:45:54.962306+0000","last_peered":"2024-04-26T01:45:54.962306+0000","last_clean":"2024-04-26T01:45:54.962306+0000","last_became_active":"2024-04-26T01:42:01.332626+0000","last_became_peered":"2024-04-26T01:42:01.332626+0000","last_unstale":"2024-04-26T01:45:54.962306+0000","last_undegraded":"2024-04-26T01:45:54.962306+0000","last_fullsized":"2024-04-26T01:45:54.962306+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:20.593730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962441+0000","last_change":"2024-04-26T01:42:05.394925+0000","last_active":"2024-04-26T01:45:54.962441+0000","last_peered":"2024-04-26T01:45:54.962441+0000","last_clean":"2024-04-26T01:45:54.962441+0000","last_became_active":"2024-04-26T01:42:05.394531+0000","last_became_peered":"2024-04-26T01:42:05.394531+0000","last_unstale":"2024-04-26T01:45:54.962441+0000","last_undegraded":"2024-04-26T01:45:54.962441+0000","last_fullsized":"2024-04-26T01:45:54.962441+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:33.177780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932471+0000","last_change":"2024-04-26T01:42:02.237482+0000","last_active":"2024-04-26T01:45:54.932471+0000","last_peered":"2024-04-26T01:45:54.932471+0000","last_clean":"2024-04-26T01:45:54.932471+0000","last_became_active":"2024-04-26T01:42:02.236752+0000","last_became_peered":"2024-04-26T01:42:02.236752+0000","last_unstale":"2024-04-26T01:45:54.932471+0000","last_undegraded":"2024-04-26T01:45:54.932471+0000","last_fullsized":"2024-04-26T01:45:54.932471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:27.211245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931723+0000","last_change":"2024-04-26T01:42:05.402157+0000","last_active":"2024-04-26T01:45:54.931723+0000","last_peered":"2024-04-26T01:45:54.931723+0000","last_clean":"2024-04-26T01:45:54.931723+0000","last_became_active":"2024-04-26T01:42:05.401889+0000","last_became_peered":"2024-04-26T01:42:05.401889+0000","last_unstale":"2024-04-26T01:45:54.931723+0000","last_undegraded":"2024-04-26T01:45:54.931723+0000","last_fullsized":"2024-04-26T01:45:54.931723+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:33.854901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962227+0000","last_change":"2024-04-26T01:42:01.339675+0000","last_active":"2024-04-26T01:45:54.962227+0000","last_peered":"2024-04-26T01:45:54.962227+0000","last_clean":"2024-04-26T01:45:54.962227+0000","last_became_active":"2024-04-26T01:42:01.339247+0000","last_became_peered":"2024-04-26T01:42:01.339247+0000","last_unstale":"2024-04-26T01:45:54.962227+0000","last_undegraded":"2024-04-26T01:45:54.962227+0000","last_fullsized":"2024-04-26T01:45:54.962227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:32.724830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931641+0000","last_change":"2024-04-26T01:42:05.412913+0000","last_active":"2024-04-26T01:45:54.931641+0000","last_peered":"2024-04-26T01:45:54.931641+0000","last_clean":"2024-04-26T01:45:54.931641+0000","last_became_active":"2024-04-26T01:42:05.412556+0000","last_became_peered":"2024-04-26T01:42:05.412556+0000","last_unstale":"2024-04-26T01:45:54.931641+0000","last_undegraded":"2024-04-26T01:45:54.931641+0000","last_fullsized":"2024-04-26T01:45:54.931641+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:00:52.798069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961962+0000","last_change":"2024-04-26T01:42:05.391235+0000","last_active":"2024-04-26T01:45:54.961962+0000","last_peered":"2024-04-26T01:45:54.961962+0000","last_clean":"2024-04-26T01:45:54.961962+0000","last_became_active":"2024-04-26T01:42:05.390717+0000","last_became_peered":"2024-04-26T01:42:05.390717+0000","last_unstale":"2024-04-26T01:45:54.961962+0000","last_undegraded":"2024-04-26T01:45:54.961962+0000","last_fullsized":"2024-04-26T01:45:54.961962+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:07.541593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"96'6","reported_seq":248,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.255628+0000","last_change":"2024-04-26T01:42:02.232823+0000","last_active":"2024-04-26T01:45:55.255628+0000","last_peered":"2024-04-26T01:45:55.255628+0000","last_clean":"2024-04-26T01:45:55.255628+0000","last_became_active":"2024-04-26T01:42:02.231899+0000","last_became_peered":"2024-04-26T01:42:02.231899+0000","last_unstale":"2024-04-26T01:45:55.255628+0000","last_undegraded":"2024-04-26T01:45:55.255628+0000","last_fullsized":"2024-04-26T01:45:55.255628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:23.287305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":209,"num_read_kb":209,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949845+0000","last_change":"2024-04-26T01:42:05.405690+0000","last_active":"2024-04-26T01:45:54.949845+0000","last_peered":"2024-04-26T01:45:54.949845+0000","last_clean":"2024-04-26T01:45:54.949845+0000","last_became_active":"2024-04-26T01:42:05.405345+0000","last_became_peered":"2024-04-26T01:42:05.405345+0000","last_unstale":"2024-04-26T01:45:54.949845+0000","last_undegraded":"2024-04-26T01:45:54.949845+0000","last_fullsized":"2024-04-26T01:45:54.949845+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:48.377916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.947235+0000","last_change":"2024-04-26T01:42:01.335721+0000","last_active":"2024-04-26T01:45:54.947235+0000","last_peered":"2024-04-26T01:45:54.947235+0000","last_clean":"2024-04-26T01:45:54.947235+0000","last_became_active":"2024-04-26T01:42:01.335149+0000","last_became_peered":"2024-04-26T01:42:01.335149+0000","last_unstale":"2024-04-26T01:45:54.947235+0000","last_undegraded":"2024-04-26T01:45:54.947235+0000","last_fullsized":"2024-04-26T01:45:54.947235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:36.372033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":369,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961908+0000","last_change":"2024-04-26T01:40:28.086333+0000","last_active":"2024-04-26T01:45:54.961908+0000","last_peered":"2024-04-26T01:45:54.961908+0000","last_clean":"2024-04-26T01:45:54.961908+0000","last_became_active":"2024-04-26T01:40:28.085902+0000","last_became_peered":"2024-04-26T01:40:28.085902+0000","last_unstale":"2024-04-26T01:45:54.961908+0000","last_undegraded":"2024-04-26T01:45:54.961908+0000","last_fullsized":"2024-04-26T01:45:54.961908+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_clean_scrub_stamp":"2024-04-26T01:36:35.357698+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:43.290117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944108+0000","last_change":"2024-04-26T01:42:05.401896+0000","last_active":"2024-04-26T01:45:54.944108+0000","last_peered":"2024-04-26T01:45:54.944108+0000","last_clean":"2024-04-26T01:45:54.944108+0000","last_became_active":"2024-04-26T01:42:05.401496+0000","last_became_peered":"2024-04-26T01:42:05.401496+0000","last_unstale":"2024-04-26T01:45:54.944108+0000","last_undegraded":"2024-04-26T01:45:54.944108+0000","last_fullsized":"2024-04-26T01:45:54.944108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:49.758444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961993+0000","last_change":"2024-04-26T01:42:01.329705+0000","last_active":"2024-04-26T01:45:54.961993+0000","last_peered":"2024-04-26T01:45:54.961993+0000","last_clean":"2024-04-26T01:45:54.961993+0000","last_became_active":"2024-04-26T01:42:01.329153+0000","last_became_peered":"2024-04-26T01:42:01.329153+0000","last_unstale":"2024-04-26T01:45:54.961993+0000","last_undegraded":"2024-04-26T01:45:54.961993+0000","last_fullsized":"2024-04-26T01:45:54.961993+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:06.747378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962915+0000","last_change":"2024-04-26T01:42:02.232148+0000","last_active":"2024-04-26T01:45:54.962915+0000","last_peered":"2024-04-26T01:45:54.962915+0000","last_clean":"2024-04-26T01:45:54.962915+0000","last_became_active":"2024-04-26T01:42:02.231739+0000","last_became_peered":"2024-04-26T01:42:02.231739+0000","last_unstale":"2024-04-26T01:45:54.962915+0000","last_undegraded":"2024-04-26T01:45:54.962915+0000","last_fullsized":"2024-04-26T01:45:54.962915+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:09.752433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.945042+0000","last_change":"2024-04-26T01:42:05.399938+0000","last_active":"2024-04-26T01:45:54.945042+0000","last_peered":"2024-04-26T01:45:54.945042+0000","last_clean":"2024-04-26T01:45:54.945042+0000","last_became_active":"2024-04-26T01:42:05.399550+0000","last_became_peered":"2024-04-26T01:42:05.399550+0000","last_unstale":"2024-04-26T01:45:54.945042+0000","last_undegraded":"2024-04-26T01:45:54.945042+0000","last_fullsized":"2024-04-26T01:45:54.945042+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:44.142284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"96'50","reported_seq":109,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.972620+0000","last_change":"2024-04-26T01:42:01.337437+0000","last_active":"2024-04-26T01:45:55.972620+0000","last_peered":"2024-04-26T01:45:55.972620+0000","last_clean":"2024-04-26T01:45:55.972620+0000","last_became_active":"2024-04-26T01:42:01.336894+0000","last_became_peered":"2024-04-26T01:42:01.336894+0000","last_unstale":"2024-04-26T01:45:55.972620+0000","last_undegraded":"2024-04-26T01:45:55.972620+0000","last_fullsized":"2024-04-26T01:45:55.972620+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:04:31.093273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":90,"num_read_kb":77,"num_write":90,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.951785+0000","last_change":"2024-04-26T01:42:05.411694+0000","last_active":"2024-04-26T01:45:54.951785+0000","last_peered":"2024-04-26T01:45:54.951785+0000","last_clean":"2024-04-26T01:45:54.951785+0000","last_became_active":"2024-04-26T01:42:05.411300+0000","last_became_peered":"2024-04-26T01:42:05.411300+0000","last_unstale":"2024-04-26T01:45:54.951785+0000","last_undegraded":"2024-04-26T01:45:54.951785+0000","last_fullsized":"2024-04-26T01:45:54.951785+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:42:08.116145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.947485+0000","last_change":"2024-04-26T01:42:01.427501+0000","last_active":"2024-04-26T01:45:54.947485+0000","last_peered":"2024-04-26T01:45:54.947485+0000","last_clean":"2024-04-26T01:45:54.947485+0000","last_became_active":"2024-04-26T01:42:01.426970+0000","last_became_peered":"2024-04-26T01:42:01.426970+0000","last_unstale":"2024-04-26T01:45:54.947485+0000","last_undegraded":"2024-04-26T01:45:54.947485+0000","last_fullsized":"2024-04-26T01:45:54.947485+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:53:53.118296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.184575+0000","last_change":"2024-04-26T01:42:05.414790+0000","last_active":"2024-04-26T01:45:55.184575+0000","last_peered":"2024-04-26T01:45:55.184575+0000","last_clean":"2024-04-26T01:45:55.184575+0000","last_became_active":"2024-04-26T01:42:05.411947+0000","last_became_peered":"2024-04-26T01:42:05.411947+0000","last_unstale":"2024-04-26T01:45:55.184575+0000","last_undegraded":"2024-04-26T01:45:55.184575+0000","last_fullsized":"2024-04-26T01:45:55.184575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:11:34.450815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961147+0000","last_change":"2024-04-26T01:42:02.230539+0000","last_active":"2024-04-26T01:45:54.961147+0000","last_peered":"2024-04-26T01:45:54.961147+0000","last_clean":"2024-04-26T01:45:54.961147+0000","last_became_active":"2024-04-26T01:42:02.230150+0000","last_became_peered":"2024-04-26T01:42:02.230150+0000","last_unstale":"2024-04-26T01:45:54.961147+0000","last_undegraded":"2024-04-26T01:45:54.961147+0000","last_fullsized":"2024-04-26T01:45:54.961147+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:08:03.073069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"95'2","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961652+0000","last_change":"2024-04-26T01:42:05.388877+0000","last_active":"2024-04-26T01:45:54.961652+0000","last_peered":"2024-04-26T01:45:54.961652+0000","last_clean":"2024-04-26T01:45:54.961652+0000","last_became_active":"2024-04-26T01:42:05.388457+0000","last_became_peered":"2024-04-26T01:42:05.388457+0000","last_unstale":"2024-04-26T01:45:54.961652+0000","last_undegraded":"2024-04-26T01:45:54.961652+0000","last_fullsized":"2024-04-26T01:45:54.961652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:54.436859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949462+0000","last_change":"2024-04-26T01:42:01.343461+0000","last_active":"2024-04-26T01:45:54.949462+0000","last_peered":"2024-04-26T01:45:54.949462+0000","last_clean":"2024-04-26T01:45:54.949462+0000","last_became_active":"2024-04-26T01:42:01.342713+0000","last_became_peered":"2024-04-26T01:42:01.342713+0000","last_unstale":"2024-04-26T01:45:54.949462+0000","last_undegraded":"2024-04-26T01:45:54.949462+0000","last_fullsized":"2024-04-26T01:45:54.949462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:38.471858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962158+0000","last_change":"2024-04-26T01:42:05.415101+0000","last_active":"2024-04-26T01:45:54.962158+0000","last_peered":"2024-04-26T01:45:54.962158+0000","last_clean":"2024-04-26T01:45:54.962158+0000","last_became_active":"2024-04-26T01:42:05.414745+0000","last_became_peered":"2024-04-26T01:42:05.414745+0000","last_unstale":"2024-04-26T01:45:54.962158+0000","last_undegraded":"2024-04-26T01:45:54.962158+0000","last_fullsized":"2024-04-26T01:45:54.962158+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:13:08.573547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"96'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931967+0000","last_change":"2024-04-26T01:42:01.339501+0000","last_active":"2024-04-26T01:45:54.931967+0000","last_peered":"2024-04-26T01:45:54.931967+0000","last_clean":"2024-04-26T01:45:54.931967+0000","last_became_active":"2024-04-26T01:42:01.339132+0000","last_became_peered":"2024-04-26T01:42:01.339132+0000","last_unstale":"2024-04-26T01:45:54.931967+0000","last_undegraded":"2024-04-26T01:45:54.931967+0000","last_fullsized":"2024-04-26T01:45:54.931967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:16.463099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949949+0000","last_change":"2024-04-26T01:42:05.417864+0000","last_active":"2024-04-26T01:45:54.949949+0000","last_peered":"2024-04-26T01:45:54.949949+0000","last_clean":"2024-04-26T01:45:54.949949+0000","last_became_active":"2024-04-26T01:42:05.417414+0000","last_became_peered":"2024-04-26T01:42:05.417414+0000","last_unstale":"2024-04-26T01:45:54.949949+0000","last_undegraded":"2024-04-26T01:45:54.949949+0000","last_fullsized":"2024-04-26T01:45:54.949949+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:23:12.793421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":50,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962222+0000","last_change":"2024-04-26T01:42:49.063769+0000","last_active":"2024-04-26T01:45:54.962222+0000","last_peered":"2024-04-26T01:45:54.962222+0000","last_clean":"2024-04-26T01:45:54.962222+0000","last_became_active":"2024-04-26T01:42:49.057463+0000","last_became_peered":"2024-04-26T01:42:49.057463+0000","last_unstale":"2024-04-26T01:45:54.962222+0000","last_undegraded":"2024-04-26T01:45:54.962222+0000","last_fullsized":"2024-04-26T01:45:54.962222+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:34:54.511304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962310+0000","last_change":"2024-04-26T01:42:05.397980+0000","last_active":"2024-04-26T01:45:54.962310+0000","last_peered":"2024-04-26T01:45:54.962310+0000","last_clean":"2024-04-26T01:45:54.962310+0000","last_became_active":"2024-04-26T01:42:05.397589+0000","last_became_peered":"2024-04-26T01:42:05.397589+0000","last_unstale":"2024-04-26T01:45:54.962310+0000","last_undegraded":"2024-04-26T01:45:54.962310+0000","last_fullsized":"2024-04-26T01:45:54.962310+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:39:55.863566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931803+0000","last_change":"2024-04-26T01:42:02.234313+0000","last_active":"2024-04-26T01:45:54.931803+0000","last_peered":"2024-04-26T01:45:54.931803+0000","last_clean":"2024-04-26T01:45:54.931803+0000","last_became_active":"2024-04-26T01:42:02.233873+0000","last_became_peered":"2024-04-26T01:42:02.233873+0000","last_unstale":"2024-04-26T01:45:54.931803+0000","last_undegraded":"2024-04-26T01:45:54.931803+0000","last_fullsized":"2024-04-26T01:45:54.931803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:34.092434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962665+0000","last_change":"2024-04-26T01:42:05.387435+0000","last_active":"2024-04-26T01:45:54.962665+0000","last_peered":"2024-04-26T01:45:54.962665+0000","last_clean":"2024-04-26T01:45:54.962665+0000","last_became_active":"2024-04-26T01:42:05.386964+0000","last_became_peered":"2024-04-26T01:42:05.386964+0000","last_unstale":"2024-04-26T01:45:54.962665+0000","last_undegraded":"2024-04-26T01:45:54.962665+0000","last_fullsized":"2024-04-26T01:45:54.962665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:12:27.905842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"96'39","reported_seq":80,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.204536+0000","last_change":"2024-04-26T01:42:01.326432+0000","last_active":"2024-04-26T01:45:55.204536+0000","last_peered":"2024-04-26T01:45:55.204536+0000","last_clean":"2024-04-26T01:45:55.204536+0000","last_became_active":"2024-04-26T01:42:01.325989+0000","last_became_peered":"2024-04-26T01:42:01.325989+0000","last_unstale":"2024-04-26T01:45:55.204536+0000","last_undegraded":"2024-04-26T01:45:55.204536+0000","last_fullsized":"2024-04-26T01:45:55.204536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:10.049563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961654+0000","last_change":"2024-04-26T01:42:05.412802+0000","last_active":"2024-04-26T01:45:54.961654+0000","last_peered":"2024-04-26T01:45:54.961654+0000","last_clean":"2024-04-26T01:45:54.961654+0000","last_became_active":"2024-04-26T01:42:05.412443+0000","last_became_peered":"2024-04-26T01:42:05.412443+0000","last_unstale":"2024-04-26T01:45:54.961654+0000","last_undegraded":"2024-04-26T01:45:54.961654+0000","last_fullsized":"2024-04-26T01:45:54.961654+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:28:56.360778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'2","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944056+0000","last_change":"2024-04-26T01:42:49.080305+0000","last_active":"2024-04-26T01:45:54.944056+0000","last_peered":"2024-04-26T01:45:54.944056+0000","last_clean":"2024-04-26T01:45:54.944056+0000","last_became_active":"2024-04-26T01:42:49.074476+0000","last_became_peered":"2024-04-26T01:42:49.074476+0000","last_unstale":"2024-04-26T01:45:54.944056+0000","last_undegraded":"2024-04-26T01:45:54.944056+0000","last_fullsized":"2024-04-26T01:45:54.944056+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:58:06.619961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962655+0000","last_change":"2024-04-26T01:42:05.387621+0000","last_active":"2024-04-26T01:45:54.962655+0000","last_peered":"2024-04-26T01:45:54.962655+0000","last_clean":"2024-04-26T01:45:54.962655+0000","last_became_active":"2024-04-26T01:42:05.387165+0000","last_became_peered":"2024-04-26T01:42:05.387165+0000","last_unstale":"2024-04-26T01:45:54.962655+0000","last_undegraded":"2024-04-26T01:45:54.962655+0000","last_fullsized":"2024-04-26T01:45:54.962655+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:37.803330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962291+0000","last_change":"2024-04-26T01:42:05.386787+0000","last_active":"2024-04-26T01:45:54.962291+0000","last_peered":"2024-04-26T01:45:54.962291+0000","last_clean":"2024-04-26T01:45:54.962291+0000","last_became_active":"2024-04-26T01:42:05.386410+0000","last_became_peered":"2024-04-26T01:42:05.386410+0000","last_unstale":"2024-04-26T01:45:54.962291+0000","last_undegraded":"2024-04-26T01:45:54.962291+0000","last_fullsized":"2024-04-26T01:45:54.962291+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:42.137191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"95'12","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931481+0000","last_change":"2024-04-26T01:42:05.392142+0000","last_active":"2024-04-26T01:45:54.931481+0000","last_peered":"2024-04-26T01:45:54.931481+0000","last_clean":"2024-04-26T01:45:54.931481+0000","last_became_active":"2024-04-26T01:42:05.391829+0000","last_became_peered":"2024-04-26T01:42:05.391829+0000","last_unstale":"2024-04-26T01:45:54.931481+0000","last_undegraded":"2024-04-26T01:45:54.931481+0000","last_fullsized":"2024-04-26T01:45:54.931481+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:00:25.934641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931859+0000","last_change":"2024-04-26T01:42:05.409184+0000","last_active":"2024-04-26T01:45:54.931859+0000","last_peered":"2024-04-26T01:45:54.931859+0000","last_clean":"2024-04-26T01:45:54.931859+0000","last_became_active":"2024-04-26T01:42:05.408894+0000","last_became_peered":"2024-04-26T01:42:05.408894+0000","last_unstale":"2024-04-26T01:45:54.931859+0000","last_undegraded":"2024-04-26T01:45:54.931859+0000","last_fullsized":"2024-04-26T01:45:54.931859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:53.291844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.183917+0000","last_change":"2024-04-26T01:42:05.406940+0000","last_active":"2024-04-26T01:45:55.183917+0000","last_peered":"2024-04-26T01:45:55.183917+0000","last_clean":"2024-04-26T01:45:55.183917+0000","last_became_active":"2024-04-26T01:42:05.406476+0000","last_became_peered":"2024-04-26T01:42:05.406476+0000","last_unstale":"2024-04-26T01:45:55.183917+0000","last_undegraded":"2024-04-26T01:45:55.183917+0000","last_fullsized":"2024-04-26T01:45:55.183917+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:21:37.734633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"95'4","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.945671+0000","last_change":"2024-04-26T01:42:05.413521+0000","last_active":"2024-04-26T01:45:54.945671+0000","last_peered":"2024-04-26T01:45:54.945671+0000","last_clean":"2024-04-26T01:45:54.945671+0000","last_became_active":"2024-04-26T01:42:05.412789+0000","last_became_peered":"2024-04-26T01:42:05.412789+0000","last_unstale":"2024-04-26T01:45:54.945671+0000","last_undegraded":"2024-04-26T01:45:54.945671+0000","last_fullsized":"2024-04-26T01:45:54.945671+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:13.471376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962356+0000","last_change":"2024-04-26T01:42:05.391395+0000","last_active":"2024-04-26T01:45:54.962356+0000","last_peered":"2024-04-26T01:45:54.962356+0000","last_clean":"2024-04-26T01:45:54.962356+0000","last_became_active":"2024-04-26T01:42:05.391048+0000","last_became_peered":"2024-04-26T01:42:05.391048+0000","last_unstale":"2024-04-26T01:45:54.962356+0000","last_undegraded":"2024-04-26T01:45:54.962356+0000","last_fullsized":"2024-04-26T01:45:54.962356+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:45:56.645454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"95'15","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931782+0000","last_change":"2024-04-26T01:42:05.386868+0000","last_active":"2024-04-26T01:45:54.931782+0000","last_peered":"2024-04-26T01:45:54.931782+0000","last_clean":"2024-04-26T01:45:54.931782+0000","last_became_active":"2024-04-26T01:42:05.386483+0000","last_became_peered":"2024-04-26T01:42:05.386483+0000","last_unstale":"2024-04-26T01:45:54.931782+0000","last_undegraded":"2024-04-26T01:45:54.931782+0000","last_fullsized":"2024-04-26T01:45:54.931782+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:42:44.453076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"95'2","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.951689+0000","last_change":"2024-04-26T01:42:05.406711+0000","last_active":"2024-04-26T01:45:54.951689+0000","last_peered":"2024-04-26T01:45:54.951689+0000","last_clean":"2024-04-26T01:45:54.951689+0000","last_became_active":"2024-04-26T01:42:05.406318+0000","last_became_peered":"2024-04-26T01:42:05.406318+0000","last_unstale":"2024-04-26T01:45:54.951689+0000","last_undegraded":"2024-04-26T01:45:54.951689+0000","last_fullsized":"2024-04-26T01:45:54.951689+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:36:16.599678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961752+0000","last_change":"2024-04-26T01:42:05.406305+0000","last_active":"2024-04-26T01:45:54.961752+0000","last_peered":"2024-04-26T01:45:54.961752+0000","last_clean":"2024-04-26T01:45:54.961752+0000","last_became_active":"2024-04-26T01:42:05.405953+0000","last_became_peered":"2024-04-26T01:42:05.405953+0000","last_unstale":"2024-04-26T01:45:54.961752+0000","last_undegraded":"2024-04-26T01:45:54.961752+0000","last_fullsized":"2024-04-26T01:45:54.961752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:36.588672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949192+0000","last_change":"2024-04-26T01:42:05.413757+0000","last_active":"2024-04-26T01:45:54.949192+0000","last_peered":"2024-04-26T01:45:54.949192+0000","last_clean":"2024-04-26T01:45:54.949192+0000","last_became_active":"2024-04-26T01:42:05.413410+0000","last_became_peered":"2024-04-26T01:42:05.413410+0000","last_unstale":"2024-04-26T01:45:54.949192+0000","last_undegraded":"2024-04-26T01:45:54.949192+0000","last_fullsized":"2024-04-26T01:45:54.949192+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:30.298443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"95'10","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961329+0000","last_change":"2024-04-26T01:42:05.412268+0000","last_active":"2024-04-26T01:45:54.961329+0000","last_peered":"2024-04-26T01:45:54.961329+0000","last_clean":"2024-04-26T01:45:54.961329+0000","last_became_active":"2024-04-26T01:42:05.411804+0000","last_became_peered":"2024-04-26T01:42:05.411804+0000","last_unstale":"2024-04-26T01:45:54.961329+0000","last_undegraded":"2024-04-26T01:45:54.961329+0000","last_fullsized":"2024-04-26T01:45:54.961329+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:47.277766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"95'11","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944497+0000","last_change":"2024-04-26T01:42:05.415563+0000","last_active":"2024-04-26T01:45:54.944497+0000","last_peered":"2024-04-26T01:45:54.944497+0000","last_clean":"2024-04-26T01:45:54.944497+0000","last_became_active":"2024-04-26T01:42:05.414818+0000","last_became_peered":"2024-04-26T01:42:05.414818+0000","last_unstale":"2024-04-26T01:45:54.944497+0000","last_undegraded":"2024-04-26T01:45:54.944497+0000","last_fullsized":"2024-04-26T01:45:54.944497+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:54.713595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962135+0000","last_change":"2024-04-26T01:42:05.392280+0000","last_active":"2024-04-26T01:45:54.962135+0000","last_peered":"2024-04-26T01:45:54.962135+0000","last_clean":"2024-04-26T01:45:54.962135+0000","last_became_active":"2024-04-26T01:42:05.391943+0000","last_became_peered":"2024-04-26T01:42:05.391943+0000","last_unstale":"2024-04-26T01:45:54.962135+0000","last_undegraded":"2024-04-26T01:45:54.962135+0000","last_fullsized":"2024-04-26T01:45:54.962135+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:38:51.560596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":601506,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1390,"num_read_kb":813,"num_write":303,"num_write_kb":1447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2039808,"data_stored":1804518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285631,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1077248,"data_stored":519573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547267,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41232,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703920,"statfs":{"total":95995035648,"available":95952814080,"internally_reserved":0,"allocated":1773568,"data_stored":1181616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710794,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28464,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716688,"statfs":{"total":95995035648,"available":95965888512,"internally_reserved":0,"allocated":1478656,"data_stored":912913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972434,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36776,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708376,"statfs":{"total":95995035648,"available":95957377024,"internally_reserved":0,"allocated":1470464,"data_stored":912721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266773,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1679360,"data_stored":1113312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430298,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":519086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40620,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704532,"statfs":{"total":95995035648,"available":95953440768,"internally_reserved":0,"allocated":1146880,"data_stored":589803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738492,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":520357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411442,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28164,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716988,"statfs":{"total":95995035648,"available":95966195712,"internally_reserved":0,"allocated":1171456,"data_stored":592386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28596,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716556,"statfs":{"total":95995035648,"available":95965753344,"internally_reserved":0,"allocated":1617920,"data_stored":1044213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888166,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":520446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691937,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37120,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708032,"statfs":{"total":95995035648,"available":95957024768,"internally_reserved":0,"allocated":1826816,"data_stored":1243673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":70924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":73507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":393842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":205025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:46:01.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:46:01.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi006 to 177.5M 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: Unable to set osd_memory_target on smithi006 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:46:01.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: Updating smithi006:/etc/ceph/ceph.conf 2024-04-26T01:46:01.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: Updating smithi029:/etc/ceph/ceph.conf 2024-04-26T01:46:01.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-26T01:46:01.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:46:01.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:46:01.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:46:01.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: pgmap v6: 97 pgs: 97 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 120 KiB/s wr, 39 op/s 2024-04-26T01:46:01.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:46:01.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:46:01.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: Updating smithi006:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:46:01.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:01 smithi006 ceph-mon[28419]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:46:01.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:01.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:01.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi006 to 177.5M 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: Unable to set osd_memory_target on smithi006 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: Updating smithi006:/etc/ceph/ceph.conf 2024-04-26T01:46:01.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: Updating smithi029:/etc/ceph/ceph.conf 2024-04-26T01:46:01.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-26T01:46:01.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:46:01.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:46:01.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:46:01.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: pgmap v6: 97 pgs: 97 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 120 KiB/s wr, 39 op/s 2024-04-26T01:46:01.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:46:01.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:46:01.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: Updating smithi006:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:46:01.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:01 smithi188 ceph-mon[32751]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:46:01.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:01.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:01.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi006 to 177.5M 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: Unable to set osd_memory_target on smithi006 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: Updating smithi006:/etc/ceph/ceph.conf 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: Updating smithi029:/etc/ceph/ceph.conf 2024-04-26T01:46:01.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-26T01:46:01.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:46:01.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:46:01.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:46:01.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: pgmap v6: 97 pgs: 97 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 120 KiB/s wr, 39 op/s 2024-04-26T01:46:01.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:46:01.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:46:01.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: Updating smithi006:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:46:01.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:01 smithi029 ceph-mon[32775]: Updating smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:46:02.160 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:02.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:02 smithi006 ceph-mon[28419]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:46:02.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:02 smithi006 ceph-mon[28419]: mgrmap e26: x(active, since 6s), standbys: y, z 2024-04-26T01:46:02.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:02 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:02.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:02 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:02.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:02 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:02.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:02 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:02.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:02 smithi006 ceph-mon[28419]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:46:02.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:02 smithi188 ceph-mon[32751]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:46:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:02 smithi188 ceph-mon[32751]: mgrmap e26: x(active, since 6s), standbys: y, z 2024-04-26T01:46:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:02 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:02 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:02 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:02 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:02 smithi188 ceph-mon[32751]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:46:02.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:02 smithi029 ceph-mon[32775]: Updating smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:46:02.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:02 smithi029 ceph-mon[32775]: mgrmap e26: x(active, since 6s), standbys: y, z 2024-04-26T01:46:02.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:02 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:02.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:02 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:02.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:02 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:02.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:02 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:02 smithi029 ceph-mon[32775]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:46:04.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.248+0000 7fcde7e00700 1 -- 172.21.15.6:0/2782246227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde810e2b0 msgr2=0x7fcde8115e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:04.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.248+0000 7fcde7e00700 1 --2- 172.21.15.6:0/2782246227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde810e2b0 0x7fcde8115e80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcde000b0d0 tx=0x7fcde000b3e0 comp rx=0 tx=0).stop 2024-04-26T01:46:04.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.249+0000 7fcde7e00700 1 -- 172.21.15.6:0/2782246227 shutdown_connections 2024-04-26T01:46:04.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.249+0000 7fcde7e00700 1 --2- 172.21.15.6:0/2782246227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde810e2b0 0x7fcde8115e80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:04.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.249+0000 7fcde7e00700 1 --2- 172.21.15.6:0/2782246227 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcde81091e0 0x7fcde81095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:04.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.249+0000 7fcde7e00700 1 --2- 172.21.15.6:0/2782246227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde8109b90 0x7fcde810dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:04.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.249+0000 7fcde7e00700 1 -- 172.21.15.6:0/2782246227 >> 172.21.15.6:0/2782246227 conn(0x7fcde80787a0 msgr2=0x7fcde8107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:04.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.249+0000 7fcde7e00700 1 -- 172.21.15.6:0/2782246227 shutdown_connections 2024-04-26T01:46:04.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.249+0000 7fcde7e00700 1 -- 172.21.15.6:0/2782246227 wait complete. 2024-04-26T01:46:04.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.250+0000 7fcde7e00700 1 Processor -- start 2024-04-26T01:46:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.250+0000 7fcde7e00700 1 -- start start 2024-04-26T01:46:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.250+0000 7fcde7e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde81091e0 0x7fcde819f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde7e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcde8109b90 0x7fcde819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde7e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde810e2b0 0x7fcde81a3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde7e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcde8119150 con 0x7fcde810e2b0 2024-04-26T01:46:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde7e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcde8118fd0 con 0x7fcde81091e0 2024-04-26T01:46:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde7e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81192d0 con 0x7fcde8109b90 2024-04-26T01:46:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde6a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde81091e0 0x7fcde819f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde6a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde81091e0 0x7fcde819f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41652/0 (socket says 172.21.15.6:41652) 2024-04-26T01:46:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde6a00700 1 -- 172.21.15.6:0/3770614638 learned_addr learned my addr 172.21.15.6:0/3770614638 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:04.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde7400700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde810e2b0 0x7fcde81a3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:04.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde6000700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcde8109b90 0x7fcde819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:04.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde6a00700 1 -- 172.21.15.6:0/3770614638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcde8109b90 msgr2=0x7fcde819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:04.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde6a00700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcde8109b90 0x7fcde819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:04.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde6a00700 1 -- 172.21.15.6:0/3770614638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde810e2b0 msgr2=0x7fcde81a3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:04.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde6a00700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde810e2b0 0x7fcde81a3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:04.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.251+0000 7fcde6a00700 1 -- 172.21.15.6:0/3770614638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde81a4570 con 0x7fcde81091e0 2024-04-26T01:46:04.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.252+0000 7fcde6a00700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde81091e0 0x7fcde819f560 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcdd800e9a0 tx=0x7fcdd800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:04.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.252+0000 7fcddf400700 1 -- 172.21.15.6:0/3770614638 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd800ca10 con 0x7fcde81091e0 2024-04-26T01:46:04.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.252+0000 7fcde7e00700 1 -- 172.21.15.6:0/3770614638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcde81a4860 con 0x7fcde81091e0 2024-04-26T01:46:04.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.252+0000 7fcde7e00700 1 -- 172.21.15.6:0/3770614638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcde81ac0e0 con 0x7fcde81091e0 2024-04-26T01:46:04.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.252+0000 7fcddf400700 1 -- 172.21.15.6:0/3770614638 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd800cbb0 con 0x7fcde81091e0 2024-04-26T01:46:04.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.252+0000 7fcddf400700 1 -- 172.21.15.6:0/3770614638 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8018420 con 0x7fcde81091e0 2024-04-26T01:46:04.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.254+0000 7fcde7e00700 1 -- 172.21.15.6:0/3770614638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc8005360 con 0x7fcde81091e0 2024-04-26T01:46:04.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.256+0000 7fcddf400700 1 -- 172.21.15.6:0/3770614638 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7fcdd8010430 con 0x7fcde81091e0 2024-04-26T01:46:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.257+0000 7fcddf400700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcdcc0a3f50 0x7fcdcc0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.257+0000 7fcddf400700 1 -- 172.21.15.6:0/3770614638 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7fcdd80d0c20 con 0x7fcde81091e0 2024-04-26T01:46:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.257+0000 7fcde6000700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcdcc0a3f50 0x7fcdcc0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:04.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.258+0000 7fcde6000700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcdcc0a3f50 0x7fcdcc0a6410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcde810f5c0 tx=0x7fcdd4009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:04.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.262+0000 7fcddf400700 1 -- 172.21.15.6:0/3770614638 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcdd809cff0 con 0x7fcde81091e0 2024-04-26T01:46:04.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.487+0000 7fcde7e00700 1 -- 172.21.15.6:0/3770614638 --> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcdc8000bf0 con 0x7fcdcc0a3f50 2024-04-26T01:46:04.486 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:04 smithi006 ceph-mon[28419]: Updating smithi006:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:46:04.486 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:04 smithi006 ceph-mon[28419]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 169 KiB/s wr, 100 op/s 2024-04-26T01:46:04.486 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:04 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:04.486 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:04 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:04.486 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:04 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:04.486 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:04 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:46:04.486 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:04 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:46:04.486 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:04 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:46:04.486 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:04 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:04.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.514+0000 7fcddf400700 1 -- 172.21.15.6:0/3770614638 <== mgr.25037 v2:172.21.15.6:6800/3353570381 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257777 (secure 0 0 0) 0x7fcdc8000bf0 con 0x7fcdcc0a3f50 2024-04-26T01:46:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:46:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.522+0000 7fcde7e00700 1 -- 172.21.15.6:0/3770614638 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcdcc0a3f50 msgr2=0x7fcdcc0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.522+0000 7fcde7e00700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcdcc0a3f50 0x7fcdcc0a6410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcde810f5c0 tx=0x7fcdd4009260 comp rx=0 tx=0).stop 2024-04-26T01:46:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.522+0000 7fcde7e00700 1 -- 172.21.15.6:0/3770614638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde81091e0 msgr2=0x7fcde819f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.522+0000 7fcde7e00700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde81091e0 0x7fcde819f560 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcdd800e9a0 tx=0x7fcdd800ecb0 comp rx=0 tx=0).stop 2024-04-26T01:46:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.523+0000 7fcde7e00700 1 -- 172.21.15.6:0/3770614638 shutdown_connections 2024-04-26T01:46:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.523+0000 7fcde7e00700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcdcc0a3f50 0x7fcdcc0a6410 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.523+0000 7fcde7e00700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde810e2b0 0x7fcde81a3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.523+0000 7fcde7e00700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde81091e0 0x7fcde819f560 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.523+0000 7fcde7e00700 1 --2- 172.21.15.6:0/3770614638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcde8109b90 0x7fcde819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.523+0000 7fcde7e00700 1 -- 172.21.15.6:0/3770614638 >> 172.21.15.6:0/3770614638 conn(0x7fcde80787a0 msgr2=0x7fcde8078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.523+0000 7fcde7e00700 1 -- 172.21.15.6:0/3770614638 shutdown_connections 2024-04-26T01:46:04.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:04.523+0000 7fcde7e00700 1 -- 172.21.15.6:0/3770614638 wait complete. 2024-04-26T01:46:04.523 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-26T01:46:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:04 smithi188 ceph-mon[32751]: Updating smithi006:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:46:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:04 smithi188 ceph-mon[32751]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 169 KiB/s wr, 100 op/s 2024-04-26T01:46:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:04 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:04 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:04 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:04 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:46:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:04 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:46:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:04 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:46:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:04 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:04 smithi029 ceph-mon[32775]: Updating smithi006:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:46:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:04 smithi029 ceph-mon[32775]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 169 KiB/s wr, 100 op/s 2024-04-26T01:46:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:04 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:04 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:04 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:04 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:46:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:04 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:46:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:04 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:46:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:04 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:46:05.441 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-04-26T01:46:02.957654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1390569,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":2026,"num_read_kb":1495,"num_write":688,"num_write_kb":5567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1001,"ondisk_log_size":1001,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":395468,"kb_used_data":16812,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124546356,"statfs":{"total":1151940427776,"available":1151535468544,"internally_reserved":0,"allocated":17215488,"data_stored":10396426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":198461,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":500,"num_read_kb":563,"num_write":202,"num_write_kb":1180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.980004"},"pg_stats":[{"pgid":"3.3f","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962627+0000","last_change":"2024-04-26T01:42:05.417304+0000","last_active":"2024-04-26T01:45:54.962627+0000","last_peered":"2024-04-26T01:45:54.962627+0000","last_clean":"2024-04-26T01:45:54.962627+0000","last_became_active":"2024-04-26T01:42:05.416999+0000","last_became_peered":"2024-04-26T01:42:05.416999+0000","last_unstale":"2024-04-26T01:45:54.962627+0000","last_undegraded":"2024-04-26T01:45:54.962627+0000","last_fullsized":"2024-04-26T01:45:54.962627+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:33:30.958943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"95'4","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.184448+0000","last_change":"2024-04-26T01:42:05.413089+0000","last_active":"2024-04-26T01:45:55.184448+0000","last_peered":"2024-04-26T01:45:55.184448+0000","last_clean":"2024-04-26T01:45:55.184448+0000","last_became_active":"2024-04-26T01:42:05.412776+0000","last_became_peered":"2024-04-26T01:42:05.412776+0000","last_unstale":"2024-04-26T01:45:55.184448+0000","last_undegraded":"2024-04-26T01:45:55.184448+0000","last_fullsized":"2024-04-26T01:45:55.184448+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:47.861340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.184102+0000","last_change":"2024-04-26T01:42:05.404864+0000","last_active":"2024-04-26T01:45:55.184102+0000","last_peered":"2024-04-26T01:45:55.184102+0000","last_clean":"2024-04-26T01:45:55.184102+0000","last_became_active":"2024-04-26T01:42:05.404529+0000","last_became_peered":"2024-04-26T01:42:05.404529+0000","last_unstale":"2024-04-26T01:45:55.184102+0000","last_undegraded":"2024-04-26T01:45:55.184102+0000","last_fullsized":"2024-04-26T01:45:55.184102+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:04:18.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.947186+0000","last_change":"2024-04-26T01:42:05.393582+0000","last_active":"2024-04-26T01:45:54.947186+0000","last_peered":"2024-04-26T01:45:54.947186+0000","last_clean":"2024-04-26T01:45:54.947186+0000","last_became_active":"2024-04-26T01:42:05.393193+0000","last_became_peered":"2024-04-26T01:42:05.393193+0000","last_unstale":"2024-04-26T01:45:54.947186+0000","last_undegraded":"2024-04-26T01:45:54.947186+0000","last_fullsized":"2024-04-26T01:45:54.947186+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:20:29.420459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"95'4","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962087+0000","last_change":"2024-04-26T01:42:05.412979+0000","last_active":"2024-04-26T01:45:54.962087+0000","last_peered":"2024-04-26T01:45:54.962087+0000","last_clean":"2024-04-26T01:45:54.962087+0000","last_became_active":"2024-04-26T01:42:05.412679+0000","last_became_peered":"2024-04-26T01:42:05.412679+0000","last_unstale":"2024-04-26T01:45:54.962087+0000","last_undegraded":"2024-04-26T01:45:54.962087+0000","last_fullsized":"2024-04-26T01:45:54.962087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:22.774960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"95'6","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931536+0000","last_change":"2024-04-26T01:42:49.062507+0000","last_active":"2024-04-26T01:45:54.931536+0000","last_peered":"2024-04-26T01:45:54.931536+0000","last_clean":"2024-04-26T01:45:54.931536+0000","last_became_active":"2024-04-26T01:42:49.062095+0000","last_became_peered":"2024-04-26T01:42:49.062095+0000","last_unstale":"2024-04-26T01:45:54.931536+0000","last_undegraded":"2024-04-26T01:45:54.931536+0000","last_fullsized":"2024-04-26T01:45:54.931536+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:47.846697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962044+0000","last_change":"2024-04-26T01:42:05.385221+0000","last_active":"2024-04-26T01:45:54.962044+0000","last_peered":"2024-04-26T01:45:54.962044+0000","last_clean":"2024-04-26T01:45:54.962044+0000","last_became_active":"2024-04-26T01:42:05.384856+0000","last_became_peered":"2024-04-26T01:42:05.384856+0000","last_unstale":"2024-04-26T01:45:54.962044+0000","last_undegraded":"2024-04-26T01:45:54.962044+0000","last_fullsized":"2024-04-26T01:45:54.962044+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:07:23.461324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949107+0000","last_change":"2024-04-26T01:42:05.404754+0000","last_active":"2024-04-26T01:45:54.949107+0000","last_peered":"2024-04-26T01:45:54.949107+0000","last_clean":"2024-04-26T01:45:54.949107+0000","last_became_active":"2024-04-26T01:42:05.404220+0000","last_became_peered":"2024-04-26T01:42:05.404220+0000","last_unstale":"2024-04-26T01:45:54.949107+0000","last_undegraded":"2024-04-26T01:45:54.949107+0000","last_fullsized":"2024-04-26T01:45:54.949107+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:50.243552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"95'11","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961852+0000","last_change":"2024-04-26T01:42:05.390972+0000","last_active":"2024-04-26T01:45:54.961852+0000","last_peered":"2024-04-26T01:45:54.961852+0000","last_clean":"2024-04-26T01:45:54.961852+0000","last_became_active":"2024-04-26T01:42:05.390599+0000","last_became_peered":"2024-04-26T01:42:05.390599+0000","last_unstale":"2024-04-26T01:45:54.961852+0000","last_undegraded":"2024-04-26T01:45:54.961852+0000","last_fullsized":"2024-04-26T01:45:54.961852+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:19:46.328640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"95'16","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932040+0000","last_change":"2024-04-26T01:42:05.392320+0000","last_active":"2024-04-26T01:45:54.932040+0000","last_peered":"2024-04-26T01:45:54.932040+0000","last_clean":"2024-04-26T01:45:54.932040+0000","last_became_active":"2024-04-26T01:42:05.391986+0000","last_became_peered":"2024-04-26T01:42:05.391986+0000","last_unstale":"2024-04-26T01:45:54.932040+0000","last_undegraded":"2024-04-26T01:45:54.932040+0000","last_fullsized":"2024-04-26T01:45:54.932040+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:17:27.302358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962508+0000","last_change":"2024-04-26T01:42:05.388077+0000","last_active":"2024-04-26T01:45:54.962508+0000","last_peered":"2024-04-26T01:45:54.962508+0000","last_clean":"2024-04-26T01:45:54.962508+0000","last_became_active":"2024-04-26T01:42:05.387545+0000","last_became_peered":"2024-04-26T01:42:05.387545+0000","last_unstale":"2024-04-26T01:45:54.962508+0000","last_undegraded":"2024-04-26T01:45:54.962508+0000","last_fullsized":"2024-04-26T01:45:54.962508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:51.288177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"95'7","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932538+0000","last_change":"2024-04-26T01:42:49.067449+0000","last_active":"2024-04-26T01:45:54.932538+0000","last_peered":"2024-04-26T01:45:54.932538+0000","last_clean":"2024-04-26T01:45:54.932538+0000","last_became_active":"2024-04-26T01:42:49.065991+0000","last_became_peered":"2024-04-26T01:42:49.065991+0000","last_unstale":"2024-04-26T01:45:54.932538+0000","last_undegraded":"2024-04-26T01:45:54.932538+0000","last_fullsized":"2024-04-26T01:45:54.932538+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:13:36.565257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"95'10","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944606+0000","last_change":"2024-04-26T01:42:05.402847+0000","last_active":"2024-04-26T01:45:54.944606+0000","last_peered":"2024-04-26T01:45:54.944606+0000","last_clean":"2024-04-26T01:45:54.944606+0000","last_became_active":"2024-04-26T01:42:05.402503+0000","last_became_peered":"2024-04-26T01:42:05.402503+0000","last_unstale":"2024-04-26T01:45:54.944606+0000","last_undegraded":"2024-04-26T01:45:54.944606+0000","last_fullsized":"2024-04-26T01:45:54.944606+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:06.935060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.183651+0000","last_change":"2024-04-26T01:42:05.405257+0000","last_active":"2024-04-26T01:45:55.183651+0000","last_peered":"2024-04-26T01:45:55.183651+0000","last_clean":"2024-04-26T01:45:55.183651+0000","last_became_active":"2024-04-26T01:42:05.404551+0000","last_became_peered":"2024-04-26T01:42:05.404551+0000","last_unstale":"2024-04-26T01:45:55.183651+0000","last_undegraded":"2024-04-26T01:45:55.183651+0000","last_fullsized":"2024-04-26T01:45:55.183651+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:20.754088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.183628+0000","last_change":"2024-04-26T01:42:05.408847+0000","last_active":"2024-04-26T01:45:55.183628+0000","last_peered":"2024-04-26T01:45:55.183628+0000","last_clean":"2024-04-26T01:45:55.183628+0000","last_became_active":"2024-04-26T01:42:05.406189+0000","last_became_peered":"2024-04-26T01:42:05.406189+0000","last_unstale":"2024-04-26T01:45:55.183628+0000","last_undegraded":"2024-04-26T01:45:55.183628+0000","last_fullsized":"2024-04-26T01:45:55.183628+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:59:58.633630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961111+0000","last_change":"2024-04-26T01:42:05.404647+0000","last_active":"2024-04-26T01:45:54.961111+0000","last_peered":"2024-04-26T01:45:54.961111+0000","last_clean":"2024-04-26T01:45:54.961111+0000","last_became_active":"2024-04-26T01:42:05.404265+0000","last_became_peered":"2024-04-26T01:42:05.404265+0000","last_unstale":"2024-04-26T01:45:54.961111+0000","last_undegraded":"2024-04-26T01:45:54.961111+0000","last_fullsized":"2024-04-26T01:45:54.961111+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:00:43.472541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962928+0000","last_change":"2024-04-26T01:42:05.417602+0000","last_active":"2024-04-26T01:45:54.962928+0000","last_peered":"2024-04-26T01:45:54.962928+0000","last_clean":"2024-04-26T01:45:54.962928+0000","last_became_active":"2024-04-26T01:42:05.417144+0000","last_became_peered":"2024-04-26T01:42:05.417144+0000","last_unstale":"2024-04-26T01:45:54.962928+0000","last_undegraded":"2024-04-26T01:45:54.962928+0000","last_fullsized":"2024-04-26T01:45:54.962928+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:34.043597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"95'10","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.934166+0000","last_change":"2024-04-26T01:42:05.392948+0000","last_active":"2024-04-26T01:45:54.934166+0000","last_peered":"2024-04-26T01:45:54.934166+0000","last_clean":"2024-04-26T01:45:54.934166+0000","last_became_active":"2024-04-26T01:42:05.392040+0000","last_became_peered":"2024-04-26T01:42:05.392040+0000","last_unstale":"2024-04-26T01:45:54.934166+0000","last_undegraded":"2024-04-26T01:45:54.934166+0000","last_fullsized":"2024-04-26T01:45:54.934166+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:17.307979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931776+0000","last_change":"2024-04-26T01:42:05.406588+0000","last_active":"2024-04-26T01:45:54.931776+0000","last_peered":"2024-04-26T01:45:54.931776+0000","last_clean":"2024-04-26T01:45:54.931776+0000","last_became_active":"2024-04-26T01:42:05.406200+0000","last_became_peered":"2024-04-26T01:42:05.406200+0000","last_unstale":"2024-04-26T01:45:54.931776+0000","last_undegraded":"2024-04-26T01:45:54.931776+0000","last_fullsized":"2024-04-26T01:45:54.931776+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:14.095383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932704+0000","last_change":"2024-04-26T01:42:01.339289+0000","last_active":"2024-04-26T01:45:54.932704+0000","last_peered":"2024-04-26T01:45:54.932704+0000","last_clean":"2024-04-26T01:45:54.932704+0000","last_became_active":"2024-04-26T01:42:01.335402+0000","last_became_peered":"2024-04-26T01:42:01.335402+0000","last_unstale":"2024-04-26T01:45:54.932704+0000","last_undegraded":"2024-04-26T01:45:54.932704+0000","last_fullsized":"2024-04-26T01:45:54.932704+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:51.411524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.950320+0000","last_change":"2024-04-26T01:42:05.409569+0000","last_active":"2024-04-26T01:45:54.950320+0000","last_peered":"2024-04-26T01:45:54.950320+0000","last_clean":"2024-04-26T01:45:54.950320+0000","last_became_active":"2024-04-26T01:42:05.407441+0000","last_became_peered":"2024-04-26T01:42:05.407441+0000","last_unstale":"2024-04-26T01:45:54.950320+0000","last_undegraded":"2024-04-26T01:45:54.950320+0000","last_fullsized":"2024-04-26T01:45:54.950320+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:36:58.043359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.951688+0000","last_change":"2024-04-26T01:42:02.231871+0000","last_active":"2024-04-26T01:45:54.951688+0000","last_peered":"2024-04-26T01:45:54.951688+0000","last_clean":"2024-04-26T01:45:54.951688+0000","last_became_active":"2024-04-26T01:42:02.230813+0000","last_became_peered":"2024-04-26T01:42:02.230813+0000","last_unstale":"2024-04-26T01:45:54.951688+0000","last_undegraded":"2024-04-26T01:45:54.951688+0000","last_fullsized":"2024-04-26T01:45:54.951688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:53.117266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.183893+0000","last_change":"2024-04-26T01:42:05.412414+0000","last_active":"2024-04-26T01:45:55.183893+0000","last_peered":"2024-04-26T01:45:55.183893+0000","last_clean":"2024-04-26T01:45:55.183893+0000","last_became_active":"2024-04-26T01:42:05.412057+0000","last_became_peered":"2024-04-26T01:42:05.412057+0000","last_unstale":"2024-04-26T01:45:55.183893+0000","last_undegraded":"2024-04-26T01:45:55.183893+0000","last_fullsized":"2024-04-26T01:45:55.183893+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:51:46.331246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.963097+0000","last_change":"2024-04-26T01:42:01.329957+0000","last_active":"2024-04-26T01:45:54.963097+0000","last_peered":"2024-04-26T01:45:54.963097+0000","last_clean":"2024-04-26T01:45:54.963097+0000","last_became_active":"2024-04-26T01:42:01.329289+0000","last_became_peered":"2024-04-26T01:42:01.329289+0000","last_unstale":"2024-04-26T01:45:54.963097+0000","last_undegraded":"2024-04-26T01:45:54.963097+0000","last_fullsized":"2024-04-26T01:45:54.963097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:46:38.983138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961476+0000","last_change":"2024-04-26T01:42:05.399438+0000","last_active":"2024-04-26T01:45:54.961476+0000","last_peered":"2024-04-26T01:45:54.961476+0000","last_clean":"2024-04-26T01:45:54.961476+0000","last_became_active":"2024-04-26T01:42:05.399082+0000","last_became_peered":"2024-04-26T01:42:05.399082+0000","last_unstale":"2024-04-26T01:45:54.961476+0000","last_undegraded":"2024-04-26T01:45:54.961476+0000","last_fullsized":"2024-04-26T01:45:54.961476+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:13.619893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"91'2","reported_seq":17,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961916+0000","last_change":"2024-04-26T01:43:49.993134+0000","last_active":"2024-04-26T01:45:54.961916+0000","last_peered":"2024-04-26T01:45:54.961916+0000","last_clean":"2024-04-26T01:45:54.961916+0000","last_became_active":"2024-04-26T01:43:49.990337+0000","last_became_peered":"2024-04-26T01:43:49.990337+0000","last_unstale":"2024-04-26T01:45:54.961916+0000","last_undegraded":"2024-04-26T01:45:54.961916+0000","last_fullsized":"2024-04-26T01:45:54.961916+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:50.502668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"95'15","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.933121+0000","last_change":"2024-04-26T01:42:49.066057+0000","last_active":"2024-04-26T01:45:54.933121+0000","last_peered":"2024-04-26T01:45:54.933121+0000","last_clean":"2024-04-26T01:45:54.933121+0000","last_became_active":"2024-04-26T01:42:49.064684+0000","last_became_peered":"2024-04-26T01:42:49.064684+0000","last_unstale":"2024-04-26T01:45:54.933121+0000","last_undegraded":"2024-04-26T01:45:54.933121+0000","last_fullsized":"2024-04-26T01:45:54.933121+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:39.159528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"96'9","reported_seq":261,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:58.971147+0000","last_change":"2024-04-26T01:42:01.332623+0000","last_active":"2024-04-26T01:45:58.971147+0000","last_peered":"2024-04-26T01:45:58.971147+0000","last_clean":"2024-04-26T01:45:58.971147+0000","last_became_active":"2024-04-26T01:42:01.332227+0000","last_became_peered":"2024-04-26T01:42:01.332227+0000","last_unstale":"2024-04-26T01:45:58.971147+0000","last_undegraded":"2024-04-26T01:45:58.971147+0000","last_fullsized":"2024-04-26T01:45:58.971147+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:58.611464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1855,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":219,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961832+0000","last_change":"2024-04-26T01:42:05.396133+0000","last_active":"2024-04-26T01:45:54.961832+0000","last_peered":"2024-04-26T01:45:54.961832+0000","last_clean":"2024-04-26T01:45:54.961832+0000","last_became_active":"2024-04-26T01:42:05.395553+0000","last_became_peered":"2024-04-26T01:42:05.395553+0000","last_unstale":"2024-04-26T01:45:54.961832+0000","last_undegraded":"2024-04-26T01:45:54.961832+0000","last_fullsized":"2024-04-26T01:45:54.961832+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:36:01.363909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931431+0000","last_change":"2024-04-26T01:42:01.340624+0000","last_active":"2024-04-26T01:45:54.931431+0000","last_peered":"2024-04-26T01:45:54.931431+0000","last_clean":"2024-04-26T01:45:54.931431+0000","last_became_active":"2024-04-26T01:42:01.340158+0000","last_became_peered":"2024-04-26T01:42:01.340158+0000","last_unstale":"2024-04-26T01:45:54.931431+0000","last_undegraded":"2024-04-26T01:45:54.931431+0000","last_fullsized":"2024-04-26T01:45:54.931431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:47:05.529922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.963143+0000","last_change":"2024-04-26T01:42:05.406354+0000","last_active":"2024-04-26T01:45:54.963143+0000","last_peered":"2024-04-26T01:45:54.963143+0000","last_clean":"2024-04-26T01:45:54.963143+0000","last_became_active":"2024-04-26T01:42:05.405989+0000","last_became_peered":"2024-04-26T01:42:05.405989+0000","last_unstale":"2024-04-26T01:45:54.963143+0000","last_undegraded":"2024-04-26T01:45:54.963143+0000","last_fullsized":"2024-04-26T01:45:54.963143+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:43.115225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944206+0000","last_change":"2024-04-26T01:42:01.340549+0000","last_active":"2024-04-26T01:45:54.944206+0000","last_peered":"2024-04-26T01:45:54.944206+0000","last_clean":"2024-04-26T01:45:54.944206+0000","last_became_active":"2024-04-26T01:42:01.339861+0000","last_became_peered":"2024-04-26T01:42:01.339861+0000","last_unstale":"2024-04-26T01:45:54.944206+0000","last_undegraded":"2024-04-26T01:45:54.944206+0000","last_fullsized":"2024-04-26T01:45:54.944206+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:02:53.154367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949325+0000","last_change":"2024-04-26T01:42:05.402664+0000","last_active":"2024-04-26T01:45:54.949325+0000","last_peered":"2024-04-26T01:45:54.949325+0000","last_clean":"2024-04-26T01:45:54.949325+0000","last_became_active":"2024-04-26T01:42:05.400785+0000","last_became_peered":"2024-04-26T01:42:05.400785+0000","last_unstale":"2024-04-26T01:45:54.949325+0000","last_undegraded":"2024-04-26T01:45:54.949325+0000","last_fullsized":"2024-04-26T01:45:54.949325+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:40:32.137752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961622+0000","last_change":"2024-04-26T01:42:01.328600+0000","last_active":"2024-04-26T01:45:54.961622+0000","last_peered":"2024-04-26T01:45:54.961622+0000","last_clean":"2024-04-26T01:45:54.961622+0000","last_became_active":"2024-04-26T01:42:01.328149+0000","last_became_peered":"2024-04-26T01:42:01.328149+0000","last_unstale":"2024-04-26T01:45:54.961622+0000","last_undegraded":"2024-04-26T01:45:54.961622+0000","last_fullsized":"2024-04-26T01:45:54.961622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:45:42.811088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"95'11","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932233+0000","last_change":"2024-04-26T01:42:49.065824+0000","last_active":"2024-04-26T01:45:54.932233+0000","last_peered":"2024-04-26T01:45:54.932233+0000","last_clean":"2024-04-26T01:45:54.932233+0000","last_became_active":"2024-04-26T01:42:49.064670+0000","last_became_peered":"2024-04-26T01:42:49.064670+0000","last_unstale":"2024-04-26T01:45:54.932233+0000","last_undegraded":"2024-04-26T01:45:54.932233+0000","last_fullsized":"2024-04-26T01:45:54.932233+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:44.790184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"95'5","reported_seq":252,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:59.972522+0000","last_change":"2024-04-26T01:42:01.335438+0000","last_active":"2024-04-26T01:45:59.972522+0000","last_peered":"2024-04-26T01:45:59.972522+0000","last_clean":"2024-04-26T01:45:59.972522+0000","last_became_active":"2024-04-26T01:42:01.334962+0000","last_became_peered":"2024-04-26T01:42:01.334962+0000","last_unstale":"2024-04-26T01:45:59.972522+0000","last_undegraded":"2024-04-26T01:45:59.972522+0000","last_fullsized":"2024-04-26T01:45:59.972522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:35:08.252274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":214,"num_read_kb":214,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"95'10","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961591+0000","last_change":"2024-04-26T01:42:05.393294+0000","last_active":"2024-04-26T01:45:54.961591+0000","last_peered":"2024-04-26T01:45:54.961591+0000","last_clean":"2024-04-26T01:45:54.961591+0000","last_became_active":"2024-04-26T01:42:05.392974+0000","last_became_peered":"2024-04-26T01:42:05.392974+0000","last_unstale":"2024-04-26T01:45:54.961591+0000","last_undegraded":"2024-04-26T01:45:54.961591+0000","last_fullsized":"2024-04-26T01:45:54.961591+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:58.188706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962548+0000","last_change":"2024-04-26T01:42:01.333505+0000","last_active":"2024-04-26T01:45:54.962548+0000","last_peered":"2024-04-26T01:45:54.962548+0000","last_clean":"2024-04-26T01:45:54.962548+0000","last_became_active":"2024-04-26T01:42:01.333062+0000","last_became_peered":"2024-04-26T01:42:01.333062+0000","last_unstale":"2024-04-26T01:45:54.962548+0000","last_undegraded":"2024-04-26T01:45:54.962548+0000","last_fullsized":"2024-04-26T01:45:54.962548+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:50:48.804443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"95'9","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.184062+0000","last_change":"2024-04-26T01:42:49.063948+0000","last_active":"2024-04-26T01:45:55.184062+0000","last_peered":"2024-04-26T01:45:55.184062+0000","last_clean":"2024-04-26T01:45:55.184062+0000","last_became_active":"2024-04-26T01:42:49.060852+0000","last_became_peered":"2024-04-26T01:42:49.060852+0000","last_unstale":"2024-04-26T01:45:55.184062+0000","last_undegraded":"2024-04-26T01:45:55.184062+0000","last_fullsized":"2024-04-26T01:45:55.184062+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.370431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.963018+0000","last_change":"2024-04-26T01:42:02.232308+0000","last_active":"2024-04-26T01:45:54.963018+0000","last_peered":"2024-04-26T01:45:54.963018+0000","last_clean":"2024-04-26T01:45:54.963018+0000","last_became_active":"2024-04-26T01:42:02.231897+0000","last_became_peered":"2024-04-26T01:42:02.231897+0000","last_unstale":"2024-04-26T01:45:54.963018+0000","last_undegraded":"2024-04-26T01:45:54.963018+0000","last_fullsized":"2024-04-26T01:45:54.963018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:36:16.847087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"95'11","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961415+0000","last_change":"2024-04-26T01:42:49.070798+0000","last_active":"2024-04-26T01:45:54.961415+0000","last_peered":"2024-04-26T01:45:54.961415+0000","last_clean":"2024-04-26T01:45:54.961415+0000","last_became_active":"2024-04-26T01:42:49.070470+0000","last_became_peered":"2024-04-26T01:42:49.070470+0000","last_unstale":"2024-04-26T01:45:54.961415+0000","last_undegraded":"2024-04-26T01:45:54.961415+0000","last_fullsized":"2024-04-26T01:45:54.961415+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:04:22.103704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"96'42","reported_seq":518,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:46:00.859508+0000","last_change":"2024-04-26T01:42:01.330822+0000","last_active":"2024-04-26T01:46:00.859508+0000","last_peered":"2024-04-26T01:46:00.859508+0000","last_clean":"2024-04-26T01:46:00.859508+0000","last_became_active":"2024-04-26T01:42:01.330337+0000","last_became_peered":"2024-04-26T01:42:01.330337+0000","last_unstale":"2024-04-26T01:46:00.859508+0000","last_undegraded":"2024-04-26T01:46:00.859508+0000","last_fullsized":"2024-04-26T01:46:00.859508+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:04.024645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72362,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":443,"num_read_kb":293,"num_write":43,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962763+0000","last_change":"2024-04-26T01:42:05.390601+0000","last_active":"2024-04-26T01:45:54.962763+0000","last_peered":"2024-04-26T01:45:54.962763+0000","last_clean":"2024-04-26T01:45:54.962763+0000","last_became_active":"2024-04-26T01:42:05.390279+0000","last_became_peered":"2024-04-26T01:42:05.390279+0000","last_unstale":"2024-04-26T01:45:54.962763+0000","last_undegraded":"2024-04-26T01:45:54.962763+0000","last_fullsized":"2024-04-26T01:45:54.962763+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:20.171769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931520+0000","last_change":"2024-04-26T01:42:01.337200+0000","last_active":"2024-04-26T01:45:54.931520+0000","last_peered":"2024-04-26T01:45:54.931520+0000","last_clean":"2024-04-26T01:45:54.931520+0000","last_became_active":"2024-04-26T01:42:01.336919+0000","last_became_peered":"2024-04-26T01:42:01.336919+0000","last_unstale":"2024-04-26T01:45:54.931520+0000","last_undegraded":"2024-04-26T01:45:54.931520+0000","last_fullsized":"2024-04-26T01:45:54.931520+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:29.799903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944349+0000","last_change":"2024-04-26T01:42:05.410081+0000","last_active":"2024-04-26T01:45:54.944349+0000","last_peered":"2024-04-26T01:45:54.944349+0000","last_clean":"2024-04-26T01:45:54.944349+0000","last_became_active":"2024-04-26T01:42:05.409341+0000","last_became_peered":"2024-04-26T01:42:05.409341+0000","last_unstale":"2024-04-26T01:45:54.944349+0000","last_undegraded":"2024-04-26T01:45:54.944349+0000","last_fullsized":"2024-04-26T01:45:54.944349+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:25:31.925296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962585+0000","last_change":"2024-04-26T01:42:01.331702+0000","last_active":"2024-04-26T01:45:54.962585+0000","last_peered":"2024-04-26T01:45:54.962585+0000","last_clean":"2024-04-26T01:45:54.962585+0000","last_became_active":"2024-04-26T01:42:01.331311+0000","last_became_peered":"2024-04-26T01:42:01.331311+0000","last_unstale":"2024-04-26T01:45:54.962585+0000","last_undegraded":"2024-04-26T01:45:54.962585+0000","last_fullsized":"2024-04-26T01:45:54.962585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:54:49.513627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"95'15","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962407+0000","last_change":"2024-04-26T01:42:05.403957+0000","last_active":"2024-04-26T01:45:54.962407+0000","last_peered":"2024-04-26T01:45:54.962407+0000","last_clean":"2024-04-26T01:45:54.962407+0000","last_became_active":"2024-04-26T01:42:05.403583+0000","last_became_peered":"2024-04-26T01:42:05.403583+0000","last_unstale":"2024-04-26T01:45:54.962407+0000","last_undegraded":"2024-04-26T01:45:54.962407+0000","last_fullsized":"2024-04-26T01:45:54.962407+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:12:20.095407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"96'7","reported_seq":260,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:46:00.967994+0000","last_change":"2024-04-26T01:42:01.332456+0000","last_active":"2024-04-26T01:46:00.967994+0000","last_peered":"2024-04-26T01:46:00.967994+0000","last_clean":"2024-04-26T01:46:00.967994+0000","last_became_active":"2024-04-26T01:42:01.331910+0000","last_became_peered":"2024-04-26T01:42:01.331910+0000","last_unstale":"2024-04-26T01:46:00.967994+0000","last_undegraded":"2024-04-26T01:46:00.967994+0000","last_fullsized":"2024-04-26T01:46:00.967994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:13:25.368205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":221,"num_read_kb":7,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"95'6","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961737+0000","last_change":"2024-04-26T01:42:49.062130+0000","last_active":"2024-04-26T01:45:54.961737+0000","last_peered":"2024-04-26T01:45:54.961737+0000","last_clean":"2024-04-26T01:45:54.961737+0000","last_became_active":"2024-04-26T01:42:49.061627+0000","last_became_peered":"2024-04-26T01:42:49.061627+0000","last_unstale":"2024-04-26T01:45:54.961737+0000","last_undegraded":"2024-04-26T01:45:54.961737+0000","last_fullsized":"2024-04-26T01:45:54.961737+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:16.101940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961990+0000","last_change":"2024-04-26T01:42:49.082618+0000","last_active":"2024-04-26T01:45:54.961990+0000","last_peered":"2024-04-26T01:45:54.961990+0000","last_clean":"2024-04-26T01:45:54.961990+0000","last_became_active":"2024-04-26T01:42:49.064654+0000","last_became_peered":"2024-04-26T01:42:49.064654+0000","last_unstale":"2024-04-26T01:45:54.961990+0000","last_undegraded":"2024-04-26T01:45:54.961990+0000","last_fullsized":"2024-04-26T01:45:54.961990+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:13:23.937059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962550+0000","last_change":"2024-04-26T01:42:05.391392+0000","last_active":"2024-04-26T01:45:54.962550+0000","last_peered":"2024-04-26T01:45:54.962550+0000","last_clean":"2024-04-26T01:45:54.962550+0000","last_became_active":"2024-04-26T01:42:05.390939+0000","last_became_peered":"2024-04-26T01:42:05.390939+0000","last_unstale":"2024-04-26T01:45:54.962550+0000","last_undegraded":"2024-04-26T01:45:54.962550+0000","last_fullsized":"2024-04-26T01:45:54.962550+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:36.141915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962306+0000","last_change":"2024-04-26T01:42:01.332932+0000","last_active":"2024-04-26T01:45:54.962306+0000","last_peered":"2024-04-26T01:45:54.962306+0000","last_clean":"2024-04-26T01:45:54.962306+0000","last_became_active":"2024-04-26T01:42:01.332626+0000","last_became_peered":"2024-04-26T01:42:01.332626+0000","last_unstale":"2024-04-26T01:45:54.962306+0000","last_undegraded":"2024-04-26T01:45:54.962306+0000","last_fullsized":"2024-04-26T01:45:54.962306+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:20.593730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962441+0000","last_change":"2024-04-26T01:42:05.394925+0000","last_active":"2024-04-26T01:45:54.962441+0000","last_peered":"2024-04-26T01:45:54.962441+0000","last_clean":"2024-04-26T01:45:54.962441+0000","last_became_active":"2024-04-26T01:42:05.394531+0000","last_became_peered":"2024-04-26T01:42:05.394531+0000","last_unstale":"2024-04-26T01:45:54.962441+0000","last_undegraded":"2024-04-26T01:45:54.962441+0000","last_fullsized":"2024-04-26T01:45:54.962441+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:33.177780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932471+0000","last_change":"2024-04-26T01:42:02.237482+0000","last_active":"2024-04-26T01:45:54.932471+0000","last_peered":"2024-04-26T01:45:54.932471+0000","last_clean":"2024-04-26T01:45:54.932471+0000","last_became_active":"2024-04-26T01:42:02.236752+0000","last_became_peered":"2024-04-26T01:42:02.236752+0000","last_unstale":"2024-04-26T01:45:54.932471+0000","last_undegraded":"2024-04-26T01:45:54.932471+0000","last_fullsized":"2024-04-26T01:45:54.932471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:27.211245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931723+0000","last_change":"2024-04-26T01:42:05.402157+0000","last_active":"2024-04-26T01:45:54.931723+0000","last_peered":"2024-04-26T01:45:54.931723+0000","last_clean":"2024-04-26T01:45:54.931723+0000","last_became_active":"2024-04-26T01:42:05.401889+0000","last_became_peered":"2024-04-26T01:42:05.401889+0000","last_unstale":"2024-04-26T01:45:54.931723+0000","last_undegraded":"2024-04-26T01:45:54.931723+0000","last_fullsized":"2024-04-26T01:45:54.931723+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:33.854901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962227+0000","last_change":"2024-04-26T01:42:01.339675+0000","last_active":"2024-04-26T01:45:54.962227+0000","last_peered":"2024-04-26T01:45:54.962227+0000","last_clean":"2024-04-26T01:45:54.962227+0000","last_became_active":"2024-04-26T01:42:01.339247+0000","last_became_peered":"2024-04-26T01:42:01.339247+0000","last_unstale":"2024-04-26T01:45:54.962227+0000","last_undegraded":"2024-04-26T01:45:54.962227+0000","last_fullsized":"2024-04-26T01:45:54.962227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:32.724830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931641+0000","last_change":"2024-04-26T01:42:05.412913+0000","last_active":"2024-04-26T01:45:54.931641+0000","last_peered":"2024-04-26T01:45:54.931641+0000","last_clean":"2024-04-26T01:45:54.931641+0000","last_became_active":"2024-04-26T01:42:05.412556+0000","last_became_peered":"2024-04-26T01:42:05.412556+0000","last_unstale":"2024-04-26T01:45:54.931641+0000","last_undegraded":"2024-04-26T01:45:54.931641+0000","last_fullsized":"2024-04-26T01:45:54.931641+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:00:52.798069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961962+0000","last_change":"2024-04-26T01:42:05.391235+0000","last_active":"2024-04-26T01:45:54.961962+0000","last_peered":"2024-04-26T01:45:54.961962+0000","last_clean":"2024-04-26T01:45:54.961962+0000","last_became_active":"2024-04-26T01:42:05.390717+0000","last_became_peered":"2024-04-26T01:42:05.390717+0000","last_unstale":"2024-04-26T01:45:54.961962+0000","last_undegraded":"2024-04-26T01:45:54.961962+0000","last_fullsized":"2024-04-26T01:45:54.961962+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:07.541593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"96'6","reported_seq":253,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:59.965227+0000","last_change":"2024-04-26T01:42:02.232823+0000","last_active":"2024-04-26T01:45:59.965227+0000","last_peered":"2024-04-26T01:45:59.965227+0000","last_clean":"2024-04-26T01:45:59.965227+0000","last_became_active":"2024-04-26T01:42:02.231899+0000","last_became_peered":"2024-04-26T01:42:02.231899+0000","last_unstale":"2024-04-26T01:45:59.965227+0000","last_undegraded":"2024-04-26T01:45:59.965227+0000","last_fullsized":"2024-04-26T01:45:59.965227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:23.287305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":214,"num_read_kb":214,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949845+0000","last_change":"2024-04-26T01:42:05.405690+0000","last_active":"2024-04-26T01:45:54.949845+0000","last_peered":"2024-04-26T01:45:54.949845+0000","last_clean":"2024-04-26T01:45:54.949845+0000","last_became_active":"2024-04-26T01:42:05.405345+0000","last_became_peered":"2024-04-26T01:42:05.405345+0000","last_unstale":"2024-04-26T01:45:54.949845+0000","last_undegraded":"2024-04-26T01:45:54.949845+0000","last_fullsized":"2024-04-26T01:45:54.949845+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:48.377916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.947235+0000","last_change":"2024-04-26T01:42:01.335721+0000","last_active":"2024-04-26T01:45:54.947235+0000","last_peered":"2024-04-26T01:45:54.947235+0000","last_clean":"2024-04-26T01:45:54.947235+0000","last_became_active":"2024-04-26T01:42:01.335149+0000","last_became_peered":"2024-04-26T01:42:01.335149+0000","last_unstale":"2024-04-26T01:45:54.947235+0000","last_undegraded":"2024-04-26T01:45:54.947235+0000","last_fullsized":"2024-04-26T01:45:54.947235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:36.372033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"96'148","reported_seq":413,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:57.098863+0000","last_change":"2024-04-26T01:40:28.086333+0000","last_active":"2024-04-26T01:45:57.098863+0000","last_peered":"2024-04-26T01:45:57.098863+0000","last_clean":"2024-04-26T01:45:57.098863+0000","last_became_active":"2024-04-26T01:40:28.085902+0000","last_became_peered":"2024-04-26T01:40:28.085902+0000","last_unstale":"2024-04-26T01:45:57.098863+0000","last_undegraded":"2024-04-26T01:45:57.098863+0000","last_fullsized":"2024-04-26T01:45:57.098863+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_clean_scrub_stamp":"2024-04-26T01:36:35.357698+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:43.290117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944108+0000","last_change":"2024-04-26T01:42:05.401896+0000","last_active":"2024-04-26T01:45:54.944108+0000","last_peered":"2024-04-26T01:45:54.944108+0000","last_clean":"2024-04-26T01:45:54.944108+0000","last_became_active":"2024-04-26T01:42:05.401496+0000","last_became_peered":"2024-04-26T01:42:05.401496+0000","last_unstale":"2024-04-26T01:45:54.944108+0000","last_undegraded":"2024-04-26T01:45:54.944108+0000","last_fullsized":"2024-04-26T01:45:54.944108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:49.758444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961993+0000","last_change":"2024-04-26T01:42:01.329705+0000","last_active":"2024-04-26T01:45:54.961993+0000","last_peered":"2024-04-26T01:45:54.961993+0000","last_clean":"2024-04-26T01:45:54.961993+0000","last_became_active":"2024-04-26T01:42:01.329153+0000","last_became_peered":"2024-04-26T01:42:01.329153+0000","last_unstale":"2024-04-26T01:45:54.961993+0000","last_undegraded":"2024-04-26T01:45:54.961993+0000","last_fullsized":"2024-04-26T01:45:54.961993+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:06.747378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962915+0000","last_change":"2024-04-26T01:42:02.232148+0000","last_active":"2024-04-26T01:45:54.962915+0000","last_peered":"2024-04-26T01:45:54.962915+0000","last_clean":"2024-04-26T01:45:54.962915+0000","last_became_active":"2024-04-26T01:42:02.231739+0000","last_became_peered":"2024-04-26T01:42:02.231739+0000","last_unstale":"2024-04-26T01:45:54.962915+0000","last_undegraded":"2024-04-26T01:45:54.962915+0000","last_fullsized":"2024-04-26T01:45:54.962915+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:09.752433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.945042+0000","last_change":"2024-04-26T01:42:05.399938+0000","last_active":"2024-04-26T01:45:54.945042+0000","last_peered":"2024-04-26T01:45:54.945042+0000","last_clean":"2024-04-26T01:45:54.945042+0000","last_became_active":"2024-04-26T01:42:05.399550+0000","last_became_peered":"2024-04-26T01:42:05.399550+0000","last_unstale":"2024-04-26T01:45:54.945042+0000","last_undegraded":"2024-04-26T01:45:54.945042+0000","last_fullsized":"2024-04-26T01:45:54.945042+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:44.142284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"96'126","reported_seq":284,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:46:00.780819+0000","last_change":"2024-04-26T01:42:01.337437+0000","last_active":"2024-04-26T01:46:00.780819+0000","last_peered":"2024-04-26T01:46:00.780819+0000","last_clean":"2024-04-26T01:46:00.780819+0000","last_became_active":"2024-04-26T01:42:01.336894+0000","last_became_peered":"2024-04-26T01:42:01.336894+0000","last_unstale":"2024-04-26T01:46:00.780819+0000","last_undegraded":"2024-04-26T01:46:00.780819+0000","last_fullsized":"2024-04-26T01:46:00.780819+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:04:31.093273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":422,"num_read_kb":376,"num_write":233,"num_write_kb":1415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.951785+0000","last_change":"2024-04-26T01:42:05.411694+0000","last_active":"2024-04-26T01:45:54.951785+0000","last_peered":"2024-04-26T01:45:54.951785+0000","last_clean":"2024-04-26T01:45:54.951785+0000","last_became_active":"2024-04-26T01:42:05.411300+0000","last_became_peered":"2024-04-26T01:42:05.411300+0000","last_unstale":"2024-04-26T01:45:54.951785+0000","last_undegraded":"2024-04-26T01:45:54.951785+0000","last_fullsized":"2024-04-26T01:45:54.951785+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:42:08.116145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.947485+0000","last_change":"2024-04-26T01:42:01.427501+0000","last_active":"2024-04-26T01:45:54.947485+0000","last_peered":"2024-04-26T01:45:54.947485+0000","last_clean":"2024-04-26T01:45:54.947485+0000","last_became_active":"2024-04-26T01:42:01.426970+0000","last_became_peered":"2024-04-26T01:42:01.426970+0000","last_unstale":"2024-04-26T01:45:54.947485+0000","last_undegraded":"2024-04-26T01:45:54.947485+0000","last_fullsized":"2024-04-26T01:45:54.947485+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:53:53.118296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.184575+0000","last_change":"2024-04-26T01:42:05.414790+0000","last_active":"2024-04-26T01:45:55.184575+0000","last_peered":"2024-04-26T01:45:55.184575+0000","last_clean":"2024-04-26T01:45:55.184575+0000","last_became_active":"2024-04-26T01:42:05.411947+0000","last_became_peered":"2024-04-26T01:42:05.411947+0000","last_unstale":"2024-04-26T01:45:55.184575+0000","last_undegraded":"2024-04-26T01:45:55.184575+0000","last_fullsized":"2024-04-26T01:45:55.184575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:11:34.450815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961147+0000","last_change":"2024-04-26T01:42:02.230539+0000","last_active":"2024-04-26T01:45:54.961147+0000","last_peered":"2024-04-26T01:45:54.961147+0000","last_clean":"2024-04-26T01:45:54.961147+0000","last_became_active":"2024-04-26T01:42:02.230150+0000","last_became_peered":"2024-04-26T01:42:02.230150+0000","last_unstale":"2024-04-26T01:45:54.961147+0000","last_undegraded":"2024-04-26T01:45:54.961147+0000","last_fullsized":"2024-04-26T01:45:54.961147+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:08:03.073069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"95'2","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961652+0000","last_change":"2024-04-26T01:42:05.388877+0000","last_active":"2024-04-26T01:45:54.961652+0000","last_peered":"2024-04-26T01:45:54.961652+0000","last_clean":"2024-04-26T01:45:54.961652+0000","last_became_active":"2024-04-26T01:42:05.388457+0000","last_became_peered":"2024-04-26T01:42:05.388457+0000","last_unstale":"2024-04-26T01:45:54.961652+0000","last_undegraded":"2024-04-26T01:45:54.961652+0000","last_fullsized":"2024-04-26T01:45:54.961652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:54.436859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949462+0000","last_change":"2024-04-26T01:42:01.343461+0000","last_active":"2024-04-26T01:45:54.949462+0000","last_peered":"2024-04-26T01:45:54.949462+0000","last_clean":"2024-04-26T01:45:54.949462+0000","last_became_active":"2024-04-26T01:42:01.342713+0000","last_became_peered":"2024-04-26T01:42:01.342713+0000","last_unstale":"2024-04-26T01:45:54.949462+0000","last_undegraded":"2024-04-26T01:45:54.949462+0000","last_fullsized":"2024-04-26T01:45:54.949462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:38.471858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962158+0000","last_change":"2024-04-26T01:42:05.415101+0000","last_active":"2024-04-26T01:45:54.962158+0000","last_peered":"2024-04-26T01:45:54.962158+0000","last_clean":"2024-04-26T01:45:54.962158+0000","last_became_active":"2024-04-26T01:42:05.414745+0000","last_became_peered":"2024-04-26T01:42:05.414745+0000","last_unstale":"2024-04-26T01:45:54.962158+0000","last_undegraded":"2024-04-26T01:45:54.962158+0000","last_fullsized":"2024-04-26T01:45:54.962158+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:13:08.573547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"96'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931967+0000","last_change":"2024-04-26T01:42:01.339501+0000","last_active":"2024-04-26T01:45:54.931967+0000","last_peered":"2024-04-26T01:45:54.931967+0000","last_clean":"2024-04-26T01:45:54.931967+0000","last_became_active":"2024-04-26T01:42:01.339132+0000","last_became_peered":"2024-04-26T01:42:01.339132+0000","last_unstale":"2024-04-26T01:45:54.931967+0000","last_undegraded":"2024-04-26T01:45:54.931967+0000","last_fullsized":"2024-04-26T01:45:54.931967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:16.463099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949949+0000","last_change":"2024-04-26T01:42:05.417864+0000","last_active":"2024-04-26T01:45:54.949949+0000","last_peered":"2024-04-26T01:45:54.949949+0000","last_clean":"2024-04-26T01:45:54.949949+0000","last_became_active":"2024-04-26T01:42:05.417414+0000","last_became_peered":"2024-04-26T01:42:05.417414+0000","last_unstale":"2024-04-26T01:45:54.949949+0000","last_undegraded":"2024-04-26T01:45:54.949949+0000","last_fullsized":"2024-04-26T01:45:54.949949+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:23:12.793421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":50,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962222+0000","last_change":"2024-04-26T01:42:49.063769+0000","last_active":"2024-04-26T01:45:54.962222+0000","last_peered":"2024-04-26T01:45:54.962222+0000","last_clean":"2024-04-26T01:45:54.962222+0000","last_became_active":"2024-04-26T01:42:49.057463+0000","last_became_peered":"2024-04-26T01:42:49.057463+0000","last_unstale":"2024-04-26T01:45:54.962222+0000","last_undegraded":"2024-04-26T01:45:54.962222+0000","last_fullsized":"2024-04-26T01:45:54.962222+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:34:54.511304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962310+0000","last_change":"2024-04-26T01:42:05.397980+0000","last_active":"2024-04-26T01:45:54.962310+0000","last_peered":"2024-04-26T01:45:54.962310+0000","last_clean":"2024-04-26T01:45:54.962310+0000","last_became_active":"2024-04-26T01:42:05.397589+0000","last_became_peered":"2024-04-26T01:42:05.397589+0000","last_unstale":"2024-04-26T01:45:54.962310+0000","last_undegraded":"2024-04-26T01:45:54.962310+0000","last_fullsized":"2024-04-26T01:45:54.962310+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:39:55.863566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931803+0000","last_change":"2024-04-26T01:42:02.234313+0000","last_active":"2024-04-26T01:45:54.931803+0000","last_peered":"2024-04-26T01:45:54.931803+0000","last_clean":"2024-04-26T01:45:54.931803+0000","last_became_active":"2024-04-26T01:42:02.233873+0000","last_became_peered":"2024-04-26T01:42:02.233873+0000","last_unstale":"2024-04-26T01:45:54.931803+0000","last_undegraded":"2024-04-26T01:45:54.931803+0000","last_fullsized":"2024-04-26T01:45:54.931803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:34.092434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962665+0000","last_change":"2024-04-26T01:42:05.387435+0000","last_active":"2024-04-26T01:45:54.962665+0000","last_peered":"2024-04-26T01:45:54.962665+0000","last_clean":"2024-04-26T01:45:54.962665+0000","last_became_active":"2024-04-26T01:42:05.386964+0000","last_became_peered":"2024-04-26T01:42:05.386964+0000","last_unstale":"2024-04-26T01:45:54.962665+0000","last_undegraded":"2024-04-26T01:45:54.962665+0000","last_fullsized":"2024-04-26T01:45:54.962665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:12:27.905842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"96'73","reported_seq":148,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:46:00.772676+0000","last_change":"2024-04-26T01:42:01.326432+0000","last_active":"2024-04-26T01:46:00.772676+0000","last_peered":"2024-04-26T01:46:00.772676+0000","last_clean":"2024-04-26T01:46:00.772676+0000","last_became_active":"2024-04-26T01:42:01.325989+0000","last_became_peered":"2024-04-26T01:42:01.325989+0000","last_unstale":"2024-04-26T01:46:00.772676+0000","last_undegraded":"2024-04-26T01:46:00.772676+0000","last_fullsized":"2024-04-26T01:46:00.772676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:10.049563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":44,"num_write":84,"num_write_kb":1075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961654+0000","last_change":"2024-04-26T01:42:05.412802+0000","last_active":"2024-04-26T01:45:54.961654+0000","last_peered":"2024-04-26T01:45:54.961654+0000","last_clean":"2024-04-26T01:45:54.961654+0000","last_became_active":"2024-04-26T01:42:05.412443+0000","last_became_peered":"2024-04-26T01:42:05.412443+0000","last_unstale":"2024-04-26T01:45:54.961654+0000","last_undegraded":"2024-04-26T01:45:54.961654+0000","last_fullsized":"2024-04-26T01:45:54.961654+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:28:56.360778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'2","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944056+0000","last_change":"2024-04-26T01:42:49.080305+0000","last_active":"2024-04-26T01:45:54.944056+0000","last_peered":"2024-04-26T01:45:54.944056+0000","last_clean":"2024-04-26T01:45:54.944056+0000","last_became_active":"2024-04-26T01:42:49.074476+0000","last_became_peered":"2024-04-26T01:42:49.074476+0000","last_unstale":"2024-04-26T01:45:54.944056+0000","last_undegraded":"2024-04-26T01:45:54.944056+0000","last_fullsized":"2024-04-26T01:45:54.944056+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:58:06.619961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962655+0000","last_change":"2024-04-26T01:42:05.387621+0000","last_active":"2024-04-26T01:45:54.962655+0000","last_peered":"2024-04-26T01:45:54.962655+0000","last_clean":"2024-04-26T01:45:54.962655+0000","last_became_active":"2024-04-26T01:42:05.387165+0000","last_became_peered":"2024-04-26T01:42:05.387165+0000","last_unstale":"2024-04-26T01:45:54.962655+0000","last_undegraded":"2024-04-26T01:45:54.962655+0000","last_fullsized":"2024-04-26T01:45:54.962655+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:37.803330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962291+0000","last_change":"2024-04-26T01:42:05.386787+0000","last_active":"2024-04-26T01:45:54.962291+0000","last_peered":"2024-04-26T01:45:54.962291+0000","last_clean":"2024-04-26T01:45:54.962291+0000","last_became_active":"2024-04-26T01:42:05.386410+0000","last_became_peered":"2024-04-26T01:42:05.386410+0000","last_unstale":"2024-04-26T01:45:54.962291+0000","last_undegraded":"2024-04-26T01:45:54.962291+0000","last_fullsized":"2024-04-26T01:45:54.962291+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:42.137191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"95'12","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931481+0000","last_change":"2024-04-26T01:42:05.392142+0000","last_active":"2024-04-26T01:45:54.931481+0000","last_peered":"2024-04-26T01:45:54.931481+0000","last_clean":"2024-04-26T01:45:54.931481+0000","last_became_active":"2024-04-26T01:42:05.391829+0000","last_became_peered":"2024-04-26T01:42:05.391829+0000","last_unstale":"2024-04-26T01:45:54.931481+0000","last_undegraded":"2024-04-26T01:45:54.931481+0000","last_fullsized":"2024-04-26T01:45:54.931481+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:00:25.934641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931859+0000","last_change":"2024-04-26T01:42:05.409184+0000","last_active":"2024-04-26T01:45:54.931859+0000","last_peered":"2024-04-26T01:45:54.931859+0000","last_clean":"2024-04-26T01:45:54.931859+0000","last_became_active":"2024-04-26T01:42:05.408894+0000","last_became_peered":"2024-04-26T01:42:05.408894+0000","last_unstale":"2024-04-26T01:45:54.931859+0000","last_undegraded":"2024-04-26T01:45:54.931859+0000","last_fullsized":"2024-04-26T01:45:54.931859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:53.291844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.183917+0000","last_change":"2024-04-26T01:42:05.406940+0000","last_active":"2024-04-26T01:45:55.183917+0000","last_peered":"2024-04-26T01:45:55.183917+0000","last_clean":"2024-04-26T01:45:55.183917+0000","last_became_active":"2024-04-26T01:42:05.406476+0000","last_became_peered":"2024-04-26T01:42:05.406476+0000","last_unstale":"2024-04-26T01:45:55.183917+0000","last_undegraded":"2024-04-26T01:45:55.183917+0000","last_fullsized":"2024-04-26T01:45:55.183917+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:21:37.734633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"95'4","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.945671+0000","last_change":"2024-04-26T01:42:05.413521+0000","last_active":"2024-04-26T01:45:54.945671+0000","last_peered":"2024-04-26T01:45:54.945671+0000","last_clean":"2024-04-26T01:45:54.945671+0000","last_became_active":"2024-04-26T01:42:05.412789+0000","last_became_peered":"2024-04-26T01:42:05.412789+0000","last_unstale":"2024-04-26T01:45:54.945671+0000","last_undegraded":"2024-04-26T01:45:54.945671+0000","last_fullsized":"2024-04-26T01:45:54.945671+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:13.471376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962356+0000","last_change":"2024-04-26T01:42:05.391395+0000","last_active":"2024-04-26T01:45:54.962356+0000","last_peered":"2024-04-26T01:45:54.962356+0000","last_clean":"2024-04-26T01:45:54.962356+0000","last_became_active":"2024-04-26T01:42:05.391048+0000","last_became_peered":"2024-04-26T01:42:05.391048+0000","last_unstale":"2024-04-26T01:45:54.962356+0000","last_undegraded":"2024-04-26T01:45:54.962356+0000","last_fullsized":"2024-04-26T01:45:54.962356+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:45:56.645454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"95'15","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931782+0000","last_change":"2024-04-26T01:42:05.386868+0000","last_active":"2024-04-26T01:45:54.931782+0000","last_peered":"2024-04-26T01:45:54.931782+0000","last_clean":"2024-04-26T01:45:54.931782+0000","last_became_active":"2024-04-26T01:42:05.386483+0000","last_became_peered":"2024-04-26T01:42:05.386483+0000","last_unstale":"2024-04-26T01:45:54.931782+0000","last_undegraded":"2024-04-26T01:45:54.931782+0000","last_fullsized":"2024-04-26T01:45:54.931782+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:42:44.453076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"95'2","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.951689+0000","last_change":"2024-04-26T01:42:05.406711+0000","last_active":"2024-04-26T01:45:54.951689+0000","last_peered":"2024-04-26T01:45:54.951689+0000","last_clean":"2024-04-26T01:45:54.951689+0000","last_became_active":"2024-04-26T01:42:05.406318+0000","last_became_peered":"2024-04-26T01:42:05.406318+0000","last_unstale":"2024-04-26T01:45:54.951689+0000","last_undegraded":"2024-04-26T01:45:54.951689+0000","last_fullsized":"2024-04-26T01:45:54.951689+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:36:16.599678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961752+0000","last_change":"2024-04-26T01:42:05.406305+0000","last_active":"2024-04-26T01:45:54.961752+0000","last_peered":"2024-04-26T01:45:54.961752+0000","last_clean":"2024-04-26T01:45:54.961752+0000","last_became_active":"2024-04-26T01:42:05.405953+0000","last_became_peered":"2024-04-26T01:42:05.405953+0000","last_unstale":"2024-04-26T01:45:54.961752+0000","last_undegraded":"2024-04-26T01:45:54.961752+0000","last_fullsized":"2024-04-26T01:45:54.961752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:36.588672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949192+0000","last_change":"2024-04-26T01:42:05.413757+0000","last_active":"2024-04-26T01:45:54.949192+0000","last_peered":"2024-04-26T01:45:54.949192+0000","last_clean":"2024-04-26T01:45:54.949192+0000","last_became_active":"2024-04-26T01:42:05.413410+0000","last_became_peered":"2024-04-26T01:42:05.413410+0000","last_unstale":"2024-04-26T01:45:54.949192+0000","last_undegraded":"2024-04-26T01:45:54.949192+0000","last_fullsized":"2024-04-26T01:45:54.949192+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:30.298443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"95'10","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961329+0000","last_change":"2024-04-26T01:42:05.412268+0000","last_active":"2024-04-26T01:45:54.961329+0000","last_peered":"2024-04-26T01:45:54.961329+0000","last_clean":"2024-04-26T01:45:54.961329+0000","last_became_active":"2024-04-26T01:42:05.411804+0000","last_became_peered":"2024-04-26T01:42:05.411804+0000","last_unstale":"2024-04-26T01:45:54.961329+0000","last_undegraded":"2024-04-26T01:45:54.961329+0000","last_fullsized":"2024-04-26T01:45:54.961329+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:47.277766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"95'11","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944497+0000","last_change":"2024-04-26T01:42:05.415563+0000","last_active":"2024-04-26T01:45:54.944497+0000","last_peered":"2024-04-26T01:45:54.944497+0000","last_clean":"2024-04-26T01:45:54.944497+0000","last_became_active":"2024-04-26T01:42:05.414818+0000","last_became_peered":"2024-04-26T01:42:05.414818+0000","last_unstale":"2024-04-26T01:45:54.944497+0000","last_undegraded":"2024-04-26T01:45:54.944497+0000","last_fullsized":"2024-04-26T01:45:54.944497+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:54.713595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962135+0000","last_change":"2024-04-26T01:42:05.392280+0000","last_active":"2024-04-26T01:45:54.962135+0000","last_peered":"2024-04-26T01:45:54.962135+0000","last_clean":"2024-04-26T01:45:54.962135+0000","last_became_active":"2024-04-26T01:42:05.391943+0000","last_became_peered":"2024-04-26T01:42:05.391943+0000","last_unstale":"2024-04-26T01:45:54.962135+0000","last_undegraded":"2024-04-26T01:45:54.962135+0000","last_fullsized":"2024-04-26T01:45:54.962135+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:38:51.560596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":799967,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1789,"num_read_kb":1165,"num_write":485,"num_write_kb":2627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2629632,"data_stored":2399901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285632,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1077248,"data_stored":519573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547268,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41676,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703476,"statfs":{"total":95995035648,"available":95952359424,"internally_reserved":0,"allocated":2109440,"data_stored":1512850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710795,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28464,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716688,"statfs":{"total":95995035648,"available":95965888512,"internally_reserved":0,"allocated":1478656,"data_stored":912913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972435,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36772,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708380,"statfs":{"total":95995035648,"available":95957381120,"internally_reserved":0,"allocated":1470464,"data_stored":912721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266774,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1679360,"data_stored":1113312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430299,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":519086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40616,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704536,"statfs":{"total":95995035648,"available":95953444864,"internally_reserved":0,"allocated":1146880,"data_stored":591241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738493,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":520357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411443,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28164,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716988,"statfs":{"total":95995035648,"available":95966195712,"internally_reserved":0,"allocated":1171456,"data_stored":592386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28908,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716244,"statfs":{"total":95995035648,"available":95965433856,"internally_reserved":0,"allocated":1814528,"data_stored":1240845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888167,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":520837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691938,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37432,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707720,"statfs":{"total":95995035648,"available":95956705280,"internally_reserved":0,"allocated":2023424,"data_stored":1440305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":72362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":74945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":721732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":330941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":393842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":403486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:46:05.445 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T01:46:05.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:46:05.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:05 smithi006 ceph-mon[28419]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:46:05.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:05 smithi188 ceph-mon[32751]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:46:05.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:05 smithi029 ceph-mon[32775]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:46:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:06.415 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:06 smithi006 ceph-mon[28419]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 131 KiB/s wr, 78 op/s 2024-04-26T01:46:06.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:06 smithi188 ceph-mon[32751]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 131 KiB/s wr, 78 op/s 2024-04-26T01:46:06.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:06 smithi029 ceph-mon[32775]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 131 KiB/s wr, 78 op/s 2024-04-26T01:46:08.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.189+0000 7f104a400700 1 -- 172.21.15.6:0/1574422261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1044074c70 msgr2=0x7f1044075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.189+0000 7f104a400700 1 --2- 172.21.15.6:0/1574422261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1044074c70 0x7f1044075050 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f103400b0a0 tx=0x7f103400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.189+0000 7f104a400700 1 -- 172.21.15.6:0/1574422261 shutdown_connections 2024-04-26T01:46:08.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.189+0000 7f104a400700 1 --2- 172.21.15.6:0/1574422261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f104410f440 0x7f1044111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.189+0000 7f104a400700 1 --2- 172.21.15.6:0/1574422261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1044074c70 0x7f1044075050 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.189+0000 7f104a400700 1 --2- 172.21.15.6:0/1574422261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1044075590 0x7f104410ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.189+0000 7f104a400700 1 -- 172.21.15.6:0/1574422261 >> 172.21.15.6:0/1574422261 conn(0x7f1044079ba0 msgr2=0x7f1044079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:08.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.189+0000 7f104a400700 1 -- 172.21.15.6:0/1574422261 shutdown_connections 2024-04-26T01:46:08.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.189+0000 7f104a400700 1 -- 172.21.15.6:0/1574422261 wait complete. 2024-04-26T01:46:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.190+0000 7f104a400700 1 Processor -- start 2024-04-26T01:46:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.191+0000 7f104a400700 1 -- start start 2024-04-26T01:46:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.191+0000 7f104a400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1044074c70 0x7f104419b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.191+0000 7f104a400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1044075590 0x7f104419b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.191+0000 7f104a400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f104410f440 0x7f104419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.191+0000 7f104a400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1044114e60 con 0x7f1044074c70 2024-04-26T01:46:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.191+0000 7f104a400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1044114ce0 con 0x7f104410f440 2024-04-26T01:46:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.191+0000 7f104a400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1044114fe0 con 0x7f1044075590 2024-04-26T01:46:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.191+0000 7f1043400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1044074c70 0x7f104419b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.191+0000 7f1043400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1044074c70 0x7f104419b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44254/0 (socket says 172.21.15.6:44254) 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.191+0000 7f1043400700 1 -- 172.21.15.6:0/607151726 learned_addr learned my addr 172.21.15.6:0/607151726 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.191+0000 7f103aa00700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1044075590 0x7f104419b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.192+0000 7f1043e00700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f104410f440 0x7f104419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.192+0000 7f1043400700 1 -- 172.21.15.6:0/607151726 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1044075590 msgr2=0x7f104419b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.192+0000 7f1043400700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1044075590 0x7f104419b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.192+0000 7f1043400700 1 -- 172.21.15.6:0/607151726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f104410f440 msgr2=0x7f104419fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.192+0000 7f1043400700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f104410f440 0x7f104419fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.192+0000 7f1043400700 1 -- 172.21.15.6:0/607151726 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10441a0370 con 0x7f1044074c70 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.192+0000 7f103aa00700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1044075590 0x7f104419b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.192+0000 7f1043400700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1044074c70 0x7f104419b3b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f103400d010 tx=0x7f1034004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.193+0000 7f1040c00700 1 -- 172.21.15.6:0/607151726 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1034021070 con 0x7f1044074c70 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.193+0000 7f1040c00700 1 -- 172.21.15.6:0/607151726 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1034007e30 con 0x7f1044074c70 2024-04-26T01:46:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.193+0000 7f104a400700 1 -- 172.21.15.6:0/607151726 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10441a0600 con 0x7f1044074c70 2024-04-26T01:46:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.193+0000 7f1040c00700 1 -- 172.21.15.6:0/607151726 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1034004640 con 0x7f1044074c70 2024-04-26T01:46:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.193+0000 7f104a400700 1 -- 172.21.15.6:0/607151726 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10441a7ee0 con 0x7f1044074c70 2024-04-26T01:46:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.194+0000 7f104a400700 1 -- 172.21.15.6:0/607151726 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1044069730 con 0x7f1044074c70 2024-04-26T01:46:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.197+0000 7f1040c00700 1 -- 172.21.15.6:0/607151726 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f1034007980 con 0x7f1044074c70 2024-04-26T01:46:08.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.199+0000 7f1040c00700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f10280a3f50 0x7f10280a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.199+0000 7f1040c00700 1 -- 172.21.15.6:0/607151726 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f103400f020 con 0x7f1044074c70 2024-04-26T01:46:08.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.199+0000 7f103aa00700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f10280a3f50 0x7f10280a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.200+0000 7f103aa00700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f10280a3f50 0x7f10280a6410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f10441110d0 tx=0x7f102c009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:08.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.202+0000 7f1040c00700 1 -- 172.21.15.6:0/607151726 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f103402a020 con 0x7f1044074c70 2024-04-26T01:46:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.431+0000 7f104a400700 1 -- 172.21.15.6:0/607151726 --> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1044101600 con 0x7f10280a3f50 2024-04-26T01:46:08.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.454+0000 7f1040c00700 1 -- 172.21.15.6:0/607151726 <== mgr.25037 v2:172.21.15.6:6800/3353570381 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257778 (secure 0 0 0) 0x7f1044101600 con 0x7f10280a3f50 2024-04-26T01:46:08.457 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:46:08.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.459+0000 7f104a400700 1 -- 172.21.15.6:0/607151726 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f10280a3f50 msgr2=0x7f10280a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.459+0000 7f104a400700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f10280a3f50 0x7f10280a6410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f10441110d0 tx=0x7f102c009260 comp rx=0 tx=0).stop 2024-04-26T01:46:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.459+0000 7f104a400700 1 -- 172.21.15.6:0/607151726 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1044074c70 msgr2=0x7f104419b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.459+0000 7f104a400700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1044074c70 0x7f104419b3b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f103400d010 tx=0x7f1034004d00 comp rx=0 tx=0).stop 2024-04-26T01:46:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.460+0000 7f104a400700 1 -- 172.21.15.6:0/607151726 shutdown_connections 2024-04-26T01:46:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.460+0000 7f104a400700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f10280a3f50 0x7f10280a6410 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.460+0000 7f104a400700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f104410f440 0x7f104419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.460+0000 7f104a400700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1044074c70 0x7f104419b3b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.460+0000 7f104a400700 1 --2- 172.21.15.6:0/607151726 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1044075590 0x7f104419b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.460+0000 7f104a400700 1 -- 172.21.15.6:0/607151726 >> 172.21.15.6:0/607151726 conn(0x7f1044079ba0 msgr2=0x7f1044100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.460+0000 7f104a400700 1 -- 172.21.15.6:0/607151726 shutdown_connections 2024-04-26T01:46:08.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:08.460+0000 7f104a400700 1 -- 172.21.15.6:0/607151726 wait complete. 2024-04-26T01:46:08.460 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-26T01:46:08.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:08 smithi006 ceph-mon[28419]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 108 KiB/s wr, 66 op/s 2024-04-26T01:46:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:08 smithi188 ceph-mon[32751]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 108 KiB/s wr, 66 op/s 2024-04-26T01:46:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:08 smithi029 ceph-mon[32775]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 108 KiB/s wr, 66 op/s 2024-04-26T01:46:09.294 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-26T01:46:06.959295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1390653,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":2054,"num_read_kb":1509,"num_write":689,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1002,"ondisk_log_size":1002,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":395460,"kb_used_data":16812,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124546364,"statfs":{"total":1151940427776,"available":1151535476736,"internally_reserved":0,"allocated":17215488,"data_stored":10398507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":198545,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":528,"num_read_kb":577,"num_write":203,"num_write_kb":1181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.981646"},"pg_stats":[{"pgid":"3.3f","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962627+0000","last_change":"2024-04-26T01:42:05.417304+0000","last_active":"2024-04-26T01:45:54.962627+0000","last_peered":"2024-04-26T01:45:54.962627+0000","last_clean":"2024-04-26T01:45:54.962627+0000","last_became_active":"2024-04-26T01:42:05.416999+0000","last_became_peered":"2024-04-26T01:42:05.416999+0000","last_unstale":"2024-04-26T01:45:54.962627+0000","last_undegraded":"2024-04-26T01:45:54.962627+0000","last_fullsized":"2024-04-26T01:45:54.962627+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:33:30.958943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"95'4","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.184448+0000","last_change":"2024-04-26T01:42:05.413089+0000","last_active":"2024-04-26T01:45:55.184448+0000","last_peered":"2024-04-26T01:45:55.184448+0000","last_clean":"2024-04-26T01:45:55.184448+0000","last_became_active":"2024-04-26T01:42:05.412776+0000","last_became_peered":"2024-04-26T01:42:05.412776+0000","last_unstale":"2024-04-26T01:45:55.184448+0000","last_undegraded":"2024-04-26T01:45:55.184448+0000","last_fullsized":"2024-04-26T01:45:55.184448+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:47.861340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.184102+0000","last_change":"2024-04-26T01:42:05.404864+0000","last_active":"2024-04-26T01:45:55.184102+0000","last_peered":"2024-04-26T01:45:55.184102+0000","last_clean":"2024-04-26T01:45:55.184102+0000","last_became_active":"2024-04-26T01:42:05.404529+0000","last_became_peered":"2024-04-26T01:42:05.404529+0000","last_unstale":"2024-04-26T01:45:55.184102+0000","last_undegraded":"2024-04-26T01:45:55.184102+0000","last_fullsized":"2024-04-26T01:45:55.184102+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:04:18.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.947186+0000","last_change":"2024-04-26T01:42:05.393582+0000","last_active":"2024-04-26T01:45:54.947186+0000","last_peered":"2024-04-26T01:45:54.947186+0000","last_clean":"2024-04-26T01:45:54.947186+0000","last_became_active":"2024-04-26T01:42:05.393193+0000","last_became_peered":"2024-04-26T01:42:05.393193+0000","last_unstale":"2024-04-26T01:45:54.947186+0000","last_undegraded":"2024-04-26T01:45:54.947186+0000","last_fullsized":"2024-04-26T01:45:54.947186+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:20:29.420459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"95'4","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962087+0000","last_change":"2024-04-26T01:42:05.412979+0000","last_active":"2024-04-26T01:45:54.962087+0000","last_peered":"2024-04-26T01:45:54.962087+0000","last_clean":"2024-04-26T01:45:54.962087+0000","last_became_active":"2024-04-26T01:42:05.412679+0000","last_became_peered":"2024-04-26T01:42:05.412679+0000","last_unstale":"2024-04-26T01:45:54.962087+0000","last_undegraded":"2024-04-26T01:45:54.962087+0000","last_fullsized":"2024-04-26T01:45:54.962087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:22.774960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"95'6","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931536+0000","last_change":"2024-04-26T01:42:49.062507+0000","last_active":"2024-04-26T01:45:54.931536+0000","last_peered":"2024-04-26T01:45:54.931536+0000","last_clean":"2024-04-26T01:45:54.931536+0000","last_became_active":"2024-04-26T01:42:49.062095+0000","last_became_peered":"2024-04-26T01:42:49.062095+0000","last_unstale":"2024-04-26T01:45:54.931536+0000","last_undegraded":"2024-04-26T01:45:54.931536+0000","last_fullsized":"2024-04-26T01:45:54.931536+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:47.846697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962044+0000","last_change":"2024-04-26T01:42:05.385221+0000","last_active":"2024-04-26T01:45:54.962044+0000","last_peered":"2024-04-26T01:45:54.962044+0000","last_clean":"2024-04-26T01:45:54.962044+0000","last_became_active":"2024-04-26T01:42:05.384856+0000","last_became_peered":"2024-04-26T01:42:05.384856+0000","last_unstale":"2024-04-26T01:45:54.962044+0000","last_undegraded":"2024-04-26T01:45:54.962044+0000","last_fullsized":"2024-04-26T01:45:54.962044+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:07:23.461324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949107+0000","last_change":"2024-04-26T01:42:05.404754+0000","last_active":"2024-04-26T01:45:54.949107+0000","last_peered":"2024-04-26T01:45:54.949107+0000","last_clean":"2024-04-26T01:45:54.949107+0000","last_became_active":"2024-04-26T01:42:05.404220+0000","last_became_peered":"2024-04-26T01:42:05.404220+0000","last_unstale":"2024-04-26T01:45:54.949107+0000","last_undegraded":"2024-04-26T01:45:54.949107+0000","last_fullsized":"2024-04-26T01:45:54.949107+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:50.243552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"95'11","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961852+0000","last_change":"2024-04-26T01:42:05.390972+0000","last_active":"2024-04-26T01:45:54.961852+0000","last_peered":"2024-04-26T01:45:54.961852+0000","last_clean":"2024-04-26T01:45:54.961852+0000","last_became_active":"2024-04-26T01:42:05.390599+0000","last_became_peered":"2024-04-26T01:42:05.390599+0000","last_unstale":"2024-04-26T01:45:54.961852+0000","last_undegraded":"2024-04-26T01:45:54.961852+0000","last_fullsized":"2024-04-26T01:45:54.961852+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:19:46.328640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"95'16","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932040+0000","last_change":"2024-04-26T01:42:05.392320+0000","last_active":"2024-04-26T01:45:54.932040+0000","last_peered":"2024-04-26T01:45:54.932040+0000","last_clean":"2024-04-26T01:45:54.932040+0000","last_became_active":"2024-04-26T01:42:05.391986+0000","last_became_peered":"2024-04-26T01:42:05.391986+0000","last_unstale":"2024-04-26T01:45:54.932040+0000","last_undegraded":"2024-04-26T01:45:54.932040+0000","last_fullsized":"2024-04-26T01:45:54.932040+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:17:27.302358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962508+0000","last_change":"2024-04-26T01:42:05.388077+0000","last_active":"2024-04-26T01:45:54.962508+0000","last_peered":"2024-04-26T01:45:54.962508+0000","last_clean":"2024-04-26T01:45:54.962508+0000","last_became_active":"2024-04-26T01:42:05.387545+0000","last_became_peered":"2024-04-26T01:42:05.387545+0000","last_unstale":"2024-04-26T01:45:54.962508+0000","last_undegraded":"2024-04-26T01:45:54.962508+0000","last_fullsized":"2024-04-26T01:45:54.962508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:51.288177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"95'7","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932538+0000","last_change":"2024-04-26T01:42:49.067449+0000","last_active":"2024-04-26T01:45:54.932538+0000","last_peered":"2024-04-26T01:45:54.932538+0000","last_clean":"2024-04-26T01:45:54.932538+0000","last_became_active":"2024-04-26T01:42:49.065991+0000","last_became_peered":"2024-04-26T01:42:49.065991+0000","last_unstale":"2024-04-26T01:45:54.932538+0000","last_undegraded":"2024-04-26T01:45:54.932538+0000","last_fullsized":"2024-04-26T01:45:54.932538+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:13:36.565257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"95'10","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944606+0000","last_change":"2024-04-26T01:42:05.402847+0000","last_active":"2024-04-26T01:45:54.944606+0000","last_peered":"2024-04-26T01:45:54.944606+0000","last_clean":"2024-04-26T01:45:54.944606+0000","last_became_active":"2024-04-26T01:42:05.402503+0000","last_became_peered":"2024-04-26T01:42:05.402503+0000","last_unstale":"2024-04-26T01:45:54.944606+0000","last_undegraded":"2024-04-26T01:45:54.944606+0000","last_fullsized":"2024-04-26T01:45:54.944606+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:06.935060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.183651+0000","last_change":"2024-04-26T01:42:05.405257+0000","last_active":"2024-04-26T01:45:55.183651+0000","last_peered":"2024-04-26T01:45:55.183651+0000","last_clean":"2024-04-26T01:45:55.183651+0000","last_became_active":"2024-04-26T01:42:05.404551+0000","last_became_peered":"2024-04-26T01:42:05.404551+0000","last_unstale":"2024-04-26T01:45:55.183651+0000","last_undegraded":"2024-04-26T01:45:55.183651+0000","last_fullsized":"2024-04-26T01:45:55.183651+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:44:20.754088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.183628+0000","last_change":"2024-04-26T01:42:05.408847+0000","last_active":"2024-04-26T01:45:55.183628+0000","last_peered":"2024-04-26T01:45:55.183628+0000","last_clean":"2024-04-26T01:45:55.183628+0000","last_became_active":"2024-04-26T01:42:05.406189+0000","last_became_peered":"2024-04-26T01:42:05.406189+0000","last_unstale":"2024-04-26T01:45:55.183628+0000","last_undegraded":"2024-04-26T01:45:55.183628+0000","last_fullsized":"2024-04-26T01:45:55.183628+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:59:58.633630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961111+0000","last_change":"2024-04-26T01:42:05.404647+0000","last_active":"2024-04-26T01:45:54.961111+0000","last_peered":"2024-04-26T01:45:54.961111+0000","last_clean":"2024-04-26T01:45:54.961111+0000","last_became_active":"2024-04-26T01:42:05.404265+0000","last_became_peered":"2024-04-26T01:42:05.404265+0000","last_unstale":"2024-04-26T01:45:54.961111+0000","last_undegraded":"2024-04-26T01:45:54.961111+0000","last_fullsized":"2024-04-26T01:45:54.961111+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:00:43.472541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962928+0000","last_change":"2024-04-26T01:42:05.417602+0000","last_active":"2024-04-26T01:45:54.962928+0000","last_peered":"2024-04-26T01:45:54.962928+0000","last_clean":"2024-04-26T01:45:54.962928+0000","last_became_active":"2024-04-26T01:42:05.417144+0000","last_became_peered":"2024-04-26T01:42:05.417144+0000","last_unstale":"2024-04-26T01:45:54.962928+0000","last_undegraded":"2024-04-26T01:45:54.962928+0000","last_fullsized":"2024-04-26T01:45:54.962928+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:08:34.043597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"95'10","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.934166+0000","last_change":"2024-04-26T01:42:05.392948+0000","last_active":"2024-04-26T01:45:54.934166+0000","last_peered":"2024-04-26T01:45:54.934166+0000","last_clean":"2024-04-26T01:45:54.934166+0000","last_became_active":"2024-04-26T01:42:05.392040+0000","last_became_peered":"2024-04-26T01:42:05.392040+0000","last_unstale":"2024-04-26T01:45:54.934166+0000","last_undegraded":"2024-04-26T01:45:54.934166+0000","last_fullsized":"2024-04-26T01:45:54.934166+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:35:17.307979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931776+0000","last_change":"2024-04-26T01:42:05.406588+0000","last_active":"2024-04-26T01:45:54.931776+0000","last_peered":"2024-04-26T01:45:54.931776+0000","last_clean":"2024-04-26T01:45:54.931776+0000","last_became_active":"2024-04-26T01:42:05.406200+0000","last_became_peered":"2024-04-26T01:42:05.406200+0000","last_unstale":"2024-04-26T01:45:54.931776+0000","last_undegraded":"2024-04-26T01:45:54.931776+0000","last_fullsized":"2024-04-26T01:45:54.931776+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:14.095383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932704+0000","last_change":"2024-04-26T01:42:01.339289+0000","last_active":"2024-04-26T01:45:54.932704+0000","last_peered":"2024-04-26T01:45:54.932704+0000","last_clean":"2024-04-26T01:45:54.932704+0000","last_became_active":"2024-04-26T01:42:01.335402+0000","last_became_peered":"2024-04-26T01:42:01.335402+0000","last_unstale":"2024-04-26T01:45:54.932704+0000","last_undegraded":"2024-04-26T01:45:54.932704+0000","last_fullsized":"2024-04-26T01:45:54.932704+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:51.411524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"95'5","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.950393+0000","last_change":"2024-04-26T01:42:05.409569+0000","last_active":"2024-04-26T01:45:54.950393+0000","last_peered":"2024-04-26T01:45:54.950393+0000","last_clean":"2024-04-26T01:45:54.950393+0000","last_became_active":"2024-04-26T01:42:05.407441+0000","last_became_peered":"2024-04-26T01:42:05.407441+0000","last_unstale":"2024-04-26T01:45:54.950393+0000","last_undegraded":"2024-04-26T01:45:54.950393+0000","last_fullsized":"2024-04-26T01:45:54.950393+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:36:58.043359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.951688+0000","last_change":"2024-04-26T01:42:02.231871+0000","last_active":"2024-04-26T01:45:54.951688+0000","last_peered":"2024-04-26T01:45:54.951688+0000","last_clean":"2024-04-26T01:45:54.951688+0000","last_became_active":"2024-04-26T01:42:02.230813+0000","last_became_peered":"2024-04-26T01:42:02.230813+0000","last_unstale":"2024-04-26T01:45:54.951688+0000","last_undegraded":"2024-04-26T01:45:54.951688+0000","last_fullsized":"2024-04-26T01:45:54.951688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:53.117266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.183893+0000","last_change":"2024-04-26T01:42:05.412414+0000","last_active":"2024-04-26T01:45:55.183893+0000","last_peered":"2024-04-26T01:45:55.183893+0000","last_clean":"2024-04-26T01:45:55.183893+0000","last_became_active":"2024-04-26T01:42:05.412057+0000","last_became_peered":"2024-04-26T01:42:05.412057+0000","last_unstale":"2024-04-26T01:45:55.183893+0000","last_undegraded":"2024-04-26T01:45:55.183893+0000","last_fullsized":"2024-04-26T01:45:55.183893+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:51:46.331246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.963097+0000","last_change":"2024-04-26T01:42:01.329957+0000","last_active":"2024-04-26T01:45:54.963097+0000","last_peered":"2024-04-26T01:45:54.963097+0000","last_clean":"2024-04-26T01:45:54.963097+0000","last_became_active":"2024-04-26T01:42:01.329289+0000","last_became_peered":"2024-04-26T01:42:01.329289+0000","last_unstale":"2024-04-26T01:45:54.963097+0000","last_undegraded":"2024-04-26T01:45:54.963097+0000","last_fullsized":"2024-04-26T01:45:54.963097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:46:38.983138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961476+0000","last_change":"2024-04-26T01:42:05.399438+0000","last_active":"2024-04-26T01:45:54.961476+0000","last_peered":"2024-04-26T01:45:54.961476+0000","last_clean":"2024-04-26T01:45:54.961476+0000","last_became_active":"2024-04-26T01:42:05.399082+0000","last_became_peered":"2024-04-26T01:42:05.399082+0000","last_unstale":"2024-04-26T01:45:54.961476+0000","last_undegraded":"2024-04-26T01:45:54.961476+0000","last_fullsized":"2024-04-26T01:45:54.961476+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:13.619893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"91'2","reported_seq":17,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961916+0000","last_change":"2024-04-26T01:43:49.993134+0000","last_active":"2024-04-26T01:45:54.961916+0000","last_peered":"2024-04-26T01:45:54.961916+0000","last_clean":"2024-04-26T01:45:54.961916+0000","last_became_active":"2024-04-26T01:43:49.990337+0000","last_became_peered":"2024-04-26T01:43:49.990337+0000","last_unstale":"2024-04-26T01:45:54.961916+0000","last_undegraded":"2024-04-26T01:45:54.961916+0000","last_fullsized":"2024-04-26T01:45:54.961916+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:50.502668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"95'15","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.933121+0000","last_change":"2024-04-26T01:42:49.066057+0000","last_active":"2024-04-26T01:45:54.933121+0000","last_peered":"2024-04-26T01:45:54.933121+0000","last_clean":"2024-04-26T01:45:54.933121+0000","last_became_active":"2024-04-26T01:42:49.064684+0000","last_became_peered":"2024-04-26T01:42:49.064684+0000","last_unstale":"2024-04-26T01:45:54.933121+0000","last_undegraded":"2024-04-26T01:45:54.933121+0000","last_fullsized":"2024-04-26T01:45:54.933121+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:39.159528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"96'9","reported_seq":266,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:46:03.980366+0000","last_change":"2024-04-26T01:42:01.332623+0000","last_active":"2024-04-26T01:46:03.980366+0000","last_peered":"2024-04-26T01:46:03.980366+0000","last_clean":"2024-04-26T01:46:03.980366+0000","last_became_active":"2024-04-26T01:42:01.332227+0000","last_became_peered":"2024-04-26T01:42:01.332227+0000","last_unstale":"2024-04-26T01:46:03.980366+0000","last_undegraded":"2024-04-26T01:46:03.980366+0000","last_fullsized":"2024-04-26T01:46:03.980366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:25:58.611464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1855,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":224,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961832+0000","last_change":"2024-04-26T01:42:05.396133+0000","last_active":"2024-04-26T01:45:54.961832+0000","last_peered":"2024-04-26T01:45:54.961832+0000","last_clean":"2024-04-26T01:45:54.961832+0000","last_became_active":"2024-04-26T01:42:05.395553+0000","last_became_peered":"2024-04-26T01:42:05.395553+0000","last_unstale":"2024-04-26T01:45:54.961832+0000","last_undegraded":"2024-04-26T01:45:54.961832+0000","last_fullsized":"2024-04-26T01:45:54.961832+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:36:01.363909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931431+0000","last_change":"2024-04-26T01:42:01.340624+0000","last_active":"2024-04-26T01:45:54.931431+0000","last_peered":"2024-04-26T01:45:54.931431+0000","last_clean":"2024-04-26T01:45:54.931431+0000","last_became_active":"2024-04-26T01:42:01.340158+0000","last_became_peered":"2024-04-26T01:42:01.340158+0000","last_unstale":"2024-04-26T01:45:54.931431+0000","last_undegraded":"2024-04-26T01:45:54.931431+0000","last_fullsized":"2024-04-26T01:45:54.931431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:47:05.529922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.963143+0000","last_change":"2024-04-26T01:42:05.406354+0000","last_active":"2024-04-26T01:45:54.963143+0000","last_peered":"2024-04-26T01:45:54.963143+0000","last_clean":"2024-04-26T01:45:54.963143+0000","last_became_active":"2024-04-26T01:42:05.405989+0000","last_became_peered":"2024-04-26T01:42:05.405989+0000","last_unstale":"2024-04-26T01:45:54.963143+0000","last_undegraded":"2024-04-26T01:45:54.963143+0000","last_fullsized":"2024-04-26T01:45:54.963143+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:21:43.115225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944206+0000","last_change":"2024-04-26T01:42:01.340549+0000","last_active":"2024-04-26T01:45:54.944206+0000","last_peered":"2024-04-26T01:45:54.944206+0000","last_clean":"2024-04-26T01:45:54.944206+0000","last_became_active":"2024-04-26T01:42:01.339861+0000","last_became_peered":"2024-04-26T01:42:01.339861+0000","last_unstale":"2024-04-26T01:45:54.944206+0000","last_undegraded":"2024-04-26T01:45:54.944206+0000","last_fullsized":"2024-04-26T01:45:54.944206+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:02:53.154367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949325+0000","last_change":"2024-04-26T01:42:05.402664+0000","last_active":"2024-04-26T01:45:54.949325+0000","last_peered":"2024-04-26T01:45:54.949325+0000","last_clean":"2024-04-26T01:45:54.949325+0000","last_became_active":"2024-04-26T01:42:05.400785+0000","last_became_peered":"2024-04-26T01:42:05.400785+0000","last_unstale":"2024-04-26T01:45:54.949325+0000","last_undegraded":"2024-04-26T01:45:54.949325+0000","last_fullsized":"2024-04-26T01:45:54.949325+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:40:32.137752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961622+0000","last_change":"2024-04-26T01:42:01.328600+0000","last_active":"2024-04-26T01:45:54.961622+0000","last_peered":"2024-04-26T01:45:54.961622+0000","last_clean":"2024-04-26T01:45:54.961622+0000","last_became_active":"2024-04-26T01:42:01.328149+0000","last_became_peered":"2024-04-26T01:42:01.328149+0000","last_unstale":"2024-04-26T01:45:54.961622+0000","last_undegraded":"2024-04-26T01:45:54.961622+0000","last_fullsized":"2024-04-26T01:45:54.961622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:45:42.811088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"95'11","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932233+0000","last_change":"2024-04-26T01:42:49.065824+0000","last_active":"2024-04-26T01:45:54.932233+0000","last_peered":"2024-04-26T01:45:54.932233+0000","last_clean":"2024-04-26T01:45:54.932233+0000","last_became_active":"2024-04-26T01:42:49.064670+0000","last_became_peered":"2024-04-26T01:42:49.064670+0000","last_unstale":"2024-04-26T01:45:54.932233+0000","last_undegraded":"2024-04-26T01:45:54.932233+0000","last_fullsized":"2024-04-26T01:45:54.932233+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:44.790184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,10],"acting":[9,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"95'5","reported_seq":256,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:46:03.979404+0000","last_change":"2024-04-26T01:42:01.335438+0000","last_active":"2024-04-26T01:46:03.979404+0000","last_peered":"2024-04-26T01:46:03.979404+0000","last_clean":"2024-04-26T01:46:03.979404+0000","last_became_active":"2024-04-26T01:42:01.334962+0000","last_became_peered":"2024-04-26T01:42:01.334962+0000","last_unstale":"2024-04-26T01:46:03.979404+0000","last_undegraded":"2024-04-26T01:46:03.979404+0000","last_fullsized":"2024-04-26T01:46:03.979404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:35:08.252274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":218,"num_read_kb":218,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"95'10","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961591+0000","last_change":"2024-04-26T01:42:05.393294+0000","last_active":"2024-04-26T01:45:54.961591+0000","last_peered":"2024-04-26T01:45:54.961591+0000","last_clean":"2024-04-26T01:45:54.961591+0000","last_became_active":"2024-04-26T01:42:05.392974+0000","last_became_peered":"2024-04-26T01:42:05.392974+0000","last_unstale":"2024-04-26T01:45:54.961591+0000","last_undegraded":"2024-04-26T01:45:54.961591+0000","last_fullsized":"2024-04-26T01:45:54.961591+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:58.188706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962548+0000","last_change":"2024-04-26T01:42:01.333505+0000","last_active":"2024-04-26T01:45:54.962548+0000","last_peered":"2024-04-26T01:45:54.962548+0000","last_clean":"2024-04-26T01:45:54.962548+0000","last_became_active":"2024-04-26T01:42:01.333062+0000","last_became_peered":"2024-04-26T01:42:01.333062+0000","last_unstale":"2024-04-26T01:45:54.962548+0000","last_undegraded":"2024-04-26T01:45:54.962548+0000","last_fullsized":"2024-04-26T01:45:54.962548+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:50:48.804443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"95'9","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.184062+0000","last_change":"2024-04-26T01:42:49.063948+0000","last_active":"2024-04-26T01:45:55.184062+0000","last_peered":"2024-04-26T01:45:55.184062+0000","last_clean":"2024-04-26T01:45:55.184062+0000","last_became_active":"2024-04-26T01:42:49.060852+0000","last_became_peered":"2024-04-26T01:42:49.060852+0000","last_unstale":"2024-04-26T01:45:55.184062+0000","last_undegraded":"2024-04-26T01:45:55.184062+0000","last_fullsized":"2024-04-26T01:45:55.184062+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.370431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.963018+0000","last_change":"2024-04-26T01:42:02.232308+0000","last_active":"2024-04-26T01:45:54.963018+0000","last_peered":"2024-04-26T01:45:54.963018+0000","last_clean":"2024-04-26T01:45:54.963018+0000","last_became_active":"2024-04-26T01:42:02.231897+0000","last_became_peered":"2024-04-26T01:42:02.231897+0000","last_unstale":"2024-04-26T01:45:54.963018+0000","last_undegraded":"2024-04-26T01:45:54.963018+0000","last_fullsized":"2024-04-26T01:45:54.963018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:36:16.847087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"95'11","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961415+0000","last_change":"2024-04-26T01:42:49.070798+0000","last_active":"2024-04-26T01:45:54.961415+0000","last_peered":"2024-04-26T01:45:54.961415+0000","last_clean":"2024-04-26T01:45:54.961415+0000","last_became_active":"2024-04-26T01:42:49.070470+0000","last_became_peered":"2024-04-26T01:42:49.070470+0000","last_unstale":"2024-04-26T01:45:54.961415+0000","last_undegraded":"2024-04-26T01:45:54.961415+0000","last_fullsized":"2024-04-26T01:45:54.961415+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:04:22.103704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"96'43","reported_seq":530,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:46:05.868696+0000","last_change":"2024-04-26T01:42:01.330822+0000","last_active":"2024-04-26T01:46:05.868696+0000","last_peered":"2024-04-26T01:46:05.868696+0000","last_clean":"2024-04-26T01:46:05.868696+0000","last_became_active":"2024-04-26T01:42:01.330337+0000","last_became_peered":"2024-04-26T01:42:01.330337+0000","last_unstale":"2024-04-26T01:46:05.868696+0000","last_undegraded":"2024-04-26T01:46:05.868696+0000","last_fullsized":"2024-04-26T01:46:05.868696+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:04.024645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72446,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":454,"num_read_kb":299,"num_write":44,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962763+0000","last_change":"2024-04-26T01:42:05.390601+0000","last_active":"2024-04-26T01:45:54.962763+0000","last_peered":"2024-04-26T01:45:54.962763+0000","last_clean":"2024-04-26T01:45:54.962763+0000","last_became_active":"2024-04-26T01:42:05.390279+0000","last_became_peered":"2024-04-26T01:42:05.390279+0000","last_unstale":"2024-04-26T01:45:54.962763+0000","last_undegraded":"2024-04-26T01:45:54.962763+0000","last_fullsized":"2024-04-26T01:45:54.962763+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:20.171769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931520+0000","last_change":"2024-04-26T01:42:01.337200+0000","last_active":"2024-04-26T01:45:54.931520+0000","last_peered":"2024-04-26T01:45:54.931520+0000","last_clean":"2024-04-26T01:45:54.931520+0000","last_became_active":"2024-04-26T01:42:01.336919+0000","last_became_peered":"2024-04-26T01:42:01.336919+0000","last_unstale":"2024-04-26T01:45:54.931520+0000","last_undegraded":"2024-04-26T01:45:54.931520+0000","last_fullsized":"2024-04-26T01:45:54.931520+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:29.799903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944349+0000","last_change":"2024-04-26T01:42:05.410081+0000","last_active":"2024-04-26T01:45:54.944349+0000","last_peered":"2024-04-26T01:45:54.944349+0000","last_clean":"2024-04-26T01:45:54.944349+0000","last_became_active":"2024-04-26T01:42:05.409341+0000","last_became_peered":"2024-04-26T01:42:05.409341+0000","last_unstale":"2024-04-26T01:45:54.944349+0000","last_undegraded":"2024-04-26T01:45:54.944349+0000","last_fullsized":"2024-04-26T01:45:54.944349+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:25:31.925296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962585+0000","last_change":"2024-04-26T01:42:01.331702+0000","last_active":"2024-04-26T01:45:54.962585+0000","last_peered":"2024-04-26T01:45:54.962585+0000","last_clean":"2024-04-26T01:45:54.962585+0000","last_became_active":"2024-04-26T01:42:01.331311+0000","last_became_peered":"2024-04-26T01:42:01.331311+0000","last_unstale":"2024-04-26T01:45:54.962585+0000","last_undegraded":"2024-04-26T01:45:54.962585+0000","last_fullsized":"2024-04-26T01:45:54.962585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:54:49.513627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"95'15","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962407+0000","last_change":"2024-04-26T01:42:05.403957+0000","last_active":"2024-04-26T01:45:54.962407+0000","last_peered":"2024-04-26T01:45:54.962407+0000","last_clean":"2024-04-26T01:45:54.962407+0000","last_became_active":"2024-04-26T01:42:05.403583+0000","last_became_peered":"2024-04-26T01:42:05.403583+0000","last_unstale":"2024-04-26T01:45:54.962407+0000","last_undegraded":"2024-04-26T01:45:54.962407+0000","last_fullsized":"2024-04-26T01:45:54.962407+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:12:20.095407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"96'7","reported_seq":264,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:46:03.974373+0000","last_change":"2024-04-26T01:42:01.332456+0000","last_active":"2024-04-26T01:46:03.974373+0000","last_peered":"2024-04-26T01:46:03.974373+0000","last_clean":"2024-04-26T01:46:03.974373+0000","last_became_active":"2024-04-26T01:42:01.331910+0000","last_became_peered":"2024-04-26T01:42:01.331910+0000","last_unstale":"2024-04-26T01:46:03.974373+0000","last_undegraded":"2024-04-26T01:46:03.974373+0000","last_fullsized":"2024-04-26T01:46:03.974373+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:13:25.368205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":225,"num_read_kb":7,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"95'6","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961737+0000","last_change":"2024-04-26T01:42:49.062130+0000","last_active":"2024-04-26T01:45:54.961737+0000","last_peered":"2024-04-26T01:45:54.961737+0000","last_clean":"2024-04-26T01:45:54.961737+0000","last_became_active":"2024-04-26T01:42:49.061627+0000","last_became_peered":"2024-04-26T01:42:49.061627+0000","last_unstale":"2024-04-26T01:45:54.961737+0000","last_undegraded":"2024-04-26T01:45:54.961737+0000","last_fullsized":"2024-04-26T01:45:54.961737+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:16.101940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961990+0000","last_change":"2024-04-26T01:42:49.082618+0000","last_active":"2024-04-26T01:45:54.961990+0000","last_peered":"2024-04-26T01:45:54.961990+0000","last_clean":"2024-04-26T01:45:54.961990+0000","last_became_active":"2024-04-26T01:42:49.064654+0000","last_became_peered":"2024-04-26T01:42:49.064654+0000","last_unstale":"2024-04-26T01:45:54.961990+0000","last_undegraded":"2024-04-26T01:45:54.961990+0000","last_fullsized":"2024-04-26T01:45:54.961990+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:13:23.937059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962550+0000","last_change":"2024-04-26T01:42:05.391392+0000","last_active":"2024-04-26T01:45:54.962550+0000","last_peered":"2024-04-26T01:45:54.962550+0000","last_clean":"2024-04-26T01:45:54.962550+0000","last_became_active":"2024-04-26T01:42:05.390939+0000","last_became_peered":"2024-04-26T01:42:05.390939+0000","last_unstale":"2024-04-26T01:45:54.962550+0000","last_undegraded":"2024-04-26T01:45:54.962550+0000","last_fullsized":"2024-04-26T01:45:54.962550+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:36.141915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962306+0000","last_change":"2024-04-26T01:42:01.332932+0000","last_active":"2024-04-26T01:45:54.962306+0000","last_peered":"2024-04-26T01:45:54.962306+0000","last_clean":"2024-04-26T01:45:54.962306+0000","last_became_active":"2024-04-26T01:42:01.332626+0000","last_became_peered":"2024-04-26T01:42:01.332626+0000","last_unstale":"2024-04-26T01:45:54.962306+0000","last_undegraded":"2024-04-26T01:45:54.962306+0000","last_fullsized":"2024-04-26T01:45:54.962306+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:20.593730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962441+0000","last_change":"2024-04-26T01:42:05.394925+0000","last_active":"2024-04-26T01:45:54.962441+0000","last_peered":"2024-04-26T01:45:54.962441+0000","last_clean":"2024-04-26T01:45:54.962441+0000","last_became_active":"2024-04-26T01:42:05.394531+0000","last_became_peered":"2024-04-26T01:42:05.394531+0000","last_unstale":"2024-04-26T01:45:54.962441+0000","last_undegraded":"2024-04-26T01:45:54.962441+0000","last_fullsized":"2024-04-26T01:45:54.962441+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:33.177780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.932471+0000","last_change":"2024-04-26T01:42:02.237482+0000","last_active":"2024-04-26T01:45:54.932471+0000","last_peered":"2024-04-26T01:45:54.932471+0000","last_clean":"2024-04-26T01:45:54.932471+0000","last_became_active":"2024-04-26T01:42:02.236752+0000","last_became_peered":"2024-04-26T01:42:02.236752+0000","last_unstale":"2024-04-26T01:45:54.932471+0000","last_undegraded":"2024-04-26T01:45:54.932471+0000","last_fullsized":"2024-04-26T01:45:54.932471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:27.211245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931723+0000","last_change":"2024-04-26T01:42:05.402157+0000","last_active":"2024-04-26T01:45:54.931723+0000","last_peered":"2024-04-26T01:45:54.931723+0000","last_clean":"2024-04-26T01:45:54.931723+0000","last_became_active":"2024-04-26T01:42:05.401889+0000","last_became_peered":"2024-04-26T01:42:05.401889+0000","last_unstale":"2024-04-26T01:45:54.931723+0000","last_undegraded":"2024-04-26T01:45:54.931723+0000","last_fullsized":"2024-04-26T01:45:54.931723+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:33.854901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962227+0000","last_change":"2024-04-26T01:42:01.339675+0000","last_active":"2024-04-26T01:45:54.962227+0000","last_peered":"2024-04-26T01:45:54.962227+0000","last_clean":"2024-04-26T01:45:54.962227+0000","last_became_active":"2024-04-26T01:42:01.339247+0000","last_became_peered":"2024-04-26T01:42:01.339247+0000","last_unstale":"2024-04-26T01:45:54.962227+0000","last_undegraded":"2024-04-26T01:45:54.962227+0000","last_fullsized":"2024-04-26T01:45:54.962227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:32.724830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931641+0000","last_change":"2024-04-26T01:42:05.412913+0000","last_active":"2024-04-26T01:45:54.931641+0000","last_peered":"2024-04-26T01:45:54.931641+0000","last_clean":"2024-04-26T01:45:54.931641+0000","last_became_active":"2024-04-26T01:42:05.412556+0000","last_became_peered":"2024-04-26T01:42:05.412556+0000","last_unstale":"2024-04-26T01:45:54.931641+0000","last_undegraded":"2024-04-26T01:45:54.931641+0000","last_fullsized":"2024-04-26T01:45:54.931641+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:00:52.798069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961962+0000","last_change":"2024-04-26T01:42:05.391235+0000","last_active":"2024-04-26T01:45:54.961962+0000","last_peered":"2024-04-26T01:45:54.961962+0000","last_clean":"2024-04-26T01:45:54.961962+0000","last_became_active":"2024-04-26T01:42:05.390717+0000","last_became_peered":"2024-04-26T01:42:05.390717+0000","last_unstale":"2024-04-26T01:45:54.961962+0000","last_undegraded":"2024-04-26T01:45:54.961962+0000","last_fullsized":"2024-04-26T01:45:54.961962+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:07.541593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"96'6","reported_seq":257,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:46:03.973599+0000","last_change":"2024-04-26T01:42:02.232823+0000","last_active":"2024-04-26T01:46:03.973599+0000","last_peered":"2024-04-26T01:46:03.973599+0000","last_clean":"2024-04-26T01:46:03.973599+0000","last_became_active":"2024-04-26T01:42:02.231899+0000","last_became_peered":"2024-04-26T01:42:02.231899+0000","last_unstale":"2024-04-26T01:46:03.973599+0000","last_undegraded":"2024-04-26T01:46:03.973599+0000","last_fullsized":"2024-04-26T01:46:03.973599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:23.287305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":218,"num_read_kb":218,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949845+0000","last_change":"2024-04-26T01:42:05.405690+0000","last_active":"2024-04-26T01:45:54.949845+0000","last_peered":"2024-04-26T01:45:54.949845+0000","last_clean":"2024-04-26T01:45:54.949845+0000","last_became_active":"2024-04-26T01:42:05.405345+0000","last_became_peered":"2024-04-26T01:42:05.405345+0000","last_unstale":"2024-04-26T01:45:54.949845+0000","last_undegraded":"2024-04-26T01:45:54.949845+0000","last_fullsized":"2024-04-26T01:45:54.949845+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:48.377916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.947235+0000","last_change":"2024-04-26T01:42:01.335721+0000","last_active":"2024-04-26T01:45:54.947235+0000","last_peered":"2024-04-26T01:45:54.947235+0000","last_clean":"2024-04-26T01:45:54.947235+0000","last_became_active":"2024-04-26T01:42:01.335149+0000","last_became_peered":"2024-04-26T01:42:01.335149+0000","last_unstale":"2024-04-26T01:45:54.947235+0000","last_undegraded":"2024-04-26T01:45:54.947235+0000","last_fullsized":"2024-04-26T01:45:54.947235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:36.372033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"96'148","reported_seq":413,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:57.098863+0000","last_change":"2024-04-26T01:40:28.086333+0000","last_active":"2024-04-26T01:45:57.098863+0000","last_peered":"2024-04-26T01:45:57.098863+0000","last_clean":"2024-04-26T01:45:57.098863+0000","last_became_active":"2024-04-26T01:40:28.085902+0000","last_became_peered":"2024-04-26T01:40:28.085902+0000","last_unstale":"2024-04-26T01:45:57.098863+0000","last_undegraded":"2024-04-26T01:45:57.098863+0000","last_fullsized":"2024-04-26T01:45:57.098863+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:36:35.357698+0000","last_clean_scrub_stamp":"2024-04-26T01:36:35.357698+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:43.290117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944108+0000","last_change":"2024-04-26T01:42:05.401896+0000","last_active":"2024-04-26T01:45:54.944108+0000","last_peered":"2024-04-26T01:45:54.944108+0000","last_clean":"2024-04-26T01:45:54.944108+0000","last_became_active":"2024-04-26T01:42:05.401496+0000","last_became_peered":"2024-04-26T01:42:05.401496+0000","last_unstale":"2024-04-26T01:45:54.944108+0000","last_undegraded":"2024-04-26T01:45:54.944108+0000","last_fullsized":"2024-04-26T01:45:54.944108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:49.758444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962025+0000","last_change":"2024-04-26T01:42:01.329705+0000","last_active":"2024-04-26T01:45:54.962025+0000","last_peered":"2024-04-26T01:45:54.962025+0000","last_clean":"2024-04-26T01:45:54.962025+0000","last_became_active":"2024-04-26T01:42:01.329153+0000","last_became_peered":"2024-04-26T01:42:01.329153+0000","last_unstale":"2024-04-26T01:45:54.962025+0000","last_undegraded":"2024-04-26T01:45:54.962025+0000","last_fullsized":"2024-04-26T01:45:54.962025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:06.747378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962915+0000","last_change":"2024-04-26T01:42:02.232148+0000","last_active":"2024-04-26T01:45:54.962915+0000","last_peered":"2024-04-26T01:45:54.962915+0000","last_clean":"2024-04-26T01:45:54.962915+0000","last_became_active":"2024-04-26T01:42:02.231739+0000","last_became_peered":"2024-04-26T01:42:02.231739+0000","last_unstale":"2024-04-26T01:45:54.962915+0000","last_undegraded":"2024-04-26T01:45:54.962915+0000","last_fullsized":"2024-04-26T01:45:54.962915+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:09.752433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.945042+0000","last_change":"2024-04-26T01:42:05.399938+0000","last_active":"2024-04-26T01:45:54.945042+0000","last_peered":"2024-04-26T01:45:54.945042+0000","last_clean":"2024-04-26T01:45:54.945042+0000","last_became_active":"2024-04-26T01:42:05.399550+0000","last_became_peered":"2024-04-26T01:42:05.399550+0000","last_unstale":"2024-04-26T01:45:54.945042+0000","last_undegraded":"2024-04-26T01:45:54.945042+0000","last_fullsized":"2024-04-26T01:45:54.945042+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:41:44.142284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"96'126","reported_seq":284,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:46:00.780819+0000","last_change":"2024-04-26T01:42:01.337437+0000","last_active":"2024-04-26T01:46:00.780819+0000","last_peered":"2024-04-26T01:46:00.780819+0000","last_clean":"2024-04-26T01:46:00.780819+0000","last_became_active":"2024-04-26T01:42:01.336894+0000","last_became_peered":"2024-04-26T01:42:01.336894+0000","last_unstale":"2024-04-26T01:46:00.780819+0000","last_undegraded":"2024-04-26T01:46:00.780819+0000","last_fullsized":"2024-04-26T01:46:00.780819+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:04:31.093273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":422,"num_read_kb":376,"num_write":233,"num_write_kb":1415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.951785+0000","last_change":"2024-04-26T01:42:05.411694+0000","last_active":"2024-04-26T01:45:54.951785+0000","last_peered":"2024-04-26T01:45:54.951785+0000","last_clean":"2024-04-26T01:45:54.951785+0000","last_became_active":"2024-04-26T01:42:05.411300+0000","last_became_peered":"2024-04-26T01:42:05.411300+0000","last_unstale":"2024-04-26T01:45:54.951785+0000","last_undegraded":"2024-04-26T01:45:54.951785+0000","last_fullsized":"2024-04-26T01:45:54.951785+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:42:08.116145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.947485+0000","last_change":"2024-04-26T01:42:01.427501+0000","last_active":"2024-04-26T01:45:54.947485+0000","last_peered":"2024-04-26T01:45:54.947485+0000","last_clean":"2024-04-26T01:45:54.947485+0000","last_became_active":"2024-04-26T01:42:01.426970+0000","last_became_peered":"2024-04-26T01:42:01.426970+0000","last_unstale":"2024-04-26T01:45:54.947485+0000","last_undegraded":"2024-04-26T01:45:54.947485+0000","last_fullsized":"2024-04-26T01:45:54.947485+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:53:53.118296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.184575+0000","last_change":"2024-04-26T01:42:05.414790+0000","last_active":"2024-04-26T01:45:55.184575+0000","last_peered":"2024-04-26T01:45:55.184575+0000","last_clean":"2024-04-26T01:45:55.184575+0000","last_became_active":"2024-04-26T01:42:05.411947+0000","last_became_peered":"2024-04-26T01:42:05.411947+0000","last_unstale":"2024-04-26T01:45:55.184575+0000","last_undegraded":"2024-04-26T01:45:55.184575+0000","last_fullsized":"2024-04-26T01:45:55.184575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:11:34.450815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961147+0000","last_change":"2024-04-26T01:42:02.230539+0000","last_active":"2024-04-26T01:45:54.961147+0000","last_peered":"2024-04-26T01:45:54.961147+0000","last_clean":"2024-04-26T01:45:54.961147+0000","last_became_active":"2024-04-26T01:42:02.230150+0000","last_became_peered":"2024-04-26T01:42:02.230150+0000","last_unstale":"2024-04-26T01:45:54.961147+0000","last_undegraded":"2024-04-26T01:45:54.961147+0000","last_fullsized":"2024-04-26T01:45:54.961147+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:08:03.073069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"95'2","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961652+0000","last_change":"2024-04-26T01:42:05.388877+0000","last_active":"2024-04-26T01:45:54.961652+0000","last_peered":"2024-04-26T01:45:54.961652+0000","last_clean":"2024-04-26T01:45:54.961652+0000","last_became_active":"2024-04-26T01:42:05.388457+0000","last_became_peered":"2024-04-26T01:42:05.388457+0000","last_unstale":"2024-04-26T01:45:54.961652+0000","last_undegraded":"2024-04-26T01:45:54.961652+0000","last_fullsized":"2024-04-26T01:45:54.961652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:54.436859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949462+0000","last_change":"2024-04-26T01:42:01.343461+0000","last_active":"2024-04-26T01:45:54.949462+0000","last_peered":"2024-04-26T01:45:54.949462+0000","last_clean":"2024-04-26T01:45:54.949462+0000","last_became_active":"2024-04-26T01:42:01.342713+0000","last_became_peered":"2024-04-26T01:42:01.342713+0000","last_unstale":"2024-04-26T01:45:54.949462+0000","last_undegraded":"2024-04-26T01:45:54.949462+0000","last_fullsized":"2024-04-26T01:45:54.949462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:38.471858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962158+0000","last_change":"2024-04-26T01:42:05.415101+0000","last_active":"2024-04-26T01:45:54.962158+0000","last_peered":"2024-04-26T01:45:54.962158+0000","last_clean":"2024-04-26T01:45:54.962158+0000","last_became_active":"2024-04-26T01:42:05.414745+0000","last_became_peered":"2024-04-26T01:42:05.414745+0000","last_unstale":"2024-04-26T01:45:54.962158+0000","last_undegraded":"2024-04-26T01:45:54.962158+0000","last_fullsized":"2024-04-26T01:45:54.962158+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:13:08.573547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"96'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931967+0000","last_change":"2024-04-26T01:42:01.339501+0000","last_active":"2024-04-26T01:45:54.931967+0000","last_peered":"2024-04-26T01:45:54.931967+0000","last_clean":"2024-04-26T01:45:54.931967+0000","last_became_active":"2024-04-26T01:42:01.339132+0000","last_became_peered":"2024-04-26T01:42:01.339132+0000","last_unstale":"2024-04-26T01:45:54.931967+0000","last_undegraded":"2024-04-26T01:45:54.931967+0000","last_fullsized":"2024-04-26T01:45:54.931967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:16.463099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949949+0000","last_change":"2024-04-26T01:42:05.417864+0000","last_active":"2024-04-26T01:45:54.949949+0000","last_peered":"2024-04-26T01:45:54.949949+0000","last_clean":"2024-04-26T01:45:54.949949+0000","last_became_active":"2024-04-26T01:42:05.417414+0000","last_became_peered":"2024-04-26T01:42:05.417414+0000","last_unstale":"2024-04-26T01:45:54.949949+0000","last_undegraded":"2024-04-26T01:45:54.949949+0000","last_fullsized":"2024-04-26T01:45:54.949949+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:23:12.793421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":50,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962222+0000","last_change":"2024-04-26T01:42:49.063769+0000","last_active":"2024-04-26T01:45:54.962222+0000","last_peered":"2024-04-26T01:45:54.962222+0000","last_clean":"2024-04-26T01:45:54.962222+0000","last_became_active":"2024-04-26T01:42:49.057463+0000","last_became_peered":"2024-04-26T01:42:49.057463+0000","last_unstale":"2024-04-26T01:45:54.962222+0000","last_undegraded":"2024-04-26T01:45:54.962222+0000","last_fullsized":"2024-04-26T01:45:54.962222+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:34:54.511304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"95'5","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962310+0000","last_change":"2024-04-26T01:42:05.397980+0000","last_active":"2024-04-26T01:45:54.962310+0000","last_peered":"2024-04-26T01:45:54.962310+0000","last_clean":"2024-04-26T01:45:54.962310+0000","last_became_active":"2024-04-26T01:42:05.397589+0000","last_became_peered":"2024-04-26T01:42:05.397589+0000","last_unstale":"2024-04-26T01:45:54.962310+0000","last_undegraded":"2024-04-26T01:45:54.962310+0000","last_fullsized":"2024-04-26T01:45:54.962310+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:39:55.863566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931803+0000","last_change":"2024-04-26T01:42:02.234313+0000","last_active":"2024-04-26T01:45:54.931803+0000","last_peered":"2024-04-26T01:45:54.931803+0000","last_clean":"2024-04-26T01:45:54.931803+0000","last_became_active":"2024-04-26T01:42:02.233873+0000","last_became_peered":"2024-04-26T01:42:02.233873+0000","last_unstale":"2024-04-26T01:45:54.931803+0000","last_undegraded":"2024-04-26T01:45:54.931803+0000","last_fullsized":"2024-04-26T01:45:54.931803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:34:34.092434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962665+0000","last_change":"2024-04-26T01:42:05.387435+0000","last_active":"2024-04-26T01:45:54.962665+0000","last_peered":"2024-04-26T01:45:54.962665+0000","last_clean":"2024-04-26T01:45:54.962665+0000","last_became_active":"2024-04-26T01:42:05.386964+0000","last_became_peered":"2024-04-26T01:42:05.386964+0000","last_unstale":"2024-04-26T01:45:54.962665+0000","last_undegraded":"2024-04-26T01:45:54.962665+0000","last_fullsized":"2024-04-26T01:45:54.962665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:12:27.905842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"96'73","reported_seq":148,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:46:00.772676+0000","last_change":"2024-04-26T01:42:01.326432+0000","last_active":"2024-04-26T01:46:00.772676+0000","last_peered":"2024-04-26T01:46:00.772676+0000","last_clean":"2024-04-26T01:46:00.772676+0000","last_became_active":"2024-04-26T01:42:01.325989+0000","last_became_peered":"2024-04-26T01:42:01.325989+0000","last_unstale":"2024-04-26T01:46:00.772676+0000","last_undegraded":"2024-04-26T01:46:00.772676+0000","last_fullsized":"2024-04-26T01:46:00.772676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:10.049563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":44,"num_write":84,"num_write_kb":1075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961654+0000","last_change":"2024-04-26T01:42:05.412802+0000","last_active":"2024-04-26T01:45:54.961654+0000","last_peered":"2024-04-26T01:45:54.961654+0000","last_clean":"2024-04-26T01:45:54.961654+0000","last_became_active":"2024-04-26T01:42:05.412443+0000","last_became_peered":"2024-04-26T01:42:05.412443+0000","last_unstale":"2024-04-26T01:45:54.961654+0000","last_undegraded":"2024-04-26T01:45:54.961654+0000","last_fullsized":"2024-04-26T01:45:54.961654+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:28:56.360778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'2","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944056+0000","last_change":"2024-04-26T01:42:49.080305+0000","last_active":"2024-04-26T01:45:54.944056+0000","last_peered":"2024-04-26T01:45:54.944056+0000","last_clean":"2024-04-26T01:45:54.944056+0000","last_became_active":"2024-04-26T01:42:49.074476+0000","last_became_peered":"2024-04-26T01:42:49.074476+0000","last_unstale":"2024-04-26T01:45:54.944056+0000","last_undegraded":"2024-04-26T01:45:54.944056+0000","last_fullsized":"2024-04-26T01:45:54.944056+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:00.294088+0000","last_clean_scrub_stamp":"2024-04-26T01:42:00.294088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:58:06.619961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962655+0000","last_change":"2024-04-26T01:42:05.387621+0000","last_active":"2024-04-26T01:45:54.962655+0000","last_peered":"2024-04-26T01:45:54.962655+0000","last_clean":"2024-04-26T01:45:54.962655+0000","last_became_active":"2024-04-26T01:42:05.387165+0000","last_became_peered":"2024-04-26T01:42:05.387165+0000","last_unstale":"2024-04-26T01:45:54.962655+0000","last_undegraded":"2024-04-26T01:45:54.962655+0000","last_fullsized":"2024-04-26T01:45:54.962655+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:37.803330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"95'9","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962291+0000","last_change":"2024-04-26T01:42:05.386787+0000","last_active":"2024-04-26T01:45:54.962291+0000","last_peered":"2024-04-26T01:45:54.962291+0000","last_clean":"2024-04-26T01:45:54.962291+0000","last_became_active":"2024-04-26T01:42:05.386410+0000","last_became_peered":"2024-04-26T01:42:05.386410+0000","last_unstale":"2024-04-26T01:45:54.962291+0000","last_undegraded":"2024-04-26T01:45:54.962291+0000","last_fullsized":"2024-04-26T01:45:54.962291+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:28:42.137191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"95'12","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931481+0000","last_change":"2024-04-26T01:42:05.392142+0000","last_active":"2024-04-26T01:45:54.931481+0000","last_peered":"2024-04-26T01:45:54.931481+0000","last_clean":"2024-04-26T01:45:54.931481+0000","last_became_active":"2024-04-26T01:42:05.391829+0000","last_became_peered":"2024-04-26T01:42:05.391829+0000","last_unstale":"2024-04-26T01:45:54.931481+0000","last_undegraded":"2024-04-26T01:45:54.931481+0000","last_fullsized":"2024-04-26T01:45:54.931481+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:00:25.934641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931859+0000","last_change":"2024-04-26T01:42:05.409184+0000","last_active":"2024-04-26T01:45:54.931859+0000","last_peered":"2024-04-26T01:45:54.931859+0000","last_clean":"2024-04-26T01:45:54.931859+0000","last_became_active":"2024-04-26T01:42:05.408894+0000","last_became_peered":"2024-04-26T01:42:05.408894+0000","last_unstale":"2024-04-26T01:45:54.931859+0000","last_undegraded":"2024-04-26T01:45:54.931859+0000","last_fullsized":"2024-04-26T01:45:54.931859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:36:53.291844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"95'7","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:55.183917+0000","last_change":"2024-04-26T01:42:05.406940+0000","last_active":"2024-04-26T01:45:55.183917+0000","last_peered":"2024-04-26T01:45:55.183917+0000","last_clean":"2024-04-26T01:45:55.183917+0000","last_became_active":"2024-04-26T01:42:05.406476+0000","last_became_peered":"2024-04-26T01:42:05.406476+0000","last_unstale":"2024-04-26T01:45:55.183917+0000","last_undegraded":"2024-04-26T01:45:55.183917+0000","last_fullsized":"2024-04-26T01:45:55.183917+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:21:37.734633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"95'4","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.945671+0000","last_change":"2024-04-26T01:42:05.413521+0000","last_active":"2024-04-26T01:45:54.945671+0000","last_peered":"2024-04-26T01:45:54.945671+0000","last_clean":"2024-04-26T01:45:54.945671+0000","last_became_active":"2024-04-26T01:42:05.412789+0000","last_became_peered":"2024-04-26T01:42:05.412789+0000","last_unstale":"2024-04-26T01:45:54.945671+0000","last_undegraded":"2024-04-26T01:45:54.945671+0000","last_fullsized":"2024-04-26T01:45:54.945671+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:13.471376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962356+0000","last_change":"2024-04-26T01:42:05.391395+0000","last_active":"2024-04-26T01:45:54.962356+0000","last_peered":"2024-04-26T01:45:54.962356+0000","last_clean":"2024-04-26T01:45:54.962356+0000","last_became_active":"2024-04-26T01:42:05.391048+0000","last_became_peered":"2024-04-26T01:42:05.391048+0000","last_unstale":"2024-04-26T01:45:54.962356+0000","last_undegraded":"2024-04-26T01:45:54.962356+0000","last_fullsized":"2024-04-26T01:45:54.962356+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:45:56.645454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"95'15","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.931782+0000","last_change":"2024-04-26T01:42:05.386868+0000","last_active":"2024-04-26T01:45:54.931782+0000","last_peered":"2024-04-26T01:45:54.931782+0000","last_clean":"2024-04-26T01:45:54.931782+0000","last_became_active":"2024-04-26T01:42:05.386483+0000","last_became_peered":"2024-04-26T01:42:05.386483+0000","last_unstale":"2024-04-26T01:45:54.931782+0000","last_undegraded":"2024-04-26T01:45:54.931782+0000","last_fullsized":"2024-04-26T01:45:54.931782+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:42:44.453076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"95'2","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.951689+0000","last_change":"2024-04-26T01:42:05.406711+0000","last_active":"2024-04-26T01:45:54.951689+0000","last_peered":"2024-04-26T01:45:54.951689+0000","last_clean":"2024-04-26T01:45:54.951689+0000","last_became_active":"2024-04-26T01:42:05.406318+0000","last_became_peered":"2024-04-26T01:42:05.406318+0000","last_unstale":"2024-04-26T01:45:54.951689+0000","last_undegraded":"2024-04-26T01:45:54.951689+0000","last_fullsized":"2024-04-26T01:45:54.951689+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:36:16.599678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"95'6","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961752+0000","last_change":"2024-04-26T01:42:05.406305+0000","last_active":"2024-04-26T01:45:54.961752+0000","last_peered":"2024-04-26T01:45:54.961752+0000","last_clean":"2024-04-26T01:45:54.961752+0000","last_became_active":"2024-04-26T01:42:05.405953+0000","last_became_peered":"2024-04-26T01:42:05.405953+0000","last_unstale":"2024-04-26T01:45:54.961752+0000","last_undegraded":"2024-04-26T01:45:54.961752+0000","last_fullsized":"2024-04-26T01:45:54.961752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:36.588672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.949192+0000","last_change":"2024-04-26T01:42:05.413757+0000","last_active":"2024-04-26T01:45:54.949192+0000","last_peered":"2024-04-26T01:45:54.949192+0000","last_clean":"2024-04-26T01:45:54.949192+0000","last_became_active":"2024-04-26T01:42:05.413410+0000","last_became_peered":"2024-04-26T01:42:05.413410+0000","last_unstale":"2024-04-26T01:45:54.949192+0000","last_undegraded":"2024-04-26T01:45:54.949192+0000","last_fullsized":"2024-04-26T01:45:54.949192+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:30.298443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"95'10","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.961329+0000","last_change":"2024-04-26T01:42:05.412268+0000","last_active":"2024-04-26T01:45:54.961329+0000","last_peered":"2024-04-26T01:45:54.961329+0000","last_clean":"2024-04-26T01:45:54.961329+0000","last_became_active":"2024-04-26T01:42:05.411804+0000","last_became_peered":"2024-04-26T01:42:05.411804+0000","last_unstale":"2024-04-26T01:45:54.961329+0000","last_undegraded":"2024-04-26T01:45:54.961329+0000","last_fullsized":"2024-04-26T01:45:54.961329+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:47.277766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"95'11","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.944497+0000","last_change":"2024-04-26T01:42:05.415563+0000","last_active":"2024-04-26T01:45:54.944497+0000","last_peered":"2024-04-26T01:45:54.944497+0000","last_clean":"2024-04-26T01:45:54.944497+0000","last_became_active":"2024-04-26T01:42:05.414818+0000","last_became_peered":"2024-04-26T01:42:05.414818+0000","last_unstale":"2024-04-26T01:45:54.944497+0000","last_undegraded":"2024-04-26T01:45:54.944497+0000","last_fullsized":"2024-04-26T01:45:54.944497+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:26:54.713595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"95'8","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:45:54.962135+0000","last_change":"2024-04-26T01:42:05.392280+0000","last_active":"2024-04-26T01:45:54.962135+0000","last_peered":"2024-04-26T01:45:54.962135+0000","last_clean":"2024-04-26T01:45:54.962135+0000","last_became_active":"2024-04-26T01:42:05.391943+0000","last_became_peered":"2024-04-26T01:42:05.391943+0000","last_unstale":"2024-04-26T01:45:54.962135+0000","last_undegraded":"2024-04-26T01:45:54.962135+0000","last_fullsized":"2024-04-26T01:45:54.962135+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:42:04.336349+0000","last_clean_scrub_stamp":"2024-04-26T01:42:04.336349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:38:51.560596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":800051,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1817,"num_read_kb":1179,"num_write":486,"num_write_kb":2628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2629632,"data_stored":2400153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":352,"ondisk_log_size":352,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285633,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1077248,"data_stored":519573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547269,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41676,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703476,"statfs":{"total":95995035648,"available":95952359424,"internally_reserved":0,"allocated":2109440,"data_stored":1512934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710796,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1478656,"data_stored":912913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972436,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36772,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708380,"statfs":{"total":95995035648,"available":95957381120,"internally_reserved":0,"allocated":1470464,"data_stored":912721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266775,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1679360,"data_stored":1113703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430300,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":519086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40616,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704536,"statfs":{"total":95995035648,"available":95953444864,"internally_reserved":0,"allocated":1146880,"data_stored":591325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":520357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411444,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1171456,"data_stored":593908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28908,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716244,"statfs":{"total":95995035648,"available":95965433856,"internally_reserved":0,"allocated":1814528,"data_stored":1240845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888168,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":520837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691939,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37432,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707720,"statfs":{"total":95995035648,"available":95956705280,"internally_reserved":0,"allocated":2023424,"data_stored":1440305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":72446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":75029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":721732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":330941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":393842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":403570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:46:09.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:46:09.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:09 smithi006 ceph-mon[28419]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:46:09.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:09 smithi188 ceph-mon[32751]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:46:09.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:09 smithi029 ceph-mon[32775]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:46:09.813 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:10.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:10 smithi006 ceph-mon[28419]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 98 KiB/s wr, 60 op/s 2024-04-26T01:46:10.591 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:10 smithi029 ceph-mon[32775]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 98 KiB/s wr, 60 op/s 2024-04-26T01:46:10.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:10 smithi188 ceph-mon[32751]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 98 KiB/s wr, 60 op/s 2024-04-26T01:46:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.960+0000 7f5a2ae00640 1 -- 172.21.15.6:0/2266129186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a24105fa0 msgr2=0x7f5a2410db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.960+0000 7f5a2ae00640 1 --2- 172.21.15.6:0/2266129186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a24105fa0 0x7f5a2410db70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c009900 tx=0x7f5a1c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.961+0000 7f5a2ae00640 1 -- 172.21.15.6:0/2266129186 shutdown_connections 2024-04-26T01:46:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.961+0000 7f5a2ae00640 1 --2- 172.21.15.6:0/2266129186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a24101880 0x7f5a241058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.961+0000 7f5a2ae00640 1 --2- 172.21.15.6:0/2266129186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a24105fa0 0x7f5a2410db70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.961+0000 7f5a2ae00640 1 --2- 172.21.15.6:0/2266129186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a24100ed0 0x7f5a241012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.961+0000 7f5a2ae00640 1 -- 172.21.15.6:0/2266129186 >> 172.21.15.6:0/2266129186 conn(0x7f5a240fc730 msgr2=0x7f5a240feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.961+0000 7f5a2ae00640 1 -- 172.21.15.6:0/2266129186 shutdown_connections 2024-04-26T01:46:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.961+0000 7f5a2ae00640 1 -- 172.21.15.6:0/2266129186 wait complete. 2024-04-26T01:46:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.962+0000 7f5a2ae00640 1 Processor -- start 2024-04-26T01:46:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.962+0000 7f5a2ae00640 1 -- start start 2024-04-26T01:46:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.962+0000 7f5a2ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a24100ed0 0x7f5a2419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.962+0000 7f5a2ae00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a24105fa0 0x7f5a2419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.962+0000 7f5a2ae00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2419efd0 0x7f5a2419f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.962+0000 7f5a2ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5a24114250 con 0x7f5a24100ed0 2024-04-26T01:46:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.962+0000 7f5a2ae00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5a241140d0 con 0x7f5a2419efd0 2024-04-26T01:46:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.962+0000 7f5a2ae00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5a241143d0 con 0x7f5a24105fa0 2024-04-26T01:46:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.963+0000 7f5a29a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a24100ed0 0x7f5a2419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.963+0000 7f5a29a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a24100ed0 0x7f5a2419a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44272/0 (socket says 172.21.15.6:44272) 2024-04-26T01:46:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.963+0000 7f5a29a00640 1 -- 172.21.15.6:0/394853891 learned_addr learned my addr 172.21.15.6:0/394853891 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.963+0000 7f5a29000640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a24105fa0 0x7f5a2419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.963+0000 7f5a2a400640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2419efd0 0x7f5a2419f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.963+0000 7f5a29a00640 1 -- 172.21.15.6:0/394853891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a24105fa0 msgr2=0x7f5a2419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.963+0000 7f5a29a00640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a24105fa0 0x7f5a2419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.963+0000 7f5a29a00640 1 -- 172.21.15.6:0/394853891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2419efd0 msgr2=0x7f5a2419f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.963+0000 7f5a29a00640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2419efd0 0x7f5a2419f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.963+0000 7f5a29a00640 1 -- 172.21.15.6:0/394853891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a2419fac0 con 0x7f5a24100ed0 2024-04-26T01:46:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.963+0000 7f5a29a00640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a24100ed0 0x7f5a2419a710 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5a1800eb90 tx=0x7f5a1800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:10.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.964+0000 7f5a22000640 1 -- 172.21.15.6:0/394853891 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a18004d20 con 0x7f5a24100ed0 2024-04-26T01:46:10.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.964+0000 7f5a22000640 1 -- 172.21.15.6:0/394853891 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a1800ee90 con 0x7f5a24100ed0 2024-04-26T01:46:10.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.964+0000 7f5a2ae00640 1 -- 172.21.15.6:0/394853891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a2419fdb0 con 0x7f5a24100ed0 2024-04-26T01:46:10.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.964+0000 7f5a22000640 1 -- 172.21.15.6:0/394853891 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a18010670 con 0x7f5a24100ed0 2024-04-26T01:46:10.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.964+0000 7f5a2ae00640 1 -- 172.21.15.6:0/394853891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a241a75a0 con 0x7f5a24100ed0 2024-04-26T01:46:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.968+0000 7f5a2f181740 1 Processor -- start 2024-04-26T01:46:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.968+0000 7f5a22000640 1 -- 172.21.15.6:0/394853891 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f5a18010850 con 0x7f5a24100ed0 2024-04-26T01:46:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.968+0000 7f5a2f181740 1 -- start start 2024-04-26T01:46:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.968+0000 7f5a2f181740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f26009b0 0x5582f2600d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.968+0000 7f5a2f181740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26012d0 0x5582f260b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.968+0000 7f5a2f181740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f260bb90 0x5582f260df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.968+0000 7f5a2f181740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5582f2566900 con 0x5582f26012d0 2024-04-26T01:46:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.968+0000 7f5a2f181740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5582f250cef0 con 0x5582f26009b0 2024-04-26T01:46:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.968+0000 7f5a2f181740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5582f25628c0 con 0x5582f260bb90 2024-04-26T01:46:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.969+0000 7f5a29000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f260bb90 0x5582f260df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.969+0000 7f5a29000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f260bb90 0x5582f260df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36134/0 (socket says 172.21.15.6:36134) 2024-04-26T01:46:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.969+0000 7f5a29a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26012d0 0x5582f260b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.969+0000 7f5a29000640 1 -- 172.21.15.6:0/873233383 learned_addr learned my addr 172.21.15.6:0/873233383 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.969+0000 7f5a2a400640 1 --2- 172.21.15.6:0/873233383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f26009b0 0x5582f2600d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.969+0000 7f5a29000640 1 -- 172.21.15.6:0/873233383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f26009b0 msgr2=0x5582f2600d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.969+0000 7f5a29000640 1 --2- 172.21.15.6:0/873233383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f26009b0 0x5582f2600d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.969+0000 7f5a29000640 1 -- 172.21.15.6:0/873233383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26012d0 msgr2=0x5582f260b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.969+0000 7f5a29000640 1 --2- 172.21.15.6:0/873233383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26012d0 0x5582f260b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.969+0000 7f5a29000640 1 -- 172.21.15.6:0/873233383 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582f2279210 con 0x5582f260bb90 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.969+0000 7f5a2a400640 1 --2- 172.21.15.6:0/873233383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f26009b0 0x5582f2600d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.970+0000 7f5a29000640 1 --2- 172.21.15.6:0/873233383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f260bb90 0x5582f260df80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009c20 tx=0x7f5a10011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=348b30786a0a6587 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.970+0000 7f5a22000640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5a0c0a3ea0 0x7f5a0c0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.970+0000 7f5a16a00640 1 -- 172.21.15.6:0/873233383 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a10004050 con 0x5582f260bb90 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.970+0000 7f5a16a00640 1 -- 172.21.15.6:0/873233383 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a100054d0 con 0x5582f260bb90 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.970+0000 7f5a22000640 1 -- 172.21.15.6:0/394853891 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f5a180d1080 con 0x7f5a24100ed0 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.970+0000 7f5a16a00640 1 -- 172.21.15.6:0/873233383 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a10004d00 con 0x5582f260bb90 2024-04-26T01:46:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.970+0000 7f5a2a400640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5a0c0a3ea0 0x7f5a0c0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.971+0000 7f5a2f181740 1 -- 172.21.15.6:0/873233383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f260bb90 msgr2=0x5582f260df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.971+0000 7f5a2f181740 1 --2- 172.21.15.6:0/873233383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f260bb90 0x5582f260df80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009c20 tx=0x7f5a10011810 comp rx=0 tx=0).stop 2024-04-26T01:46:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.971+0000 7f5a2a400640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5a0c0a3ea0 0x7f5a0c0a6360 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c006010 tx=0x7f5a1c005a40 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.971+0000 7f5a2f181740 1 -- 172.21.15.6:0/873233383 shutdown_connections 2024-04-26T01:46:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.971+0000 7f5a2f181740 1 --2- 172.21.15.6:0/873233383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26012d0 0x5582f260b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.971+0000 7f5a2f181740 1 --2- 172.21.15.6:0/873233383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f260bb90 0x5582f260df80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.971+0000 7f5a2f181740 1 --2- 172.21.15.6:0/873233383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f26009b0 0x5582f2600d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.971+0000 7f5a2f181740 1 -- 172.21.15.6:0/873233383 >> 172.21.15.6:0/873233383 conn(0x5582f25fcec0 msgr2=0x5582f25fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.971+0000 7f5a2f181740 1 -- 172.21.15.6:0/873233383 shutdown_connections 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.971+0000 7f5a2f181740 1 -- 172.21.15.6:0/873233383 wait complete. 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.972+0000 7f5a2f181740 1 -- start start 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.972+0000 7f5a2f181740 1 Processor -- start 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.972+0000 7f5a2f181740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26009b0 0x5582f2608aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.972+0000 7f5a2f181740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f26012d0 0x5582f2608fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.972+0000 7f5a2f181740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f260bb90 0x5582f2609520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.972+0000 7f5a2f181740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5582f260e7c0 con 0x5582f26009b0 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.972+0000 7f5a2f181740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5582f2520310 con 0x5582f26012d0 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.972+0000 7f5a2f181740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5582f260e940 con 0x5582f260bb90 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.973+0000 7f5a2f181740 10 client.0 _ll_register_callbacks cb 0x5582f24eecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.973+0000 7f5a29a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26009b0 0x5582f2608aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.973+0000 7f5a29000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f26012d0 0x5582f2608fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.973+0000 7f5a29a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26009b0 0x5582f2608aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44286/0 (socket says 172.21.15.6:44286) 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.973+0000 7f5a29a00640 1 -- 172.21.15.6:0/4059398361 learned_addr learned my addr 172.21.15.6:0/4059398361 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.973+0000 7f5a29000640 1 -- 172.21.15.6:0/4059398361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f260bb90 msgr2=0x5582f2609520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.973+0000 7f5a29000640 1 --2- 172.21.15.6:0/4059398361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f260bb90 0x5582f2609520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.973+0000 7f5a29000640 1 -- 172.21.15.6:0/4059398361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26009b0 msgr2=0x5582f2608aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.973+0000 7f5a29000640 1 --2- 172.21.15.6:0/4059398361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26009b0 0x5582f2608aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.973+0000 7f5a29000640 1 -- 172.21.15.6:0/4059398361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582f2345620 con 0x5582f26012d0 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.973+0000 7f5a29a00640 1 --2- 172.21.15.6:0/4059398361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26009b0 0x5582f2608aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.974+0000 7f5a29000640 1 --2- 172.21.15.6:0/4059398361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f26012d0 0x5582f2608fe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5a10011d10 tx=0x7f5a10004590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c5f98d97edad04f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.974+0000 7f5a0b400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.974+0000 7f5a0b400640 1 -- 172.21.15.6:0/4059398361 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a10029070 con 0x5582f26012d0 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.974+0000 7f5a0b400640 1 -- 172.21.15.6:0/4059398361 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f5a10003a00 con 0x5582f26012d0 2024-04-26T01:46:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.974+0000 7f5a2f181740 1 -- 172.21.15.6:0/4059398361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a100054d0 con 0x5582f26012d0 2024-04-26T01:46:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.974+0000 7f5a0b400640 1 -- 172.21.15.6:0/4059398361 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5a10035050 con 0x5582f26012d0 2024-04-26T01:46:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.974+0000 7f5a2f181740 10 client.25163 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.974+0000 7f5a2f181740 1 -- 172.21.15.6:0/4059398361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5582f261a9a0 con 0x5582f26012d0 2024-04-26T01:46:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.975+0000 7f5a0b400640 1 -- 172.21.15.6:0/4059398361 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a100054d0 con 0x5582f26012d0 2024-04-26T01:46:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.975+0000 7f5a0b400640 1 -- 172.21.15.6:0/4059398361 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5a10004f30 con 0x5582f26012d0 2024-04-26T01:46:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.975+0000 7f5a2f181740 10 client.25163 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.975+0000 7f5a2f181740 10 client.25163 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.975+0000 7f5a2f181740 20 client.25163 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.975+0000 7f5a2f181740 1 --2- 172.21.15.6:0/4059398361 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5582f261bd50 0x5582f261e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.975+0000 7f5a2f181740 4 client.25163 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.975+0000 7f5a2f181740 1 -- 172.21.15.6:0/4059398361 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582f257a8b0 con 0x5582f261bd50 2024-04-26T01:46:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.976+0000 7f5a29a00640 1 --2- 172.21.15.6:0/4059398361 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5582f261bd50 0x5582f261e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.976+0000 7f5a29a00640 1 --2- 172.21.15.6:0/4059398361 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5582f261bd50 0x5582f261e140 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5bc8a0a0325cc7f server_cookie=72bfb69345d9ff16 in_seq=0 out_seq=0 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.976+0000 7f5a0b400640 10 client.25163 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.977+0000 7f5a0b400640 1 -- 172.21.15.6:0/4059398361 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582f257a8b0 con 0x5582f261bd50 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.977+0000 7f5a0b400640 10 client.25163 handle_command_reply: tid=0 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.977+0000 7f5a2f181740 1 client.25163 shutdown 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.977+0000 7f5a2f181740 1 -- 172.21.15.6:0/4059398361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f26012d0 msgr2=0x5582f2608fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.977+0000 7f5a2f181740 1 --2- 172.21.15.6:0/4059398361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f26012d0 0x5582f2608fe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5a10011d10 tx=0x7f5a10004590 comp rx=0 tx=0).stop 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.977+0000 7f5a2f181740 1 -- 172.21.15.6:0/4059398361 shutdown_connections 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.977+0000 7f5a2f181740 1 --2- 172.21.15.6:0/4059398361 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5582f261bd50 0x5582f261e140 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.977+0000 7f5a2f181740 1 --2- 172.21.15.6:0/4059398361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f260bb90 0x5582f2609520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.977+0000 7f5a2f181740 1 --2- 172.21.15.6:0/4059398361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f26012d0 0x5582f2608fe0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.977+0000 7f5a0b400640 0 client.25163 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.977+0000 7f5a2f181740 1 --2- 172.21.15.6:0/4059398361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26009b0 0x5582f2608aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.977+0000 7f5a2f181740 1 -- 172.21.15.6:0/4059398361 >> 172.21.15.6:0/4059398361 conn(0x5582f25fcec0 msgr2=0x5582f2602c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.978+0000 7f5a2f181740 1 -- 172.21.15.6:0/4059398361 shutdown_connections 2024-04-26T01:46:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.978+0000 7f5a2f181740 1 -- 172.21.15.6:0/4059398361 wait complete. 2024-04-26T01:46:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.978+0000 7f5a2f181740 20 client.25163 trim_cache size 0 max 16384 2024-04-26T01:46:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.995+0000 7f5a2f181740 1 Processor -- start 2024-04-26T01:46:10.993 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.995+0000 7f5a2f181740 1 -- start start 2024-04-26T01:46:10.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.995+0000 7f5a2f181740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26009b0 0x5582f2617680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.995+0000 7f5a2f181740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26012d0 0x5582f2609170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.995+0000 7f5a2f181740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f260bb90 0x5582f2623910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.995+0000 7f5a2f181740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5a241b2500 con 0x5582f26012d0 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.995+0000 7f5a2f181740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5a241afdd0 con 0x5582f260bb90 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.995+0000 7f5a2f181740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5a241adc20 con 0x5582f26009b0 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.996+0000 7f5a29a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26009b0 0x5582f2617680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.996+0000 7f5a29a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26009b0 0x5582f2617680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36166/0 (socket says 172.21.15.6:36166) 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.996+0000 7f5a29a00640 1 -- 172.21.15.6:0/939455867 learned_addr learned my addr 172.21.15.6:0/939455867 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.996+0000 7f5a29000640 1 --2- 172.21.15.6:0/939455867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26012d0 0x5582f2609170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.996+0000 7f5a29a00640 1 -- 172.21.15.6:0/939455867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f260bb90 msgr2=0x5582f2623910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.996+0000 7f5a29a00640 1 --2- 172.21.15.6:0/939455867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f260bb90 0x5582f2623910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.996+0000 7f5a29a00640 1 -- 172.21.15.6:0/939455867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26012d0 msgr2=0x5582f2609170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.996+0000 7f5a29a00640 1 --2- 172.21.15.6:0/939455867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26012d0 0x5582f2609170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.996+0000 7f5a29a00640 1 -- 172.21.15.6:0/939455867 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582f260e340 con 0x5582f26009b0 2024-04-26T01:46:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.996+0000 7f5a29a00640 1 --2- 172.21.15.6:0/939455867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26009b0 0x5582f2617680 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5a18099780 tx=0x7f5a180bd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74fac097a10af287 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.997+0000 7f5a16000640 1 -- 172.21.15.6:0/939455867 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a1809c8f0 con 0x5582f26009b0 2024-04-26T01:46:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.997+0000 7f5a16000640 1 -- 172.21.15.6:0/939455867 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a180a0030 con 0x5582f26009b0 2024-04-26T01:46:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.997+0000 7f5a16000640 1 -- 172.21.15.6:0/939455867 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a1809c710 con 0x5582f26009b0 2024-04-26T01:46:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.997+0000 7f5a2f181740 1 -- 172.21.15.6:0/939455867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26009b0 msgr2=0x5582f2617680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.997+0000 7f5a2f181740 1 --2- 172.21.15.6:0/939455867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26009b0 0x5582f2617680 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5a18099780 tx=0x7f5a180bd040 comp rx=0 tx=0).stop 2024-04-26T01:46:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.997+0000 7f5a2f181740 1 -- 172.21.15.6:0/939455867 shutdown_connections 2024-04-26T01:46:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.997+0000 7f5a2f181740 1 --2- 172.21.15.6:0/939455867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26012d0 0x5582f2609170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.997+0000 7f5a2f181740 1 --2- 172.21.15.6:0/939455867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f260bb90 0x5582f2623910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.997+0000 7f5a2f181740 1 --2- 172.21.15.6:0/939455867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26009b0 0x5582f2617680 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.997+0000 7f5a2f181740 1 -- 172.21.15.6:0/939455867 >> 172.21.15.6:0/939455867 conn(0x5582f25fcec0 msgr2=0x5582f2608a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.998+0000 7f5a2f181740 1 -- 172.21.15.6:0/939455867 shutdown_connections 2024-04-26T01:46:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.998+0000 7f5a2f181740 1 -- 172.21.15.6:0/939455867 wait complete. 2024-04-26T01:46:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.998+0000 7f5a2f181740 1 -- start start 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.998+0000 7f5a2f181740 1 Processor -- start 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a2f181740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26009b0 0x5582f2611510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a2f181740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26012d0 0x5582f2611a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a2f181740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f260bb90 0x5582f2611f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a2f181740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5a241ae290 con 0x5582f26009b0 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a2f181740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5a241af700 con 0x5582f260bb90 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a2f181740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5a241ae960 con 0x5582f26012d0 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a2f181740 10 client.0 _ll_register_callbacks cb 0x7f5a10004450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a29000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26012d0 0x5582f2611a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a2a400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f260bb90 0x5582f2611f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a29000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26012d0 0x5582f2611a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36168/0 (socket says 172.21.15.6:36168) 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a29000640 1 -- 172.21.15.6:0/1157316204 learned_addr learned my addr 172.21.15.6:0/1157316204 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a29a00640 1 --2- 172.21.15.6:0/1157316204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26009b0 0x5582f2611510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a29000640 1 -- 172.21.15.6:0/1157316204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f260bb90 msgr2=0x5582f2611f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a29000640 1 --2- 172.21.15.6:0/1157316204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f260bb90 0x5582f2611f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a29000640 1 -- 172.21.15.6:0/1157316204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26009b0 msgr2=0x5582f2611510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a29000640 1 --2- 172.21.15.6:0/1157316204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26009b0 0x5582f2611510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:10.999+0000 7f5a29000640 1 -- 172.21.15.6:0/1157316204 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582f2681190 con 0x5582f26012d0 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.000+0000 7f5a29000640 1 --2- 172.21.15.6:0/1157316204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26012d0 0x5582f2611a50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x5582f2608300 tx=0x7f5a10005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ab856edcdc6b0ec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.000+0000 7f5a0b400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.000+0000 7f5a0b400640 1 -- 172.21.15.6:0/1157316204 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a10029040 con 0x5582f26012d0 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.000+0000 7f5a2f181740 1 -- 172.21.15.6:0/1157316204 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a180a0030 con 0x5582f26012d0 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.000+0000 7f5a0b400640 1 -- 172.21.15.6:0/1157316204 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f5a10003800 con 0x5582f26012d0 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.000+0000 7f5a0b400640 1 -- 172.21.15.6:0/1157316204 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5a10041050 con 0x5582f26012d0 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.000+0000 7f5a2f181740 10 client.25201 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.000+0000 7f5a2f181740 1 -- 172.21.15.6:0/1157316204 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5582f2682470 con 0x5582f26012d0 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.001+0000 7f5a0b400640 1 -- 172.21.15.6:0/1157316204 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a180a0030 con 0x5582f26012d0 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.001+0000 7f5a0b400640 1 -- 172.21.15.6:0/1157316204 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5a10023cb0 con 0x5582f26012d0 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.001+0000 7f5a2f181740 10 client.25201 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.001+0000 7f5a2f181740 10 client.25201 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.001+0000 7f5a2f181740 20 client.25201 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.001+0000 7f5a2f181740 1 --2- 172.21.15.6:0/1157316204 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5582f2682c20 0x5582f2685040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.001+0000 7f5a2f181740 4 client.25201 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.001+0000 7f5a2f181740 1 -- 172.21.15.6:0/1157316204 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5582f2574320 con 0x5582f2682c20 2024-04-26T01:46:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.002+0000 7f5a29a00640 1 --2- 172.21.15.6:0/1157316204 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5582f2682c20 0x5582f2685040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.002+0000 7f5a29a00640 1 --2- 172.21.15.6:0/1157316204 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5582f2682c20 0x5582f2685040 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e122ba1dd94a9e18 server_cookie=2f1cff20f4f0562a in_seq=0 out_seq=0 2024-04-26T01:46:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.002+0000 7f5a0b400640 10 client.25201 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:11.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a0b400640 1 -- 172.21.15.6:0/1157316204 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5582f2574320 con 0x5582f2682c20 2024-04-26T01:46:11.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a0b400640 10 client.25201 handle_command_reply: tid=0 2024-04-26T01:46:11.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a2f181740 1 client.25201 shutdown 2024-04-26T01:46:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a2f181740 1 -- 172.21.15.6:0/1157316204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26012d0 msgr2=0x5582f2611a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a2f181740 1 --2- 172.21.15.6:0/1157316204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26012d0 0x5582f2611a50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x5582f2608300 tx=0x7f5a10005f10 comp rx=0 tx=0).stop 2024-04-26T01:46:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a2f181740 1 -- 172.21.15.6:0/1157316204 shutdown_connections 2024-04-26T01:46:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a2f181740 1 --2- 172.21.15.6:0/1157316204 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5582f2682c20 0x5582f2685040 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a2f181740 1 --2- 172.21.15.6:0/1157316204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5582f260bb90 0x5582f2611f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a2f181740 1 --2- 172.21.15.6:0/1157316204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582f26012d0 0x5582f2611a50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a0b400640 0 client.25201 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a2f181740 1 --2- 172.21.15.6:0/1157316204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5582f26009b0 0x5582f2611510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a2f181740 1 -- 172.21.15.6:0/1157316204 >> 172.21.15.6:0/1157316204 conn(0x5582f25fcec0 msgr2=0x5582f26092d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a2f181740 1 -- 172.21.15.6:0/1157316204 shutdown_connections 2024-04-26T01:46:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.016+0000 7f5a2f181740 1 -- 172.21.15.6:0/1157316204 wait complete. 2024-04-26T01:46:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.017+0000 7f5a2f181740 20 client.25201 trim_cache size 0 max 16384 2024-04-26T01:46:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:46:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:46:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:46:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:46:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.017+0000 7f5a2ae00640 1 -- 172.21.15.6:0/394853891 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5a0c0a3ea0 msgr2=0x7f5a0c0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.017+0000 7f5a2ae00640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5a0c0a3ea0 0x7f5a0c0a6360 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c006010 tx=0x7f5a1c005a40 comp rx=0 tx=0).stop 2024-04-26T01:46:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.017+0000 7f5a2ae00640 1 -- 172.21.15.6:0/394853891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a24100ed0 msgr2=0x7f5a2419a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.017+0000 7f5a2ae00640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a24100ed0 0x7f5a2419a710 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5a1800eb90 tx=0x7f5a1800c6a0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.018+0000 7f5a2ae00640 1 -- 172.21.15.6:0/394853891 shutdown_connections 2024-04-26T01:46:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.018+0000 7f5a2ae00640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5a0c0a3ea0 0x7f5a0c0a6360 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.018+0000 7f5a2ae00640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2419efd0 0x7f5a2419f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.018+0000 7f5a2ae00640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a24105fa0 0x7f5a2419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.018+0000 7f5a2ae00640 1 --2- 172.21.15.6:0/394853891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a24100ed0 0x7f5a2419a710 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.018+0000 7f5a2ae00640 1 -- 172.21.15.6:0/394853891 >> 172.21.15.6:0/394853891 conn(0x7f5a240fc730 msgr2=0x7f5a240fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.018+0000 7f5a2ae00640 1 -- 172.21.15.6:0/394853891 shutdown_connections 2024-04-26T01:46:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:11.018+0000 7f5a2ae00640 1 -- 172.21.15.6:0/394853891 wait complete. 2024-04-26T01:46:11.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.819+0000 7fab2b600700 1 -- 172.21.15.6:0/3854572714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab24108a30 msgr2=0x7fab24108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:11.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.819+0000 7fab2b600700 1 --2- 172.21.15.6:0/3854572714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab24108a30 0x7fab24108e10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fab18009990 tx=0x7fab18009ca0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.819+0000 7fab2b600700 1 -- 172.21.15.6:0/3854572714 shutdown_connections 2024-04-26T01:46:11.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.819+0000 7fab2b600700 1 --2- 172.21.15.6:0/3854572714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab2406b0f0 0x7fab2410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.820+0000 7fab2b600700 1 --2- 172.21.15.6:0/3854572714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab24108a30 0x7fab24108e10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.820+0000 7fab2b600700 1 --2- 172.21.15.6:0/3854572714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab2406a750 0x7fab2406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.820+0000 7fab2b600700 1 -- 172.21.15.6:0/3854572714 >> 172.21.15.6:0/3854572714 conn(0x7fab240788f0 msgr2=0x7fab24078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:11.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.820+0000 7fab2b600700 1 -- 172.21.15.6:0/3854572714 shutdown_connections 2024-04-26T01:46:11.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.820+0000 7fab2b600700 1 -- 172.21.15.6:0/3854572714 wait complete. 2024-04-26T01:46:11.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.821+0000 7fab2b600700 1 Processor -- start 2024-04-26T01:46:11.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.821+0000 7fab2b600700 1 -- start start 2024-04-26T01:46:11.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.821+0000 7fab2b600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab2406a750 0x7fab24101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:11.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.821+0000 7fab2b600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2406b0f0 0x7fab24101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:11.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.821+0000 7fab2b600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab24108a30 0x7fab24102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:11.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.821+0000 7fab2b600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fab24112c00 con 0x7fab24108a30 2024-04-26T01:46:11.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.821+0000 7fab2b600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fab24112a80 con 0x7fab2406b0f0 2024-04-26T01:46:11.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.821+0000 7fab2b600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fab24112d80 con 0x7fab2406a750 2024-04-26T01:46:11.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.822+0000 7fab23e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2406b0f0 0x7fab24101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:11.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.822+0000 7fab23e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2406b0f0 0x7fab24101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:51260/0 (socket says 172.21.15.6:51260) 2024-04-26T01:46:11.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.822+0000 7fab23e00700 1 -- 172.21.15.6:0/3047042643 learned_addr learned my addr 172.21.15.6:0/3047042643 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:11.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.822+0000 7fab29600700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab24108a30 0x7fab24102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:11.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.822+0000 7fab28c00700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab2406a750 0x7fab24101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:11.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.822+0000 7fab23e00700 1 -- 172.21.15.6:0/3047042643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab2406a750 msgr2=0x7fab24101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:11.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.822+0000 7fab23e00700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab2406a750 0x7fab24101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.822+0000 7fab23e00700 1 -- 172.21.15.6:0/3047042643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab24108a30 msgr2=0x7fab24102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:11.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.822+0000 7fab23e00700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab24108a30 0x7fab24102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:11.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.822+0000 7fab23e00700 1 -- 172.21.15.6:0/3047042643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab241a76e0 con 0x7fab2406b0f0 2024-04-26T01:46:11.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.823+0000 7fab23e00700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2406b0f0 0x7fab24101f50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fab1000e9a0 tx=0x7fab1000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.823+0000 7fab21600700 1 -- 172.21.15.6:0/3047042643 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1000ca10 con 0x7fab2406b0f0 2024-04-26T01:46:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.823+0000 7fab2b600700 1 -- 172.21.15.6:0/3047042643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab241a79d0 con 0x7fab2406b0f0 2024-04-26T01:46:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.823+0000 7fab21600700 1 -- 172.21.15.6:0/3047042643 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab1000cbb0 con 0x7fab2406b0f0 2024-04-26T01:46:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.823+0000 7fab21600700 1 -- 172.21.15.6:0/3047042643 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab10018680 con 0x7fab2406b0f0 2024-04-26T01:46:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.823+0000 7fab2b600700 1 -- 172.21.15.6:0/3047042643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab241a7f10 con 0x7fab2406b0f0 2024-04-26T01:46:11.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.824+0000 7fab2b600700 1 -- 172.21.15.6:0/3047042643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab24069730 con 0x7fab2406b0f0 2024-04-26T01:46:11.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.826+0000 7fab21600700 1 -- 172.21.15.6:0/3047042643 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7fab10010430 con 0x7fab2406b0f0 2024-04-26T01:46:11.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.827+0000 7fab21600700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fab0c0a3ed0 0x7fab0c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:11.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.828+0000 7fab21600700 1 -- 172.21.15.6:0/3047042643 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7fab100d0d80 con 0x7fab2406b0f0 2024-04-26T01:46:11.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.828+0000 7fab28c00700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fab0c0a3ed0 0x7fab0c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:11.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.829+0000 7fab28c00700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fab0c0a3ed0 0x7fab0c0a6390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fab18006010 tx=0x7fab1800b260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:11.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:11.830+0000 7fab21600700 1 -- 172.21.15.6:0/3047042643 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fab1009d150 con 0x7fab2406b0f0 2024-04-26T01:46:12.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.114+0000 7fab2b600700 1 -- 172.21.15.6:0/3047042643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab2402e140 con 0x7fab2406b0f0 2024-04-26T01:46:12.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.115+0000 7fab21600700 1 -- 172.21.15.6:0/3047042643 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11850 (secure 0 0 0) 0x7fab1009ca50 con 0x7fab2406b0f0 2024-04-26T01:46:12.114 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:46:12.114 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:45:54.644030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24512,24505]},"id":1}]} 2024-04-26T01:46:12.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.118+0000 7fab2b600700 1 -- 172.21.15.6:0/3047042643 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fab0c0a3ed0 msgr2=0x7fab0c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:12.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.118+0000 7fab2b600700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fab0c0a3ed0 0x7fab0c0a6390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fab18006010 tx=0x7fab1800b260 comp rx=0 tx=0).stop 2024-04-26T01:46:12.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.118+0000 7fab2b600700 1 -- 172.21.15.6:0/3047042643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2406b0f0 msgr2=0x7fab24101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:12.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.118+0000 7fab2b600700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2406b0f0 0x7fab24101f50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fab1000e9a0 tx=0x7fab1000ecb0 comp rx=0 tx=0).stop 2024-04-26T01:46:12.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.118+0000 7fab2b600700 1 -- 172.21.15.6:0/3047042643 shutdown_connections 2024-04-26T01:46:12.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.118+0000 7fab2b600700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fab0c0a3ed0 0x7fab0c0a6390 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:12.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.118+0000 7fab2b600700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab24108a30 0x7fab24102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:12.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.118+0000 7fab2b600700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab2406a750 0x7fab24101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:12.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.118+0000 7fab2b600700 1 --2- 172.21.15.6:0/3047042643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2406b0f0 0x7fab24101f50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:12.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.118+0000 7fab2b600700 1 -- 172.21.15.6:0/3047042643 >> 172.21.15.6:0/3047042643 conn(0x7fab240788f0 msgr2=0x7fab240ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:12.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.119+0000 7fab2b600700 1 -- 172.21.15.6:0/3047042643 shutdown_connections 2024-04-26T01:46:12.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:12.119+0000 7fab2b600700 1 -- 172.21.15.6:0/3047042643 wait complete. 2024-04-26T01:46:12.119 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-04-26T01:46:12.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:12 smithi006 ceph-mon[28419]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 98 KiB/s wr, 61 op/s 2024-04-26T01:46:12.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:12 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3047042643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:12.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:12 smithi188 ceph-mon[32751]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 98 KiB/s wr, 61 op/s 2024-04-26T01:46:12.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:12 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3047042643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:12.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:12 smithi029 ceph-mon[32775]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 98 KiB/s wr, 61 op/s 2024-04-26T01:46:12.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:12 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3047042643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:12.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:46:13.418 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:14.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:14 smithi006 ceph-mon[28419]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 49 KiB/s wr, 47 op/s 2024-04-26T01:46:14.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:14 smithi188 ceph-mon[32751]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 49 KiB/s wr, 47 op/s 2024-04-26T01:46:14.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:14 smithi029 ceph-mon[32775]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 49 KiB/s wr, 47 op/s 2024-04-26T01:46:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.630+0000 7f84a2000700 1 -- 172.21.15.6:0/4014899977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f849c1092d0 msgr2=0x7f849c1096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.630+0000 7f84a2000700 1 --2- 172.21.15.6:0/4014899977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f849c1092d0 0x7f849c1096b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f848c009920 tx=0x7f848c009c30 comp rx=0 tx=0).stop 2024-04-26T01:46:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.630+0000 7f84a2000700 1 -- 172.21.15.6:0/4014899977 shutdown_connections 2024-04-26T01:46:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.630+0000 7f84a2000700 1 --2- 172.21.15.6:0/4014899977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f849c10e3b0 0x7f849c115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.630+0000 7f84a2000700 1 --2- 172.21.15.6:0/4014899977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f849c1092d0 0x7f849c1096b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.630+0000 7f84a2000700 1 --2- 172.21.15.6:0/4014899977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f849c109c80 0x7f849c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.630+0000 7f84a2000700 1 -- 172.21.15.6:0/4014899977 >> 172.21.15.6:0/4014899977 conn(0x7f849c076210 msgr2=0x7f849c107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.630+0000 7f84a2000700 1 -- 172.21.15.6:0/4014899977 shutdown_connections 2024-04-26T01:46:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.630+0000 7f84a2000700 1 -- 172.21.15.6:0/4014899977 wait complete. 2024-04-26T01:46:15.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.631+0000 7f84a2000700 1 Processor -- start 2024-04-26T01:46:15.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.631+0000 7f84a2000700 1 -- start start 2024-04-26T01:46:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.632+0000 7f84a2000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f849c1092d0 0x7f849c1a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.632+0000 7f84a2000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f849c109c80 0x7f849c1a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.632+0000 7f84a2000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f849c10e3b0 0x7f849c19f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.632+0000 7f84a2000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f849c119290 con 0x7f849c10e3b0 2024-04-26T01:46:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.632+0000 7f84a2000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f849c119110 con 0x7f849c109c80 2024-04-26T01:46:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.632+0000 7f84a2000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f849c119410 con 0x7f849c1092d0 2024-04-26T01:46:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.632+0000 7f849aa00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f849c109c80 0x7f849c1a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.632+0000 7f849aa00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f849c109c80 0x7f849c1a5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:51268/0 (socket says 172.21.15.6:51268) 2024-04-26T01:46:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.632+0000 7f849aa00700 1 -- 172.21.15.6:0/3462240450 learned_addr learned my addr 172.21.15.6:0/3462240450 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.632+0000 7f849b400700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f849c1092d0 0x7f849c1a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.632+0000 7f849be00700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f849c10e3b0 0x7f849c19f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.633+0000 7f849aa00700 1 -- 172.21.15.6:0/3462240450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f849c1092d0 msgr2=0x7f849c1a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:15.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.633+0000 7f849aa00700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f849c1092d0 0x7f849c1a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:15.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.633+0000 7f849aa00700 1 -- 172.21.15.6:0/3462240450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f849c10e3b0 msgr2=0x7f849c19f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:15.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.633+0000 7f849aa00700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f849c10e3b0 0x7f849c19f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:15.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.633+0000 7f849aa00700 1 -- 172.21.15.6:0/3462240450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849c19ff30 con 0x7f849c109c80 2024-04-26T01:46:15.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.633+0000 7f849aa00700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f849c109c80 0x7f849c1a5ac0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f848800ea30 tx=0x7f848800ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:15.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.633+0000 7f8493e00700 1 -- 172.21.15.6:0/3462240450 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848800cbd0 con 0x7f849c109c80 2024-04-26T01:46:15.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.633+0000 7f84a2000700 1 -- 172.21.15.6:0/3462240450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f849c1a0220 con 0x7f849c109c80 2024-04-26T01:46:15.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.634+0000 7f84a2000700 1 -- 172.21.15.6:0/3462240450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f849c1ac500 con 0x7f849c109c80 2024-04-26T01:46:15.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.634+0000 7f8493e00700 1 -- 172.21.15.6:0/3462240450 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84880044d0 con 0x7f849c109c80 2024-04-26T01:46:15.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.634+0000 7f8493e00700 1 -- 172.21.15.6:0/3462240450 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488003dc0 con 0x7f849c109c80 2024-04-26T01:46:15.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.635+0000 7f84a2000700 1 -- 172.21.15.6:0/3462240450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f849c10a7b0 con 0x7f849c109c80 2024-04-26T01:46:15.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.637+0000 7f8493e00700 1 -- 172.21.15.6:0/3462240450 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f8488004020 con 0x7f849c109c80 2024-04-26T01:46:15.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.638+0000 7f8493e00700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f84800a3ed0 0x7f84800a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:15.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.639+0000 7f8493e00700 1 -- 172.21.15.6:0/3462240450 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f84880d0c60 con 0x7f849c109c80 2024-04-26T01:46:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.639+0000 7f849b400700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f84800a3ed0 0x7f84800a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.639+0000 7f849b400700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f84800a3ed0 0x7f84800a6390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f848c009680 tx=0x7f848c019040 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:15.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.643+0000 7f8493e00700 1 -- 172.21.15.6:0/3462240450 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8488016330 con 0x7f849c109c80 2024-04-26T01:46:15.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.925+0000 7f84a2000700 1 -- 172.21.15.6:0/3462240450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f849c1a0b80 con 0x7f849c109c80 2024-04-26T01:46:15.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.927+0000 7f8493e00700 1 -- 172.21.15.6:0/3462240450 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11850 (secure 0 0 0) 0x7f848809d030 con 0x7f849c109c80 2024-04-26T01:46:15.925 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:46:15.926 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:45:54.644030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24512,24505]},"id":1}]} 2024-04-26T01:46:15.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.930+0000 7f84a2000700 1 -- 172.21.15.6:0/3462240450 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f84800a3ed0 msgr2=0x7f84800a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:15.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.930+0000 7f84a2000700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f84800a3ed0 0x7f84800a6390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f848c009680 tx=0x7f848c019040 comp rx=0 tx=0).stop 2024-04-26T01:46:15.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.930+0000 7f84a2000700 1 -- 172.21.15.6:0/3462240450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f849c109c80 msgr2=0x7f849c1a5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:15.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.930+0000 7f84a2000700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f849c109c80 0x7f849c1a5ac0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f848800ea30 tx=0x7f848800ed40 comp rx=0 tx=0).stop 2024-04-26T01:46:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.930+0000 7f84a2000700 1 -- 172.21.15.6:0/3462240450 shutdown_connections 2024-04-26T01:46:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.930+0000 7f84a2000700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f84800a3ed0 0x7f84800a6390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.930+0000 7f84a2000700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f849c10e3b0 0x7f849c19f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.930+0000 7f84a2000700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f849c1092d0 0x7f849c1a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.930+0000 7f84a2000700 1 --2- 172.21.15.6:0/3462240450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f849c109c80 0x7f849c1a5ac0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.930+0000 7f84a2000700 1 -- 172.21.15.6:0/3462240450 >> 172.21.15.6:0/3462240450 conn(0x7f849c076210 msgr2=0x7f849c1086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.930+0000 7f84a2000700 1 -- 172.21.15.6:0/3462240450 shutdown_connections 2024-04-26T01:46:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:15.930+0000 7f84a2000700 1 -- 172.21.15.6:0/3462240450 wait complete. 2024-04-26T01:46:15.930 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-04-26T01:46:16.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:16 smithi188 ceph-mon[32751]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:46:16.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:16 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3462240450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:16.658 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:16 smithi006 ceph-mon[28419]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:46:16.658 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:16 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3462240450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:16.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:16 smithi029 ceph-mon[32775]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:46:16.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:16 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3462240450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:16.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:46:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:18.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:18 smithi006 ceph-mon[28419]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-26T01:46:18.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:18 smithi188 ceph-mon[32751]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-26T01:46:18.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:18 smithi029 ceph-mon[32775]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-26T01:46:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.572+0000 7f2b94a00700 1 -- 172.21.15.6:0/1928560974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b9006afe0 msgr2=0x7f2b9006b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.572+0000 7f2b94a00700 1 --2- 172.21.15.6:0/1928560974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b9006afe0 0x7f2b9006b3c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009990 tx=0x7f2b80009ca0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.573+0000 7f2b94a00700 1 -- 172.21.15.6:0/1928560974 shutdown_connections 2024-04-26T01:46:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.573+0000 7f2b94a00700 1 --2- 172.21.15.6:0/1928560974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b9010f4e0 0x7f2b901118d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.573+0000 7f2b94a00700 1 --2- 172.21.15.6:0/1928560974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b9006afe0 0x7f2b9006b3c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.573+0000 7f2b94a00700 1 --2- 172.21.15.6:0/1928560974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9006b900 0x7f2b9010efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.573+0000 7f2b94a00700 1 -- 172.21.15.6:0/1928560974 >> 172.21.15.6:0/1928560974 conn(0x7f2b900788b0 msgr2=0x7f2b90078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:19.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.573+0000 7f2b94a00700 1 -- 172.21.15.6:0/1928560974 shutdown_connections 2024-04-26T01:46:19.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.573+0000 7f2b94a00700 1 -- 172.21.15.6:0/1928560974 wait complete. 2024-04-26T01:46:19.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.574+0000 7f2b94a00700 1 Processor -- start 2024-04-26T01:46:19.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.574+0000 7f2b94a00700 1 -- start start 2024-04-26T01:46:19.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.575+0000 7f2b94a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b9006afe0 0x7f2b9019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.575+0000 7f2b94a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b9006b900 0x7f2b9019b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.575+0000 7f2b94a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9010f4e0 0x7f2b9019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.575+0000 7f2b94a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2b90114df0 con 0x7f2b9006b900 2024-04-26T01:46:19.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.575+0000 7f2b94a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2b90114c70 con 0x7f2b9010f4e0 2024-04-26T01:46:19.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.575+0000 7f2b94a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2b90114f70 con 0x7f2b9006afe0 2024-04-26T01:46:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.575+0000 7f2b85400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b9006b900 0x7f2b9019b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.575+0000 7f2b8de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b9006afe0 0x7f2b9019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.575+0000 7f2b85400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b9006b900 0x7f2b9019b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47754/0 (socket says 172.21.15.6:47754) 2024-04-26T01:46:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.575+0000 7f2b85400700 1 -- 172.21.15.6:0/3384399619 learned_addr learned my addr 172.21.15.6:0/3384399619 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.575+0000 7f2b8e800700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9010f4e0 0x7f2b9019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.576+0000 7f2b8de00700 1 -- 172.21.15.6:0/3384399619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9010f4e0 msgr2=0x7f2b9019fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.576+0000 7f2b8de00700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9010f4e0 0x7f2b9019fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.576+0000 7f2b8de00700 1 -- 172.21.15.6:0/3384399619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b9006b900 msgr2=0x7f2b9019b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.576+0000 7f2b8de00700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b9006b900 0x7f2b9019b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.576+0000 7f2b8de00700 1 -- 172.21.15.6:0/3384399619 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b901a0310 con 0x7f2b9006afe0 2024-04-26T01:46:19.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.576+0000 7f2b85400700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b9006b900 0x7f2b9019b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:19.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.576+0000 7f2b8de00700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b9006afe0 0x7f2b9019b300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2b80006010 tx=0x7f2b800057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:19.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.576+0000 7f2b87400700 1 -- 172.21.15.6:0/3384399619 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80027070 con 0x7f2b9006afe0 2024-04-26T01:46:19.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.576+0000 7f2b94a00700 1 -- 172.21.15.6:0/3384399619 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b901a05a0 con 0x7f2b9006afe0 2024-04-26T01:46:19.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.576+0000 7f2b87400700 1 -- 172.21.15.6:0/3384399619 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b80022470 con 0x7f2b9006afe0 2024-04-26T01:46:19.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.576+0000 7f2b87400700 1 -- 172.21.15.6:0/3384399619 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80017650 con 0x7f2b9006afe0 2024-04-26T01:46:19.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.577+0000 7f2b94a00700 1 -- 172.21.15.6:0/3384399619 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b901a7e80 con 0x7f2b9006afe0 2024-04-26T01:46:19.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.578+0000 7f2b94a00700 1 -- 172.21.15.6:0/3384399619 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b90069730 con 0x7f2b9006afe0 2024-04-26T01:46:19.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.580+0000 7f2b87400700 1 -- 172.21.15.6:0/3384399619 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f2b8002a070 con 0x7f2b9006afe0 2024-04-26T01:46:19.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.581+0000 7f2b87400700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2b740a3ed0 0x7f2b740a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:19.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.581+0000 7f2b87400700 1 -- 172.21.15.6:0/3384399619 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f2b800d1770 con 0x7f2b9006afe0 2024-04-26T01:46:19.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.581+0000 7f2b85400700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2b740a3ed0 0x7f2b740a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:19.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.582+0000 7f2b85400700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2b740a3ed0 0x7f2b740a6390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2b90102ef0 tx=0x7f2b7800b040 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:19.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.583+0000 7f2b87400700 1 -- 172.21.15.6:0/3384399619 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b80013070 con 0x7f2b9006afe0 2024-04-26T01:46:19.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.816+0000 7f2b94a00700 1 -- 172.21.15.6:0/3384399619 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b9002df80 con 0x7f2b9006afe0 2024-04-26T01:46:19.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.818+0000 7f2b87400700 1 -- 172.21.15.6:0/3384399619 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+23052 (secure 0 0 0) 0x7f2b8009dcf0 con 0x7f2b9006afe0 2024-04-26T01:46:19.817 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:46:19.817 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":96,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:32:29.071544+0000","modified":"2024-04-26T01:45:54.919795+0000","last_up_change":"2024-04-26T01:40:49.915924+0000","last_in_change":"2024-04-26T01:40:29.559647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:36:35.182726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:41:59.612167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:42:04.106467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f164c51d-ce5d-4140-b9e3-a4b84088f5ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6803","nonce":76518486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6805","nonce":76518486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6809","nonce":76518486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6807","nonce":76518486}]},"public_addr":"172.21.15.6:6803/76518486","cluster_addr":"172.21.15.6:6805/76518486","heartbeat_back_addr":"172.21.15.6:6809/76518486","heartbeat_front_addr":"172.21.15.6:6807/76518486","state":["exists","up"]},{"osd":1,"uuid":"57e875dd-ddf1-4ee4-a2a7-9589d7f42818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6801","nonce":400769116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6803","nonce":400769116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6807","nonce":400769116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6805","nonce":400769116}]},"public_addr":"172.21.15.29:6801/400769116","cluster_addr":"172.21.15.29:6803/400769116","heartbeat_back_addr":"172.21.15.29:6807/400769116","heartbeat_front_addr":"172.21.15.29:6805/400769116","state":["exists","up"]},{"osd":2,"uuid":"77e0c294-c48a-4318-938b-4604a533a2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6801","nonce":3508067672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6803","nonce":3508067672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6807","nonce":3508067672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6805","nonce":3508067672}]},"public_addr":"172.21.15.188:6801/3508067672","cluster_addr":"172.21.15.188:6803/3508067672","heartbeat_back_addr":"172.21.15.188:6807/3508067672","heartbeat_front_addr":"172.21.15.188:6805/3508067672","state":["exists","up"]},{"osd":3,"uuid":"6393dac5-996c-47af-9687-0f1393ecc483","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6811","nonce":1987311118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6813","nonce":1987311118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6817","nonce":1987311118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6815","nonce":1987311118}]},"public_addr":"172.21.15.6:6811/1987311118","cluster_addr":"172.21.15.6:6813/1987311118","heartbeat_back_addr":"172.21.15.6:6817/1987311118","heartbeat_front_addr":"172.21.15.6:6815/1987311118","state":["exists","up"]},{"osd":4,"uuid":"f2c1a535-26a0-46c6-a6d0-c2529c786739","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6809","nonce":2125876609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6811","nonce":2125876609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6815","nonce":2125876609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6813","nonce":2125876609}]},"public_addr":"172.21.15.29:6809/2125876609","cluster_addr":"172.21.15.29:6811/2125876609","heartbeat_back_addr":"172.21.15.29:6815/2125876609","heartbeat_front_addr":"172.21.15.29:6813/2125876609","state":["exists","up"]},{"osd":5,"uuid":"9788bb0a-0d79-4743-bce6-80f5342e8f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6809","nonce":2092077007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6811","nonce":2092077007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6815","nonce":2092077007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6813","nonce":2092077007}]},"public_addr":"172.21.15.188:6809/2092077007","cluster_addr":"172.21.15.188:6811/2092077007","heartbeat_back_addr":"172.21.15.188:6815/2092077007","heartbeat_front_addr":"172.21.15.188:6813/2092077007","state":["exists","up"]},{"osd":6,"uuid":"fd8584e1-69ee-4c30-85e4-a637357d1b6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6819","nonce":3256016695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6821","nonce":3256016695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6825","nonce":3256016695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6823","nonce":3256016695}]},"public_addr":"172.21.15.6:6819/3256016695","cluster_addr":"172.21.15.6:6821/3256016695","heartbeat_back_addr":"172.21.15.6:6825/3256016695","heartbeat_front_addr":"172.21.15.6:6823/3256016695","state":["exists","up"]},{"osd":7,"uuid":"6b7e6784-d574-42da-8416-a74a3b6b3755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6817","nonce":3478594037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6819","nonce":3478594037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6823","nonce":3478594037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6821","nonce":3478594037}]},"public_addr":"172.21.15.29:6817/3478594037","cluster_addr":"172.21.15.29:6819/3478594037","heartbeat_back_addr":"172.21.15.29:6823/3478594037","heartbeat_front_addr":"172.21.15.29:6821/3478594037","state":["exists","up"]},{"osd":8,"uuid":"30cd2ea1-2767-4062-8bd9-47214e4b4b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6817","nonce":3290364376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6819","nonce":3290364376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6823","nonce":3290364376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6821","nonce":3290364376}]},"public_addr":"172.21.15.188:6817/3290364376","cluster_addr":"172.21.15.188:6819/3290364376","heartbeat_back_addr":"172.21.15.188:6823/3290364376","heartbeat_front_addr":"172.21.15.188:6821/3290364376","state":["exists","up"]},{"osd":9,"uuid":"f033cdc0-1178-4a7f-a338-69ff32416087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6827","nonce":1918893912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6829","nonce":1918893912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6833","nonce":1918893912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6831","nonce":1918893912}]},"public_addr":"172.21.15.6:6827/1918893912","cluster_addr":"172.21.15.6:6829/1918893912","heartbeat_back_addr":"172.21.15.6:6833/1918893912","heartbeat_front_addr":"172.21.15.6:6831/1918893912","state":["exists","up"]},{"osd":10,"uuid":"04e9b5ef-ac0c-482b-a69e-a7121b36e7da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6825","nonce":1395777729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6827","nonce":1395777729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6831","nonce":1395777729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6829","nonce":1395777729}]},"public_addr":"172.21.15.29:6825/1395777729","cluster_addr":"172.21.15.29:6827/1395777729","heartbeat_back_addr":"172.21.15.29:6831/1395777729","heartbeat_front_addr":"172.21.15.29:6829/1395777729","state":["exists","up"]},{"osd":11,"uuid":"cb1638ce-4189-4241-b63e-b63c607e90ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6825","nonce":3322197880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6827","nonce":3322197880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6831","nonce":3322197880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6829","nonce":3322197880}]},"public_addr":"172.21.15.188:6825/3322197880","cluster_addr":"172.21.15.188:6827/3322197880","heartbeat_back_addr":"172.21.15.188:6831/3322197880","heartbeat_front_addr":"172.21.15.188:6829/3322197880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:41.553455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:07.641467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:31.748891+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:03.743553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:30.065262+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:54.976527+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:29.348206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:56.285928+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:21.246409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:54.468561+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:22.093015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:47.342379+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.19","mappings":[{"from":3,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6801/809793211":"2024-04-27T01:45:54.919747+0000","172.21.15.6:6800/809793211":"2024-04-27T01:45:54.919747+0000","172.21.15.6:0/286280546":"2024-04-27T01:45:54.919747+0000","172.21.15.6:0/402969404":"2024-04-27T01:45:54.919747+0000","172.21.15.6:0/2707582744":"2024-04-27T01:45:54.919747+0000","172.21.15.6:0/3143272062":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1325707596":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6800/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6801/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1359518071":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/1823104535":"2024-04-27T01:45:54.919747+0000","172.21.15.6:6801/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/2813598711":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/3140150861":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6800/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/65701414":"2024-04-27T01:33:47.021882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:46:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.822+0000 7f2b94a00700 1 -- 172.21.15.6:0/3384399619 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2b740a3ed0 msgr2=0x7f2b740a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.822+0000 7f2b94a00700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2b740a3ed0 0x7f2b740a6390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2b90102ef0 tx=0x7f2b7800b040 comp rx=0 tx=0).stop 2024-04-26T01:46:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.822+0000 7f2b94a00700 1 -- 172.21.15.6:0/3384399619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b9006afe0 msgr2=0x7f2b9019b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.822+0000 7f2b94a00700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b9006afe0 0x7f2b9019b300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2b80006010 tx=0x7f2b800057c0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.823+0000 7f2b94a00700 1 -- 172.21.15.6:0/3384399619 shutdown_connections 2024-04-26T01:46:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.823+0000 7f2b94a00700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2b740a3ed0 0x7f2b740a6390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.823+0000 7f2b94a00700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9010f4e0 0x7f2b9019fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.823+0000 7f2b94a00700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b9006afe0 0x7f2b9019b300 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.823+0000 7f2b94a00700 1 --2- 172.21.15.6:0/3384399619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b9006b900 0x7f2b9019b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.823+0000 7f2b94a00700 1 -- 172.21.15.6:0/3384399619 >> 172.21.15.6:0/3384399619 conn(0x7f2b900788b0 msgr2=0x7f2b901109b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:19.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.823+0000 7f2b94a00700 1 -- 172.21.15.6:0/3384399619 shutdown_connections 2024-04-26T01:46:19.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:19.823+0000 7f2b94a00700 1 -- 172.21.15.6:0/3384399619 wait complete. 2024-04-26T01:46:20.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:20 smithi188 ceph-mon[32751]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-04-26T01:46:20.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:20 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3384399619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:46:20.666 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:20 smithi006 ceph-mon[28419]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-04-26T01:46:20.666 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:20 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3384399619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:46:20.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:20 smithi029 ceph-mon[32775]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-04-26T01:46:20.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:20 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3384399619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:46:20.731 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-04-26T01:46:21.201 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:22.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:22 smithi006 ceph-mon[28419]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-04-26T01:46:22.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:22 smithi188 ceph-mon[32751]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-04-26T01:46:22.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:22 smithi029 ceph-mon[32775]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-04-26T01:46:24.092 INFO:teuthology.orchestra.run.smithi006.stdout:{"run_dir":"/var/run/ceph"} 2024-04-26T01:46:24.092 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-04-26T01:46:24.093 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 3448 2024-04-26T01:46:24.093 DEBUG:tasks.quiescer:Yielding 2024-04-26T01:46:24.093 INFO:teuthology.run_tasks:Running task workunit... 2024-04-26T01:46:24.107 INFO:tasks.workunit:Pulling workunits from ref afa1933b0fdbb6c99c947d1eda34d661d23cd327 2024-04-26T01:46:24.108 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-26T01:46:24.108 INFO:tasks.workunit:timeout=3h 2024-04-26T01:46:24.108 INFO:tasks.workunit:cleanup=True 2024-04-26T01:46:24.108 DEBUG:teuthology.orchestra.run.smithi006:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-26T01:46:24.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:46:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.158+0000 7f967c600640 1 -- 172.21.15.6:0/767514808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9674107280 msgr2=0x7f967410ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.158+0000 7f967c600640 1 --2- 172.21.15.6:0/767514808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9674107280 0x7f967410ee50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f966c009ca0 tx=0x7f966c02a8c0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.159+0000 7f967c600640 1 -- 172.21.15.6:0/767514808 shutdown_connections 2024-04-26T01:46:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.159+0000 7f967c600640 1 --2- 172.21.15.6:0/767514808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9674107280 0x7f967410ee50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.159+0000 7f967c600640 1 --2- 172.21.15.6:0/767514808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9674102b60 0x7f9674106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.159+0000 7f967c600640 1 --2- 172.21.15.6:0/767514808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96741021b0 0x7f9674102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.159+0000 7f967c600640 1 -- 172.21.15.6:0/767514808 >> 172.21.15.6:0/767514808 conn(0x7f96740fda10 msgr2=0x7f96740ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.159+0000 7f967c600640 1 -- 172.21.15.6:0/767514808 shutdown_connections 2024-04-26T01:46:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.159+0000 7f967c600640 1 -- 172.21.15.6:0/767514808 wait complete. 2024-04-26T01:46:24.158 INFO:teuthology.orchestra.run.smithi006.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-26T01:46:24.159 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-26T01:46:24.159 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 67h/103d Inode: 1099511628283 Links: 2 2024-04-26T01:46:24.159 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:46:24.159 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-26T01:46:24.159 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-26 01:42:46.710828294 +0000 2024-04-26T01:46:24.159 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-26 01:42:46.710828294 +0000 2024-04-26T01:46:24.159 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-26 01:45:53.602855595 +0000 2024-04-26T01:46:24.159 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-04-26 01:42:46.710828294 +0000 2024-04-26T01:46:24.160 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-26T01:46:24.160 DEBUG:teuthology.orchestra.run.smithi006:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-26T01:46:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.160+0000 7f967c600640 1 Processor -- start 2024-04-26T01:46:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.160+0000 7f967c600640 1 -- start start 2024-04-26T01:46:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.160+0000 7f967c600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96741021b0 0x7f96741a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.160+0000 7f967c600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9674107280 0x7f96741a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.160+0000 7f967c600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f967419a7b0 0x7f967419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.160+0000 7f967c600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f96741121d0 con 0x7f96741021b0 2024-04-26T01:46:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.160+0000 7f967c600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9674112050 con 0x7f967419a7b0 2024-04-26T01:46:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.160+0000 7f967c600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9674112350 con 0x7f9674107280 2024-04-26T01:46:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.161+0000 7f9679c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96741021b0 0x7f96741a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.161+0000 7f9679c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96741021b0 0x7f96741a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47774/0 (socket says 172.21.15.6:47774) 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.161+0000 7f9679c00640 1 -- 172.21.15.6:0/1071727083 learned_addr learned my addr 172.21.15.6:0/1071727083 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.161+0000 7f9679200640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9674107280 0x7f96741a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.161+0000 7f967a600640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f967419a7b0 0x7f967419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.161+0000 7f9679c00640 1 -- 172.21.15.6:0/1071727083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9674107280 msgr2=0x7f96741a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.161+0000 7f9679c00640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9674107280 0x7f96741a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.161+0000 7f9679c00640 1 -- 172.21.15.6:0/1071727083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f967419a7b0 msgr2=0x7f967419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.161+0000 7f9679c00640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f967419a7b0 0x7f967419ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.161+0000 7f9679c00640 1 -- 172.21.15.6:0/1071727083 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967419b3b0 con 0x7f96741021b0 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.161+0000 7f967a600640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f967419a7b0 0x7f967419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.162+0000 7f9679c00640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96741021b0 0x7f96741a06a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f966800e9f0 tx=0x7f966800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.162+0000 7f9672000640 1 -- 172.21.15.6:0/1071727083 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966800cd80 con 0x7f96741021b0 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.162+0000 7f9672000640 1 -- 172.21.15.6:0/1071727083 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96680108e0 con 0x7f96741021b0 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.162+0000 7f967c600640 1 -- 172.21.15.6:0/1071727083 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96741a7440 con 0x7f96741021b0 2024-04-26T01:46:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.162+0000 7f9672000640 1 -- 172.21.15.6:0/1071727083 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9668018980 con 0x7f96741021b0 2024-04-26T01:46:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.162+0000 7f967c600640 1 -- 172.21.15.6:0/1071727083 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96741a78a0 con 0x7f96741021b0 2024-04-26T01:46:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.166+0000 7f9672000640 1 -- 172.21.15.6:0/1071727083 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f9668010430 con 0x7f96741021b0 2024-04-26T01:46:24.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.167+0000 7f967f40d740 1 Processor -- start 2024-04-26T01:46:24.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.167+0000 7f967f40d740 1 -- start start 2024-04-26T01:46:24.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f9672000640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f965c0a3eb0 0x7f965c0a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f9672000640 1 -- 172.21.15.6:0/1071727083 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f96680d0e70 con 0x7f96741021b0 2024-04-26T01:46:24.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f967f40d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 0x558c29931c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f967f40d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 0x558c2993c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f967f40d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c2993ca00 0x558c2993edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f967f40d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x558c29897900 con 0x558c29931820 2024-04-26T01:46:24.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f967f40d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558c2983def0 con 0x558c2993ca00 2024-04-26T01:46:24.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f967f40d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558c298938c0 con 0x558c29932140 2024-04-26T01:46:24.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f967a600640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f965c0a3eb0 0x7f965c0a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f9679c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 0x558c29931c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f9679c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 0x558c29931c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47778/0 (socket says 172.21.15.6:47778) 2024-04-26T01:46:24.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f9679c00640 1 -- 172.21.15.6:0/613930136 learned_addr learned my addr 172.21.15.6:0/613930136 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:24.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.168+0000 7f9679200640 1 --2- 172.21.15.6:0/613930136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 0x558c2993c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.169+0000 7f9679200640 1 -- 172.21.15.6:0/613930136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c2993ca00 msgr2=0x558c2993edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:24.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.169+0000 7f9679200640 1 --2- 172.21.15.6:0/613930136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c2993ca00 0x558c2993edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.169+0000 7f9679200640 1 -- 172.21.15.6:0/613930136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 msgr2=0x558c29931c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.169+0000 7f9679200640 1 --2- 172.21.15.6:0/613930136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 0x558c29931c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.169+0000 7f9679200640 1 -- 172.21.15.6:0/613930136 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c295aa210 con 0x558c29932140 2024-04-26T01:46:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.169+0000 7f9679c00640 1 --2- 172.21.15.6:0/613930136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 0x558c29931c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.169+0000 7f9679200640 1 --2- 172.21.15.6:0/613930136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 0x558c2993c4c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f967406b650 tx=0x7f9660011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acccba48b683b0e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.169+0000 7f967a600640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f965c0a3eb0 0x7f965c0a6370 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f966c006010 tx=0x7f966c02f470 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.169+0000 7f9666a00640 1 -- 172.21.15.6:0/613930136 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660004050 con 0x558c29932140 2024-04-26T01:46:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.170+0000 7f9666a00640 1 -- 172.21.15.6:0/613930136 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96600041f0 con 0x558c29932140 2024-04-26T01:46:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.170+0000 7f9666a00640 1 -- 172.21.15.6:0/613930136 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660019660 con 0x558c29932140 2024-04-26T01:46:24.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.170+0000 7f967f40d740 1 -- 172.21.15.6:0/613930136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 msgr2=0x558c2993c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.170+0000 7f967f40d740 1 --2- 172.21.15.6:0/613930136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 0x558c2993c4c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f967406b650 tx=0x7f9660011810 comp rx=0 tx=0).stop 2024-04-26T01:46:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.171+0000 7f967f40d740 1 -- 172.21.15.6:0/613930136 shutdown_connections 2024-04-26T01:46:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.171+0000 7f967f40d740 1 --2- 172.21.15.6:0/613930136 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c2993ca00 0x558c2993edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.171+0000 7f967f40d740 1 --2- 172.21.15.6:0/613930136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 0x558c2993c4c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.171+0000 7f967f40d740 1 --2- 172.21.15.6:0/613930136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 0x558c29931c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.171+0000 7f967f40d740 1 -- 172.21.15.6:0/613930136 >> 172.21.15.6:0/613930136 conn(0x558c2992dd30 msgr2=0x558c2992e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.171+0000 7f967f40d740 1 -- 172.21.15.6:0/613930136 shutdown_connections 2024-04-26T01:46:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.171+0000 7f967f40d740 1 -- 172.21.15.6:0/613930136 wait complete. 2024-04-26T01:46:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.171+0000 7f967f40d740 1 -- start start 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.171+0000 7f967f40d740 1 Processor -- start 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.172+0000 7f967f40d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29931820 0x558c29939910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.172+0000 7f967f40d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29932140 0x558c29939e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.172+0000 7f967f40d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c2993ca00 0x558c2993a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.172+0000 7f967f40d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x558c2993f630 con 0x558c29932140 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.172+0000 7f967f40d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558c29851310 con 0x558c29931820 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.172+0000 7f967f40d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558c2993f7b0 con 0x558c2993ca00 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.172+0000 7f967f40d740 10 client.0 _ll_register_callbacks cb 0x558c2981fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.172+0000 7f967a600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c2993ca00 0x558c2993a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.172+0000 7f9679200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29932140 0x558c29939e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.172+0000 7f9679c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29931820 0x558c29939910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.172+0000 7f967a600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c2993ca00 0x558c2993a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:40876/0 (socket says 172.21.15.6:40876) 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.172+0000 7f967a600640 1 -- 172.21.15.6:0/1945323885 learned_addr learned my addr 172.21.15.6:0/1945323885 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.173+0000 7f9679c00640 1 -- 172.21.15.6:0/1945323885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c2993ca00 msgr2=0x558c2993a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.173+0000 7f9679c00640 1 --2- 172.21.15.6:0/1945323885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c2993ca00 0x558c2993a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.173+0000 7f9679c00640 1 -- 172.21.15.6:0/1945323885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29932140 msgr2=0x558c29939e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.173+0000 7f9679c00640 1 --2- 172.21.15.6:0/1945323885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29932140 0x558c29939e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.173+0000 7f9679c00640 1 -- 172.21.15.6:0/1945323885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c29676620 con 0x558c29931820 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.173+0000 7f9679c00640 1 --2- 172.21.15.6:0/1945323885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29931820 0x558c29939910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f966809c880 tx=0x7f966809d280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=816d0c8db6f459b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.173+0000 7f965b400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.173+0000 7f965b400640 1 -- 172.21.15.6:0/1945323885 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96680a57f0 con 0x558c29931820 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.173+0000 7f967f40d740 1 -- 172.21.15.6:0/1945323885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96600041f0 con 0x558c29931820 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.173+0000 7f965b400640 1 -- 172.21.15.6:0/1945323885 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f96680ce6e0 con 0x558c29931820 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.174+0000 7f965b400640 1 -- 172.21.15.6:0/1945323885 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f96680ce8e0 con 0x558c29931820 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.174+0000 7f967f40d740 10 client.25193 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.174+0000 7f967f40d740 1 -- 172.21.15.6:0/1945323885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558c2994b900 con 0x558c29931820 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.174+0000 7f965b400640 1 -- 172.21.15.6:0/1945323885 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96600041f0 con 0x558c29931820 2024-04-26T01:46:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.174+0000 7f965b400640 1 -- 172.21.15.6:0/1945323885 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f96680cfce0 con 0x558c29931820 2024-04-26T01:46:24.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.174+0000 7f967f40d740 10 client.25193 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:24.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.174+0000 7f967f40d740 10 client.25193 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:24.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.174+0000 7f967f40d740 20 client.25193 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:24.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.174+0000 7f967f40d740 1 --2- 172.21.15.6:0/1945323885 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558c2994cd70 0x558c2994f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.174+0000 7f967f40d740 4 client.25193 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:24.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.174+0000 7f967f40d740 1 -- 172.21.15.6:0/1945323885 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c298ab8b0 con 0x558c2994cd70 2024-04-26T01:46:24.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.175+0000 7f9679200640 1 --2- 172.21.15.6:0/1945323885 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558c2994cd70 0x558c2994f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.175+0000 7f9679200640 1 --2- 172.21.15.6:0/1945323885 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558c2994cd70 0x558c2994f160 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f837ba70c4878869 server_cookie=6fe6763bc7209c5f in_seq=0 out_seq=0 2024-04-26T01:46:24.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.175+0000 7f965b400640 10 client.25193 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.176+0000 7f965b400640 1 -- 172.21.15.6:0/1945323885 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c298ab8b0 con 0x558c2994cd70 2024-04-26T01:46:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.176+0000 7f965b400640 10 client.25193 handle_command_reply: tid=0 2024-04-26T01:46:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.176+0000 7f967f40d740 1 client.25193 shutdown 2024-04-26T01:46:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.177+0000 7f967f40d740 1 -- 172.21.15.6:0/1945323885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29931820 msgr2=0x558c29939910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.177+0000 7f967f40d740 1 --2- 172.21.15.6:0/1945323885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29931820 0x558c29939910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f966809c880 tx=0x7f966809d280 comp rx=0 tx=0).stop 2024-04-26T01:46:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.177+0000 7f967f40d740 1 -- 172.21.15.6:0/1945323885 shutdown_connections 2024-04-26T01:46:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.177+0000 7f967f40d740 1 --2- 172.21.15.6:0/1945323885 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558c2994cd70 0x558c2994f160 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.177+0000 7f967f40d740 1 --2- 172.21.15.6:0/1945323885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29932140 0x558c29939e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.177+0000 7f967f40d740 1 --2- 172.21.15.6:0/1945323885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c2993ca00 0x558c2993a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.177+0000 7f967f40d740 1 --2- 172.21.15.6:0/1945323885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29931820 0x558c29939910 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.177+0000 7f967f40d740 1 -- 172.21.15.6:0/1945323885 >> 172.21.15.6:0/1945323885 conn(0x558c2992dd30 msgr2=0x558c29933ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.177+0000 7f965b400640 0 client.25193 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.177+0000 7f967f40d740 1 -- 172.21.15.6:0/1945323885 shutdown_connections 2024-04-26T01:46:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.177+0000 7f967f40d740 1 -- 172.21.15.6:0/1945323885 wait complete. 2024-04-26T01:46:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.177+0000 7f967f40d740 20 client.25193 trim_cache size 0 max 16384 2024-04-26T01:46:24.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.198+0000 7f967f40d740 1 Processor -- start 2024-04-26T01:46:24.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.198+0000 7f967f40d740 1 -- start start 2024-04-26T01:46:24.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.198+0000 7f967f40d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29931820 0x558c299484f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.198+0000 7f967f40d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 0x558c2992cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.198+0000 7f967f40d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c2993ca00 0x558c29954a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.197 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.198+0000 7f967f40d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f96741b2830 con 0x558c2993ca00 2024-04-26T01:46:24.197 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.198+0000 7f967f40d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f96741b0100 con 0x558c29931820 2024-04-26T01:46:24.197 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.198+0000 7f967f40d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f96741adf50 con 0x558c29932140 2024-04-26T01:46:24.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.199+0000 7f9679200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 0x558c2992cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.199+0000 7f967a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c2993ca00 0x558c29954a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.199+0000 7f9679200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 0x558c2992cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:40880/0 (socket says 172.21.15.6:40880) 2024-04-26T01:46:24.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.199+0000 7f9679200640 1 -- 172.21.15.6:0/1041651264 learned_addr learned my addr 172.21.15.6:0/1041651264 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:24.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.199+0000 7f9679c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29931820 0x558c299484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.199+0000 7f9679200640 1 -- 172.21.15.6:0/1041651264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29931820 msgr2=0x558c299484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:24.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.199+0000 7f9679200640 1 --2- 172.21.15.6:0/1041651264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29931820 0x558c299484f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.199+0000 7f9679200640 1 -- 172.21.15.6:0/1041651264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c2993ca00 msgr2=0x558c29954a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:24.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.199+0000 7f9679200640 1 --2- 172.21.15.6:0/1041651264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c2993ca00 0x558c29954a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.199+0000 7f9679200640 1 -- 172.21.15.6:0/1041651264 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c2994f4d0 con 0x558c29932140 2024-04-26T01:46:24.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.199+0000 7f9679200640 1 --2- 172.21.15.6:0/1041651264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 0x558c2992cd10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x558c29932dc0 tx=0x7f9660009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f918ceb6b2f7378c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:24.199 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.200+0000 7f9666000640 1 -- 172.21.15.6:0/1041651264 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660031070 con 0x558c29932140 2024-04-26T01:46:24.199 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.200+0000 7f9666000640 1 -- 172.21.15.6:0/1041651264 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9660009df0 con 0x558c29932140 2024-04-26T01:46:24.199 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.200+0000 7f9666000640 1 -- 172.21.15.6:0/1041651264 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660005030 con 0x558c29932140 2024-04-26T01:46:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.201+0000 7f967f40d740 1 -- 172.21.15.6:0/1041651264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 msgr2=0x558c2992cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.201+0000 7f967f40d740 1 --2- 172.21.15.6:0/1041651264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 0x558c2992cd10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x558c29932dc0 tx=0x7f9660009680 comp rx=0 tx=0).stop 2024-04-26T01:46:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.201+0000 7f967f40d740 1 -- 172.21.15.6:0/1041651264 shutdown_connections 2024-04-26T01:46:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.201+0000 7f967f40d740 1 --2- 172.21.15.6:0/1041651264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c2993ca00 0x558c29954a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.201+0000 7f967f40d740 1 --2- 172.21.15.6:0/1041651264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c29932140 0x558c2992cd10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.201+0000 7f967f40d740 1 --2- 172.21.15.6:0/1041651264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29931820 0x558c299484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.201+0000 7f967f40d740 1 -- 172.21.15.6:0/1041651264 >> 172.21.15.6:0/1041651264 conn(0x558c2992dd30 msgr2=0x558c2992c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.201+0000 7f967f40d740 1 -- 172.21.15.6:0/1041651264 shutdown_connections 2024-04-26T01:46:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.201+0000 7f967f40d740 1 -- 172.21.15.6:0/1041651264 wait complete. 2024-04-26T01:46:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.202+0000 7f967f40d740 1 -- start start 2024-04-26T01:46:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.202+0000 7f967f40d740 1 Processor -- start 2024-04-26T01:46:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.202+0000 7f967f40d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 0x558c29944ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.202+0000 7f967f40d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29932140 0x558c29942230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.202+0000 7f967f40d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c2993ca00 0x558c29942840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.202+0000 7f967f40d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f96741ae5c0 con 0x558c29931820 2024-04-26T01:46:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.202+0000 7f967f40d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f96741afa30 con 0x558c29932140 2024-04-26T01:46:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.202+0000 7f967f40d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f96741aec90 con 0x558c2993ca00 2024-04-26T01:46:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.202+0000 7f967f40d740 10 client.0 _ll_register_callbacks cb 0x7f966809c4e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f9679c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 0x558c29944ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f9679c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 0x558c29944ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47796/0 (socket says 172.21.15.6:47796) 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f9679c00640 1 -- 172.21.15.6:0/796950744 learned_addr learned my addr 172.21.15.6:0/796950744 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f9679200640 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29932140 0x558c29942230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f967a600640 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c2993ca00 0x558c29942840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f9679c00640 1 -- 172.21.15.6:0/796950744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c2993ca00 msgr2=0x558c29942840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f9679c00640 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c2993ca00 0x558c29942840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f9679c00640 1 -- 172.21.15.6:0/796950744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29932140 msgr2=0x558c29942230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f9679c00640 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29932140 0x558c29942230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f9679c00640 1 -- 172.21.15.6:0/796950744 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c29943010 con 0x558c29931820 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f967a600640 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c2993ca00 0x558c29942840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f9679c00640 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 0x558c29944ea0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9668009250 tx=0x7f96680ce540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206712339c1c41ec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.203+0000 7f965b400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.204+0000 7f965b400640 1 -- 172.21.15.6:0/796950744 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96680a57f0 con 0x558c29931820 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.204+0000 7f965b400640 1 -- 172.21.15.6:0/796950744 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f966809bc50 con 0x558c29931820 2024-04-26T01:46:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.204+0000 7f967f40d740 1 -- 172.21.15.6:0/796950744 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9660009df0 con 0x558c29931820 2024-04-26T01:46:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.204+0000 7f965b400640 1 -- 172.21.15.6:0/796950744 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f96680b11a0 con 0x558c29931820 2024-04-26T01:46:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.204+0000 7f967f40d740 10 client.15096 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.204+0000 7f967f40d740 1 -- 172.21.15.6:0/796950744 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558c299b33b0 con 0x558c29931820 2024-04-26T01:46:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.204+0000 7f965b400640 1 -- 172.21.15.6:0/796950744 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9660009df0 con 0x558c29931820 2024-04-26T01:46:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.205+0000 7f965b400640 1 -- 172.21.15.6:0/796950744 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f96680a20c0 con 0x558c29931820 2024-04-26T01:46:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.205+0000 7f967f40d740 10 client.15096 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.205+0000 7f967f40d740 10 client.15096 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.205+0000 7f967f40d740 20 client.15096 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.205+0000 7f967f40d740 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558c299b3890 0x558c299b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.205+0000 7f967f40d740 4 client.15096 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.205+0000 7f967f40d740 1 -- 172.21.15.6:0/796950744 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558c298a5320 con 0x558c299b3890 2024-04-26T01:46:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.205+0000 7f9679200640 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558c299b3890 0x558c299b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.206+0000 7f965b400640 10 client.15096 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.206+0000 7f9679200640 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558c299b3890 0x558c299b5c80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a863cc65be73bfc1 server_cookie=bd07243ccb9e7fb8 in_seq=0 out_seq=0 2024-04-26T01:46:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.212+0000 7f965b400640 1 -- 172.21.15.6:0/796950744 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558c298a5320 con 0x558c299b3890 2024-04-26T01:46:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.212+0000 7f965b400640 10 client.15096 handle_command_reply: tid=0 2024-04-26T01:46:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.212+0000 7f967f40d740 1 client.15096 shutdown 2024-04-26T01:46:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.213+0000 7f967f40d740 1 -- 172.21.15.6:0/796950744 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 msgr2=0x558c29944ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.213+0000 7f967f40d740 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 0x558c29944ea0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9668009250 tx=0x7f96680ce540 comp rx=0 tx=0).stop 2024-04-26T01:46:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.213+0000 7f967f40d740 1 -- 172.21.15.6:0/796950744 shutdown_connections 2024-04-26T01:46:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.213+0000 7f967f40d740 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558c299b3890 0x558c299b5c80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.213+0000 7f967f40d740 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558c2993ca00 0x558c29942840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.213+0000 7f967f40d740 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c29932140 0x558c29942230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.213+0000 7f967f40d740 1 --2- 172.21.15.6:0/796950744 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558c29931820 0x558c29944ea0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.213+0000 7f967f40d740 1 -- 172.21.15.6:0/796950744 >> 172.21.15.6:0/796950744 conn(0x558c2992dd30 msgr2=0x558c29933a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.213+0000 7f965b400640 0 client.15096 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.213+0000 7f967f40d740 1 -- 172.21.15.6:0/796950744 shutdown_connections 2024-04-26T01:46:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.213+0000 7f967f40d740 1 -- 172.21.15.6:0/796950744 wait complete. 2024-04-26T01:46:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.213+0000 7f967f40d740 20 client.15096 trim_cache size 0 max 16384 2024-04-26T01:46:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:46:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:46:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:46:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:46:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.214+0000 7f967c600640 1 -- 172.21.15.6:0/1071727083 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f965c0a3eb0 msgr2=0x7f965c0a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.214+0000 7f967c600640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f965c0a3eb0 0x7f965c0a6370 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f966c006010 tx=0x7f966c02f470 comp rx=0 tx=0).stop 2024-04-26T01:46:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.214+0000 7f967c600640 1 -- 172.21.15.6:0/1071727083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96741021b0 msgr2=0x7f96741a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.214+0000 7f967c600640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96741021b0 0x7f96741a06a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f966800e9f0 tx=0x7f966800eeb0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.214+0000 7f967c600640 1 -- 172.21.15.6:0/1071727083 shutdown_connections 2024-04-26T01:46:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.214+0000 7f967c600640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f965c0a3eb0 0x7f965c0a6370 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.214+0000 7f967c600640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f967419a7b0 0x7f967419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.214+0000 7f967c600640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9674107280 0x7f96741a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.214+0000 7f967c600640 1 --2- 172.21.15.6:0/1071727083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96741021b0 0x7f96741a06a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.214+0000 7f967c600640 1 -- 172.21.15.6:0/1071727083 >> 172.21.15.6:0/1071727083 conn(0x7f96740fda10 msgr2=0x7f96740ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.214+0000 7f967c600640 1 -- 172.21.15.6:0/1071727083 shutdown_connections 2024-04-26T01:46:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:24.214+0000 7f967c600640 1 -- 172.21.15.6:0/1071727083 wait complete. 2024-04-26T01:46:24.244 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout afa1933b0fdbb6c99c947d1eda34d661d23cd327 2024-04-26T01:46:24.342 INFO:tasks.workunit.client.0.smithi006.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-26T01:46:24.551 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:24 smithi188 ceph-mon[32751]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-04-26T01:46:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:24 smithi029 ceph-mon[32775]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-04-26T01:46:24.702 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:24 smithi006 ceph-mon[28419]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-04-26T01:46:26.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:26 smithi006 ceph-mon[28419]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:46:26.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:26 smithi188 ceph-mon[32751]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:46:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:26 smithi029 ceph-mon[32775]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:46:26.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.806+0000 7ff6daa00700 1 -- 172.21.15.6:0/2065793644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6d4107990 msgr2=0x7ff6d4107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:26.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.806+0000 7ff6daa00700 1 --2- 172.21.15.6:0/2065793644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6d4107990 0x7ff6d4107d70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff6c40098f0 tx=0x7ff6c4009c00 comp rx=0 tx=0).stop 2024-04-26T01:46:26.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.807+0000 7ff6daa00700 1 -- 172.21.15.6:0/2065793644 shutdown_connections 2024-04-26T01:46:26.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.807+0000 7ff6daa00700 1 --2- 172.21.15.6:0/2065793644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4102300 0x7ff6d410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:26.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.807+0000 7ff6daa00700 1 --2- 172.21.15.6:0/2065793644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6d4107990 0x7ff6d4107d70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.807+0000 7ff6daa00700 1 --2- 172.21.15.6:0/2065793644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d4101940 0x7ff6d4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.807+0000 7ff6daa00700 1 -- 172.21.15.6:0/2065793644 >> 172.21.15.6:0/2065793644 conn(0x7ff6d40788f0 msgr2=0x7ff6d4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.807+0000 7ff6daa00700 1 -- 172.21.15.6:0/2065793644 shutdown_connections 2024-04-26T01:46:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.807+0000 7ff6daa00700 1 -- 172.21.15.6:0/2065793644 wait complete. 2024-04-26T01:46:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.808+0000 7ff6daa00700 1 Processor -- start 2024-04-26T01:46:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.808+0000 7ff6daa00700 1 -- start start 2024-04-26T01:46:26.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.809+0000 7ff6daa00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4101940 0x7ff6d41034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:26.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.809+0000 7ff6daa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d4102300 0x7ff6d4103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:26.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.809+0000 7ff6daa00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6d4107990 0x7ff6d4103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:26.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.809+0000 7ff6daa00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d4111b00 con 0x7ff6d4107990 2024-04-26T01:46:26.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.809+0000 7ff6daa00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d4111980 con 0x7ff6d4101940 2024-04-26T01:46:26.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.809+0000 7ff6daa00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d4111c80 con 0x7ff6d4102300 2024-04-26T01:46:26.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.809+0000 7ff6d3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4101940 0x7ff6d41034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:26.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.809+0000 7ff6d3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4101940 0x7ff6d41034f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:51028/0 (socket says 172.21.15.6:51028) 2024-04-26T01:46:26.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.809+0000 7ff6d3e00700 1 -- 172.21.15.6:0/2532305155 learned_addr learned my addr 172.21.15.6:0/2532305155 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:26.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.809+0000 7ff6d3400700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d4102300 0x7ff6d4103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:26.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.809+0000 7ff6d8a00700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6d4107990 0x7ff6d4103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:26.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.810+0000 7ff6d3e00700 1 -- 172.21.15.6:0/2532305155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d4102300 msgr2=0x7ff6d4103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:26.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.810+0000 7ff6d3e00700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d4102300 0x7ff6d4103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:26.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.810+0000 7ff6d3e00700 1 -- 172.21.15.6:0/2532305155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6d4107990 msgr2=0x7ff6d4103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:26.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.810+0000 7ff6d3e00700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6d4107990 0x7ff6d4103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:26.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.810+0000 7ff6d3e00700 1 -- 172.21.15.6:0/2532305155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d41a7750 con 0x7ff6d4101940 2024-04-26T01:46:26.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.810+0000 7ff6d8a00700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6d4107990 0x7ff6d4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:46:26.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.810+0000 7ff6d3e00700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4101940 0x7ff6d41034f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400f990 tx=0x7ff6c400f9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:26.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.810+0000 7ff6d0c00700 1 -- 172.21.15.6:0/2532305155 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4027070 con 0x7ff6d4101940 2024-04-26T01:46:26.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.811+0000 7ff6daa00700 1 -- 172.21.15.6:0/2532305155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d41a79e0 con 0x7ff6d4101940 2024-04-26T01:46:26.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.811+0000 7ff6d0c00700 1 -- 172.21.15.6:0/2532305155 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6c40055f0 con 0x7ff6d4101940 2024-04-26T01:46:26.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.811+0000 7ff6d0c00700 1 -- 172.21.15.6:0/2532305155 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4004930 con 0x7ff6d4101940 2024-04-26T01:46:26.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.811+0000 7ff6daa00700 1 -- 172.21.15.6:0/2532305155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d41a7e90 con 0x7ff6d4101940 2024-04-26T01:46:26.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.812+0000 7ff6daa00700 1 -- 172.21.15.6:0/2532305155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6d4069730 con 0x7ff6d4101940 2024-04-26T01:46:26.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.814+0000 7ff6d0c00700 1 -- 172.21.15.6:0/2532305155 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7ff6c402a070 con 0x7ff6d4101940 2024-04-26T01:46:26.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.816+0000 7ff6d0c00700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff6b80a3ed0 0x7ff6b80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:26.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.816+0000 7ff6d0c00700 1 -- 172.21.15.6:0/2532305155 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7ff6c40d1c30 con 0x7ff6d4101940 2024-04-26T01:46:26.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.816+0000 7ff6d3400700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff6b80a3ed0 0x7ff6b80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:26.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.817+0000 7ff6d3400700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff6b80a3ed0 0x7ff6b80a6390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4078f60 tx=0x7ff6c0009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:26.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:26.818+0000 7ff6d0c00700 1 -- 172.21.15.6:0/2532305155 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6c4031070 con 0x7ff6d4101940 2024-04-26T01:46:27.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.153+0000 7ff6daa00700 1 -- 172.21.15.6:0/2532305155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6d41a80c0 con 0x7ff6d4101940 2024-04-26T01:46:27.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.154+0000 7ff6d0c00700 1 -- 172.21.15.6:0/2532305155 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11850 (secure 0 0 0) 0x7ff6c4013030 con 0x7ff6d4101940 2024-04-26T01:46:27.152 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:46:27.152 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:45:54.644030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24512,24505]},"id":1}]} 2024-04-26T01:46:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.157+0000 7ff6daa00700 1 -- 172.21.15.6:0/2532305155 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff6b80a3ed0 msgr2=0x7ff6b80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.157+0000 7ff6daa00700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff6b80a3ed0 0x7ff6b80a6390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4078f60 tx=0x7ff6c0009260 comp rx=0 tx=0).stop 2024-04-26T01:46:27.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.157+0000 7ff6daa00700 1 -- 172.21.15.6:0/2532305155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4101940 msgr2=0x7ff6d41034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:27.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.157+0000 7ff6daa00700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4101940 0x7ff6d41034f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400f990 tx=0x7ff6c400f9c0 comp rx=0 tx=0).stop 2024-04-26T01:46:27.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.157+0000 7ff6daa00700 1 -- 172.21.15.6:0/2532305155 shutdown_connections 2024-04-26T01:46:27.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.157+0000 7ff6daa00700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff6b80a3ed0 0x7ff6b80a6390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:27.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.157+0000 7ff6daa00700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6d4107990 0x7ff6d4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:27.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.157+0000 7ff6daa00700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4101940 0x7ff6d41034f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:27.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.157+0000 7ff6daa00700 1 --2- 172.21.15.6:0/2532305155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d4102300 0x7ff6d4103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:27.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.157+0000 7ff6daa00700 1 -- 172.21.15.6:0/2532305155 >> 172.21.15.6:0/2532305155 conn(0x7ff6d40788f0 msgr2=0x7ff6d406a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:27.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.157+0000 7ff6daa00700 1 -- 172.21.15.6:0/2532305155 shutdown_connections 2024-04-26T01:46:27.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:27.157+0000 7ff6daa00700 1 -- 172.21.15.6:0/2532305155 wait complete. 2024-04-26T01:46:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-04-26T01:46:27.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:27 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2532305155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:27.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:27 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2532305155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:27.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:27 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2532305155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:28.216 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T01:42:08.299563+0000', 'modified': '2024-04-26T01:45:54.644030+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24523, 'mds_1': 24512, 'mds_2': 24505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6837/402171153', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6836', 'nonce': 402171153}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 402171153}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'g', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.6:6839/3936238112', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6838', 'nonce': 3936238112}, {'type': 'v1', 'addr': '172.21.15.6:6839', 'nonce': 3936238112}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6835/1255666766', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6834', 'nonce': 1255666766}, {'type': 'v1', 'addr': '172.21.15.188:6835', 'nonce': 1255666766}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.188:6837/489767141', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6836', 'nonce': 489767141}, {'type': 'v1', 'addr': '172.21.15.188:6837', 'nonce': 489767141}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24512': {'gid': 24512, 'name': 'b', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.29:6839/2325582359', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6838', 'nonce': 2325582359}, {'type': 'v1', 'addr': '172.21.15.29:6839', 'nonce': 2325582359}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.188:6839/674623421', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6838', 'nonce': 674623421}, {'type': 'v1', 'addr': '172.21.15.188:6839', 'nonce': 674623421}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24523, 'qdb_cluster': [24523, 24512, 24505]} 2024-04-26T01:46:28.217 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T01:46:28.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell mds.c status 2024-04-26T01:46:28.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:28 smithi006 ceph-mon[28419]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.2 KiB/s wr, 13 op/s 2024-04-26T01:46:28.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:28 smithi188 ceph-mon[32751]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.2 KiB/s wr, 13 op/s 2024-04-26T01:46:28.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:28 smithi029 ceph-mon[32775]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.2 KiB/s wr, 13 op/s 2024-04-26T01:46:28.696 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:30.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:30 smithi188 ceph-mon[32751]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T01:46:30.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:30 smithi029 ceph-mon[32775]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T01:46:30.695 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:30 smithi006 ceph-mon[28419]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T01:46:30.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.874+0000 7f59df800700 1 -- 172.21.15.6:0/365082146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59d8074c70 msgr2=0x7f59d8075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:30.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.874+0000 7f59df800700 1 --2- 172.21.15.6:0/365082146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59d8074c70 0x7f59d8075050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f59d000b0a0 tx=0x7f59d000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.874+0000 7f59df800700 1 -- 172.21.15.6:0/365082146 shutdown_connections 2024-04-26T01:46:30.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.874+0000 7f59df800700 1 --2- 172.21.15.6:0/365082146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d810f470 0x7f59d8111860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.874+0000 7f59df800700 1 --2- 172.21.15.6:0/365082146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59d8074c70 0x7f59d8075050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.874+0000 7f59df800700 1 --2- 172.21.15.6:0/365082146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8075590 0x7f59d810ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.874+0000 7f59df800700 1 -- 172.21.15.6:0/365082146 >> 172.21.15.6:0/365082146 conn(0x7f59d8079bf0 msgr2=0x7f59d807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:30.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.874+0000 7f59df800700 1 -- 172.21.15.6:0/365082146 shutdown_connections 2024-04-26T01:46:30.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.874+0000 7f59df800700 1 -- 172.21.15.6:0/365082146 wait complete. 2024-04-26T01:46:30.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.875+0000 7f59df800700 1 Processor -- start 2024-04-26T01:46:30.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.876+0000 7f59df800700 1 -- start start 2024-04-26T01:46:30.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.876+0000 7f59df800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8074c70 0x7f59d819b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.876+0000 7f59df800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d8075590 0x7f59d819b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.876+0000 7f59df800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59d810f470 0x7f59d819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.876+0000 7f59df800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f59d8114eb0 con 0x7f59d8074c70 2024-04-26T01:46:30.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.876+0000 7f59df800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f59d8114d30 con 0x7f59d810f470 2024-04-26T01:46:30.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.876+0000 7f59df800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f59d8115030 con 0x7f59d8075590 2024-04-26T01:46:30.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.877+0000 7f59dce00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8074c70 0x7f59d819b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.877+0000 7f59dce00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8074c70 0x7f59d819b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43016/0 (socket says 172.21.15.6:43016) 2024-04-26T01:46:30.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.877+0000 7f59dce00700 1 -- 172.21.15.6:0/4246669644 learned_addr learned my addr 172.21.15.6:0/4246669644 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:30.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.877+0000 7f59d7e00700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d8075590 0x7f59d819b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.877+0000 7f59dd800700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59d810f470 0x7f59d819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.877+0000 7f59dce00700 1 -- 172.21.15.6:0/4246669644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d8075590 msgr2=0x7f59d819b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:30.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.877+0000 7f59dce00700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d8075590 0x7f59d819b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.877+0000 7f59dce00700 1 -- 172.21.15.6:0/4246669644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59d810f470 msgr2=0x7f59d819fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:30.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.877+0000 7f59dce00700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59d810f470 0x7f59d819fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.877+0000 7f59dce00700 1 -- 172.21.15.6:0/4246669644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d81a0430 con 0x7f59d8074c70 2024-04-26T01:46:30.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.877+0000 7f59dd800700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59d810f470 0x7f59d819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:30.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.878+0000 7f59dce00700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8074c70 0x7f59d819b420 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f59d000d010 tx=0x7f59d0004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:30.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.878+0000 7f59d5600700 1 -- 172.21.15.6:0/4246669644 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0021070 con 0x7f59d8074c70 2024-04-26T01:46:30.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.878+0000 7f59d5600700 1 -- 172.21.15.6:0/4246669644 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59d0007c40 con 0x7f59d8074c70 2024-04-26T01:46:30.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.878+0000 7f59d5600700 1 -- 172.21.15.6:0/4246669644 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0004710 con 0x7f59d8074c70 2024-04-26T01:46:30.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.878+0000 7f59df800700 1 -- 172.21.15.6:0/4246669644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d81a06c0 con 0x7f59d8074c70 2024-04-26T01:46:30.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.878+0000 7f59df800700 1 -- 172.21.15.6:0/4246669644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d81a7fa0 con 0x7f59d8074c70 2024-04-26T01:46:30.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.882+0000 7f59d5600700 1 -- 172.21.15.6:0/4246669644 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f59d0003a00 con 0x7f59d8074c70 2024-04-26T01:46:30.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.883+0000 7f59ea0a7b80 1 Processor -- start 2024-04-26T01:46:30.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.884+0000 7f59ea0a7b80 1 -- start start 2024-04-26T01:46:30.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.884+0000 7f59ea0a7b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 0x55ff0287f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.884+0000 7f59ea0a7b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0287fc10 0x55ff0288a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.884+0000 7f59ea0a7b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0288aab0 0x55ff0288cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.884+0000 7f59d5600700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f59bc0a3f50 0x7f59bc0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.884+0000 7f59ea0a7b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55ff027d2e40 con 0x55ff0287fc10 2024-04-26T01:46:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.884+0000 7f59ea0a7b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ff027796e0 con 0x55ff0288aab0 2024-04-26T01:46:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.884+0000 7f59ea0a7b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ff027cee00 con 0x55ff0287f2f0 2024-04-26T01:46:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.884+0000 7f59d5600700 1 -- 172.21.15.6:0/4246669644 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f59d000f020 con 0x7f59d8074c70 2024-04-26T01:46:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59dd800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 0x55ff0287f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59dce00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0287fc10 0x55ff0288a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59dd800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 0x55ff0287f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56536/0 (socket says 172.21.15.6:56536) 2024-04-26T01:46:30.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59dd800700 1 -- 172.21.15.6:0/2478586794 learned_addr learned my addr 172.21.15.6:0/2478586794 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:30.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59dce00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0287fc10 0x55ff0288a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43028/0 (socket says 172.21.15.6:43028) 2024-04-26T01:46:30.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59dd800700 1 --2- 172.21.15.6:0/2478586794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0288aab0 0x55ff0288cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59d7e00700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f59bc0a3f50 0x7f59bc0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59dd800700 1 -- 172.21.15.6:0/2478586794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0288aab0 msgr2=0x55ff0288cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59dd800700 1 --2- 172.21.15.6:0/2478586794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0288aab0 0x55ff0288cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59dd800700 1 -- 172.21.15.6:0/2478586794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0287fc10 msgr2=0x55ff0288a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59dd800700 1 --2- 172.21.15.6:0/2478586794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0287fc10 0x55ff0288a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59dd800700 1 -- 172.21.15.6:0/2478586794 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff0276f630 con 0x55ff0287f2f0 2024-04-26T01:46:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.885+0000 7f59dd800700 1 --2- 172.21.15.6:0/2478586794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 0x55ff0287f6d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f59c80127f0 tx=0x7f59c8009790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70eaa9bd38b283b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.886+0000 7f59ce000700 1 -- 172.21.15.6:0/2478586794 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c8012e10 con 0x55ff0287f2f0 2024-04-26T01:46:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.886+0000 7f59d7e00700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f59bc0a3f50 0x7f59bc0a6410 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f59d8102630 tx=0x7f59c4009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.886+0000 7f59ce000700 1 -- 172.21.15.6:0/2478586794 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59c800f900 con 0x55ff0287f2f0 2024-04-26T01:46:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.886+0000 7f59ce000700 1 -- 172.21.15.6:0/2478586794 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c801e7e0 con 0x55ff0287f2f0 2024-04-26T01:46:30.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.887+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/2478586794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 msgr2=0x55ff0287f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:30.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.887+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/2478586794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 0x55ff0287f6d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f59c80127f0 tx=0x7f59c8009790 comp rx=0 tx=0).stop 2024-04-26T01:46:30.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.887+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/2478586794 shutdown_connections 2024-04-26T01:46:30.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.887+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/2478586794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0288aab0 0x55ff0288cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.887+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/2478586794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 0x55ff0287f6d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.887+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/2478586794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0287fc10 0x55ff0288a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.887+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/2478586794 >> 172.21.15.6:0/2478586794 conn(0x55ff027ae550 msgr2=0x55ff0287b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:30.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.887+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/2478586794 shutdown_connections 2024-04-26T01:46:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.887+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/2478586794 wait complete. 2024-04-26T01:46:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.888+0000 7f59ea0a7b80 1 -- start start 2024-04-26T01:46:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.888+0000 7f59ea0a7b80 1 Processor -- start 2024-04-26T01:46:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59ea0a7b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 0x55ff0288a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59ea0a7b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287fc10 0x55ff02887500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59ea0a7b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0288aab0 0x55ff02887a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59ea0a7b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55ff0288d5a0 con 0x55ff0288aab0 2024-04-26T01:46:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59ea0a7b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ff0278c810 con 0x55ff0287f2f0 2024-04-26T01:46:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59ea0a7b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ff0288d720 con 0x55ff0287fc10 2024-04-26T01:46:30.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59ea0a7b80 10 client.0 _ll_register_callbacks cb 0x55ff0276b6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:30.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59d7e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287fc10 0x55ff02887500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59dce00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 0x55ff0288a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59d7e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287fc10 0x55ff02887500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56548/0 (socket says 172.21.15.6:56548) 2024-04-26T01:46:30.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59d7e00700 1 -- 172.21.15.6:0/1722085360 learned_addr learned my addr 172.21.15.6:0/1722085360 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:30.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59dd800700 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0288aab0 0x55ff02887a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.889+0000 7f59d7e00700 1 -- 172.21.15.6:0/1722085360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 msgr2=0x55ff0288a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.890+0000 7f59d7e00700 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 0x55ff0288a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.890+0000 7f59d7e00700 1 -- 172.21.15.6:0/1722085360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0288aab0 msgr2=0x55ff02887a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.890+0000 7f59d7e00700 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0288aab0 0x55ff02887a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.890+0000 7f59d7e00700 1 -- 172.21.15.6:0/1722085360 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff02720880 con 0x55ff0287fc10 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.890+0000 7f59dce00700 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 0x55ff0288a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.890+0000 7f59d7e00700 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287fc10 0x55ff02887500 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f59c4011fd0 tx=0x7f59c40047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c0230ddb7a51f26 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.890+0000 7f59c2a00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.890+0000 7f59c2a00700 1 -- 172.21.15.6:0/1722085360 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c4004020 con 0x55ff0287fc10 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.890+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1722085360 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59c800f900 con 0x55ff0287fc10 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.890+0000 7f59c2a00700 1 -- 172.21.15.6:0/1722085360 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f59c40041c0 con 0x55ff0287fc10 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.891+0000 7f59c2a00700 1 -- 172.21.15.6:0/1722085360 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f59c40043c0 con 0x55ff0287fc10 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.891+0000 7f59ea0a7b80 10 client.25255 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.891+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1722085360 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ff02769aa0 con 0x55ff0287fc10 2024-04-26T01:46:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.891+0000 7f59c2a00700 1 -- 172.21.15.6:0/1722085360 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59c800f900 con 0x55ff0287fc10 2024-04-26T01:46:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.891+0000 7f59c2a00700 1 -- 172.21.15.6:0/1722085360 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f59c401e8d0 con 0x55ff0287fc10 2024-04-26T01:46:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.891+0000 7f59ea0a7b80 10 client.25255 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.891+0000 7f59ea0a7b80 10 client.25255 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T01:46:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.891+0000 7f59ea0a7b80 20 client.25255 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.891+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55ff0289a320 0x55ff0289b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.892+0000 7f59ea0a7b80 4 client.25255 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.892+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1722085360 --> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff027e0630 con 0x55ff0289a320 2024-04-26T01:46:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.892+0000 7f59dce00700 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55ff0289a320 0x55ff0289b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.892+0000 7f59dce00700 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55ff0289a320 0x55ff0289b790 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aea5d2e156db1a05 server_cookie=b75e583ac86e7b56 in_seq=0 out_seq=0 2024-04-26T01:46:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.892+0000 7f59c2a00700 10 client.25255 ms_handle_connect on v2:172.21.15.188:6836/489767141 2024-04-26T01:46:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.893+0000 7f59c2a00700 1 -- 172.21.15.6:0/1722085360 <== mds.2 v2:172.21.15.188:6836/489767141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ff027e0630 con 0x55ff0289a320 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.893+0000 7f59c2a00700 10 client.25255 handle_command_reply: tid=0 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.893+0000 7f59ea0a7b80 1 client.25255 shutdown 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.893+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1722085360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287fc10 msgr2=0x55ff02887500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.893+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287fc10 0x55ff02887500 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f59c4011fd0 tx=0x7f59c40047c0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.893+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1722085360 shutdown_connections 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.893+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55ff0289a320 0x55ff0289b790 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.893+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0288aab0 0x55ff02887a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.893+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 0x55ff0288a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.893+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/1722085360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287fc10 0x55ff02887500 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.893+0000 7f59c2a00700 0 client.25255 ms_handle_reset on v2:172.21.15.188:6836/489767141 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.893+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1722085360 >> 172.21.15.6:0/1722085360 conn(0x55ff027ae550 msgr2=0x55ff0287a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.894+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1722085360 shutdown_connections 2024-04-26T01:46:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.894+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1722085360 wait complete. 2024-04-26T01:46:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.894+0000 7f59ea0a7b80 20 client.25255 trim_cache size 0 max 16384 2024-04-26T01:46:30.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.918+0000 7f59ea0a7b80 1 Processor -- start 2024-04-26T01:46:30.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.918+0000 7f59ea0a7b80 1 -- start start 2024-04-26T01:46:30.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.919+0000 7f59ea0a7b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 0x55ff028879f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.919+0000 7f59ea0a7b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0287fc10 0x55ff0287b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.919+0000 7f59ea0a7b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0288aab0 0x55ff0287b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.919+0000 7f59ea0a7b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55ff02899ad0 con 0x55ff0287fc10 2024-04-26T01:46:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.919+0000 7f59ea0a7b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ff0289a1a0 con 0x55ff0287f2f0 2024-04-26T01:46:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.919+0000 7f59ea0a7b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ff0289a870 con 0x55ff0288aab0 2024-04-26T01:46:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.919+0000 7f59d7e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0287fc10 0x55ff0287b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.919+0000 7f59dce00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 0x55ff028879f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.919+0000 7f59dce00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 0x55ff028879f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:35904/0 (socket says 172.21.15.6:35904) 2024-04-26T01:46:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.919+0000 7f59dce00700 1 -- 172.21.15.6:0/227139863 learned_addr learned my addr 172.21.15.6:0/227139863 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.919+0000 7f59dd800700 1 --2- 172.21.15.6:0/227139863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0288aab0 0x55ff0287b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.920+0000 7f59dce00700 1 -- 172.21.15.6:0/227139863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0288aab0 msgr2=0x55ff0287b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.920+0000 7f59dce00700 1 --2- 172.21.15.6:0/227139863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0288aab0 0x55ff0287b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.920+0000 7f59dce00700 1 -- 172.21.15.6:0/227139863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0287fc10 msgr2=0x55ff0287b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:30.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.920+0000 7f59dce00700 1 --2- 172.21.15.6:0/227139863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0287fc10 0x55ff0287b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.920+0000 7f59dce00700 1 -- 172.21.15.6:0/227139863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff02885a70 con 0x55ff0287f2f0 2024-04-26T01:46:30.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.920+0000 7f59dce00700 1 --2- 172.21.15.6:0/227139863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 0x55ff028879f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f59d009a740 tx=0x7f59d000b740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93ccbd30fe012650 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:30.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.920+0000 7f59cd600700 1 -- 172.21.15.6:0/227139863 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d00a2120 con 0x55ff0287f2f0 2024-04-26T01:46:30.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.920+0000 7f59cd600700 1 -- 172.21.15.6:0/227139863 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59d00a6950 con 0x55ff0287f2f0 2024-04-26T01:46:30.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.920+0000 7f59cd600700 1 -- 172.21.15.6:0/227139863 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d009e230 con 0x55ff0287f2f0 2024-04-26T01:46:30.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.921+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/227139863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 msgr2=0x55ff028879f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:30.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.921+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/227139863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 0x55ff028879f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f59d009a740 tx=0x7f59d000b740 comp rx=0 tx=0).stop 2024-04-26T01:46:30.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.921+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/227139863 shutdown_connections 2024-04-26T01:46:30.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.921+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/227139863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0288aab0 0x55ff0287b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.921+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/227139863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287f2f0 0x55ff028879f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.921+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/227139863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0287fc10 0x55ff0287b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.921+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/227139863 >> 172.21.15.6:0/227139863 conn(0x55ff027ae550 msgr2=0x55ff028872e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.921+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/227139863 shutdown_connections 2024-04-26T01:46:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.921+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/227139863 wait complete. 2024-04-26T01:46:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.922+0000 7f59ea0a7b80 1 -- start start 2024-04-26T01:46:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.922+0000 7f59ea0a7b80 1 Processor -- start 2024-04-26T01:46:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.922+0000 7f59ea0a7b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 0x55ff02884580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.922+0000 7f59ea0a7b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287fc10 0x55ff02884ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59ea0a7b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0288aab0 0x55ff028a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59ea0a7b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55ff0289b610 con 0x55ff0288aab0 2024-04-26T01:46:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59ea0a7b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ff0289af40 con 0x55ff0287fc10 2024-04-26T01:46:30.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59ea0a7b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ff0289bce0 con 0x55ff0287f2f0 2024-04-26T01:46:30.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59ea0a7b80 10 client.0 _ll_register_callbacks cb 0x55ff0276b6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:30.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59dd800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0288aab0 0x55ff028a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59dce00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 0x55ff02884580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59d7e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287fc10 0x55ff02884ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59dce00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 0x55ff02884580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56564/0 (socket says 172.21.15.6:56564) 2024-04-26T01:46:30.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59dce00700 1 -- 172.21.15.6:0/1261483641 learned_addr learned my addr 172.21.15.6:0/1261483641 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:30.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59d7e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287fc10 0x55ff02884ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:35912/0 (socket says 172.21.15.6:35912) 2024-04-26T01:46:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59dce00700 1 -- 172.21.15.6:0/1261483641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287fc10 msgr2=0x55ff02884ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59dce00700 1 --2- 172.21.15.6:0/1261483641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287fc10 0x55ff02884ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59dce00700 1 -- 172.21.15.6:0/1261483641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0288aab0 msgr2=0x55ff028a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59dce00700 1 --2- 172.21.15.6:0/1261483641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0288aab0 0x55ff028a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.923+0000 7f59dce00700 1 -- 172.21.15.6:0/1261483641 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff028a15f0 con 0x55ff0287f2f0 2024-04-26T01:46:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.924+0000 7f59dce00700 1 --2- 172.21.15.6:0/1261483641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 0x55ff02884580 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f59d0004240 tx=0x7f59d0007df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9eaa399b78b1a7be server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.924+0000 7f59c2a00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:46:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.924+0000 7f59c2a00700 1 -- 172.21.15.6:0/1261483641 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d000d7a0 con 0x55ff0287f2f0 2024-04-26T01:46:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.924+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1261483641 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59d00a6950 con 0x55ff0287f2f0 2024-04-26T01:46:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.924+0000 7f59c2a00700 1 -- 172.21.15.6:0/1261483641 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f59d000d940 con 0x55ff0287f2f0 2024-04-26T01:46:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.925+0000 7f59c2a00700 1 -- 172.21.15.6:0/1261483641 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f59d009dfd0 con 0x55ff0287f2f0 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.925+0000 7f59ea0a7b80 10 client.25261 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.925+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1261483641 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ff02901a90 con 0x55ff0287f2f0 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.925+0000 7f59c2a00700 1 -- 172.21.15.6:0/1261483641 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59d00a6950 con 0x55ff0287f2f0 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.925+0000 7f59c2a00700 1 -- 172.21.15.6:0/1261483641 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f59d0011070 con 0x55ff0287f2f0 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.925+0000 7f59ea0a7b80 10 client.25261 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.925+0000 7f59ea0a7b80 10 client.25261 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.925+0000 7f59ea0a7b80 20 client.25261 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.925+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/1261483641 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55ff029062b0 0x55ff02904fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.925+0000 7f59ea0a7b80 4 client.25261 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.925+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1261483641 --> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] -- command(tid 0: {"prefix": "status"}) -- 0x55ff027da6d0 con 0x55ff029062b0 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.926+0000 7f59d7e00700 1 --2- 172.21.15.6:0/1261483641 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55ff029062b0 0x55ff02904fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.926+0000 7f59d7e00700 1 --2- 172.21.15.6:0/1261483641 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55ff029062b0 0x55ff02904fc0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5eac3195fb277f11 server_cookie=11da0416391d05f8 in_seq=0 out_seq=0 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.926+0000 7f59c2a00700 10 client.25261 ms_handle_connect on v2:172.21.15.188:6836/489767141 2024-04-26T01:46:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.926+0000 7f59c2a00700 1 -- 172.21.15.6:0/1261483641 <== mds.2 v2:172.21.15.188:6836/489767141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ff027da6d0 con 0x55ff029062b0 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.926+0000 7f59c2a00700 10 client.25261 handle_command_reply: tid=0 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.926+0000 7f59ea0a7b80 1 client.25261 shutdown 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.927+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1261483641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 msgr2=0x55ff02884580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.927+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/1261483641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 0x55ff02884580 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f59d0004240 tx=0x7f59d0007df0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.927+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1261483641 shutdown_connections 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.927+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/1261483641 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x55ff029062b0 0x55ff02904fc0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.927+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/1261483641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ff0288aab0 0x55ff028a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.927+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/1261483641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ff0287f2f0 0x55ff02884580 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.927+0000 7f59ea0a7b80 1 --2- 172.21.15.6:0/1261483641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ff0287fc10 0x55ff02884ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.927+0000 7f59c2a00700 0 client.25261 ms_handle_reset on v2:172.21.15.188:6836/489767141 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.927+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1261483641 >> 172.21.15.6:0/1261483641 conn(0x55ff027ae550 msgr2=0x55ff02882380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.927+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1261483641 shutdown_connections 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.927+0000 7f59ea0a7b80 1 -- 172.21.15.6:0/1261483641 wait complete. 2024-04-26T01:46:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.927+0000 7f59ea0a7b80 20 client.25261 trim_cache size 0 max 16384 2024-04-26T01:46:30.929 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:46:30.929 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:46:30.929 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 2, 2024-04-26T01:46:30.929 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 24505, 2024-04-26T01:46:30.929 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-26T01:46:30.929 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-26T01:46:30.929 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-26T01:46:30.929 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 246.45700125799999, 2024-04-26T01:46:30.929 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 26, 2024-04-26T01:46:30.929 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 96, 2024-04-26T01:46:30.929 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 96, 2024-04-26T01:46:30.930 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 301.95819584899999 2024-04-26T01:46:30.930 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:46:30.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.928+0000 7f59df800700 1 -- 172.21.15.6:0/4246669644 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f59bc0a3f50 msgr2=0x7f59bc0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:30.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.928+0000 7f59df800700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f59bc0a3f50 0x7f59bc0a6410 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f59d8102630 tx=0x7f59c4009260 comp rx=0 tx=0).stop 2024-04-26T01:46:30.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.928+0000 7f59df800700 1 -- 172.21.15.6:0/4246669644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8074c70 msgr2=0x7f59d819b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:30.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.928+0000 7f59df800700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8074c70 0x7f59d819b420 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f59d000d010 tx=0x7f59d0004d00 comp rx=0 tx=0).stop 2024-04-26T01:46:30.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.928+0000 7f59df800700 1 -- 172.21.15.6:0/4246669644 shutdown_connections 2024-04-26T01:46:30.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.928+0000 7f59df800700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f59bc0a3f50 0x7f59bc0a6410 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.928+0000 7f59df800700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59d810f470 0x7f59d819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.928+0000 7f59df800700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8074c70 0x7f59d819b420 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.928+0000 7f59df800700 1 --2- 172.21.15.6:0/4246669644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d8075590 0x7f59d819b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:30.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.928+0000 7f59df800700 1 -- 172.21.15.6:0/4246669644 >> 172.21.15.6:0/4246669644 conn(0x7f59d8079bf0 msgr2=0x7f59d8100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:30.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.929+0000 7f59df800700 1 -- 172.21.15.6:0/4246669644 shutdown_connections 2024-04-26T01:46:30.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:30.929+0000 7f59df800700 1 -- 172.21.15.6:0/4246669644 wait complete. 2024-04-26T01:46:31.890 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T01:46:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:32.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:32 smithi029 ceph-mon[32775]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T01:46:32.763 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:32 smithi006 ceph-mon[28419]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T01:46:32.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:32 smithi188 ceph-mon[32751]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T01:46:34.493 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:34 smithi006 ceph-mon[28419]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-26T01:46:34.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:34 smithi029 ceph-mon[32775]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-26T01:46:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.785+0000 7f2988200700 1 -- 172.21.15.6:0/1458787554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2980103450 msgr2=0x7f298010f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.785+0000 7f2988200700 1 --2- 172.21.15.6:0/1458787554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2980103450 0x7f298010f8b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2978009f90 tx=0x7f297800b330 comp rx=0 tx=0).stop 2024-04-26T01:46:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.785+0000 7f2988200700 1 -- 172.21.15.6:0/1458787554 shutdown_connections 2024-04-26T01:46:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.785+0000 7f2988200700 1 --2- 172.21.15.6:0/1458787554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2980103450 0x7f298010f8b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.785+0000 7f2988200700 1 --2- 172.21.15.6:0/1458787554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2980108a50 0x7f2980108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.785+0000 7f2988200700 1 --2- 172.21.15.6:0/1458787554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2980102a90 0x7f2980102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.785+0000 7f2988200700 1 -- 172.21.15.6:0/1458787554 >> 172.21.15.6:0/1458787554 conn(0x7f2980079ba0 msgr2=0x7f2980079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.786+0000 7f2988200700 1 -- 172.21.15.6:0/1458787554 shutdown_connections 2024-04-26T01:46:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.786+0000 7f2988200700 1 -- 172.21.15.6:0/1458787554 wait complete. 2024-04-26T01:46:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.787+0000 7f2988200700 1 Processor -- start 2024-04-26T01:46:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.787+0000 7f2988200700 1 -- start start 2024-04-26T01:46:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.788+0000 7f2988200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2980102a90 0x7f2980074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.788+0000 7f2988200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2980103450 0x7f2980075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.788+0000 7f2988200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2980108a50 0x7f29800757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.788+0000 7f2988200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2980112be0 con 0x7f2980103450 2024-04-26T01:46:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.788+0000 7f2988200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2980112a60 con 0x7f2980108a50 2024-04-26T01:46:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.788+0000 7f2988200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2980112d60 con 0x7f2980102a90 2024-04-26T01:46:34.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.788+0000 7f2984e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2980103450 0x7f2980075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.789+0000 7f2985800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2980102a90 0x7f2980074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.789+0000 7f2984e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2980103450 0x7f2980075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43066/0 (socket says 172.21.15.6:43066) 2024-04-26T01:46:34.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.789+0000 7f2984e00700 1 -- 172.21.15.6:0/3146143391 learned_addr learned my addr 172.21.15.6:0/3146143391 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:34.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.789+0000 7f2986200700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2980108a50 0x7f29800757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.789+0000 7f2984e00700 1 -- 172.21.15.6:0/3146143391 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2980102a90 msgr2=0x7f2980074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:34.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.789+0000 7f2984e00700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2980102a90 0x7f2980074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.789+0000 7f2984e00700 1 -- 172.21.15.6:0/3146143391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2980108a50 msgr2=0x7f29800757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:34.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.789+0000 7f2984e00700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2980108a50 0x7f29800757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.789+0000 7f2984e00700 1 -- 172.21.15.6:0/3146143391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29801a7640 con 0x7f2980103450 2024-04-26T01:46:34.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.789+0000 7f2986200700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2980108a50 0x7f29800757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:34.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.790+0000 7f2984e00700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2980103450 0x7f2980075280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f296c00e990 tx=0x7f296c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:34.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.790+0000 7f297e000700 1 -- 172.21.15.6:0/3146143391 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c00cb70 con 0x7f2980103450 2024-04-26T01:46:34.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.790+0000 7f297e000700 1 -- 172.21.15.6:0/3146143391 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f296c0044d0 con 0x7f2980103450 2024-04-26T01:46:34.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.790+0000 7f297e000700 1 -- 172.21.15.6:0/3146143391 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c005020 con 0x7f2980103450 2024-04-26T01:46:34.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.790+0000 7f2988200700 1 -- 172.21.15.6:0/3146143391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29801a7930 con 0x7f2980103450 2024-04-26T01:46:34.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.791+0000 7f2988200700 1 -- 172.21.15.6:0/3146143391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29801a7e20 con 0x7f2980103450 2024-04-26T01:46:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.794+0000 7f297e000700 1 -- 172.21.15.6:0/3146143391 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f296c004020 con 0x7f2980103450 2024-04-26T01:46:34.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.796+0000 7f2992ab8b80 1 Processor -- start 2024-04-26T01:46:34.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.796+0000 7f2992ab8b80 1 -- start start 2024-04-26T01:46:34.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.796+0000 7f297e000700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f29680a3ed0 0x7f29680a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.796+0000 7f2992ab8b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db832f0 0x55891db836d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.796+0000 7f2992ab8b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891db8e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.796+0000 7f297e000700 1 -- 172.21.15.6:0/3146143391 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f296c0d0960 con 0x7f2980103450 2024-04-26T01:46:34.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.796+0000 7f2992ab8b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 0x55891db90ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.796+0000 7f2992ab8b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55891dad6e40 con 0x55891db832f0 2024-04-26T01:46:34.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.796+0000 7f2992ab8b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55891da7d6e0 con 0x55891db8eab0 2024-04-26T01:46:34.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.796+0000 7f2992ab8b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55891dad2e00 con 0x55891db83c10 2024-04-26T01:46:34.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.797+0000 7f2984e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891db8e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.797+0000 7f2986200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 0x55891db90ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.797+0000 7f2984e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891db8e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56594/0 (socket says 172.21.15.6:56594) 2024-04-26T01:46:34.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.797+0000 7f2986200700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f29680a3ed0 0x7f29680a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.797+0000 7f2984e00700 1 -- 172.21.15.6:0/4256932663 learned_addr learned my addr 172.21.15.6:0/4256932663 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:34.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.797+0000 7f2985800700 1 --2- 172.21.15.6:0/4256932663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db832f0 0x55891db836d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.797+0000 7f2984e00700 1 -- 172.21.15.6:0/4256932663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 msgr2=0x55891db90ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:34.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.797+0000 7f2984e00700 1 --2- 172.21.15.6:0/4256932663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 0x55891db90ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.797+0000 7f2984e00700 1 -- 172.21.15.6:0/4256932663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db832f0 msgr2=0x55891db836d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:34.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.797+0000 7f2984e00700 1 --2- 172.21.15.6:0/4256932663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db832f0 0x55891db836d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.798+0000 7f2984e00700 1 -- 172.21.15.6:0/4256932663 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55891da73630 con 0x55891db83c10 2024-04-26T01:46:34.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.798+0000 7f2986200700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f29680a3ed0 0x7f29680a6390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f297800b6c0 tx=0x7f2978009320 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:34.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.798+0000 7f2984e00700 1 --2- 172.21.15.6:0/4256932663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891db8e570 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f296c099000 tx=0x7f296c09c1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55b58bd02d077a49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:34.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.799+0000 7f2972a00700 1 -- 172.21.15.6:0/4256932663 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c01f040 con 0x55891db83c10 2024-04-26T01:46:34.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.799+0000 7f2972a00700 1 -- 172.21.15.6:0/4256932663 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f296c004310 con 0x55891db83c10 2024-04-26T01:46:34.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.799+0000 7f2972a00700 1 -- 172.21.15.6:0/4256932663 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c0a52a0 con 0x55891db83c10 2024-04-26T01:46:34.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.799+0000 7f2992ab8b80 1 -- 172.21.15.6:0/4256932663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 msgr2=0x55891db8e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:34.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.799+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/4256932663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891db8e570 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f296c099000 tx=0x7f296c09c1f0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.800+0000 7f2992ab8b80 1 -- 172.21.15.6:0/4256932663 shutdown_connections 2024-04-26T01:46:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.800+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/4256932663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 0x55891db90ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.800+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/4256932663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db832f0 0x55891db836d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.800+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/4256932663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891db8e570 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.800+0000 7f2992ab8b80 1 -- 172.21.15.6:0/4256932663 >> 172.21.15.6:0/4256932663 conn(0x55891dab2550 msgr2=0x55891db7f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.800+0000 7f2992ab8b80 1 -- 172.21.15.6:0/4256932663 shutdown_connections 2024-04-26T01:46:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.800+0000 7f2992ab8b80 1 -- 172.21.15.6:0/4256932663 wait complete. 2024-04-26T01:46:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.800+0000 7f2992ab8b80 1 -- start start 2024-04-26T01:46:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.801+0000 7f2992ab8b80 1 Processor -- start 2024-04-26T01:46:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.801+0000 7f2992ab8b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db832f0 0x55891db8e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.801+0000 7f2992ab8b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891db8b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.801+0000 7f2992ab8b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db8eab0 0x55891db8ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.801+0000 7f2992ab8b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55891db915a0 con 0x55891db8eab0 2024-04-26T01:46:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.801+0000 7f2992ab8b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55891da90810 con 0x55891db832f0 2024-04-26T01:46:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.801+0000 7f2992ab8b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55891db91720 con 0x55891db83c10 2024-04-26T01:46:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.801+0000 7f2992ab8b80 10 client.0 _ll_register_callbacks cb 0x55891da6f6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2984e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891db8b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2985800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db832f0 0x55891db8e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2984e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891db8b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56610/0 (socket says 172.21.15.6:56610) 2024-04-26T01:46:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2984e00700 1 -- 172.21.15.6:0/690226883 learned_addr learned my addr 172.21.15.6:0/690226883 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2986200700 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db8eab0 0x55891db8ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2984e00700 1 -- 172.21.15.6:0/690226883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db832f0 msgr2=0x55891db8e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2984e00700 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db832f0 0x55891db8e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2984e00700 1 -- 172.21.15.6:0/690226883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db8eab0 msgr2=0x55891db8ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2984e00700 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db8eab0 0x55891db8ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2984e00700 1 -- 172.21.15.6:0/690226883 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55891da24880 con 0x55891db83c10 2024-04-26T01:46:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2986200700 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db8eab0 0x55891db8ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2984e00700 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891db8b550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f296c00c550 tx=0x7f296c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee99cb04ab5bc3ae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.802+0000 7f2967400700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:46:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.803+0000 7f2967400700 1 -- 172.21.15.6:0/690226883 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c0b9020 con 0x55891db83c10 2024-04-26T01:46:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.803+0000 7f2967400700 1 -- 172.21.15.6:0/690226883 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f296c09a930 con 0x55891db83c10 2024-04-26T01:46:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.803+0000 7f2992ab8b80 1 -- 172.21.15.6:0/690226883 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55891dae4630 con 0x55891db83c10 2024-04-26T01:46:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.803+0000 7f2967400700 1 -- 172.21.15.6:0/690226883 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f296c0b3030 con 0x55891db83c10 2024-04-26T01:46:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.803+0000 7f2992ab8b80 10 client.25273 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.803+0000 7f2992ab8b80 1 -- 172.21.15.6:0/690226883 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55891da6daa0 con 0x55891db83c10 2024-04-26T01:46:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.803+0000 7f2967400700 1 -- 172.21.15.6:0/690226883 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55891dae4630 con 0x55891db83c10 2024-04-26T01:46:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.804+0000 7f2967400700 1 -- 172.21.15.6:0/690226883 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f296c00f040 con 0x55891db83c10 2024-04-26T01:46:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.804+0000 7f2992ab8b80 10 client.25273 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.804+0000 7f2992ab8b80 10 client.25273 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:46:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.804+0000 7f2992ab8b80 20 client.25273 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.804+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x55891db9cff0 0x55891db9f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.804+0000 7f2992ab8b80 4 client.25273 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.804+0000 7f2992ab8b80 1 -- 172.21.15.6:0/690226883 --> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55891dade6d0 con 0x55891db9cff0 2024-04-26T01:46:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.806+0000 7f2985800700 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x55891db9cff0 0x55891db9f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.807+0000 7f2985800700 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x55891db9cff0 0x55891db9f190 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=70a0e30262ada1ff server_cookie=a09ad6fba8c78921 in_seq=0 out_seq=0 2024-04-26T01:46:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.807+0000 7f2967400700 10 client.25273 ms_handle_connect on v2:172.21.15.29:6838/2325582359 2024-04-26T01:46:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2967400700 1 -- 172.21.15.6:0/690226883 <== mds.1 v2:172.21.15.29:6838/2325582359 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55891dade6d0 con 0x55891db9cff0 2024-04-26T01:46:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2967400700 10 client.25273 handle_command_reply: tid=0 2024-04-26T01:46:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2992ab8b80 1 client.25273 shutdown 2024-04-26T01:46:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2992ab8b80 1 -- 172.21.15.6:0/690226883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 msgr2=0x55891db8b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891db8b550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f296c00c550 tx=0x7f296c000bc0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2992ab8b80 1 -- 172.21.15.6:0/690226883 shutdown_connections 2024-04-26T01:46:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db8eab0 0x55891db8ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x55891db9cff0 0x55891db9f190 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db832f0 0x55891db8e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2967400700 0 client.25273 ms_handle_reset on v2:172.21.15.29:6838/2325582359 2024-04-26T01:46:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/690226883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891db8b550 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2992ab8b80 1 -- 172.21.15.6:0/690226883 >> 172.21.15.6:0/690226883 conn(0x55891dab2550 msgr2=0x55891db85c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2992ab8b80 1 -- 172.21.15.6:0/690226883 shutdown_connections 2024-04-26T01:46:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.808+0000 7f2992ab8b80 1 -- 172.21.15.6:0/690226883 wait complete. 2024-04-26T01:46:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.809+0000 7f2992ab8b80 20 client.25273 trim_cache size 0 max 16384 2024-04-26T01:46:34.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.831+0000 7f2992ab8b80 1 Processor -- start 2024-04-26T01:46:34.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.831+0000 7f2992ab8b80 1 -- start start 2024-04-26T01:46:34.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.831+0000 7f2992ab8b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db832f0 0x55891db9d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.831+0000 7f2992ab8b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db83c10 0x55891db8a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.831+0000 7f2992ab8b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 0x55891db86c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.831+0000 7f2992ab8b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55891db9d4d0 con 0x55891db83c10 2024-04-26T01:46:34.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.831+0000 7f2992ab8b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55891db9dba0 con 0x55891db8eab0 2024-04-26T01:46:34.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.831+0000 7f2992ab8b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55891db9e270 con 0x55891db832f0 2024-04-26T01:46:34.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.832+0000 7f2985800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db832f0 0x55891db9d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.832+0000 7f2985800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db832f0 0x55891db9d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56620/0 (socket says 172.21.15.6:56620) 2024-04-26T01:46:34.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.832+0000 7f2985800700 1 -- 172.21.15.6:0/2981253791 learned_addr learned my addr 172.21.15.6:0/2981253791 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:34.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.832+0000 7f2986200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 0x55891db86c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.832+0000 7f2984e00700 1 --2- 172.21.15.6:0/2981253791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db83c10 0x55891db8a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.832+0000 7f2985800700 1 -- 172.21.15.6:0/2981253791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 msgr2=0x55891db86c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:34.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.832+0000 7f2985800700 1 --2- 172.21.15.6:0/2981253791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 0x55891db86c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.832+0000 7f2985800700 1 -- 172.21.15.6:0/2981253791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db83c10 msgr2=0x55891db8a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:34.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.832+0000 7f2985800700 1 --2- 172.21.15.6:0/2981253791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db83c10 0x55891db8a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.832+0000 7f2985800700 1 -- 172.21.15.6:0/2981253791 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55891dba00f0 con 0x55891db832f0 2024-04-26T01:46:34.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.833+0000 7f2985800700 1 --2- 172.21.15.6:0/2981253791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db832f0 0x55891db9d650 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x55891db9f9f0 tx=0x7f2974011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=758b0d1ba1e89d43 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:34.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.833+0000 7f2972000700 1 -- 172.21.15.6:0/2981253791 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2974011cf0 con 0x55891db832f0 2024-04-26T01:46:34.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.833+0000 7f2972000700 1 -- 172.21.15.6:0/2981253791 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f297401d8e0 con 0x55891db832f0 2024-04-26T01:46:34.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.833+0000 7f2972000700 1 -- 172.21.15.6:0/2981253791 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2974010c50 con 0x55891db832f0 2024-04-26T01:46:34.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.834+0000 7f2992ab8b80 1 -- 172.21.15.6:0/2981253791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db832f0 msgr2=0x55891db9d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:34.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.834+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/2981253791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db832f0 0x55891db9d650 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x55891db9f9f0 tx=0x7f2974011720 comp rx=0 tx=0).stop 2024-04-26T01:46:34.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.834+0000 7f2992ab8b80 1 -- 172.21.15.6:0/2981253791 shutdown_connections 2024-04-26T01:46:34.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.834+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/2981253791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 0x55891db86c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.834+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/2981253791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db832f0 0x55891db9d650 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.834+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/2981253791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db83c10 0x55891db8a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.834+0000 7f2992ab8b80 1 -- 172.21.15.6:0/2981253791 >> 172.21.15.6:0/2981253791 conn(0x55891dab2550 msgr2=0x55891db86510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:34.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.834+0000 7f2992ab8b80 1 -- 172.21.15.6:0/2981253791 shutdown_connections 2024-04-26T01:46:34.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.834+0000 7f2992ab8b80 1 -- 172.21.15.6:0/2981253791 wait complete. 2024-04-26T01:46:34.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.835+0000 7f2992ab8b80 1 -- start start 2024-04-26T01:46:34.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.835+0000 7f2992ab8b80 1 Processor -- start 2024-04-26T01:46:34.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.835+0000 7f2992ab8b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db832f0 0x55891dba1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.835+0000 7f2992ab8b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891dba1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.835+0000 7f2992ab8b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 0x55891dba6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.835+0000 7f2992ab8b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55891db9f010 con 0x55891db832f0 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.835+0000 7f2992ab8b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55891db9e940 con 0x55891db8eab0 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.835+0000 7f2992ab8b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55891db9f6e0 con 0x55891db83c10 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.836+0000 7f2992ab8b80 10 client.0 _ll_register_callbacks cb 0x55891da6f6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.836+0000 7f2985800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db832f0 0x55891dba1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.836+0000 7f2985800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db832f0 0x55891dba1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43098/0 (socket says 172.21.15.6:43098) 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.836+0000 7f2985800700 1 -- 172.21.15.6:0/3683460032 learned_addr learned my addr 172.21.15.6:0/3683460032 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.836+0000 7f2984e00700 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891dba1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.836+0000 7f2986200700 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 0x55891dba6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.836+0000 7f2985800700 1 -- 172.21.15.6:0/3683460032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 msgr2=0x55891dba1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.836+0000 7f2985800700 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891dba1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.836+0000 7f2985800700 1 -- 172.21.15.6:0/3683460032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 msgr2=0x55891dba6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:34.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.836+0000 7f2985800700 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 0x55891dba6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.836+0000 7f2985800700 1 -- 172.21.15.6:0/3683460032 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55891dba6c40 con 0x55891db832f0 2024-04-26T01:46:34.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.836+0000 7f2984e00700 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891dba1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:34.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.837+0000 7f2985800700 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db832f0 0x55891dba1790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f29740107d0 tx=0x7f2974010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3e81b85a74a229 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:34.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.837+0000 7f2967400700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:46:34.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.837+0000 7f2967400700 1 -- 172.21.15.6:0/3683460032 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297401d630 con 0x55891db832f0 2024-04-26T01:46:34.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.837+0000 7f2967400700 1 -- 172.21.15.6:0/3683460032 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f2974011af0 con 0x55891db832f0 2024-04-26T01:46:34.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.837+0000 7f2992ab8b80 1 -- 172.21.15.6:0/3683460032 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f297401d8e0 con 0x55891db832f0 2024-04-26T01:46:34.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.838+0000 7f2967400700 1 -- 172.21.15.6:0/3683460032 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2974011d30 con 0x55891db832f0 2024-04-26T01:46:34.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.838+0000 7f2992ab8b80 10 client.15135 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:34.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.838+0000 7f2992ab8b80 1 -- 172.21.15.6:0/3683460032 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55891dba7f70 con 0x55891db832f0 2024-04-26T01:46:34.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.838+0000 7f2967400700 1 -- 172.21.15.6:0/3683460032 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f297401d8e0 con 0x55891db832f0 2024-04-26T01:46:34.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.838+0000 7f2967400700 1 -- 172.21.15.6:0/3683460032 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f29740053b0 con 0x55891db832f0 2024-04-26T01:46:34.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.838+0000 7f2992ab8b80 10 client.15135 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:34.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.838+0000 7f2992ab8b80 10 client.15135 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:46:34.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.838+0000 7f2992ab8b80 20 client.15135 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:34.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.838+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x55891dc0a6f0 0x55891dc09400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:34.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.838+0000 7f2992ab8b80 4 client.15135 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:46:34.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.838+0000 7f2992ab8b80 1 -- 172.21.15.6:0/3683460032 --> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] -- command(tid 0: {"prefix": "status"}) -- 0x55891d957200 con 0x55891dc0a6f0 2024-04-26T01:46:34.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.839+0000 7f2984e00700 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x55891dc0a6f0 0x55891dc09400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.839+0000 7f2984e00700 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x55891dc0a6f0 0x55891dc09400 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=27199e19694f6b47 server_cookie=65b300107ec826d5 in_seq=0 out_seq=0 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.839+0000 7f2967400700 10 client.15135 ms_handle_connect on v2:172.21.15.29:6838/2325582359 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.839+0000 7f2967400700 1 -- 172.21.15.6:0/3683460032 <== mds.1 v2:172.21.15.29:6838/2325582359 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55891d957200 con 0x55891dc0a6f0 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.839+0000 7f2967400700 10 client.15135 handle_command_reply: tid=0 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.839+0000 7f2992ab8b80 1 client.15135 shutdown 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.840+0000 7f2992ab8b80 1 -- 172.21.15.6:0/3683460032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db832f0 msgr2=0x55891dba1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.840+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db832f0 0x55891dba1790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f29740107d0 tx=0x7f2974010430 comp rx=0 tx=0).stop 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.840+0000 7f2992ab8b80 1 -- 172.21.15.6:0/3683460032 shutdown_connections 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.840+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x55891dc0a6f0 0x55891dc09400 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.840+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55891db8eab0 0x55891dba6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.840+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55891db832f0 0x55891dba1790 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.840+0000 7f2992ab8b80 1 --2- 172.21.15.6:0/3683460032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55891db83c10 0x55891dba1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.840+0000 7f2992ab8b80 1 -- 172.21.15.6:0/3683460032 >> 172.21.15.6:0/3683460032 conn(0x55891dab2550 msgr2=0x55891db7e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:34.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.840+0000 7f2967400700 0 client.15135 ms_handle_reset on v2:172.21.15.29:6838/2325582359 2024-04-26T01:46:34.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.841+0000 7f2992ab8b80 1 -- 172.21.15.6:0/3683460032 shutdown_connections 2024-04-26T01:46:34.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.841+0000 7f2992ab8b80 1 -- 172.21.15.6:0/3683460032 wait complete. 2024-04-26T01:46:34.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.841+0000 7f2992ab8b80 20 client.15135 trim_cache size 0 max 16384 2024-04-26T01:46:34.841 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:46:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:46:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 1, 2024-04-26T01:46:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 24512, 2024-04-26T01:46:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-26T01:46:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-26T01:46:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-26T01:46:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 251.38067423699999, 2024-04-26T01:46:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 26, 2024-04-26T01:46:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 96, 2024-04-26T01:46:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 96, 2024-04-26T01:46:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 285.95015876299999 2024-04-26T01:46:34.842 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:46:34.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.842+0000 7f2988200700 1 -- 172.21.15.6:0/3146143391 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f29680a3ed0 msgr2=0x7f29680a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:34.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.842+0000 7f2988200700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f29680a3ed0 0x7f29680a6390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f297800b6c0 tx=0x7f2978009320 comp rx=0 tx=0).stop 2024-04-26T01:46:34.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.842+0000 7f2988200700 1 -- 172.21.15.6:0/3146143391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2980103450 msgr2=0x7f2980075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:34.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.842+0000 7f2988200700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2980103450 0x7f2980075280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f296c00e990 tx=0x7f296c00ed50 comp rx=0 tx=0).stop 2024-04-26T01:46:34.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.842+0000 7f2988200700 1 -- 172.21.15.6:0/3146143391 shutdown_connections 2024-04-26T01:46:34.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.842+0000 7f2988200700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f29680a3ed0 0x7f29680a6390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.842+0000 7f2988200700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2980108a50 0x7f29800757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.842+0000 7f2988200700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2980102a90 0x7f2980074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.842+0000 7f2988200700 1 --2- 172.21.15.6:0/3146143391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2980103450 0x7f2980075280 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:34.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.842+0000 7f2988200700 1 -- 172.21.15.6:0/3146143391 >> 172.21.15.6:0/3146143391 conn(0x7f2980079ba0 msgr2=0x7f2980101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:34.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.842+0000 7f2988200700 1 -- 172.21.15.6:0/3146143391 shutdown_connections 2024-04-26T01:46:34.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:34.842+0000 7f2988200700 1 -- 172.21.15.6:0/3146143391 wait complete. 2024-04-26T01:46:34.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:34 smithi188 ceph-mon[32751]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-26T01:46:35.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T01:46:36.233 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:36.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:36 smithi006 ceph-mon[28419]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T01:46:36.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:36 smithi029 ceph-mon[32775]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T01:46:36.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:36 smithi188 ceph-mon[32751]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T01:46:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.366+0000 7f28b5a00640 1 -- 172.21.15.6:0/1211341943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28b0100ed0 msgr2=0x7f28b01012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.366+0000 7f28b5a00640 1 --2- 172.21.15.6:0/1211341943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28b0100ed0 0x7f28b01012b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f28a0009900 tx=0x7f28a002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.366+0000 7f28b5a00640 1 -- 172.21.15.6:0/1211341943 shutdown_connections 2024-04-26T01:46:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.366+0000 7f28b5a00640 1 --2- 172.21.15.6:0/1211341943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b0105fa0 0x7f28b010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.366+0000 7f28b5a00640 1 --2- 172.21.15.6:0/1211341943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28b0101880 0x7f28b01058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.366+0000 7f28b5a00640 1 --2- 172.21.15.6:0/1211341943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28b0100ed0 0x7f28b01012b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.366+0000 7f28b5a00640 1 -- 172.21.15.6:0/1211341943 >> 172.21.15.6:0/1211341943 conn(0x7f28b00fc730 msgr2=0x7f28b00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.366+0000 7f28b5a00640 1 -- 172.21.15.6:0/1211341943 shutdown_connections 2024-04-26T01:46:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.366+0000 7f28b5a00640 1 -- 172.21.15.6:0/1211341943 wait complete. 2024-04-26T01:46:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.367+0000 7f28b5a00640 1 Processor -- start 2024-04-26T01:46:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.368+0000 7f28b5a00640 1 -- start start 2024-04-26T01:46:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.368+0000 7f28b5a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b0100ed0 0x7f28b01a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.368+0000 7f28b5a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28b0105fa0 0x7f28b01a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.368+0000 7f28b5a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28b019a5a0 0x7f28b019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.368+0000 7f28b5a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f28b0110f00 con 0x7f28b019a5a0 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.368+0000 7f28b5a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f28b0110d80 con 0x7f28b0100ed0 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.368+0000 7f28b5a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f28b0111080 con 0x7f28b0105fa0 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.368+0000 7f28afe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b0100ed0 0x7f28b01a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.368+0000 7f28afe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b0100ed0 0x7f28b01a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:35996/0 (socket says 172.21.15.6:35996) 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.368+0000 7f28afe00640 1 -- 172.21.15.6:0/1164131127 learned_addr learned my addr 172.21.15.6:0/1164131127 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.368+0000 7f28b5000640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28b019a5a0 0x7f28b019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.368+0000 7f28af400640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28b0105fa0 0x7f28b01a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.369+0000 7f28b5000640 1 -- 172.21.15.6:0/1164131127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28b0105fa0 msgr2=0x7f28b01a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.369+0000 7f28b5000640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28b0105fa0 0x7f28b01a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.369+0000 7f28b5000640 1 -- 172.21.15.6:0/1164131127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b0100ed0 msgr2=0x7f28b01a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.369+0000 7f28b5000640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b0100ed0 0x7f28b01a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.369+0000 7f28b5000640 1 -- 172.21.15.6:0/1164131127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b019b0c0 con 0x7f28b019a5a0 2024-04-26T01:46:37.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.369+0000 7f28b5000640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28b019a5a0 0x7f28b019aa50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f28a800e970 tx=0x7f28a800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:37.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.369+0000 7f28acc00640 1 -- 172.21.15.6:0/1164131127 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a800ccc0 con 0x7f28b019a5a0 2024-04-26T01:46:37.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.369+0000 7f28acc00640 1 -- 172.21.15.6:0/1164131127 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28a8005020 con 0x7f28b019a5a0 2024-04-26T01:46:37.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.369+0000 7f28b5a00640 1 -- 172.21.15.6:0/1164131127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b019b3b0 con 0x7f28b019a5a0 2024-04-26T01:46:37.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.369+0000 7f28acc00640 1 -- 172.21.15.6:0/1164131127 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a8004a00 con 0x7f28b019a5a0 2024-04-26T01:46:37.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.370+0000 7f28b5a00640 1 -- 172.21.15.6:0/1164131127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b01a7600 con 0x7f28b019a5a0 2024-04-26T01:46:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.373+0000 7f28b9d8c740 1 Processor -- start 2024-04-26T01:46:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.373+0000 7f28acc00640 1 -- 172.21.15.6:0/1164131127 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f28a8004ba0 con 0x7f28b019a5a0 2024-04-26T01:46:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.373+0000 7f28b9d8c740 1 -- start start 2024-04-26T01:46:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.374+0000 7f28b9d8c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7c9b0 0x55f114d7cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.374+0000 7f28b9d8c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7d2d0 0x55f114d87650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.374+0000 7f28b9d8c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d87b90 0x55f114d89f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.374+0000 7f28b9d8c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55f114ce2900 con 0x55f114d87b90 2024-04-26T01:46:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.374+0000 7f28b9d8c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f114c88ef0 con 0x55f114d7c9b0 2024-04-26T01:46:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.374+0000 7f28b9d8c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f114cde8c0 con 0x55f114d7d2d0 2024-04-26T01:46:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.374+0000 7f28af400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7c9b0 0x55f114d7cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.374+0000 7f28afe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7d2d0 0x55f114d87650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.374+0000 7f28af400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7c9b0 0x55f114d7cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:35998/0 (socket says 172.21.15.6:35998) 2024-04-26T01:46:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.374+0000 7f28af400640 1 -- 172.21.15.6:0/1520458193 learned_addr learned my addr 172.21.15.6:0/1520458193 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.374+0000 7f28af400640 1 --2- 172.21.15.6:0/1520458193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d87b90 0x55f114d89f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.375+0000 7f28afe00640 1 -- 172.21.15.6:0/1520458193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7c9b0 msgr2=0x55f114d7cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.375+0000 7f28afe00640 1 --2- 172.21.15.6:0/1520458193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7c9b0 0x55f114d7cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.375+0000 7f28afe00640 1 -- 172.21.15.6:0/1520458193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d87b90 msgr2=0x55f114d89f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.375+0000 7f28afe00640 1 --2- 172.21.15.6:0/1520458193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d87b90 0x55f114d89f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.375+0000 7f28afe00640 1 -- 172.21.15.6:0/1520458193 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1149f5210 con 0x55f114d7d2d0 2024-04-26T01:46:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.375+0000 7f28acc00640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f28940a3ea0 0x7f28940a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.375+0000 7f28acc00640 1 -- 172.21.15.6:0/1164131127 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f28a80d09b0 con 0x7f28b019a5a0 2024-04-26T01:46:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.375+0000 7f28af400640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f28940a3ea0 0x7f28940a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.375+0000 7f28afe00640 1 --2- 172.21.15.6:0/1520458193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7d2d0 0x55f114d87650 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f28a0004770 tx=0x7f28a0031b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc7cec1668ef9721 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.375+0000 7f28a5600640 1 -- 172.21.15.6:0/1520458193 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a0040070 con 0x55f114d7d2d0 2024-04-26T01:46:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.375+0000 7f28a5600640 1 -- 172.21.15.6:0/1520458193 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28a0004030 con 0x55f114d7d2d0 2024-04-26T01:46:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.375+0000 7f28a5600640 1 -- 172.21.15.6:0/1520458193 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a0035040 con 0x55f114d7d2d0 2024-04-26T01:46:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.376+0000 7f28b9d8c740 1 -- 172.21.15.6:0/1520458193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7d2d0 msgr2=0x55f114d87650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.376+0000 7f28af400640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f28940a3ea0 0x7f28940a6360 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x55f114d7a2e0 tx=0x7f2898004ab0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.376+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/1520458193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7d2d0 0x55f114d87650 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f28a0004770 tx=0x7f28a0031b10 comp rx=0 tx=0).stop 2024-04-26T01:46:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.376+0000 7f28b9d8c740 1 -- 172.21.15.6:0/1520458193 shutdown_connections 2024-04-26T01:46:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.376+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/1520458193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d87b90 0x55f114d89f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.376+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/1520458193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7d2d0 0x55f114d87650 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.376+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/1520458193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7c9b0 0x55f114d7cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.376+0000 7f28b9d8c740 1 -- 172.21.15.6:0/1520458193 >> 172.21.15.6:0/1520458193 conn(0x55f114d78ec0 msgr2=0x55f114d792a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.376+0000 7f28b9d8c740 1 -- 172.21.15.6:0/1520458193 shutdown_connections 2024-04-26T01:46:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.377+0000 7f28b9d8c740 1 -- 172.21.15.6:0/1520458193 wait complete. 2024-04-26T01:46:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.377+0000 7f28b9d8c740 1 -- start start 2024-04-26T01:46:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.377+0000 7f28b9d8c740 1 Processor -- start 2024-04-26T01:46:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.378+0000 7f28b9d8c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 0x55f114d84b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.378+0000 7f28b9d8c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7d2d0 0x55f114d85040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.378+0000 7f28b9d8c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d87b90 0x55f114d85580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.378+0000 7f28b9d8c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55f114d8a760 con 0x55f114d87b90 2024-04-26T01:46:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.378+0000 7f28b9d8c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f114c9c310 con 0x55f114d7d2d0 2024-04-26T01:46:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.378+0000 7f28b9d8c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f114d8a8e0 con 0x55f114d7c9b0 2024-04-26T01:46:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.378+0000 7f28b9d8c740 10 client.0 _ll_register_callbacks cb 0x55f114c6acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.378+0000 7f28afe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 0x55f114d84b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.378+0000 7f28af400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7d2d0 0x55f114d85040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.378+0000 7f28afe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 0x55f114d84b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56654/0 (socket says 172.21.15.6:56654) 2024-04-26T01:46:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.378+0000 7f28b5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d87b90 0x55f114d85580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.378+0000 7f28afe00640 1 -- 172.21.15.6:0/536419504 learned_addr learned my addr 172.21.15.6:0/536419504 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:37.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.379+0000 7f28afe00640 1 -- 172.21.15.6:0/536419504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7d2d0 msgr2=0x55f114d85040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:37.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.379+0000 7f28afe00640 1 --2- 172.21.15.6:0/536419504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7d2d0 0x55f114d85040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.379+0000 7f28afe00640 1 -- 172.21.15.6:0/536419504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d87b90 msgr2=0x55f114d85580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:37.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.379+0000 7f28afe00640 1 --2- 172.21.15.6:0/536419504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d87b90 0x55f114d85580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.379+0000 7f28afe00640 1 -- 172.21.15.6:0/536419504 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f114ac1620 con 0x55f114d7c9b0 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.379+0000 7f28b5000640 1 --2- 172.21.15.6:0/536419504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d87b90 0x55f114d85580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.379+0000 7f28afe00640 1 --2- 172.21.15.6:0/536419504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 0x55f114d84b00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f28a0004490 tx=0x7f28a00044c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0b2f99b29793a3e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.379+0000 7f289e000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.379+0000 7f289e000640 1 -- 172.21.15.6:0/536419504 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a004c020 con 0x55f114d7c9b0 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.379+0000 7f289e000640 1 -- 172.21.15.6:0/536419504 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f28a00314a0 con 0x55f114d7c9b0 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.379+0000 7f28b9d8c740 1 -- 172.21.15.6:0/536419504 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28a0004030 con 0x55f114d7c9b0 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.379+0000 7f289e000640 1 -- 172.21.15.6:0/536419504 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f28a0057050 con 0x55f114d7c9b0 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.380+0000 7f28b9d8c740 10 client.25291 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.380+0000 7f28b9d8c740 1 -- 172.21.15.6:0/536419504 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f114d8a760 con 0x55f114d7c9b0 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.380+0000 7f289e000640 1 -- 172.21.15.6:0/536419504 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28a0004030 con 0x55f114d7c9b0 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.380+0000 7f289e000640 1 -- 172.21.15.6:0/536419504 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f28a00317a0 con 0x55f114d7c9b0 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.380+0000 7f28b9d8c740 10 client.25291 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.380+0000 7f28b9d8c740 10 client.25291 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.380+0000 7f28b9d8c740 20 client.25291 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.380+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/536419504 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f114d97b20 0x55f114d99f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.380+0000 7f28b9d8c740 4 client.25291 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.380+0000 7f28b9d8c740 1 -- 172.21.15.6:0/536419504 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f114cf68b0 con 0x55f114d97b20 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.381+0000 7f28af400640 1 --2- 172.21.15.6:0/536419504 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f114d97b20 0x55f114d99f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.381+0000 7f289e000640 10 client.25291 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.381+0000 7f28af400640 1 --2- 172.21.15.6:0/536419504 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f114d97b20 0x55f114d99f10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da27f81283829d1c server_cookie=20b501c194392b48 in_seq=0 out_seq=0 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.382+0000 7f289e000640 1 -- 172.21.15.6:0/536419504 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f114cf68b0 con 0x55f114d97b20 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.382+0000 7f289e000640 10 client.25291 handle_command_reply: tid=0 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.382+0000 7f28b9d8c740 1 client.25291 shutdown 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.382+0000 7f28b9d8c740 1 -- 172.21.15.6:0/536419504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 msgr2=0x55f114d84b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.382+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/536419504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 0x55f114d84b00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f28a0004490 tx=0x7f28a00044c0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.383+0000 7f28b9d8c740 1 -- 172.21.15.6:0/536419504 shutdown_connections 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.383+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/536419504 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f114d97b20 0x55f114d99f10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.383+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/536419504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d87b90 0x55f114d85580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.383+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/536419504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7d2d0 0x55f114d85040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.383+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/536419504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 0x55f114d84b00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.383+0000 7f28b9d8c740 1 -- 172.21.15.6:0/536419504 >> 172.21.15.6:0/536419504 conn(0x55f114d78ec0 msgr2=0x55f114d7ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:37.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.383+0000 7f289e000640 0 client.25291 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:37.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.383+0000 7f28b9d8c740 1 -- 172.21.15.6:0/536419504 shutdown_connections 2024-04-26T01:46:37.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.383+0000 7f28b9d8c740 1 -- 172.21.15.6:0/536419504 wait complete. 2024-04-26T01:46:37.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.383+0000 7f28b9d8c740 20 client.25291 trim_cache size 0 max 16384 2024-04-26T01:46:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28b9d8c740 1 Processor -- start 2024-04-26T01:46:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28b9d8c740 1 -- start start 2024-04-26T01:46:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28b9d8c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 0x55f114d93620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28b9d8c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 0x55f114d7ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28b9d8c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d87b90 0x55f114d9f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28b9d8c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f28b01b2570 con 0x55f114d7d2d0 2024-04-26T01:46:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28b9d8c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f28b01afe40 con 0x55f114d87b90 2024-04-26T01:46:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28b9d8c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f28b01adc90 con 0x55f114d7c9b0 2024-04-26T01:46:37.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28af400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 0x55f114d7ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28afe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 0x55f114d93620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28b5000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d87b90 0x55f114d9f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28af400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 0x55f114d7ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43132/0 (socket says 172.21.15.6:43132) 2024-04-26T01:46:37.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.400+0000 7f28af400640 1 -- 172.21.15.6:0/1278969740 learned_addr learned my addr 172.21.15.6:0/1278969740 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:37.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.401+0000 7f28afe00640 1 -- 172.21.15.6:0/1278969740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d87b90 msgr2=0x55f114d9f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:37.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.401+0000 7f28afe00640 1 --2- 172.21.15.6:0/1278969740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d87b90 0x55f114d9f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.401+0000 7f28afe00640 1 -- 172.21.15.6:0/1278969740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 msgr2=0x55f114d7ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:37.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.401+0000 7f28afe00640 1 --2- 172.21.15.6:0/1278969740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 0x55f114d7ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.401+0000 7f28afe00640 1 -- 172.21.15.6:0/1278969740 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f114d7dc60 con 0x55f114d7c9b0 2024-04-26T01:46:37.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.401+0000 7f28af400640 1 --2- 172.21.15.6:0/1278969740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 0x55f114d7ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:37.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.401+0000 7f28b5000640 1 --2- 172.21.15.6:0/1278969740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d87b90 0x55f114d9f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.401+0000 7f28afe00640 1 --2- 172.21.15.6:0/1278969740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 0x55f114d93620 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f28a0003690 tx=0x7f28a002a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58cf07391c8ba32b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.402+0000 7f28a4c00640 1 -- 172.21.15.6:0/1278969740 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a004c040 con 0x55f114d7c9b0 2024-04-26T01:46:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.402+0000 7f28a4c00640 1 -- 172.21.15.6:0/1278969740 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28a0031950 con 0x55f114d7c9b0 2024-04-26T01:46:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.402+0000 7f28a4c00640 1 -- 172.21.15.6:0/1278969740 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a0040070 con 0x55f114d7c9b0 2024-04-26T01:46:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.402+0000 7f28b9d8c740 1 -- 172.21.15.6:0/1278969740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 msgr2=0x55f114d93620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.402+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/1278969740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 0x55f114d93620 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f28a0003690 tx=0x7f28a002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.402+0000 7f28b9d8c740 1 -- 172.21.15.6:0/1278969740 shutdown_connections 2024-04-26T01:46:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.402+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/1278969740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 0x55f114d7ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.402+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/1278969740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d87b90 0x55f114d9f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.402+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/1278969740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d7c9b0 0x55f114d93620 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.402+0000 7f28b9d8c740 1 -- 172.21.15.6:0/1278969740 >> 172.21.15.6:0/1278969740 conn(0x55f114d78ec0 msgr2=0x55f114d7e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.402+0000 7f28b9d8c740 1 -- 172.21.15.6:0/1278969740 shutdown_connections 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.402+0000 7f28b9d8c740 1 -- 172.21.15.6:0/1278969740 wait complete. 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.403+0000 7f28b9d8c740 1 -- start start 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.403+0000 7f28b9d8c740 1 Processor -- start 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28b9d8c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7c9b0 0x55f114d8fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28b9d8c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 0x55f114d8d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28b9d8c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d87b90 0x55f114d8d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28b9d8c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f28b01ae300 con 0x55f114d7d2d0 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28b9d8c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f28b01af770 con 0x55f114d7c9b0 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28b9d8c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f28b01ae9d0 con 0x55f114d87b90 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28b9d8c740 10 client.0 _ll_register_callbacks cb 0x7f28a0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28af400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 0x55f114d8d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28b5000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d87b90 0x55f114d8d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28af400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 0x55f114d8d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43140/0 (socket says 172.21.15.6:43140) 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28af400640 1 -- 172.21.15.6:0/2011689549 learned_addr learned my addr 172.21.15.6:0/2011689549 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28afe00640 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7c9b0 0x55f114d8fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28af400640 1 -- 172.21.15.6:0/2011689549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d87b90 msgr2=0x55f114d8d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28af400640 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d87b90 0x55f114d8d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28af400640 1 -- 172.21.15.6:0/2011689549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7c9b0 msgr2=0x55f114d8fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28af400640 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7c9b0 0x55f114d8fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28af400640 1 -- 172.21.15.6:0/2011689549 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f114d8df80 con 0x55f114d7d2d0 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28b5000640 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d87b90 0x55f114d8d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.404+0000 7f28afe00640 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7c9b0 0x55f114d8fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.405+0000 7f28af400640 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 0x55f114d8d240 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55f114d7a050 tx=0x7f2898004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268211c430b11974 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.405+0000 7f289e000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.405+0000 7f289e000640 1 -- 172.21.15.6:0/2011689549 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2898020810 con 0x55f114d7d2d0 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.405+0000 7f289e000640 1 -- 172.21.15.6:0/2011689549 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f28980209b0 con 0x55f114d7d2d0 2024-04-26T01:46:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.405+0000 7f28b9d8c740 1 -- 172.21.15.6:0/2011689549 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28a0031950 con 0x55f114d7d2d0 2024-04-26T01:46:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.405+0000 7f289e000640 1 -- 172.21.15.6:0/2011689549 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f289803d050 con 0x55f114d7d2d0 2024-04-26T01:46:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.405+0000 7f28b9d8c740 10 client.15153 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.405+0000 7f28b9d8c740 1 -- 172.21.15.6:0/2011689549 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f114dfdf70 con 0x55f114d7d2d0 2024-04-26T01:46:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.406+0000 7f289e000640 1 -- 172.21.15.6:0/2011689549 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28a0031950 con 0x55f114d7d2d0 2024-04-26T01:46:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.406+0000 7f289e000640 1 -- 172.21.15.6:0/2011689549 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f289802f640 con 0x55f114d7d2d0 2024-04-26T01:46:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.406+0000 7f28b9d8c740 10 client.15153 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.406+0000 7f28b9d8c740 10 client.15153 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.406+0000 7f28b9d8c740 20 client.15153 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.406+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f114dfe750 0x55f114e00b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.406+0000 7f28b9d8c740 4 client.15153 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.406+0000 7f28b9d8c740 1 -- 172.21.15.6:0/2011689549 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f114cf0320 con 0x55f114dfe750 2024-04-26T01:46:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.406+0000 7f28afe00640 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f114dfe750 0x55f114e00b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.407+0000 7f28afe00640 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f114dfe750 0x55f114e00b70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60b9ef299b2309f5 server_cookie=e15d5c1ceb22ef0a in_seq=0 out_seq=0 2024-04-26T01:46:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.407+0000 7f289e000640 10 client.15153 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f289e000640 1 -- 172.21.15.6:0/2011689549 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f114cf0320 con 0x55f114dfe750 2024-04-26T01:46:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f289e000640 10 client.15153 handle_command_reply: tid=0 2024-04-26T01:46:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f28b9d8c740 1 client.15153 shutdown 2024-04-26T01:46:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f28b9d8c740 1 -- 172.21.15.6:0/2011689549 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 msgr2=0x55f114d8d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 0x55f114d8d240 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55f114d7a050 tx=0x7f2898004a40 comp rx=0 tx=0).stop 2024-04-26T01:46:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f28b9d8c740 1 -- 172.21.15.6:0/2011689549 shutdown_connections 2024-04-26T01:46:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f114dfe750 0x55f114e00b70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f114d7d2d0 0x55f114d8d240 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f289e000640 0 client.15153 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f114d87b90 0x55f114d8d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f28b9d8c740 1 --2- 172.21.15.6:0/2011689549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f114d7c9b0 0x55f114d8fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f28b9d8c740 1 -- 172.21.15.6:0/2011689549 >> 172.21.15.6:0/2011689549 conn(0x55f114d78ec0 msgr2=0x55f114d7ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.417+0000 7f28b9d8c740 1 -- 172.21.15.6:0/2011689549 shutdown_connections 2024-04-26T01:46:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.418+0000 7f28b9d8c740 1 -- 172.21.15.6:0/2011689549 wait complete. 2024-04-26T01:46:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.418+0000 7f28b9d8c740 20 client.15153 trim_cache size 0 max 16384 2024-04-26T01:46:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:46:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:46:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:46:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:46:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.419+0000 7f28b5a00640 1 -- 172.21.15.6:0/1164131127 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f28940a3ea0 msgr2=0x7f28940a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.419+0000 7f28b5a00640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f28940a3ea0 0x7f28940a6360 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x55f114d7a2e0 tx=0x7f2898004ab0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.419+0000 7f28b5a00640 1 -- 172.21.15.6:0/1164131127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28b019a5a0 msgr2=0x7f28b019aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.419+0000 7f28b5a00640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28b019a5a0 0x7f28b019aa50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f28a800e970 tx=0x7f28a800ee30 comp rx=0 tx=0).stop 2024-04-26T01:46:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.419+0000 7f28b5a00640 1 -- 172.21.15.6:0/1164131127 shutdown_connections 2024-04-26T01:46:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.419+0000 7f28b5a00640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f28940a3ea0 0x7f28940a6360 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.419+0000 7f28b5a00640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28b019a5a0 0x7f28b019aa50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.419+0000 7f28b5a00640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28b0105fa0 0x7f28b01a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.419+0000 7f28b5a00640 1 --2- 172.21.15.6:0/1164131127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b0100ed0 0x7f28b01a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.419+0000 7f28b5a00640 1 -- 172.21.15.6:0/1164131127 >> 172.21.15.6:0/1164131127 conn(0x7f28b00fc730 msgr2=0x7f28b00fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:37.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.419+0000 7f28b5a00640 1 -- 172.21.15.6:0/1164131127 shutdown_connections 2024-04-26T01:46:37.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:37.419+0000 7f28b5a00640 1 -- 172.21.15.6:0/1164131127 wait complete. 2024-04-26T01:46:38.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.639+0000 7f2bea800700 1 -- 172.21.15.6:0/2556960031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2be4075590 msgr2=0x7f2be410ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:38.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.639+0000 7f2bea800700 1 --2- 172.21.15.6:0/2556960031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2be4075590 0x7f2be410ef00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009990 tx=0x7f2bd0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.639+0000 7f2bea800700 1 -- 172.21.15.6:0/2556960031 shutdown_connections 2024-04-26T01:46:38.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.639+0000 7f2bea800700 1 --2- 172.21.15.6:0/2556960031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be410f440 0x7f2be4111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.639+0000 7f2bea800700 1 --2- 172.21.15.6:0/2556960031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2be4074c70 0x7f2be4075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.639+0000 7f2bea800700 1 --2- 172.21.15.6:0/2556960031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2be4075590 0x7f2be410ef00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.639+0000 7f2bea800700 1 -- 172.21.15.6:0/2556960031 >> 172.21.15.6:0/2556960031 conn(0x7f2be4079bf0 msgr2=0x7f2be407a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:38.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.639+0000 7f2bea800700 1 -- 172.21.15.6:0/2556960031 shutdown_connections 2024-04-26T01:46:38.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.639+0000 7f2bea800700 1 -- 172.21.15.6:0/2556960031 wait complete. 2024-04-26T01:46:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.641+0000 7f2bea800700 1 Processor -- start 2024-04-26T01:46:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.641+0000 7f2bea800700 1 -- start start 2024-04-26T01:46:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.641+0000 7f2bea800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be4074c70 0x7f2be419b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.641+0000 7f2bea800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2be4075590 0x7f2be419b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.641+0000 7f2bea800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2be410f440 0x7f2be419fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.641+0000 7f2bea800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2be4114ed0 con 0x7f2be4074c70 2024-04-26T01:46:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.641+0000 7f2bea800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2be4114d50 con 0x7f2be410f440 2024-04-26T01:46:38.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.641+0000 7f2bea800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2be4115050 con 0x7f2be4075590 2024-04-26T01:46:38.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.642+0000 7f2be2a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2be4075590 0x7f2be419b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.642+0000 7f2be2a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2be4075590 0x7f2be419b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38066/0 (socket says 172.21.15.6:38066) 2024-04-26T01:46:38.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.642+0000 7f2be2a00700 1 -- 172.21.15.6:0/1205881222 learned_addr learned my addr 172.21.15.6:0/1205881222 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:38.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.642+0000 7f2be3e00700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2be410f440 0x7f2be419fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.642+0000 7f2be3400700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be4074c70 0x7f2be419b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.642+0000 7f2be3e00700 1 -- 172.21.15.6:0/1205881222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2be4075590 msgr2=0x7f2be419b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:38.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.642+0000 7f2be3e00700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2be4075590 0x7f2be419b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.642+0000 7f2be3e00700 1 -- 172.21.15.6:0/1205881222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be4074c70 msgr2=0x7f2be419b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:38.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.642+0000 7f2be3e00700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be4074c70 0x7f2be419b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.642+0000 7f2be3e00700 1 -- 172.21.15.6:0/1205881222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be41a0450 con 0x7f2be410f440 2024-04-26T01:46:38.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.643+0000 7f2be3e00700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2be410f440 0x7f2be419fd10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2be406b270 tx=0x7f2bdc00ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:38.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.643+0000 7f2bdbe00700 1 -- 172.21.15.6:0/1205881222 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc0092e0 con 0x7f2be410f440 2024-04-26T01:46:38.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.643+0000 7f2bdbe00700 1 -- 172.21.15.6:0/1205881222 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bdc004bb0 con 0x7f2be410f440 2024-04-26T01:46:38.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.643+0000 7f2bea800700 1 -- 172.21.15.6:0/1205881222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be41a06e0 con 0x7f2be410f440 2024-04-26T01:46:38.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.643+0000 7f2bdbe00700 1 -- 172.21.15.6:0/1205881222 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc004020 con 0x7f2be410f440 2024-04-26T01:46:38.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.643+0000 7f2bea800700 1 -- 172.21.15.6:0/1205881222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be41a7fc0 con 0x7f2be410f440 2024-04-26T01:46:38.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.648+0000 7f2bdbe00700 1 -- 172.21.15.6:0/1205881222 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f2bdc01b030 con 0x7f2be410f440 2024-04-26T01:46:38.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.649+0000 7f2bf5191b80 1 Processor -- start 2024-04-26T01:46:38.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.649+0000 7f2bf5191b80 1 -- start start 2024-04-26T01:46:38.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.650+0000 7f2bf5191b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f862f0 0x55d1f0f866d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.650+0000 7f2bf5191b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0f91570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.650+0000 7f2bf5191b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f91ab0 0x55d1f0f93ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.650+0000 7f2bf5191b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0ed9e40 con 0x55d1f0f862f0 2024-04-26T01:46:38.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.650+0000 7f2bf5191b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0e806e0 con 0x55d1f0f91ab0 2024-04-26T01:46:38.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.650+0000 7f2bf5191b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0ed5e00 con 0x55d1f0f86c10 2024-04-26T01:46:38.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.650+0000 7f2be2a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f862f0 0x55d1f0f866d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.650+0000 7f2be2a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f862f0 0x55d1f0f866d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51408/0 (socket says 172.21.15.6:51408) 2024-04-26T01:46:38.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.650+0000 7f2be2a00700 1 -- 172.21.15.6:0/3114616309 learned_addr learned my addr 172.21.15.6:0/3114616309 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:38.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.650+0000 7f2be3400700 1 --2- 172.21.15.6:0/3114616309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0f91570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.651+0000 7f2be2a00700 1 --2- 172.21.15.6:0/3114616309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f91ab0 0x55d1f0f93ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.651+0000 7f2bdbe00700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2bc80a3f50 0x7f2bc80a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.651+0000 7f2be3400700 1 -- 172.21.15.6:0/3114616309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f91ab0 msgr2=0x55d1f0f93ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:38.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.651+0000 7f2be3400700 1 --2- 172.21.15.6:0/3114616309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f91ab0 0x55d1f0f93ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.651+0000 7f2be3400700 1 -- 172.21.15.6:0/3114616309 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f862f0 msgr2=0x55d1f0f866d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:38.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.651+0000 7f2be3400700 1 --2- 172.21.15.6:0/3114616309 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f862f0 0x55d1f0f866d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.651+0000 7f2be3400700 1 -- 172.21.15.6:0/3114616309 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1f0e76630 con 0x55d1f0f86c10 2024-04-26T01:46:38.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.651+0000 7f2bdbe00700 1 -- 172.21.15.6:0/1205881222 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f2bdc0d03c0 con 0x7f2be410f440 2024-04-26T01:46:38.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.651+0000 7f2be3e00700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2bc80a3f50 0x7f2bc80a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.652+0000 7f2be3400700 1 --2- 172.21.15.6:0/3114616309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0f91570 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009cb0 tx=0x7f2bd400f3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c775628a395e76f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:38.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.652+0000 7f2be2a00700 1 --2- 172.21.15.6:0/3114616309 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f862f0 0x55d1f0f866d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:38.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.652+0000 7f2bd8c00700 1 -- 172.21.15.6:0/3114616309 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd400fb50 con 0x55d1f0f86c10 2024-04-26T01:46:38.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.652+0000 7f2bd8c00700 1 -- 172.21.15.6:0/3114616309 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bd40044e0 con 0x55d1f0f86c10 2024-04-26T01:46:38.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.652+0000 7f2bd8c00700 1 -- 172.21.15.6:0/3114616309 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4005020 con 0x55d1f0f86c10 2024-04-26T01:46:38.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.653+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3114616309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 msgr2=0x55d1f0f91570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:38.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.653+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/3114616309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0f91570 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009cb0 tx=0x7f2bd400f3c0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.653+0000 7f2be3e00700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2bc80a3f50 0x7f2bc80a6410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0cd970 tx=0x7f2bdc009da0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:38.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.653+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3114616309 shutdown_connections 2024-04-26T01:46:38.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.653+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/3114616309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f91ab0 0x55d1f0f93ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.653+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/3114616309 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f862f0 0x55d1f0f866d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.653+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/3114616309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0f91570 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.653+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3114616309 >> 172.21.15.6:0/3114616309 conn(0x55d1f0eb5550 msgr2=0x55d1f0f82790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.653+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3114616309 shutdown_connections 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.653+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3114616309 wait complete. 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.654+0000 7f2bf5191b80 1 -- start start 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.654+0000 7f2bf5191b80 1 Processor -- start 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.655+0000 7f2bf5191b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f862f0 0x55d1f0f911c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.655+0000 7f2bf5191b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0f8e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.655+0000 7f2bf5191b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f91ab0 0x55d1f0f8ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.655+0000 7f2bf5191b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0f945a0 con 0x55d1f0f91ab0 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.655+0000 7f2bf5191b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0e93810 con 0x55d1f0f862f0 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.655+0000 7f2bf5191b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0f94720 con 0x55d1f0f86c10 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.655+0000 7f2bf5191b80 10 client.0 _ll_register_callbacks cb 0x55d1f0e726d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.655+0000 7f2be3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0f8e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.655+0000 7f2be3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0f8e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38082/0 (socket says 172.21.15.6:38082) 2024-04-26T01:46:38.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.655+0000 7f2be3400700 1 -- 172.21.15.6:0/782095558 learned_addr learned my addr 172.21.15.6:0/782095558 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:38.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.655+0000 7f2be2a00700 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f91ab0 0x55d1f0f8ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.655+0000 7f2be2a00700 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f862f0 0x55d1f0f911c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.656+0000 7f2be2a00700 1 -- 172.21.15.6:0/782095558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 msgr2=0x55d1f0f8e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:38.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.656+0000 7f2be2a00700 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0f8e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.656+0000 7f2be2a00700 1 -- 172.21.15.6:0/782095558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f862f0 msgr2=0x55d1f0f911c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:38.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.656+0000 7f2be2a00700 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f862f0 0x55d1f0f911c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.656+0000 7f2be2a00700 1 -- 172.21.15.6:0/782095558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1f0e27880 con 0x55d1f0f91ab0 2024-04-26T01:46:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.656+0000 7f2be2a00700 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f91ab0 0x55d1f0f8ea00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0005800 tx=0x7f2bd0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb9fbb37f8fd9a6d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.656+0000 7f2bcd600700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:46:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.656+0000 7f2bcd600700 1 -- 172.21.15.6:0/782095558 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0021040 con 0x55d1f0f91ab0 2024-04-26T01:46:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.656+0000 7f2bcd600700 1 -- 172.21.15.6:0/782095558 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f2bd000f460 con 0x55d1f0f91ab0 2024-04-26T01:46:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.657+0000 7f2bf5191b80 1 -- 172.21.15.6:0/782095558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bd40044e0 con 0x55d1f0f91ab0 2024-04-26T01:46:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.657+0000 7f2bcd600700 1 -- 172.21.15.6:0/782095558 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2bd000be70 con 0x55d1f0f91ab0 2024-04-26T01:46:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.657+0000 7f2bf5191b80 10 client.15165 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.657+0000 7f2bf5191b80 1 -- 172.21.15.6:0/782095558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d1f0e70aa0 con 0x55d1f0f91ab0 2024-04-26T01:46:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.657+0000 7f2bcd600700 1 -- 172.21.15.6:0/782095558 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bd40044e0 con 0x55d1f0f91ab0 2024-04-26T01:46:38.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.657+0000 7f2bcd600700 1 -- 172.21.15.6:0/782095558 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2bd0027070 con 0x55d1f0f91ab0 2024-04-26T01:46:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.658+0000 7f2bf5191b80 10 client.15165 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.658+0000 7f2bf5191b80 10 client.15165 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:46:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.658+0000 7f2bf5191b80 20 client.15165 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.658+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d1f0fa1180 0x55d1f0fa25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.658+0000 7f2bf5191b80 4 client.15165 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.658+0000 7f2bf5191b80 1 -- 172.21.15.6:0/782095558 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1f0ee7630 con 0x55d1f0fa1180 2024-04-26T01:46:38.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.658+0000 7f2be3400700 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d1f0fa1180 0x55d1f0fa25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.659+0000 7f2be3400700 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d1f0fa1180 0x55d1f0fa25f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aae397b9dc70c63f server_cookie=48a159b78f8e26a4 in_seq=0 out_seq=0 2024-04-26T01:46:38.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.659+0000 7f2bcd600700 10 client.15165 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:38.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bcd600700 1 -- 172.21.15.6:0/782095558 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d1f0ee7630 con 0x55d1f0fa1180 2024-04-26T01:46:38.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bcd600700 10 client.15165 handle_command_reply: tid=0 2024-04-26T01:46:38.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bf5191b80 1 client.15165 shutdown 2024-04-26T01:46:38.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bf5191b80 1 -- 172.21.15.6:0/782095558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f91ab0 msgr2=0x55d1f0f8ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f91ab0 0x55d1f0f8ea00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0005800 tx=0x7f2bd0000bc0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bf5191b80 1 -- 172.21.15.6:0/782095558 shutdown_connections 2024-04-26T01:46:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f91ab0 0x55d1f0f8ea00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f862f0 0x55d1f0f911c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d1f0fa1180 0x55d1f0fa25f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/782095558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0f8e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bf5191b80 1 -- 172.21.15.6:0/782095558 >> 172.21.15.6:0/782095558 conn(0x55d1f0eb5550 msgr2=0x55d1f0f88c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bcd600700 0 client.15165 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bf5191b80 1 -- 172.21.15.6:0/782095558 shutdown_connections 2024-04-26T01:46:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bf5191b80 1 -- 172.21.15.6:0/782095558 wait complete. 2024-04-26T01:46:38.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.660+0000 7f2bf5191b80 20 client.15165 trim_cache size 0 max 16384 2024-04-26T01:46:38.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.686+0000 7f2bf5191b80 1 Processor -- start 2024-04-26T01:46:38.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.686+0000 7f2bf5191b80 1 -- start start 2024-04-26T01:46:38.684 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:38 smithi006 ceph-mon[28419]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-26T01:46:38.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.687+0000 7f2bf5191b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f862f0 0x55d1f0fa00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.687+0000 7f2bf5191b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f86c10 0x55d1f0faba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.687+0000 7f2bf5191b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f91ab0 0x55d1f0fabf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.687+0000 7f2bf5191b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0fa0930 con 0x55d1f0f91ab0 2024-04-26T01:46:38.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.687+0000 7f2bf5191b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0fa1000 con 0x55d1f0f86c10 2024-04-26T01:46:38.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.687+0000 7f2bf5191b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0fa16d0 con 0x55d1f0f862f0 2024-04-26T01:46:38.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.687+0000 7f2be2a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f91ab0 0x55d1f0fabf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.687+0000 7f2be3400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f86c10 0x55d1f0faba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.687+0000 7f2be2a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f862f0 0x55d1f0fa00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.687+0000 7f2be3400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f86c10 0x55d1f0faba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55810/0 (socket says 172.21.15.6:55810) 2024-04-26T01:46:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.687+0000 7f2be3400700 1 -- 172.21.15.6:0/2865363757 learned_addr learned my addr 172.21.15.6:0/2865363757 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.688+0000 7f2be3400700 1 -- 172.21.15.6:0/2865363757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f862f0 msgr2=0x55d1f0fa00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.688+0000 7f2be3400700 1 --2- 172.21.15.6:0/2865363757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f862f0 0x55d1f0fa00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.688+0000 7f2be3400700 1 -- 172.21.15.6:0/2865363757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f91ab0 msgr2=0x55d1f0fabf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.688+0000 7f2be3400700 1 --2- 172.21.15.6:0/2865363757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f91ab0 0x55d1f0fabf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.688+0000 7f2be3400700 1 -- 172.21.15.6:0/2865363757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1f0f90e00 con 0x55d1f0f86c10 2024-04-26T01:46:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.688+0000 7f2be2a00700 1 --2- 172.21.15.6:0/2865363757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f862f0 0x55d1f0fa00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.689+0000 7f2be3400700 1 --2- 172.21.15.6:0/2865363757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f86c10 0x55d1f0faba50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4000c00 tx=0x7f2bd400f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=931eeef85b3c9fc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:38.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.689+0000 7f2bcfe00700 1 -- 172.21.15.6:0/2865363757 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4038070 con 0x55d1f0f86c10 2024-04-26T01:46:38.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.689+0000 7f2bcfe00700 1 -- 172.21.15.6:0/2865363757 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bd400fc10 con 0x55d1f0f86c10 2024-04-26T01:46:38.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.689+0000 7f2bcfe00700 1 -- 172.21.15.6:0/2865363757 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4003680 con 0x55d1f0f86c10 2024-04-26T01:46:38.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.690+0000 7f2bf5191b80 1 -- 172.21.15.6:0/2865363757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f86c10 msgr2=0x55d1f0faba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:38.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.690+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/2865363757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f86c10 0x55d1f0faba50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4000c00 tx=0x7f2bd400f3c0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.690+0000 7f2bf5191b80 1 -- 172.21.15.6:0/2865363757 shutdown_connections 2024-04-26T01:46:38.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.690+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/2865363757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f91ab0 0x55d1f0fabf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.690+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/2865363757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f862f0 0x55d1f0fa00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.690+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/2865363757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f86c10 0x55d1f0faba50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.690+0000 7f2bf5191b80 1 -- 172.21.15.6:0/2865363757 >> 172.21.15.6:0/2865363757 conn(0x55d1f0eb5550 msgr2=0x55d1f0f7e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:38.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.691+0000 7f2bf5191b80 1 -- 172.21.15.6:0/2865363757 shutdown_connections 2024-04-26T01:46:38.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.691+0000 7f2bf5191b80 1 -- 172.21.15.6:0/2865363757 wait complete. 2024-04-26T01:46:38.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.691+0000 7f2bf5191b80 1 -- start start 2024-04-26T01:46:38.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.691+0000 7f2bf5191b80 1 Processor -- start 2024-04-26T01:46:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.692+0000 7f2bf5191b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f862f0 0x55d1f0fa54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.692+0000 7f2bf5191b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0fa5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.692+0000 7f2bf5191b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f91ab0 0x55d1f0fa5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.692+0000 7f2bf5191b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0fa2470 con 0x55d1f0f862f0 2024-04-26T01:46:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.692+0000 7f2bf5191b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0fa1da0 con 0x55d1f0f91ab0 2024-04-26T01:46:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.692+0000 7f2bf5191b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0fa2b40 con 0x55d1f0f86c10 2024-04-26T01:46:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.692+0000 7f2bf5191b80 10 client.0 _ll_register_callbacks cb 0x55d1f0e726d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.692+0000 7f2be3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0fa5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.693+0000 7f2be3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0fa5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38104/0 (socket says 172.21.15.6:38104) 2024-04-26T01:46:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.693+0000 7f2be3400700 1 -- 172.21.15.6:0/3847291285 learned_addr learned my addr 172.21.15.6:0/3847291285 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.693+0000 7f2be2a00700 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f862f0 0x55d1f0fa54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.693+0000 7f2be2a00700 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f91ab0 0x55d1f0fa5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:38 smithi029 ceph-mon[32775]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-26T01:46:38.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.693+0000 7f2be3400700 1 -- 172.21.15.6:0/3847291285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f91ab0 msgr2=0x55d1f0fa5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:38.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.693+0000 7f2be3400700 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f91ab0 0x55d1f0fa5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.693+0000 7f2be3400700 1 -- 172.21.15.6:0/3847291285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f862f0 msgr2=0x55d1f0fa54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:38.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.693+0000 7f2be3400700 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f862f0 0x55d1f0fa54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.693+0000 7f2be3400700 1 -- 172.21.15.6:0/3847291285 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1f0fa9d40 con 0x55d1f0f86c10 2024-04-26T01:46:38.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.693+0000 7f2be2a00700 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f91ab0 0x55d1f0fa5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:38.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.693+0000 7f2be3400700 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0fa5a10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4005fe0 tx=0x7f2bd400fdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51cbf20e81d2be31 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:38.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.693+0000 7f2bcd600700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:46:38.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.694+0000 7f2bcd600700 1 -- 172.21.15.6:0/3847291285 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4043060 con 0x55d1f0f86c10 2024-04-26T01:46:38.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.694+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3847291285 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bd400fc10 con 0x55d1f0f86c10 2024-04-26T01:46:38.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.694+0000 7f2bcd600700 1 -- 172.21.15.6:0/3847291285 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f2bd4004eb0 con 0x55d1f0f86c10 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.694+0000 7f2bcd600700 1 -- 172.21.15.6:0/3847291285 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2bd4038040 con 0x55d1f0f86c10 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.694+0000 7f2bf5191b80 10 client.25315 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.694+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3847291285 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d1f1008650 con 0x55d1f0f86c10 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.695+0000 7f2bcd600700 1 -- 172.21.15.6:0/3847291285 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bd400fc10 con 0x55d1f0f86c10 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.695+0000 7f2bcd600700 1 -- 172.21.15.6:0/3847291285 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2bd4003ac0 con 0x55d1f0f86c10 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.695+0000 7f2bf5191b80 10 client.25315 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.695+0000 7f2bf5191b80 10 client.25315 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.695+0000 7f2bf5191b80 20 client.25315 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.695+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d1f100cb60 0x55d1f100b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.695+0000 7f2bf5191b80 4 client.25315 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.695+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3847291285 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "status"}) -- 0x55d1f0ee16d0 con 0x55d1f100cb60 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.695+0000 7f2be2a00700 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d1f100cb60 0x55d1f100b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.696+0000 7f2be2a00700 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d1f100cb60 0x55d1f100b870 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84d5e5a65a5aa7c8 server_cookie=63896c481823d328 in_seq=0 out_seq=0 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.696+0000 7f2bcd600700 10 client.25315 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.696+0000 7f2bcd600700 1 -- 172.21.15.6:0/3847291285 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d1f0ee16d0 con 0x55d1f100cb60 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.696+0000 7f2bcd600700 10 client.25315 handle_command_reply: tid=0 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.696+0000 7f2bf5191b80 1 client.25315 shutdown 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.696+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3847291285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 msgr2=0x55d1f0fa5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.696+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0fa5a10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4005fe0 tx=0x7f2bd400fdc0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.697+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3847291285 shutdown_connections 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.697+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d1f0f91ab0 0x55d1f0fa5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.697+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d1f0f862f0 0x55d1f0fa54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.697+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d1f100cb60 0x55d1f100b870 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.697+0000 7f2bf5191b80 1 --2- 172.21.15.6:0/3847291285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d1f0f86c10 0x55d1f0fa5a10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.697+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3847291285 >> 172.21.15.6:0/3847291285 conn(0x55d1f0eb5550 msgr2=0x55d1f0fa00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.697+0000 7f2bcd600700 0 client.25315 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.697+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3847291285 shutdown_connections 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.697+0000 7f2bf5191b80 1 -- 172.21.15.6:0/3847291285 wait complete. 2024-04-26T01:46:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.697+0000 7f2bf5191b80 20 client.25315 trim_cache size 0 max 16384 2024-04-26T01:46:38.697 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:46:38.697 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a5b72bee-036c-11ef-bc93-c7b262605968", 2024-04-26T01:46:38.697 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-04-26T01:46:38.697 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 24523, 2024-04-26T01:46:38.697 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-26T01:46:38.697 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-26T01:46:38.697 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-26T01:46:38.697 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 270.29127193300002, 2024-04-26T01:46:38.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 26, 2024-04-26T01:46:38.697 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 96, 2024-04-26T01:46:38.698 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 96, 2024-04-26T01:46:38.698 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 304.27328852300002 2024-04-26T01:46:38.698 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:46:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.698+0000 7f2bea800700 1 -- 172.21.15.6:0/1205881222 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2bc80a3f50 msgr2=0x7f2bc80a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.698+0000 7f2bea800700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2bc80a3f50 0x7f2bc80a6410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0cd970 tx=0x7f2bdc009da0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.698+0000 7f2bea800700 1 -- 172.21.15.6:0/1205881222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2be410f440 msgr2=0x7f2be419fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.698+0000 7f2bea800700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2be410f440 0x7f2be419fd10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2be406b270 tx=0x7f2bdc00ef90 comp rx=0 tx=0).stop 2024-04-26T01:46:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.698+0000 7f2bea800700 1 -- 172.21.15.6:0/1205881222 shutdown_connections 2024-04-26T01:46:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.698+0000 7f2bea800700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2bc80a3f50 0x7f2bc80a6410 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.698+0000 7f2bea800700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2be410f440 0x7f2be419fd10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.698+0000 7f2bea800700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be4074c70 0x7f2be419b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.698+0000 7f2bea800700 1 --2- 172.21.15.6:0/1205881222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2be4075590 0x7f2be419b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.698+0000 7f2bea800700 1 -- 172.21.15.6:0/1205881222 >> 172.21.15.6:0/1205881222 conn(0x7f2be4079bf0 msgr2=0x7f2be4100ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.698+0000 7f2bea800700 1 -- 172.21.15.6:0/1205881222 shutdown_connections 2024-04-26T01:46:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:38.699+0000 7f2bea800700 1 -- 172.21.15.6:0/1205881222 wait complete. 2024-04-26T01:46:38.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:38 smithi188 ceph-mon[32751]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-26T01:46:39.507 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-26T01:46:40.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:40 smithi029 ceph-mon[32775]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T01:46:40.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:40 smithi006 ceph-mon[28419]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T01:46:40.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:40 smithi188 ceph-mon[32751]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T01:46:42.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:42 smithi029 ceph-mon[32775]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:46:42.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:42 smithi006 ceph-mon[28419]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:46:42.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:42 smithi188 ceph-mon[32751]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:46:44.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:44 smithi029 ceph-mon[32775]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T01:46:44.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:44 smithi006 ceph-mon[28419]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T01:46:44.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:44 smithi188 ceph-mon[32751]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T01:46:46.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:46 smithi029 ceph-mon[32775]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:46:46.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:46 smithi006 ceph-mon[28419]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:46:46.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:46 smithi188 ceph-mon[32751]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:46:48.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:48 smithi029 ceph-mon[32775]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T01:46:48.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:48 smithi006 ceph-mon[28419]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T01:46:48.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:48 smithi188 ceph-mon[32751]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T01:46:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.599+0000 7f73ffc00640 1 -- 172.21.15.6:0/2307884638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f8103b00 msgr2=0x7f73f810bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.599+0000 7f73ffc00640 1 --2- 172.21.15.6:0/2307884638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f8103b00 0x7f73f810bfe0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f73e4009900 tx=0x7f73e402a480 comp rx=0 tx=0).stop 2024-04-26T01:46:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.600+0000 7f73ffc00640 1 -- 172.21.15.6:0/2307884638 shutdown_connections 2024-04-26T01:46:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.600+0000 7f73ffc00640 1 --2- 172.21.15.6:0/2307884638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f810c5b0 0x7f73f810e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.600+0000 7f73ffc00640 1 --2- 172.21.15.6:0/2307884638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f8103b00 0x7f73f810bfe0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.600+0000 7f73ffc00640 1 --2- 172.21.15.6:0/2307884638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73f81031e0 0x7f73f81035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.600+0000 7f73ffc00640 1 -- 172.21.15.6:0/2307884638 >> 172.21.15.6:0/2307884638 conn(0x7f73f8077fd0 msgr2=0x7f73f80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.600+0000 7f73ffc00640 1 -- 172.21.15.6:0/2307884638 shutdown_connections 2024-04-26T01:46:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.600+0000 7f73ffc00640 1 -- 172.21.15.6:0/2307884638 wait complete. 2024-04-26T01:46:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.601+0000 7f73ffc00640 1 Processor -- start 2024-04-26T01:46:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.601+0000 7f73ffc00640 1 -- start start 2024-04-26T01:46:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.601+0000 7f73ffc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f81031e0 0x7f73f81a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73ffc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f8103b00 0x7f73f81a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73ffc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73f810c5b0 0x7f73f819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73ffc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f73f8112240 con 0x7f73f810c5b0 2024-04-26T01:46:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73ffc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81120c0 con 0x7f73f8103b00 2024-04-26T01:46:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73ffc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81123c0 con 0x7f73f81031e0 2024-04-26T01:46:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73fd200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f81031e0 0x7f73f81a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73fd200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f81031e0 0x7f73f81a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:33848/0 (socket says 172.21.15.6:33848) 2024-04-26T01:46:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73fd200640 1 -- 172.21.15.6:0/4238589372 learned_addr learned my addr 172.21.15.6:0/4238589372 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73f7e00640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f8103b00 0x7f73f81a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73fdc00640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73f810c5b0 0x7f73f819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73fd200640 1 -- 172.21.15.6:0/4238589372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f8103b00 msgr2=0x7f73f81a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73fd200640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f8103b00 0x7f73f81a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73fd200640 1 -- 172.21.15.6:0/4238589372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73f810c5b0 msgr2=0x7f73f819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73fd200640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73f810c5b0 0x7f73f819a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.602+0000 7f73fd200640 1 -- 172.21.15.6:0/4238589372 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f819af90 con 0x7f73f81031e0 2024-04-26T01:46:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.603+0000 7f73fd200640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f81031e0 0x7f73f81a0650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f73e800ebc0 tx=0x7f73e800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.603+0000 7f73f5600640 1 -- 172.21.15.6:0/4238589372 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8010630 con 0x7f73f81031e0 2024-04-26T01:46:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.603+0000 7f73ffc00640 1 -- 172.21.15.6:0/4238589372 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f819b280 con 0x7f73f81031e0 2024-04-26T01:46:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.603+0000 7f73f5600640 1 -- 172.21.15.6:0/4238589372 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e8010c80 con 0x7f73f81031e0 2024-04-26T01:46:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.603+0000 7f73f5600640 1 -- 172.21.15.6:0/4238589372 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8018b60 con 0x7f73f81031e0 2024-04-26T01:46:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.603+0000 7f73ffc00640 1 -- 172.21.15.6:0/4238589372 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f81a74a0 con 0x7f73f81031e0 2024-04-26T01:46:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.606+0000 7f73f5600640 1 -- 172.21.15.6:0/4238589372 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f73e8018d00 con 0x7f73f81031e0 2024-04-26T01:46:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.606+0000 7f7402a4d740 1 Processor -- start 2024-04-26T01:46:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.607+0000 7f7402a4d740 1 -- start start 2024-04-26T01:46:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.607+0000 7f7402a4d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e60820 0x564572e60c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.607+0000 7f7402a4d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e61140 0x564572e6b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.607+0000 7f7402a4d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e6ba00 0x564572e6ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.607+0000 7f7402a4d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564572dc6900 con 0x564572e61140 2024-04-26T01:46:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.607+0000 7f7402a4d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564572d6cef0 con 0x564572e60820 2024-04-26T01:46:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.607+0000 7f7402a4d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564572dc28c0 con 0x564572e6ba00 2024-04-26T01:46:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.607+0000 7f73fdc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e60820 0x564572e60c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.608+0000 7f73fd200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e61140 0x564572e6b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.608+0000 7f73fdc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e60820 0x564572e60c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59708/0 (socket says 172.21.15.6:59708) 2024-04-26T01:46:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.608+0000 7f73fdc00640 1 -- 172.21.15.6:0/3441483766 learned_addr learned my addr 172.21.15.6:0/3441483766 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.608+0000 7f73f7e00640 1 --2- 172.21.15.6:0/3441483766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e6ba00 0x564572e6ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.608+0000 7f73fdc00640 1 -- 172.21.15.6:0/3441483766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e6ba00 msgr2=0x564572e6ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.608+0000 7f73fdc00640 1 --2- 172.21.15.6:0/3441483766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e6ba00 0x564572e6ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.608+0000 7f73fdc00640 1 -- 172.21.15.6:0/3441483766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e61140 msgr2=0x564572e6b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.608+0000 7f73fdc00640 1 --2- 172.21.15.6:0/3441483766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e61140 0x564572e6b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.608+0000 7f73fdc00640 1 -- 172.21.15.6:0/3441483766 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564572ad9210 con 0x564572e60820 2024-04-26T01:46:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.608+0000 7f73fdc00640 1 --2- 172.21.15.6:0/3441483766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e60820 0x564572e60c00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009b30 tx=0x7f73f00113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=154b68703e3e1984 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.609+0000 7f73f5600640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f73dc0a3e30 0x7f73dc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.609+0000 7f73ee000640 1 -- 172.21.15.6:0/3441483766 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0011e10 con 0x564572e60820 2024-04-26T01:46:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.609+0000 7f73ee000640 1 -- 172.21.15.6:0/3441483766 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73f0012900 con 0x564572e60820 2024-04-26T01:46:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.609+0000 7f73ee000640 1 -- 172.21.15.6:0/3441483766 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f000f910 con 0x564572e60820 2024-04-26T01:46:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.609+0000 7f73f5600640 1 -- 172.21.15.6:0/4238589372 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f73e80d11c0 con 0x7f73f81031e0 2024-04-26T01:46:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.609+0000 7f73fd200640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f73dc0a3e30 0x7f73dc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.610+0000 7f7402a4d740 1 -- 172.21.15.6:0/3441483766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e60820 msgr2=0x564572e60c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.610+0000 7f7402a4d740 1 --2- 172.21.15.6:0/3441483766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e60820 0x564572e60c00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009b30 tx=0x7f73f00113e0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.610+0000 7f73fd200640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f73dc0a3e30 0x7f73dc0a62f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f73e80ce770 tx=0x7f73e80d4c50 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.610+0000 7f7402a4d740 1 -- 172.21.15.6:0/3441483766 shutdown_connections 2024-04-26T01:46:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.610+0000 7f7402a4d740 1 --2- 172.21.15.6:0/3441483766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e61140 0x564572e6b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.610+0000 7f7402a4d740 1 --2- 172.21.15.6:0/3441483766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e6ba00 0x564572e6ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.610+0000 7f7402a4d740 1 --2- 172.21.15.6:0/3441483766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e60820 0x564572e60c00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.610+0000 7f7402a4d740 1 -- 172.21.15.6:0/3441483766 >> 172.21.15.6:0/3441483766 conn(0x564572e5cd30 msgr2=0x564572e5d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.610+0000 7f7402a4d740 1 -- 172.21.15.6:0/3441483766 shutdown_connections 2024-04-26T01:46:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.610+0000 7f7402a4d740 1 -- 172.21.15.6:0/3441483766 wait complete. 2024-04-26T01:46:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.610+0000 7f7402a4d740 1 -- start start 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.611+0000 7f7402a4d740 1 Processor -- start 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.611+0000 7f7402a4d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e60820 0x564572e68980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.611+0000 7f7402a4d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e68ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.611+0000 7f7402a4d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e6ba00 0x564572e69400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.611+0000 7f7402a4d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564572e6e630 con 0x564572e60820 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.611+0000 7f7402a4d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564572d80310 con 0x564572e61140 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.611+0000 7f7402a4d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564572e6e7b0 con 0x564572e6ba00 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.611+0000 7f7402a4d740 10 client.0 _ll_register_callbacks cb 0x564572d4ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.611+0000 7f73fdc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e60820 0x564572e68980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.611+0000 7f73fdc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e60820 0x564572e68980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45974/0 (socket says 172.21.15.6:45974) 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.611+0000 7f73fdc00640 1 -- 172.21.15.6:0/3897650723 learned_addr learned my addr 172.21.15.6:0/3897650723 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.612+0000 7f73f7e00640 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e68ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.612+0000 7f73fdc00640 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e6ba00 0x564572e69400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.612+0000 7f73f7e00640 1 -- 172.21.15.6:0/3897650723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e6ba00 msgr2=0x564572e69400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.612+0000 7f73f7e00640 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e6ba00 0x564572e69400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.612+0000 7f73f7e00640 1 -- 172.21.15.6:0/3897650723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e60820 msgr2=0x564572e68980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.612+0000 7f73f7e00640 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e60820 0x564572e68980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.612+0000 7f73f7e00640 1 -- 172.21.15.6:0/3897650723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564572ba5620 con 0x564572e61140 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.612+0000 7f73f7e00640 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e68ec0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f73e402aa70 tx=0x7f73e40043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dd9e2b6a0019b5c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.612+0000 7f73e2a00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.612+0000 7f73e2a00640 1 -- 172.21.15.6:0/3897650723 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e4035040 con 0x564572e61140 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.613+0000 7f73e2a00640 1 -- 172.21.15.6:0/3897650723 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f73e4005450 con 0x564572e61140 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.613+0000 7f7402a4d740 1 -- 172.21.15.6:0/3897650723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73f0012900 con 0x564572e61140 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.613+0000 7f73e2a00640 1 -- 172.21.15.6:0/3897650723 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f73e4005650 con 0x564572e61140 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.613+0000 7f7402a4d740 10 client.25289 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.613+0000 7f7402a4d740 1 -- 172.21.15.6:0/3897650723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564572e7a870 con 0x564572e61140 2024-04-26T01:46:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.613+0000 7f73e2a00640 1 -- 172.21.15.6:0/3897650723 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73f0012900 con 0x564572e61140 2024-04-26T01:46:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.613+0000 7f73e2a00640 1 -- 172.21.15.6:0/3897650723 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f73e4003a00 con 0x564572e61140 2024-04-26T01:46:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.614+0000 7f7402a4d740 10 client.25289 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.614+0000 7f7402a4d740 10 client.25289 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.614+0000 7f7402a4d740 20 client.25289 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.614+0000 7f7402a4d740 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564572e7bd30 0x564572e7e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.614+0000 7f7402a4d740 4 client.25289 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.614+0000 7f7402a4d740 1 -- 172.21.15.6:0/3897650723 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564572dda8b0 con 0x564572e7bd30 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.614+0000 7f73fdc00640 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564572e7bd30 0x564572e7e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.614+0000 7f73fdc00640 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564572e7bd30 0x564572e7e120 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4da9bb694df87f42 server_cookie=58b63f3e670ca529 in_seq=0 out_seq=0 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.614+0000 7f73e2a00640 10 client.25289 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.615+0000 7f73e2a00640 1 -- 172.21.15.6:0/3897650723 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564572dda8b0 con 0x564572e7bd30 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.615+0000 7f73e2a00640 10 client.25289 handle_command_reply: tid=0 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.615+0000 7f7402a4d740 1 client.25289 shutdown 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.616+0000 7f7402a4d740 1 -- 172.21.15.6:0/3897650723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 msgr2=0x564572e68ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.616+0000 7f7402a4d740 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e68ec0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f73e402aa70 tx=0x7f73e40043b0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.616+0000 7f7402a4d740 1 -- 172.21.15.6:0/3897650723 shutdown_connections 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.616+0000 7f7402a4d740 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564572e7bd30 0x564572e7e120 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.616+0000 7f7402a4d740 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e6ba00 0x564572e69400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.616+0000 7f7402a4d740 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e68ec0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.616+0000 7f7402a4d740 1 --2- 172.21.15.6:0/3897650723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e60820 0x564572e68980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.616+0000 7f73e2a00640 0 client.25289 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.616+0000 7f7402a4d740 1 -- 172.21.15.6:0/3897650723 >> 172.21.15.6:0/3897650723 conn(0x564572e5cd30 msgr2=0x564572e62ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.616+0000 7f7402a4d740 1 -- 172.21.15.6:0/3897650723 shutdown_connections 2024-04-26T01:46:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.616+0000 7f7402a4d740 1 -- 172.21.15.6:0/3897650723 wait complete. 2024-04-26T01:46:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.616+0000 7f7402a4d740 20 client.25289 trim_cache size 0 max 16384 2024-04-26T01:46:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.636+0000 7f7402a4d740 1 Processor -- start 2024-04-26T01:46:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.636+0000 7f7402a4d740 1 -- start start 2024-04-26T01:46:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.637+0000 7f7402a4d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e60820 0x564572e774f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.637+0000 7f7402a4d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e631e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.637+0000 7f7402a4d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e6ba00 0x564572e839c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.637+0000 7f7402a4d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81b2790 con 0x564572e6ba00 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.637+0000 7f7402a4d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81b0060 con 0x564572e61140 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.637+0000 7f7402a4d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81adeb0 con 0x564572e60820 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.637+0000 7f73f7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e631e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.637+0000 7f73f7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e631e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59738/0 (socket says 172.21.15.6:59738) 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.637+0000 7f73f7e00640 1 -- 172.21.15.6:0/620312121 learned_addr learned my addr 172.21.15.6:0/620312121 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.637+0000 7f73fdc00640 1 --2- 172.21.15.6:0/620312121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e60820 0x564572e774f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.637+0000 7f73fdc00640 1 --2- 172.21.15.6:0/620312121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e6ba00 0x564572e839c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.638+0000 7f73f7e00640 1 -- 172.21.15.6:0/620312121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e60820 msgr2=0x564572e774f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.638+0000 7f73f7e00640 1 --2- 172.21.15.6:0/620312121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e60820 0x564572e774f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.638+0000 7f73f7e00640 1 -- 172.21.15.6:0/620312121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e6ba00 msgr2=0x564572e839c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.638+0000 7f73f7e00640 1 --2- 172.21.15.6:0/620312121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e6ba00 0x564572e839c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.638+0000 7f73f7e00640 1 -- 172.21.15.6:0/620312121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564572e60f70 con 0x564572e61140 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.638+0000 7f73f7e00640 1 --2- 172.21.15.6:0/620312121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e631e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f73e4005ab0 tx=0x7f73e4001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33cf8c4eaecb72a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.638+0000 7f73ed600640 1 -- 172.21.15.6:0/620312121 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e4035070 con 0x564572e61140 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.638+0000 7f73ed600640 1 -- 172.21.15.6:0/620312121 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e4041070 con 0x564572e61140 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.638+0000 7f73ed600640 1 -- 172.21.15.6:0/620312121 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e4053070 con 0x564572e61140 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.639+0000 7f7402a4d740 1 -- 172.21.15.6:0/620312121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 msgr2=0x564572e631e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.639+0000 7f7402a4d740 1 --2- 172.21.15.6:0/620312121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e631e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f73e4005ab0 tx=0x7f73e4001e70 comp rx=0 tx=0).stop 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.639+0000 7f7402a4d740 1 -- 172.21.15.6:0/620312121 shutdown_connections 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.639+0000 7f7402a4d740 1 --2- 172.21.15.6:0/620312121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e6ba00 0x564572e839c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.639+0000 7f7402a4d740 1 --2- 172.21.15.6:0/620312121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e631e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.639+0000 7f7402a4d740 1 --2- 172.21.15.6:0/620312121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e60820 0x564572e774f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.639+0000 7f7402a4d740 1 -- 172.21.15.6:0/620312121 >> 172.21.15.6:0/620312121 conn(0x564572e5cd30 msgr2=0x564572e6aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.639+0000 7f7402a4d740 1 -- 172.21.15.6:0/620312121 shutdown_connections 2024-04-26T01:46:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.639+0000 7f7402a4d740 1 -- 172.21.15.6:0/620312121 wait complete. 2024-04-26T01:46:50.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.640+0000 7f7402a4d740 1 -- start start 2024-04-26T01:46:50.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.640+0000 7f7402a4d740 1 Processor -- start 2024-04-26T01:46:50.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.640+0000 7f7402a4d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e60820 0x564572e71630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.640+0000 7f7402a4d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e71b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.640+0000 7f7402a4d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e6ba00 0x564572e720b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.640+0000 7f7402a4d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81ae520 con 0x564572e6ba00 2024-04-26T01:46:50.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.640+0000 7f7402a4d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81af990 con 0x564572e61140 2024-04-26T01:46:50.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.640+0000 7f7402a4d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81aebf0 con 0x564572e60820 2024-04-26T01:46:50.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.641+0000 7f7402a4d740 10 client.0 _ll_register_callbacks cb 0x7f73e4004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.641+0000 7f73fdc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e6ba00 0x564572e720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.641+0000 7f73fdc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e6ba00 0x564572e720b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45986/0 (socket says 172.21.15.6:45986) 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.641+0000 7f73fdc00640 1 -- 172.21.15.6:0/583391489 learned_addr learned my addr 172.21.15.6:0/583391489 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.641+0000 7f73f7e00640 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e71b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.641+0000 7f73fdc00640 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e60820 0x564572e71630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.641+0000 7f73f7e00640 1 -- 172.21.15.6:0/583391489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e60820 msgr2=0x564572e71630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.641+0000 7f73f7e00640 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e60820 0x564572e71630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.641+0000 7f73f7e00640 1 -- 172.21.15.6:0/583391489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e6ba00 msgr2=0x564572e720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.641+0000 7f73f7e00640 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e6ba00 0x564572e720b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.641+0000 7f73f7e00640 1 -- 172.21.15.6:0/583391489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564572ee1440 con 0x564572e61140 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.642+0000 7f73f7e00640 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e71b70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x564572e589e0 tx=0x7f73e402ac00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ed67e14ec85f4d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.642+0000 7f73e2a00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.642+0000 7f73e2a00640 1 -- 172.21.15.6:0/583391489 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e4035070 con 0x564572e61140 2024-04-26T01:46:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.642+0000 7f73e2a00640 1 -- 172.21.15.6:0/583391489 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f73e40037f0 con 0x564572e61140 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.642+0000 7f7402a4d740 1 -- 172.21.15.6:0/583391489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73e4041070 con 0x564572e61140 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.642+0000 7f73e2a00640 1 -- 172.21.15.6:0/583391489 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f73e4053040 con 0x564572e61140 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.642+0000 7f7402a4d740 10 client.25295 fetch_fsmap learned FSMap version 26 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.642+0000 7f7402a4d740 1 -- 172.21.15.6:0/583391489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564572ee2730 con 0x564572e61140 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.643+0000 7f73e2a00640 1 -- 172.21.15.6:0/583391489 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e4041070 con 0x564572e61140 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.643+0000 7f73e2a00640 1 -- 172.21.15.6:0/583391489 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f73e40046c0 con 0x564572e61140 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.643+0000 7f7402a4d740 10 client.25295 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.643+0000 7f7402a4d740 10 client.25295 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.643+0000 7f7402a4d740 20 client.25295 populate_metadata read hostname 'smithi006' 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.643+0000 7f7402a4d740 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564572ee2de0 0x564572ee51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.643+0000 7f7402a4d740 4 client.25295 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.643+0000 7f7402a4d740 1 -- 172.21.15.6:0/583391489 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564572dd4320 con 0x564572ee2de0 2024-04-26T01:46:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.643+0000 7f73fdc00640 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564572ee2de0 0x564572ee51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.644+0000 7f73fdc00640 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564572ee2de0 0x564572ee51d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f46cb45047c16cc server_cookie=f1e6f7671d62cea in_seq=0 out_seq=0 2024-04-26T01:46:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.644+0000 7f73e2a00640 10 client.25295 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f73e2a00640 1 -- 172.21.15.6:0/583391489 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564572dd4320 con 0x564572ee2de0 2024-04-26T01:46:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f73e2a00640 10 client.25295 handle_command_reply: tid=0 2024-04-26T01:46:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f7402a4d740 1 client.25295 shutdown 2024-04-26T01:46:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f7402a4d740 1 -- 172.21.15.6:0/583391489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 msgr2=0x564572e71b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f7402a4d740 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e71b70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x564572e589e0 tx=0x7f73e402ac00 comp rx=0 tx=0).stop 2024-04-26T01:46:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f7402a4d740 1 -- 172.21.15.6:0/583391489 shutdown_connections 2024-04-26T01:46:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f7402a4d740 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564572ee2de0 0x564572ee51d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f7402a4d740 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564572e6ba00 0x564572e720b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f7402a4d740 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564572e61140 0x564572e71b70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f7402a4d740 1 --2- 172.21.15.6:0/583391489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564572e60820 0x564572e71630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f73e2a00640 0 client.25295 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:46:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f7402a4d740 1 -- 172.21.15.6:0/583391489 >> 172.21.15.6:0/583391489 conn(0x564572e5cd30 msgr2=0x564572e68920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f7402a4d740 1 -- 172.21.15.6:0/583391489 shutdown_connections 2024-04-26T01:46:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f7402a4d740 1 -- 172.21.15.6:0/583391489 wait complete. 2024-04-26T01:46:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.648+0000 7f7402a4d740 20 client.25295 trim_cache size 0 max 16384 2024-04-26T01:46:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:46:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:46:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:46:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:46:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.649+0000 7f73ffc00640 1 -- 172.21.15.6:0/4238589372 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f73dc0a3e30 msgr2=0x7f73dc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.649+0000 7f73ffc00640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f73dc0a3e30 0x7f73dc0a62f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f73e80ce770 tx=0x7f73e80d4c50 comp rx=0 tx=0).stop 2024-04-26T01:46:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.649+0000 7f73ffc00640 1 -- 172.21.15.6:0/4238589372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f81031e0 msgr2=0x7f73f81a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.649+0000 7f73ffc00640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f81031e0 0x7f73f81a0650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f73e800ebc0 tx=0x7f73e800c6a0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.649+0000 7f73ffc00640 1 -- 172.21.15.6:0/4238589372 shutdown_connections 2024-04-26T01:46:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.649+0000 7f73ffc00640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f73dc0a3e30 0x7f73dc0a62f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.649+0000 7f73ffc00640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73f810c5b0 0x7f73f819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.649+0000 7f73ffc00640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f8103b00 0x7f73f81a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.649+0000 7f73ffc00640 1 --2- 172.21.15.6:0/4238589372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f81031e0 0x7f73f81a0650 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.649+0000 7f73ffc00640 1 -- 172.21.15.6:0/4238589372 >> 172.21.15.6:0/4238589372 conn(0x7f73f8077fd0 msgr2=0x7f73f81021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.650+0000 7f73ffc00640 1 -- 172.21.15.6:0/4238589372 shutdown_connections 2024-04-26T01:46:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:46:50.650+0000 7f73ffc00640 1 -- 172.21.15.6:0/4238589372 wait complete. 2024-04-26T01:46:50.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:50 smithi006 ceph-mon[28419]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:46:50.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:50 smithi188 ceph-mon[32751]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:46:50.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:50 smithi029 ceph-mon[32775]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:46:52.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:52 smithi006 ceph-mon[28419]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-26T01:46:52.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:52 smithi188 ceph-mon[32751]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-26T01:46:52.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:52 smithi029 ceph-mon[32775]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-26T01:46:54.509 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.6 2024-04-26T01:46:54.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:46:54.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:54 smithi006 ceph-mon[28419]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T01:46:54.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:54 smithi188 ceph-mon[32751]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T01:46:54.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:54 smithi029 ceph-mon[32775]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T01:46:55.012 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:56.749 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:56 smithi006 ceph-mon[28419]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:46:56.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:56 smithi188 ceph-mon[32751]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:46:56.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:56 smithi029 ceph-mon[32775]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:46:58.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:58 smithi006 ceph-mon[28419]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T01:46:58.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.866+0000 7f0ab0e00700 1 -- 172.21.15.6:0/276023578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aac107870 msgr2=0x7f0aac107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:58.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.866+0000 7f0ab0e00700 1 --2- 172.21.15.6:0/276023578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aac107870 0x7f0aac107c50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c009990 tx=0x7f0a9c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:46:58.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.866+0000 7f0ab0e00700 1 -- 172.21.15.6:0/276023578 shutdown_connections 2024-04-26T01:46:58.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.867+0000 7f0ab0e00700 1 --2- 172.21.15.6:0/276023578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aac102230 0x7f0aac10e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:58.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.867+0000 7f0ab0e00700 1 --2- 172.21.15.6:0/276023578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aac107870 0x7f0aac107c50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:58.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.867+0000 7f0ab0e00700 1 --2- 172.21.15.6:0/276023578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0aac101870 0x7f0aac101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:58.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.867+0000 7f0ab0e00700 1 -- 172.21.15.6:0/276023578 >> 172.21.15.6:0/276023578 conn(0x7f0aac0fd360 msgr2=0x7f0aac0ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:58.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.867+0000 7f0ab0e00700 1 -- 172.21.15.6:0/276023578 shutdown_connections 2024-04-26T01:46:58.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.867+0000 7f0ab0e00700 1 -- 172.21.15.6:0/276023578 wait complete. 2024-04-26T01:46:58.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.868+0000 7f0ab0e00700 1 Processor -- start 2024-04-26T01:46:58.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.869+0000 7f0ab0e00700 1 -- start start 2024-04-26T01:46:58.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.869+0000 7f0ab0e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0aac101870 0x7f0aac199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:58.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.869+0000 7f0ab0e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aac102230 0x7f0aac199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:58.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.869+0000 7f0ab0e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aac107870 0x7f0aac19d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:58.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.869+0000 7f0ab0e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0aac111ac0 con 0x7f0aac107870 2024-04-26T01:46:58.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.869+0000 7f0ab0e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0aac111940 con 0x7f0aac102230 2024-04-26T01:46:58.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.869+0000 7f0ab0e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0aac111c40 con 0x7f0aac101870 2024-04-26T01:46:58.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.870+0000 7f0aaaa00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aac102230 0x7f0aac199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:58.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.870+0000 7f0aaaa00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aac102230 0x7f0aac199580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53802/0 (socket says 172.21.15.6:53802) 2024-04-26T01:46:58.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.870+0000 7f0aaaa00700 1 -- 172.21.15.6:0/117003402 learned_addr learned my addr 172.21.15.6:0/117003402 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:46:58.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.870+0000 7f0aab400700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0aac101870 0x7f0aac199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:58.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.870+0000 7f0aaaa00700 1 -- 172.21.15.6:0/117003402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0aac101870 msgr2=0x7f0aac199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:58.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.870+0000 7f0aaaa00700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0aac101870 0x7f0aac199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:58.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.870+0000 7f0aaaa00700 1 -- 172.21.15.6:0/117003402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aac107870 msgr2=0x7f0aac19d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:58.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.870+0000 7f0aaaa00700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aac107870 0x7f0aac19d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:58.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.870+0000 7f0aaaa00700 1 -- 172.21.15.6:0/117003402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aac19e050 con 0x7f0aac102230 2024-04-26T01:46:58.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.871+0000 7f0aaaa00700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aac102230 0x7f0aac199580 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800e9c0 tx=0x7f0a9800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:58.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.871+0000 7f0aa3e00700 1 -- 172.21.15.6:0/117003402 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a9800cbd0 con 0x7f0aac102230 2024-04-26T01:46:58.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.871+0000 7f0aa3e00700 1 -- 172.21.15.6:0/117003402 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a9800cd70 con 0x7f0aac102230 2024-04-26T01:46:58.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.871+0000 7f0aa3e00700 1 -- 172.21.15.6:0/117003402 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a98005680 con 0x7f0aac102230 2024-04-26T01:46:58.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.871+0000 7f0ab0e00700 1 -- 172.21.15.6:0/117003402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aac19e340 con 0x7f0aac102230 2024-04-26T01:46:58.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.871+0000 7f0ab0e00700 1 -- 172.21.15.6:0/117003402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aac1a5bc0 con 0x7f0aac102230 2024-04-26T01:46:58.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.873+0000 7f0ab0e00700 1 -- 172.21.15.6:0/117003402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aac069730 con 0x7f0aac102230 2024-04-26T01:46:58.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.878+0000 7f0aa3e00700 1 -- 172.21.15.6:0/117003402 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f0a98004750 con 0x7f0aac102230 2024-04-26T01:46:58.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.879+0000 7f0aa3e00700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0a900a3f50 0x7f0a900a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:58.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.879+0000 7f0aa3e00700 1 -- 172.21.15.6:0/117003402 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f0a980d0cf0 con 0x7f0aac102230 2024-04-26T01:46:58.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.880+0000 7f0aab400700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0a900a3f50 0x7f0a900a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:58.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.881+0000 7f0aab400700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0a900a3f50 0x7f0a900a6410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c006010 tx=0x7f0a9c00b260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:58.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:58.884+0000 7f0aa3e00700 1 -- 172.21.15.6:0/117003402 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a9809d0c0 con 0x7f0aac102230 2024-04-26T01:46:58.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:58 smithi188 ceph-mon[32751]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T01:46:58.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:58 smithi029 ceph-mon[32775]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T01:46:59.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.243+0000 7f0ab0e00700 1 -- 172.21.15.6:0/117003402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0aac19a3a0 con 0x7f0aac102230 2024-04-26T01:46:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.244+0000 7f0aa3e00700 1 -- 172.21.15.6:0/117003402 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11850 (secure 0 0 0) 0x7f0a9809c9c0 con 0x7f0aac102230 2024-04-26T01:46:59.242 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:46:59.242 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:45:54.644030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1255666766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1255666766},{"type":"v1","addr":"172.21.15.188:6835","nonce":1255666766}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24512,24505]},"id":1}]} 2024-04-26T01:46:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.248+0000 7f0ab0e00700 1 -- 172.21.15.6:0/117003402 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0a900a3f50 msgr2=0x7f0a900a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.248+0000 7f0ab0e00700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0a900a3f50 0x7f0a900a6410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c006010 tx=0x7f0a9c00b260 comp rx=0 tx=0).stop 2024-04-26T01:46:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.248+0000 7f0ab0e00700 1 -- 172.21.15.6:0/117003402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aac102230 msgr2=0x7f0aac199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.248+0000 7f0ab0e00700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aac102230 0x7f0aac199580 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800e9c0 tx=0x7f0a9800ecd0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.248+0000 7f0ab0e00700 1 -- 172.21.15.6:0/117003402 shutdown_connections 2024-04-26T01:46:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.248+0000 7f0ab0e00700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0a900a3f50 0x7f0a900a6410 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.248+0000 7f0ab0e00700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aac107870 0x7f0aac19d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.248+0000 7f0ab0e00700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0aac101870 0x7f0aac199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.248+0000 7f0ab0e00700 1 --2- 172.21.15.6:0/117003402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0aac102230 0x7f0aac199580 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.249+0000 7f0ab0e00700 1 -- 172.21.15.6:0/117003402 >> 172.21.15.6:0/117003402 conn(0x7f0aac0fd360 msgr2=0x7f0aac105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.249+0000 7f0ab0e00700 1 -- 172.21.15.6:0/117003402 shutdown_connections 2024-04-26T01:46:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:46:59.249+0000 7f0ab0e00700 1 -- 172.21.15.6:0/117003402 wait complete. 2024-04-26T01:46:59.247 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-04-26T01:46:59.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:46:59 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/117003402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:59.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:46:59 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/117003402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:59.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:46:59 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/117003402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:00.068 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 85.6 --await-for 42.0' 2024-04-26T01:47:00.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 85.6 --await-for 42.0 2024-04-26T01:47:00.601 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:47:00.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:00 smithi006 ceph-mon[28419]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:47:00.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:00 smithi188 ceph-mon[32751]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:47:00.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:00 smithi029 ceph-mon[32775]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:47:02.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:02 smithi006 ceph-mon[28419]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-04-26T01:47:02.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.798+0000 7f33faa00700 1 -- 172.21.15.6:0/2691630017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33f4104410 msgr2=0x7f33f41088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:02.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.798+0000 7f33faa00700 1 --2- 172.21.15.6:0/2691630017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33f4104410 0x7f33f41088c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009990 tx=0x7f33e0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.798+0000 7f33faa00700 1 -- 172.21.15.6:0/2691630017 shutdown_connections 2024-04-26T01:47:02.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.798+0000 7f33faa00700 1 --2- 172.21.15.6:0/2691630017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f4108ff0 0x7f33f410f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.798+0000 7f33faa00700 1 --2- 172.21.15.6:0/2691630017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4103a60 0x7f33f4103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.798+0000 7f33faa00700 1 --2- 172.21.15.6:0/2691630017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33f4104410 0x7f33f41088c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.798+0000 7f33faa00700 1 -- 172.21.15.6:0/2691630017 >> 172.21.15.6:0/2691630017 conn(0x7f33f40fd290 msgr2=0x7f33f40ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:02.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.798+0000 7f33faa00700 1 -- 172.21.15.6:0/2691630017 shutdown_connections 2024-04-26T01:47:02.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.798+0000 7f33faa00700 1 -- 172.21.15.6:0/2691630017 wait complete. 2024-04-26T01:47:02.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.799+0000 7f33faa00700 1 Processor -- start 2024-04-26T01:47:02.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.799+0000 7f33faa00700 1 -- start start 2024-04-26T01:47:02.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.800+0000 7f33faa00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4103a60 0x7f33f419b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.800+0000 7f33faa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33f4104410 0x7f33f419b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.800+0000 7f33faa00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f4108ff0 0x7f33f419fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.800+0000 7f33faa00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f33f4112c20 con 0x7f33f4103a60 2024-04-26T01:47:02.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.800+0000 7f33faa00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f33f4112aa0 con 0x7f33f4108ff0 2024-04-26T01:47:02.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.800+0000 7f33faa00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f33f4112da0 con 0x7f33f4104410 2024-04-26T01:47:02.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.800+0000 7f33fa000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f4108ff0 0x7f33f419fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.800+0000 7f33fa000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f4108ff0 0x7f33f419fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53816/0 (socket says 172.21.15.6:53816) 2024-04-26T01:47:02.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.800+0000 7f33fa000700 1 -- 172.21.15.6:0/1931775854 learned_addr learned my addr 172.21.15.6:0/1931775854 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:02.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.801+0000 7f33f8c00700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33f4104410 0x7f33f419b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.801+0000 7f33f9600700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4103a60 0x7f33f419b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.801+0000 7f33f9600700 1 -- 172.21.15.6:0/1931775854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33f4104410 msgr2=0x7f33f419b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:02.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.801+0000 7f33f9600700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33f4104410 0x7f33f419b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.801+0000 7f33f9600700 1 -- 172.21.15.6:0/1931775854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f4108ff0 msgr2=0x7f33f419fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:02.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.801+0000 7f33f9600700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f4108ff0 0x7f33f419fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.801+0000 7f33f9600700 1 -- 172.21.15.6:0/1931775854 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33f41a0250 con 0x7f33f4103a60 2024-04-26T01:47:02.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.802+0000 7f33f9600700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4103a60 0x7f33f419b240 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f33e800e9c0 tx=0x7f33e800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:02.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.802+0000 7f33f2000700 1 -- 172.21.15.6:0/1931775854 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e8004220 con 0x7f33f4103a60 2024-04-26T01:47:02.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.802+0000 7f33faa00700 1 -- 172.21.15.6:0/1931775854 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33f41a0540 con 0x7f33f4103a60 2024-04-26T01:47:02.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.802+0000 7f33faa00700 1 -- 172.21.15.6:0/1931775854 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33f41a7dc0 con 0x7f33f4103a60 2024-04-26T01:47:02.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.803+0000 7f33f2000700 1 -- 172.21.15.6:0/1931775854 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33e80043c0 con 0x7f33f4103a60 2024-04-26T01:47:02.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.803+0000 7f33f2000700 1 -- 172.21.15.6:0/1931775854 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e8003c50 con 0x7f33f4103a60 2024-04-26T01:47:02.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.807+0000 7f3406906b80 1 Processor -- start 2024-04-26T01:47:02.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.808+0000 7f3406906b80 1 -- start start 2024-04-26T01:47:02.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.808+0000 7f3406906b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d5f30 0x5631687d6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.808+0000 7f3406906b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d6850 0x5631687e11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.808+0000 7f3406906b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687e1700 0x5631687e3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.808+0000 7f3406906b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x563168729020 con 0x5631687e1700 2024-04-26T01:47:02.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.808+0000 7f3406906b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5631686cf8c0 con 0x5631687d6850 2024-04-26T01:47:02.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.808+0000 7f3406906b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563168724fe0 con 0x5631687d5f30 2024-04-26T01:47:02.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.809+0000 7f33fa000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d5f30 0x5631687d6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.809+0000 7f33fa000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d5f30 0x5631687d6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:39900/0 (socket says 172.21.15.6:39900) 2024-04-26T01:47:02.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.809+0000 7f33fa000700 1 -- 172.21.15.6:0/2611479986 learned_addr learned my addr 172.21.15.6:0/2611479986 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:02.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.809+0000 7f33f8c00700 1 --2- 172.21.15.6:0/2611479986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687e1700 0x5631687e3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.810+0000 7f33f2000700 1 -- 172.21.15.6:0/1931775854 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f33e800c7c0 con 0x7f33f4103a60 2024-04-26T01:47:02.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.810+0000 7f33f8c00700 1 -- 172.21.15.6:0/2611479986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d5f30 msgr2=0x5631687d6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:02.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.810+0000 7f33f8c00700 1 --2- 172.21.15.6:0/2611479986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d5f30 0x5631687d6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.810+0000 7f33f8c00700 1 -- 172.21.15.6:0/2611479986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d6850 msgr2=0x5631687e11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.810+0000 7f33f9600700 1 --2- 172.21.15.6:0/2611479986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d6850 0x5631687e11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.811+0000 7f33f8c00700 1 --2- 172.21.15.6:0/2611479986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d6850 0x5631687e11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.811+0000 7f33f8c00700 1 -- 172.21.15.6:0/2611479986 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631686c5810 con 0x5631687e1700 2024-04-26T01:47:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.811+0000 7f33f9600700 1 --2- 172.21.15.6:0/2611479986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d6850 0x5631687e11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:47:02.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.812+0000 7f33f8c00700 1 --2- 172.21.15.6:0/2611479986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687e1700 0x5631687e3af0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f33e00098e0 tx=0x7f33e0005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7869b2ffe52881c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:02.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.812+0000 7f33e6a00700 1 -- 172.21.15.6:0/2611479986 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e0027070 con 0x5631687e1700 2024-04-26T01:47:02.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.812+0000 7f33e6a00700 1 -- 172.21.15.6:0/2611479986 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33e000bca0 con 0x5631687e1700 2024-04-26T01:47:02.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.812+0000 7f33e6a00700 1 -- 172.21.15.6:0/2611479986 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e00039f0 con 0x5631687e1700 2024-04-26T01:47:02.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.813+0000 7f3406906b80 1 -- 172.21.15.6:0/2611479986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687e1700 msgr2=0x5631687e3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.813+0000 7f3406906b80 1 --2- 172.21.15.6:0/2611479986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687e1700 0x5631687e3af0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f33e00098e0 tx=0x7f33e0005bc0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.815+0000 7f3406906b80 1 -- 172.21.15.6:0/2611479986 shutdown_connections 2024-04-26T01:47:02.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.815+0000 7f3406906b80 1 --2- 172.21.15.6:0/2611479986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687e1700 0x5631687e3af0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.815+0000 7f3406906b80 1 --2- 172.21.15.6:0/2611479986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d5f30 0x5631687d6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.815+0000 7f3406906b80 1 --2- 172.21.15.6:0/2611479986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d6850 0x5631687e11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.815+0000 7f3406906b80 1 -- 172.21.15.6:0/2611479986 >> 172.21.15.6:0/2611479986 conn(0x5631687050e0 msgr2=0x5631687d2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:02.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.815+0000 7f33f2000700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f33dc0a4070 0x7f33dc0a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.816+0000 7f33f2000700 1 -- 172.21.15.6:0/1931775854 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f33e80d0c00 con 0x7f33f4103a60 2024-04-26T01:47:02.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.816+0000 7f33f8c00700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f33dc0a4070 0x7f33dc0a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.816+0000 7f3406906b80 1 -- 172.21.15.6:0/2611479986 shutdown_connections 2024-04-26T01:47:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.817+0000 7f3406906b80 1 -- 172.21.15.6:0/2611479986 wait complete. 2024-04-26T01:47:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.817+0000 7f3406906b80 1 -- start start 2024-04-26T01:47:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.817+0000 7f33f8c00700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f33dc0a4070 0x7f33dc0a6530 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f33e0005b40 tx=0x7f33e001a040 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.818+0000 7f3406906b80 1 Processor -- start 2024-04-26T01:47:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.818+0000 7f3406906b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d5f30 0x5631687e0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.818+0000 7f3406906b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d6850 0x5631687de0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.818+0000 7f3406906b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687e1700 0x5631687de610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.818+0000 7f3406906b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5631687f5140 con 0x5631687e1700 2024-04-26T01:47:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.818+0000 7f3406906b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5631686e29f0 con 0x5631687d5f30 2024-04-26T01:47:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.818+0000 7f3406906b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5631687f52c0 con 0x5631687d6850 2024-04-26T01:47:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.818+0000 7f3406906b80 10 client.0 _ll_register_callbacks cb 0x5631686c18b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.819+0000 7f33f9600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d5f30 0x5631687e0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.819+0000 7f33f9600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d5f30 0x5631687e0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53832/0 (socket says 172.21.15.6:53832) 2024-04-26T01:47:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.819+0000 7f33f9600700 1 -- 172.21.15.6:0/4114046776 learned_addr learned my addr 172.21.15.6:0/4114046776 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.819+0000 7f33f8c00700 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d6850 0x5631687de0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.819+0000 7f33fa000700 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687e1700 0x5631687de610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.820+0000 7f33f9600700 1 -- 172.21.15.6:0/4114046776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d6850 msgr2=0x5631687de0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.820+0000 7f33f9600700 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d6850 0x5631687de0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.820+0000 7f33f9600700 1 -- 172.21.15.6:0/4114046776 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687e1700 msgr2=0x5631687de610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.820+0000 7f33f9600700 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687e1700 0x5631687de610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.820+0000 7f33f9600700 1 -- 172.21.15.6:0/4114046776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563168676a60 con 0x5631687d5f30 2024-04-26T01:47:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.820+0000 7f33fa000700 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687e1700 0x5631687de610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:47:02.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.820+0000 7f33f9600700 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d5f30 0x5631687e0dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f33e800f010 tx=0x7f33e809c2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54fe658ebf82e61e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.820+0000 7f33db400700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:47:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.820+0000 7f33db400700 1 -- 172.21.15.6:0/4114046776 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e809c8d0 con 0x5631687d5f30 2024-04-26T01:47:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.821+0000 7f33db400700 1 -- 172.21.15.6:0/4114046776 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f33e80870b0 con 0x5631687d5f30 2024-04-26T01:47:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.821+0000 7f3406906b80 1 -- 172.21.15.6:0/4114046776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33e000bca0 con 0x5631687d5f30 2024-04-26T01:47:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.821+0000 7f33db400700 1 -- 172.21.15.6:0/4114046776 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f33e80872b0 con 0x5631687d5f30 2024-04-26T01:47:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.821+0000 7f3406906b80 10 client.25310 fetch_fsmap learned FSMap version 26 2024-04-26T01:47:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.821+0000 7f3406906b80 1 -- 172.21.15.6:0/4114046776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5631686bfc80 con 0x5631687d5f30 2024-04-26T01:47:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.822+0000 7f33db400700 1 -- 172.21.15.6:0/4114046776 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33e000bca0 con 0x5631687d5f30 2024-04-26T01:47:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.822+0000 7f33db400700 1 -- 172.21.15.6:0/4114046776 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f33e80ce430 con 0x5631687d5f30 2024-04-26T01:47:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.822+0000 7f3406906b80 10 client.25310 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:47:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.822+0000 7f3406906b80 10 client.25310 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:47:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.822+0000 7f3406906b80 20 client.25310 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.822+0000 7f3406906b80 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563168801e40 0x563168804300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.823+0000 7f3406906b80 4 client.25310 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.823+0000 7f3406906b80 1 -- 172.21.15.6:0/4114046776 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563168736810 con 0x563168801e40 2024-04-26T01:47:02.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.823+0000 7f33f8c00700 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563168801e40 0x563168804300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.824+0000 7f33f8c00700 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563168801e40 0x563168804300 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7a3fb123ee47888 server_cookie=1d2df56573dc3f6c in_seq=0 out_seq=0 2024-04-26T01:47:02.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.824+0000 7f33db400700 10 client.25310 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.825+0000 7f33db400700 1 -- 172.21.15.6:0/4114046776 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563168736810 con 0x563168801e40 2024-04-26T01:47:02.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.825+0000 7f33db400700 10 client.25310 handle_command_reply: tid=0 2024-04-26T01:47:02.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.825+0000 7f3406906b80 1 client.25310 shutdown 2024-04-26T01:47:02.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.825+0000 7f3406906b80 1 -- 172.21.15.6:0/4114046776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d5f30 msgr2=0x5631687e0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:02.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.825+0000 7f3406906b80 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d5f30 0x5631687e0dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f33e800f010 tx=0x7f33e809c2e0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.827+0000 7f3406906b80 1 -- 172.21.15.6:0/4114046776 shutdown_connections 2024-04-26T01:47:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.827+0000 7f3406906b80 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687e1700 0x5631687de610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.827+0000 7f3406906b80 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687d5f30 0x5631687e0dd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.827+0000 7f3406906b80 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563168801e40 0x563168804300 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.827+0000 7f3406906b80 1 --2- 172.21.15.6:0/4114046776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d6850 0x5631687de0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.827+0000 7f33db400700 0 client.25310 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.827+0000 7f3406906b80 1 -- 172.21.15.6:0/4114046776 >> 172.21.15.6:0/4114046776 conn(0x5631687050e0 msgr2=0x5631687d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.827+0000 7f3406906b80 1 -- 172.21.15.6:0/4114046776 shutdown_connections 2024-04-26T01:47:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.827+0000 7f3406906b80 1 -- 172.21.15.6:0/4114046776 wait complete. 2024-04-26T01:47:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.828+0000 7f3406906b80 20 client.25310 trim_cache size 0 max 16384 2024-04-26T01:47:02.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.864+0000 7f3406906b80 1 Processor -- start 2024-04-26T01:47:02.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.865+0000 7f3406906b80 1 -- start start 2024-04-26T01:47:02.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.865+0000 7f3406906b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d5f30 0x563168809c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.865+0000 7f3406906b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d6850 0x56316880a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.865+0000 7f3406906b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687e1700 0x5631687fa560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.865+0000 7f3406906b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f33d400ff40 con 0x5631687d6850 2024-04-26T01:47:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.865+0000 7f3406906b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f33d400d7d0 con 0x5631687e1700 2024-04-26T01:47:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.865+0000 7f3406906b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f33d400b620 con 0x5631687d5f30 2024-04-26T01:47:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.866+0000 7f33f8c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d6850 0x56316880a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.866+0000 7f33f8c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d6850 0x56316880a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58694/0 (socket says 172.21.15.6:58694) 2024-04-26T01:47:02.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.866+0000 7f33f8c00700 1 -- 172.21.15.6:0/3088299939 learned_addr learned my addr 172.21.15.6:0/3088299939 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:02.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.866+0000 7f33f8c00700 1 -- 172.21.15.6:0/3088299939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d5f30 msgr2=0x563168809c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:47:02.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.866+0000 7f33f8c00700 1 --2- 172.21.15.6:0/3088299939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d5f30 0x563168809c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.866+0000 7f33f8c00700 1 -- 172.21.15.6:0/3088299939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687e1700 msgr2=0x5631687fa560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:47:02.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.866+0000 7f33f8c00700 1 --2- 172.21.15.6:0/3088299939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687e1700 0x5631687fa560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.866+0000 7f33f8c00700 1 -- 172.21.15.6:0/3088299939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563168804980 con 0x5631687d6850 2024-04-26T01:47:02.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.866+0000 7f33f8c00700 1 --2- 172.21.15.6:0/3088299939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d6850 0x56316880a150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f33e0005850 tx=0x7f33e0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf07db603dad3b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.867+0000 7f33e6000700 1 -- 172.21.15.6:0/3088299939 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e0040070 con 0x5631687d6850 2024-04-26T01:47:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.867+0000 7f33e6000700 1 -- 172.21.15.6:0/3088299939 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33e0028040 con 0x5631687d6850 2024-04-26T01:47:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.867+0000 7f33e6000700 1 -- 172.21.15.6:0/3088299939 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e003b620 con 0x5631687d6850 2024-04-26T01:47:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.868+0000 7f3406906b80 1 -- 172.21.15.6:0/3088299939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d6850 msgr2=0x56316880a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.868+0000 7f3406906b80 1 --2- 172.21.15.6:0/3088299939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d6850 0x56316880a150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f33e0005850 tx=0x7f33e0000bc0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.868+0000 7f3406906b80 1 -- 172.21.15.6:0/3088299939 shutdown_connections 2024-04-26T01:47:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.868+0000 7f3406906b80 1 --2- 172.21.15.6:0/3088299939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687e1700 0x5631687fa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.868+0000 7f3406906b80 1 --2- 172.21.15.6:0/3088299939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687d5f30 0x563168809c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.868+0000 7f3406906b80 1 --2- 172.21.15.6:0/3088299939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d6850 0x56316880a150 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f33e0005850 tx=0x7f33e0000bc0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.868+0000 7f3406906b80 1 -- 172.21.15.6:0/3088299939 >> 172.21.15.6:0/3088299939 conn(0x5631687050e0 msgr2=0x5631687e0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.869+0000 7f3406906b80 1 -- 172.21.15.6:0/3088299939 shutdown_connections 2024-04-26T01:47:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.869+0000 7f3406906b80 1 -- 172.21.15.6:0/3088299939 wait complete. 2024-04-26T01:47:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.870+0000 7f3406906b80 1 -- start start 2024-04-26T01:47:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.871+0000 7f3406906b80 1 Processor -- start 2024-04-26T01:47:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.871+0000 7f3406906b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d5f30 0x5631687f7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.871+0000 7f3406906b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687e1700 0x5631687f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.871+0000 7f3406906b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687f7fc0 0x563168868490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.871+0000 7f3406906b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f33d400bc90 con 0x5631687d5f30 2024-04-26T01:47:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.871+0000 7f3406906b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f33d400d100 con 0x5631687f7fc0 2024-04-26T01:47:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.871+0000 7f3406906b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f33d400c360 con 0x5631687e1700 2024-04-26T01:47:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.871+0000 7f3406906b80 10 client.0 _ll_register_callbacks cb 0x5631686c18b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.872+0000 7f33f9600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d5f30 0x5631687f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.872+0000 7f33fa000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687f7fc0 0x563168868490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.872+0000 7f33fa000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687f7fc0 0x563168868490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53850/0 (socket says 172.21.15.6:53850) 2024-04-26T01:47:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.872+0000 7f33fa000700 1 -- 172.21.15.6:0/4148478467 learned_addr learned my addr 172.21.15.6:0/4148478467 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.872+0000 7f33f8c00700 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687e1700 0x5631687f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.872+0000 7f33f8c00700 1 -- 172.21.15.6:0/4148478467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687f7fc0 msgr2=0x563168868490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.872+0000 7f33f8c00700 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687f7fc0 0x563168868490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.872+0000 7f33f8c00700 1 -- 172.21.15.6:0/4148478467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d5f30 msgr2=0x5631687f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.872+0000 7f33f8c00700 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d5f30 0x5631687f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.872+0000 7f33f8c00700 1 -- 172.21.15.6:0/4148478467 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631687d7d80 con 0x5631687e1700 2024-04-26T01:47:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.873+0000 7f33fa000700 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687f7fc0 0x563168868490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.873+0000 7f33f9600700 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d5f30 0x5631687f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.873+0000 7f33f8c00700 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687e1700 0x5631687f7a80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x5631687dc630 tx=0x7f33e000b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73f7bf304c74d4b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.873+0000 7f33db400700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:47:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.873+0000 7f33db400700 1 -- 172.21.15.6:0/4148478467 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e0048060 con 0x5631687e1700 2024-04-26T01:47:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.874+0000 7f3406906b80 1 -- 172.21.15.6:0/4148478467 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33e0028040 con 0x5631687e1700 2024-04-26T01:47:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.874+0000 7f33db400700 1 -- 172.21.15.6:0/4148478467 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f33e0003680 con 0x5631687e1700 2024-04-26T01:47:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.874+0000 7f33db400700 1 -- 172.21.15.6:0/4148478467 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f33e0040070 con 0x5631687e1700 2024-04-26T01:47:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.874+0000 7f3406906b80 10 client.25351 fetch_fsmap learned FSMap version 26 2024-04-26T01:47:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.874+0000 7f3406906b80 1 -- 172.21.15.6:0/4148478467 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5631688698f0 con 0x5631687e1700 2024-04-26T01:47:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.875+0000 7f33db400700 1 -- 172.21.15.6:0/4148478467 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33e0028040 con 0x5631687e1700 2024-04-26T01:47:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.875+0000 7f33db400700 1 -- 172.21.15.6:0/4148478467 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f33e0021070 con 0x5631687e1700 2024-04-26T01:47:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.875+0000 7f3406906b80 10 client.25351 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:47:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.875+0000 7f3406906b80 10 client.25351 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:47:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.875+0000 7f3406906b80 20 client.25351 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.875+0000 7f3406906b80 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563168869b20 0x56316886bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.875+0000 7f3406906b80 4 client.25351 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.6, "await_for": 42.0}] 2024-04-26T01:47:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.875+0000 7f3406906b80 1 -- 172.21.15.6:0/4148478467 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.6, "await_for": 42.0}) -- 0x5631687308b0 con 0x563168869b20 2024-04-26T01:47:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.876+0000 7f33f9600700 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563168869b20 0x56316886bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.876+0000 7f33f9600700 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563168869b20 0x56316886bf10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2c1468a12facd08 server_cookie=decbe297bbe6bcf6 in_seq=0 out_seq=0 2024-04-26T01:47:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.876+0000 7f33db400700 10 client.25351 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:02.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.888+0000 7f33db400700 1 -- 172.21.15.6:0/4148478467 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5631687308b0 con 0x563168869b20 2024-04-26T01:47:02.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.888+0000 7f33db400700 10 client.25351 handle_command_reply: tid=0 2024-04-26T01:47:02.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.888+0000 7f3406906b80 1 client.25351 shutdown 2024-04-26T01:47:02.887 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:47:02.887 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 26, 2024-04-26T01:47:02.887 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 2, 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "74c6c8bd": { 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 2, 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 85.6, 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:47:02.888 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:47:02.889 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:47:02.889 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:47:02.889 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:47:02.889 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:47:02.889 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:47:02.889 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:47:02.889 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:47:02.889 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:47:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:02 smithi188 ceph-mon[32751]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-04-26T01:47:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.888+0000 7f3406906b80 1 -- 172.21.15.6:0/4148478467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687e1700 msgr2=0x5631687f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.889+0000 7f3406906b80 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687e1700 0x5631687f7a80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x5631687dc630 tx=0x7f33e000b8a0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.889+0000 7f3406906b80 1 -- 172.21.15.6:0/4148478467 shutdown_connections 2024-04-26T01:47:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.889+0000 7f3406906b80 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631687f7fc0 0x563168868490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.889+0000 7f3406906b80 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5631687d5f30 0x5631687f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.889+0000 7f3406906b80 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563168869b20 0x56316886bf10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.889+0000 7f3406906b80 1 --2- 172.21.15.6:0/4148478467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631687e1700 0x5631687f7a80 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x5631687dc630 tx=0x7f33e000b8a0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.889+0000 7f33db400700 0 client.25351 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.889+0000 7f3406906b80 1 -- 172.21.15.6:0/4148478467 >> 172.21.15.6:0/4148478467 conn(0x5631687050e0 msgr2=0x5631687d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.889+0000 7f3406906b80 1 -- 172.21.15.6:0/4148478467 shutdown_connections 2024-04-26T01:47:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.889+0000 7f3406906b80 1 -- 172.21.15.6:0/4148478467 wait complete. 2024-04-26T01:47:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.889+0000 7f3406906b80 20 client.25351 trim_cache size 0 max 16384 2024-04-26T01:47:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.890+0000 7f33faa00700 1 -- 172.21.15.6:0/1931775854 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f33dc0a4070 msgr2=0x7f33dc0a6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.890+0000 7f33faa00700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f33dc0a4070 0x7f33dc0a6530 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f33e0005b40 tx=0x7f33e001a040 comp rx=0 tx=0).stop 2024-04-26T01:47:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.890+0000 7f33faa00700 1 -- 172.21.15.6:0/1931775854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4103a60 msgr2=0x7f33f419b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.890+0000 7f33faa00700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4103a60 0x7f33f419b240 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f33e800e9c0 tx=0x7f33e800ecd0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.891+0000 7f33faa00700 1 -- 172.21.15.6:0/1931775854 shutdown_connections 2024-04-26T01:47:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.891+0000 7f33faa00700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f33dc0a4070 0x7f33dc0a6530 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.891+0000 7f33faa00700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f4108ff0 0x7f33f419fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.891+0000 7f33faa00700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4103a60 0x7f33f419b240 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.891+0000 7f33faa00700 1 --2- 172.21.15.6:0/1931775854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33f4104410 0x7f33f419b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:02.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.891+0000 7f33faa00700 1 -- 172.21.15.6:0/1931775854 >> 172.21.15.6:0/1931775854 conn(0x7f33f40fd290 msgr2=0x7f33f4101fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:02.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.894+0000 7f33faa00700 1 -- 172.21.15.6:0/1931775854 shutdown_connections 2024-04-26T01:47:02.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:02.894+0000 7f33faa00700 1 -- 172.21.15.6:0/1931775854 wait complete. 2024-04-26T01:47:02.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:02 smithi029 ceph-mon[32775]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-04-26T01:47:03.733 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '74c6c8bd', quiesce took 9.2 seconds. Will release after: 25.6 2024-04-26T01:47:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.830+0000 7f8d18400640 1 -- 172.21.15.6:0/2698000960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d101043c0 msgr2=0x7f8d101047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.830+0000 7f8d18400640 1 --2- 172.21.15.6:0/2698000960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d101043c0 0x7f8d101047a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009900 tx=0x7f8d0402a3f0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.831+0000 7f8d18400640 1 -- 172.21.15.6:0/2698000960 shutdown_connections 2024-04-26T01:47:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.831+0000 7f8d18400640 1 --2- 172.21.15.6:0/2698000960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d10104d70 0x7f8d10108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.831+0000 7f8d18400640 1 --2- 172.21.15.6:0/2698000960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d10109490 0x7f8d10111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.831+0000 7f8d18400640 1 --2- 172.21.15.6:0/2698000960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d101043c0 0x7f8d101047a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.831+0000 7f8d18400640 1 -- 172.21.15.6:0/2698000960 >> 172.21.15.6:0/2698000960 conn(0x7f8d100ffc40 msgr2=0x7f8d10102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.831+0000 7f8d18400640 1 -- 172.21.15.6:0/2698000960 shutdown_connections 2024-04-26T01:47:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.831+0000 7f8d18400640 1 -- 172.21.15.6:0/2698000960 wait complete. 2024-04-26T01:47:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.832+0000 7f8d18400640 1 Processor -- start 2024-04-26T01:47:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.832+0000 7f8d18400640 1 -- start start 2024-04-26T01:47:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.832+0000 7f8d18400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d101043c0 0x7f8d101a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.832+0000 7f8d18400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d10109490 0x7f8d101a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.832+0000 7f8d18400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1019a7e0 0x7f8d1019ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.832+0000 7f8d18400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8d10114400 con 0x7f8d10109490 2024-04-26T01:47:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.832+0000 7f8d18400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8d10114280 con 0x7f8d1019a7e0 2024-04-26T01:47:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.832+0000 7f8d18400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8d10114580 con 0x7f8d101043c0 2024-04-26T01:47:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.833+0000 7f8d15a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d101043c0 0x7f8d101a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.833+0000 7f8d15a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d101043c0 0x7f8d101a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:39928/0 (socket says 172.21.15.6:39928) 2024-04-26T01:47:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.833+0000 7f8d15a00640 1 -- 172.21.15.6:0/851462217 learned_addr learned my addr 172.21.15.6:0/851462217 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.833+0000 7f8d16400640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1019a7e0 0x7f8d1019ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.833+0000 7f8d15000640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d10109490 0x7f8d101a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.833+0000 7f8d15a00640 1 -- 172.21.15.6:0/851462217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1019a7e0 msgr2=0x7f8d1019ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.833+0000 7f8d15a00640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1019a7e0 0x7f8d1019ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.833+0000 7f8d15a00640 1 -- 172.21.15.6:0/851462217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d10109490 msgr2=0x7f8d101a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.833+0000 7f8d15a00640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d10109490 0x7f8d101a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.833+0000 7f8d15a00640 1 -- 172.21.15.6:0/851462217 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1019b370 con 0x7f8d101043c0 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.833+0000 7f8d16400640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1019a7e0 0x7f8d1019ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.833+0000 7f8d15000640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d10109490 0x7f8d101a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.834+0000 7f8d15a00640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d101043c0 0x7f8d101a0670 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8d04005940 tx=0x7f8d040040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.834+0000 7f8d0e000640 1 -- 172.21.15.6:0/851462217 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0403f030 con 0x7f8d101043c0 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.834+0000 7f8d18400640 1 -- 172.21.15.6:0/851462217 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1019b600 con 0x7f8d101043c0 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.834+0000 7f8d0e000640 1 -- 172.21.15.6:0/851462217 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d040044c0 con 0x7f8d101043c0 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.834+0000 7f8d0e000640 1 -- 172.21.15.6:0/851462217 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04035070 con 0x7f8d101043c0 2024-04-26T01:47:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.834+0000 7f8d18400640 1 -- 172.21.15.6:0/851462217 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d101a77e0 con 0x7f8d101043c0 2024-04-26T01:47:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.839+0000 7f8d1b20e740 1 Processor -- start 2024-04-26T01:47:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.839+0000 7f8d1b20e740 1 -- start start 2024-04-26T01:47:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.839+0000 7f8d0e000640 1 -- 172.21.15.6:0/851462217 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f8d04042030 con 0x7f8d101043c0 2024-04-26T01:47:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.839+0000 7f8d1b20e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06e2820 0x55cbe06e2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.839+0000 7f8d1b20e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 0x55cbe06ed4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.840+0000 7f8d1b20e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06eda00 0x55cbe06efdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.840+0000 7f8d1b20e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55cbe0648900 con 0x55cbe06e3140 2024-04-26T01:47:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.840+0000 7f8d1b20e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cbe05eeef0 con 0x55cbe06eda00 2024-04-26T01:47:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.840+0000 7f8d1b20e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55cbe06448c0 con 0x55cbe06e2820 2024-04-26T01:47:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.840+0000 7f8d15a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 0x55cbe06ed4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.840+0000 7f8d15a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 0x55cbe06ed4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58710/0 (socket says 172.21.15.6:58710) 2024-04-26T01:47:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.840+0000 7f8d15a00640 1 -- 172.21.15.6:0/793332680 learned_addr learned my addr 172.21.15.6:0/793332680 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.840+0000 7f8d16400640 1 --2- 172.21.15.6:0/793332680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06e2820 0x55cbe06e2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.840+0000 7f8d15000640 1 --2- 172.21.15.6:0/793332680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06eda00 0x55cbe06efdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.841+0000 7f8d16400640 1 -- 172.21.15.6:0/793332680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06eda00 msgr2=0x55cbe06efdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.841+0000 7f8d16400640 1 --2- 172.21.15.6:0/793332680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06eda00 0x55cbe06efdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.841+0000 7f8d16400640 1 -- 172.21.15.6:0/793332680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 msgr2=0x55cbe06ed4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.841+0000 7f8d16400640 1 --2- 172.21.15.6:0/793332680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 0x55cbe06ed4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.841+0000 7f8d16400640 1 -- 172.21.15.6:0/793332680 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbe035b210 con 0x55cbe06e2820 2024-04-26T01:47:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.841+0000 7f8d15a00640 1 --2- 172.21.15.6:0/793332680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 0x55cbe06ed4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.841+0000 7f8d16400640 1 --2- 172.21.15.6:0/793332680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06e2820 0x55cbe06e2c00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8d08005fd0 tx=0x7f8d0800fb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bc5e0a029262292 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.841+0000 7f8d02a00640 1 -- 172.21.15.6:0/793332680 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08004b20 con 0x55cbe06e2820 2024-04-26T01:47:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.841+0000 7f8d02a00640 1 -- 172.21.15.6:0/793332680 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d0800ee40 con 0x55cbe06e2820 2024-04-26T01:47:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.841+0000 7f8d02a00640 1 -- 172.21.15.6:0/793332680 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d080042b0 con 0x55cbe06e2820 2024-04-26T01:47:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.842+0000 7f8d0e000640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8cf80a3eb0 0x7f8cf80a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.842+0000 7f8d0e000640 1 -- 172.21.15.6:0/851462217 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f8d040f0660 con 0x7f8d101043c0 2024-04-26T01:47:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.842+0000 7f8d15a00640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8cf80a3eb0 0x7f8cf80a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.842+0000 7f8d1b20e740 1 -- 172.21.15.6:0/793332680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06e2820 msgr2=0x55cbe06e2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.842+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/793332680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06e2820 0x55cbe06e2c00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8d08005fd0 tx=0x7f8d0800fb70 comp rx=0 tx=0).stop 2024-04-26T01:47:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.843+0000 7f8d1b20e740 1 -- 172.21.15.6:0/793332680 shutdown_connections 2024-04-26T01:47:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.843+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/793332680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 0x55cbe06ed4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.843+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/793332680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06eda00 0x55cbe06efdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.843+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/793332680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06e2820 0x55cbe06e2c00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.843+0000 7f8d1b20e740 1 -- 172.21.15.6:0/793332680 >> 172.21.15.6:0/793332680 conn(0x55cbe06ded30 msgr2=0x55cbe06df110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.843+0000 7f8d15a00640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8cf80a3eb0 0x7f8cf80a6370 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d040c4b80 tx=0x7f8d040f6530 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.843+0000 7f8d1b20e740 1 -- 172.21.15.6:0/793332680 shutdown_connections 2024-04-26T01:47:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.843+0000 7f8d1b20e740 1 -- 172.21.15.6:0/793332680 wait complete. 2024-04-26T01:47:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.844+0000 7f8d1b20e740 1 -- start start 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.844+0000 7f8d1b20e740 1 Processor -- start 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.844+0000 7f8d1b20e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e2820 0x55cbe06ea9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.844+0000 7f8d1b20e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e3140 0x55cbe06eaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.845+0000 7f8d1b20e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06eda00 0x55cbe06eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.845+0000 7f8d1b20e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55cbe06f0630 con 0x55cbe06e2820 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.845+0000 7f8d1b20e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cbe0602310 con 0x55cbe06e3140 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.845+0000 7f8d1b20e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55cbe06f07b0 con 0x55cbe06eda00 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.845+0000 7f8d1b20e740 10 client.0 _ll_register_callbacks cb 0x55cbe05d0cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.845+0000 7f8d16400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e2820 0x55cbe06ea9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.845+0000 7f8d16400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e2820 0x55cbe06ea9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58726/0 (socket says 172.21.15.6:58726) 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.845+0000 7f8d16400640 1 -- 172.21.15.6:0/1504902179 learned_addr learned my addr 172.21.15.6:0/1504902179 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.845+0000 7f8d15000640 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e3140 0x55cbe06eaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.846+0000 7f8d15000640 1 -- 172.21.15.6:0/1504902179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06eda00 msgr2=0x55cbe06eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.846+0000 7f8d15000640 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06eda00 0x55cbe06eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.846+0000 7f8d15000640 1 -- 172.21.15.6:0/1504902179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e2820 msgr2=0x55cbe06ea9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.846+0000 7f8d15000640 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e2820 0x55cbe06ea9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.846+0000 7f8d15000640 1 -- 172.21.15.6:0/1504902179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbe0427620 con 0x55cbe06e3140 2024-04-26T01:47:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.846+0000 7f8d16400640 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e2820 0x55cbe06ea9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.847+0000 7f8d15000640 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e3140 0x55cbe06eaf00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8d101000d0 tx=0x7f8cfc0039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a24e4d7efc71818 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.847+0000 7f8cf7400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:47:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.847+0000 7f8cf7400640 1 -- 172.21.15.6:0/1504902179 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc012860 con 0x55cbe06e3140 2024-04-26T01:47:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.847+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1504902179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cbe065c8b0 con 0x55cbe06e3140 2024-04-26T01:47:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.847+0000 7f8cf7400640 1 -- 172.21.15.6:0/1504902179 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f8cfc012a00 con 0x55cbe06e3140 2024-04-26T01:47:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.847+0000 7f8cf7400640 1 -- 172.21.15.6:0/1504902179 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8cfc012c00 con 0x55cbe06e3140 2024-04-26T01:47:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.847+0000 7f8d1b20e740 10 client.25325 fetch_fsmap learned FSMap version 26 2024-04-26T01:47:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.847+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1504902179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cbe06fc9b0 con 0x55cbe06e3140 2024-04-26T01:47:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.848+0000 7f8cf7400640 1 -- 172.21.15.6:0/1504902179 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55cbe065c8b0 con 0x55cbe06e3140 2024-04-26T01:47:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.848+0000 7f8cf7400640 1 -- 172.21.15.6:0/1504902179 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8cfc01a030 con 0x55cbe06e3140 2024-04-26T01:47:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.848+0000 7f8d1b20e740 10 client.25325 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:47:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.848+0000 7f8d1b20e740 10 client.25325 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.848+0000 7f8d1b20e740 20 client.25325 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.848+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cbe06fde20 0x55cbe0700230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.848+0000 7f8d1b20e740 4 client.25325 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.848+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1504902179 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbe0656320 con 0x55cbe06fde20 2024-04-26T01:47:03.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.848+0000 7f8d16400640 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cbe06fde20 0x55cbe0700230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.849+0000 7f8d16400640 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cbe06fde20 0x55cbe0700230 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8044ca794780c117 server_cookie=1da83164a3e55645 in_seq=0 out_seq=0 2024-04-26T01:47:03.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.849+0000 7f8cf7400640 10 client.25325 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:03.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.850+0000 7f8cf7400640 1 -- 172.21.15.6:0/1504902179 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cbe0656320 con 0x55cbe06fde20 2024-04-26T01:47:03.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.850+0000 7f8cf7400640 10 client.25325 handle_command_reply: tid=0 2024-04-26T01:47:03.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.850+0000 7f8d1b20e740 1 client.25325 shutdown 2024-04-26T01:47:03.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.850+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1504902179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e3140 msgr2=0x55cbe06eaf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:03.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.850+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e3140 0x55cbe06eaf00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8d101000d0 tx=0x7f8cfc0039c0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.850+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1504902179 shutdown_connections 2024-04-26T01:47:03.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.850+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cbe06fde20 0x55cbe0700230 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.850+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06eda00 0x55cbe06eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.850+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e3140 0x55cbe06eaf00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.850+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/1504902179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e2820 0x55cbe06ea9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.850+0000 7f8cf7400640 0 client.25325 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:03.849 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.851+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1504902179 >> 172.21.15.6:0/1504902179 conn(0x55cbe06ded30 msgr2=0x55cbe06e4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:03.849 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.851+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1504902179 shutdown_connections 2024-04-26T01:47:03.849 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.851+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1504902179 wait complete. 2024-04-26T01:47:03.849 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.851+0000 7f8d1b20e740 20 client.25325 trim_cache size 0 max 16384 2024-04-26T01:47:03.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.867+0000 7f8d1b20e740 1 Processor -- start 2024-04-26T01:47:03.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.867+0000 7f8d1b20e740 1 -- start start 2024-04-26T01:47:03.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.868+0000 7f8d1b20e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 0x55cbe06f94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.868+0000 7f8d1b20e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06e3140 0x55cbe06eb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.868+0000 7f8d1b20e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06eda00 0x55cbe06e48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.868+0000 7f8d1b20e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101ae3c0 con 0x55cbe06eda00 2024-04-26T01:47:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.868+0000 7f8d1b20e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101abc90 con 0x55cbe06e2820 2024-04-26T01:47:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.868+0000 7f8d1b20e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101a9ae0 con 0x55cbe06e3140 2024-04-26T01:47:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.868+0000 7f8d16400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06eda00 0x55cbe06e48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.868+0000 7f8d15000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06e3140 0x55cbe06eb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.868+0000 7f8d16400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06eda00 0x55cbe06e48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58736/0 (socket says 172.21.15.6:58736) 2024-04-26T01:47:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.868+0000 7f8d16400640 1 -- 172.21.15.6:0/2222333488 learned_addr learned my addr 172.21.15.6:0/2222333488 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.868+0000 7f8d16400640 1 --2- 172.21.15.6:0/2222333488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 0x55cbe06f94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.869+0000 7f8d16400640 1 -- 172.21.15.6:0/2222333488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06e3140 msgr2=0x55cbe06eb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.869+0000 7f8d16400640 1 --2- 172.21.15.6:0/2222333488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06e3140 0x55cbe06eb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.869+0000 7f8d16400640 1 -- 172.21.15.6:0/2222333488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06eda00 msgr2=0x55cbe06e48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.869+0000 7f8d16400640 1 --2- 172.21.15.6:0/2222333488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06eda00 0x55cbe06e48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.869+0000 7f8d16400640 1 -- 172.21.15.6:0/2222333488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbe07008b0 con 0x55cbe06e2820 2024-04-26T01:47:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.869+0000 7f8d15000640 1 --2- 172.21.15.6:0/2222333488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06e3140 0x55cbe06eb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.869+0000 7f8d16400640 1 --2- 172.21.15.6:0/2222333488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 0x55cbe06f94f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8d08005fd0 tx=0x7f8d0800fb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82ac3952e7201aed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.870+0000 7f8d02000640 1 -- 172.21.15.6:0/2222333488 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08003d40 con 0x55cbe06e2820 2024-04-26T01:47:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.870+0000 7f8d02000640 1 -- 172.21.15.6:0/2222333488 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d0800e990 con 0x55cbe06e2820 2024-04-26T01:47:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.870+0000 7f8d02000640 1 -- 172.21.15.6:0/2222333488 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08035530 con 0x55cbe06e2820 2024-04-26T01:47:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.870+0000 7f8d1b20e740 1 -- 172.21.15.6:0/2222333488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 msgr2=0x55cbe06f94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.870+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/2222333488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 0x55cbe06f94f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8d08005fd0 tx=0x7f8d0800fb70 comp rx=0 tx=0).stop 2024-04-26T01:47:03.868 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.871+0000 7f8d1b20e740 1 -- 172.21.15.6:0/2222333488 shutdown_connections 2024-04-26T01:47:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.871+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/2222333488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06eda00 0x55cbe06e48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.871+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/2222333488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06e3140 0x55cbe06eb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.871+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/2222333488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 0x55cbe06f94f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.871+0000 7f8d1b20e740 1 -- 172.21.15.6:0/2222333488 >> 172.21.15.6:0/2222333488 conn(0x55cbe06ded30 msgr2=0x55cbe06e41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.871+0000 7f8d1b20e740 1 -- 172.21.15.6:0/2222333488 shutdown_connections 2024-04-26T01:47:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.871+0000 7f8d1b20e740 1 -- 172.21.15.6:0/2222333488 wait complete. 2024-04-26T01:47:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.871+0000 7f8d1b20e740 1 -- start start 2024-04-26T01:47:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.871+0000 7f8d1b20e740 1 Processor -- start 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.872+0000 7f8d1b20e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 0x55cbe06f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.872+0000 7f8d1b20e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 0x55cbe06f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.872+0000 7f8d1b20e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06eda00 0x55cbe07623b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.872+0000 7f8d1b20e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101aa150 con 0x55cbe06e3140 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.872+0000 7f8d1b20e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101ab5c0 con 0x55cbe06e2820 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.872+0000 7f8d1b20e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101aa820 con 0x55cbe06eda00 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.872+0000 7f8d1b20e740 10 client.0 _ll_register_callbacks cb 0x7f8cfc009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.872+0000 7f8d16400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 0x55cbe06f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.872+0000 7f8d16400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 0x55cbe06f29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53910/0 (socket says 172.21.15.6:53910) 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.872+0000 7f8d16400640 1 -- 172.21.15.6:0/1184484979 learned_addr learned my addr 172.21.15.6:0/1184484979 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.872+0000 7f8d16400640 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06eda00 0x55cbe07623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.873+0000 7f8d15000640 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 0x55cbe06f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.873+0000 7f8d16400640 1 -- 172.21.15.6:0/1184484979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06eda00 msgr2=0x55cbe07623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.873+0000 7f8d16400640 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06eda00 0x55cbe07623b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.873+0000 7f8d16400640 1 -- 172.21.15.6:0/1184484979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 msgr2=0x55cbe06f2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.873+0000 7f8d16400640 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 0x55cbe06f2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.873+0000 7f8d16400640 1 -- 172.21.15.6:0/1184484979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbe06f37e0 con 0x55cbe06e2820 2024-04-26T01:47:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.873+0000 7f8d15000640 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 0x55cbe06f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.873+0000 7f8d16400640 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 0x55cbe06f29d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009fe0 tx=0x7f8d08030040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edce1a5db5153d97 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.873+0000 7f8cf7400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:47:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.874+0000 7f8cf7400640 1 -- 172.21.15.6:0/1184484979 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d080353d0 con 0x55cbe06e2820 2024-04-26T01:47:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.874+0000 7f8cf7400640 1 -- 172.21.15.6:0/1184484979 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8556+0+0 (secure 0 0 0) 0x7f8d08035710 con 0x55cbe06e2820 2024-04-26T01:47:03.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.874+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1184484979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d0800e990 con 0x55cbe06e2820 2024-04-26T01:47:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.874+0000 7f8cf7400640 1 -- 172.21.15.6:0/1184484979 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8d08035910 con 0x55cbe06e2820 2024-04-26T01:47:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.874+0000 7f8d1b20e740 10 client.25331 fetch_fsmap learned FSMap version 26 2024-04-26T01:47:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.874+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1184484979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cbe07639d0 con 0x55cbe06e2820 2024-04-26T01:47:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.874+0000 7f8cf7400640 1 -- 172.21.15.6:0/1184484979 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d0800e990 con 0x55cbe06e2820 2024-04-26T01:47:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.875+0000 7f8cf7400640 1 -- 172.21.15.6:0/1184484979 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8d08041040 con 0x55cbe06e2820 2024-04-26T01:47:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.875+0000 7f8d1b20e740 10 client.25331 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T01:47:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.875+0000 7f8d1b20e740 10 client.25331 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.875+0000 7f8d1b20e740 20 client.25331 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.875+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cbe0763e50 0x55cbe0766240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.875+0000 7f8d1b20e740 4 client.25331 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:47:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.875+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1184484979 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cbe0650190 con 0x55cbe0763e50 2024-04-26T01:47:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.875+0000 7f8d15000640 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cbe0763e50 0x55cbe0766240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.877+0000 7f8d15000640 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cbe0763e50 0x55cbe0766240 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=589b09bbf2cd65d4 server_cookie=dab9449207a7f90c in_seq=0 out_seq=0 2024-04-26T01:47:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:03.877+0000 7f8cf7400640 10 client.25331 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.435+0000 7f8cf7400640 1 -- 172.21.15.6:0/1184484979 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cbe0650190 con 0x55cbe0763e50 2024-04-26T01:47:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.435+0000 7f8cf7400640 10 client.25331 handle_command_reply: tid=0 2024-04-26T01:47:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.435+0000 7f8d1b20e740 1 client.25331 shutdown 2024-04-26T01:47:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.436+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1184484979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 msgr2=0x55cbe06f29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.436+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 0x55cbe06f29d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009fe0 tx=0x7f8d08030040 comp rx=0 tx=0).stop 2024-04-26T01:47:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.436+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1184484979 shutdown_connections 2024-04-26T01:47:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.436+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cbe0763e50 0x55cbe0766240 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.436+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cbe06e3140 0x55cbe06f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.436+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cbe06eda00 0x55cbe07623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.436+0000 7f8d1b20e740 1 --2- 172.21.15.6:0/1184484979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cbe06e2820 0x55cbe06f29d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.436+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1184484979 >> 172.21.15.6:0/1184484979 conn(0x55cbe06ded30 msgr2=0x55cbe06e4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.436+0000 7f8cf7400640 0 client.25331 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.436+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1184484979 shutdown_connections 2024-04-26T01:47:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.436+0000 7f8d1b20e740 1 -- 172.21.15.6:0/1184484979 wait complete. 2024-04-26T01:47:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.437+0000 7f8d1b20e740 20 client.25331 trim_cache size 0 max 16384 2024-04-26T01:47:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:47:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:47:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:47:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:47:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.438+0000 7f8d18400640 1 -- 172.21.15.6:0/851462217 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8cf80a3eb0 msgr2=0x7f8cf80a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.438+0000 7f8d18400640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8cf80a3eb0 0x7f8cf80a6370 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d040c4b80 tx=0x7f8d040f6530 comp rx=0 tx=0).stop 2024-04-26T01:47:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.438+0000 7f8d18400640 1 -- 172.21.15.6:0/851462217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d101043c0 msgr2=0x7f8d101a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.438+0000 7f8d18400640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d101043c0 0x7f8d101a0670 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8d04005940 tx=0x7f8d040040c0 comp rx=0 tx=0).stop 2024-04-26T01:47:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.441+0000 7f8d18400640 1 -- 172.21.15.6:0/851462217 shutdown_connections 2024-04-26T01:47:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.441+0000 7f8d18400640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8cf80a3eb0 0x7f8cf80a6370 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.441+0000 7f8d18400640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d10109490 0x7f8d101a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.441+0000 7f8d18400640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1019a7e0 0x7f8d1019ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.441+0000 7f8d18400640 1 --2- 172.21.15.6:0/851462217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d101043c0 0x7f8d101a0670 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.441+0000 7f8d18400640 1 -- 172.21.15.6:0/851462217 >> 172.21.15.6:0/851462217 conn(0x7f8d100ffc40 msgr2=0x7f8d10100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.441+0000 7f8d18400640 1 -- 172.21.15.6:0/851462217 shutdown_connections 2024-04-26T01:47:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:04.441+0000 7f8d18400640 1 -- 172.21.15.6:0/851462217 wait complete. 2024-04-26T01:47:04.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:04 smithi006 ceph-mon[28419]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-26T01:47:04.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:04 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:47:04.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-mon[32751]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-26T01:47:04.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:47:04.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:04 smithi029 ceph-mon[32775]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-26T01:47:04.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:04 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:47:05.390 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: *** Caught signal (Segmentation fault) ** 2024-04-26T01:47:05.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: in thread 7f10a33ed700 thread_name:md_log_replay 2024-04-26T01:47:05.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-26T01:47:05.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f10b47aed20] 2024-04-26T01:47:05.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 2: /usr/bin/ceph-mds(+0x54938f) [0x55ecfb7b838f] 2024-04-26T01:47:05.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55ecfb7bbb23] 2024-04-26T01:47:05.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 4: (MDLog::_replay_thread()+0x617) [0x55ecfb816647] 2024-04-26T01:47:05.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55ecfb81fd31] 2024-04-26T01:47:05.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f10b47a41ca] 2024-04-26T01:47:05.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 7: clone() 2024-04-26T01:47:05.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 2024-04-26T01:47:04.970+0000 7f10a33ed700 -1 *** Caught signal (Segmentation fault) ** 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: in thread 7f10a33ed700 thread_name:md_log_replay 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f10b47aed20] 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 2: /usr/bin/ceph-mds(+0x54938f) [0x55ecfb7b838f] 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55ecfb7bbb23] 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 4: (MDLog::_replay_thread()+0x617) [0x55ecfb816647] 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55ecfb81fd31] 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f10b47a41ca] 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 7: clone() 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 0> 2024-04-26T01:47:04.970+0000 7f10a33ed700 -1 *** Caught signal (Segmentation fault) ** 2024-04-26T01:47:05.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: in thread 7f10a33ed700 thread_name:md_log_replay 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f10b47aed20] 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 2: /usr/bin/ceph-mds(+0x54938f) [0x55ecfb7b838f] 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55ecfb7bbb23] 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 4: (MDLog::_replay_thread()+0x617) [0x55ecfb816647] 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55ecfb81fd31] 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f10b47a41ca] 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 7: clone() 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 0> 2024-04-26T01:47:04.970+0000 7f10a33ed700 -1 *** Caught signal (Segmentation fault) ** 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: in thread 7f10a33ed700 thread_name:md_log_replay 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 2024-04-26T01:47:05.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-26T01:47:05.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f10b47aed20] 2024-04-26T01:47:05.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 2: /usr/bin/ceph-mds(+0x54938f) [0x55ecfb7b838f] 2024-04-26T01:47:05.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55ecfb7bbb23] 2024-04-26T01:47:05.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 4: (MDLog::_replay_thread()+0x617) [0x55ecfb816647] 2024-04-26T01:47:05.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55ecfb81fd31] 2024-04-26T01:47:05.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f10b47a41ca] 2024-04-26T01:47:05.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 7: clone() 2024-04-26T01:47:05.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T01:47:05.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[63602]: 2024-04-26T01:47:05.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:05 smithi188 podman[68006]: 2024-04-26 01:47:05.027684776 +0000 UTC m=+0.020660622 container died 146c931191428cac4535f6747a2e2f9d41c14c7eb00ebeab027ab3b83b2517da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-26T01:47:05.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:05 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:47:05.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:05 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:47:05.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:05 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:47:05.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:05 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:47:05.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:05 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:47:05.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:05 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:47:05.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:05 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:47:05.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:05 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:47:05.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:05 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:47:06.287 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:06 smithi188 podman[68006]: 2024-04-26 01:47:06.034404599 +0000 UTC m=+1.027380455 container remove 146c931191428cac4535f6747a2e2f9d41c14c7eb00ebeab027ab3b83b2517da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T01:47:06.287 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:06 smithi188 systemd[1]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=139/n/a 2024-04-26T01:47:06.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:06 smithi188 ceph-mon[32751]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-26T01:47:06.639 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:06 smithi188 systemd[1]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-26T01:47:06.640 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:06 smithi188 systemd[1]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.f.service: Consumed 1.098s CPU time. 2024-04-26T01:47:06.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:06 smithi006 ceph-mon[28419]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-26T01:47:06.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:06 smithi029 ceph-mon[32775]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-26T01:47:08.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:08 smithi006 ceph-mon[28419]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 24 op/s 2024-04-26T01:47:08.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:08 smithi188 ceph-mon[32751]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 24 op/s 2024-04-26T01:47:08.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:08 smithi029 ceph-mon[32775]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 24 op/s 2024-04-26T01:47:10.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:10 smithi006 ceph-mon[28419]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-04-26T01:47:10.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:10 smithi188 ceph-mon[32751]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-04-26T01:47:10.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:10 smithi029 ceph-mon[32775]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-04-26T01:47:12.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:12 smithi006 ceph-mon[28419]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 23 op/s 2024-04-26T01:47:12.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:12 smithi188 ceph-mon[32751]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 23 op/s 2024-04-26T01:47:12.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:12 smithi029 ceph-mon[32775]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 23 op/s 2024-04-26T01:47:14.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:14 smithi006 ceph-mon[28419]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 19 op/s 2024-04-26T01:47:14.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:14 smithi188 ceph-mon[32751]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 19 op/s 2024-04-26T01:47:14.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:14 smithi029 ceph-mon[32775]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 19 op/s 2024-04-26T01:47:16.745 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:16 smithi188 ceph-mon[32751]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T01:47:16.745 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:16 smithi188 systemd[1]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-04-26T01:47:16.745 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:16 smithi188 systemd[1]: Stopped Ceph mds.f for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:47:16.746 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:16 smithi188 systemd[1]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.f.service: Consumed 1.098s CPU time. 2024-04-26T01:47:16.746 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:16 smithi188 systemd[1]: Starting Ceph mds.f for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T01:47:16.746 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:16 smithi188 podman[68134]: 2024-04-26 01:47:16.529277756 +0000 UTC m=+0.019542738 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:47:16.746 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:16 smithi188 podman[68134]: 2024-04-26 01:47:16.642363155 +0000 UTC m=+0.132628129 container create 2aa418024757e4413413f89dcf27cae30090281135eb22f3021c5524156bc5c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:47:16.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:16 smithi006 ceph-mon[28419]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T01:47:16.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:16 smithi029 ceph-mon[32775]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T01:47:17.139 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:16 smithi188 podman[68134]: 2024-04-26 01:47:16.846246091 +0000 UTC m=+0.336511063 container init 2aa418024757e4413413f89dcf27cae30090281135eb22f3021c5524156bc5c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-26T01:47:17.139 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:16 smithi188 podman[68134]: 2024-04-26 01:47:16.848810131 +0000 UTC m=+0.339075101 container start 2aa418024757e4413413f89dcf27cae30090281135eb22f3021c5524156bc5c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:47:17.140 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:16 smithi188 ceph-mds[68153]: -- 172.21.15.188:0/542188291 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e1174d3a40 con 0x55e1174a7000 2024-04-26T01:47:17.140 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: starting mds.f at 2024-04-26T01:47:17.140 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:16 smithi188 bash[68134]: 2aa418024757e4413413f89dcf27cae30090281135eb22f3021c5524156bc5c7 2024-04-26T01:47:17.140 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:16 smithi188 systemd[1]: Started Ceph mds.f for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T01:47:17.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.603+0000 7f74c5400640 1 -- 172.21.15.6:0/3559794874 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c0101fd0 msgr2=0x7f74c01023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:17.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.603+0000 7f74c5400640 1 --2- 172.21.15.6:0/3559794874 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c0101fd0 0x7f74c01023b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009900 tx=0x7f74b002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.603+0000 7f74c5400640 1 -- 172.21.15.6:0/3559794874 shutdown_connections 2024-04-26T01:47:17.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.603+0000 7f74c5400640 1 --2- 172.21.15.6:0/3559794874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c01070a0 0x7f74c010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.604+0000 7f74c5400640 1 --2- 172.21.15.6:0/3559794874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74c0102980 0x7f74c01069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.604+0000 7f74c5400640 1 --2- 172.21.15.6:0/3559794874 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c0101fd0 0x7f74c01023b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.604+0000 7f74c5400640 1 -- 172.21.15.6:0/3559794874 >> 172.21.15.6:0/3559794874 conn(0x7f74c00fd890 msgr2=0x7f74c00ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:17.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.604+0000 7f74c5400640 1 -- 172.21.15.6:0/3559794874 shutdown_connections 2024-04-26T01:47:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.604+0000 7f74c5400640 1 -- 172.21.15.6:0/3559794874 wait complete. 2024-04-26T01:47:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.605+0000 7f74c5400640 1 Processor -- start 2024-04-26T01:47:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.605+0000 7f74c5400640 1 -- start start 2024-04-26T01:47:17.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.606+0000 7f74c5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c0101fd0 0x7f74c01a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.606+0000 7f74c5400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74c01070a0 0x7f74c01a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.606+0000 7f74c5400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c019a5d0 0x7f74c019aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.606+0000 7f74c5400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f74c0112050 con 0x7f74c0101fd0 2024-04-26T01:47:17.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.606+0000 7f74c5400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f74c0111ed0 con 0x7f74c01070a0 2024-04-26T01:47:17.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.606+0000 7f74c5400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f74c01121d0 con 0x7f74c019a5d0 2024-04-26T01:47:17.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.606+0000 7f74c4a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c019a5d0 0x7f74c019aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.606+0000 7f74c4a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c019a5d0 0x7f74c019aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56810/0 (socket says 172.21.15.6:56810) 2024-04-26T01:47:17.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.606+0000 7f74c4a00640 1 -- 172.21.15.6:0/3542776463 learned_addr learned my addr 172.21.15.6:0/3542776463 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:17.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.606+0000 7f74bf400640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74c01070a0 0x7f74c01a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.606+0000 7f74bfe00640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c0101fd0 0x7f74c01a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.607+0000 7f74bf400640 1 -- 172.21.15.6:0/3542776463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c019a5d0 msgr2=0x7f74c019aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:17.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.607+0000 7f74bf400640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c019a5d0 0x7f74c019aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.607+0000 7f74bf400640 1 -- 172.21.15.6:0/3542776463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c0101fd0 msgr2=0x7f74c01a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:17.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.607+0000 7f74bf400640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c0101fd0 0x7f74c01a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.607+0000 7f74bf400640 1 -- 172.21.15.6:0/3542776463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c019b0f0 con 0x7f74c01070a0 2024-04-26T01:47:17.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.607+0000 7f74bf400640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74c01070a0 0x7f74c01a0960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00e9c0 tx=0x7f74ac00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:17.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.607+0000 7f74bcc00640 1 -- 172.21.15.6:0/3542776463 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac00ccc0 con 0x7f74c01070a0 2024-04-26T01:47:17.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.607+0000 7f74c5400640 1 -- 172.21.15.6:0/3542776463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c019b3e0 con 0x7f74c01070a0 2024-04-26T01:47:17.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.608+0000 7f74bcc00640 1 -- 172.21.15.6:0/3542776463 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74ac005020 con 0x7f74c01070a0 2024-04-26T01:47:17.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.608+0000 7f74bcc00640 1 -- 172.21.15.6:0/3542776463 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac004a00 con 0x7f74c01070a0 2024-04-26T01:47:17.607 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.608+0000 7f74c5400640 1 -- 172.21.15.6:0/3542776463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74c01a7630 con 0x7f74c01070a0 2024-04-26T01:47:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.611+0000 7f74c9813740 1 Processor -- start 2024-04-26T01:47:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.611+0000 7f74c9813740 1 -- start start 2024-04-26T01:47:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.612+0000 7f74c9813740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 0x55f65990bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.612+0000 7f74c9813740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f659916650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.612+0000 7f74c9813740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f659916b90 0x55f659918f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.612+0000 7f74c9813740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55f659871900 con 0x55f65990b9b0 2024-04-26T01:47:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.612+0000 7f74c9813740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f659817ef0 con 0x55f659916b90 2024-04-26T01:47:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.612+0000 7f74c9813740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f65986d8c0 con 0x55f65990c2d0 2024-04-26T01:47:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.612+0000 7f74c4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 0x55f65990bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.612+0000 7f74c4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 0x55f65990bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43612/0 (socket says 172.21.15.6:43612) 2024-04-26T01:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.612+0000 7f74c4a00640 1 -- 172.21.15.6:0/3126716793 learned_addr learned my addr 172.21.15.6:0/3126716793 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.613+0000 7f74bfe00640 1 --2- 172.21.15.6:0/3126716793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f659916650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.613+0000 7f74c4a00640 1 -- 172.21.15.6:0/3126716793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 msgr2=0x55f659916650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.613+0000 7f74c4a00640 1 --2- 172.21.15.6:0/3126716793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f659916650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.613+0000 7f74c4a00640 1 -- 172.21.15.6:0/3126716793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f659916b90 msgr2=0x55f659918f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:47:17.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.613+0000 7f74c4a00640 1 --2- 172.21.15.6:0/3126716793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f659916b90 0x55f659918f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.613+0000 7f74c4a00640 1 -- 172.21.15.6:0/3126716793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f659584210 con 0x55f65990b9b0 2024-04-26T01:47:17.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.613+0000 7f74bcc00640 1 -- 172.21.15.6:0/3542776463 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) ==== 138878+0+0 (secure 0 0 0) 0x7f74ac0051d0 con 0x7f74c01070a0 2024-04-26T01:47:17.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.614+0000 7f74c4a00640 1 --2- 172.21.15.6:0/3126716793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 0x55f65990bd90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f74b8010450 tx=0x7f74b8010910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3220775f1a78d52d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:17.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.614+0000 7f74b5200640 1 -- 172.21.15.6:0/3126716793 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b8011760 con 0x55f65990b9b0 2024-04-26T01:47:17.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.614+0000 7f74b5200640 1 -- 172.21.15.6:0/3126716793 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74b8011db0 con 0x55f65990b9b0 2024-04-26T01:47:17.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.614+0000 7f74b5200640 1 -- 172.21.15.6:0/3126716793 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b8018380 con 0x55f65990b9b0 2024-04-26T01:47:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.615+0000 7f74bcc00640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f74a40a3f20 0x7f74a40a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.615+0000 7f74c9813740 1 -- 172.21.15.6:0/3126716793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 msgr2=0x55f65990bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.615+0000 7f74c9813740 1 --2- 172.21.15.6:0/3126716793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 0x55f65990bd90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f74b8010450 tx=0x7f74b8010910 comp rx=0 tx=0).stop 2024-04-26T01:47:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.615+0000 7f74bcc00640 1 -- 172.21.15.6:0/3542776463 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f74ac0d0a30 con 0x7f74c01070a0 2024-04-26T01:47:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.615+0000 7f74c9813740 1 -- 172.21.15.6:0/3126716793 shutdown_connections 2024-04-26T01:47:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.615+0000 7f74c9813740 1 --2- 172.21.15.6:0/3126716793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f659916b90 0x55f659918f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.615+0000 7f74c9813740 1 --2- 172.21.15.6:0/3126716793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f659916650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.615+0000 7f74c9813740 1 --2- 172.21.15.6:0/3126716793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 0x55f65990bd90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.615+0000 7f74c9813740 1 -- 172.21.15.6:0/3126716793 >> 172.21.15.6:0/3126716793 conn(0x55f659907ec0 msgr2=0x55f6599082a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.616+0000 7f74bfe00640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f74a40a3f20 0x7f74a40a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.616+0000 7f74c9813740 1 -- 172.21.15.6:0/3126716793 shutdown_connections 2024-04-26T01:47:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.616+0000 7f74c9813740 1 -- 172.21.15.6:0/3126716793 wait complete. 2024-04-26T01:47:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.616+0000 7f74bfe00640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f74a40a3f20 0x7f74a40a63e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f74b0005ab0 tx=0x7f74b0005a20 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.616+0000 7f74c9813740 1 -- start start 2024-04-26T01:47:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.617+0000 7f74c9813740 1 Processor -- start 2024-04-26T01:47:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.617+0000 7f74c9813740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990b9b0 0x55f659913bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.617+0000 7f74c9813740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f65990c2d0 0x55f659914100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.617+0000 7f74c9813740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f659916b90 0x55f659914640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.617+0000 7f74c9813740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55f6599197c0 con 0x55f659916b90 2024-04-26T01:47:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.617+0000 7f74c9813740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f65982b310 con 0x55f65990c2d0 2024-04-26T01:47:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.617+0000 7f74c9813740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f659919940 con 0x55f65990b9b0 2024-04-26T01:47:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.617+0000 7f74c9813740 10 client.0 _ll_register_callbacks cb 0x55f6597f9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.618+0000 7f74bfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990b9b0 0x55f659913bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.618+0000 7f74bfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990b9b0 0x55f659913bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56832/0 (socket says 172.21.15.6:56832) 2024-04-26T01:47:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.618+0000 7f74bfe00640 1 -- 172.21.15.6:0/593082100 learned_addr learned my addr 172.21.15.6:0/593082100 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.618+0000 7f74bfe00640 1 -- 172.21.15.6:0/593082100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f65990c2d0 msgr2=0x55f659914100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:47:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.618+0000 7f74bf400640 1 --2- 172.21.15.6:0/593082100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f65990c2d0 0x55f659914100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.618+0000 7f74bfe00640 1 --2- 172.21.15.6:0/593082100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f65990c2d0 0x55f659914100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.618+0000 7f74bfe00640 1 -- 172.21.15.6:0/593082100 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f659916b90 msgr2=0x55f659914640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.618+0000 7f74bfe00640 1 --2- 172.21.15.6:0/593082100 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f659916b90 0x55f659914640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.618+0000 7f74bfe00640 1 -- 172.21.15.6:0/593082100 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f659650620 con 0x55f65990b9b0 2024-04-26T01:47:17.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.619+0000 7f74bfe00640 1 --2- 172.21.15.6:0/593082100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990b9b0 0x55f659913bc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009900 tx=0x7f74b0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eff90c50cf4ecab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:17.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.619+0000 7f74a9600640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:47:17.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.619+0000 7f74a9600640 1 -- 172.21.15.6:0/593082100 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b004e070 con 0x55f65990b9b0 2024-04-26T01:47:17.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.619+0000 7f74a9600640 1 -- 172.21.15.6:0/593082100 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f74b00037f0 con 0x55f65990b9b0 2024-04-26T01:47:17.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.619+0000 7f74c9813740 1 -- 172.21.15.6:0/593082100 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74b8011db0 con 0x55f65990b9b0 2024-04-26T01:47:17.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.619+0000 7f74a9600640 1 -- 172.21.15.6:0/593082100 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f74b0058050 con 0x55f65990b9b0 2024-04-26T01:47:17.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.619+0000 7f74c9813740 10 client.25384 fetch_fsmap learned FSMap version 28 2024-04-26T01:47:17.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.619+0000 7f74c9813740 1 -- 172.21.15.6:0/593082100 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f65992e2c0 con 0x55f65990b9b0 2024-04-26T01:47:17.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.619+0000 7f74a9600640 1 -- 172.21.15.6:0/593082100 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74b8011db0 con 0x55f65990b9b0 2024-04-26T01:47:17.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.620+0000 7f74a9600640 1 -- 172.21.15.6:0/593082100 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f74b0032750 con 0x55f65990b9b0 2024-04-26T01:47:17.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.620+0000 7f74c9813740 10 client.25384 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.620+0000 7f74c9813740 10 client.25384 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.620+0000 7f74c9813740 20 client.25384 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.620+0000 7f74c9813740 1 --2- 172.21.15.6:0/593082100 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f65992f7d0 0x55f659931bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.620+0000 7f74c9813740 4 client.25384 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.620+0000 7f74c9813740 1 -- 172.21.15.6:0/593082100 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6598858b0 con 0x55f65992f7d0 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.620+0000 7f74bf400640 1 --2- 172.21.15.6:0/593082100 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f65992f7d0 0x55f659931bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.620+0000 7f74bf400640 1 --2- 172.21.15.6:0/593082100 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f65992f7d0 0x55f659931bc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4084623ee48baa57 server_cookie=2e5667ac1c07ae8c in_seq=0 out_seq=0 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.620+0000 7f74a9600640 10 client.25384 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.621+0000 7f74a9600640 1 -- 172.21.15.6:0/593082100 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f6598858b0 con 0x55f65992f7d0 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.621+0000 7f74a9600640 10 client.25384 handle_command_reply: tid=0 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.621+0000 7f74c9813740 1 client.25384 shutdown 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.621+0000 7f74c9813740 1 -- 172.21.15.6:0/593082100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990b9b0 msgr2=0x55f659913bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.621+0000 7f74c9813740 1 --2- 172.21.15.6:0/593082100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990b9b0 0x55f659913bc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009900 tx=0x7f74b0009260 comp rx=0 tx=0).stop 2024-04-26T01:47:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.622+0000 7f74c9813740 1 -- 172.21.15.6:0/593082100 shutdown_connections 2024-04-26T01:47:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.622+0000 7f74c9813740 1 --2- 172.21.15.6:0/593082100 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f65992f7d0 0x55f659931bc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.622+0000 7f74c9813740 1 --2- 172.21.15.6:0/593082100 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f659916b90 0x55f659914640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.622+0000 7f74c9813740 1 --2- 172.21.15.6:0/593082100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f65990c2d0 0x55f659914100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.622+0000 7f74c9813740 1 --2- 172.21.15.6:0/593082100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990b9b0 0x55f659913bc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.622+0000 7f74c9813740 1 -- 172.21.15.6:0/593082100 >> 172.21.15.6:0/593082100 conn(0x55f659907ec0 msgr2=0x55f65990dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.622+0000 7f74a9600640 0 client.25384 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.622+0000 7f74c9813740 1 -- 172.21.15.6:0/593082100 shutdown_connections 2024-04-26T01:47:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.622+0000 7f74c9813740 1 -- 172.21.15.6:0/593082100 wait complete. 2024-04-26T01:47:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.622+0000 7f74c9813740 20 client.25384 trim_cache size 0 max 16384 2024-04-26T01:47:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.638+0000 7f74c9813740 1 Processor -- start 2024-04-26T01:47:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.638+0000 7f74c9813740 1 -- start start 2024-04-26T01:47:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.638+0000 7f74c9813740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f65990b9b0 0x55f659912b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.638+0000 7f74c9813740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f6599077a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.638+0000 7f74c9813740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f659916b90 0x55f65990e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.638+0000 7f74c9813740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f74c01ae1a0 con 0x55f659916b90 2024-04-26T01:47:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.638+0000 7f74c9813740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f74c01aba70 con 0x55f65990b9b0 2024-04-26T01:47:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.638+0000 7f74c9813740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f74c01a9920 con 0x55f65990c2d0 2024-04-26T01:47:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.638+0000 7f74bfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f65990b9b0 0x55f659912b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.638+0000 7f74bfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f65990b9b0 0x55f659912b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:44800/0 (socket says 172.21.15.6:44800) 2024-04-26T01:47:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.638+0000 7f74bfe00640 1 -- 172.21.15.6:0/106587785 learned_addr learned my addr 172.21.15.6:0/106587785 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.638+0000 7f74bf400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f6599077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.639+0000 7f74c4a00640 1 --2- 172.21.15.6:0/106587785 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f659916b90 0x55f65990e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.639+0000 7f74bf400640 1 -- 172.21.15.6:0/106587785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f65990b9b0 msgr2=0x55f659912b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.639+0000 7f74bf400640 1 --2- 172.21.15.6:0/106587785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f65990b9b0 0x55f659912b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.639+0000 7f74bf400640 1 -- 172.21.15.6:0/106587785 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f659916b90 msgr2=0x55f65990e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.639+0000 7f74bf400640 1 --2- 172.21.15.6:0/106587785 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f659916b90 0x55f65990e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.639+0000 7f74bf400640 1 -- 172.21.15.6:0/106587785 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f659914b20 con 0x55f65990c2d0 2024-04-26T01:47:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.639+0000 7f74bfe00640 1 --2- 172.21.15.6:0/106587785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f65990b9b0 0x55f659912b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.639+0000 7f74bf400640 1 --2- 172.21.15.6:0/106587785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f6599077a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0cdf80 tx=0x7f74ac098a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=654ca4608046958c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.639+0000 7f74abe00640 1 -- 172.21.15.6:0/106587785 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac01c040 con 0x55f65990c2d0 2024-04-26T01:47:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.639+0000 7f74abe00640 1 -- 172.21.15.6:0/106587785 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74ac0870d0 con 0x55f65990c2d0 2024-04-26T01:47:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.639+0000 7f74abe00640 1 -- 172.21.15.6:0/106587785 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac0a5140 con 0x55f65990c2d0 2024-04-26T01:47:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.640+0000 7f74c9813740 1 -- 172.21.15.6:0/106587785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 msgr2=0x55f6599077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.640+0000 7f74c9813740 1 --2- 172.21.15.6:0/106587785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f6599077a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0cdf80 tx=0x7f74ac098a30 comp rx=0 tx=0).stop 2024-04-26T01:47:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.640+0000 7f74c9813740 1 -- 172.21.15.6:0/106587785 shutdown_connections 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.640+0000 7f74c9813740 1 --2- 172.21.15.6:0/106587785 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f659916b90 0x55f65990e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.640+0000 7f74c9813740 1 --2- 172.21.15.6:0/106587785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f6599077a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.640+0000 7f74c9813740 1 --2- 172.21.15.6:0/106587785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f65990b9b0 0x55f659912b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.640+0000 7f74c9813740 1 -- 172.21.15.6:0/106587785 >> 172.21.15.6:0/106587785 conn(0x55f659907ec0 msgr2=0x55f659914410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.640+0000 7f74c9813740 1 -- 172.21.15.6:0/106587785 shutdown_connections 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.640+0000 7f74c9813740 1 -- 172.21.15.6:0/106587785 wait complete. 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.641+0000 7f74c9813740 1 -- start start 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.641+0000 7f74c9813740 1 Processor -- start 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74c9813740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 0x55f6599277f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74c9813740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f659924b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74c9813740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f659916b90 0x55f6599250c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74c9813740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f74c01a9f70 con 0x55f65990b9b0 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74c9813740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f74c01ab3a0 con 0x55f659916b90 2024-04-26T01:47:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74c9813740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f74c01aa600 con 0x55f65990c2d0 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74c9813740 10 client.0 _ll_register_callbacks cb 0x7f74b0005040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 0x55f6599277f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74bf400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f659924b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74bf400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f659924b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56858/0 (socket says 172.21.15.6:56858) 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 0x55f6599277f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43640/0 (socket says 172.21.15.6:43640) 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74bf400640 1 -- 172.21.15.6:0/2299797599 learned_addr learned my addr 172.21.15.6:0/2299797599 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74bf400640 1 -- 172.21.15.6:0/2299797599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f659916b90 msgr2=0x55f6599250c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74bf400640 1 --2- 172.21.15.6:0/2299797599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f659916b90 0x55f6599250c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74bf400640 1 -- 172.21.15.6:0/2299797599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 msgr2=0x55f6599277f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74bf400640 1 --2- 172.21.15.6:0/2299797599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 0x55f6599277f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.642+0000 7f74bf400640 1 -- 172.21.15.6:0/2299797599 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6599258f0 con 0x55f65990c2d0 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.643+0000 7f74a9600640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.643+0000 7f74bf400640 1 --2- 172.21.15.6:0/2299797599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f659924b80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f74ac09c690 tx=0x7f74ac0a5370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72656b32f72f1ba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.643+0000 7f74a9600640 1 -- 172.21.15.6:0/2299797599 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac082060 con 0x55f65990c2d0 2024-04-26T01:47:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.643+0000 7f74a9600640 1 -- 172.21.15.6:0/2299797599 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f74ac098d70 con 0x55f65990c2d0 2024-04-26T01:47:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.643+0000 7f74c9813740 1 -- 172.21.15.6:0/2299797599 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74ac0870d0 con 0x55f65990c2d0 2024-04-26T01:47:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.643+0000 7f74a9600640 1 -- 172.21.15.6:0/2299797599 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f74ac0bb050 con 0x55f65990c2d0 2024-04-26T01:47:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.643+0000 7f74c9813740 10 client.25390 fetch_fsmap learned FSMap version 28 2024-04-26T01:47:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.643+0000 7f74c9813740 1 -- 172.21.15.6:0/2299797599 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f659995dc0 con 0x55f65990c2d0 2024-04-26T01:47:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.644+0000 7f74a9600640 1 -- 172.21.15.6:0/2299797599 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74ac0870d0 con 0x55f65990c2d0 2024-04-26T01:47:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.644+0000 7f74a9600640 1 -- 172.21.15.6:0/2299797599 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f74ac0b5030 con 0x55f65990c2d0 2024-04-26T01:47:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.644+0000 7f74c9813740 10 client.25390 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:47:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.644+0000 7f74c9813740 10 client.25390 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.644+0000 7f74c9813740 20 client.25390 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.644+0000 7f74c9813740 1 --2- 172.21.15.6:0/2299797599 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f659996450 0x55f659998870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.644+0000 7f74c9813740 4 client.25390 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:47:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.644+0000 7f74c9813740 1 -- 172.21.15.6:0/2299797599 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f65987f320 con 0x55f659996450 2024-04-26T01:47:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.644+0000 7f74bfe00640 1 --2- 172.21.15.6:0/2299797599 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f659996450 0x55f659998870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.645+0000 7f74bfe00640 1 --2- 172.21.15.6:0/2299797599 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f659996450 0x55f659998870 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=552962f5cb9bdb10 server_cookie=dcf87fe38949d569 in_seq=0 out_seq=0 2024-04-26T01:47:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.645+0000 7f74a9600640 10 client.25390 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.651+0000 7f74a9600640 1 -- 172.21.15.6:0/2299797599 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f65987f320 con 0x55f659996450 2024-04-26T01:47:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.651+0000 7f74a9600640 10 client.25390 handle_command_reply: tid=0 2024-04-26T01:47:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.651+0000 7f74c9813740 1 client.25390 shutdown 2024-04-26T01:47:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.652+0000 7f74c9813740 1 -- 172.21.15.6:0/2299797599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 msgr2=0x55f659924b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.652+0000 7f74c9813740 1 --2- 172.21.15.6:0/2299797599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f659924b80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f74ac09c690 tx=0x7f74ac0a5370 comp rx=0 tx=0).stop 2024-04-26T01:47:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.652+0000 7f74c9813740 1 -- 172.21.15.6:0/2299797599 shutdown_connections 2024-04-26T01:47:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.652+0000 7f74c9813740 1 --2- 172.21.15.6:0/2299797599 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f659996450 0x55f659998870 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.652+0000 7f74c9813740 1 --2- 172.21.15.6:0/2299797599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f659916b90 0x55f6599250c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.652+0000 7f74c9813740 1 --2- 172.21.15.6:0/2299797599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f65990c2d0 0x55f659924b80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.652+0000 7f74c9813740 1 --2- 172.21.15.6:0/2299797599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f65990b9b0 0x55f6599277f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.652+0000 7f74c9813740 1 -- 172.21.15.6:0/2299797599 >> 172.21.15.6:0/2299797599 conn(0x55f659907ec0 msgr2=0x55f6599153f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.652+0000 7f74a9600640 0 client.25390 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.652+0000 7f74c9813740 1 -- 172.21.15.6:0/2299797599 shutdown_connections 2024-04-26T01:47:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.652+0000 7f74c9813740 1 -- 172.21.15.6:0/2299797599 wait complete. 2024-04-26T01:47:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.652+0000 7f74c9813740 20 client.25390 trim_cache size 0 max 16384 2024-04-26T01:47:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:47:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:47:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:47:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:47:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.653+0000 7f74c5400640 1 -- 172.21.15.6:0/3542776463 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f74a40a3f20 msgr2=0x7f74a40a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.653+0000 7f74c5400640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f74a40a3f20 0x7f74a40a63e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f74b0005ab0 tx=0x7f74b0005a20 comp rx=0 tx=0).stop 2024-04-26T01:47:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.653+0000 7f74c5400640 1 -- 172.21.15.6:0/3542776463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74c01070a0 msgr2=0x7f74c01a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.653+0000 7f74c5400640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74c01070a0 0x7f74c01a0960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00e9c0 tx=0x7f74ac00ee80 comp rx=0 tx=0).stop 2024-04-26T01:47:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.653+0000 7f74c5400640 1 -- 172.21.15.6:0/3542776463 shutdown_connections 2024-04-26T01:47:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.653+0000 7f74c5400640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f74a40a3f20 0x7f74a40a63e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.653+0000 7f74c5400640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74c019a5d0 0x7f74c019aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.653+0000 7f74c5400640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74c01070a0 0x7f74c01a0960 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.653+0000 7f74c5400640 1 --2- 172.21.15.6:0/3542776463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c0101fd0 0x7f74c01a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.653+0000 7f74c5400640 1 -- 172.21.15.6:0/3542776463 >> 172.21.15.6:0/3542776463 conn(0x7f74c00fd890 msgr2=0x7f74c00ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.653+0000 7f74c5400640 1 -- 172.21.15.6:0/3542776463 shutdown_connections 2024-04-26T01:47:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:17.653+0000 7f74c5400640 1 -- 172.21.15.6:0/3542776463 wait complete. 2024-04-26T01:47:17.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:17 smithi006 ceph-mon[28419]: daemon mds.f restarted 2024-04-26T01:47:17.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:17 smithi188 ceph-mon[32751]: daemon mds.f restarted 2024-04-26T01:47:17.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:17 smithi029 ceph-mon[32775]: daemon mds.f restarted 2024-04-26T01:47:18.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:18 smithi006 ceph-mon[28419]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-04-26T01:47:18.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:18 smithi006 ceph-mon[28419]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T01:47:18.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:18 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6834/47657351,v1:172.21.15.188:6835/47657351] up:boot 2024-04-26T01:47:18.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:18 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:47:18.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:18 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:47:18.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:18 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:18.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:18 smithi188 ceph-mon[32751]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-04-26T01:47:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:18 smithi188 ceph-mon[32751]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T01:47:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:18 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6834/47657351,v1:172.21.15.188:6835/47657351] up:boot 2024-04-26T01:47:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:18 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:47:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:18 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:47:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:18 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:18.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:18 smithi029 ceph-mon[32775]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-04-26T01:47:18.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:18 smithi029 ceph-mon[32775]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T01:47:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:18 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6834/47657351,v1:172.21.15.188:6835/47657351] up:boot 2024-04-26T01:47:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:18 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:47:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:18 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:47:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:18 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:20.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:20 smithi006 ceph-mon[28419]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 102 B/s wr, 4 op/s 2024-04-26T01:47:20.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:20 smithi188 ceph-mon[32751]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 102 B/s wr, 4 op/s 2024-04-26T01:47:20.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:20 smithi029 ceph-mon[32775]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 102 B/s wr, 4 op/s 2024-04-26T01:47:22.530 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:22 smithi006 ceph-mon[28419]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-26T01:47:22.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:22 smithi188 ceph-mon[32751]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-26T01:47:22.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:22 smithi029 ceph-mon[32775]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-26T01:47:24.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:24 smithi006 ceph-mon[28419]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-26T01:47:24.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:24 smithi188 ceph-mon[32751]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-26T01:47:24.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:24 smithi029 ceph-mon[32775]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-26T01:47:26.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:26 smithi006 ceph-mon[28419]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-26T01:47:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:26 smithi188 ceph-mon[32751]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-26T01:47:26.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:26 smithi029 ceph-mon[32775]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-26T01:47:27.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:27 smithi188 ceph-mon[32751]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-26T01:47:27.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:27 smithi188 ceph-mon[32751]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-04-26T01:47:27.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:27 smithi029 ceph-mon[32775]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-26T01:47:27.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:27 smithi029 ceph-mon[32775]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-04-26T01:47:28.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:27 smithi006 ceph-mon[28419]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-26T01:47:28.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:27 smithi006 ceph-mon[28419]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-04-26T01:47:29.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:47:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:47:30.300 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:30 smithi006 ceph-mon[28419]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 982 B/s wr, 7 op/s 2024-04-26T01:47:30.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:30 smithi188 ceph-mon[32751]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 982 B/s wr, 7 op/s 2024-04-26T01:47:30.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:30 smithi029 ceph-mon[32775]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 982 B/s wr, 7 op/s 2024-04-26T01:47:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.799+0000 7fbf8d800640 1 -- 172.21.15.6:0/960166875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf88105fa0 msgr2=0x7fbf8810db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.799+0000 7fbf8d800640 1 --2- 172.21.15.6:0/960166875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf88105fa0 0x7fbf8810db70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009ca0 tx=0x7fbf8002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.799+0000 7fbf8d800640 1 -- 172.21.15.6:0/960166875 shutdown_connections 2024-04-26T01:47:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.799+0000 7fbf8d800640 1 --2- 172.21.15.6:0/960166875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf88101880 0x7fbf881058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.799+0000 7fbf8d800640 1 --2- 172.21.15.6:0/960166875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf88105fa0 0x7fbf8810db70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.799+0000 7fbf8d800640 1 --2- 172.21.15.6:0/960166875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88100ed0 0x7fbf881012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.799+0000 7fbf8d800640 1 -- 172.21.15.6:0/960166875 >> 172.21.15.6:0/960166875 conn(0x7fbf880fc730 msgr2=0x7fbf880feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.800+0000 7fbf8d800640 1 -- 172.21.15.6:0/960166875 shutdown_connections 2024-04-26T01:47:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.800+0000 7fbf8d800640 1 -- 172.21.15.6:0/960166875 wait complete. 2024-04-26T01:47:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.801+0000 7fbf8d800640 1 Processor -- start 2024-04-26T01:47:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.801+0000 7fbf8d800640 1 -- start start 2024-04-26T01:47:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.801+0000 7fbf8d800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88100ed0 0x7fbf88102960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.801+0000 7fbf8d800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf88105fa0 0x7fbf88102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.801+0000 7fbf8d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf881033e0 0x7fbf881a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.801+0000 7fbf8d800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbf88110f00 con 0x7fbf881033e0 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.801+0000 7fbf8d800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbf88110d80 con 0x7fbf88100ed0 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.801+0000 7fbf8d800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbf88111080 con 0x7fbf88105fa0 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.802+0000 7fbf87e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88100ed0 0x7fbf88102960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.802+0000 7fbf8ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf881033e0 0x7fbf881a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.802+0000 7fbf87400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf88105fa0 0x7fbf88102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.802+0000 7fbf87e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88100ed0 0x7fbf88102960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:50972/0 (socket says 172.21.15.6:50972) 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.802+0000 7fbf87e00640 1 -- 172.21.15.6:0/2688267681 learned_addr learned my addr 172.21.15.6:0/2688267681 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.802+0000 7fbf87e00640 1 -- 172.21.15.6:0/2688267681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf88105fa0 msgr2=0x7fbf88102ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.802+0000 7fbf87e00640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf88105fa0 0x7fbf88102ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.802+0000 7fbf87e00640 1 -- 172.21.15.6:0/2688267681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf881033e0 msgr2=0x7fbf881a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.802+0000 7fbf87e00640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf881033e0 0x7fbf881a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.802+0000 7fbf87e00640 1 -- 172.21.15.6:0/2688267681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf881a6e40 con 0x7fbf88100ed0 2024-04-26T01:47:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.802+0000 7fbf87400640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf88105fa0 0x7fbf88102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.802+0000 7fbf87e00640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88100ed0 0x7fbf88102960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800e9f0 tx=0x7fbf7800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.803+0000 7fbf84c00640 1 -- 172.21.15.6:0/2688267681 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7800cd80 con 0x7fbf88100ed0 2024-04-26T01:47:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.803+0000 7fbf8d800640 1 -- 172.21.15.6:0/2688267681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf881a7130 con 0x7fbf88100ed0 2024-04-26T01:47:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.803+0000 7fbf84c00640 1 -- 172.21.15.6:0/2688267681 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf780054d0 con 0x7fbf88100ed0 2024-04-26T01:47:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.803+0000 7fbf84c00640 1 -- 172.21.15.6:0/2688267681 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf78010430 con 0x7fbf88100ed0 2024-04-26T01:47:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.803+0000 7fbf8d800640 1 -- 172.21.15.6:0/2688267681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf881a7680 con 0x7fbf88100ed0 2024-04-26T01:47:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.806+0000 7fbf84c00640 1 -- 172.21.15.6:0/2688267681 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fbf780046c0 con 0x7fbf88100ed0 2024-04-26T01:47:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.807+0000 7fbf91bb8740 1 Processor -- start 2024-04-26T01:47:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.807+0000 7fbf91bb8740 1 -- start start 2024-04-26T01:47:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.807+0000 7fbf91bb8740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 0x55d610a2cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.807+0000 7fbf91bb8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a37650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.807+0000 7fbf91bb8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a37b90 0x55d610a39f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.807+0000 7fbf91bb8740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d610992900 con 0x55d610a2c9b0 2024-04-26T01:47:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.807+0000 7fbf91bb8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d610938ef0 con 0x55d610a2d2d0 2024-04-26T01:47:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.807+0000 7fbf91bb8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d61098e8c0 con 0x55d610a37b90 2024-04-26T01:47:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.807+0000 7fbf8ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 0x55d610a2cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf8ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 0x55d610a2cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35454/0 (socket says 172.21.15.6:35454) 2024-04-26T01:47:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf8ce00640 1 -- 172.21.15.6:0/2186713220 learned_addr learned my addr 172.21.15.6:0/2186713220 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf87400640 1 --2- 172.21.15.6:0/2186713220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a37b90 0x55d610a39f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf87e00640 1 --2- 172.21.15.6:0/2186713220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a37650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf8ce00640 1 -- 172.21.15.6:0/2186713220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a37b90 msgr2=0x55d610a39f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf8ce00640 1 --2- 172.21.15.6:0/2186713220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a37b90 0x55d610a39f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf8ce00640 1 -- 172.21.15.6:0/2186713220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 msgr2=0x55d610a37650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf8ce00640 1 --2- 172.21.15.6:0/2186713220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a37650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf8ce00640 1 -- 172.21.15.6:0/2186713220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6106a5210 con 0x55d610a2c9b0 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf87e00640 1 --2- 172.21.15.6:0/2186713220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a37650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf84c00640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbf6c0a3db0 0x7fbf6c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf84c00640 1 -- 172.21.15.6:0/2688267681 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7fbf780d0b00 con 0x7fbf88100ed0 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf8ce00640 1 --2- 172.21.15.6:0/2186713220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 0x55d610a2cd90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009410 tx=0x7fbf80005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de3934338248cbc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.808+0000 7fbf87e00640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbf6c0a3db0 0x7fbf6c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.809+0000 7fbf7d600640 1 -- 172.21.15.6:0/2186713220 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80031cf0 con 0x55d610a2c9b0 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.809+0000 7fbf7d600640 1 -- 172.21.15.6:0/2186713220 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf80036030 con 0x55d610a2c9b0 2024-04-26T01:47:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.809+0000 7fbf7d600640 1 -- 172.21.15.6:0/2186713220 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80004240 con 0x55d610a2c9b0 2024-04-26T01:47:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.809+0000 7fbf91bb8740 1 -- 172.21.15.6:0/2186713220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 msgr2=0x55d610a2cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.809+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/2186713220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 0x55d610a2cd90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009410 tx=0x7fbf80005ef0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.809+0000 7fbf87e00640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbf6c0a3db0 0x7fbf6c0a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55d610a2df50 tx=0x7fbf78009720 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.810+0000 7fbf91bb8740 1 -- 172.21.15.6:0/2186713220 shutdown_connections 2024-04-26T01:47:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.810+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/2186713220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a37b90 0x55d610a39f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.810+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/2186713220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a37650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.810+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/2186713220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 0x55d610a2cd90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.810+0000 7fbf91bb8740 1 -- 172.21.15.6:0/2186713220 >> 172.21.15.6:0/2186713220 conn(0x55d610a28ec0 msgr2=0x55d610a292a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.810+0000 7fbf91bb8740 1 -- 172.21.15.6:0/2186713220 shutdown_connections 2024-04-26T01:47:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.810+0000 7fbf91bb8740 1 -- 172.21.15.6:0/2186713220 wait complete. 2024-04-26T01:47:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.810+0000 7fbf91bb8740 1 -- start start 2024-04-26T01:47:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.810+0000 7fbf91bb8740 1 Processor -- start 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.811+0000 7fbf91bb8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a2c9b0 0x55d610a34af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.811+0000 7fbf91bb8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a35030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.811+0000 7fbf91bb8740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 0x55d610a35570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.811+0000 7fbf91bb8740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d610a3a5e0 con 0x55d610a37b90 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.811+0000 7fbf91bb8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d61094c310 con 0x55d610a2d2d0 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.811+0000 7fbf91bb8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d610a3a760 con 0x55d610a2c9b0 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.811+0000 7fbf91bb8740 10 client.0 _ll_register_callbacks cb 0x55d61091acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.811+0000 7fbf8ce00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a2c9b0 0x55d610a34af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.811+0000 7fbf8ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 0x55d610a35570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.812+0000 7fbf8ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 0x55d610a35570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35470/0 (socket says 172.21.15.6:35470) 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.812+0000 7fbf8ce00640 1 -- 172.21.15.6:0/3366747701 learned_addr learned my addr 172.21.15.6:0/3366747701 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.812+0000 7fbf87400640 1 --2- 172.21.15.6:0/3366747701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a35030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.812+0000 7fbf8ce00640 1 -- 172.21.15.6:0/3366747701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a2c9b0 msgr2=0x55d610a34af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.812+0000 7fbf8ce00640 1 --2- 172.21.15.6:0/3366747701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a2c9b0 0x55d610a34af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.812+0000 7fbf8ce00640 1 -- 172.21.15.6:0/3366747701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 msgr2=0x55d610a35030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.812+0000 7fbf8ce00640 1 --2- 172.21.15.6:0/3366747701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a35030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.812+0000 7fbf8ce00640 1 -- 172.21.15.6:0/3366747701 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d610771620 con 0x55d610a37b90 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.812+0000 7fbf8ce00640 1 --2- 172.21.15.6:0/3366747701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 0x55d610a35570 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009a60 tx=0x7fbf8002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a52522adbe0e69 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.812+0000 7fbf72000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.812+0000 7fbf72000640 1 -- 172.21.15.6:0/3366747701 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80059060 con 0x55d610a37b90 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.812+0000 7fbf72000640 1 -- 172.21.15.6:0/3366747701 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7fbf80004620 con 0x55d610a37b90 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.813+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3366747701 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf80036030 con 0x55d610a37b90 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.813+0000 7fbf72000640 1 -- 172.21.15.6:0/3366747701 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbf8004a040 con 0x55d610a37b90 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.813+0000 7fbf91bb8740 10 client.15255 fetch_fsmap learned FSMap version 28 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.813+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3366747701 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d610a3a5e0 con 0x55d610a37b90 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.813+0000 7fbf72000640 1 -- 172.21.15.6:0/3366747701 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf80036030 con 0x55d610a37b90 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.813+0000 7fbf72000640 1 -- 172.21.15.6:0/3366747701 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7fbf80043070 con 0x55d610a37b90 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.813+0000 7fbf91bb8740 10 client.15255 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:47:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.813+0000 7fbf91bb8740 10 client.15255 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.813+0000 7fbf91bb8740 20 client.15255 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.813+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/3366747701 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d610a479c0 0x55d610a49db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.813+0000 7fbf91bb8740 4 client.15255 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.813+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3366747701 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6109a68b0 con 0x55d610a479c0 2024-04-26T01:47:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.814+0000 7fbf87400640 1 --2- 172.21.15.6:0/3366747701 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d610a479c0 0x55d610a49db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.814+0000 7fbf87400640 1 --2- 172.21.15.6:0/3366747701 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d610a479c0 0x55d610a49db0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfc7e24c789805a5 server_cookie=8abf97ab2600b24b in_seq=0 out_seq=0 2024-04-26T01:47:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.814+0000 7fbf72000640 10 client.15255 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.815+0000 7fbf72000640 1 -- 172.21.15.6:0/3366747701 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d6109a68b0 con 0x55d610a479c0 2024-04-26T01:47:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.815+0000 7fbf72000640 10 client.15255 handle_command_reply: tid=0 2024-04-26T01:47:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.815+0000 7fbf91bb8740 1 client.15255 shutdown 2024-04-26T01:47:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.815+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3366747701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 msgr2=0x55d610a35570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.815+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/3366747701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 0x55d610a35570 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009a60 tx=0x7fbf8002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.815+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3366747701 shutdown_connections 2024-04-26T01:47:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.815+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/3366747701 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d610a479c0 0x55d610a49db0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.815+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/3366747701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 0x55d610a35570 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.815+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/3366747701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a35030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.815+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/3366747701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a2c9b0 0x55d610a34af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.815+0000 7fbf72000640 0 client.15255 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.815+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3366747701 >> 172.21.15.6:0/3366747701 conn(0x55d610a28ec0 msgr2=0x55d610a2ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.816+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3366747701 shutdown_connections 2024-04-26T01:47:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.816+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3366747701 wait complete. 2024-04-26T01:47:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.816+0000 7fbf91bb8740 20 client.15255 trim_cache size 0 max 16384 2024-04-26T01:47:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.832+0000 7fbf91bb8740 1 Processor -- start 2024-04-26T01:47:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.832+0000 7fbf91bb8740 1 -- start start 2024-04-26T01:47:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf91bb8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a2c9b0 0x55d610a434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf91bb8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a4f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf91bb8740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 0x55d610a4fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf91bb8740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbf881b23f0 con 0x55d610a37b90 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf91bb8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbf881afcc0 con 0x55d610a2d2d0 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf91bb8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbf881adb10 con 0x55d610a2c9b0 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf8ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 0x55d610a4fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf87400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a4f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf87400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a4f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:50994/0 (socket says 172.21.15.6:50994) 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf87400640 1 -- 172.21.15.6:0/3142636204 learned_addr learned my addr 172.21.15.6:0/3142636204 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf8ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 0x55d610a4fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35474/0 (socket says 172.21.15.6:35474) 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf8ce00640 1 --2- 172.21.15.6:0/3142636204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a2c9b0 0x55d610a434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf87400640 1 -- 172.21.15.6:0/3142636204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a2c9b0 msgr2=0x55d610a434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf87400640 1 --2- 172.21.15.6:0/3142636204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a2c9b0 0x55d610a434a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf87400640 1 -- 172.21.15.6:0/3142636204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 msgr2=0x55d610a4fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf87400640 1 --2- 172.21.15.6:0/3142636204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 0x55d610a4fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.833+0000 7fbf87400640 1 -- 172.21.15.6:0/3142636204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d610a2a0f0 con 0x55d610a2d2d0 2024-04-26T01:47:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.834+0000 7fbf87400640 1 --2- 172.21.15.6:0/3142636204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a4f6a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55d610a2a050 tx=0x7fbf74005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b19a37c3413be01 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.834+0000 7fbf7cc00640 1 -- 172.21.15.6:0/3142636204 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf74023030 con 0x55d610a2d2d0 2024-04-26T01:47:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.834+0000 7fbf7cc00640 1 -- 172.21.15.6:0/3142636204 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf74004030 con 0x55d610a2d2d0 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.834+0000 7fbf7cc00640 1 -- 172.21.15.6:0/3142636204 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf74003c60 con 0x55d610a2d2d0 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.835+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3142636204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 msgr2=0x55d610a4f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.835+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/3142636204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a4f6a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55d610a2a050 tx=0x7fbf74005f90 comp rx=0 tx=0).stop 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.835+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3142636204 shutdown_connections 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.835+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/3142636204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a37b90 0x55d610a4fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.835+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/3142636204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a4f6a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.835+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/3142636204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a2c9b0 0x55d610a434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.835+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3142636204 >> 172.21.15.6:0/3142636204 conn(0x55d610a28ec0 msgr2=0x55d610a292a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.835+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3142636204 shutdown_connections 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.835+0000 7fbf91bb8740 1 -- 172.21.15.6:0/3142636204 wait complete. 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.835+0000 7fbf91bb8740 1 -- start start 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.836+0000 7fbf91bb8740 1 Processor -- start 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.836+0000 7fbf91bb8740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 0x55d610a3d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.836+0000 7fbf91bb8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a3d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.836+0000 7fbf91bb8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a37b90 0x55d610a3dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.836+0000 7fbf91bb8740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbf881ae180 con 0x55d610a2c9b0 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.836+0000 7fbf91bb8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbf881af5f0 con 0x55d610a2d2d0 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.836+0000 7fbf91bb8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbf881ae850 con 0x55d610a37b90 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.836+0000 7fbf91bb8740 10 client.0 _ll_register_callbacks cb 0x7fbf80003a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.836+0000 7fbf8ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 0x55d610a3d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.836+0000 7fbf8ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 0x55d610a3d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35476/0 (socket says 172.21.15.6:35476) 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.836+0000 7fbf8ce00640 1 -- 172.21.15.6:0/806403775 learned_addr learned my addr 172.21.15.6:0/806403775 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.837+0000 7fbf87400640 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a3d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.837+0000 7fbf8ce00640 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a37b90 0x55d610a3dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.837+0000 7fbf8ce00640 1 -- 172.21.15.6:0/806403775 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a37b90 msgr2=0x55d610a3dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.837+0000 7fbf8ce00640 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a37b90 0x55d610a3dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.837+0000 7fbf8ce00640 1 -- 172.21.15.6:0/806403775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 msgr2=0x55d610a3d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.837+0000 7fbf8ce00640 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a3d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.837+0000 7fbf8ce00640 1 -- 172.21.15.6:0/806403775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d610aacf20 con 0x55d610a2c9b0 2024-04-26T01:47:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.837+0000 7fbf8ce00640 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 0x55d610a3d320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbf8002ade0 tx=0x7fbf80003bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901bc5275dec9348 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.837+0000 7fbf72000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:47:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.837+0000 7fbf72000640 1 -- 172.21.15.6:0/806403775 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80059040 con 0x55d610a2c9b0 2024-04-26T01:47:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf72000640 1 -- 172.21.15.6:0/806403775 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7fbf8002f870 con 0x55d610a2c9b0 2024-04-26T01:47:30.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf91bb8740 1 -- 172.21.15.6:0/806403775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf74004030 con 0x55d610a2c9b0 2024-04-26T01:47:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf72000640 1 -- 172.21.15.6:0/806403775 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbf8002fa70 con 0x55d610a2c9b0 2024-04-26T01:47:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf91bb8740 10 client.15261 fetch_fsmap learned FSMap version 28 2024-04-26T01:47:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf91bb8740 1 -- 172.21.15.6:0/806403775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d610aae150 con 0x55d610a2c9b0 2024-04-26T01:47:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf72000640 1 -- 172.21.15.6:0/806403775 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf74004030 con 0x55d610a2c9b0 2024-04-26T01:47:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf72000640 1 -- 172.21.15.6:0/806403775 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7fbf80043070 con 0x55d610a2c9b0 2024-04-26T01:47:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf91bb8740 10 client.15261 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:47:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf91bb8740 10 client.15261 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf91bb8740 20 client.15261 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d610aae870 0x55d610ab0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf91bb8740 4 client.15261 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:47:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.838+0000 7fbf91bb8740 1 -- 172.21.15.6:0/806403775 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d6109a0320 con 0x55d610aae870 2024-04-26T01:47:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.839+0000 7fbf87400640 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d610aae870 0x55d610ab0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.839+0000 7fbf87400640 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d610aae870 0x55d610ab0c90 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95301492ba6f9011 server_cookie=5ce51bf29788d321 in_seq=0 out_seq=0 2024-04-26T01:47:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.839+0000 7fbf72000640 10 client.15261 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf72000640 1 -- 172.21.15.6:0/806403775 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d6109a0320 con 0x55d610aae870 2024-04-26T01:47:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf72000640 10 client.15261 handle_command_reply: tid=0 2024-04-26T01:47:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf91bb8740 1 client.15261 shutdown 2024-04-26T01:47:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf91bb8740 1 -- 172.21.15.6:0/806403775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 msgr2=0x55d610a3d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 0x55d610a3d320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbf8002ade0 tx=0x7fbf80003bf0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf91bb8740 1 -- 172.21.15.6:0/806403775 shutdown_connections 2024-04-26T01:47:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d610aae870 0x55d610ab0c90 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d610a37b90 0x55d610a3dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d610a2d2d0 0x55d610a3d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf91bb8740 1 --2- 172.21.15.6:0/806403775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d610a2c9b0 0x55d610a3d320 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf72000640 0 client.15261 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf91bb8740 1 -- 172.21.15.6:0/806403775 >> 172.21.15.6:0/806403775 conn(0x55d610a28ec0 msgr2=0x55d610a34ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf91bb8740 1 -- 172.21.15.6:0/806403775 shutdown_connections 2024-04-26T01:47:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf91bb8740 1 -- 172.21.15.6:0/806403775 wait complete. 2024-04-26T01:47:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.845+0000 7fbf91bb8740 20 client.15261 trim_cache size 0 max 16384 2024-04-26T01:47:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:47:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:47:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:47:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:47:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.846+0000 7fbf8d800640 1 -- 172.21.15.6:0/2688267681 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbf6c0a3db0 msgr2=0x7fbf6c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.846+0000 7fbf8d800640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbf6c0a3db0 0x7fbf6c0a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55d610a2df50 tx=0x7fbf78009720 comp rx=0 tx=0).stop 2024-04-26T01:47:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.846+0000 7fbf8d800640 1 -- 172.21.15.6:0/2688267681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88100ed0 msgr2=0x7fbf88102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.846+0000 7fbf8d800640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88100ed0 0x7fbf88102960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800e9f0 tx=0x7fbf7800edf0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.846+0000 7fbf8d800640 1 -- 172.21.15.6:0/2688267681 shutdown_connections 2024-04-26T01:47:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.846+0000 7fbf8d800640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbf6c0a3db0 0x7fbf6c0a6270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.846+0000 7fbf8d800640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf881033e0 0x7fbf881a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.846+0000 7fbf8d800640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf88105fa0 0x7fbf88102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.846+0000 7fbf8d800640 1 --2- 172.21.15.6:0/2688267681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88100ed0 0x7fbf88102960 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.846+0000 7fbf8d800640 1 -- 172.21.15.6:0/2688267681 >> 172.21.15.6:0/2688267681 conn(0x7fbf880fc730 msgr2=0x7fbf88069b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.846+0000 7fbf8d800640 1 -- 172.21.15.6:0/2688267681 shutdown_connections 2024-04-26T01:47:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:30.847+0000 7fbf8d800640 1 -- 172.21.15.6:0/2688267681 wait complete. 2024-04-26T01:47:31.870 INFO:tasks.workunit.client.0.smithi006.stderr:Updating files: 60% (7549/12492) Updating files: 61% (7621/12492) Updating files: 62% (7746/12492) Updating files: 63% (7870/12492) Updating files: 64% (7995/12492) Updating files: 65% (8120/12492) Updating files: 66% (8245/12492) Updating files: 67% (8370/12492) Updating files: 68% (8495/12492) Updating files: 69% (8620/12492) Updating files: 70% (8745/12492) Updating files: 71% (8870/12492) Updating files: 72% (8995/12492) Updating files: 73% (9120/12492) Updating files: 74% (9245/12492) Updating files: 75% (9369/12492) Updating files: 76% (9494/12492) Updating files: 77% (9619/12492) Updating files: 78% (9744/12492) Updating files: 79% (9869/12492) Updating files: 80% (9994/12492) Updating files: 81% (10119/12492) Updating files: 82% (10244/12492) Updating files: 83% (10369/12492) Updating files: 84% (10494/12492) Updating files: 85% (10619/12492) Updating files: 86% (10744/12492) Updating files: 87% (10869/12492) Updating files: 88% (10993/12492) Updating files: 89% (11118/12492) Updating files: 90% (11243/12492) Updating files: 91% (11368/12492) Updating files: 92% (11493/12492) Updating files: 93% (11618/12492) Updating files: 94% (11743/12492) Updating files: 95% (11868/12492) Updating files: 95% (11953/12492) Updating files: 96% (11993/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-26T01:47:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.206+0000 7f9786400700 1 -- 172.21.15.6:0/2283723837 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97801138d0 msgr2=0x7f9780115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.206+0000 7f9786400700 1 --2- 172.21.15.6:0/2283723837 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97801138d0 0x7f9780115cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f977800b0a0 tx=0x7f977800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.206+0000 7f9786400700 1 -- 172.21.15.6:0/2283723837 shutdown_connections 2024-04-26T01:47:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.206+0000 7f9786400700 1 --2- 172.21.15.6:0/2283723837 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97801138d0 0x7f9780115cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.206+0000 7f9786400700 1 --2- 172.21.15.6:0/2283723837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9780076f70 0x7f9780077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.206+0000 7f9786400700 1 --2- 172.21.15.6:0/2283723837 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9780077890 0x7f9780113390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.206+0000 7f9786400700 1 -- 172.21.15.6:0/2283723837 >> 172.21.15.6:0/2283723837 conn(0x7f9780100720 msgr2=0x7f9780102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.206+0000 7f9786400700 1 -- 172.21.15.6:0/2283723837 shutdown_connections 2024-04-26T01:47:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.206+0000 7f9786400700 1 -- 172.21.15.6:0/2283723837 wait complete. 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.207+0000 7f9786400700 1 Processor -- start 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.208+0000 7f9786400700 1 -- start start 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.208+0000 7f9786400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9780076f70 0x7f978019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.208+0000 7f9786400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9780077890 0x7f978019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.208+0000 7f9786400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97801138d0 0x7f97801a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.208+0000 7f9786400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9780119250 con 0x7f9780076f70 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.208+0000 7f9786400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f97801190d0 con 0x7f97801138d0 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.208+0000 7f9786400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f97801193d0 con 0x7f9780077890 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.208+0000 7f977f400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9780076f70 0x7f978019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.208+0000 7f977ea00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9780077890 0x7f978019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.208+0000 7f977f400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9780076f70 0x7f978019f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35496/0 (socket says 172.21.15.6:35496) 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.208+0000 7f977f400700 1 -- 172.21.15.6:0/3844789084 learned_addr learned my addr 172.21.15.6:0/3844789084 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.209+0000 7f977fe00700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97801138d0 0x7f97801a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.209+0000 7f977ea00700 1 -- 172.21.15.6:0/3844789084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97801138d0 msgr2=0x7f97801a3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.209+0000 7f977ea00700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97801138d0 0x7f97801a3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.209+0000 7f977ea00700 1 -- 172.21.15.6:0/3844789084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9780076f70 msgr2=0x7f978019f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.209+0000 7f977ea00700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9780076f70 0x7f978019f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.209+0000 7f977ea00700 1 -- 172.21.15.6:0/3844789084 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97801a4700 con 0x7f9780077890 2024-04-26T01:47:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.209+0000 7f977ea00700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9780077890 0x7f978019fc80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f976c00e9f0 tx=0x7f976c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.210+0000 7f9777e00700 1 -- 172.21.15.6:0/3844789084 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c00cb70 con 0x7f9780077890 2024-04-26T01:47:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.210+0000 7f9777e00700 1 -- 172.21.15.6:0/3844789084 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f976c00ee90 con 0x7f9780077890 2024-04-26T01:47:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.210+0000 7f9786400700 1 -- 172.21.15.6:0/3844789084 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97801a49f0 con 0x7f9780077890 2024-04-26T01:47:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.210+0000 7f9777e00700 1 -- 172.21.15.6:0/3844789084 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c003d40 con 0x7f9780077890 2024-04-26T01:47:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.210+0000 7f9786400700 1 -- 172.21.15.6:0/3844789084 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97801ac270 con 0x7f9780077890 2024-04-26T01:47:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.211+0000 7f9786400700 1 -- 172.21.15.6:0/3844789084 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9780069730 con 0x7f9780077890 2024-04-26T01:47:32.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.213+0000 7f9777e00700 1 -- 172.21.15.6:0/3844789084 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f976c003f00 con 0x7f9780077890 2024-04-26T01:47:32.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.214+0000 7f9777e00700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f97640a3db0 0x7f97640a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:32.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.214+0000 7f9777e00700 1 -- 172.21.15.6:0/3844789084 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f976c0d0a80 con 0x7f9780077890 2024-04-26T01:47:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.215+0000 7f977f400700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f97640a3db0 0x7f97640a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.215+0000 7f977f400700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f97640a3db0 0x7f97640a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9770009d20 tx=0x7f9770009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:32.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.217+0000 7f9777e00700 1 -- 172.21.15.6:0/3844789084 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f976c09d0a0 con 0x7f9780077890 2024-04-26T01:47:32.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:32 smithi006 ceph-mon[28419]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-26T01:47:32.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:32 smithi188 ceph-mon[32751]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-26T01:47:32.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:32 smithi029 ceph-mon[32775]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr:Note: switching to 'afa1933b0fdbb6c99c947d1eda34d661d23cd327'. 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr:state without impacting any branches by switching back to a branch. 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr: git switch -c 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr:Or undo this operation with: 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr: git switch - 2024-04-26T01:47:32.457 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-26T01:47:32.458 INFO:tasks.workunit.client.0.smithi006.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-26T01:47:32.458 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-26T01:47:32.458 INFO:tasks.workunit.client.0.smithi006.stderr:HEAD is now at afa1933b0fd qa/suites/fs: add quiescer to the fs suite 2024-04-26T01:47:32.461 DEBUG:teuthology.orchestra.run.smithi006:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-26T01:47:32.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.494+0000 7f9786400700 1 -- 172.21.15.6:0/3844789084 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f978002df80 con 0x7f9780077890 2024-04-26T01:47:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.495+0000 7f9777e00700 1 -- 172.21.15.6:0/3844789084 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11844 (secure 0 0 0) 0x7f976c09c9a0 con 0x7f9780077890 2024-04-26T01:47:32.494 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:47:32.494 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:47:17.538822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25372":{"gid":25372,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/47657351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":47657351},{"type":"v1","addr":"172.21.15.188:6835","nonce":47657351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24512,24505]},"id":1}]} 2024-04-26T01:47:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.498+0000 7f9786400700 1 -- 172.21.15.6:0/3844789084 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f97640a3db0 msgr2=0x7f97640a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.498+0000 7f9786400700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f97640a3db0 0x7f97640a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9770009d20 tx=0x7f9770009260 comp rx=0 tx=0).stop 2024-04-26T01:47:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.498+0000 7f9786400700 1 -- 172.21.15.6:0/3844789084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9780077890 msgr2=0x7f978019fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.498+0000 7f9786400700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9780077890 0x7f978019fc80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f976c00e9f0 tx=0x7f976c00ed00 comp rx=0 tx=0).stop 2024-04-26T01:47:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.498+0000 7f9786400700 1 -- 172.21.15.6:0/3844789084 shutdown_connections 2024-04-26T01:47:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.498+0000 7f9786400700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f97640a3db0 0x7f97640a6270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.498+0000 7f9786400700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97801138d0 0x7f97801a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.498+0000 7f9786400700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9780076f70 0x7f978019f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.498+0000 7f9786400700 1 --2- 172.21.15.6:0/3844789084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9780077890 0x7f978019fc80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.498+0000 7f9786400700 1 -- 172.21.15.6:0/3844789084 >> 172.21.15.6:0/3844789084 conn(0x7f9780100720 msgr2=0x7f9780102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.498+0000 7f9786400700 1 -- 172.21.15.6:0/3844789084 shutdown_connections 2024-04-26T01:47:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:32.498+0000 7f9786400700 1 -- 172.21.15.6:0/3844789084 wait complete. 2024-04-26T01:47:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 28 2024-04-26T01:47:32.553 INFO:tasks.workunit.client.0.smithi006.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-26T01:47:32.558 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-26T01:47:32.558 INFO:tasks.workunit.client.0.smithi006.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-26T01:47:33.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:33 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3844789084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:33.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:33 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3844789084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:33.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:33 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3844789084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:33.546 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id 74c6c8bd --release --await' 2024-04-26T01:47:33.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id 74c6c8bd --release --await 2024-04-26T01:47:33.962 INFO:tasks.workunit.client.0.smithi006.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-26T01:47:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:47:34.081 INFO:tasks.workunit.client.0.smithi006.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-26T01:47:34.167 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-26T01:47:34.172 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-26T01:47:34.172 INFO:tasks.workunit.client.0.smithi006.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-26T01:47:34.256 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-26T01:47:34.267 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T01:47:34.267 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-26T01:47:34.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:34 smithi006 ceph-mon[28419]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-26T01:47:34.318 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-04-26T01:47:34.320 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-04-26T01:47:34.320 DEBUG:teuthology.orchestra.run.smithi006:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afa1933b0fdbb6c99c947d1eda34d661d23cd327 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-04-26T01:47:34.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:34 smithi188 ceph-mon[32751]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-26T01:47:34.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:34 smithi029 ceph-mon[32775]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-26T01:47:36.181 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:36 smithi006 ceph-mon[28419]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-26T01:47:36.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.370+0000 7f6884000700 1 -- 172.21.15.6:0/2294988478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f687c1092d0 msgr2=0x7f687c1096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.370+0000 7f6884000700 1 --2- 172.21.15.6:0/2294988478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f687c1092d0 0x7f687c1096b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f68700098f0 tx=0x7f6870009c00 comp rx=0 tx=0).stop 2024-04-26T01:47:36.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.370+0000 7f6884000700 1 -- 172.21.15.6:0/2294988478 shutdown_connections 2024-04-26T01:47:36.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.370+0000 7f6884000700 1 --2- 172.21.15.6:0/2294988478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f687c10e3b0 0x7f687c115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.370+0000 7f6884000700 1 --2- 172.21.15.6:0/2294988478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f687c1092d0 0x7f687c1096b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.370+0000 7f6884000700 1 --2- 172.21.15.6:0/2294988478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f687c109c80 0x7f687c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.370+0000 7f6884000700 1 -- 172.21.15.6:0/2294988478 >> 172.21.15.6:0/2294988478 conn(0x7f687c076210 msgr2=0x7f687c107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:36.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.371+0000 7f6884000700 1 -- 172.21.15.6:0/2294988478 shutdown_connections 2024-04-26T01:47:36.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.371+0000 7f6884000700 1 -- 172.21.15.6:0/2294988478 wait complete. 2024-04-26T01:47:36.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.372+0000 7f6884000700 1 Processor -- start 2024-04-26T01:47:36.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.372+0000 7f6884000700 1 -- start start 2024-04-26T01:47:36.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.373+0000 7f6884000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f687c1092d0 0x7f687c19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.373+0000 7f6884000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f687c109c80 0x7f687c19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.373+0000 7f6884000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f687c10e3b0 0x7f687c1a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.373+0000 7f6884000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f687c119240 con 0x7f687c109c80 2024-04-26T01:47:36.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.373+0000 7f6884000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1190c0 con 0x7f687c1092d0 2024-04-26T01:47:36.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.373+0000 7f6884000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1193c0 con 0x7f687c10e3b0 2024-04-26T01:47:36.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.373+0000 7f6881600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f687c1092d0 0x7f687c19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.373+0000 7f6880c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f687c109c80 0x7f687c19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.373+0000 7f6881600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f687c1092d0 0x7f687c19f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:51038/0 (socket says 172.21.15.6:51038) 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.373+0000 7f6881600700 1 -- 172.21.15.6:0/1711599675 learned_addr learned my addr 172.21.15.6:0/1711599675 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.373+0000 7f6882000700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f687c10e3b0 0x7f687c1a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.374+0000 7f6881600700 1 -- 172.21.15.6:0/1711599675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f687c10e3b0 msgr2=0x7f687c1a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.374+0000 7f6881600700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f687c10e3b0 0x7f687c1a4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.374+0000 7f6881600700 1 -- 172.21.15.6:0/1711599675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f687c109c80 msgr2=0x7f687c19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.374+0000 7f6881600700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f687c109c80 0x7f687c19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.374+0000 7f6881600700 1 -- 172.21.15.6:0/1711599675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687c1a4740 con 0x7f687c1092d0 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.374+0000 7f6881600700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f687c1092d0 0x7f687c19f730 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6870009230 tx=0x7f687000f9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.374+0000 7f687a000700 1 -- 172.21.15.6:0/1711599675 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870027070 con 0x7f687c1092d0 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.374+0000 7f687a000700 1 -- 172.21.15.6:0/1711599675 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68700055f0 con 0x7f687c1092d0 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.374+0000 7f6884000700 1 -- 172.21.15.6:0/1711599675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f687c1a49d0 con 0x7f687c1092d0 2024-04-26T01:47:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.374+0000 7f687a000700 1 -- 172.21.15.6:0/1711599675 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870004930 con 0x7f687c1092d0 2024-04-26T01:47:36.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.375+0000 7f6884000700 1 -- 172.21.15.6:0/1711599675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f687c1ac2b0 con 0x7f687c1092d0 2024-04-26T01:47:36.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.378+0000 7f687a000700 1 -- 172.21.15.6:0/1711599675 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f687002a070 con 0x7f687c1092d0 2024-04-26T01:47:36.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.380+0000 7f688e8b7b80 1 Processor -- start 2024-04-26T01:47:36.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.380+0000 7f688e8b7b80 1 -- start start 2024-04-26T01:47:36.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.380+0000 7f687a000700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68640a3e30 0x7f68640a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.380+0000 7f687a000700 1 -- 172.21.15.6:0/1711599675 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f68700d10e0 con 0x7f687c1092d0 2024-04-26T01:47:36.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f688e8b7b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fa8920 0x55b8e2fa8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f688e8b7b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 0x55b8e2fb3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f6882000700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68640a3e30 0x7f68640a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f688e8b7b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f688e8b7b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55b8e2efbf90 con 0x55b8e2fa8920 2024-04-26T01:47:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f688e8b7b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b8e2ea2830 con 0x55b8e2fa9240 2024-04-26T01:47:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f688e8b7b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b8e2ef7f50 con 0x55b8e2fb40f0 2024-04-26T01:47:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f6881600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fa8920 0x55b8e2fa8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f6880c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 0x55b8e2fb3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f6881600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fa8920 0x55b8e2fa8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35516/0 (socket says 172.21.15.6:35516) 2024-04-26T01:47:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f6880c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 0x55b8e2fb3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:51052/0 (socket says 172.21.15.6:51052) 2024-04-26T01:47:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f6881600700 1 -- 172.21.15.6:0/2443830382 learned_addr learned my addr 172.21.15.6:0/2443830382 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.381+0000 7f6882000700 1 --2- 172.21.15.6:0/2443830382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.382+0000 7f6880c00700 1 -- 172.21.15.6:0/2443830382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fb40f0 msgr2=0x55b8e2fb64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.382+0000 7f6880c00700 1 --2- 172.21.15.6:0/2443830382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.382+0000 7f6880c00700 1 -- 172.21.15.6:0/2443830382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fa8920 msgr2=0x55b8e2fa8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.382+0000 7f6880c00700 1 --2- 172.21.15.6:0/2443830382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fa8920 0x55b8e2fa8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.382+0000 7f6880c00700 1 -- 172.21.15.6:0/2443830382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8e2e98780 con 0x55b8e2fa9240 2024-04-26T01:47:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.382+0000 7f6881600700 1 --2- 172.21.15.6:0/2443830382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fa8920 0x55b8e2fa8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.382+0000 7f6882000700 1 --2- 172.21.15.6:0/2443830382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.382+0000 7f6880c00700 1 --2- 172.21.15.6:0/2443830382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 0x55b8e2fb3bb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f687c06c260 tx=0x7f6868011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a04bdacb270206a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.382+0000 7f6882000700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68640a3e30 0x7f68640a62f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f687400f940 tx=0x7f6874009920 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.382+0000 7f686ea00700 1 -- 172.21.15.6:0/2443830382 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6868011c90 con 0x55b8e2fa9240 2024-04-26T01:47:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.382+0000 7f686ea00700 1 -- 172.21.15.6:0/2443830382 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6868011e30 con 0x55b8e2fa9240 2024-04-26T01:47:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.382+0000 7f686ea00700 1 -- 172.21.15.6:0/2443830382 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6868005280 con 0x55b8e2fa9240 2024-04-26T01:47:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.383+0000 7f688e8b7b80 1 -- 172.21.15.6:0/2443830382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 msgr2=0x55b8e2fb3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.383+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/2443830382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 0x55b8e2fb3bb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f687c06c260 tx=0x7f6868011440 comp rx=0 tx=0).stop 2024-04-26T01:47:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.383+0000 7f688e8b7b80 1 -- 172.21.15.6:0/2443830382 shutdown_connections 2024-04-26T01:47:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.383+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/2443830382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.383+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/2443830382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fa8920 0x55b8e2fa8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.383+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/2443830382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 0x55b8e2fb3bb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.383+0000 7f688e8b7b80 1 -- 172.21.15.6:0/2443830382 >> 172.21.15.6:0/2443830382 conn(0x55b8e2ed76a0 msgr2=0x55b8e2fa4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.383+0000 7f688e8b7b80 1 -- 172.21.15.6:0/2443830382 shutdown_connections 2024-04-26T01:47:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.383+0000 7f688e8b7b80 1 -- 172.21.15.6:0/2443830382 wait complete. 2024-04-26T01:47:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.384+0000 7f688e8b7b80 1 -- start start 2024-04-26T01:47:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.384+0000 7f688e8b7b80 1 Processor -- start 2024-04-26T01:47:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f688e8b7b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 0x55b8e2fb37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f688e8b7b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 0x55b8e2fb0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f688e8b7b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f688e8b7b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55b8e2fb6c10 con 0x55b8e2fb40f0 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f688e8b7b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b8e2eb5960 con 0x55b8e2fa9240 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f688e8b7b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b8e2fb6d90 con 0x55b8e2fa8920 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f688e8b7b80 10 client.0 _ll_register_callbacks cb 0x55b8e2e94820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f6881600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 0x55b8e2fb37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f6882000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f6880c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 0x55b8e2fb0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f6881600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 0x55b8e2fb37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37510/0 (socket says 172.21.15.6:37510) 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f6881600700 1 -- 172.21.15.6:0/3420287633 learned_addr learned my addr 172.21.15.6:0/3420287633 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f6880c00700 1 -- 172.21.15.6:0/3420287633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 msgr2=0x55b8e2fb37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f6880c00700 1 --2- 172.21.15.6:0/3420287633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 0x55b8e2fb37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f6880c00700 1 -- 172.21.15.6:0/3420287633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fb40f0 msgr2=0x55b8e2fb0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f6880c00700 1 --2- 172.21.15.6:0/3420287633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f6880c00700 1 -- 172.21.15.6:0/3420287633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8e2e499d0 con 0x55b8e2fa9240 2024-04-26T01:47:36.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.385+0000 7f6881600700 1 --2- 172.21.15.6:0/3420287633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 0x55b8e2fb37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:36.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.386+0000 7f6880c00700 1 --2- 172.21.15.6:0/3420287633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 0x55b8e2fb0ab0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f687c06c260 tx=0x7f6868004840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c1036616883fb1f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:36.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.386+0000 7f6863400700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:47:36.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.386+0000 7f6863400700 1 -- 172.21.15.6:0/3420287633 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6868004af0 con 0x55b8e2fa9240 2024-04-26T01:47:36.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.386+0000 7f6863400700 1 -- 172.21.15.6:0/3420287633 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f6868004260 con 0x55b8e2fa9240 2024-04-26T01:47:36.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.386+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3420287633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6868011e30 con 0x55b8e2fa9240 2024-04-26T01:47:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.387+0000 7f6863400700 1 -- 172.21.15.6:0/3420287633 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6868030050 con 0x55b8e2fa9240 2024-04-26T01:47:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.387+0000 7f688e8b7b80 10 client.25379 fetch_fsmap learned FSMap version 28 2024-04-26T01:47:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.387+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3420287633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b8e2e92bf0 con 0x55b8e2fa9240 2024-04-26T01:47:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.387+0000 7f6863400700 1 -- 172.21.15.6:0/3420287633 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6868011e30 con 0x55b8e2fa9240 2024-04-26T01:47:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.387+0000 7f6863400700 1 -- 172.21.15.6:0/3420287633 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f6868018070 con 0x55b8e2fa9240 2024-04-26T01:47:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.388+0000 7f688e8b7b80 10 client.25379 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:47:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.388+0000 7f688e8b7b80 10 client.25379 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:47:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.388+0000 7f688e8b7b80 20 client.25379 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.388+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/3420287633 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8e2fc3790 0x55b8e2fc5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.388+0000 7f688e8b7b80 4 client.25379 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.388+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3420287633 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8e2f09780 con 0x55b8e2fc3790 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.388+0000 7f6881600700 1 --2- 172.21.15.6:0/3420287633 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8e2fc3790 0x55b8e2fc5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.389+0000 7f6881600700 1 --2- 172.21.15.6:0/3420287633 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8e2fc3790 0x55b8e2fc5b80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee1e2bf6c53b9f23 server_cookie=4f5170a21a2eccc9 in_seq=0 out_seq=0 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.389+0000 7f6863400700 10 client.25379 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.389+0000 7f6863400700 1 -- 172.21.15.6:0/3420287633 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b8e2f09780 con 0x55b8e2fc3790 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.389+0000 7f6863400700 10 client.25379 handle_command_reply: tid=0 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f688e8b7b80 1 client.25379 shutdown 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3420287633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 msgr2=0x55b8e2fb0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/3420287633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 0x55b8e2fb0ab0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f687c06c260 tx=0x7f6868004840 comp rx=0 tx=0).stop 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3420287633 shutdown_connections 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/3420287633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/3420287633 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8e2fc3790 0x55b8e2fc5b80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/3420287633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 0x55b8e2fb37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f6863400700 0 client.25379 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:36.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/3420287633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa9240 0x55b8e2fb0ab0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3420287633 >> 172.21.15.6:0/3420287633 conn(0x55b8e2ed76a0 msgr2=0x55b8e2fa49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:36.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3420287633 shutdown_connections 2024-04-26T01:47:36.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3420287633 wait complete. 2024-04-26T01:47:36.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.390+0000 7f688e8b7b80 20 client.25379 trim_cache size 0 max 16384 2024-04-26T01:47:36.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:36 smithi188 ceph-mon[32751]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-26T01:47:36.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.412+0000 7f688e8b7b80 1 Processor -- start 2024-04-26T01:47:36.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.412+0000 7f688e8b7b80 1 -- start start 2024-04-26T01:47:36.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.412+0000 7f688e8b7b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa8920 0x55b8e2fbfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.412+0000 7f688e8b7b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa9240 0x55b8e2fcb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.412+0000 7f688e8b7b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fcb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.412+0000 7f688e8b7b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1b61d0 con 0x55b8e2fb40f0 2024-04-26T01:47:36.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.412+0000 7f688e8b7b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1b3a60 con 0x55b8e2fa8920 2024-04-26T01:47:36.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.412+0000 7f688e8b7b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1b18b0 con 0x55b8e2fa9240 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.413+0000 7f6880c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa9240 0x55b8e2fcb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.413+0000 7f6881600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa8920 0x55b8e2fbfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.413+0000 7f6880c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa9240 0x55b8e2fcb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37526/0 (socket says 172.21.15.6:37526) 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.413+0000 7f6882000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fcb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.413+0000 7f6880c00700 1 -- 172.21.15.6:0/3432908807 learned_addr learned my addr 172.21.15.6:0/3432908807 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.413+0000 7f6880c00700 1 -- 172.21.15.6:0/3432908807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa8920 msgr2=0x55b8e2fbfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.413+0000 7f6880c00700 1 --2- 172.21.15.6:0/3432908807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa8920 0x55b8e2fbfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.413+0000 7f6880c00700 1 -- 172.21.15.6:0/3432908807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fb40f0 msgr2=0x55b8e2fcb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.413+0000 7f6880c00700 1 --2- 172.21.15.6:0/3432908807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fcb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.413+0000 7f6880c00700 1 -- 172.21.15.6:0/3432908807 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8e2fc0080 con 0x55b8e2fa9240 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.414+0000 7f6880c00700 1 --2- 172.21.15.6:0/3432908807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa9240 0x55b8e2fcb2a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f686802e040 tx=0x7f6868001e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f6afd2b947c7daa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.414+0000 7f686e000700 1 -- 172.21.15.6:0/3432908807 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6868004af0 con 0x55b8e2fa9240 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.414+0000 7f686e000700 1 -- 172.21.15.6:0/3432908807 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6868015030 con 0x55b8e2fa9240 2024-04-26T01:47:36.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.414+0000 7f686e000700 1 -- 172.21.15.6:0/3432908807 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68680038e0 con 0x55b8e2fa9240 2024-04-26T01:47:36.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.414+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3432908807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa9240 msgr2=0x55b8e2fcb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.414+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/3432908807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa9240 0x55b8e2fcb2a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f686802e040 tx=0x7f6868001e50 comp rx=0 tx=0).stop 2024-04-26T01:47:36.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.414+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3432908807 shutdown_connections 2024-04-26T01:47:36.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.414+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/3432908807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fcb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.414+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/3432908807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fa8920 0x55b8e2fbfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.414+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/3432908807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa9240 0x55b8e2fcb2a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.414+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3432908807 >> 172.21.15.6:0/3432908807 conn(0x55b8e2ed76a0 msgr2=0x55b8e2fac1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.415+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3432908807 shutdown_connections 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.415+0000 7f688e8b7b80 1 -- 172.21.15.6:0/3432908807 wait complete. 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.415+0000 7f688e8b7b80 1 -- start start 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.415+0000 7f688e8b7b80 1 Processor -- start 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.416+0000 7f688e8b7b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 0x55b8e2fbc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.416+0000 7f688e8b7b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fa9240 0x55b8e2fbc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.416+0000 7f688e8b7b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.416+0000 7f688e8b7b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1b1f20 con 0x55b8e2fa9240 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.416+0000 7f688e8b7b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1b3390 con 0x55b8e2fb40f0 2024-04-26T01:47:36.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.416+0000 7f688e8b7b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1b25f0 con 0x55b8e2fa8920 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.416+0000 7f688e8b7b80 10 client.0 _ll_register_callbacks cb 0x55b8e2e94820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.416+0000 7f6880c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fa9240 0x55b8e2fbc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.416+0000 7f6881600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 0x55b8e2fbc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.416+0000 7f6881600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 0x55b8e2fbc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37536/0 (socket says 172.21.15.6:37536) 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.416+0000 7f6881600700 1 -- 172.21.15.6:0/1815315835 learned_addr learned my addr 172.21.15.6:0/1815315835 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.416+0000 7f6882000700 1 --2- 172.21.15.6:0/1815315835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.417+0000 7f6881600700 1 -- 172.21.15.6:0/1815315835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fb40f0 msgr2=0x55b8e2fb9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.417+0000 7f6881600700 1 --2- 172.21.15.6:0/1815315835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.417+0000 7f6881600700 1 -- 172.21.15.6:0/1815315835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fa9240 msgr2=0x55b8e2fbc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.417+0000 7f6881600700 1 --2- 172.21.15.6:0/1815315835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fa9240 0x55b8e2fbc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.417+0000 7f6881600700 1 -- 172.21.15.6:0/1815315835 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8e2fb9860 con 0x55b8e2fa8920 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.417+0000 7f6881600700 1 --2- 172.21.15.6:0/1815315835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 0x55b8e2fbc050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6870099500 tx=0x7f6870019040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b0153841c1f5f91 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.417+0000 7f6863400700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.417+0000 7f6863400700 1 -- 172.21.15.6:0/1815315835 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68700a1560 con 0x55b8e2fa8920 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.417+0000 7f6863400700 1 -- 172.21.15.6:0/1815315835 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f6870099820 con 0x55b8e2fa8920 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.417+0000 7f688e8b7b80 1 -- 172.21.15.6:0/1815315835 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6868015030 con 0x55b8e2fa8920 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.418+0000 7f6863400700 1 -- 172.21.15.6:0/1815315835 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f68700a5d90 con 0x55b8e2fa8920 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.418+0000 7f688e8b7b80 10 client.25432 fetch_fsmap learned FSMap version 28 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.418+0000 7f688e8b7b80 1 -- 172.21.15.6:0/1815315835 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b8e302a700 con 0x55b8e2fa8920 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.418+0000 7f6863400700 1 -- 172.21.15.6:0/1815315835 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6868015030 con 0x55b8e2fa8920 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.418+0000 7f6863400700 1 -- 172.21.15.6:0/1815315835 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f687009d410 con 0x55b8e2fa8920 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.418+0000 7f688e8b7b80 10 client.25432 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.418+0000 7f688e8b7b80 10 client.25432 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.418+0000 7f688e8b7b80 20 client.25432 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.418+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/1815315835 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8e302ad60 0x55b8e302d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.418+0000 7f688e8b7b80 4 client.25432 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "74c6c8bd", "release": true, "await": true}] 2024-04-26T01:47:36.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.418+0000 7f688e8b7b80 1 -- 172.21.15.6:0/1815315835 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "74c6c8bd", "release": true, "await": true}) -- 0x55b8e2f03820 con 0x55b8e302ad60 2024-04-26T01:47:36.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.419+0000 7f6880c00700 1 --2- 172.21.15.6:0/1815315835 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8e302ad60 0x55b8e302d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.419+0000 7f6880c00700 1 --2- 172.21.15.6:0/1815315835 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8e302ad60 0x55b8e302d180 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82c9f8332866ee7 server_cookie=74e19106b66c4129 in_seq=0 out_seq=0 2024-04-26T01:47:36.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.419+0000 7f6863400700 10 client.25432 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.423+0000 7f6863400700 1 -- 172.21.15.6:0/1815315835 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55b8e2f03820 con 0x55b8e302ad60 2024-04-26T01:47:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.423+0000 7f6863400700 10 client.25432 handle_command_reply: tid=0 2024-04-26T01:47:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.423+0000 7f688e8b7b80 1 client.25432 shutdown 2024-04-26T01:47:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.423+0000 7f688e8b7b80 1 -- 172.21.15.6:0/1815315835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 msgr2=0x55b8e2fbc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.423+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/1815315835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 0x55b8e2fbc050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6870099500 tx=0x7f6870019040 comp rx=0 tx=0).stop 2024-04-26T01:47:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.424+0000 7f688e8b7b80 1 -- 172.21.15.6:0/1815315835 shutdown_connections 2024-04-26T01:47:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.424+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/1815315835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8e2fb40f0 0x55b8e2fb9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.424+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/1815315835 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8e302ad60 0x55b8e302d180 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.424+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/1815315835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8e2fa8920 0x55b8e2fbc050 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.424+0000 7f688e8b7b80 1 --2- 172.21.15.6:0/1815315835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8e2fa9240 0x55b8e2fbc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.424+0000 7f6863400700 0 client.25432 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.424+0000 7f688e8b7b80 1 -- 172.21.15.6:0/1815315835 >> 172.21.15.6:0/1815315835 conn(0x55b8e2ed76a0 msgr2=0x55b8e2fb22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.424+0000 7f688e8b7b80 1 -- 172.21.15.6:0/1815315835 shutdown_connections 2024-04-26T01:47:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.424+0000 7f688e8b7b80 1 -- 172.21.15.6:0/1815315835 wait complete. 2024-04-26T01:47:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.424+0000 7f688e8b7b80 20 client.25432 trim_cache size 0 max 16384 2024-04-26T01:47:36.424 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:47:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 28, 2024-04-26T01:47:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:47:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 4, 2024-04-26T01:47:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:47:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "74c6c8bd": { 2024-04-26T01:47:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 4, 2024-04-26T01:47:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:47:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 85.6, 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:47:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:47:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:47:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:47:36.426 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:47:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.425+0000 7f6884000700 1 -- 172.21.15.6:0/1711599675 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68640a3e30 msgr2=0x7f68640a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.425+0000 7f6884000700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68640a3e30 0x7f68640a62f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f687400f940 tx=0x7f6874009920 comp rx=0 tx=0).stop 2024-04-26T01:47:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.425+0000 7f6884000700 1 -- 172.21.15.6:0/1711599675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f687c1092d0 msgr2=0x7f687c19f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.425+0000 7f6884000700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f687c1092d0 0x7f687c19f730 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6870009230 tx=0x7f687000f9b0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.425+0000 7f6884000700 1 -- 172.21.15.6:0/1711599675 shutdown_connections 2024-04-26T01:47:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.425+0000 7f6884000700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68640a3e30 0x7f68640a62f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.425+0000 7f6884000700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f687c10e3b0 0x7f687c1a4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.425+0000 7f6884000700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f687c1092d0 0x7f687c19f730 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.425+0000 7f6884000700 1 --2- 172.21.15.6:0/1711599675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f687c109c80 0x7f687c19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.425+0000 7f6884000700 1 -- 172.21.15.6:0/1711599675 >> 172.21.15.6:0/1711599675 conn(0x7f687c076210 msgr2=0x7f687c10d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.425+0000 7f6884000700 1 -- 172.21.15.6:0/1711599675 shutdown_connections 2024-04-26T01:47:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:47:36.426+0000 7f6884000700 1 -- 172.21.15.6:0/1711599675 wait complete. 2024-04-26T01:47:36.436 INFO:tasks.workunit.client.0.smithi006.stderr:+ dbench 1 2024-04-26T01:47:36.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:36 smithi029 ceph-mon[32775]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-26T01:47:36.443 INFO:tasks.workunit.client.0.smithi006.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-26T01:47:36.443 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-26T01:47:36.443 INFO:tasks.workunit.client.0.smithi006.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-26T01:47:36.448 INFO:tasks.workunit.client.0.smithi006.stdout:0 of 1 processes prepared for launch 0 sec 2024-04-26T01:47:36.548 INFO:tasks.workunit.client.0.smithi006.stdout:1 of 1 processes prepared for launch 0 sec 2024-04-26T01:47:36.548 INFO:tasks.workunit.client.0.smithi006.stdout:releasing clients 2024-04-26T01:47:37.416 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '74c6c8bd', seconds elapsed: 42.9 2024-04-26T01:47:37.549 INFO:tasks.workunit.client.0.smithi006.stdout: 1 1698 26.22 MB/sec warmup 1 sec latency 14.412 ms 2024-04-26T01:47:38.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:38 smithi188 ceph-mon[32751]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-26T01:47:38.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:38 smithi029 ceph-mon[32775]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-26T01:47:38.549 INFO:tasks.workunit.client.0.smithi006.stdout: 1 4022 20.03 MB/sec warmup 2 sec latency 9.898 ms 2024-04-26T01:47:38.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:38 smithi006 ceph-mon[28419]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-26T01:47:39.549 INFO:tasks.workunit.client.0.smithi006.stdout: 1 6060 15.95 MB/sec warmup 3 sec latency 12.766 ms 2024-04-26T01:47:40.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:40 smithi188 ceph-mon[32751]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-26T01:47:40.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:40 smithi029 ceph-mon[32775]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-26T01:47:40.549 INFO:tasks.workunit.client.0.smithi006.stdout: 1 7834 14.88 MB/sec warmup 4 sec latency 12.280 ms 2024-04-26T01:47:40.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:40 smithi006 ceph-mon[28419]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-26T01:47:41.549 INFO:tasks.workunit.client.0.smithi006.stdout: 1 10118 14.35 MB/sec warmup 5 sec latency 10.665 ms 2024-04-26T01:47:42.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:42 smithi188 ceph-mon[32751]: pgmap v57: 97 pgs: 97 active+clean; 6.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 455 KiB/s wr, 59 op/s 2024-04-26T01:47:42.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:42 smithi029 ceph-mon[32775]: pgmap v57: 97 pgs: 97 active+clean; 6.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 455 KiB/s wr, 59 op/s 2024-04-26T01:47:42.549 INFO:tasks.workunit.client.0.smithi006.stdout: 1 11721 13.37 MB/sec warmup 6 sec latency 19.291 ms 2024-04-26T01:47:42.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:42 smithi006 ceph-mon[28419]: pgmap v57: 97 pgs: 97 active+clean; 6.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 455 KiB/s wr, 59 op/s 2024-04-26T01:47:43.549 INFO:tasks.workunit.client.0.smithi006.stdout: 1 14204 13.68 MB/sec warmup 7 sec latency 15.273 ms 2024-04-26T01:47:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.993+0000 7f09ed600640 1 -- 172.21.15.6:0/4203601070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e8102f30 msgr2=0x7f09e8103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.993+0000 7f09ed600640 1 --2- 172.21.15.6:0/4203601070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e8102f30 0x7f09e8103310 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009900 tx=0x7f09d802a480 comp rx=0 tx=0).stop 2024-04-26T01:47:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.993+0000 7f09ed600640 1 -- 172.21.15.6:0/4203601070 shutdown_connections 2024-04-26T01:47:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.993+0000 7f09ed600640 1 --2- 172.21.15.6:0/4203601070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09e8103850 0x7f09e810bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.993+0000 7f09ed600640 1 --2- 172.21.15.6:0/4203601070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09e810c350 0x7f09e810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.993+0000 7f09ed600640 1 --2- 172.21.15.6:0/4203601070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e8102f30 0x7f09e8103310 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.993+0000 7f09ed600640 1 -- 172.21.15.6:0/4203601070 >> 172.21.15.6:0/4203601070 conn(0x7f09e80fc730 msgr2=0x7f09e80feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.993+0000 7f09ed600640 1 -- 172.21.15.6:0/4203601070 shutdown_connections 2024-04-26T01:47:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.993+0000 7f09ed600640 1 -- 172.21.15.6:0/4203601070 wait complete. 2024-04-26T01:47:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.994+0000 7f09ed600640 1 Processor -- start 2024-04-26T01:47:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.994+0000 7f09ed600640 1 -- start start 2024-04-26T01:47:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ed600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09e8102f30 0x7f09e81985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ed600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09e8103850 0x7f09e8198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ed600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e810c350 0x7f09e819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ed600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f09e8112010 con 0x7f09e8103850 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ed600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f09e8111e90 con 0x7f09e8102f30 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ed600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f09e8112190 con 0x7f09e810c350 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09e8103850 0x7f09e8198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ecc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e810c350 0x7f09e819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09e7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09e8102f30 0x7f09e81985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ecc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e810c350 0x7f09e819ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58602/0 (socket says 172.21.15.6:58602) 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ecc00640 1 -- 172.21.15.6:0/3312829814 learned_addr learned my addr 172.21.15.6:0/3312829814 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ecc00640 1 -- 172.21.15.6:0/3312829814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09e8102f30 msgr2=0x7f09e81985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ecc00640 1 --2- 172.21.15.6:0/3312829814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09e8102f30 0x7f09e81985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ecc00640 1 -- 172.21.15.6:0/3312829814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09e8103850 msgr2=0x7f09e8198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ecc00640 1 --2- 172.21.15.6:0/3312829814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09e8103850 0x7f09e8198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.995+0000 7f09ecc00640 1 -- 172.21.15.6:0/3312829814 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e819d550 con 0x7f09e810c350 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.996+0000 7f09ecc00640 1 --2- 172.21.15.6:0/3312829814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e810c350 0x7f09e819ce20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f09e000eb10 tx=0x7f09e000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.996+0000 7f09e5600640 1 -- 172.21.15.6:0/3312829814 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0004b20 con 0x7f09e810c350 2024-04-26T01:47:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.996+0000 7f09e5600640 1 -- 172.21.15.6:0/3312829814 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09e0004cc0 con 0x7f09e810c350 2024-04-26T01:47:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.996+0000 7f09e5600640 1 -- 172.21.15.6:0/3312829814 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0010670 con 0x7f09e810c350 2024-04-26T01:47:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.996+0000 7f09ed600640 1 -- 172.21.15.6:0/3312829814 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e819d840 con 0x7f09e810c350 2024-04-26T01:47:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:43.996+0000 7f09ed600640 1 -- 172.21.15.6:0/3312829814 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e81a50d0 con 0x7f09e810c350 2024-04-26T01:47:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.000+0000 7f09f1bd0740 1 Processor -- start 2024-04-26T01:47:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.000+0000 7f09e5600640 1 -- 172.21.15.6:0/3312829814 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f09e0005020 con 0x7f09e810c350 2024-04-26T01:47:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.000+0000 7f09f1bd0740 1 -- start start 2024-04-26T01:47:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.000+0000 7f09f1bd0740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909926dc0 0x5629099271a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.000+0000 7f09f1bd0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5629099276e0 0x562909931a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.000+0000 7f09f1bd0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909931fa0 0x562909934390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.000+0000 7f09f1bd0740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x562909888900 con 0x562909926dc0 2024-04-26T01:47:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.000+0000 7f09f1bd0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56290982eef0 con 0x5629099276e0 2024-04-26T01:47:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.000+0000 7f09f1bd0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5629098848c0 con 0x562909931fa0 2024-04-26T01:47:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.001+0000 7f09dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909926dc0 0x5629099271a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.001+0000 7f09dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909926dc0 0x5629099271a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35448/0 (socket says 172.21.15.6:35448) 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.001+0000 7f09dfe00640 1 -- 172.21.15.6:0/2974239943 learned_addr learned my addr 172.21.15.6:0/2974239943 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.001+0000 7f09e7e00640 1 --2- 172.21.15.6:0/2974239943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5629099276e0 0x562909931a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.001+0000 7f09dfe00640 1 --2- 172.21.15.6:0/2974239943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909931fa0 0x562909934390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.001+0000 7f09e7e00640 1 -- 172.21.15.6:0/2974239943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909931fa0 msgr2=0x562909934390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.001+0000 7f09e7e00640 1 --2- 172.21.15.6:0/2974239943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909931fa0 0x562909934390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.001+0000 7f09e7e00640 1 -- 172.21.15.6:0/2974239943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909926dc0 msgr2=0x5629099271a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.001+0000 7f09e7e00640 1 --2- 172.21.15.6:0/2974239943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909926dc0 0x5629099271a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.001+0000 7f09e7e00640 1 -- 172.21.15.6:0/2974239943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56290959b210 con 0x5629099276e0 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.001+0000 7f09e5600640 1 --2- 172.21.15.6:0/3312829814 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f09cc0a3d80 0x7f09cc0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.002+0000 7f09e7e00640 1 --2- 172.21.15.6:0/2974239943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5629099276e0 0x562909931a60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f09d802aa70 tx=0x7f09d8004190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=926424180f6d18a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.002+0000 7f09e5600640 1 -- 172.21.15.6:0/3312829814 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f09e00d0cd0 con 0x7f09e810c350 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.002+0000 7f09dfe00640 1 --2- 172.21.15.6:0/3312829814 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f09cc0a3d80 0x7f09cc0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.002+0000 7f09dd200640 1 -- 172.21.15.6:0/2974239943 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d8035040 con 0x5629099276e0 2024-04-26T01:47:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.002+0000 7f09dd200640 1 -- 172.21.15.6:0/2974239943 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09d80054f0 con 0x5629099276e0 2024-04-26T01:47:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.002+0000 7f09dd200640 1 -- 172.21.15.6:0/2974239943 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d80035f0 con 0x5629099276e0 2024-04-26T01:47:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.003+0000 7f09f1bd0740 1 -- 172.21.15.6:0/2974239943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5629099276e0 msgr2=0x562909931a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.003+0000 7f09dfe00640 1 --2- 172.21.15.6:0/3312829814 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f09cc0a3d80 0x7f09cc0a6240 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x5629099246f0 tx=0x7f09d000b5a0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.003+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/2974239943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5629099276e0 0x562909931a60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f09d802aa70 tx=0x7f09d8004190 comp rx=0 tx=0).stop 2024-04-26T01:47:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.003+0000 7f09f1bd0740 1 -- 172.21.15.6:0/2974239943 shutdown_connections 2024-04-26T01:47:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.003+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/2974239943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909931fa0 0x562909934390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.003+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/2974239943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5629099276e0 0x562909931a60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.003+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/2974239943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909926dc0 0x5629099271a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.003+0000 7f09f1bd0740 1 -- 172.21.15.6:0/2974239943 >> 172.21.15.6:0/2974239943 conn(0x5629099232d0 msgr2=0x5629099236b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.003+0000 7f09f1bd0740 1 -- 172.21.15.6:0/2974239943 shutdown_connections 2024-04-26T01:47:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.003+0000 7f09f1bd0740 1 -- 172.21.15.6:0/2974239943 wait complete. 2024-04-26T01:47:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.004+0000 7f09f1bd0740 1 -- start start 2024-04-26T01:47:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.004+0000 7f09f1bd0740 1 Processor -- start 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.004+0000 7f09f1bd0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909926dc0 0x56290992efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.004+0000 7f09f1bd0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5629099276e0 0x56290992f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.004+0000 7f09f1bd0740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 0x56290992fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.004+0000 7f09f1bd0740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x562909934b70 con 0x562909931fa0 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.004+0000 7f09f1bd0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562909842310 con 0x5629099276e0 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.004+0000 7f09f1bd0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562909934cf0 con 0x562909926dc0 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.004+0000 7f09f1bd0740 10 client.0 _ll_register_callbacks cb 0x562909810cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09e7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909926dc0 0x56290992efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09dfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5629099276e0 0x56290992f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09e7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909926dc0 0x56290992efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58608/0 (socket says 172.21.15.6:58608) 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09e7e00640 1 -- 172.21.15.6:0/1846404382 learned_addr learned my addr 172.21.15.6:0/1846404382 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09ecc00640 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 0x56290992fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09e7e00640 1 -- 172.21.15.6:0/1846404382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5629099276e0 msgr2=0x56290992f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09e7e00640 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5629099276e0 0x56290992f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09e7e00640 1 -- 172.21.15.6:0/1846404382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 msgr2=0x56290992fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09e7e00640 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 0x56290992fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09e7e00640 1 -- 172.21.15.6:0/1846404382 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562909667620 con 0x562909926dc0 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09ecc00640 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 0x56290992fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09dfe00640 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5629099276e0 0x56290992f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09e7e00640 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909926dc0 0x56290992efb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f09d802a4c0 tx=0x7f09d8004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ba1f90f3f45ff83 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.005+0000 7f09d5600640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.006+0000 7f09d5600640 1 -- 172.21.15.6:0/1846404382 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d8053060 con 0x562909926dc0 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.006+0000 7f09d5600640 1 -- 172.21.15.6:0/1846404382 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f09d8004e60 con 0x562909926dc0 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.006+0000 7f09f1bd0740 1 -- 172.21.15.6:0/1846404382 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09d80054f0 con 0x562909926dc0 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.006+0000 7f09d5600640 1 -- 172.21.15.6:0/1846404382 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f09d8035050 con 0x562909926dc0 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.006+0000 7f09f1bd0740 10 client.25444 fetch_fsmap learned FSMap version 28 2024-04-26T01:47:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.006+0000 7f09f1bd0740 1 -- 172.21.15.6:0/1846404382 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562909934b70 con 0x562909926dc0 2024-04-26T01:47:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.006+0000 7f09d5600640 1 -- 172.21.15.6:0/1846404382 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09d80054f0 con 0x562909926dc0 2024-04-26T01:47:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.007+0000 7f09d5600640 1 -- 172.21.15.6:0/1846404382 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f09d8003c40 con 0x562909926dc0 2024-04-26T01:47:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.007+0000 7f09f1bd0740 10 client.25444 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:47:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.007+0000 7f09f1bd0740 10 client.25444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.007+0000 7f09f1bd0740 20 client.25444 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.007+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562909941fe0 0x5629099443d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.007+0000 7f09f1bd0740 4 client.25444 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.007+0000 7f09f1bd0740 1 -- 172.21.15.6:0/1846404382 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56290989c8b0 con 0x562909941fe0 2024-04-26T01:47:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.007+0000 7f09dfe00640 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562909941fe0 0x5629099443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.008+0000 7f09dfe00640 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562909941fe0 0x5629099443d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75a3ee34e449fcf server_cookie=36c1b826753baa18 in_seq=0 out_seq=0 2024-04-26T01:47:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.008+0000 7f09d5600640 10 client.25444 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09d5600640 1 -- 172.21.15.6:0/1846404382 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56290989c8b0 con 0x562909941fe0 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09d5600640 10 client.25444 handle_command_reply: tid=0 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09f1bd0740 1 client.25444 shutdown 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09f1bd0740 1 -- 172.21.15.6:0/1846404382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909926dc0 msgr2=0x56290992efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909926dc0 0x56290992efb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f09d802a4c0 tx=0x7f09d8004770 comp rx=0 tx=0).stop 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09f1bd0740 1 -- 172.21.15.6:0/1846404382 shutdown_connections 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562909941fe0 0x5629099443d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 0x56290992fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09d5600640 0 client.25444 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5629099276e0 0x56290992f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/1846404382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909926dc0 0x56290992efb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09f1bd0740 1 -- 172.21.15.6:0/1846404382 >> 172.21.15.6:0/1846404382 conn(0x5629099232d0 msgr2=0x562909929050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09f1bd0740 1 -- 172.21.15.6:0/1846404382 shutdown_connections 2024-04-26T01:47:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.009+0000 7f09f1bd0740 1 -- 172.21.15.6:0/1846404382 wait complete. 2024-04-26T01:47:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.010+0000 7f09f1bd0740 20 client.25444 trim_cache size 0 max 16384 2024-04-26T01:47:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.026+0000 7f09f1bd0740 1 Processor -- start 2024-04-26T01:47:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.026+0000 7f09f1bd0740 1 -- start start 2024-04-26T01:47:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.026+0000 7f09f1bd0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562909926dc0 0x56290993da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09f1bd0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629099276e0 0x562909929780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09f1bd0740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 0x56290991dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09f1bd0740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f09e81ae140 con 0x562909931fa0 2024-04-26T01:47:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09f1bd0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f09e81aba10 con 0x562909926dc0 2024-04-26T01:47:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09f1bd0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f09e81a9860 con 0x5629099276e0 2024-04-26T01:47:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09ecc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 0x56290991dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09dfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629099276e0 0x562909929780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09dfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629099276e0 0x562909929780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58620/0 (socket says 172.21.15.6:58620) 2024-04-26T01:47:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09dfe00640 1 -- 172.21.15.6:0/2178606973 learned_addr learned my addr 172.21.15.6:0/2178606973 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09ecc00640 1 --2- 172.21.15.6:0/2178606973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 0x56290991dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35460/0 (socket says 172.21.15.6:35460) 2024-04-26T01:47:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09e7e00640 1 --2- 172.21.15.6:0/2178606973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562909926dc0 0x56290993da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09ecc00640 1 -- 172.21.15.6:0/2178606973 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629099276e0 msgr2=0x562909929780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09ecc00640 1 --2- 172.21.15.6:0/2178606973 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629099276e0 0x562909929780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09ecc00640 1 -- 172.21.15.6:0/2178606973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562909926dc0 msgr2=0x56290993da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09ecc00640 1 --2- 172.21.15.6:0/2178606973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562909926dc0 0x56290993da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09ecc00640 1 -- 172.21.15.6:0/2178606973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562909927d50 con 0x562909931fa0 2024-04-26T01:47:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.027+0000 7f09e7e00640 1 --2- 172.21.15.6:0/2178606973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562909926dc0 0x56290993da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.028+0000 7f09ecc00640 1 --2- 172.21.15.6:0/2178606973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 0x56290991dea0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f09e00ce3a0 tx=0x7f09e009ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff42176a9b5079c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.028+0000 7f09d7e00640 1 -- 172.21.15.6:0/2178606973 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e009cbf0 con 0x562909931fa0 2024-04-26T01:47:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.028+0000 7f09d7e00640 1 -- 172.21.15.6:0/2178606973 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09e00a0fa0 con 0x562909931fa0 2024-04-26T01:47:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.028+0000 7f09d7e00640 1 -- 172.21.15.6:0/2178606973 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e00bd0c0 con 0x562909931fa0 2024-04-26T01:47:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.029+0000 7f09f1bd0740 1 -- 172.21.15.6:0/2178606973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 msgr2=0x56290991dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.029+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/2178606973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 0x56290991dea0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f09e00ce3a0 tx=0x7f09e009ca80 comp rx=0 tx=0).stop 2024-04-26T01:47:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.029+0000 7f09f1bd0740 1 -- 172.21.15.6:0/2178606973 shutdown_connections 2024-04-26T01:47:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.029+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/2178606973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562909931fa0 0x56290991dea0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.029+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/2178606973 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5629099276e0 0x562909929780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.029+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/2178606973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562909926dc0 0x56290993da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.029+0000 7f09f1bd0740 1 -- 172.21.15.6:0/2178606973 >> 172.21.15.6:0/2178606973 conn(0x5629099232d0 msgr2=0x56290991d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.029+0000 7f09f1bd0740 1 -- 172.21.15.6:0/2178606973 shutdown_connections 2024-04-26T01:47:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.029+0000 7f09f1bd0740 1 -- 172.21.15.6:0/2178606973 wait complete. 2024-04-26T01:47:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.029+0000 7f09f1bd0740 1 -- start start 2024-04-26T01:47:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09f1bd0740 1 Processor -- start 2024-04-26T01:47:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09f1bd0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562909926dc0 0x56290993a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09f1bd0740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5629099276e0 0x562909937840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09f1bd0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909931fa0 0x562909937f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09f1bd0740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f09e81a9ed0 con 0x5629099276e0 2024-04-26T01:47:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09f1bd0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f09e81ab340 con 0x562909926dc0 2024-04-26T01:47:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09f1bd0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f09e81aa5a0 con 0x562909931fa0 2024-04-26T01:47:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09f1bd0740 10 client.0 _ll_register_callbacks cb 0x7f09d8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5629099276e0 0x562909937840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09ecc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909931fa0 0x562909937f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09e7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562909926dc0 0x56290993a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5629099276e0 0x562909937840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35472/0 (socket says 172.21.15.6:35472) 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09ecc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909931fa0 0x562909937f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58626/0 (socket says 172.21.15.6:58626) 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.030+0000 7f09dfe00640 1 -- 172.21.15.6:0/555769663 learned_addr learned my addr 172.21.15.6:0/555769663 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.031+0000 7f09ecc00640 1 -- 172.21.15.6:0/555769663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562909926dc0 msgr2=0x56290993a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.031+0000 7f09ecc00640 1 --2- 172.21.15.6:0/555769663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562909926dc0 0x56290993a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.031+0000 7f09ecc00640 1 -- 172.21.15.6:0/555769663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5629099276e0 msgr2=0x562909937840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.031+0000 7f09ecc00640 1 --2- 172.21.15.6:0/555769663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5629099276e0 0x562909937840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.031+0000 7f09ecc00640 1 -- 172.21.15.6:0/555769663 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629099a7370 con 0x562909931fa0 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.031+0000 7f09e7e00640 1 --2- 172.21.15.6:0/555769663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562909926dc0 0x56290993a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.031+0000 7f09ecc00640 1 --2- 172.21.15.6:0/555769663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909931fa0 0x562909937f30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f09e00ce3d0 tx=0x7f09e000c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b84b6d7aded660c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.031+0000 7f09d5600640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.031+0000 7f09d5600640 1 -- 172.21.15.6:0/555769663 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e00bb070 con 0x562909931fa0 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.031+0000 7f09d5600640 1 -- 172.21.15.6:0/555769663 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f09e00bd050 con 0x562909931fa0 2024-04-26T01:47:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.031+0000 7f09f1bd0740 1 -- 172.21.15.6:0/555769663 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09e00a0fa0 con 0x562909931fa0 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.032+0000 7f09d5600640 1 -- 172.21.15.6:0/555769663 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f09e00bd250 con 0x562909931fa0 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.032+0000 7f09f1bd0740 10 client.25450 fetch_fsmap learned FSMap version 28 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.032+0000 7f09f1bd0740 1 -- 172.21.15.6:0/555769663 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5629099a8520 con 0x562909931fa0 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.032+0000 7f09d5600640 1 -- 172.21.15.6:0/555769663 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09e00a0fa0 con 0x562909931fa0 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.032+0000 7f09d5600640 1 -- 172.21.15.6:0/555769663 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f09e00ceaf0 con 0x562909931fa0 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.032+0000 7f09f1bd0740 10 client.25450 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.032+0000 7f09f1bd0740 10 client.25450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.032+0000 7f09f1bd0740 20 client.25450 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.032+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/555769663 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5629099a8ae0 0x5629099aaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.032+0000 7f09f1bd0740 4 client.25450 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.032+0000 7f09f1bd0740 1 -- 172.21.15.6:0/555769663 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562909896320 con 0x5629099a8ae0 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.032+0000 7f09e7e00640 1 --2- 172.21.15.6:0/555769663 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5629099a8ae0 0x5629099aaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.033+0000 7f09e7e00640 1 --2- 172.21.15.6:0/555769663 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5629099a8ae0 0x5629099aaed0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=198555709b43e17f server_cookie=af8c007a62b9942b in_seq=0 out_seq=0 2024-04-26T01:47:44.035 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.033+0000 7f09d5600640 10 client.25450 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:44.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.158+0000 7f09d5600640 1 -- 172.21.15.6:0/555769663 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562909896320 con 0x5629099a8ae0 2024-04-26T01:47:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.158+0000 7f09d5600640 10 client.25450 handle_command_reply: tid=0 2024-04-26T01:47:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.158+0000 7f09f1bd0740 1 client.25450 shutdown 2024-04-26T01:47:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.159+0000 7f09f1bd0740 1 -- 172.21.15.6:0/555769663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909931fa0 msgr2=0x562909937f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.159+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/555769663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909931fa0 0x562909937f30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f09e00ce3d0 tx=0x7f09e000c950 comp rx=0 tx=0).stop 2024-04-26T01:47:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.159+0000 7f09f1bd0740 1 -- 172.21.15.6:0/555769663 shutdown_connections 2024-04-26T01:47:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.159+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/555769663 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5629099a8ae0 0x5629099aaed0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.159+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/555769663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5629099276e0 0x562909937840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.159+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/555769663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562909931fa0 0x562909937f30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.159+0000 7f09d5600640 0 client.25450 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.159+0000 7f09f1bd0740 1 --2- 172.21.15.6:0/555769663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562909926dc0 0x56290993a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.159+0000 7f09f1bd0740 1 -- 172.21.15.6:0/555769663 >> 172.21.15.6:0/555769663 conn(0x5629099232d0 msgr2=0x56290992eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.159+0000 7f09f1bd0740 1 -- 172.21.15.6:0/555769663 shutdown_connections 2024-04-26T01:47:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.159+0000 7f09f1bd0740 1 -- 172.21.15.6:0/555769663 wait complete. 2024-04-26T01:47:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.159+0000 7f09f1bd0740 20 client.25450 trim_cache size 0 max 16384 2024-04-26T01:47:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:47:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:47:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:47:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:47:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.160+0000 7f09ed600640 1 -- 172.21.15.6:0/3312829814 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f09cc0a3d80 msgr2=0x7f09cc0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.160+0000 7f09ed600640 1 --2- 172.21.15.6:0/3312829814 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f09cc0a3d80 0x7f09cc0a6240 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x5629099246f0 tx=0x7f09d000b5a0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.160+0000 7f09ed600640 1 -- 172.21.15.6:0/3312829814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e810c350 msgr2=0x7f09e819ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.160+0000 7f09ed600640 1 --2- 172.21.15.6:0/3312829814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e810c350 0x7f09e819ce20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f09e000eb10 tx=0x7f09e000c2d0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.160+0000 7f09ed600640 1 -- 172.21.15.6:0/3312829814 shutdown_connections 2024-04-26T01:47:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.160+0000 7f09ed600640 1 --2- 172.21.15.6:0/3312829814 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f09cc0a3d80 0x7f09cc0a6240 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.160+0000 7f09ed600640 1 --2- 172.21.15.6:0/3312829814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09e8103850 0x7f09e8198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.160+0000 7f09ed600640 1 --2- 172.21.15.6:0/3312829814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09e810c350 0x7f09e819ce20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.160+0000 7f09ed600640 1 --2- 172.21.15.6:0/3312829814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09e8102f30 0x7f09e81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.160+0000 7f09ed600640 1 -- 172.21.15.6:0/3312829814 >> 172.21.15.6:0/3312829814 conn(0x7f09e80fc730 msgr2=0x7f09e80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.160+0000 7f09ed600640 1 -- 172.21.15.6:0/3312829814 shutdown_connections 2024-04-26T01:47:44.161 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:44.160+0000 7f09ed600640 1 -- 172.21.15.6:0/3312829814 wait complete. 2024-04-26T01:47:44.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:44 smithi188 ceph-mon[32751]: pgmap v58: 97 pgs: 97 active+clean; 6.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 455 KiB/s wr, 56 op/s 2024-04-26T01:47:44.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:44 smithi029 ceph-mon[32775]: pgmap v58: 97 pgs: 97 active+clean; 6.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 455 KiB/s wr, 56 op/s 2024-04-26T01:47:44.550 INFO:tasks.workunit.client.0.smithi006.stdout: 1 15754 12.82 MB/sec warmup 8 sec latency 26.973 ms 2024-04-26T01:47:44.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:44 smithi006 ceph-mon[28419]: pgmap v58: 97 pgs: 97 active+clean; 6.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 455 KiB/s wr, 56 op/s 2024-04-26T01:47:45.550 INFO:tasks.workunit.client.0.smithi006.stdout: 1 18110 12.95 MB/sec warmup 9 sec latency 8.330 ms 2024-04-26T01:47:46.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:46 smithi188 ceph-mon[32751]: pgmap v59: 97 pgs: 97 active+clean; 6.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 455 KiB/s wr, 56 op/s 2024-04-26T01:47:46.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:46 smithi029 ceph-mon[32775]: pgmap v59: 97 pgs: 97 active+clean; 6.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 455 KiB/s wr, 56 op/s 2024-04-26T01:47:46.550 INFO:tasks.workunit.client.0.smithi006.stdout: 1 20181 12.36 MB/sec warmup 10 sec latency 11.920 ms 2024-04-26T01:47:46.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:46 smithi006 ceph-mon[28419]: pgmap v59: 97 pgs: 97 active+clean; 6.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 455 KiB/s wr, 56 op/s 2024-04-26T01:47:47.550 INFO:tasks.workunit.client.0.smithi006.stdout: 1 21991 12.35 MB/sec warmup 11 sec latency 12.660 ms 2024-04-26T01:47:48.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:48 smithi188 ceph-mon[32751]: pgmap v60: 97 pgs: 97 active+clean; 6.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 981 KiB/s wr, 192 op/s 2024-04-26T01:47:48.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:48 smithi029 ceph-mon[32775]: pgmap v60: 97 pgs: 97 active+clean; 6.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 981 KiB/s wr, 192 op/s 2024-04-26T01:47:48.550 INFO:tasks.workunit.client.0.smithi006.stdout: 1 24346 12.36 MB/sec warmup 12 sec latency 9.305 ms 2024-04-26T01:47:48.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:48 smithi006 ceph-mon[28419]: pgmap v60: 97 pgs: 97 active+clean; 6.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 981 KiB/s wr, 192 op/s 2024-04-26T01:47:49.550 INFO:tasks.workunit.client.0.smithi006.stdout: 1 25913 12.06 MB/sec warmup 13 sec latency 12.163 ms 2024-04-26T01:47:50.369 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:50 smithi006 ceph-mon[28419]: pgmap v61: 97 pgs: 97 active+clean; 6.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 980 KiB/s wr, 189 op/s 2024-04-26T01:47:50.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:50 smithi188 ceph-mon[32751]: pgmap v61: 97 pgs: 97 active+clean; 6.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 980 KiB/s wr, 189 op/s 2024-04-26T01:47:50.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:50 smithi029 ceph-mon[32775]: pgmap v61: 97 pgs: 97 active+clean; 6.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 980 KiB/s wr, 189 op/s 2024-04-26T01:47:50.550 INFO:tasks.workunit.client.0.smithi006.stdout: 1 28505 12.32 MB/sec warmup 14 sec latency 15.155 ms 2024-04-26T01:47:51.550 INFO:tasks.workunit.client.0.smithi006.stdout: 1 29870 11.93 MB/sec warmup 15 sec latency 14.109 ms 2024-04-26T01:47:52.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:52 smithi188 ceph-mon[32751]: pgmap v62: 97 pgs: 97 active+clean; 11 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.4 MiB/s wr, 249 op/s 2024-04-26T01:47:52.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:52 smithi029 ceph-mon[32775]: pgmap v62: 97 pgs: 97 active+clean; 11 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.4 MiB/s wr, 249 op/s 2024-04-26T01:47:52.550 INFO:tasks.workunit.client.0.smithi006.stdout: 1 32286 12.05 MB/sec warmup 16 sec latency 8.576 ms 2024-04-26T01:47:52.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:52 smithi006 ceph-mon[28419]: pgmap v62: 97 pgs: 97 active+clean; 11 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.4 MiB/s wr, 249 op/s 2024-04-26T01:47:53.551 INFO:tasks.workunit.client.0.smithi006.stdout: 1 34382 11.78 MB/sec warmup 17 sec latency 11.934 ms 2024-04-26T01:47:54.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:54 smithi188 ceph-mon[32751]: pgmap v63: 97 pgs: 97 active+clean; 11 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 998 KiB/s wr, 196 op/s 2024-04-26T01:47:54.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:54 smithi029 ceph-mon[32775]: pgmap v63: 97 pgs: 97 active+clean; 11 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 998 KiB/s wr, 196 op/s 2024-04-26T01:47:54.551 INFO:tasks.workunit.client.0.smithi006.stdout: 1 36099 11.79 MB/sec warmup 18 sec latency 18.104 ms 2024-04-26T01:47:54.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:54 smithi006 ceph-mon[28419]: pgmap v63: 97 pgs: 97 active+clean; 11 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 998 KiB/s wr, 196 op/s 2024-04-26T01:47:55.551 INFO:tasks.workunit.client.0.smithi006.stdout: 1 38111 11.59 MB/sec warmup 19 sec latency 27.862 ms 2024-04-26T01:47:56.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:56 smithi188 ceph-mon[32751]: pgmap v64: 97 pgs: 97 active+clean; 11 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 998 KiB/s wr, 196 op/s 2024-04-26T01:47:56.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:56 smithi029 ceph-mon[32775]: pgmap v64: 97 pgs: 97 active+clean; 11 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 998 KiB/s wr, 196 op/s 2024-04-26T01:47:56.551 INFO:tasks.workunit.client.0.smithi006.stdout: 1 39722 11.57 MB/sec warmup 20 sec latency 17.827 ms 2024-04-26T01:47:56.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:56 smithi006 ceph-mon[28419]: pgmap v64: 97 pgs: 97 active+clean; 11 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 998 KiB/s wr, 196 op/s 2024-04-26T01:47:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.326+0000 7fc840a00640 1 -- 172.21.15.6:0/3591757532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc83c101030 msgr2=0x7fc83c10ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.326+0000 7fc840a00640 1 --2- 172.21.15.6:0/3591757532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc83c101030 0x7fc83c10ec10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc834009cd0 tx=0x7fc83402a8d0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.326+0000 7fc840a00640 1 -- 172.21.15.6:0/3591757532 shutdown_connections 2024-04-26T01:47:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.326+0000 7fc840a00640 1 --2- 172.21.15.6:0/3591757532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc83c100670 0x7fc83c100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.326+0000 7fc840a00640 1 --2- 172.21.15.6:0/3591757532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc83c101030 0x7fc83c10ec10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.326+0000 7fc840a00640 1 --2- 172.21.15.6:0/3591757532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c107f00 0x7fc83c1082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.326+0000 7fc840a00640 1 -- 172.21.15.6:0/3591757532 >> 172.21.15.6:0/3591757532 conn(0x7fc83c075fc0 msgr2=0x7fc83c0763d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.326+0000 7fc840a00640 1 -- 172.21.15.6:0/3591757532 shutdown_connections 2024-04-26T01:47:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.327+0000 7fc840a00640 1 -- 172.21.15.6:0/3591757532 wait complete. 2024-04-26T01:47:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.328+0000 7fc840a00640 1 Processor -- start 2024-04-26T01:47:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.328+0000 7fc840a00640 1 -- start start 2024-04-26T01:47:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.328+0000 7fc840a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc83c100670 0x7fc83c101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.328+0000 7fc840a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc83c101030 0x7fc83c101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.328+0000 7fc840a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c107f00 0x7fc83c1022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.328+0000 7fc840a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1120c0 con 0x7fc83c101030 2024-04-26T01:47:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.328+0000 7fc840a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c111f40 con 0x7fc83c100670 2024-04-26T01:47:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.328+0000 7fc840a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c112240 con 0x7fc83c107f00 2024-04-26T01:47:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.328+0000 7fc83be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c107f00 0x7fc83c1022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.328+0000 7fc83be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c107f00 0x7fc83c1022d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:53344/0 (socket says 172.21.15.6:53344) 2024-04-26T01:47:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.328+0000 7fc83be00640 1 -- 172.21.15.6:0/2412938860 learned_addr learned my addr 172.21.15.6:0/2412938860 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.329+0000 7fc83b400640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc83c100670 0x7fc83c101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.329+0000 7fc83aa00640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc83c101030 0x7fc83c101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.329+0000 7fc83be00640 1 -- 172.21.15.6:0/2412938860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc83c100670 msgr2=0x7fc83c101850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.329+0000 7fc83be00640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc83c100670 0x7fc83c101850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.329+0000 7fc83be00640 1 -- 172.21.15.6:0/2412938860 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc83c101030 msgr2=0x7fc83c101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.329+0000 7fc83be00640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc83c101030 0x7fc83c101d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.329+0000 7fc83be00640 1 -- 172.21.15.6:0/2412938860 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c1a6ac0 con 0x7fc83c107f00 2024-04-26T01:47:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.329+0000 7fc83b400640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc83c100670 0x7fc83c101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.329+0000 7fc83be00640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c107f00 0x7fc83c1022d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc83402adf0 tx=0x7fc834032c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.330+0000 7fc833e00640 1 -- 172.21.15.6:0/2412938860 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834032de0 con 0x7fc83c107f00 2024-04-26T01:47:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.330+0000 7fc833e00640 1 -- 172.21.15.6:0/2412938860 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc834003e20 con 0x7fc83c107f00 2024-04-26T01:47:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.330+0000 7fc840a00640 1 -- 172.21.15.6:0/2412938860 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83c1a6d50 con 0x7fc83c107f00 2024-04-26T01:47:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.330+0000 7fc833e00640 1 -- 172.21.15.6:0/2412938860 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8340305f0 con 0x7fc83c107f00 2024-04-26T01:47:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.330+0000 7fc840a00640 1 -- 172.21.15.6:0/2412938860 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc83c1a72a0 con 0x7fc83c107f00 2024-04-26T01:47:57.331 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.333+0000 7fc833e00640 1 -- 172.21.15.6:0/2412938860 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fc834003970 con 0x7fc83c107f00 2024-04-26T01:47:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.333+0000 7fc844b22740 1 Processor -- start 2024-04-26T01:47:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.334+0000 7fc844b22740 1 -- start start 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.334+0000 7fc844b22740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861cf4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.334+0000 7fc844b22740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 0x562861cff650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.334+0000 7fc844b22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861d01f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.334+0000 7fc844b22740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x562861c5a900 con 0x562861cf52d0 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.334+0000 7fc844b22740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562861c00ef0 con 0x562861cf49b0 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.334+0000 7fc844b22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562861c568c0 con 0x562861cffb90 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.334+0000 7fc83aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861cf4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.334+0000 7fc83aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861cf4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:46786/0 (socket says 172.21.15.6:46786) 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.334+0000 7fc83aa00640 1 -- 172.21.15.6:0/4091976341 learned_addr learned my addr 172.21.15.6:0/4091976341 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.334+0000 7fc83b400640 1 --2- 172.21.15.6:0/4091976341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 0x562861cff650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.334+0000 7fc83aa00640 1 --2- 172.21.15.6:0/4091976341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861d01f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.335+0000 7fc83aa00640 1 -- 172.21.15.6:0/4091976341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 msgr2=0x562861d01f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.335+0000 7fc83aa00640 1 --2- 172.21.15.6:0/4091976341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861d01f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.335+0000 7fc83aa00640 1 -- 172.21.15.6:0/4091976341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 msgr2=0x562861cff650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.335+0000 7fc83aa00640 1 --2- 172.21.15.6:0/4091976341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 0x562861cff650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.335+0000 7fc83aa00640 1 -- 172.21.15.6:0/4091976341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56286196d210 con 0x562861cf49b0 2024-04-26T01:47:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.335+0000 7fc83aa00640 1 --2- 172.21.15.6:0/4091976341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861cf4d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc828009fd0 tx=0x7fc828005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a5dd4f10318338b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.335+0000 7fc833e00640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc8200a3d00 0x7fc8200a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.335+0000 7fc827e00640 1 -- 172.21.15.6:0/4091976341 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc828004050 con 0x562861cf49b0 2024-04-26T01:47:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.336+0000 7fc827e00640 1 -- 172.21.15.6:0/4091976341 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8280054d0 con 0x562861cf49b0 2024-04-26T01:47:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.336+0000 7fc827e00640 1 -- 172.21.15.6:0/4091976341 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc828004ba0 con 0x562861cf49b0 2024-04-26T01:47:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.336+0000 7fc833e00640 1 -- 172.21.15.6:0/2412938860 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7fc8340f1570 con 0x7fc83c107f00 2024-04-26T01:47:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.336+0000 7fc83b400640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc8200a3d00 0x7fc8200a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.337+0000 7fc83b400640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc8200a3d00 0x7fc8200a61c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x562861cf5f50 tx=0x7fc82c005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:57.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.337+0000 7fc844b22740 1 -- 172.21.15.6:0/4091976341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 msgr2=0x562861cf4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:57.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.337+0000 7fc844b22740 1 --2- 172.21.15.6:0/4091976341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861cf4d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc828009fd0 tx=0x7fc828005850 comp rx=0 tx=0).stop 2024-04-26T01:47:57.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.337+0000 7fc844b22740 1 -- 172.21.15.6:0/4091976341 shutdown_connections 2024-04-26T01:47:57.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.337+0000 7fc844b22740 1 --2- 172.21.15.6:0/4091976341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 0x562861cff650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.337+0000 7fc844b22740 1 --2- 172.21.15.6:0/4091976341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861d01f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.337+0000 7fc844b22740 1 --2- 172.21.15.6:0/4091976341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861cf4d90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.337+0000 7fc844b22740 1 -- 172.21.15.6:0/4091976341 >> 172.21.15.6:0/4091976341 conn(0x562861cf0ec0 msgr2=0x562861cf12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:57.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.337+0000 7fc844b22740 1 -- 172.21.15.6:0/4091976341 shutdown_connections 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.337+0000 7fc844b22740 1 -- 172.21.15.6:0/4091976341 wait complete. 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.337+0000 7fc844b22740 1 -- start start 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.338+0000 7fc844b22740 1 Processor -- start 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.338+0000 7fc844b22740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861cfcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.338+0000 7fc844b22740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 0x562861cfd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.338+0000 7fc844b22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861cfd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.338+0000 7fc844b22740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x562861d027c0 con 0x562861cf52d0 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.338+0000 7fc844b22740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562861c14310 con 0x562861cf49b0 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.338+0000 7fc844b22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562861d02940 con 0x562861cffb90 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.338+0000 7fc844b22740 10 client.0 _ll_register_callbacks cb 0x562861be2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc83b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861cfcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc83b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861cfcb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:46794/0 (socket says 172.21.15.6:46794) 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc83aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 0x562861cfd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc83b400640 1 -- 172.21.15.6:0/3244666498 learned_addr learned my addr 172.21.15.6:0/3244666498 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc83be00640 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861cfd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc83b400640 1 -- 172.21.15.6:0/3244666498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 msgr2=0x562861cfd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc83b400640 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861cfd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc83b400640 1 -- 172.21.15.6:0/3244666498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 msgr2=0x562861cfd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc83b400640 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 0x562861cfd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc83b400640 1 -- 172.21.15.6:0/3244666498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562861a39620 con 0x562861cf49b0 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc83be00640 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861cfd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc83b400640 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861cfcb80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00fb30 tx=0x7fc82c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8b6f5984493765e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.339+0000 7fc824c00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.340+0000 7fc824c00640 1 -- 172.21.15.6:0/3244666498 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c02e060 con 0x562861cf49b0 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.340+0000 7fc824c00640 1 -- 172.21.15.6:0/3244666498 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7fc82c018030 con 0x562861cf49b0 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.340+0000 7fc844b22740 1 -- 172.21.15.6:0/3244666498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8280054d0 con 0x562861cf49b0 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.340+0000 7fc824c00640 1 -- 172.21.15.6:0/3244666498 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc82c01e650 con 0x562861cf49b0 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.340+0000 7fc844b22740 10 client.25415 fetch_fsmap learned FSMap version 28 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.340+0000 7fc844b22740 1 -- 172.21.15.6:0/3244666498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562861d0eb30 con 0x562861cf49b0 2024-04-26T01:47:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.340+0000 7fc824c00640 1 -- 172.21.15.6:0/3244666498 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8280054d0 con 0x562861cf49b0 2024-04-26T01:47:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.340+0000 7fc824c00640 1 -- 172.21.15.6:0/3244666498 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7fc82c031900 con 0x562861cf49b0 2024-04-26T01:47:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.340+0000 7fc844b22740 10 client.25415 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:47:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.341+0000 7fc844b22740 10 client.25415 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.341+0000 7fc844b22740 20 client.25415 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.341+0000 7fc844b22740 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562861d10060 0x562861d12470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.341+0000 7fc844b22740 4 client.25415 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.341+0000 7fc844b22740 1 -- 172.21.15.6:0/3244666498 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562861c6e8b0 con 0x562861d10060 2024-04-26T01:47:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.341+0000 7fc83aa00640 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562861d10060 0x562861d12470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.341+0000 7fc83aa00640 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562861d10060 0x562861d12470 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=629fcdec4892f7a0 server_cookie=ad13d0cbab953c0c in_seq=0 out_seq=0 2024-04-26T01:47:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.341+0000 7fc824c00640 10 client.25415 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.342+0000 7fc824c00640 1 -- 172.21.15.6:0/3244666498 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562861c6e8b0 con 0x562861d10060 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.342+0000 7fc824c00640 10 client.25415 handle_command_reply: tid=0 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.342+0000 7fc844b22740 1 client.25415 shutdown 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.343+0000 7fc844b22740 1 -- 172.21.15.6:0/3244666498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 msgr2=0x562861cfcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.343+0000 7fc844b22740 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861cfcb80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00fb30 tx=0x7fc82c004730 comp rx=0 tx=0).stop 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.343+0000 7fc844b22740 1 -- 172.21.15.6:0/3244666498 shutdown_connections 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.343+0000 7fc844b22740 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562861d10060 0x562861d12470 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.343+0000 7fc844b22740 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 0x562861cfd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.343+0000 7fc844b22740 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861cfd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.343+0000 7fc844b22740 1 --2- 172.21.15.6:0/3244666498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861cfcb80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.343+0000 7fc824c00640 0 client.25415 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.343+0000 7fc844b22740 1 -- 172.21.15.6:0/3244666498 >> 172.21.15.6:0/3244666498 conn(0x562861cf0ec0 msgr2=0x562861cf6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.343+0000 7fc844b22740 1 -- 172.21.15.6:0/3244666498 shutdown_connections 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.343+0000 7fc844b22740 1 -- 172.21.15.6:0/3244666498 wait complete. 2024-04-26T01:47:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.343+0000 7fc844b22740 20 client.25415 trim_cache size 0 max 16384 2024-04-26T01:47:57.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc844b22740 1 Processor -- start 2024-04-26T01:47:57.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc844b22740 1 -- start start 2024-04-26T01:47:57.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc844b22740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf49b0 0x562861d0b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc844b22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cf52d0 0x562861cf7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc844b22740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cffb90 0x562861cefea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc844b22740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1b2480 con 0x562861cf49b0 2024-04-26T01:47:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc844b22740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1afd50 con 0x562861cffb90 2024-04-26T01:47:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc844b22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1adba0 con 0x562861cf52d0 2024-04-26T01:47:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc83be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cffb90 0x562861cefea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc83aa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cf52d0 0x562861cf7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc83be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cffb90 0x562861cefea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:46806/0 (socket says 172.21.15.6:46806) 2024-04-26T01:47:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc83be00640 1 -- 172.21.15.6:0/746213805 learned_addr learned my addr 172.21.15.6:0/746213805 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.364+0000 7fc83aa00640 1 --2- 172.21.15.6:0/746213805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cf52d0 0x562861cf7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:53358/0 (socket says 172.21.15.6:53358) 2024-04-26T01:47:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.365+0000 7fc83b400640 1 --2- 172.21.15.6:0/746213805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf49b0 0x562861d0b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.365+0000 7fc83be00640 1 -- 172.21.15.6:0/746213805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cf52d0 msgr2=0x562861cf7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.365+0000 7fc83be00640 1 --2- 172.21.15.6:0/746213805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cf52d0 0x562861cf7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.365+0000 7fc83be00640 1 -- 172.21.15.6:0/746213805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf49b0 msgr2=0x562861d0b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.365+0000 7fc83be00640 1 --2- 172.21.15.6:0/746213805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf49b0 0x562861d0b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.365+0000 7fc83be00640 1 -- 172.21.15.6:0/746213805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562861d01f80 con 0x562861cffb90 2024-04-26T01:47:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.365+0000 7fc83be00640 1 --2- 172.21.15.6:0/746213805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cffb90 0x562861cefea0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc8340bd530 tx=0x7fc834032f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7addd113061e371 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:57.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.365+0000 7fc827400640 1 -- 172.21.15.6:0/746213805 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8340bdca0 con 0x562861cffb90 2024-04-26T01:47:57.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.365+0000 7fc827400640 1 -- 172.21.15.6:0/746213805 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc834036030 con 0x562861cffb90 2024-04-26T01:47:57.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.365+0000 7fc827400640 1 -- 172.21.15.6:0/746213805 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8340b9c10 con 0x562861cffb90 2024-04-26T01:47:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.366+0000 7fc844b22740 1 -- 172.21.15.6:0/746213805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cffb90 msgr2=0x562861cefea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.366+0000 7fc844b22740 1 --2- 172.21.15.6:0/746213805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cffb90 0x562861cefea0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc8340bd530 tx=0x7fc834032f80 comp rx=0 tx=0).stop 2024-04-26T01:47:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.366+0000 7fc844b22740 1 -- 172.21.15.6:0/746213805 shutdown_connections 2024-04-26T01:47:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.366+0000 7fc844b22740 1 --2- 172.21.15.6:0/746213805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cffb90 0x562861cefea0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.366+0000 7fc844b22740 1 --2- 172.21.15.6:0/746213805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cf52d0 0x562861cf7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.366+0000 7fc844b22740 1 --2- 172.21.15.6:0/746213805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf49b0 0x562861d0b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.366+0000 7fc844b22740 1 -- 172.21.15.6:0/746213805 >> 172.21.15.6:0/746213805 conn(0x562861cf0ec0 msgr2=0x562861cef790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.366+0000 7fc844b22740 1 -- 172.21.15.6:0/746213805 shutdown_connections 2024-04-26T01:47:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.366+0000 7fc844b22740 1 -- 172.21.15.6:0/746213805 wait complete. 2024-04-26T01:47:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.367+0000 7fc844b22740 1 -- start start 2024-04-26T01:47:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.367+0000 7fc844b22740 1 Processor -- start 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.367+0000 7fc844b22740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861d6e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.367+0000 7fc844b22740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 0x562861d6eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.367+0000 7fc844b22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861d04f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.367+0000 7fc844b22740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1ae210 con 0x562861cf52d0 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.367+0000 7fc844b22740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1af680 con 0x562861cf49b0 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.367+0000 7fc844b22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1ae8e0 con 0x562861cffb90 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.368+0000 7fc844b22740 10 client.0 _ll_register_callbacks cb 0x7fc82c00f9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.368+0000 7fc83be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861d04f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.368+0000 7fc83be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861d04f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:53368/0 (socket says 172.21.15.6:53368) 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.368+0000 7fc83be00640 1 -- 172.21.15.6:0/1665116068 learned_addr learned my addr 172.21.15.6:0/1665116068 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.368+0000 7fc83b400640 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861d6e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.368+0000 7fc83aa00640 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 0x562861d6eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.368+0000 7fc83be00640 1 -- 172.21.15.6:0/1665116068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 msgr2=0x562861d6e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.368+0000 7fc83be00640 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861d6e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.368+0000 7fc83be00640 1 -- 172.21.15.6:0/1665116068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 msgr2=0x562861d6eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.368+0000 7fc83be00640 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 0x562861d6eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.368+0000 7fc83be00640 1 -- 172.21.15.6:0/1665116068 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562861d05650 con 0x562861cffb90 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.369+0000 7fc83be00640 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861d04f90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc8340b9a00 tx=0x7fc834009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e82d0e472b5835d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.369+0000 7fc824c00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.369+0000 7fc824c00640 1 -- 172.21.15.6:0/1665116068 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8340a2060 con 0x562861cffb90 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.369+0000 7fc824c00640 1 -- 172.21.15.6:0/1665116068 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7fc8340dc2c0 con 0x562861cffb90 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.369+0000 7fc844b22740 1 -- 172.21.15.6:0/1665116068 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc834036030 con 0x562861cffb90 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.369+0000 7fc824c00640 1 -- 172.21.15.6:0/1665116068 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc8340bdca0 con 0x562861cffb90 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.369+0000 7fc844b22740 10 client.25468 fetch_fsmap learned FSMap version 28 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.369+0000 7fc844b22740 1 -- 172.21.15.6:0/1665116068 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562861d76180 con 0x562861cffb90 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.369+0000 7fc824c00640 1 -- 172.21.15.6:0/1665116068 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc834036030 con 0x562861cffb90 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.370+0000 7fc824c00640 1 -- 172.21.15.6:0/1665116068 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7fc8340c5d80 con 0x562861cffb90 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.370+0000 7fc844b22740 10 client.25468 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.370+0000 7fc844b22740 10 client.25468 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:47:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.370+0000 7fc844b22740 20 client.25468 populate_metadata read hostname 'smithi006' 2024-04-26T01:47:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.370+0000 7fc844b22740 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562861d76790 0x562861d78b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.370+0000 7fc844b22740 4 client.25468 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:47:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.370+0000 7fc844b22740 1 -- 172.21.15.6:0/1665116068 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562861cf8780 con 0x562861d76790 2024-04-26T01:47:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.370+0000 7fc83b400640 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562861d76790 0x562861d78b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.371+0000 7fc83b400640 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562861d76790 0x562861d78b80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a62c710546b0f216 server_cookie=1d33228b9b760013 in_seq=0 out_seq=0 2024-04-26T01:47:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.371+0000 7fc824c00640 10 client.25468 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.492+0000 7fc824c00640 1 -- 172.21.15.6:0/1665116068 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562861cf8780 con 0x562861d76790 2024-04-26T01:47:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.492+0000 7fc824c00640 10 client.25468 handle_command_reply: tid=0 2024-04-26T01:47:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.492+0000 7fc844b22740 1 client.25468 shutdown 2024-04-26T01:47:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.493+0000 7fc844b22740 1 -- 172.21.15.6:0/1665116068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 msgr2=0x562861d04f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.493+0000 7fc844b22740 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861d04f90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc8340b9a00 tx=0x7fc834009a20 comp rx=0 tx=0).stop 2024-04-26T01:47:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.493+0000 7fc844b22740 1 -- 172.21.15.6:0/1665116068 shutdown_connections 2024-04-26T01:47:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.493+0000 7fc844b22740 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562861d76790 0x562861d78b80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.493+0000 7fc844b22740 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562861cf52d0 0x562861d6eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.493+0000 7fc844b22740 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562861cffb90 0x562861d04f90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.493+0000 7fc824c00640 0 client.25468 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:47:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.493+0000 7fc844b22740 1 --2- 172.21.15.6:0/1665116068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562861cf49b0 0x562861d6e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.493+0000 7fc844b22740 1 -- 172.21.15.6:0/1665116068 >> 172.21.15.6:0/1665116068 conn(0x562861cf0ec0 msgr2=0x562861cef790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.493+0000 7fc844b22740 1 -- 172.21.15.6:0/1665116068 shutdown_connections 2024-04-26T01:47:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.493+0000 7fc844b22740 1 -- 172.21.15.6:0/1665116068 wait complete. 2024-04-26T01:47:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.493+0000 7fc844b22740 20 client.25468 trim_cache size 0 max 16384 2024-04-26T01:47:57.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:47:57.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:47:57.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:47:57.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:47:57.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.494+0000 7fc840a00640 1 -- 172.21.15.6:0/2412938860 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc8200a3d00 msgr2=0x7fc8200a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.494+0000 7fc840a00640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc8200a3d00 0x7fc8200a61c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x562861cf5f50 tx=0x7fc82c005f90 comp rx=0 tx=0).stop 2024-04-26T01:47:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.494+0000 7fc840a00640 1 -- 172.21.15.6:0/2412938860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c107f00 msgr2=0x7fc83c1022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.494+0000 7fc840a00640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c107f00 0x7fc83c1022d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc83402adf0 tx=0x7fc834032c70 comp rx=0 tx=0).stop 2024-04-26T01:47:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.494+0000 7fc840a00640 1 -- 172.21.15.6:0/2412938860 shutdown_connections 2024-04-26T01:47:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.494+0000 7fc840a00640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc8200a3d00 0x7fc8200a61c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.494+0000 7fc840a00640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc83c101030 0x7fc83c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.494+0000 7fc840a00640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc83c107f00 0x7fc83c1022d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.494+0000 7fc840a00640 1 --2- 172.21.15.6:0/2412938860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc83c100670 0x7fc83c101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.494+0000 7fc840a00640 1 -- 172.21.15.6:0/2412938860 >> 172.21.15.6:0/2412938860 conn(0x7fc83c075fc0 msgr2=0x7fc83c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.495+0000 7fc840a00640 1 -- 172.21.15.6:0/2412938860 shutdown_connections 2024-04-26T01:47:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:47:57.495+0000 7fc840a00640 1 -- 172.21.15.6:0/2412938860 wait complete. 2024-04-26T01:47:57.551 INFO:tasks.workunit.client.0.smithi006.stdout: 1 41659 11.40 MB/sec warmup 21 sec latency 34.903 ms 2024-04-26T01:47:58.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:58 smithi188 ceph-mon[32751]: pgmap v65: 97 pgs: 97 active+clean; 15 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.5 MiB/s wr, 257 op/s 2024-04-26T01:47:58.390 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:58 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: ignoring --setuser ceph since I am not root 2024-04-26T01:47:58.390 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:58 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: ignoring --setgroup ceph since I am not root 2024-04-26T01:47:58.390 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:58 smithi188 ceph-mds[68153]: -- 172.21.15.188:0/820489056 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55badd635a40 con 0x55badd609000 2024-04-26T01:47:58.390 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:47:58 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: starting mds.f at 2024-04-26T01:47:58.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:58 smithi029 ceph-mon[32775]: pgmap v65: 97 pgs: 97 active+clean; 15 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.5 MiB/s wr, 257 op/s 2024-04-26T01:47:58.551 INFO:tasks.workunit.client.0.smithi006.stdout: 1 43162 11.38 MB/sec warmup 22 sec latency 20.053 ms 2024-04-26T01:47:58.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:58 smithi006 ceph-mon[28419]: pgmap v65: 97 pgs: 97 active+clean; 15 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.5 MiB/s wr, 257 op/s 2024-04-26T01:47:59.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:47:59 smithi029 ceph-mon[32775]: daemon mds.f restarted 2024-04-26T01:47:59.551 INFO:tasks.workunit.client.0.smithi006.stdout: 1 45033 11.20 MB/sec warmup 23 sec latency 16.595 ms 2024-04-26T01:47:59.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:47:59 smithi006 ceph-mon[28419]: daemon mds.f restarted 2024-04-26T01:47:59.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:47:59 smithi188 ceph-mon[32751]: daemon mds.f restarted 2024-04-26T01:48:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:00 smithi029 ceph-mon[32775]: pgmap v66: 97 pgs: 97 active+clean; 15 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 983 KiB/s wr, 121 op/s 2024-04-26T01:48:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:00 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:48:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:00 smithi029 ceph-mon[32775]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T01:48:00.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:00 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6834/1633748308,v1:172.21.15.188:6835/1633748308] up:boot 2024-04-26T01:48:00.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:00 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:48:00.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:00 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:48:00.554 INFO:tasks.workunit.client.0.smithi006.stdout: 1 46511 11.23 MB/sec warmup 24 sec latency 62.765 ms 2024-04-26T01:48:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:00 smithi006 ceph-mon[28419]: pgmap v66: 97 pgs: 97 active+clean; 15 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 983 KiB/s wr, 121 op/s 2024-04-26T01:48:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:00 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:48:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:00 smithi006 ceph-mon[28419]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T01:48:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:00 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6834/1633748308,v1:172.21.15.188:6835/1633748308] up:boot 2024-04-26T01:48:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:00 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:48:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:00 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:48:00.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:00 smithi188 ceph-mon[32751]: pgmap v66: 97 pgs: 97 active+clean; 15 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 983 KiB/s wr, 121 op/s 2024-04-26T01:48:00.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:00 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:48:00.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:00 smithi188 ceph-mon[32751]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T01:48:00.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:00 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6834/1633748308,v1:172.21.15.188:6835/1633748308] up:boot 2024-04-26T01:48:00.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:00 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:48:00.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:00 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:48:01.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:01 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:48:01.552 INFO:tasks.workunit.client.0.smithi006.stdout: 1 46745 10.78 MB/sec warmup 25 sec latency 631.673 ms 2024-04-26T01:48:01.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:01 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:48:01.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:01 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:48:02.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:02 smithi029 ceph-mon[32775]: pgmap v68: 97 pgs: 97 active+clean; 11 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.1 MiB/s wr, 251 op/s 2024-04-26T01:48:02.522 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:02 smithi006 ceph-mon[28419]: pgmap v68: 97 pgs: 97 active+clean; 11 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.1 MiB/s wr, 251 op/s 2024-04-26T01:48:02.552 INFO:tasks.workunit.client.0.smithi006.stdout: 1 46745 10.37 MB/sec warmup 26 sec latency 1631.822 ms 2024-04-26T01:48:02.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:02 smithi188 ceph-mon[32751]: pgmap v68: 97 pgs: 97 active+clean; 11 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.1 MiB/s wr, 251 op/s 2024-04-26T01:48:03.552 INFO:tasks.workunit.client.0.smithi006.stdout: 1 46745 9.98 MB/sec warmup 27 sec latency 2631.989 ms 2024-04-26T01:48:04.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:04 smithi029 ceph-mon[32775]: pgmap v69: 97 pgs: 97 active+clean; 11 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.1 MiB/s wr, 251 op/s 2024-04-26T01:48:04.552 INFO:tasks.workunit.client.0.smithi006.stdout: 1 47635 9.81 MB/sec warmup 28 sec latency 3012.323 ms 2024-04-26T01:48:04.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:04 smithi006 ceph-mon[28419]: pgmap v69: 97 pgs: 97 active+clean; 11 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.1 MiB/s wr, 251 op/s 2024-04-26T01:48:04.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:04 smithi188 ceph-mon[32751]: pgmap v69: 97 pgs: 97 active+clean; 11 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.1 MiB/s wr, 251 op/s 2024-04-26T01:48:05.552 INFO:tasks.workunit.client.0.smithi006.stdout: 1 49355 9.75 MB/sec warmup 29 sec latency 20.373 ms 2024-04-26T01:48:06.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:06 smithi029 ceph-mon[32775]: pgmap v70: 97 pgs: 97 active+clean; 11 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.1 MiB/s wr, 251 op/s 2024-04-26T01:48:06.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:06 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:48:06.553 INFO:tasks.workunit.client.0.smithi006.stdout: 1 50229 9.62 MB/sec warmup 30 sec latency 58.563 ms 2024-04-26T01:48:06.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:06 smithi006 ceph-mon[28419]: pgmap v70: 97 pgs: 97 active+clean; 11 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.1 MiB/s wr, 251 op/s 2024-04-26T01:48:06.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:06 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:48:06.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:06 smithi188 ceph-mon[32751]: pgmap v70: 97 pgs: 97 active+clean; 11 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.1 MiB/s wr, 251 op/s 2024-04-26T01:48:06.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:06 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:48:07.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:07 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:48:07.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:07 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:48:07.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:07 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:48:07.553 INFO:tasks.workunit.client.0.smithi006.stdout: 1 51376 9.48 MB/sec warmup 31 sec latency 138.637 ms 2024-04-26T01:48:07.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:07 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:48:07.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:07 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:48:07.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:07 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:48:07.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:07 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:48:07.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:07 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:48:07.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:07 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:48:08.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:08 smithi029 ceph-mon[32775]: pgmap v71: 97 pgs: 97 active+clean; 32 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 3.0 MiB/s wr, 244 op/s 2024-04-26T01:48:08.553 INFO:tasks.workunit.client.0.smithi006.stdout: 1 52656 9.41 MB/sec warmup 32 sec latency 320.106 ms 2024-04-26T01:48:08.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:08 smithi006 ceph-mon[28419]: pgmap v71: 97 pgs: 97 active+clean; 32 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 3.0 MiB/s wr, 244 op/s 2024-04-26T01:48:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:08 smithi188 ceph-mon[32751]: pgmap v71: 97 pgs: 97 active+clean; 32 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 3.0 MiB/s wr, 244 op/s 2024-04-26T01:48:09.553 INFO:tasks.workunit.client.0.smithi006.stdout: 1 53649 9.33 MB/sec warmup 33 sec latency 49.268 ms 2024-04-26T01:48:10.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:10 smithi029 ceph-mon[32775]: pgmap v72: 97 pgs: 97 active+clean; 32 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 3.0 MiB/s wr, 244 op/s 2024-04-26T01:48:10.553 INFO:tasks.workunit.client.0.smithi006.stdout: 1 54742 9.20 MB/sec warmup 34 sec latency 16.651 ms 2024-04-26T01:48:10.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:10 smithi006 ceph-mon[28419]: pgmap v72: 97 pgs: 97 active+clean; 32 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 3.0 MiB/s wr, 244 op/s 2024-04-26T01:48:10.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:10 smithi188 ceph-mon[32751]: pgmap v72: 97 pgs: 97 active+clean; 32 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 3.0 MiB/s wr, 244 op/s 2024-04-26T01:48:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.652+0000 7f2efec00640 1 -- 172.21.15.6:0/1597766149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef81021b0 msgr2=0x7f2ef8102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.652+0000 7f2efec00640 1 --2- 172.21.15.6:0/1597766149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef81021b0 0x7f2ef8102590 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009900 tx=0x7f2ee802a3f0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.652+0000 7f2efec00640 1 -- 172.21.15.6:0/1597766149 shutdown_connections 2024-04-26T01:48:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.652+0000 7f2efec00640 1 --2- 172.21.15.6:0/1597766149 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ef8102b60 0x7f2ef8106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.652+0000 7f2efec00640 1 --2- 172.21.15.6:0/1597766149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef8107280 0x7f2ef810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.652+0000 7f2efec00640 1 --2- 172.21.15.6:0/1597766149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef81021b0 0x7f2ef8102590 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.652+0000 7f2efec00640 1 -- 172.21.15.6:0/1597766149 >> 172.21.15.6:0/1597766149 conn(0x7f2ef80fda10 msgr2=0x7f2ef80ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.653+0000 7f2efec00640 1 -- 172.21.15.6:0/1597766149 shutdown_connections 2024-04-26T01:48:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.653+0000 7f2efec00640 1 -- 172.21.15.6:0/1597766149 wait complete. 2024-04-26T01:48:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.654+0000 7f2efec00640 1 Processor -- start 2024-04-26T01:48:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.654+0000 7f2efec00640 1 -- start start 2024-04-26T01:48:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.655+0000 7f2efec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ef81021b0 0x7f2ef81a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.655+0000 7f2efec00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef8107280 0x7f2ef81a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.655+0000 7f2efec00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef819a7d0 0x7f2ef819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.655+0000 7f2efec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef8114450 con 0x7f2ef81021b0 2024-04-26T01:48:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.655+0000 7f2efec00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef81142d0 con 0x7f2ef819a7d0 2024-04-26T01:48:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.655+0000 7f2efec00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef81145d0 con 0x7f2ef8107280 2024-04-26T01:48:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.655+0000 7f2ef7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ef81021b0 0x7f2ef81a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.655+0000 7f2ef7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ef81021b0 0x7f2ef81a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59290/0 (socket says 172.21.15.6:59290) 2024-04-26T01:48:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.655+0000 7f2ef7e00640 1 -- 172.21.15.6:0/539897242 learned_addr learned my addr 172.21.15.6:0/539897242 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.655+0000 7f2efcc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef819a7d0 0x7f2ef819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.655+0000 7f2ef7400640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef8107280 0x7f2ef81a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.656+0000 7f2ef7e00640 1 -- 172.21.15.6:0/539897242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef8107280 msgr2=0x7f2ef81a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.656+0000 7f2ef7e00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef8107280 0x7f2ef81a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.656+0000 7f2ef7e00640 1 -- 172.21.15.6:0/539897242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef819a7d0 msgr2=0x7f2ef819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.656+0000 7f2ef7e00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef819a7d0 0x7f2ef819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.656+0000 7f2ef7e00640 1 -- 172.21.15.6:0/539897242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef819b360 con 0x7f2ef81021b0 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.656+0000 7f2ef7400640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef8107280 0x7f2ef81a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.656+0000 7f2efcc00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef819a7d0 0x7f2ef819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.656+0000 7f2ef7e00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ef81021b0 0x7f2ef81a0660 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8005ab0 tx=0x7f2ee8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.656+0000 7f2ef4c00640 1 -- 172.21.15.6:0/539897242 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee803f070 con 0x7f2ef81021b0 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.657+0000 7f2ef4c00640 1 -- 172.21.15.6:0/539897242 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ee8031b60 con 0x7f2ef81021b0 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.657+0000 7f2ef4c00640 1 -- 172.21.15.6:0/539897242 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8035030 con 0x7f2ef81021b0 2024-04-26T01:48:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.657+0000 7f2efec00640 1 -- 172.21.15.6:0/539897242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ef819b5f0 con 0x7f2ef81021b0 2024-04-26T01:48:10.656 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.657+0000 7f2efec00640 1 -- 172.21.15.6:0/539897242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ef81a7750 con 0x7f2ef81021b0 2024-04-26T01:48:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.660+0000 7f2f019dd740 1 Processor -- start 2024-04-26T01:48:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.660+0000 7f2ef4c00640 1 -- 172.21.15.6:0/539897242 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f2ee8042070 con 0x7f2ef81021b0 2024-04-26T01:48:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.661+0000 7f2f019dd740 1 -- start start 2024-04-26T01:48:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.661+0000 7f2f019dd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a3820 0x55ec403a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.661+0000 7f2f019dd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a4140 0x55ec403ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.661+0000 7f2f019dd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403aea00 0x55ec403b0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.661+0000 7f2f019dd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55ec40309900 con 0x55ec403aea00 2024-04-26T01:48:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.661+0000 7f2f019dd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ec402afef0 con 0x55ec403a3820 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.661+0000 7f2f019dd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ec403058c0 con 0x55ec403a4140 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.661+0000 7f2efcc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a3820 0x55ec403a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.661+0000 7f2ef7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a4140 0x55ec403ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.661+0000 7f2efcc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a3820 0x55ec403a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:49178/0 (socket says 172.21.15.6:49178) 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.661+0000 7f2efcc00640 1 -- 172.21.15.6:0/2128432099 learned_addr learned my addr 172.21.15.6:0/2128432099 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.662+0000 7f2ef7400640 1 --2- 172.21.15.6:0/2128432099 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403aea00 0x55ec403b0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.662+0000 7f2efcc00640 1 -- 172.21.15.6:0/2128432099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a4140 msgr2=0x55ec403ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.662+0000 7f2efcc00640 1 --2- 172.21.15.6:0/2128432099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a4140 0x55ec403ae4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.662+0000 7f2efcc00640 1 -- 172.21.15.6:0/2128432099 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403aea00 msgr2=0x55ec403b0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.662+0000 7f2efcc00640 1 --2- 172.21.15.6:0/2128432099 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403aea00 0x55ec403b0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.662+0000 7f2efcc00640 1 -- 172.21.15.6:0/2128432099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec4001c210 con 0x55ec403a3820 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.662+0000 7f2efcc00640 1 --2- 172.21.15.6:0/2128432099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a3820 0x55ec403a3c00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0011790 tx=0x7f2ef0011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=316799854b826711 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.662+0000 7f2ef4c00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2edc0a3d90 0x7f2edc0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.662+0000 7f2eed200640 1 -- 172.21.15.6:0/2128432099 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0003c30 con 0x55ec403a3820 2024-04-26T01:48:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.662+0000 7f2eed200640 1 -- 172.21.15.6:0/2128432099 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ef000fe50 con 0x55ec403a3820 2024-04-26T01:48:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.662+0000 7f2ef4c00640 1 -- 172.21.15.6:0/539897242 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f2ee80f0f80 con 0x7f2ef81021b0 2024-04-26T01:48:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.663+0000 7f2eed200640 1 -- 172.21.15.6:0/2128432099 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef00193e0 con 0x55ec403a3820 2024-04-26T01:48:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.663+0000 7f2ef7e00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2edc0a3d90 0x7f2edc0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.663+0000 7f2f019dd740 1 -- 172.21.15.6:0/2128432099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a3820 msgr2=0x55ec403a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.663+0000 7f2ef7e00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2edc0a3d90 0x7f2edc0a6250 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x55ec403a4dc0 tx=0x7f2ee80f4a30 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.663+0000 7f2f019dd740 1 --2- 172.21.15.6:0/2128432099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a3820 0x55ec403a3c00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0011790 tx=0x7f2ef0011c50 comp rx=0 tx=0).stop 2024-04-26T01:48:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.663+0000 7f2f019dd740 1 -- 172.21.15.6:0/2128432099 shutdown_connections 2024-04-26T01:48:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.663+0000 7f2f019dd740 1 --2- 172.21.15.6:0/2128432099 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403aea00 0x55ec403b0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.664+0000 7f2f019dd740 1 --2- 172.21.15.6:0/2128432099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a4140 0x55ec403ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.664+0000 7f2f019dd740 1 --2- 172.21.15.6:0/2128432099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a3820 0x55ec403a3c00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.664+0000 7f2f019dd740 1 -- 172.21.15.6:0/2128432099 >> 172.21.15.6:0/2128432099 conn(0x55ec4039fd30 msgr2=0x55ec403a0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.664+0000 7f2f019dd740 1 -- 172.21.15.6:0/2128432099 shutdown_connections 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.664+0000 7f2f019dd740 1 -- 172.21.15.6:0/2128432099 wait complete. 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.664+0000 7f2f019dd740 1 -- start start 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.664+0000 7f2f019dd740 1 Processor -- start 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2f019dd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403a3820 0x55ec403ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2f019dd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a4140 0x55ec403abf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2f019dd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403aea00 0x55ec403ac440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2f019dd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55ec403b1630 con 0x55ec403a3820 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2f019dd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ec402c3310 con 0x55ec403a4140 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2f019dd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ec403b17b0 con 0x55ec403aea00 2024-04-26T01:48:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2f019dd740 10 client.0 _ll_register_callbacks cb 0x55ec40291cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2ef7400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a4140 0x55ec403abf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2efcc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403a3820 0x55ec403ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2ef7400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a4140 0x55ec403abf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:49182/0 (socket says 172.21.15.6:49182) 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2ef7400640 1 -- 172.21.15.6:0/862244595 learned_addr learned my addr 172.21.15.6:0/862244595 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2efcc00640 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403aea00 0x55ec403ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.665+0000 7f2ef7400640 1 -- 172.21.15.6:0/862244595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403aea00 msgr2=0x55ec403ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2ef7400640 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403aea00 0x55ec403ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2ef7400640 1 -- 172.21.15.6:0/862244595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403a3820 msgr2=0x55ec403ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2ef7400640 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403a3820 0x55ec403ab9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2ef7400640 1 -- 172.21.15.6:0/862244595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec400e8620 con 0x55ec403a4140 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2efcc00640 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403a3820 0x55ec403ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2ef7400640 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a4140 0x55ec403abf00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4013790 tx=0x7f2ee4013c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caa5ce3c6c79f358 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2ee1600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2ee1600640 1 -- 172.21.15.6:0/862244595 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4020060 con 0x55ec403a4140 2024-04-26T01:48:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2ee1600640 1 -- 172.21.15.6:0/862244595 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f2ee4011dd0 con 0x55ec403a4140 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2f019dd740 1 -- 172.21.15.6:0/862244595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ef000fe50 con 0x55ec403a4140 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2ee1600640 1 -- 172.21.15.6:0/862244595 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2ee400f450 con 0x55ec403a4140 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2f019dd740 10 client.25439 fetch_fsmap learned FSMap version 31 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.666+0000 7f2f019dd740 1 -- 172.21.15.6:0/862244595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ec403bd960 con 0x55ec403a4140 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.667+0000 7f2ee1600640 1 -- 172.21.15.6:0/862244595 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ef000fe50 con 0x55ec403a4140 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.667+0000 7f2ee1600640 1 -- 172.21.15.6:0/862244595 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f2ee401a030 con 0x55ec403a4140 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.667+0000 7f2f019dd740 10 client.25439 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.667+0000 7f2f019dd740 10 client.25439 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.667+0000 7f2f019dd740 20 client.25439 populate_metadata read hostname 'smithi006' 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.667+0000 7f2f019dd740 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55ec403bee70 0x55ec403c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.667+0000 7f2f019dd740 4 client.25439 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.667+0000 7f2f019dd740 1 -- 172.21.15.6:0/862244595 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec4031d8b0 con 0x55ec403bee70 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.667+0000 7f2efcc00640 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55ec403bee70 0x55ec403c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.668+0000 7f2efcc00640 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55ec403bee70 0x55ec403c12a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=107198fb621f3db8 server_cookie=140e6f8b2f40b567 in_seq=0 out_seq=0 2024-04-26T01:48:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.668+0000 7f2ee1600640 10 client.25439 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.668+0000 7f2ee1600640 1 -- 172.21.15.6:0/862244595 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec4031d8b0 con 0x55ec403bee70 2024-04-26T01:48:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.668+0000 7f2ee1600640 10 client.25439 handle_command_reply: tid=0 2024-04-26T01:48:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2f019dd740 1 client.25439 shutdown 2024-04-26T01:48:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2f019dd740 1 -- 172.21.15.6:0/862244595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a4140 msgr2=0x55ec403abf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2f019dd740 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a4140 0x55ec403abf00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4013790 tx=0x7f2ee4013c50 comp rx=0 tx=0).stop 2024-04-26T01:48:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2f019dd740 1 -- 172.21.15.6:0/862244595 shutdown_connections 2024-04-26T01:48:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2f019dd740 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55ec403bee70 0x55ec403c12a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2f019dd740 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403aea00 0x55ec403ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2ee1600640 0 client.25439 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2f019dd740 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a4140 0x55ec403abf00 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4013790 tx=0x7f2ee4013c50 comp rx=0 tx=0).stop 2024-04-26T01:48:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2f019dd740 1 --2- 172.21.15.6:0/862244595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403a3820 0x55ec403ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2f019dd740 1 -- 172.21.15.6:0/862244595 >> 172.21.15.6:0/862244595 conn(0x55ec4039fd30 msgr2=0x55ec403a5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2f019dd740 1 -- 172.21.15.6:0/862244595 shutdown_connections 2024-04-26T01:48:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2f019dd740 1 -- 172.21.15.6:0/862244595 wait complete. 2024-04-26T01:48:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.669+0000 7f2f019dd740 20 client.25439 trim_cache size 0 max 16384 2024-04-26T01:48:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.687+0000 7f2f019dd740 1 Processor -- start 2024-04-26T01:48:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.687+0000 7f2f019dd740 1 -- start start 2024-04-26T01:48:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.687+0000 7f2f019dd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 0x55ec4039f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.687+0000 7f2f019dd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403aea00 0x55ec403ba4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.687+0000 7f2f019dd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a7390 0x55ec403c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.687+0000 7f2f019dd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef81b2810 con 0x55ec403aea00 2024-04-26T01:48:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.687+0000 7f2f019dd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef81b0080 con 0x55ec403a7390 2024-04-26T01:48:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.687+0000 7f2f019dd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef81aded0 con 0x55ec403a3820 2024-04-26T01:48:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.688+0000 7f2ef7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403aea00 0x55ec403ba4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.688+0000 7f2efcc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 0x55ec4039f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.688+0000 7f2ef7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403aea00 0x55ec403ba4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59324/0 (socket says 172.21.15.6:59324) 2024-04-26T01:48:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.688+0000 7f2ef7400640 1 -- 172.21.15.6:0/2658314885 learned_addr learned my addr 172.21.15.6:0/2658314885 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.688+0000 7f2efcc00640 1 --2- 172.21.15.6:0/2658314885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a7390 0x55ec403c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.688+0000 7f2ef7400640 1 -- 172.21.15.6:0/2658314885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 msgr2=0x55ec4039f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.688+0000 7f2ef7400640 1 --2- 172.21.15.6:0/2658314885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 0x55ec4039f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.688+0000 7f2ef7400640 1 -- 172.21.15.6:0/2658314885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a7390 msgr2=0x55ec403c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.688+0000 7f2ef7400640 1 --2- 172.21.15.6:0/2658314885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a7390 0x55ec403c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.688+0000 7f2ef7400640 1 -- 172.21.15.6:0/2658314885 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec4039b690 con 0x55ec403aea00 2024-04-26T01:48:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.688+0000 7f2efcc00640 1 --2- 172.21.15.6:0/2658314885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 0x55ec4039f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.689+0000 7f2ef7400640 1 --2- 172.21.15.6:0/2658314885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403aea00 0x55ec403ba4f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4018040 tx=0x7f2ee4009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fec75835a7dfd90 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.689+0000 7f2ee3e00640 1 -- 172.21.15.6:0/2658314885 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee400ea20 con 0x55ec403aea00 2024-04-26T01:48:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.689+0000 7f2ee3e00640 1 -- 172.21.15.6:0/2658314885 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ee400ebc0 con 0x55ec403aea00 2024-04-26T01:48:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.689+0000 7f2ee3e00640 1 -- 172.21.15.6:0/2658314885 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40035f0 con 0x55ec403aea00 2024-04-26T01:48:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.690+0000 7f2f019dd740 1 -- 172.21.15.6:0/2658314885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403aea00 msgr2=0x55ec403ba4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.690+0000 7f2f019dd740 1 --2- 172.21.15.6:0/2658314885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403aea00 0x55ec403ba4f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4018040 tx=0x7f2ee4009770 comp rx=0 tx=0).stop 2024-04-26T01:48:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.690+0000 7f2f019dd740 1 -- 172.21.15.6:0/2658314885 shutdown_connections 2024-04-26T01:48:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.690+0000 7f2f019dd740 1 --2- 172.21.15.6:0/2658314885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403aea00 0x55ec403ba4f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.690+0000 7f2f019dd740 1 --2- 172.21.15.6:0/2658314885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403a7390 0x55ec403c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.690+0000 7f2f019dd740 1 --2- 172.21.15.6:0/2658314885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 0x55ec4039f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.690+0000 7f2f019dd740 1 -- 172.21.15.6:0/2658314885 >> 172.21.15.6:0/2658314885 conn(0x55ec4039fd30 msgr2=0x55ec403a6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.690+0000 7f2f019dd740 1 -- 172.21.15.6:0/2658314885 shutdown_connections 2024-04-26T01:48:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.690+0000 7f2f019dd740 1 -- 172.21.15.6:0/2658314885 wait complete. 2024-04-26T01:48:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.690+0000 7f2f019dd740 1 -- start start 2024-04-26T01:48:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.691+0000 7f2f019dd740 1 Processor -- start 2024-04-26T01:48:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.691+0000 7f2f019dd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 0x55ec403b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.691+0000 7f2f019dd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403a7390 0x55ec403b4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.691+0000 7f2f019dd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403aea00 0x55ec403b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.691+0000 7f2f019dd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef81ae540 con 0x55ec403a7390 2024-04-26T01:48:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.691+0000 7f2f019dd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef81af9b0 con 0x55ec403aea00 2024-04-26T01:48:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.691+0000 7f2f019dd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef81aec10 con 0x55ec403a3820 2024-04-26T01:48:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.691+0000 7f2f019dd740 10 client.0 _ll_register_callbacks cb 0x7f2ee4009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.691+0000 7f2efcc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 0x55ec403b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.691+0000 7f2efcc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 0x55ec403b4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36344/0 (socket says 172.21.15.6:36344) 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.692+0000 7f2efcc00640 1 -- 172.21.15.6:0/1808606164 learned_addr learned my addr 172.21.15.6:0/1808606164 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.692+0000 7f2ef7400640 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403a7390 0x55ec403b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.692+0000 7f2efcc00640 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403aea00 0x55ec403b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.692+0000 7f2efcc00640 1 -- 172.21.15.6:0/1808606164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403aea00 msgr2=0x55ec403b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.692+0000 7f2efcc00640 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403aea00 0x55ec403b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.692+0000 7f2efcc00640 1 -- 172.21.15.6:0/1808606164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403a7390 msgr2=0x55ec403b4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.692+0000 7f2efcc00640 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403a7390 0x55ec403b4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.692+0000 7f2efcc00640 1 -- 172.21.15.6:0/1808606164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec4042cc00 con 0x55ec403a3820 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.692+0000 7f2efcc00640 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 0x55ec403b4350 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0005c40 tx=0x7f2ee4009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b02d645527794179 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.692+0000 7f2ee1600640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.692+0000 7f2ee1600640 1 -- 172.21.15.6:0/1808606164 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef001f040 con 0x55ec403a3820 2024-04-26T01:48:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.693+0000 7f2ee1600640 1 -- 172.21.15.6:0/1808606164 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f2ef000fa20 con 0x55ec403a3820 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.693+0000 7f2f019dd740 1 -- 172.21.15.6:0/1808606164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ee400ebc0 con 0x55ec403a3820 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.693+0000 7f2ee1600640 1 -- 172.21.15.6:0/1808606164 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2ef000fc20 con 0x55ec403a3820 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.693+0000 7f2f019dd740 10 client.25492 fetch_fsmap learned FSMap version 31 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.693+0000 7f2f019dd740 1 -- 172.21.15.6:0/1808606164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ec4042de10 con 0x55ec403a3820 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.693+0000 7f2ee1600640 1 -- 172.21.15.6:0/1808606164 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ee400ebc0 con 0x55ec403a3820 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.694+0000 7f2ee1600640 1 -- 172.21.15.6:0/1808606164 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f2ef0025040 con 0x55ec403a3820 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.694+0000 7f2f019dd740 10 client.25492 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.694+0000 7f2f019dd740 10 client.25492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.694+0000 7f2f019dd740 20 client.25492 populate_metadata read hostname 'smithi006' 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.694+0000 7f2f019dd740 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55ec4042e480 0x55ec404308a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.694+0000 7f2f019dd740 4 client.25492 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.694+0000 7f2f019dd740 1 -- 172.21.15.6:0/1808606164 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ec40317320 con 0x55ec4042e480 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.694+0000 7f2ef7400640 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55ec4042e480 0x55ec404308a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.694+0000 7f2ef7400640 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55ec4042e480 0x55ec404308a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce75a4c1e1544a58 server_cookie=8f0e629cb6a0f505 in_seq=0 out_seq=0 2024-04-26T01:48:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.694+0000 7f2ee1600640 10 client.25492 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2ee1600640 1 -- 172.21.15.6:0/1808606164 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ec40317320 con 0x55ec4042e480 2024-04-26T01:48:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2ee1600640 10 client.25492 handle_command_reply: tid=0 2024-04-26T01:48:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2f019dd740 1 client.25492 shutdown 2024-04-26T01:48:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2f019dd740 1 -- 172.21.15.6:0/1808606164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 msgr2=0x55ec403b4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2f019dd740 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 0x55ec403b4350 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0005c40 tx=0x7f2ee4009c30 comp rx=0 tx=0).stop 2024-04-26T01:48:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2f019dd740 1 -- 172.21.15.6:0/1808606164 shutdown_connections 2024-04-26T01:48:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2f019dd740 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55ec4042e480 0x55ec404308a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2f019dd740 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ec403a7390 0x55ec403b4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2f019dd740 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ec403aea00 0x55ec403b4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2f019dd740 1 --2- 172.21.15.6:0/1808606164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ec403a3820 0x55ec403b4350 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2ee1600640 0 client.25492 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2f019dd740 1 -- 172.21.15.6:0/1808606164 >> 172.21.15.6:0/1808606164 conn(0x55ec4039fd30 msgr2=0x55ec403ab980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2f019dd740 1 -- 172.21.15.6:0/1808606164 shutdown_connections 2024-04-26T01:48:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.743+0000 7f2f019dd740 1 -- 172.21.15.6:0/1808606164 wait complete. 2024-04-26T01:48:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.744+0000 7f2f019dd740 20 client.25492 trim_cache size 0 max 16384 2024-04-26T01:48:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:48:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:48:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:48:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:48:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.744+0000 7f2efec00640 1 -- 172.21.15.6:0/539897242 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2edc0a3d90 msgr2=0x7f2edc0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.744+0000 7f2efec00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2edc0a3d90 0x7f2edc0a6250 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x55ec403a4dc0 tx=0x7f2ee80f4a30 comp rx=0 tx=0).stop 2024-04-26T01:48:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.744+0000 7f2efec00640 1 -- 172.21.15.6:0/539897242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ef81021b0 msgr2=0x7f2ef81a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.744+0000 7f2efec00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ef81021b0 0x7f2ef81a0660 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8005ab0 tx=0x7f2ee8003c10 comp rx=0 tx=0).stop 2024-04-26T01:48:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.745+0000 7f2efec00640 1 -- 172.21.15.6:0/539897242 shutdown_connections 2024-04-26T01:48:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.745+0000 7f2efec00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2edc0a3d90 0x7f2edc0a6250 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.745+0000 7f2efec00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef819a7d0 0x7f2ef819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.745+0000 7f2efec00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef8107280 0x7f2ef81a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.745+0000 7f2efec00640 1 --2- 172.21.15.6:0/539897242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ef81021b0 0x7f2ef81a0660 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.745+0000 7f2efec00640 1 -- 172.21.15.6:0/539897242 >> 172.21.15.6:0/539897242 conn(0x7f2ef80fda10 msgr2=0x7f2ef80fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.745+0000 7f2efec00640 1 -- 172.21.15.6:0/539897242 shutdown_connections 2024-04-26T01:48:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:10.745+0000 7f2efec00640 1 -- 172.21.15.6:0/539897242 wait complete. 2024-04-26T01:48:11.553 INFO:tasks.workunit.client.0.smithi006.stdout: 1 56481 9.19 MB/sec warmup 35 sec latency 36.439 ms 2024-04-26T01:48:12.553 INFO:tasks.workunit.client.0.smithi006.stdout: 1 57338 9.08 MB/sec warmup 36 sec latency 13.208 ms 2024-04-26T01:48:12.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:12 smithi006 ceph-mon[28419]: pgmap v73: 97 pgs: 97 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.9 MiB/s wr, 397 op/s 2024-04-26T01:48:12.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:12 smithi188 ceph-mon[32751]: pgmap v73: 97 pgs: 97 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.9 MiB/s wr, 397 op/s 2024-04-26T01:48:12.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:12 smithi029 ceph-mon[32775]: pgmap v73: 97 pgs: 97 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.9 MiB/s wr, 397 op/s 2024-04-26T01:48:13.554 INFO:tasks.workunit.client.0.smithi006.stdout: 1 58980 8.99 MB/sec warmup 37 sec latency 14.930 ms 2024-04-26T01:48:14.554 INFO:tasks.workunit.client.0.smithi006.stdout: 1 60535 9.10 MB/sec warmup 38 sec latency 16.332 ms 2024-04-26T01:48:14.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:14 smithi006 ceph-mon[28419]: pgmap v74: 97 pgs: 97 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.4 MiB/s wr, 244 op/s 2024-04-26T01:48:14.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:14 smithi188 ceph-mon[32751]: pgmap v74: 97 pgs: 97 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.4 MiB/s wr, 244 op/s 2024-04-26T01:48:14.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:14 smithi029 ceph-mon[32775]: pgmap v74: 97 pgs: 97 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.4 MiB/s wr, 244 op/s 2024-04-26T01:48:15.554 INFO:tasks.workunit.client.0.smithi006.stdout: 1 61203 8.89 MB/sec warmup 39 sec latency 49.461 ms 2024-04-26T01:48:16.554 INFO:tasks.workunit.client.0.smithi006.stdout: 1 62794 8.84 MB/sec warmup 40 sec latency 17.885 ms 2024-04-26T01:48:16.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:16 smithi006 ceph-mon[28419]: pgmap v75: 97 pgs: 97 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.4 MiB/s wr, 244 op/s 2024-04-26T01:48:16.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:16 smithi188 ceph-mon[32751]: pgmap v75: 97 pgs: 97 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.4 MiB/s wr, 244 op/s 2024-04-26T01:48:16.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:16 smithi029 ceph-mon[32775]: pgmap v75: 97 pgs: 97 active+clean; 27 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.4 MiB/s wr, 244 op/s 2024-04-26T01:48:17.554 INFO:tasks.workunit.client.0.smithi006.stdout: 1 64173 8.90 MB/sec warmup 41 sec latency 15.935 ms 2024-04-26T01:48:18.554 INFO:tasks.workunit.client.0.smithi006.stdout: 1 65120 8.74 MB/sec warmup 42 sec latency 20.195 ms 2024-04-26T01:48:18.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:18 smithi006 ceph-mon[28419]: pgmap v76: 97 pgs: 97 active+clean; 30 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.7 MiB/s wr, 309 op/s 2024-04-26T01:48:18.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:18 smithi188 ceph-mon[32751]: pgmap v76: 97 pgs: 97 active+clean; 30 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.7 MiB/s wr, 309 op/s 2024-04-26T01:48:18.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:18 smithi029 ceph-mon[32775]: pgmap v76: 97 pgs: 97 active+clean; 30 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.7 MiB/s wr, 309 op/s 2024-04-26T01:48:19.554 INFO:tasks.workunit.client.0.smithi006.stdout: 1 66777 8.77 MB/sec warmup 43 sec latency 18.050 ms 2024-04-26T01:48:20.554 INFO:tasks.workunit.client.0.smithi006.stdout: 1 67813 8.73 MB/sec warmup 44 sec latency 15.579 ms 2024-04-26T01:48:20.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:20 smithi006 ceph-mon[28419]: pgmap v77: 97 pgs: 97 active+clean; 30 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 255 op/s 2024-04-26T01:48:20.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:20 smithi188 ceph-mon[32751]: pgmap v77: 97 pgs: 97 active+clean; 30 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 255 op/s 2024-04-26T01:48:20.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:20 smithi029 ceph-mon[32775]: pgmap v77: 97 pgs: 97 active+clean; 30 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 255 op/s 2024-04-26T01:48:21.554 INFO:tasks.workunit.client.0.smithi006.stdout: 1 68846 8.65 MB/sec warmup 45 sec latency 19.476 ms 2024-04-26T01:48:22.555 INFO:tasks.workunit.client.0.smithi006.stdout: 1 70399 8.63 MB/sec warmup 46 sec latency 17.944 ms 2024-04-26T01:48:22.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:22 smithi006 ceph-mon[28419]: pgmap v78: 97 pgs: 97 active+clean; 34 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.0 MiB/s wr, 319 op/s 2024-04-26T01:48:22.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:22 smithi188 ceph-mon[32751]: pgmap v78: 97 pgs: 97 active+clean; 34 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.0 MiB/s wr, 319 op/s 2024-04-26T01:48:22.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:22 smithi029 ceph-mon[32775]: pgmap v78: 97 pgs: 97 active+clean; 34 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.0 MiB/s wr, 319 op/s 2024-04-26T01:48:23.555 INFO:tasks.workunit.client.0.smithi006.stdout: 1 71366 8.58 MB/sec warmup 47 sec latency 18.857 ms 2024-04-26T01:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.890+0000 7fcc7b200640 1 -- 172.21.15.6:0/2373372866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc7410e660 msgr2=0x7fcc7410ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.890+0000 7fcc7b200640 1 --2- 172.21.15.6:0/2373372866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc7410e660 0x7fcc7410ea40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009930 tx=0x7fcc6402a440 comp rx=0 tx=0).stop 2024-04-26T01:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.891+0000 7fcc7b200640 1 -- 172.21.15.6:0/2373372866 shutdown_connections 2024-04-26T01:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.891+0000 7fcc7b200640 1 --2- 172.21.15.6:0/2373372866 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc74108f60 0x7fcc74115420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.891+0000 7fcc7b200640 1 --2- 172.21.15.6:0/2373372866 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc741085a0 0x7fcc74108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.891+0000 7fcc7b200640 1 --2- 172.21.15.6:0/2373372866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc7410e660 0x7fcc7410ea40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.891+0000 7fcc7b200640 1 -- 172.21.15.6:0/2373372866 >> 172.21.15.6:0/2373372866 conn(0x7fcc74076460 msgr2=0x7fcc74078880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.891+0000 7fcc7b200640 1 -- 172.21.15.6:0/2373372866 shutdown_connections 2024-04-26T01:48:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.891+0000 7fcc7b200640 1 -- 172.21.15.6:0/2373372866 wait complete. 2024-04-26T01:48:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.892+0000 7fcc7b200640 1 Processor -- start 2024-04-26T01:48:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.892+0000 7fcc7b200640 1 -- start start 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.892+0000 7fcc7b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc741085a0 0x7fcc7419ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc7b200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc74108f60 0x7fcc7419f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc7b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc7410e660 0x7fcc741a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc7b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcc74118860 con 0x7fcc7410e660 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc7b200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcc741186e0 con 0x7fcc74108f60 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc7b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcc741189e0 con 0x7fcc741085a0 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc6be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc74108f60 0x7fcc7419f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc73e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc741085a0 0x7fcc7419ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc6be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc74108f60 0x7fcc7419f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39044/0 (socket says 172.21.15.6:39044) 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc73e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc741085a0 0x7fcc7419ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58938/0 (socket says 172.21.15.6:58938) 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc6be00640 1 -- 172.21.15.6:0/2085677060 learned_addr learned my addr 172.21.15.6:0/2085677060 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc79200640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc7410e660 0x7fcc741a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc6be00640 1 -- 172.21.15.6:0/2085677060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc741085a0 msgr2=0x7fcc7419ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc6be00640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc741085a0 0x7fcc7419ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc6be00640 1 -- 172.21.15.6:0/2085677060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc7410e660 msgr2=0x7fcc741a35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc6be00640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc7410e660 0x7fcc741a35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc6be00640 1 -- 172.21.15.6:0/2085677060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc741a3d10 con 0x7fcc74108f60 2024-04-26T01:48:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.893+0000 7fcc79200640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc7410e660 0x7fcc741a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.894+0000 7fcc6be00640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc74108f60 0x7fcc7419f260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c00ea60 tx=0x7fcc5c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.894+0000 7fcc71600640 1 -- 172.21.15.6:0/2085677060 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c00cde0 con 0x7fcc74108f60 2024-04-26T01:48:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.894+0000 7fcc71600640 1 -- 172.21.15.6:0/2085677060 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc5c0035f0 con 0x7fcc74108f60 2024-04-26T01:48:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.894+0000 7fcc71600640 1 -- 172.21.15.6:0/2085677060 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c010690 con 0x7fcc74108f60 2024-04-26T01:48:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.894+0000 7fcc7b200640 1 -- 172.21.15.6:0/2085677060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc741a4000 con 0x7fcc74108f60 2024-04-26T01:48:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.894+0000 7fcc7b200640 1 -- 172.21.15.6:0/2085677060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc741ab890 con 0x7fcc74108f60 2024-04-26T01:48:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.897+0000 7fcc71600640 1 -- 172.21.15.6:0/2085677060 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fcc5c0037a0 con 0x7fcc74108f60 2024-04-26T01:48:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.898+0000 7fcc7dff1740 1 Processor -- start 2024-04-26T01:48:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.898+0000 7fcc7dff1740 1 -- start start 2024-04-26T01:48:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.898+0000 7fcc7dff1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681bacac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.898+0000 7fcc7dff1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bacb140 0x55681bad54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.898+0000 7fcc7dff1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bad5a00 0x55681bad7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.898+0000 7fcc7dff1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55681ba30900 con 0x55681baca820 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.898+0000 7fcc7dff1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55681b9d6ef0 con 0x55681bad5a00 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.898+0000 7fcc7dff1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55681ba2c8c0 con 0x55681bacb140 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc73e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bacb140 0x55681bad54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc79200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681bacac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc73e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bacb140 0x55681bad54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58948/0 (socket says 172.21.15.6:58948) 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc73e00640 1 -- 172.21.15.6:0/1416859362 learned_addr learned my addr 172.21.15.6:0/1416859362 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc6be00640 1 --2- 172.21.15.6:0/1416859362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bad5a00 0x55681bad7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc73e00640 1 -- 172.21.15.6:0/1416859362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bad5a00 msgr2=0x55681bad7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc73e00640 1 --2- 172.21.15.6:0/1416859362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bad5a00 0x55681bad7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc73e00640 1 -- 172.21.15.6:0/1416859362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 msgr2=0x55681bacac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc73e00640 1 --2- 172.21.15.6:0/1416859362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681bacac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc73e00640 1 -- 172.21.15.6:0/1416859362 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55681b743210 con 0x55681bacb140 2024-04-26T01:48:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc6be00640 1 --2- 172.21.15.6:0/1416859362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bad5a00 0x55681bad7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc79200640 1 --2- 172.21.15.6:0/1416859362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681bacac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc71600640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcc580a3d10 0x7fcc580a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.899+0000 7fcc73e00640 1 --2- 172.21.15.6:0/1416859362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bacb140 0x55681bad54c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcc64006010 tx=0x7fcc64004170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2cb06e11337e4a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.900+0000 7fcc71600640 1 -- 172.21.15.6:0/2085677060 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fcc5c0d0bd0 con 0x7fcc74108f60 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.900+0000 7fcc79200640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcc580a3d10 0x7fcc580a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.900+0000 7fcc69600640 1 -- 172.21.15.6:0/1416859362 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc64035040 con 0x55681bacb140 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.900+0000 7fcc69600640 1 -- 172.21.15.6:0/1416859362 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc640054f0 con 0x55681bacb140 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.900+0000 7fcc69600640 1 -- 172.21.15.6:0/1416859362 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc640035f0 con 0x55681bacb140 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.900+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1416859362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bacb140 msgr2=0x55681bad54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.900+0000 7fcc79200640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcc580a3d10 0x7fcc580a61d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009690 tx=0x7fcc6c009620 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.900+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1416859362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bacb140 0x55681bad54c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcc64006010 tx=0x7fcc64004170 comp rx=0 tx=0).stop 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.901+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1416859362 shutdown_connections 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.901+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1416859362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bad5a00 0x55681bad7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.901+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1416859362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bacb140 0x55681bad54c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.901+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1416859362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681bacac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.901+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1416859362 >> 172.21.15.6:0/1416859362 conn(0x55681bac6d30 msgr2=0x55681bac7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.901+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1416859362 shutdown_connections 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.901+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1416859362 wait complete. 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.901+0000 7fcc7dff1740 1 -- start start 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.901+0000 7fcc7dff1740 1 Processor -- start 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.902+0000 7fcc7dff1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681bad2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.902+0000 7fcc7dff1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bacb140 0x55681bad2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.902+0000 7fcc7dff1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bad5a00 0x55681bad33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.902+0000 7fcc7dff1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55681bad85d0 con 0x55681baca820 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.902+0000 7fcc7dff1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55681b9ea310 con 0x55681bacb140 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.902+0000 7fcc7dff1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55681bad8750 con 0x55681bad5a00 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.902+0000 7fcc7dff1740 10 client.0 _ll_register_callbacks cb 0x55681b9b8cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.902+0000 7fcc6be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bacb140 0x55681bad2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.902+0000 7fcc73e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681bad2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.902+0000 7fcc79200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bad5a00 0x55681bad33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.902+0000 7fcc6be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bacb140 0x55681bad2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39068/0 (socket says 172.21.15.6:39068) 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.902+0000 7fcc6be00640 1 -- 172.21.15.6:0/467826258 learned_addr learned my addr 172.21.15.6:0/467826258 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc6be00640 1 -- 172.21.15.6:0/467826258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bad5a00 msgr2=0x55681bad33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc6be00640 1 --2- 172.21.15.6:0/467826258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bad5a00 0x55681bad33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc6be00640 1 -- 172.21.15.6:0/467826258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 msgr2=0x55681bad2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc6be00640 1 --2- 172.21.15.6:0/467826258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681bad2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc6be00640 1 -- 172.21.15.6:0/467826258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55681b80f620 con 0x55681bacb140 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc6be00640 1 --2- 172.21.15.6:0/467826258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bacb140 0x55681bad2e90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009920 tx=0x7fcc5c098e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b53dfc7f1c22c0cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc62000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc62000640 1 -- 172.21.15.6:0/467826258 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c09d1f0 con 0x55681bacb140 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc62000640 1 -- 172.21.15.6:0/467826258 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fcc5c004000 con 0x55681bacb140 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc7dff1740 1 -- 172.21.15.6:0/467826258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc640054f0 con 0x55681bacb140 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc62000640 1 -- 172.21.15.6:0/467826258 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcc5c09caf0 con 0x55681bacb140 2024-04-26T01:48:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc7dff1740 10 client.25457 fetch_fsmap learned FSMap version 31 2024-04-26T01:48:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.903+0000 7fcc7dff1740 1 -- 172.21.15.6:0/467826258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55681bae47e0 con 0x55681bacb140 2024-04-26T01:48:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.904+0000 7fcc62000640 1 -- 172.21.15.6:0/467826258 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc640054f0 con 0x55681bacb140 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.904+0000 7fcc62000640 1 -- 172.21.15.6:0/467826258 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fcc5c0ce520 con 0x55681bacb140 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.904+0000 7fcc7dff1740 10 client.25457 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.904+0000 7fcc7dff1740 10 client.25457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.904+0000 7fcc7dff1740 20 client.25457 populate_metadata read hostname 'smithi006' 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.904+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/467826258 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55681bae5cc0 0x55681bae8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.904+0000 7fcc7dff1740 4 client.25457 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.904+0000 7fcc7dff1740 1 -- 172.21.15.6:0/467826258 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55681ba448b0 con 0x55681bae5cc0 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.904+0000 7fcc73e00640 1 --2- 172.21.15.6:0/467826258 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55681bae5cc0 0x55681bae8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.905+0000 7fcc73e00640 1 --2- 172.21.15.6:0/467826258 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55681bae5cc0 0x55681bae8110 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ae83a071cd2f9ec server_cookie=9aa5aa3a7a018e95 in_seq=0 out_seq=0 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.905+0000 7fcc62000640 10 client.25457 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.906+0000 7fcc62000640 1 -- 172.21.15.6:0/467826258 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55681ba448b0 con 0x55681bae5cc0 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.906+0000 7fcc62000640 10 client.25457 handle_command_reply: tid=0 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.906+0000 7fcc7dff1740 1 client.25457 shutdown 2024-04-26T01:48:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.906+0000 7fcc7dff1740 1 -- 172.21.15.6:0/467826258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bacb140 msgr2=0x55681bad2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.906+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/467826258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bacb140 0x55681bad2e90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009920 tx=0x7fcc5c098e80 comp rx=0 tx=0).stop 2024-04-26T01:48:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.906+0000 7fcc7dff1740 1 -- 172.21.15.6:0/467826258 shutdown_connections 2024-04-26T01:48:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.906+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/467826258 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55681bae5cc0 0x55681bae8110 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.906+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/467826258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bad5a00 0x55681bad33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.906+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/467826258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bacb140 0x55681bad2e90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.906+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/467826258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681bad2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.906+0000 7fcc62000640 0 client.25457 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.906+0000 7fcc7dff1740 1 -- 172.21.15.6:0/467826258 >> 172.21.15.6:0/467826258 conn(0x55681bac6d30 msgr2=0x55681baccab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.907+0000 7fcc7dff1740 1 -- 172.21.15.6:0/467826258 shutdown_connections 2024-04-26T01:48:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.907+0000 7fcc7dff1740 1 -- 172.21.15.6:0/467826258 wait complete. 2024-04-26T01:48:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.907+0000 7fcc7dff1740 20 client.25457 trim_cache size 0 max 16384 2024-04-26T01:48:23.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.925+0000 7fcc7dff1740 1 Processor -- start 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.925+0000 7fcc7dff1740 1 -- start start 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.925+0000 7fcc7dff1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681baca820 0x55681bae1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.925+0000 7fcc7dff1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681bacb140 0x55681bad3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.925+0000 7fcc7dff1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bad5a00 0x55681baed9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.925+0000 7fcc7dff1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcc741b6c00 con 0x55681bacb140 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.925+0000 7fcc7dff1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcc741b4470 con 0x55681bad5a00 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.925+0000 7fcc7dff1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcc741b22c0 con 0x55681baca820 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.926+0000 7fcc73e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681baca820 0x55681bae1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.926+0000 7fcc73e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681baca820 0x55681bae1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58968/0 (socket says 172.21.15.6:58968) 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.926+0000 7fcc73e00640 1 -- 172.21.15.6:0/1303391243 learned_addr learned my addr 172.21.15.6:0/1303391243 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.926+0000 7fcc6be00640 1 --2- 172.21.15.6:0/1303391243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681bacb140 0x55681bad3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.926+0000 7fcc79200640 1 --2- 172.21.15.6:0/1303391243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bad5a00 0x55681baed9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.926+0000 7fcc73e00640 1 -- 172.21.15.6:0/1303391243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bad5a00 msgr2=0x55681baed9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.926+0000 7fcc73e00640 1 --2- 172.21.15.6:0/1303391243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bad5a00 0x55681baed9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.926+0000 7fcc73e00640 1 -- 172.21.15.6:0/1303391243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681bacb140 msgr2=0x55681bad3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.926+0000 7fcc73e00640 1 --2- 172.21.15.6:0/1303391243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681bacb140 0x55681bad3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.926+0000 7fcc73e00640 1 -- 172.21.15.6:0/1303391243 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55681bac7f60 con 0x55681baca820 2024-04-26T01:48:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.927+0000 7fcc73e00640 1 --2- 172.21.15.6:0/1303391243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681baca820 0x55681bae1490 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcc6402aa30 tx=0x7fcc64005900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47e0087c6b1b79d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.927+0000 7fcc68c00640 1 -- 172.21.15.6:0/1303391243 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc64035040 con 0x55681baca820 2024-04-26T01:48:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.927+0000 7fcc68c00640 1 -- 172.21.15.6:0/1303391243 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc64059070 con 0x55681baca820 2024-04-26T01:48:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.927+0000 7fcc68c00640 1 -- 172.21.15.6:0/1303391243 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc64003850 con 0x55681baca820 2024-04-26T01:48:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.928+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1303391243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681baca820 msgr2=0x55681bae1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.928+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1303391243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681baca820 0x55681bae1490 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcc6402aa30 tx=0x7fcc64005900 comp rx=0 tx=0).stop 2024-04-26T01:48:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.928+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1303391243 shutdown_connections 2024-04-26T01:48:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.928+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1303391243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681bacb140 0x55681bad3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.928+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1303391243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bad5a00 0x55681baed9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.928+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1303391243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681baca820 0x55681bae1490 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.928+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1303391243 >> 172.21.15.6:0/1303391243 conn(0x55681bac6d30 msgr2=0x55681bad2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.928+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1303391243 shutdown_connections 2024-04-26T01:48:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.928+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1303391243 wait complete. 2024-04-26T01:48:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.929+0000 7fcc7dff1740 1 -- start start 2024-04-26T01:48:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.929+0000 7fcc7dff1740 1 Processor -- start 2024-04-26T01:48:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.930+0000 7fcc7dff1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681badaf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.930+0000 7fcc7dff1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bacb140 0x55681badb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.930+0000 7fcc7dff1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bad5a00 0x55681bb43200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.930+0000 7fcc7dff1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcc741b2930 con 0x55681baca820 2024-04-26T01:48:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.930+0000 7fcc7dff1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcc741b3da0 con 0x55681bacb140 2024-04-26T01:48:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.930+0000 7fcc7dff1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcc741b3000 con 0x55681bad5a00 2024-04-26T01:48:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.930+0000 7fcc7dff1740 10 client.0 _ll_register_callbacks cb 0x7fcc5c009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.930+0000 7fcc79200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bad5a00 0x55681bb43200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.930+0000 7fcc79200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bad5a00 0x55681bb43200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58972/0 (socket says 172.21.15.6:58972) 2024-04-26T01:48:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.930+0000 7fcc79200640 1 -- 172.21.15.6:0/1769446192 learned_addr learned my addr 172.21.15.6:0/1769446192 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.930+0000 7fcc6be00640 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bacb140 0x55681badb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.930+0000 7fcc73e00640 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681badaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.931+0000 7fcc79200640 1 -- 172.21.15.6:0/1769446192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bacb140 msgr2=0x55681badb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.931+0000 7fcc79200640 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bacb140 0x55681badb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.931+0000 7fcc79200640 1 -- 172.21.15.6:0/1769446192 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 msgr2=0x55681badaf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.931+0000 7fcc79200640 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681badaf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.931+0000 7fcc79200640 1 -- 172.21.15.6:0/1769446192 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55681bb43840 con 0x55681bad5a00 2024-04-26T01:48:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.931+0000 7fcc73e00640 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681badaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:48:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.931+0000 7fcc79200640 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bad5a00 0x55681bb43200 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c012c70 tx=0x7fcc6c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1195eae84c8b3466 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.931+0000 7fcc62000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:48:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.932+0000 7fcc62000640 1 -- 172.21.15.6:0/1769446192 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c004b20 con 0x55681bad5a00 2024-04-26T01:48:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.932+0000 7fcc62000640 1 -- 172.21.15.6:0/1769446192 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fcc6c004cc0 con 0x55681bad5a00 2024-04-26T01:48:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.932+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1769446192 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc64059070 con 0x55681bad5a00 2024-04-26T01:48:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.932+0000 7fcc62000640 1 -- 172.21.15.6:0/1769446192 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcc6c003970 con 0x55681bad5a00 2024-04-26T01:48:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.933+0000 7fcc7dff1740 10 client.25510 fetch_fsmap learned FSMap version 31 2024-04-26T01:48:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.933+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1769446192 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55681bb4be00 con 0x55681bad5a00 2024-04-26T01:48:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.933+0000 7fcc62000640 1 -- 172.21.15.6:0/1769446192 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc64059070 con 0x55681bad5a00 2024-04-26T01:48:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.934+0000 7fcc62000640 1 -- 172.21.15.6:0/1769446192 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fcc6c029410 con 0x55681bad5a00 2024-04-26T01:48:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.934+0000 7fcc7dff1740 10 client.25510 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:48:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.934+0000 7fcc7dff1740 10 client.25510 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:48:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.934+0000 7fcc7dff1740 20 client.25510 populate_metadata read hostname 'smithi006' 2024-04-26T01:48:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.934+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55681bae5cc0 0x55681bb4e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.934+0000 7fcc7dff1740 4 client.25510 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:48:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.934+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1769446192 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55681ba3e320 con 0x55681bae5cc0 2024-04-26T01:48:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.934+0000 7fcc73e00640 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55681bae5cc0 0x55681bb4e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.935+0000 7fcc73e00640 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55681bae5cc0 0x55681bb4e6a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46582d0c748bb9de server_cookie=8bfb1fe6f7ecc8e7 in_seq=0 out_seq=0 2024-04-26T01:48:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.935+0000 7fcc62000640 10 client.25510 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.986+0000 7fcc62000640 1 -- 172.21.15.6:0/1769446192 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55681ba3e320 con 0x55681bae5cc0 2024-04-26T01:48:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.986+0000 7fcc62000640 10 client.25510 handle_command_reply: tid=0 2024-04-26T01:48:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.986+0000 7fcc7dff1740 1 client.25510 shutdown 2024-04-26T01:48:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.986+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1769446192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bad5a00 msgr2=0x55681bb43200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.986+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bad5a00 0x55681bb43200 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c012c70 tx=0x7fcc6c004a40 comp rx=0 tx=0).stop 2024-04-26T01:48:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.986+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1769446192 shutdown_connections 2024-04-26T01:48:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.986+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55681bae5cc0 0x55681bb4e6a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.986+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55681bad5a00 0x55681bb43200 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.986+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681bacb140 0x55681badb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.986+0000 7fcc7dff1740 1 --2- 172.21.15.6:0/1769446192 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55681baca820 0x55681badaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.986+0000 7fcc62000640 0 client.25510 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.986+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1769446192 >> 172.21.15.6:0/1769446192 conn(0x55681bac6d30 msgr2=0x55681bad31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.987+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1769446192 shutdown_connections 2024-04-26T01:48:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.987+0000 7fcc7dff1740 1 -- 172.21.15.6:0/1769446192 wait complete. 2024-04-26T01:48:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.987+0000 7fcc7dff1740 20 client.25510 trim_cache size 0 max 16384 2024-04-26T01:48:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:48:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:48:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:48:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:48:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.988+0000 7fcc7b200640 1 -- 172.21.15.6:0/2085677060 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcc580a3d10 msgr2=0x7fcc580a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.988+0000 7fcc7b200640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcc580a3d10 0x7fcc580a61d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009690 tx=0x7fcc6c009620 comp rx=0 tx=0).stop 2024-04-26T01:48:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.988+0000 7fcc7b200640 1 -- 172.21.15.6:0/2085677060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc74108f60 msgr2=0x7fcc7419f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.988+0000 7fcc7b200640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc74108f60 0x7fcc7419f260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c00ea60 tx=0x7fcc5c00ef20 comp rx=0 tx=0).stop 2024-04-26T01:48:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.988+0000 7fcc7b200640 1 -- 172.21.15.6:0/2085677060 shutdown_connections 2024-04-26T01:48:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.988+0000 7fcc7b200640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcc580a3d10 0x7fcc580a61d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.988+0000 7fcc7b200640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc7410e660 0x7fcc741a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.988+0000 7fcc7b200640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc74108f60 0x7fcc7419f260 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.988+0000 7fcc7b200640 1 --2- 172.21.15.6:0/2085677060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc741085a0 0x7fcc7419ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.988+0000 7fcc7b200640 1 -- 172.21.15.6:0/2085677060 >> 172.21.15.6:0/2085677060 conn(0x7fcc74076460 msgr2=0x7fcc74077d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.988+0000 7fcc7b200640 1 -- 172.21.15.6:0/2085677060 shutdown_connections 2024-04-26T01:48:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:23.988+0000 7fcc7b200640 1 -- 172.21.15.6:0/2085677060 wait complete. 2024-04-26T01:48:24.555 INFO:tasks.workunit.client.0.smithi006.stdout: 1 72572 8.51 MB/sec warmup 48 sec latency 29.074 ms 2024-04-26T01:48:24.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:24 smithi006 ceph-mon[28419]: pgmap v79: 97 pgs: 97 active+clean; 34 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 746 KiB/s wr, 129 op/s 2024-04-26T01:48:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:24 smithi188 ceph-mon[32751]: pgmap v79: 97 pgs: 97 active+clean; 34 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 746 KiB/s wr, 129 op/s 2024-04-26T01:48:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:24 smithi029 ceph-mon[32775]: pgmap v79: 97 pgs: 97 active+clean; 34 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 746 KiB/s wr, 129 op/s 2024-04-26T01:48:24.961 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T01:48:24.961 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-26T01:48:24.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:48:25.455 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:48:25.555 INFO:tasks.workunit.client.0.smithi006.stdout: 1 74523 8.59 MB/sec warmup 49 sec latency 15.399 ms 2024-04-26T01:48:26.555 INFO:tasks.workunit.client.0.smithi006.stdout: 1 75225 8.46 MB/sec warmup 50 sec latency 21.885 ms 2024-04-26T01:48:26.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:26 smithi006 ceph-mon[28419]: pgmap v80: 97 pgs: 97 active+clean; 34 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 746 KiB/s wr, 129 op/s 2024-04-26T01:48:26.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:26 smithi188 ceph-mon[32751]: pgmap v80: 97 pgs: 97 active+clean; 34 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 746 KiB/s wr, 129 op/s 2024-04-26T01:48:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:26 smithi029 ceph-mon[32775]: pgmap v80: 97 pgs: 97 active+clean; 34 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 746 KiB/s wr, 129 op/s 2024-04-26T01:48:27.555 INFO:tasks.workunit.client.0.smithi006.stdout: 1 76921 8.42 MB/sec warmup 51 sec latency 15.885 ms 2024-04-26T01:48:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.730+0000 7f1f86800700 1 -- 172.21.15.6:0/1448917459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80103450 msgr2=0x7f1f8010f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.730+0000 7f1f86800700 1 --2- 172.21.15.6:0/1448917459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80103450 0x7f1f8010f8b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009f90 tx=0x7f1f7800b330 comp rx=0 tx=0).stop 2024-04-26T01:48:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.730+0000 7f1f86800700 1 -- 172.21.15.6:0/1448917459 shutdown_connections 2024-04-26T01:48:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.730+0000 7f1f86800700 1 --2- 172.21.15.6:0/1448917459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80103450 0x7f1f8010f8b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.730+0000 7f1f86800700 1 --2- 172.21.15.6:0/1448917459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f80108a50 0x7f1f80108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.730+0000 7f1f86800700 1 --2- 172.21.15.6:0/1448917459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f80102a90 0x7f1f80102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.730+0000 7f1f86800700 1 -- 172.21.15.6:0/1448917459 >> 172.21.15.6:0/1448917459 conn(0x7f1f80079ba0 msgr2=0x7f1f80079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:27.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.730+0000 7f1f86800700 1 -- 172.21.15.6:0/1448917459 shutdown_connections 2024-04-26T01:48:27.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.730+0000 7f1f86800700 1 -- 172.21.15.6:0/1448917459 wait complete. 2024-04-26T01:48:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.731+0000 7f1f86800700 1 Processor -- start 2024-04-26T01:48:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.732+0000 7f1f86800700 1 -- start start 2024-04-26T01:48:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.732+0000 7f1f86800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f80102a90 0x7f1f8019b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.732+0000 7f1f86800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f80103450 0x7f1f8019b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.732+0000 7f1f86800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80108a50 0x7f1f8019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.732+0000 7f1f86800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1f80112c50 con 0x7f1f80103450 2024-04-26T01:48:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.732+0000 7f1f86800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1f80112ad0 con 0x7f1f80108a50 2024-04-26T01:48:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.732+0000 7f1f86800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1f80112dd0 con 0x7f1f80102a90 2024-04-26T01:48:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.732+0000 7f1f7f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f80102a90 0x7f1f8019b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.732+0000 7f1f7f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f80102a90 0x7f1f8019b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58986/0 (socket says 172.21.15.6:58986) 2024-04-26T01:48:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.732+0000 7f1f7f400700 1 -- 172.21.15.6:0/2474903164 learned_addr learned my addr 172.21.15.6:0/2474903164 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:27.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.733+0000 7f1f7fe00700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80108a50 0x7f1f8019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:27.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.733+0000 7f1f7fe00700 1 -- 172.21.15.6:0/2474903164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f80102a90 msgr2=0x7f1f8019b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:27.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.733+0000 7f1f7fe00700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f80102a90 0x7f1f8019b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:27.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.733+0000 7f1f7fe00700 1 -- 172.21.15.6:0/2474903164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f80103450 msgr2=0x7f1f8019b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:27.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.733+0000 7f1f7fe00700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f80103450 0x7f1f8019b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.733+0000 7f1f7fe00700 1 -- 172.21.15.6:0/2474903164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f801a02e0 con 0x7f1f80108a50 2024-04-26T01:48:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.733+0000 7f1f7f400700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f80102a90 0x7f1f8019b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.733+0000 7f1f7fe00700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80108a50 0x7f1f8019fba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800b6c0 tx=0x7f1f7800cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.734+0000 7f1f7cc00700 1 -- 172.21.15.6:0/2474903164 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f78021070 con 0x7f1f80108a50 2024-04-26T01:48:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.734+0000 7f1f7cc00700 1 -- 172.21.15.6:0/2474903164 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f78007ac0 con 0x7f1f80108a50 2024-04-26T01:48:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.734+0000 7f1f7cc00700 1 -- 172.21.15.6:0/2474903164 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f78005040 con 0x7f1f80108a50 2024-04-26T01:48:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.734+0000 7f1f86800700 1 -- 172.21.15.6:0/2474903164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f801a0570 con 0x7f1f80108a50 2024-04-26T01:48:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.734+0000 7f1f86800700 1 -- 172.21.15.6:0/2474903164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f801a7e50 con 0x7f1f80108a50 2024-04-26T01:48:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.735+0000 7f1f86800700 1 -- 172.21.15.6:0/2474903164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f80069730 con 0x7f1f80108a50 2024-04-26T01:48:27.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.737+0000 7f1f7cc00700 1 -- 172.21.15.6:0/2474903164 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f1f78024070 con 0x7f1f80108a50 2024-04-26T01:48:27.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.739+0000 7f1f7cc00700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1f640a3e30 0x7f1f640a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:27.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.739+0000 7f1f7cc00700 1 -- 172.21.15.6:0/2474903164 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f1f7800f020 con 0x7f1f80108a50 2024-04-26T01:48:27.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.739+0000 7f1f7f400700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1f640a3e30 0x7f1f640a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:27.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.740+0000 7f1f7f400700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1f640a3e30 0x7f1f640a62f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1f70011550 tx=0x7f1f70009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:27.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:27.741+0000 7f1f7cc00700 1 -- 172.21.15.6:0/2474903164 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f7802b020 con 0x7f1f80108a50 2024-04-26T01:48:28.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.051+0000 7f1f86800700 1 -- 172.21.15.6:0/2474903164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f801a80e0 con 0x7f1f80108a50 2024-04-26T01:48:28.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.052+0000 7f1f7cc00700 1 -- 172.21.15.6:0/2474903164 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11853 (secure 0 0 0) 0x7f1f7809e600 con 0x7f1f80108a50 2024-04-26T01:48:28.050 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:48:28.050 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:47:59.436110+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25474":{"gid":25474,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1633748308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1633748308},{"type":"v1","addr":"172.21.15.188:6835","nonce":1633748308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:48:28.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.056+0000 7f1f86800700 1 -- 172.21.15.6:0/2474903164 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1f640a3e30 msgr2=0x7f1f640a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.056+0000 7f1f86800700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1f640a3e30 0x7f1f640a62f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1f70011550 tx=0x7f1f70009260 comp rx=0 tx=0).stop 2024-04-26T01:48:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.056+0000 7f1f86800700 1 -- 172.21.15.6:0/2474903164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80108a50 msgr2=0x7f1f8019fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.056+0000 7f1f86800700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80108a50 0x7f1f8019fba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800b6c0 tx=0x7f1f7800cc30 comp rx=0 tx=0).stop 2024-04-26T01:48:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.056+0000 7f1f86800700 1 -- 172.21.15.6:0/2474903164 shutdown_connections 2024-04-26T01:48:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.056+0000 7f1f86800700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1f640a3e30 0x7f1f640a62f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.056+0000 7f1f86800700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80108a50 0x7f1f8019fba0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.056+0000 7f1f86800700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f80102a90 0x7f1f8019b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:28.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.056+0000 7f1f86800700 1 --2- 172.21.15.6:0/2474903164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f80103450 0x7f1f8019b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:28.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.056+0000 7f1f86800700 1 -- 172.21.15.6:0/2474903164 >> 172.21.15.6:0/2474903164 conn(0x7f1f80079ba0 msgr2=0x7f1f8010df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:28.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.057+0000 7f1f86800700 1 -- 172.21.15.6:0/2474903164 shutdown_connections 2024-04-26T01:48:28.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:28.057+0000 7f1f86800700 1 -- 172.21.15.6:0/2474903164 wait complete. 2024-04-26T01:48:28.056 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 31 2024-04-26T01:48:28.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:28 smithi006 ceph-mon[28419]: pgmap v81: 97 pgs: 97 active+clean; 33 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 225 op/s 2024-04-26T01:48:28.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:28 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2474903164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:48:28.555 INFO:tasks.workunit.client.0.smithi006.stdout: 1 78366 8.49 MB/sec warmup 52 sec latency 15.459 ms 2024-04-26T01:48:28.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:28 smithi188 ceph-mon[32751]: pgmap v81: 97 pgs: 97 active+clean; 33 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 225 op/s 2024-04-26T01:48:28.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:28 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2474903164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:48:28.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:28 smithi029 ceph-mon[32775]: pgmap v81: 97 pgs: 97 active+clean; 33 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 225 op/s 2024-04-26T01:48:28.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:28 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2474903164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:48:28.960 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-26T01:48:28.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-26T01:48:29.432 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:48:29.555 INFO:tasks.workunit.client.0.smithi006.stdout: 1 79146 8.37 MB/sec warmup 53 sec latency 81.298 ms 2024-04-26T01:48:30.555 INFO:tasks.workunit.client.0.smithi006.stdout: 1 80804 8.38 MB/sec warmup 54 sec latency 17.892 ms 2024-04-26T01:48:30.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:30 smithi006 ceph-mon[28419]: pgmap v82: 97 pgs: 97 active+clean; 33 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 762 KiB/s wr, 160 op/s 2024-04-26T01:48:30.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:30 smithi188 ceph-mon[32751]: pgmap v82: 97 pgs: 97 active+clean; 33 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 762 KiB/s wr, 160 op/s 2024-04-26T01:48:30.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:30 smithi029 ceph-mon[32775]: pgmap v82: 97 pgs: 97 active+clean; 33 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 762 KiB/s wr, 160 op/s 2024-04-26T01:48:31.555 INFO:tasks.workunit.client.0.smithi006.stdout: 1 82005 8.38 MB/sec warmup 55 sec latency 31.266 ms 2024-04-26T01:48:32.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.494+0000 7fda1b800700 1 -- 172.21.15.6:0/1669861762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda14075590 msgr2=0x7fda1410ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:32.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.494+0000 7fda1b800700 1 --2- 172.21.15.6:0/1669861762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda14075590 0x7fda1410ef60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fda00009990 tx=0x7fda00009ca0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.494+0000 7fda1b800700 1 -- 172.21.15.6:0/1669861762 shutdown_connections 2024-04-26T01:48:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.494+0000 7fda1b800700 1 --2- 172.21.15.6:0/1669861762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda1410f4a0 0x7fda14111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.494+0000 7fda1b800700 1 --2- 172.21.15.6:0/1669861762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda14074c70 0x7fda14075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.494+0000 7fda1b800700 1 --2- 172.21.15.6:0/1669861762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda14075590 0x7fda1410ef60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.494+0000 7fda1b800700 1 -- 172.21.15.6:0/1669861762 >> 172.21.15.6:0/1669861762 conn(0x7fda14079b10 msgr2=0x7fda14079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.495+0000 7fda1b800700 1 -- 172.21.15.6:0/1669861762 shutdown_connections 2024-04-26T01:48:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.495+0000 7fda1b800700 1 -- 172.21.15.6:0/1669861762 wait complete. 2024-04-26T01:48:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.496+0000 7fda1b800700 1 Processor -- start 2024-04-26T01:48:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.496+0000 7fda1b800700 1 -- start start 2024-04-26T01:48:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.497+0000 7fda1b800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda14074c70 0x7fda1419b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.497+0000 7fda1b800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda14075590 0x7fda1419b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.497+0000 7fda1b800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda1410f4a0 0x7fda1419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.495 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:32 smithi006 ceph-mon[28419]: pgmap v83: 97 pgs: 97 active+clean; 37 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 224 op/s 2024-04-26T01:48:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.497+0000 7fda1b800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fda14114e70 con 0x7fda1410f4a0 2024-04-26T01:48:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.497+0000 7fda1b800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fda14114cf0 con 0x7fda14075590 2024-04-26T01:48:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.497+0000 7fda1b800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fda14114ff0 con 0x7fda14074c70 2024-04-26T01:48:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.497+0000 7fda18e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda14074c70 0x7fda1419b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.497+0000 7fda18e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda14074c70 0x7fda1419b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45950/0 (socket says 172.21.15.6:45950) 2024-04-26T01:48:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.497+0000 7fda18e00700 1 -- 172.21.15.6:0/1579308323 learned_addr learned my addr 172.21.15.6:0/1579308323 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.497+0000 7fda13e00700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda14075590 0x7fda1419b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.498+0000 7fda18e00700 1 -- 172.21.15.6:0/1579308323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda14075590 msgr2=0x7fda1419b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.498+0000 7fda18e00700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda14075590 0x7fda1419b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.498+0000 7fda19800700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda1410f4a0 0x7fda1419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.498+0000 7fda18e00700 1 -- 172.21.15.6:0/1579308323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda1410f4a0 msgr2=0x7fda1419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.498+0000 7fda18e00700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda1410f4a0 0x7fda1419fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.498+0000 7fda18e00700 1 -- 172.21.15.6:0/1579308323 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda141a0290 con 0x7fda14074c70 2024-04-26T01:48:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.498+0000 7fda18e00700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda14074c70 0x7fda1419b280 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fda0800e9c0 tx=0x7fda0800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.498+0000 7fda11600700 1 -- 172.21.15.6:0/1579308323 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0800cbd0 con 0x7fda14074c70 2024-04-26T01:48:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.499+0000 7fda11600700 1 -- 172.21.15.6:0/1579308323 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda0800cd70 con 0x7fda14074c70 2024-04-26T01:48:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.499+0000 7fda11600700 1 -- 172.21.15.6:0/1579308323 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08005610 con 0x7fda14074c70 2024-04-26T01:48:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.499+0000 7fda1b800700 1 -- 172.21.15.6:0/1579308323 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda141a0580 con 0x7fda14074c70 2024-04-26T01:48:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.499+0000 7fda1b800700 1 -- 172.21.15.6:0/1579308323 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda141a7e00 con 0x7fda14074c70 2024-04-26T01:48:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.502+0000 7fda11600700 1 -- 172.21.15.6:0/1579308323 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fda08004750 con 0x7fda14074c70 2024-04-26T01:48:32.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.504+0000 7fda11600700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd9fc0a3ed0 0x7fd9fc0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.504+0000 7fda260fdb80 1 Processor -- start 2024-04-26T01:48:32.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.504+0000 7fda11600700 1 -- 172.21.15.6:0/1579308323 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fda080d0b60 con 0x7fda14074c70 2024-04-26T01:48:32.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.504+0000 7fda260fdb80 1 -- start start 2024-04-26T01:48:32.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.504+0000 7fda13e00700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd9fc0a3ed0 0x7fd9fc0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.504+0000 7fda260fdb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x563714935d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.504+0000 7fda260fdb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637149362d0 0x563714940c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.504+0000 7fda260fdb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563714941180 0x563714943570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.504+0000 7fda260fdb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x563714889020 con 0x563714941180 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.504+0000 7fda260fdb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56371482f8c0 con 0x5637149362d0 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.504+0000 7fda260fdb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563714884fe0 con 0x5637149359b0 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.505+0000 7fda18e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x563714935d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.505+0000 7fda18e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x563714935d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45960/0 (socket says 172.21.15.6:45960) 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.505+0000 7fda18e00700 1 -- 172.21.15.6:0/4082714096 learned_addr learned my addr 172.21.15.6:0/4082714096 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.505+0000 7fda13e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637149362d0 0x563714940c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.505+0000 7fda18e00700 1 -- 172.21.15.6:0/4082714096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637149362d0 msgr2=0x563714940c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.505+0000 7fda18e00700 1 --2- 172.21.15.6:0/4082714096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637149362d0 0x563714940c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.505+0000 7fda13e00700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd9fc0a3ed0 0x7fd9fc0a6390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda141111a0 tx=0x7fda0000b260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.505+0000 7fda18e00700 1 -- 172.21.15.6:0/4082714096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563714941180 msgr2=0x563714943570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.505+0000 7fda18e00700 1 --2- 172.21.15.6:0/4082714096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563714941180 0x563714943570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.505+0000 7fda18e00700 1 -- 172.21.15.6:0/4082714096 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563714825810 con 0x5637149359b0 2024-04-26T01:48:32.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.505+0000 7fda18e00700 1 --2- 172.21.15.6:0/4082714096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x563714935d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fda080a51c0 tx=0x7fda0809c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f58da129746ca082 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.506+0000 7fda06000700 1 -- 172.21.15.6:0/4082714096 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080bd0d0 con 0x5637149359b0 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.506+0000 7fda06000700 1 -- 172.21.15.6:0/4082714096 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda080a0e30 con 0x5637149359b0 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.506+0000 7fda06000700 1 -- 172.21.15.6:0/4082714096 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080a5260 con 0x5637149359b0 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.507+0000 7fda260fdb80 1 -- 172.21.15.6:0/4082714096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 msgr2=0x563714935d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.507+0000 7fda260fdb80 1 --2- 172.21.15.6:0/4082714096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x563714935d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fda080a51c0 tx=0x7fda0809c4b0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.507+0000 7fda260fdb80 1 -- 172.21.15.6:0/4082714096 shutdown_connections 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.507+0000 7fda260fdb80 1 --2- 172.21.15.6:0/4082714096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563714941180 0x563714943570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.507+0000 7fda260fdb80 1 --2- 172.21.15.6:0/4082714096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x563714935d90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.507+0000 7fda260fdb80 1 --2- 172.21.15.6:0/4082714096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637149362d0 0x563714940c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.507+0000 7fda260fdb80 1 -- 172.21.15.6:0/4082714096 >> 172.21.15.6:0/4082714096 conn(0x563714864730 msgr2=0x563714931e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.507+0000 7fda260fdb80 1 -- 172.21.15.6:0/4082714096 shutdown_connections 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.507+0000 7fda260fdb80 1 -- 172.21.15.6:0/4082714096 wait complete. 2024-04-26T01:48:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.508+0000 7fda260fdb80 1 -- start start 2024-04-26T01:48:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.508+0000 7fda260fdb80 1 Processor -- start 2024-04-26T01:48:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda260fdb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149359b0 0x563714940970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda260fdb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637149362d0 0x56371493dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda260fdb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563714941180 0x56371493e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda260fdb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x563714943ca0 con 0x5637149359b0 2024-04-26T01:48:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda260fdb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5637148429f0 con 0x5637149362d0 2024-04-26T01:48:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda260fdb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563714943e20 con 0x563714941180 2024-04-26T01:48:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda260fdb80 10 client.0 _ll_register_callbacks cb 0x5637148218b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda13e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637149362d0 0x56371493dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda18e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149359b0 0x563714940970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda13e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637149362d0 0x56371493dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36802/0 (socket says 172.21.15.6:36802) 2024-04-26T01:48:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda19800700 1 --2- 172.21.15.6:0/1734101519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563714941180 0x56371493e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda13e00700 1 -- 172.21.15.6:0/1734101519 learned_addr learned my addr 172.21.15.6:0/1734101519 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda13e00700 1 -- 172.21.15.6:0/1734101519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563714941180 msgr2=0x56371493e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda13e00700 1 --2- 172.21.15.6:0/1734101519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563714941180 0x56371493e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda13e00700 1 -- 172.21.15.6:0/1734101519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149359b0 msgr2=0x563714940970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda13e00700 1 --2- 172.21.15.6:0/1734101519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149359b0 0x563714940970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.509+0000 7fda13e00700 1 -- 172.21.15.6:0/1734101519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637147d6a60 con 0x5637149362d0 2024-04-26T01:48:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.510+0000 7fda13e00700 1 --2- 172.21.15.6:0/1734101519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637149362d0 0x56371493dc70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fda00015000 tx=0x7fda00009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7840ceaa6c7ec061 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.510+0000 7fd9faa00700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:48:32.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.510+0000 7fd9faa00700 1 -- 172.21.15.6:0/1734101519 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00033020 con 0x5637149362d0 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.510+0000 7fda260fdb80 1 -- 172.21.15.6:0/1734101519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda080a0e30 con 0x5637149362d0 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.510+0000 7fd9faa00700 1 -- 172.21.15.6:0/1734101519 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fda0000f460 con 0x5637149362d0 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.510+0000 7fd9faa00700 1 -- 172.21.15.6:0/1734101519 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fda0003e050 con 0x5637149362d0 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.510+0000 7fda260fdb80 10 client.25481 fetch_fsmap learned FSMap version 31 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.510+0000 7fda260fdb80 1 -- 172.21.15.6:0/1734101519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56371481fc80 con 0x5637149362d0 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.511+0000 7fd9faa00700 1 -- 172.21.15.6:0/1734101519 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda080a0e30 con 0x5637149362d0 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.511+0000 7fd9faa00700 1 -- 172.21.15.6:0/1734101519 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fda0002f4b0 con 0x5637149362d0 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.511+0000 7fda260fdb80 10 client.25481 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.511+0000 7fda260fdb80 10 client.25481 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.511+0000 7fda260fdb80 20 client.25481 populate_metadata read hostname 'smithi006' 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.511+0000 7fda260fdb80 1 --2- 172.21.15.6:0/1734101519 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563714950a30 0x563714952e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.511+0000 7fda260fdb80 4 client.25481 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.511+0000 7fda260fdb80 1 -- 172.21.15.6:0/1734101519 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563714896810 con 0x563714950a30 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.511+0000 7fda18e00700 1 --2- 172.21.15.6:0/1734101519 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563714950a30 0x563714952e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.512+0000 7fda18e00700 1 --2- 172.21.15.6:0/1734101519 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563714950a30 0x563714952e20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12c6145249a86c42 server_cookie=4bb651c32e8bbfa0 in_seq=0 out_seq=0 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.512+0000 7fd9faa00700 10 client.25481 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.512+0000 7fd9faa00700 1 -- 172.21.15.6:0/1734101519 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563714896810 con 0x563714950a30 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.512+0000 7fd9faa00700 10 client.25481 handle_command_reply: tid=0 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.512+0000 7fda260fdb80 1 client.25481 shutdown 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.513+0000 7fda260fdb80 1 -- 172.21.15.6:0/1734101519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637149362d0 msgr2=0x56371493dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.513+0000 7fda260fdb80 1 --2- 172.21.15.6:0/1734101519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637149362d0 0x56371493dc70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fda00015000 tx=0x7fda00009310 comp rx=0 tx=0).stop 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.513+0000 7fda260fdb80 1 -- 172.21.15.6:0/1734101519 shutdown_connections 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.513+0000 7fda260fdb80 1 --2- 172.21.15.6:0/1734101519 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563714950a30 0x563714952e20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.513+0000 7fda260fdb80 1 --2- 172.21.15.6:0/1734101519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563714941180 0x56371493e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.513+0000 7fda260fdb80 1 --2- 172.21.15.6:0/1734101519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149359b0 0x563714940970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.513+0000 7fda260fdb80 1 --2- 172.21.15.6:0/1734101519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637149362d0 0x56371493dc70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.513+0000 7fda260fdb80 1 -- 172.21.15.6:0/1734101519 >> 172.21.15.6:0/1734101519 conn(0x563714864730 msgr2=0x563714931a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:32.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.513+0000 7fd9faa00700 0 client.25481 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:32.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.513+0000 7fda260fdb80 1 -- 172.21.15.6:0/1734101519 shutdown_connections 2024-04-26T01:48:32.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.513+0000 7fda260fdb80 1 -- 172.21.15.6:0/1734101519 wait complete. 2024-04-26T01:48:32.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.513+0000 7fda260fdb80 20 client.25481 trim_cache size 0 max 16384 2024-04-26T01:48:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.535+0000 7fda260fdb80 1 Processor -- start 2024-04-26T01:48:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.535+0000 7fda260fdb80 1 -- start start 2024-04-26T01:48:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.536+0000 7fda260fdb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x56371494cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.536+0000 7fda260fdb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149362d0 0x56371493fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.536+0000 7fda260fdb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 0x5637149490b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.536+0000 7fda260fdb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fda141b1e60 con 0x5637149362d0 2024-04-26T01:48:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.536+0000 7fda260fdb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fda141af690 con 0x563714941180 2024-04-26T01:48:32.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.536+0000 7fda260fdb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fda141ad4e0 con 0x5637149359b0 2024-04-26T01:48:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.536+0000 7fda19800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 0x5637149490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.536+0000 7fda13e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149362d0 0x56371493fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.536+0000 7fda19800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 0x5637149490b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36808/0 (socket says 172.21.15.6:36808) 2024-04-26T01:48:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.536+0000 7fda19800700 1 -- 172.21.15.6:0/868478957 learned_addr learned my addr 172.21.15.6:0/868478957 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.536+0000 7fda18e00700 1 --2- 172.21.15.6:0/868478957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x56371494cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.537+0000 7fda19800700 1 -- 172.21.15.6:0/868478957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 msgr2=0x56371494cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.537+0000 7fda19800700 1 --2- 172.21.15.6:0/868478957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x56371494cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.537+0000 7fda19800700 1 -- 172.21.15.6:0/868478957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149362d0 msgr2=0x56371493fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:32.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.537+0000 7fda19800700 1 --2- 172.21.15.6:0/868478957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149362d0 0x56371493fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.537+0000 7fda19800700 1 -- 172.21.15.6:0/868478957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637149366b0 con 0x563714941180 2024-04-26T01:48:32.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.537+0000 7fda13e00700 1 --2- 172.21.15.6:0/868478957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149362d0 0x56371493fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:32.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.537+0000 7fda19800700 1 --2- 172.21.15.6:0/868478957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 0x5637149490b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009460 tx=0x7fda0c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de50e5b928f3c8a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:32.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.537+0000 7fda05600700 1 -- 172.21.15.6:0/868478957 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c01c070 con 0x563714941180 2024-04-26T01:48:32.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.537+0000 7fda05600700 1 -- 172.21.15.6:0/868478957 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda0c017b30 con 0x563714941180 2024-04-26T01:48:32.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.537+0000 7fda05600700 1 -- 172.21.15.6:0/868478957 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c016c80 con 0x563714941180 2024-04-26T01:48:32.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.538+0000 7fda260fdb80 1 -- 172.21.15.6:0/868478957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 msgr2=0x5637149490b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.538+0000 7fda260fdb80 1 --2- 172.21.15.6:0/868478957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 0x5637149490b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009460 tx=0x7fda0c009920 comp rx=0 tx=0).stop 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.538+0000 7fda260fdb80 1 -- 172.21.15.6:0/868478957 shutdown_connections 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.538+0000 7fda260fdb80 1 --2- 172.21.15.6:0/868478957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 0x5637149490b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.538+0000 7fda260fdb80 1 --2- 172.21.15.6:0/868478957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x56371494cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.538+0000 7fda260fdb80 1 --2- 172.21.15.6:0/868478957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149362d0 0x56371493fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.538+0000 7fda260fdb80 1 -- 172.21.15.6:0/868478957 >> 172.21.15.6:0/868478957 conn(0x563714864730 msgr2=0x56371493f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.538+0000 7fda260fdb80 1 -- 172.21.15.6:0/868478957 shutdown_connections 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.539+0000 7fda260fdb80 1 -- 172.21.15.6:0/868478957 wait complete. 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.539+0000 7fda260fdb80 1 -- start start 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.539+0000 7fda260fdb80 1 Processor -- start 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.540+0000 7fda260fdb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x563714947c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.540+0000 7fda260fdb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149362d0 0x563714948160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.540+0000 7fda260fdb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 0x5637149486a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.540+0000 7fda260fdb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fda141adb50 con 0x5637149362d0 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.540+0000 7fda260fdb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fda141aefc0 con 0x563714941180 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.540+0000 7fda260fdb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fda141ae220 con 0x5637149359b0 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.540+0000 7fda260fdb80 10 client.0 _ll_register_callbacks cb 0x5637148218b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.540+0000 7fda18e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x563714947c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.540+0000 7fda18e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x563714947c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45982/0 (socket says 172.21.15.6:45982) 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.540+0000 7fda18e00700 1 -- 172.21.15.6:0/456878827 learned_addr learned my addr 172.21.15.6:0/456878827 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.540+0000 7fda19800700 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 0x5637149486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.540+0000 7fda13e00700 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149362d0 0x563714948160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.541+0000 7fda19800700 1 -- 172.21.15.6:0/456878827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 msgr2=0x563714947c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.541+0000 7fda19800700 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x563714947c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.541+0000 7fda19800700 1 -- 172.21.15.6:0/456878827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149362d0 msgr2=0x563714948160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.541+0000 7fda19800700 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149362d0 0x563714948160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.541+0000 7fda19800700 1 -- 172.21.15.6:0/456878827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637149b70c0 con 0x563714941180 2024-04-26T01:48:32.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.541+0000 7fda18e00700 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x563714947c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:32.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.541+0000 7fda19800700 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 0x5637149486a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fda0c01a040 tx=0x7fda0c004d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bfc4fdc73cdac11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:32.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.541+0000 7fd9faa00700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:48:32.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.541+0000 7fd9faa00700 1 -- 172.21.15.6:0/456878827 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c024020 con 0x563714941180 2024-04-26T01:48:32.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.542+0000 7fd9faa00700 1 -- 172.21.15.6:0/456878827 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fda0c003c00 con 0x563714941180 2024-04-26T01:48:32.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.542+0000 7fda260fdb80 1 -- 172.21.15.6:0/456878827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda0c017b30 con 0x563714941180 2024-04-26T01:48:32.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.542+0000 7fd9faa00700 1 -- 172.21.15.6:0/456878827 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fda0c01c040 con 0x563714941180 2024-04-26T01:48:32.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.542+0000 7fda260fdb80 10 client.25487 fetch_fsmap learned FSMap version 31 2024-04-26T01:48:32.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.542+0000 7fda260fdb80 1 -- 172.21.15.6:0/456878827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5637149b8380 con 0x563714941180 2024-04-26T01:48:32.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.542+0000 7fd9faa00700 1 -- 172.21.15.6:0/456878827 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda0c017b30 con 0x563714941180 2024-04-26T01:48:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.543+0000 7fd9faa00700 1 -- 172.21.15.6:0/456878827 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fda0c01f030 con 0x563714941180 2024-04-26T01:48:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.543+0000 7fda260fdb80 10 client.25487 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:48:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.543+0000 7fda260fdb80 10 client.25487 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:48:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.543+0000 7fda260fdb80 20 client.25487 populate_metadata read hostname 'smithi006' 2024-04-26T01:48:32.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.543+0000 7fda260fdb80 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5637149b89c0 0x5637149bade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:32.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.543+0000 7fda260fdb80 4 client.25487 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-26T01:48:32.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.543+0000 7fda260fdb80 1 -- 172.21.15.6:0/456878827 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x5637148908b0 con 0x5637149b89c0 2024-04-26T01:48:32.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.543+0000 7fda18e00700 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5637149b89c0 0x5637149bade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:32.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.544+0000 7fda18e00700 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5637149b89c0 0x5637149bade0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb027d75b4ef27a9 server_cookie=39840e35b3727be8 in_seq=0 out_seq=0 2024-04-26T01:48:32.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.544+0000 7fd9faa00700 10 client.25487 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:32.555 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83230 8.32 MB/sec warmup 56 sec latency 39.823 ms 2024-04-26T01:48:32.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.611+0000 7fd9faa00700 1 -- 172.21.15.6:0/456878827 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5637148908b0 con 0x5637149b89c0 2024-04-26T01:48:32.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.611+0000 7fd9faa00700 10 client.25487 handle_command_reply: tid=0 2024-04-26T01:48:32.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.611+0000 7fda260fdb80 1 client.25487 shutdown 2024-04-26T01:48:32.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.612+0000 7fda260fdb80 1 -- 172.21.15.6:0/456878827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 msgr2=0x5637149486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:32.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.612+0000 7fda260fdb80 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 0x5637149486a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fda0c01a040 tx=0x7fda0c004d80 comp rx=0 tx=0).stop 2024-04-26T01:48:32.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.612+0000 7fda260fdb80 1 -- 172.21.15.6:0/456878827 shutdown_connections 2024-04-26T01:48:32.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.612+0000 7fda260fdb80 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563714941180 0x5637149486a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.612+0000 7fda260fdb80 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5637149b89c0 0x5637149bade0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.612+0000 7fda260fdb80 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637149359b0 0x563714947c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.612+0000 7fda260fdb80 1 --2- 172.21.15.6:0/456878827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637149362d0 0x563714948160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.612+0000 7fda260fdb80 1 -- 172.21.15.6:0/456878827 >> 172.21.15.6:0/456878827 conn(0x563714864730 msgr2=0x56371493f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:32.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.612+0000 7fd9faa00700 0 client.25487 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:32.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.612+0000 7fda260fdb80 1 -- 172.21.15.6:0/456878827 shutdown_connections 2024-04-26T01:48:32.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.613+0000 7fda260fdb80 1 -- 172.21.15.6:0/456878827 wait complete. 2024-04-26T01:48:32.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.613+0000 7fda260fdb80 20 client.25487 trim_cache size 0 max 16384 2024-04-26T01:48:32.612 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:48:32.612 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 31, 2024-04-26T01:48:32.612 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 6, 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "c54a4666": { 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 6, 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 90.0, 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:48:32.613 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:48:32.614 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:48:32.614 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:48:32.614 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:48:32.614 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:48:32.614 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:48:32.614 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:48:32.614 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:48:32.614 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:48:32.614 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:48:32.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.614+0000 7fda1b800700 1 -- 172.21.15.6:0/1579308323 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd9fc0a3ed0 msgr2=0x7fd9fc0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:32.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.614+0000 7fda1b800700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd9fc0a3ed0 0x7fd9fc0a6390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda141111a0 tx=0x7fda0000b260 comp rx=0 tx=0).stop 2024-04-26T01:48:32.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.614+0000 7fda1b800700 1 -- 172.21.15.6:0/1579308323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda14074c70 msgr2=0x7fda1419b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:32.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.614+0000 7fda1b800700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda14074c70 0x7fda1419b280 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fda0800e9c0 tx=0x7fda0800ecd0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.614+0000 7fda1b800700 1 -- 172.21.15.6:0/1579308323 shutdown_connections 2024-04-26T01:48:32.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.614+0000 7fda1b800700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd9fc0a3ed0 0x7fd9fc0a6390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.614+0000 7fda1b800700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda1410f4a0 0x7fda1419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.614+0000 7fda1b800700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda14074c70 0x7fda1419b280 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.614+0000 7fda1b800700 1 --2- 172.21.15.6:0/1579308323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda14075590 0x7fda1419b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:32.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.614+0000 7fda1b800700 1 -- 172.21.15.6:0/1579308323 >> 172.21.15.6:0/1579308323 conn(0x7fda14079b10 msgr2=0x7fda140ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:32.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.615+0000 7fda1b800700 1 -- 172.21.15.6:0/1579308323 shutdown_connections 2024-04-26T01:48:32.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:48:32.615+0000 7fda1b800700 1 -- 172.21.15.6:0/1579308323 wait complete. 2024-04-26T01:48:32.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:32 smithi188 ceph-mon[32751]: pgmap v83: 97 pgs: 97 active+clean; 37 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 224 op/s 2024-04-26T01:48:32.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:32 smithi029 ceph-mon[32775]: pgmap v83: 97 pgs: 97 active+clean; 37 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 224 op/s 2024-04-26T01:48:33.546 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c54a4666', quiesce took 8.6 seconds. Will release after: 30.0 2024-04-26T01:48:33.556 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 8.18 MB/sec warmup 57 sec latency 948.366 ms 2024-04-26T01:48:34.556 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 8.03 MB/sec warmup 58 sec latency 1948.493 ms 2024-04-26T01:48:34.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:34 smithi006 ceph-mon[28419]: pgmap v84: 97 pgs: 97 active+clean; 37 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 800 KiB/s wr, 160 op/s 2024-04-26T01:48:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:34 smithi188 ceph-mon[32751]: pgmap v84: 97 pgs: 97 active+clean; 37 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 800 KiB/s wr, 160 op/s 2024-04-26T01:48:34.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:34 smithi029 ceph-mon[32775]: pgmap v84: 97 pgs: 97 active+clean; 37 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 800 KiB/s wr, 160 op/s 2024-04-26T01:48:35.556 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 7.90 MB/sec warmup 59 sec latency 2948.584 ms 2024-04-26T01:48:36.556 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 7.77 MB/sec warmup 60 sec latency 3948.744 ms 2024-04-26T01:48:36.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:36 smithi006 ceph-mon[28419]: pgmap v85: 97 pgs: 97 active+clean; 37 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 800 KiB/s wr, 160 op/s 2024-04-26T01:48:36.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:36 smithi188 ceph-mon[32751]: pgmap v85: 97 pgs: 97 active+clean; 37 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 800 KiB/s wr, 160 op/s 2024-04-26T01:48:36.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:36 smithi029 ceph-mon[32775]: pgmap v85: 97 pgs: 97 active+clean; 37 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 800 KiB/s wr, 160 op/s 2024-04-26T01:48:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.138+0000 7ff407a00640 1 -- 172.21.15.6:0/1186448049 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff400074c20 msgr2=0x7ff40010e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.138+0000 7ff407a00640 1 --2- 172.21.15.6:0/1186448049 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff400074c20 0x7ff40010e2c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009900 tx=0x7ff3ec02a480 comp rx=0 tx=0).stop 2024-04-26T01:48:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.139+0000 7ff407a00640 1 -- 172.21.15.6:0/1186448049 shutdown_connections 2024-04-26T01:48:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.139+0000 7ff407a00640 1 --2- 172.21.15.6:0/1186448049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff40010e800 0x7ff400110bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.139+0000 7ff407a00640 1 --2- 172.21.15.6:0/1186448049 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff400074c20 0x7ff40010e2c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.139+0000 7ff407a00640 1 --2- 172.21.15.6:0/1186448049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff400074300 0x7ff4000746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.139+0000 7ff407a00640 1 -- 172.21.15.6:0/1186448049 >> 172.21.15.6:0/1186448049 conn(0x7ff4000fda10 msgr2=0x7ff4000ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.139+0000 7ff407a00640 1 -- 172.21.15.6:0/1186448049 shutdown_connections 2024-04-26T01:48:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.139+0000 7ff407a00640 1 -- 172.21.15.6:0/1186448049 wait complete. 2024-04-26T01:48:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.140+0000 7ff407a00640 1 Processor -- start 2024-04-26T01:48:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.140+0000 7ff407a00640 1 -- start start 2024-04-26T01:48:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.140+0000 7ff407a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff400074300 0x7ff4001a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.140+0000 7ff407a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff400074c20 0x7ff4001a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.140+0000 7ff407a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff40010e800 0x7ff40019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.140+0000 7ff407a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff400114410 con 0x7ff40010e800 2024-04-26T01:48:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.140+0000 7ff407a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff400114290 con 0x7ff400074300 2024-04-26T01:48:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.140+0000 7ff407a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff400114590 con 0x7ff400074c20 2024-04-26T01:48:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.141+0000 7ff405a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff40010e800 0x7ff40019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.141+0000 7ff405000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff400074300 0x7ff4001a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.141+0000 7ff3ffe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff400074c20 0x7ff4001a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.141+0000 7ff405000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff400074300 0x7ff4001a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36848/0 (socket says 172.21.15.6:36848) 2024-04-26T01:48:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.141+0000 7ff3ffe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff400074c20 0x7ff4001a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:46002/0 (socket says 172.21.15.6:46002) 2024-04-26T01:48:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.141+0000 7ff405000640 1 -- 172.21.15.6:0/2191839991 learned_addr learned my addr 172.21.15.6:0/2191839991 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.141+0000 7ff405a00640 1 -- 172.21.15.6:0/2191839991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff400074c20 msgr2=0x7ff4001a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.141+0000 7ff405a00640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff400074c20 0x7ff4001a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.141+0000 7ff405a00640 1 -- 172.21.15.6:0/2191839991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff400074300 msgr2=0x7ff4001a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.141+0000 7ff405a00640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff400074300 0x7ff4001a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.141+0000 7ff405a00640 1 -- 172.21.15.6:0/2191839991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40019afa0 con 0x7ff40010e800 2024-04-26T01:48:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.141+0000 7ff405000640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff400074300 0x7ff4001a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.142+0000 7ff405a00640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff40010e800 0x7ff40019a7d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009fd0 tx=0x7ff3f800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.142+0000 7ff3fd600640 1 -- 172.21.15.6:0/2191839991 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f800ce20 con 0x7ff40010e800 2024-04-26T01:48:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.142+0000 7ff407a00640 1 -- 172.21.15.6:0/2191839991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff40019b290 con 0x7ff40010e800 2024-04-26T01:48:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.142+0000 7ff3fd600640 1 -- 172.21.15.6:0/2191839991 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f8005020 con 0x7ff40010e800 2024-04-26T01:48:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.142+0000 7ff3fd600640 1 -- 172.21.15.6:0/2191839991 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8018670 con 0x7ff40010e800 2024-04-26T01:48:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.142+0000 7ff407a00640 1 -- 172.21.15.6:0/2191839991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4001a74b0 con 0x7ff40010e800 2024-04-26T01:48:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.145+0000 7ff40a78d740 1 Processor -- start 2024-04-26T01:48:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.146+0000 7ff3fd600640 1 -- 172.21.15.6:0/2191839991 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7ff3f80054d0 con 0x7ff40010e800 2024-04-26T01:48:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.146+0000 7ff40a78d740 1 -- start start 2024-04-26T01:48:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.146+0000 7ff40a78d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302164820 0x562302164c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.146+0000 7ff40a78d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 0x56230216f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.146+0000 7ff40a78d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x562302171df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.146+0000 7ff40a78d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5623020ca900 con 0x56230216fa00 2024-04-26T01:48:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.146+0000 7ff40a78d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562302070ef0 con 0x562302164820 2024-04-26T01:48:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.146+0000 7ff40a78d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5623020c68c0 con 0x562302165140 2024-04-26T01:48:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.147+0000 7ff405000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 0x56230216f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.147+0000 7ff405000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 0x56230216f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:46006/0 (socket says 172.21.15.6:46006) 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.147+0000 7ff405000640 1 -- 172.21.15.6:0/1497690799 learned_addr learned my addr 172.21.15.6:0/1497690799 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.147+0000 7ff3ffe00640 1 --2- 172.21.15.6:0/1497690799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302164820 0x562302164c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.147+0000 7ff3ffe00640 1 --2- 172.21.15.6:0/1497690799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x562302171df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.147+0000 7ff405000640 1 -- 172.21.15.6:0/1497690799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302164820 msgr2=0x562302164c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.147+0000 7ff405000640 1 --2- 172.21.15.6:0/1497690799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302164820 0x562302164c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.147+0000 7ff405000640 1 -- 172.21.15.6:0/1497690799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 msgr2=0x562302171df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.147+0000 7ff405000640 1 --2- 172.21.15.6:0/1497690799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x562302171df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.147+0000 7ff405000640 1 -- 172.21.15.6:0/1497690799 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562301ddd210 con 0x562302165140 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.147+0000 7ff3fd600640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff3e80a3d10 0x7ff3e80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.147+0000 7ff405000640 1 --2- 172.21.15.6:0/1497690799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 0x56230216f4c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4010580 tx=0x7ff3f4010a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95342525501751ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.148+0000 7ff3fd600640 1 -- 172.21.15.6:0/2191839991 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7ff3f80d0e70 con 0x7ff40010e800 2024-04-26T01:48:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.148+0000 7ff3ffe00640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff3e80a3d10 0x7ff3e80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.148+0000 7ff3f2000640 1 -- 172.21.15.6:0/1497690799 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f400e890 con 0x562302165140 2024-04-26T01:48:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.148+0000 7ff3f2000640 1 -- 172.21.15.6:0/1497690799 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f401c480 con 0x562302165140 2024-04-26T01:48:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.148+0000 7ff3f2000640 1 -- 172.21.15.6:0/1497690799 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f40116a0 con 0x562302165140 2024-04-26T01:48:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.149+0000 7ff40a78d740 1 -- 172.21.15.6:0/1497690799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 msgr2=0x56230216f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.149+0000 7ff3ffe00640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff3e80a3d10 0x7ff3e80a61d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec005ab0 tx=0x7ff3ec005a20 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.149+0000 7ff40a78d740 1 --2- 172.21.15.6:0/1497690799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 0x56230216f4c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4010580 tx=0x7ff3f4010a40 comp rx=0 tx=0).stop 2024-04-26T01:48:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.149+0000 7ff40a78d740 1 -- 172.21.15.6:0/1497690799 shutdown_connections 2024-04-26T01:48:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.149+0000 7ff40a78d740 1 --2- 172.21.15.6:0/1497690799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x562302171df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.149+0000 7ff40a78d740 1 --2- 172.21.15.6:0/1497690799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 0x56230216f4c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.149+0000 7ff40a78d740 1 --2- 172.21.15.6:0/1497690799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302164820 0x562302164c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.149+0000 7ff40a78d740 1 -- 172.21.15.6:0/1497690799 >> 172.21.15.6:0/1497690799 conn(0x562302160d30 msgr2=0x562302161110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.149+0000 7ff40a78d740 1 -- 172.21.15.6:0/1497690799 shutdown_connections 2024-04-26T01:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.149+0000 7ff40a78d740 1 -- 172.21.15.6:0/1497690799 wait complete. 2024-04-26T01:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.149+0000 7ff40a78d740 1 -- start start 2024-04-26T01:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.150+0000 7ff40a78d740 1 Processor -- start 2024-04-26T01:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.150+0000 7ff40a78d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562302164820 0x56230216c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.150+0000 7ff40a78d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 0x56230216cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.150+0000 7ff40a78d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56230216fa00 0x56230216d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.150+0000 7ff40a78d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x562302172630 con 0x562302164820 2024-04-26T01:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.150+0000 7ff40a78d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562302084310 con 0x56230216fa00 2024-04-26T01:48:37.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.150+0000 7ff40a78d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5623021727b0 con 0x562302165140 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.150+0000 7ff40a78d740 10 client.0 _ll_register_callbacks cb 0x562302052cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.150+0000 7ff405000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562302164820 0x56230216c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.150+0000 7ff3ffe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 0x56230216cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.151+0000 7ff405000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562302164820 0x56230216c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46626/0 (socket says 172.21.15.6:46626) 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.151+0000 7ff405000640 1 -- 172.21.15.6:0/463257555 learned_addr learned my addr 172.21.15.6:0/463257555 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.151+0000 7ff405a00640 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56230216fa00 0x56230216d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.151+0000 7ff3ffe00640 1 -- 172.21.15.6:0/463257555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56230216fa00 msgr2=0x56230216d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.151+0000 7ff3ffe00640 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56230216fa00 0x56230216d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.151+0000 7ff3ffe00640 1 -- 172.21.15.6:0/463257555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562302164820 msgr2=0x56230216c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.151+0000 7ff3ffe00640 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562302164820 0x56230216c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.151+0000 7ff3ffe00640 1 -- 172.21.15.6:0/463257555 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562301ea9620 con 0x562302165140 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.151+0000 7ff405000640 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562302164820 0x56230216c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.151+0000 7ff3ffe00640 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 0x56230216cf20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec02a9a0 tx=0x7ff3ec005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9655ee46edc79b46 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.151+0000 7ff3e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.152+0000 7ff3e6a00640 1 -- 172.21.15.6:0/463257555 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec0318b0 con 0x562302165140 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.152+0000 7ff3e6a00640 1 -- 172.21.15.6:0/463257555 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7ff3ec031a50 con 0x562302165140 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.152+0000 7ff40a78d740 1 -- 172.21.15.6:0/463257555 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3f401c480 con 0x562302165140 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.152+0000 7ff3e6a00640 1 -- 172.21.15.6:0/463257555 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff3ec031c50 con 0x562302165140 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.152+0000 7ff40a78d740 10 client.25546 fetch_fsmap learned FSMap version 31 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.152+0000 7ff40a78d740 1 -- 172.21.15.6:0/463257555 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56230217e8e0 con 0x562302165140 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.152+0000 7ff3e6a00640 1 -- 172.21.15.6:0/463257555 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f401c480 con 0x562302165140 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.153+0000 7ff3e6a00640 1 -- 172.21.15.6:0/463257555 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7ff3ec044030 con 0x562302165140 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.153+0000 7ff40a78d740 10 client.25546 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.153+0000 7ff40a78d740 10 client.25546 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.153+0000 7ff40a78d740 20 client.25546 populate_metadata read hostname 'smithi006' 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.153+0000 7ff40a78d740 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56230217fda0 0x5623021821f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.153+0000 7ff40a78d740 4 client.25546 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.153+0000 7ff40a78d740 1 -- 172.21.15.6:0/463257555 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623020de8b0 con 0x56230217fda0 2024-04-26T01:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.153+0000 7ff405000640 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56230217fda0 0x5623021821f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.154+0000 7ff405000640 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56230217fda0 0x5623021821f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80253960172353a server_cookie=ef4393b8be23e464 in_seq=0 out_seq=0 2024-04-26T01:48:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.154+0000 7ff3e6a00640 10 client.25546 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff3e6a00640 1 -- 172.21.15.6:0/463257555 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623020de8b0 con 0x56230217fda0 2024-04-26T01:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff3e6a00640 10 client.25546 handle_command_reply: tid=0 2024-04-26T01:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff40a78d740 1 client.25546 shutdown 2024-04-26T01:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff40a78d740 1 -- 172.21.15.6:0/463257555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 msgr2=0x56230216cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff40a78d740 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 0x56230216cf20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec02a9a0 tx=0x7ff3ec005780 comp rx=0 tx=0).stop 2024-04-26T01:48:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff40a78d740 1 -- 172.21.15.6:0/463257555 shutdown_connections 2024-04-26T01:48:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff40a78d740 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56230217fda0 0x5623021821f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff40a78d740 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56230216fa00 0x56230216d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff40a78d740 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302165140 0x56230216cf20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff3e6a00640 0 client.25546 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff40a78d740 1 --2- 172.21.15.6:0/463257555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562302164820 0x56230216c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff40a78d740 1 -- 172.21.15.6:0/463257555 >> 172.21.15.6:0/463257555 conn(0x562302160d30 msgr2=0x562302166a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff40a78d740 1 -- 172.21.15.6:0/463257555 shutdown_connections 2024-04-26T01:48:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff40a78d740 1 -- 172.21.15.6:0/463257555 wait complete. 2024-04-26T01:48:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.155+0000 7ff40a78d740 20 client.25546 trim_cache size 0 max 16384 2024-04-26T01:48:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.172+0000 7ff40a78d740 1 Processor -- start 2024-04-26T01:48:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.172+0000 7ff40a78d740 1 -- start start 2024-04-26T01:48:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.172+0000 7ff40a78d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302164820 0x56230217b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.172+0000 7ff40a78d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302165140 0x562302187b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.172+0000 7ff40a78d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x562302188040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.172+0000 7ff40a78d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff4001b28b0 con 0x56230216fa00 2024-04-26T01:48:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.172+0000 7ff40a78d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff4001b0120 con 0x562302165140 2024-04-26T01:48:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.172+0000 7ff40a78d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff4001adf70 con 0x562302164820 2024-04-26T01:48:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff405000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302164820 0x56230217b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff3ffe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302165140 0x562302187b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff405a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x562302188040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff3ffe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302165140 0x562302187b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36862/0 (socket says 172.21.15.6:36862) 2024-04-26T01:48:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff3ffe00640 1 -- 172.21.15.6:0/605996560 learned_addr learned my addr 172.21.15.6:0/605996560 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff405000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302164820 0x56230217b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:46036/0 (socket says 172.21.15.6:46036) 2024-04-26T01:48:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff3ffe00640 1 -- 172.21.15.6:0/605996560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302164820 msgr2=0x56230217b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff3ffe00640 1 --2- 172.21.15.6:0/605996560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302164820 0x56230217b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff3ffe00640 1 -- 172.21.15.6:0/605996560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 msgr2=0x562302188040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff3ffe00640 1 --2- 172.21.15.6:0/605996560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x562302188040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff3ffe00640 1 -- 172.21.15.6:0/605996560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623021677d0 con 0x562302165140 2024-04-26T01:48:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff405000640 1 --2- 172.21.15.6:0/605996560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302164820 0x56230217b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.173+0000 7ff3ffe00640 1 --2- 172.21.15.6:0/605996560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302165140 0x562302187b00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec042040 tx=0x7ff3ec02a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c45649f868cea391 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.174+0000 7ff3f1600640 1 -- 172.21.15.6:0/605996560 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec05d030 con 0x562302165140 2024-04-26T01:48:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.174+0000 7ff3f1600640 1 -- 172.21.15.6:0/605996560 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3ec04c070 con 0x562302165140 2024-04-26T01:48:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.174+0000 7ff3f1600640 1 -- 172.21.15.6:0/605996560 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec03cdc0 con 0x562302165140 2024-04-26T01:48:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.174+0000 7ff40a78d740 1 -- 172.21.15.6:0/605996560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302165140 msgr2=0x562302187b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.174+0000 7ff40a78d740 1 --2- 172.21.15.6:0/605996560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302165140 0x562302187b00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec042040 tx=0x7ff3ec02a480 comp rx=0 tx=0).stop 2024-04-26T01:48:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.174+0000 7ff40a78d740 1 -- 172.21.15.6:0/605996560 shutdown_connections 2024-04-26T01:48:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.174+0000 7ff40a78d740 1 --2- 172.21.15.6:0/605996560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x562302188040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.174+0000 7ff40a78d740 1 --2- 172.21.15.6:0/605996560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302165140 0x562302187b00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.174+0000 7ff40a78d740 1 --2- 172.21.15.6:0/605996560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302164820 0x56230217b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.174+0000 7ff40a78d740 1 -- 172.21.15.6:0/605996560 >> 172.21.15.6:0/605996560 conn(0x562302160d30 msgr2=0x562302167ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.175+0000 7ff40a78d740 1 -- 172.21.15.6:0/605996560 shutdown_connections 2024-04-26T01:48:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.175+0000 7ff40a78d740 1 -- 172.21.15.6:0/605996560 wait complete. 2024-04-26T01:48:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.175+0000 7ff40a78d740 1 -- start start 2024-04-26T01:48:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.175+0000 7ff40a78d740 1 Processor -- start 2024-04-26T01:48:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.176+0000 7ff40a78d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302164820 0x562302174c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.176+0000 7ff40a78d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302165140 0x5623021751c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.176+0000 7ff40a78d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x5623021e6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.176+0000 7ff40a78d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff4001ae5e0 con 0x56230216fa00 2024-04-26T01:48:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.176+0000 7ff40a78d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff4001afa50 con 0x562302165140 2024-04-26T01:48:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.176+0000 7ff40a78d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff4001aecb0 con 0x562302164820 2024-04-26T01:48:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.176+0000 7ff40a78d740 10 client.0 _ll_register_callbacks cb 0x7ff3ec004450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.176+0000 7ff405a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x5623021e6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.176+0000 7ff405a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x5623021e6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46652/0 (socket says 172.21.15.6:46652) 2024-04-26T01:48:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.176+0000 7ff405a00640 1 -- 172.21.15.6:0/980693807 learned_addr learned my addr 172.21.15.6:0/980693807 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.176+0000 7ff405000640 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302164820 0x562302174c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.176+0000 7ff3ffe00640 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302165140 0x5623021751c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff405a00640 1 -- 172.21.15.6:0/980693807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302164820 msgr2=0x562302174c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff405a00640 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302164820 0x562302174c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff405a00640 1 -- 172.21.15.6:0/980693807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302165140 msgr2=0x5623021751c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff405a00640 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302165140 0x5623021751c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff405a00640 1 -- 172.21.15.6:0/980693807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623021e7140 con 0x56230216fa00 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff405a00640 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x5623021e6b50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800c760 tx=0x7ff3f809ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbaef9614a33c864 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff3e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff3e6a00640 1 -- 172.21.15.6:0/980693807 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f80ce7c0 con 0x56230216fa00 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff3e6a00640 1 -- 172.21.15.6:0/980693807 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7ff3f80ce960 con 0x56230216fa00 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff40a78d740 1 -- 172.21.15.6:0/980693807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3ec04c070 con 0x56230216fa00 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff3e6a00640 1 -- 172.21.15.6:0/980693807 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff3f80a5570 con 0x56230216fa00 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff40a78d740 10 client.15387 fetch_fsmap learned FSMap version 31 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.177+0000 7ff40a78d740 1 -- 172.21.15.6:0/980693807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5623021ee530 con 0x56230216fa00 2024-04-26T01:48:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.178+0000 7ff3e6a00640 1 -- 172.21.15.6:0/980693807 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3ec04c070 con 0x56230216fa00 2024-04-26T01:48:37.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.178+0000 7ff3e6a00640 1 -- 172.21.15.6:0/980693807 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7ff3f80bd1d0 con 0x56230216fa00 2024-04-26T01:48:37.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.178+0000 7ff40a78d740 10 client.15387 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:48:37.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.178+0000 7ff40a78d740 10 client.15387 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:48:37.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.178+0000 7ff40a78d740 20 client.15387 populate_metadata read hostname 'smithi006' 2024-04-26T01:48:37.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.178+0000 7ff40a78d740 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5623021eecb0 0x5623021f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:37.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.178+0000 7ff40a78d740 4 client.15387 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:48:37.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.178+0000 7ff40a78d740 1 -- 172.21.15.6:0/980693807 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5623020d8320 con 0x5623021eecb0 2024-04-26T01:48:37.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.178+0000 7ff405000640 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5623021eecb0 0x5623021f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:37.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.179+0000 7ff405000640 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5623021eecb0 0x5623021f10d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a8b6058beabe4 server_cookie=4927e891118cbe8d in_seq=0 out_seq=0 2024-04-26T01:48:37.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.179+0000 7ff3e6a00640 10 client.15387 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:37.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.219+0000 7ff3e6a00640 1 -- 172.21.15.6:0/980693807 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5623020d8320 con 0x5623021eecb0 2024-04-26T01:48:37.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.219+0000 7ff3e6a00640 10 client.15387 handle_command_reply: tid=0 2024-04-26T01:48:37.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.219+0000 7ff40a78d740 1 client.15387 shutdown 2024-04-26T01:48:37.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.220+0000 7ff40a78d740 1 -- 172.21.15.6:0/980693807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 msgr2=0x5623021e6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.220+0000 7ff40a78d740 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x5623021e6b50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800c760 tx=0x7ff3f809ce50 comp rx=0 tx=0).stop 2024-04-26T01:48:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.220+0000 7ff40a78d740 1 -- 172.21.15.6:0/980693807 shutdown_connections 2024-04-26T01:48:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.220+0000 7ff40a78d740 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5623021eecb0 0x5623021f10d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.220+0000 7ff40a78d740 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56230216fa00 0x5623021e6b50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.220+0000 7ff40a78d740 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562302165140 0x5623021751c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.220+0000 7ff40a78d740 1 --2- 172.21.15.6:0/980693807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562302164820 0x562302174c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.220+0000 7ff40a78d740 1 -- 172.21.15.6:0/980693807 >> 172.21.15.6:0/980693807 conn(0x562302160d30 msgr2=0x5623021661d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.220+0000 7ff3e6a00640 0 client.15387 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.220+0000 7ff40a78d740 1 -- 172.21.15.6:0/980693807 shutdown_connections 2024-04-26T01:48:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.220+0000 7ff40a78d740 1 -- 172.21.15.6:0/980693807 wait complete. 2024-04-26T01:48:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.220+0000 7ff40a78d740 20 client.15387 trim_cache size 0 max 16384 2024-04-26T01:48:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:48:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:48:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:48:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:48:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.221+0000 7ff407a00640 1 -- 172.21.15.6:0/2191839991 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff3e80a3d10 msgr2=0x7ff3e80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.221+0000 7ff407a00640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff3e80a3d10 0x7ff3e80a61d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec005ab0 tx=0x7ff3ec005a20 comp rx=0 tx=0).stop 2024-04-26T01:48:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.221+0000 7ff407a00640 1 -- 172.21.15.6:0/2191839991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff40010e800 msgr2=0x7ff40019a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.221+0000 7ff407a00640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff40010e800 0x7ff40019a7d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009fd0 tx=0x7ff3f800ef10 comp rx=0 tx=0).stop 2024-04-26T01:48:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.221+0000 7ff407a00640 1 -- 172.21.15.6:0/2191839991 shutdown_connections 2024-04-26T01:48:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.221+0000 7ff407a00640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff3e80a3d10 0x7ff3e80a61d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.221+0000 7ff407a00640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff40010e800 0x7ff40019a7d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.221+0000 7ff407a00640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff400074c20 0x7ff4001a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.221+0000 7ff407a00640 1 --2- 172.21.15.6:0/2191839991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff400074300 0x7ff4001a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.221+0000 7ff407a00640 1 -- 172.21.15.6:0/2191839991 >> 172.21.15.6:0/2191839991 conn(0x7ff4000fda10 msgr2=0x7ff4000ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.222+0000 7ff407a00640 1 -- 172.21.15.6:0/2191839991 shutdown_connections 2024-04-26T01:48:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:37.222+0000 7ff407a00640 1 -- 172.21.15.6:0/2191839991 wait complete. 2024-04-26T01:48:37.556 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 7.64 MB/sec warmup 61 sec latency 4948.908 ms 2024-04-26T01:48:38.557 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 7.52 MB/sec warmup 62 sec latency 5949.551 ms 2024-04-26T01:48:38.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:38 smithi006 ceph-mon[28419]: pgmap v86: 97 pgs: 97 active+clean; 38 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 950 KiB/s wr, 191 op/s 2024-04-26T01:48:38.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:38 smithi188 ceph-mon[32751]: pgmap v86: 97 pgs: 97 active+clean; 38 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 950 KiB/s wr, 191 op/s 2024-04-26T01:48:38.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:38 smithi029 ceph-mon[32775]: pgmap v86: 97 pgs: 97 active+clean; 38 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 950 KiB/s wr, 191 op/s 2024-04-26T01:48:39.558 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 7.40 MB/sec warmup 63 sec latency 6949.689 ms 2024-04-26T01:48:40.557 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 7.28 MB/sec warmup 64 sec latency 7949.849 ms 2024-04-26T01:48:40.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:40 smithi006 ceph-mon[28419]: pgmap v87: 97 pgs: 97 active+clean; 38 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 542 KiB/s wr, 94 op/s 2024-04-26T01:48:40.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:40 smithi188 ceph-mon[32751]: pgmap v87: 97 pgs: 97 active+clean; 38 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 542 KiB/s wr, 94 op/s 2024-04-26T01:48:40.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:40 smithi029 ceph-mon[32775]: pgmap v87: 97 pgs: 97 active+clean; 38 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 542 KiB/s wr, 94 op/s 2024-04-26T01:48:41.557 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 7.17 MB/sec warmup 65 sec latency 8950.007 ms 2024-04-26T01:48:42.558 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 7.06 MB/sec warmup 66 sec latency 9950.151 ms 2024-04-26T01:48:42.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:42 smithi006 ceph-mon[28419]: pgmap v88: 97 pgs: 97 active+clean; 34 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 138 op/s 2024-04-26T01:48:42.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:42 smithi188 ceph-mon[32751]: pgmap v88: 97 pgs: 97 active+clean; 34 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 138 op/s 2024-04-26T01:48:42.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:42 smithi029 ceph-mon[32775]: pgmap v88: 97 pgs: 97 active+clean; 34 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 138 op/s 2024-04-26T01:48:43.558 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 6.95 MB/sec warmup 67 sec latency 10950.309 ms 2024-04-26T01:48:44.558 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 6.85 MB/sec warmup 68 sec latency 11950.467 ms 2024-04-26T01:48:44.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:44 smithi006 ceph-mon[28419]: pgmap v89: 97 pgs: 97 active+clean; 34 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 683 KiB/s wr, 74 op/s 2024-04-26T01:48:44.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:44 smithi188 ceph-mon[32751]: pgmap v89: 97 pgs: 97 active+clean; 34 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 683 KiB/s wr, 74 op/s 2024-04-26T01:48:44.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:44 smithi029 ceph-mon[32775]: pgmap v89: 97 pgs: 97 active+clean; 34 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 683 KiB/s wr, 74 op/s 2024-04-26T01:48:45.558 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 6.75 MB/sec warmup 69 sec latency 12950.561 ms 2024-04-26T01:48:46.558 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 6.66 MB/sec warmup 70 sec latency 13950.663 ms 2024-04-26T01:48:46.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:46 smithi006 ceph-mon[28419]: pgmap v90: 97 pgs: 97 active+clean; 34 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 683 KiB/s wr, 74 op/s 2024-04-26T01:48:46.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:46 smithi188 ceph-mon[32751]: pgmap v90: 97 pgs: 97 active+clean; 34 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 683 KiB/s wr, 74 op/s 2024-04-26T01:48:46.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:46 smithi029 ceph-mon[32775]: pgmap v90: 97 pgs: 97 active+clean; 34 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 683 KiB/s wr, 74 op/s 2024-04-26T01:48:47.558 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 6.56 MB/sec warmup 71 sec latency 14950.760 ms 2024-04-26T01:48:48.558 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 6.47 MB/sec warmup 72 sec latency 15950.858 ms 2024-04-26T01:48:48.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:48 smithi006 ceph-mon[28419]: pgmap v91: 97 pgs: 97 active+clean; 34 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-04-26T01:48:48.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:48 smithi188 ceph-mon[32751]: pgmap v91: 97 pgs: 97 active+clean; 34 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-04-26T01:48:48.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:48 smithi029 ceph-mon[32775]: pgmap v91: 97 pgs: 97 active+clean; 34 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-04-26T01:48:49.558 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 6.38 MB/sec warmup 73 sec latency 16951.007 ms 2024-04-26T01:48:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.371+0000 7f1869200640 1 -- 172.21.15.6:0/1288866005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186410e6f0 msgr2=0x7f1864110ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.371+0000 7f1869200640 1 --2- 172.21.15.6:0/1288866005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186410e6f0 0x7f1864110ae0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f185c009be0 tx=0x7f185c02a6c0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.371+0000 7f1869200640 1 -- 172.21.15.6:0/1288866005 shutdown_connections 2024-04-26T01:48:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.371+0000 7f1869200640 1 --2- 172.21.15.6:0/1288866005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186410e6f0 0x7f1864110ae0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.371+0000 7f1869200640 1 --2- 172.21.15.6:0/1288866005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18641014f0 0x7f186410e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.371+0000 7f1869200640 1 --2- 172.21.15.6:0/1288866005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1864100b40 0x7f1864100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.371+0000 7f1869200640 1 -- 172.21.15.6:0/1288866005 >> 172.21.15.6:0/1288866005 conn(0x7f18640fc730 msgr2=0x7f18640feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.371+0000 7f1869200640 1 -- 172.21.15.6:0/1288866005 shutdown_connections 2024-04-26T01:48:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.371+0000 7f1869200640 1 -- 172.21.15.6:0/1288866005 wait complete. 2024-04-26T01:48:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.372+0000 7f1869200640 1 Processor -- start 2024-04-26T01:48:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.372+0000 7f1869200640 1 -- start start 2024-04-26T01:48:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.372+0000 7f1869200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1864100b40 0x7f1864102d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.372+0000 7f1869200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18641014f0 0x7f18641032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.373+0000 7f1869200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186410e6f0 0x7f1864103810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.373+0000 7f1869200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18641141d0 con 0x7f186410e6f0 2024-04-26T01:48:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.373+0000 7f1869200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1864114050 con 0x7f18641014f0 2024-04-26T01:48:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.373+0000 7f1869200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1864114350 con 0x7f1864100b40 2024-04-26T01:48:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.373+0000 7f1863400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1864100b40 0x7f1864102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.373+0000 7f1862a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18641014f0 0x7f18641032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.373+0000 7f1863400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1864100b40 0x7f1864102d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44138/0 (socket says 172.21.15.6:44138) 2024-04-26T01:48:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.373+0000 7f1863400640 1 -- 172.21.15.6:0/3570388623 learned_addr learned my addr 172.21.15.6:0/3570388623 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.373+0000 7f1863e00640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186410e6f0 0x7f1864103810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.374+0000 7f1863400640 1 -- 172.21.15.6:0/3570388623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18641014f0 msgr2=0x7f18641032d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.374+0000 7f1863400640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18641014f0 0x7f18641032d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.374+0000 7f1863400640 1 -- 172.21.15.6:0/3570388623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186410e6f0 msgr2=0x7f1864103810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.374+0000 7f1863400640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186410e6f0 0x7f1864103810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.374+0000 7f1863400640 1 -- 172.21.15.6:0/3570388623 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18641aaea0 con 0x7f1864100b40 2024-04-26T01:48:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.374+0000 7f1862a00640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18641014f0 0x7f18641032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.374+0000 7f1863400640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1864100b40 0x7f1864102d90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f185400ea90 tx=0x7f185400ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.374+0000 7f185be00640 1 -- 172.21.15.6:0/3570388623 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185400ce40 con 0x7f1864100b40 2024-04-26T01:48:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.374+0000 7f185be00640 1 -- 172.21.15.6:0/3570388623 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18540035f0 con 0x7f1864100b40 2024-04-26T01:48:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.374+0000 7f185be00640 1 -- 172.21.15.6:0/3570388623 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1854010670 con 0x7f1864100b40 2024-04-26T01:48:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.374+0000 7f1869200640 1 -- 172.21.15.6:0/3570388623 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18641ab190 con 0x7f1864100b40 2024-04-26T01:48:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.375+0000 7f1869200640 1 -- 172.21.15.6:0/3570388623 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18641ab690 con 0x7f1864100b40 2024-04-26T01:48:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.378+0000 7f186d60b740 1 Processor -- start 2024-04-26T01:48:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.378+0000 7f185be00640 1 -- 172.21.15.6:0/3570388623 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f1854003aa0 con 0x7f1864100b40 2024-04-26T01:48:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.378+0000 7f186d60b740 1 -- start start 2024-04-26T01:48:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.378+0000 7f186d60b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd339b0 0x55a70cd33d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.378+0000 7f186d60b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd342d0 0x55a70cd3e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.378+0000 7f186d60b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd3eb90 0x55a70cd40f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.378+0000 7f186d60b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a70cc99900 con 0x55a70cd342d0 2024-04-26T01:48:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.378+0000 7f186d60b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a70cc3fef0 con 0x55a70cd3eb90 2024-04-26T01:48:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.378+0000 7f186d60b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a70cc958c0 con 0x55a70cd339b0 2024-04-26T01:48:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.379+0000 7f1863e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd339b0 0x55a70cd33d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.379+0000 7f1863400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd342d0 0x55a70cd3e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.379+0000 7f1863e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd339b0 0x55a70cd33d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44148/0 (socket says 172.21.15.6:44148) 2024-04-26T01:48:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.379+0000 7f1863e00640 1 -- 172.21.15.6:0/25964140 learned_addr learned my addr 172.21.15.6:0/25964140 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.379+0000 7f1862a00640 1 --2- 172.21.15.6:0/25964140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd3eb90 0x55a70cd40f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.379+0000 7f1862a00640 1 -- 172.21.15.6:0/25964140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd339b0 msgr2=0x55a70cd33d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.379+0000 7f1862a00640 1 --2- 172.21.15.6:0/25964140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd339b0 0x55a70cd33d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.379+0000 7f1862a00640 1 -- 172.21.15.6:0/25964140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd342d0 msgr2=0x55a70cd3e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.379+0000 7f1862a00640 1 --2- 172.21.15.6:0/25964140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd342d0 0x55a70cd3e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.379+0000 7f1862a00640 1 -- 172.21.15.6:0/25964140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a70c9ac210 con 0x55a70cd3eb90 2024-04-26T01:48:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.380+0000 7f1862a00640 1 --2- 172.21.15.6:0/25964140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd3eb90 0x55a70cd40f80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1864104590 tx=0x7f184c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64d2de878cd55955 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.380+0000 7f185be00640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f18480a3d00 0x7f18480a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.380+0000 7f1858c00640 1 -- 172.21.15.6:0/25964140 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c004020 con 0x55a70cd3eb90 2024-04-26T01:48:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.380+0000 7f1858c00640 1 -- 172.21.15.6:0/25964140 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f184c0041c0 con 0x55a70cd3eb90 2024-04-26T01:48:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.380+0000 7f1858c00640 1 -- 172.21.15.6:0/25964140 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c00f8b0 con 0x55a70cd3eb90 2024-04-26T01:48:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.380+0000 7f185be00640 1 -- 172.21.15.6:0/3570388623 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f18540d0c90 con 0x7f1864100b40 2024-04-26T01:48:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.380+0000 7f1863e00640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f18480a3d00 0x7f18480a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.381+0000 7f186d60b740 1 -- 172.21.15.6:0/25964140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd3eb90 msgr2=0x55a70cd40f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.381+0000 7f186d60b740 1 --2- 172.21.15.6:0/25964140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd3eb90 0x55a70cd40f80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1864104590 tx=0x7f184c0118d0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.381+0000 7f186d60b740 1 -- 172.21.15.6:0/25964140 shutdown_connections 2024-04-26T01:48:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.381+0000 7f186d60b740 1 --2- 172.21.15.6:0/25964140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd342d0 0x55a70cd3e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.381+0000 7f186d60b740 1 --2- 172.21.15.6:0/25964140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd3eb90 0x55a70cd40f80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.381+0000 7f186d60b740 1 --2- 172.21.15.6:0/25964140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd339b0 0x55a70cd33d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.381+0000 7f186d60b740 1 -- 172.21.15.6:0/25964140 >> 172.21.15.6:0/25964140 conn(0x55a70cd2fec0 msgr2=0x55a70cd302a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.381+0000 7f1863e00640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f18480a3d00 0x7f18480a61c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f185c02abe0 tx=0x7f185c005fd0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.381+0000 7f186d60b740 1 -- 172.21.15.6:0/25964140 shutdown_connections 2024-04-26T01:48:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.381+0000 7f186d60b740 1 -- 172.21.15.6:0/25964140 wait complete. 2024-04-26T01:48:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.381+0000 7f186d60b740 1 -- start start 2024-04-26T01:48:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f186d60b740 1 Processor -- start 2024-04-26T01:48:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f186d60b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd339b0 0x55a70cd3bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f186d60b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd342d0 0x55a70cd3c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f186d60b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd3eb90 0x55a70cd3c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f186d60b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a70cd417c0 con 0x55a70cd342d0 2024-04-26T01:48:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f186d60b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a70cc53310 con 0x55a70cd3eb90 2024-04-26T01:48:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f186d60b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a70cd41940 con 0x55a70cd339b0 2024-04-26T01:48:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f186d60b740 10 client.0 _ll_register_callbacks cb 0x55a70cc21cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f1862a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd342d0 0x55a70cd3c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f1862a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd342d0 0x55a70cd3c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56486/0 (socket says 172.21.15.6:56486) 2024-04-26T01:48:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f1862a00640 1 -- 172.21.15.6:0/3467352071 learned_addr learned my addr 172.21.15.6:0/3467352071 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f1863e00640 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd3eb90 0x55a70cd3c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.382+0000 7f1863400640 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd339b0 0x55a70cd3bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.383+0000 7f1862a00640 1 -- 172.21.15.6:0/3467352071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd339b0 msgr2=0x55a70cd3bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.383+0000 7f1862a00640 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd339b0 0x55a70cd3bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.383+0000 7f1862a00640 1 -- 172.21.15.6:0/3467352071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd3eb90 msgr2=0x55a70cd3c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.383+0000 7f1862a00640 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd3eb90 0x55a70cd3c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.383+0000 7f1862a00640 1 -- 172.21.15.6:0/3467352071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a70ca78620 con 0x55a70cd342d0 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.383+0000 7f1862a00640 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd342d0 0x55a70cd3c010 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f184c01b040 tx=0x7f184c005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8ee6afd75d8c74 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.383+0000 7f1851600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.383+0000 7f1851600640 1 -- 172.21.15.6:0/3467352071 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c027030 con 0x55a70cd342d0 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.383+0000 7f1851600640 1 -- 172.21.15.6:0/3467352071 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f184c016040 con 0x55a70cd342d0 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.383+0000 7f186d60b740 1 -- 172.21.15.6:0/3467352071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f184c0041c0 con 0x55a70cd342d0 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.384+0000 7f1851600640 1 -- 172.21.15.6:0/3467352071 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f184c031050 con 0x55a70cd342d0 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.384+0000 7f186d60b740 10 client.15399 fetch_fsmap learned FSMap version 31 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.384+0000 7f186d60b740 1 -- 172.21.15.6:0/3467352071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a70cd4d9d0 con 0x55a70cd342d0 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.384+0000 7f1851600640 1 -- 172.21.15.6:0/3467352071 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f184c0041c0 con 0x55a70cd342d0 2024-04-26T01:48:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.384+0000 7f1851600640 1 -- 172.21.15.6:0/3467352071 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f184c020030 con 0x55a70cd342d0 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.384+0000 7f186d60b740 10 client.15399 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.384+0000 7f186d60b740 10 client.15399 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.385+0000 7f186d60b740 20 client.15399 populate_metadata read hostname 'smithi006' 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.385+0000 7f186d60b740 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a70cd4ede0 0x55a70cd511d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.385+0000 7f186d60b740 4 client.15399 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.385+0000 7f186d60b740 1 -- 172.21.15.6:0/3467352071 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a70ccad8b0 con 0x55a70cd4ede0 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.385+0000 7f1863400640 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a70cd4ede0 0x55a70cd511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.385+0000 7f1863400640 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a70cd4ede0 0x55a70cd511d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78af37ed8c467730 server_cookie=6b72d0ded013f246 in_seq=0 out_seq=0 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.385+0000 7f1851600640 10 client.15399 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.386+0000 7f1851600640 1 -- 172.21.15.6:0/3467352071 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a70ccad8b0 con 0x55a70cd4ede0 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.386+0000 7f1851600640 10 client.15399 handle_command_reply: tid=0 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.386+0000 7f186d60b740 1 client.15399 shutdown 2024-04-26T01:48:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.386+0000 7f186d60b740 1 -- 172.21.15.6:0/3467352071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd342d0 msgr2=0x55a70cd3c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.386+0000 7f186d60b740 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd342d0 0x55a70cd3c010 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f184c01b040 tx=0x7f184c005df0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.387+0000 7f186d60b740 1 -- 172.21.15.6:0/3467352071 shutdown_connections 2024-04-26T01:48:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.387+0000 7f186d60b740 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a70cd4ede0 0x55a70cd511d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.387+0000 7f186d60b740 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd342d0 0x55a70cd3c010 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.387+0000 7f1851600640 0 client.15399 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.387+0000 7f186d60b740 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd3eb90 0x55a70cd3c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.387+0000 7f186d60b740 1 --2- 172.21.15.6:0/3467352071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd339b0 0x55a70cd3bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.387+0000 7f186d60b740 1 -- 172.21.15.6:0/3467352071 >> 172.21.15.6:0/3467352071 conn(0x55a70cd2fec0 msgr2=0x55a70cd35c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.387+0000 7f186d60b740 1 -- 172.21.15.6:0/3467352071 shutdown_connections 2024-04-26T01:48:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.387+0000 7f186d60b740 1 -- 172.21.15.6:0/3467352071 wait complete. 2024-04-26T01:48:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.387+0000 7f186d60b740 20 client.15399 trim_cache size 0 max 16384 2024-04-26T01:48:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.403+0000 7f186d60b740 1 Processor -- start 2024-04-26T01:48:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.403+0000 7f186d60b740 1 -- start start 2024-04-26T01:48:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f186d60b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 0x55a70cd4a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f186d60b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd342d0 0x55a70cd36370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f186d60b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd3eb90 0x55a70cd3c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f186d60b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18641b6ad0 con 0x55a70cd339b0 2024-04-26T01:48:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f186d60b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f18641b4340 con 0x55a70cd342d0 2024-04-26T01:48:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f186d60b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f18641b2190 con 0x55a70cd3eb90 2024-04-26T01:48:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f1863400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 0x55a70cd4a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f1863400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 0x55a70cd4a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56500/0 (socket says 172.21.15.6:56500) 2024-04-26T01:48:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f1863400640 1 -- 172.21.15.6:0/3756269817 learned_addr learned my addr 172.21.15.6:0/3756269817 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f1863e00640 1 --2- 172.21.15.6:0/3756269817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd3eb90 0x55a70cd3c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f1862a00640 1 --2- 172.21.15.6:0/3756269817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd342d0 0x55a70cd36370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f1863400640 1 -- 172.21.15.6:0/3756269817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd3eb90 msgr2=0x55a70cd3c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f1863400640 1 --2- 172.21.15.6:0/3756269817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd3eb90 0x55a70cd3c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f1863400640 1 -- 172.21.15.6:0/3756269817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd342d0 msgr2=0x55a70cd36370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f1863400640 1 --2- 172.21.15.6:0/3756269817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd342d0 0x55a70cd36370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f1863400640 1 -- 172.21.15.6:0/3756269817 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a70cd368b0 con 0x55a70cd339b0 2024-04-26T01:48:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.404+0000 7f1863e00640 1 --2- 172.21.15.6:0/3756269817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd3eb90 0x55a70cd3c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.405+0000 7f1863400640 1 --2- 172.21.15.6:0/3756269817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 0x55a70cd4a680 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1854099580 tx=0x7f18540d4740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226e9c9a8e1cfac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.405+0000 7f1853e00640 1 -- 172.21.15.6:0/3756269817 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185409c980 con 0x55a70cd339b0 2024-04-26T01:48:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.405+0000 7f1853e00640 1 -- 172.21.15.6:0/3756269817 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f185409d2b0 con 0x55a70cd339b0 2024-04-26T01:48:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.405+0000 7f1853e00640 1 -- 172.21.15.6:0/3756269817 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185409c510 con 0x55a70cd339b0 2024-04-26T01:48:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.406+0000 7f186d60b740 1 -- 172.21.15.6:0/3756269817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 msgr2=0x55a70cd4a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.406+0000 7f186d60b740 1 --2- 172.21.15.6:0/3756269817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 0x55a70cd4a680 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1854099580 tx=0x7f18540d4740 comp rx=0 tx=0).stop 2024-04-26T01:48:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.406+0000 7f186d60b740 1 -- 172.21.15.6:0/3756269817 shutdown_connections 2024-04-26T01:48:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.406+0000 7f186d60b740 1 --2- 172.21.15.6:0/3756269817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd3eb90 0x55a70cd3c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.406+0000 7f186d60b740 1 --2- 172.21.15.6:0/3756269817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd342d0 0x55a70cd36370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.406+0000 7f186d60b740 1 --2- 172.21.15.6:0/3756269817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 0x55a70cd4a680 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.406+0000 7f186d60b740 1 -- 172.21.15.6:0/3756269817 >> 172.21.15.6:0/3756269817 conn(0x55a70cd2fec0 msgr2=0x55a70cd3ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.406+0000 7f186d60b740 1 -- 172.21.15.6:0/3756269817 shutdown_connections 2024-04-26T01:48:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.406+0000 7f186d60b740 1 -- 172.21.15.6:0/3756269817 wait complete. 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.406+0000 7f186d60b740 1 -- start start 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.406+0000 7f186d60b740 1 Processor -- start 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.407+0000 7f186d60b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 0x55a70cd47070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.407+0000 7f186d60b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd342d0 0x55a70cd44450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.407+0000 7f186d60b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd3eb90 0x55a70cd44a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.407+0000 7f186d60b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18641b2800 con 0x55a70cd339b0 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.407+0000 7f186d60b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f18641b3c70 con 0x55a70cd342d0 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.407+0000 7f186d60b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f18641b2ed0 con 0x55a70cd3eb90 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.407+0000 7f186d60b740 10 client.0 _ll_register_callbacks cb 0x7f184c009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.407+0000 7f1863400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 0x55a70cd47070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.407+0000 7f1863e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd3eb90 0x55a70cd44a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.407+0000 7f1863400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 0x55a70cd47070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56506/0 (socket says 172.21.15.6:56506) 2024-04-26T01:48:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.407+0000 7f1863400640 1 -- 172.21.15.6:0/2798906084 learned_addr learned my addr 172.21.15.6:0/2798906084 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.407+0000 7f1862a00640 1 --2- 172.21.15.6:0/2798906084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd342d0 0x55a70cd44450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f1863400640 1 -- 172.21.15.6:0/2798906084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd3eb90 msgr2=0x55a70cd44a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f1863400640 1 --2- 172.21.15.6:0/2798906084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd3eb90 0x55a70cd44a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f1863400640 1 -- 172.21.15.6:0/2798906084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd342d0 msgr2=0x55a70cd44450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f1863400640 1 --2- 172.21.15.6:0/2798906084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd342d0 0x55a70cd44450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f1863400640 1 -- 172.21.15.6:0/2798906084 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a70cd45290 con 0x55a70cd339b0 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f1863400640 1 --2- 172.21.15.6:0/2798906084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 0x55a70cd47070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f18540a52f0 tx=0x7f185409cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf5cec107dde984 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f1851600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f1851600640 1 -- 172.21.15.6:0/2798906084 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185409c6f0 con 0x55a70cd339b0 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f1851600640 1 -- 172.21.15.6:0/2798906084 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f1854099220 con 0x55a70cd339b0 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f186d60b740 1 -- 172.21.15.6:0/2798906084 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f185409d2b0 con 0x55a70cd339b0 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f1851600640 1 -- 172.21.15.6:0/2798906084 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f18540a0f60 con 0x55a70cd339b0 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f186d60b740 10 client.15405 fetch_fsmap learned FSMap version 31 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.408+0000 7f186d60b740 1 -- 172.21.15.6:0/2798906084 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a70cdb5360 con 0x55a70cd339b0 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.409+0000 7f1851600640 1 -- 172.21.15.6:0/2798906084 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f185409d2b0 con 0x55a70cd339b0 2024-04-26T01:48:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.409+0000 7f1851600640 1 -- 172.21.15.6:0/2798906084 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f1854004000 con 0x55a70cd339b0 2024-04-26T01:48:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.409+0000 7f186d60b740 10 client.15405 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:48:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.409+0000 7f186d60b740 10 client.15405 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:48:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.409+0000 7f186d60b740 20 client.15405 populate_metadata read hostname 'smithi006' 2024-04-26T01:48:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.409+0000 7f186d60b740 1 --2- 172.21.15.6:0/2798906084 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a70cdb5a90 0x55a70cdb7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.409+0000 7f186d60b740 4 client.15405 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:48:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.409+0000 7f186d60b740 1 -- 172.21.15.6:0/2798906084 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a70cca7320 con 0x55a70cdb5a90 2024-04-26T01:48:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.409+0000 7f1862a00640 1 --2- 172.21.15.6:0/2798906084 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a70cdb5a90 0x55a70cdb7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.410+0000 7f1862a00640 1 --2- 172.21.15.6:0/2798906084 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a70cdb5a90 0x55a70cdb7eb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=539ab59055a59bd server_cookie=bab1869bcf3589c9 in_seq=0 out_seq=0 2024-04-26T01:48:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.410+0000 7f1851600640 10 client.15405 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.442+0000 7f1851600640 1 -- 172.21.15.6:0/2798906084 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a70cca7320 con 0x55a70cdb5a90 2024-04-26T01:48:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.442+0000 7f1851600640 10 client.15405 handle_command_reply: tid=0 2024-04-26T01:48:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.442+0000 7f186d60b740 1 client.15405 shutdown 2024-04-26T01:48:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.442+0000 7f186d60b740 1 -- 172.21.15.6:0/2798906084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 msgr2=0x55a70cd47070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.442+0000 7f186d60b740 1 --2- 172.21.15.6:0/2798906084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 0x55a70cd47070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f18540a52f0 tx=0x7f185409cca0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.442+0000 7f186d60b740 1 -- 172.21.15.6:0/2798906084 shutdown_connections 2024-04-26T01:48:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.442+0000 7f186d60b740 1 --2- 172.21.15.6:0/2798906084 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a70cdb5a90 0x55a70cdb7eb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.442+0000 7f186d60b740 1 --2- 172.21.15.6:0/2798906084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a70cd3eb90 0x55a70cd44a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.442+0000 7f186d60b740 1 --2- 172.21.15.6:0/2798906084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a70cd342d0 0x55a70cd44450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.442+0000 7f186d60b740 1 --2- 172.21.15.6:0/2798906084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a70cd339b0 0x55a70cd47070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.442+0000 7f1851600640 0 client.15405 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:48:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.442+0000 7f186d60b740 1 -- 172.21.15.6:0/2798906084 >> 172.21.15.6:0/2798906084 conn(0x55a70cd2fec0 msgr2=0x55a70cd3c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:50.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.443+0000 7f186d60b740 1 -- 172.21.15.6:0/2798906084 shutdown_connections 2024-04-26T01:48:50.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.443+0000 7f186d60b740 1 -- 172.21.15.6:0/2798906084 wait complete. 2024-04-26T01:48:50.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.443+0000 7f186d60b740 20 client.15405 trim_cache size 0 max 16384 2024-04-26T01:48:50.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:48:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:48:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:48:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:48:50.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.444+0000 7f1869200640 1 -- 172.21.15.6:0/3570388623 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f18480a3d00 msgr2=0x7f18480a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.444+0000 7f1869200640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f18480a3d00 0x7f18480a61c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f185c02abe0 tx=0x7f185c005fd0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.444+0000 7f1869200640 1 -- 172.21.15.6:0/3570388623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1864100b40 msgr2=0x7f1864102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.444+0000 7f1869200640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1864100b40 0x7f1864102d90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f185400ea90 tx=0x7f185400ef50 comp rx=0 tx=0).stop 2024-04-26T01:48:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.444+0000 7f1869200640 1 -- 172.21.15.6:0/3570388623 shutdown_connections 2024-04-26T01:48:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.444+0000 7f1869200640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f18480a3d00 0x7f18480a61c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.444+0000 7f1869200640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f186410e6f0 0x7f1864103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.444+0000 7f1869200640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18641014f0 0x7f18641032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.444+0000 7f1869200640 1 --2- 172.21.15.6:0/3570388623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1864100b40 0x7f1864102d90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.444+0000 7f1869200640 1 -- 172.21.15.6:0/3570388623 >> 172.21.15.6:0/3570388623 conn(0x7f18640fc730 msgr2=0x7f18640feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.444+0000 7f1869200640 1 -- 172.21.15.6:0/3570388623 shutdown_connections 2024-04-26T01:48:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:48:50.444+0000 7f1869200640 1 -- 172.21.15.6:0/3570388623 wait complete. 2024-04-26T01:48:50.558 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 6.30 MB/sec warmup 74 sec latency 17951.163 ms 2024-04-26T01:48:50.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:50 smithi006 ceph-mon[28419]: pgmap v92: 97 pgs: 97 active+clean; 34 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 875 KiB/s wr, 65 op/s 2024-04-26T01:48:50.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:50 smithi188 ceph-mon[32751]: pgmap v92: 97 pgs: 97 active+clean; 34 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 875 KiB/s wr, 65 op/s 2024-04-26T01:48:50.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:50 smithi029 ceph-mon[32775]: pgmap v92: 97 pgs: 97 active+clean; 34 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 875 KiB/s wr, 65 op/s 2024-04-26T01:48:51.559 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 6.21 MB/sec warmup 75 sec latency 18951.294 ms 2024-04-26T01:48:52.560 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 6.13 MB/sec warmup 76 sec latency 19951.430 ms 2024-04-26T01:48:52.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:52 smithi006 ceph-mon[28419]: pgmap v93: 97 pgs: 97 active+clean; 33 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.3 MiB/s wr, 95 op/s 2024-04-26T01:48:52.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:52 smithi188 ceph-mon[32751]: pgmap v93: 97 pgs: 97 active+clean; 33 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.3 MiB/s wr, 95 op/s 2024-04-26T01:48:52.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:52 smithi029 ceph-mon[32775]: pgmap v93: 97 pgs: 97 active+clean; 33 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.3 MiB/s wr, 95 op/s 2024-04-26T01:48:53.559 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 6.05 MB/sec warmup 77 sec latency 20951.522 ms 2024-04-26T01:48:54.559 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.97 MB/sec warmup 78 sec latency 21951.685 ms 2024-04-26T01:48:54.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:54 smithi006 ceph-mon[28419]: pgmap v94: 97 pgs: 97 active+clean; 33 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 838 KiB/s wr, 51 op/s 2024-04-26T01:48:54.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:54 smithi188 ceph-mon[32751]: pgmap v94: 97 pgs: 97 active+clean; 33 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 838 KiB/s wr, 51 op/s 2024-04-26T01:48:54.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:54 smithi029 ceph-mon[32775]: pgmap v94: 97 pgs: 97 active+clean; 33 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 838 KiB/s wr, 51 op/s 2024-04-26T01:48:55.559 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.90 MB/sec warmup 79 sec latency 22951.834 ms 2024-04-26T01:48:56.559 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.82 MB/sec warmup 80 sec latency 23951.994 ms 2024-04-26T01:48:56.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:56 smithi006 ceph-mon[28419]: pgmap v95: 97 pgs: 97 active+clean; 33 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 838 KiB/s wr, 51 op/s 2024-04-26T01:48:56.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:56 smithi188 ceph-mon[32751]: pgmap v95: 97 pgs: 97 active+clean; 33 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 838 KiB/s wr, 51 op/s 2024-04-26T01:48:56.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:56 smithi029 ceph-mon[32775]: pgmap v95: 97 pgs: 97 active+clean; 33 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 838 KiB/s wr, 51 op/s 2024-04-26T01:48:57.559 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.75 MB/sec warmup 81 sec latency 24952.134 ms 2024-04-26T01:48:58.560 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.68 MB/sec warmup 82 sec latency 25952.292 ms 2024-04-26T01:48:58.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:48:58 smithi006 ceph-mon[28419]: pgmap v96: 97 pgs: 97 active+clean; 32 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-04-26T01:48:58.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:48:58 smithi188 ceph-mon[32751]: pgmap v96: 97 pgs: 97 active+clean; 32 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-04-26T01:48:58.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:48:58 smithi029 ceph-mon[32775]: pgmap v96: 97 pgs: 97 active+clean; 32 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-04-26T01:48:59.560 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.61 MB/sec warmup 83 sec latency 26952.457 ms 2024-04-26T01:49:00.560 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.55 MB/sec warmup 84 sec latency 27952.592 ms 2024-04-26T01:49:00.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:00 smithi006 ceph-mon[28419]: pgmap v97: 97 pgs: 97 active+clean; 32 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 972 KiB/s wr, 46 op/s 2024-04-26T01:49:00.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:00 smithi188 ceph-mon[32751]: pgmap v97: 97 pgs: 97 active+clean; 32 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 972 KiB/s wr, 46 op/s 2024-04-26T01:49:00.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:00 smithi029 ceph-mon[32775]: pgmap v97: 97 pgs: 97 active+clean; 32 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 972 KiB/s wr, 46 op/s 2024-04-26T01:49:01.560 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.48 MB/sec warmup 85 sec latency 28952.690 ms 2024-04-26T01:49:02.560 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.42 MB/sec warmup 86 sec latency 29952.791 ms 2024-04-26T01:49:02.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:02 smithi188 ceph-mon[32751]: pgmap v98: 97 pgs: 97 active+clean; 35 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-04-26T01:49:02.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:02 smithi029 ceph-mon[32775]: pgmap v98: 97 pgs: 97 active+clean; 35 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-04-26T01:49:02.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:02 smithi006 ceph-mon[28419]: pgmap v98: 97 pgs: 97 active+clean; 35 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-04-26T01:49:03.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:49:03.560 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.36 MB/sec warmup 87 sec latency 30952.863 ms 2024-04-26T01:49:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.596+0000 7fcccf000640 1 -- 172.21.15.6:0/4192264753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccc81043c0 msgr2=0x7fccc81047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.596+0000 7fcccf000640 1 --2- 172.21.15.6:0/4192264753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccc81043c0 0x7fccc81047a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009860 tx=0x7fccb802a340 comp rx=0 tx=0).stop 2024-04-26T01:49:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.596+0000 7fcccf000640 1 -- 172.21.15.6:0/4192264753 shutdown_connections 2024-04-26T01:49:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.596+0000 7fcccf000640 1 --2- 172.21.15.6:0/4192264753 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccc8104d70 0x7fccc8108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.596+0000 7fcccf000640 1 --2- 172.21.15.6:0/4192264753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc8109490 0x7fccc8111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.596+0000 7fcccf000640 1 --2- 172.21.15.6:0/4192264753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccc81043c0 0x7fccc81047a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.596+0000 7fcccf000640 1 -- 172.21.15.6:0/4192264753 >> 172.21.15.6:0/4192264753 conn(0x7fccc80ffc40 msgr2=0x7fccc8102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.596+0000 7fcccf000640 1 -- 172.21.15.6:0/4192264753 shutdown_connections 2024-04-26T01:49:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.596+0000 7fcccf000640 1 -- 172.21.15.6:0/4192264753 wait complete. 2024-04-26T01:49:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.597+0000 7fcccf000640 1 Processor -- start 2024-04-26T01:49:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.597+0000 7fcccf000640 1 -- start start 2024-04-26T01:49:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.598+0000 7fcccf000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccc81043c0 0x7fccc8076460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.598+0000 7fcccf000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccc8109490 0x7fccc80769a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.598+0000 7fcccf000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc8076ee0 0x7fccc81aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.598+0000 7fcccf000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fccc8114400 con 0x7fccc81043c0 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.598+0000 7fcccf000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fccc8114280 con 0x7fccc8076ee0 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.598+0000 7fcccf000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fccc8114580 con 0x7fccc8109490 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.598+0000 7fccc7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccc81043c0 0x7fccc8076460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.598+0000 7fccc7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccc81043c0 0x7fccc8076460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43222/0 (socket says 172.21.15.6:43222) 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.598+0000 7fccc7e00640 1 -- 172.21.15.6:0/179028319 learned_addr learned my addr 172.21.15.6:0/179028319 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.598+0000 7fccc7400640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccc8109490 0x7fccc80769a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.598+0000 7fcccd000640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc8076ee0 0x7fccc81aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.599+0000 7fccc7e00640 1 -- 172.21.15.6:0/179028319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccc8109490 msgr2=0x7fccc80769a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.599+0000 7fccc7e00640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccc8109490 0x7fccc80769a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.599+0000 7fccc7e00640 1 -- 172.21.15.6:0/179028319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc8076ee0 msgr2=0x7fccc81aaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.599+0000 7fccc7e00640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc8076ee0 0x7fccc81aaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.599+0000 7fccc7e00640 1 -- 172.21.15.6:0/179028319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc81ab4b0 con 0x7fccc81043c0 2024-04-26T01:49:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.599+0000 7fccc7e00640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccc81043c0 0x7fccc8076460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fccb8035040 tx=0x7fccb8032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.600+0000 7fccc4c00640 1 -- 172.21.15.6:0/179028319 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb8037040 con 0x7fccc81043c0 2024-04-26T01:49:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.600+0000 7fcccf000640 1 -- 172.21.15.6:0/179028319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc81ab740 con 0x7fccc81043c0 2024-04-26T01:49:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.600+0000 7fccc4c00640 1 -- 172.21.15.6:0/179028319 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccb8041070 con 0x7fccc81043c0 2024-04-26T01:49:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.600+0000 7fccc4c00640 1 -- 172.21.15.6:0/179028319 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb8003850 con 0x7fccc81043c0 2024-04-26T01:49:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.600+0000 7fcccf000640 1 -- 172.21.15.6:0/179028319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc81abba0 con 0x7fccc81043c0 2024-04-26T01:49:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.603+0000 7fccd1d85740 1 Processor -- start 2024-04-26T01:49:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.603+0000 7fccd1d85740 1 -- start start 2024-04-26T01:49:03.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.604+0000 7fccd1d85740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56025267b820 0x56025267bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.604+0000 7fccd1d85740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267c140 0x5602526864c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.604+0000 7fccd1d85740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560252686a00 0x560252688df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.604+0000 7fccd1d85740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5602525e1900 con 0x56025267b820 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.604+0000 7fccd1d85740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560252587ef0 con 0x56025267c140 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.604+0000 7fccd1d85740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5602525dd8c0 con 0x560252686a00 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.604+0000 7fcccd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56025267b820 0x56025267bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.604+0000 7fcccd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56025267b820 0x56025267bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43232/0 (socket says 172.21.15.6:43232) 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.604+0000 7fcccd000640 1 -- 172.21.15.6:0/1160436412 learned_addr learned my addr 172.21.15.6:0/1160436412 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.604+0000 7fccc7400640 1 --2- 172.21.15.6:0/1160436412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560252686a00 0x560252688df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.605+0000 7fcccd000640 1 -- 172.21.15.6:0/1160436412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560252686a00 msgr2=0x560252688df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.605+0000 7fcccd000640 1 --2- 172.21.15.6:0/1160436412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560252686a00 0x560252688df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.605+0000 7fcccd000640 1 -- 172.21.15.6:0/1160436412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267c140 msgr2=0x5602526864c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:49:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.605+0000 7fcccd000640 1 --2- 172.21.15.6:0/1160436412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267c140 0x5602526864c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.605+0000 7fcccd000640 1 -- 172.21.15.6:0/1160436412 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602522f4210 con 0x56025267b820 2024-04-26T01:49:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.605+0000 7fcccd000640 1 --2- 172.21.15.6:0/1160436412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56025267b820 0x56025267bc00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009f50 tx=0x7fccc00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe251ef2f5952211 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.605+0000 7fccc4c00640 1 -- 172.21.15.6:0/179028319 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fccb80039f0 con 0x7fccc81043c0 2024-04-26T01:49:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.605+0000 7fccbd600640 1 -- 172.21.15.6:0/1160436412 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0004d20 con 0x56025267b820 2024-04-26T01:49:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.605+0000 7fccbd600640 1 -- 172.21.15.6:0/1160436412 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccc0004050 con 0x56025267b820 2024-04-26T01:49:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.605+0000 7fccbd600640 1 -- 172.21.15.6:0/1160436412 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0005220 con 0x56025267b820 2024-04-26T01:49:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.606+0000 7fccd1d85740 1 -- 172.21.15.6:0/1160436412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56025267b820 msgr2=0x56025267bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.606+0000 7fccd1d85740 1 --2- 172.21.15.6:0/1160436412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56025267b820 0x56025267bc00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009f50 tx=0x7fccc00118d0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.606+0000 7fccd1d85740 1 -- 172.21.15.6:0/1160436412 shutdown_connections 2024-04-26T01:49:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.606+0000 7fccd1d85740 1 --2- 172.21.15.6:0/1160436412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560252686a00 0x560252688df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.606+0000 7fccd1d85740 1 --2- 172.21.15.6:0/1160436412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267c140 0x5602526864c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.606+0000 7fccd1d85740 1 --2- 172.21.15.6:0/1160436412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56025267b820 0x56025267bc00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.606+0000 7fccd1d85740 1 -- 172.21.15.6:0/1160436412 >> 172.21.15.6:0/1160436412 conn(0x560252677d30 msgr2=0x560252678110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.606+0000 7fccd1d85740 1 -- 172.21.15.6:0/1160436412 shutdown_connections 2024-04-26T01:49:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.606+0000 7fccd1d85740 1 -- 172.21.15.6:0/1160436412 wait complete. 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.607+0000 7fccc4c00640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fccac0a3c20 0x7fccac0a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.607+0000 7fccc4c00640 1 -- 172.21.15.6:0/179028319 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fccb80f01d0 con 0x7fccc81043c0 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.607+0000 7fccc7400640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fccac0a3c20 0x7fccac0a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.607+0000 7fccd1d85740 1 -- start start 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.607+0000 7fccd1d85740 1 Processor -- start 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.608+0000 7fccc7400640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fccac0a3c20 0x7fccac0a60e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x560252679150 tx=0x7fccb4009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.608+0000 7fccd1d85740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267b820 0x5602526839b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.608+0000 7fccd1d85740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267c140 0x560252683ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.608+0000 7fccd1d85740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x560252684430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.608+0000 7fccd1d85740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560252689630 con 0x560252686a00 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.608+0000 7fccd1d85740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56025259b310 con 0x56025267b820 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.608+0000 7fccd1d85740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5602526897b0 con 0x56025267c140 2024-04-26T01:49:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.608+0000 7fccd1d85740 10 client.0 _ll_register_callbacks cb 0x560252569cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.608+0000 7fcccd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x560252684430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.608+0000 7fccc7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267b820 0x5602526839b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.608+0000 7fcccd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x560252684430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43246/0 (socket says 172.21.15.6:43246) 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.609+0000 7fcccd000640 1 -- 172.21.15.6:0/3097694472 learned_addr learned my addr 172.21.15.6:0/3097694472 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.609+0000 7fccc7400640 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267c140 0x560252683ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.609+0000 7fcccd000640 1 -- 172.21.15.6:0/3097694472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267c140 msgr2=0x560252683ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.609+0000 7fcccd000640 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267c140 0x560252683ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.609+0000 7fcccd000640 1 -- 172.21.15.6:0/3097694472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267b820 msgr2=0x5602526839b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.609+0000 7fcccd000640 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267b820 0x5602526839b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.609+0000 7fcccd000640 1 -- 172.21.15.6:0/3097694472 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602523c0620 con 0x560252686a00 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.609+0000 7fccc7e00640 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267b820 0x5602526839b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.609+0000 7fcccd000640 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x560252684430 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fccc001b040 tx=0x7fccc0004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12bdaa1ef8e15ed6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.609+0000 7fccb2000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:49:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.610+0000 7fccb2000640 1 -- 172.21.15.6:0/3097694472 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0027070 con 0x560252686a00 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.610+0000 7fccb2000640 1 -- 172.21.15.6:0/3097694472 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fccc0005050 con 0x560252686a00 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.610+0000 7fccd1d85740 1 -- 172.21.15.6:0/3097694472 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccc0004050 con 0x560252686a00 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.610+0000 7fccb2000640 1 -- 172.21.15.6:0/3097694472 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fccc0009be0 con 0x560252686a00 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.610+0000 7fccd1d85740 10 client.15417 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.610+0000 7fccd1d85740 1 -- 172.21.15.6:0/3097694472 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5602526958f0 con 0x560252686a00 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.610+0000 7fccb2000640 1 -- 172.21.15.6:0/3097694472 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccc0004050 con 0x560252686a00 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.611+0000 7fccb2000640 1 -- 172.21.15.6:0/3097694472 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fccc000f5f0 con 0x560252686a00 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.611+0000 7fccd1d85740 10 client.15417 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.611+0000 7fccd1d85740 10 client.15417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.611+0000 7fccd1d85740 20 client.15417 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.611+0000 7fccd1d85740 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560252696d00 0x5602526990f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.611+0000 7fccd1d85740 4 client.15417 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.611+0000 7fccd1d85740 1 -- 172.21.15.6:0/3097694472 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602525f58b0 con 0x560252696d00 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.611+0000 7fccc7e00640 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560252696d00 0x5602526990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.612+0000 7fccc7e00640 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560252696d00 0x5602526990f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=681052446ed2884a server_cookie=905fa87e2ee18e0e in_seq=0 out_seq=0 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.612+0000 7fccb2000640 10 client.15417 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.612+0000 7fccb2000640 1 -- 172.21.15.6:0/3097694472 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602525f58b0 con 0x560252696d00 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.612+0000 7fccb2000640 10 client.15417 handle_command_reply: tid=0 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.612+0000 7fccd1d85740 1 client.15417 shutdown 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.613+0000 7fccd1d85740 1 -- 172.21.15.6:0/3097694472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 msgr2=0x560252684430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.613+0000 7fccd1d85740 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x560252684430 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fccc001b040 tx=0x7fccc0004410 comp rx=0 tx=0).stop 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.613+0000 7fccd1d85740 1 -- 172.21.15.6:0/3097694472 shutdown_connections 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.613+0000 7fccd1d85740 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560252696d00 0x5602526990f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.613+0000 7fccd1d85740 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x560252684430 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.613+0000 7fccd1d85740 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267c140 0x560252683ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.613+0000 7fccd1d85740 1 --2- 172.21.15.6:0/3097694472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267b820 0x5602526839b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.613+0000 7fccb2000640 0 client.15417 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.613+0000 7fccd1d85740 1 -- 172.21.15.6:0/3097694472 >> 172.21.15.6:0/3097694472 conn(0x560252677d30 msgr2=0x56025267dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.613+0000 7fccd1d85740 1 -- 172.21.15.6:0/3097694472 shutdown_connections 2024-04-26T01:49:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.613+0000 7fccd1d85740 1 -- 172.21.15.6:0/3097694472 wait complete. 2024-04-26T01:49:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.613+0000 7fccd1d85740 20 client.15417 trim_cache size 0 max 16384 2024-04-26T01:49:03.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.634+0000 7fccd1d85740 1 Processor -- start 2024-04-26T01:49:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fccd1d85740 1 -- start start 2024-04-26T01:49:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fccd1d85740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267b820 0x5602526924f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fccd1d85740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267c140 0x56025267e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fccd1d85740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x560252676d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fccd1d85740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fccc81b6c90 con 0x560252686a00 2024-04-26T01:49:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fccd1d85740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fccc81b4500 con 0x56025267b820 2024-04-26T01:49:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fccd1d85740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fccc81b2350 con 0x56025267c140 2024-04-26T01:49:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fcccd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x560252676d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fccc7400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267c140 0x56025267e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fccc7400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267c140 0x56025267e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:54738/0 (socket says 172.21.15.6:54738) 2024-04-26T01:49:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fccc7400640 1 -- 172.21.15.6:0/2669866570 learned_addr learned my addr 172.21.15.6:0/2669866570 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:03.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fcccd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x560252676d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43252/0 (socket says 172.21.15.6:43252) 2024-04-26T01:49:03.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.635+0000 7fccc7e00640 1 --2- 172.21.15.6:0/2669866570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267b820 0x5602526924f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.636+0000 7fccc7400640 1 -- 172.21.15.6:0/2669866570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267b820 msgr2=0x5602526924f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:03.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.636+0000 7fccc7400640 1 --2- 172.21.15.6:0/2669866570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267b820 0x5602526924f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.636+0000 7fccc7400640 1 -- 172.21.15.6:0/2669866570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 msgr2=0x560252676d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:03.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.636+0000 7fccc7400640 1 --2- 172.21.15.6:0/2669866570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x560252676d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.636+0000 7fccc7400640 1 -- 172.21.15.6:0/2669866570 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56025267bf70 con 0x56025267c140 2024-04-26T01:49:03.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.636+0000 7fccc7400640 1 --2- 172.21.15.6:0/2669866570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267c140 0x56025267e1e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fccc806b6e0 tx=0x7fccb4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93b1cda32da23d28 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:03.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.636+0000 7fccbcc00640 1 -- 172.21.15.6:0/2669866570 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb401cd80 con 0x56025267c140 2024-04-26T01:49:03.636 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.637+0000 7fccbcc00640 1 -- 172.21.15.6:0/2669866570 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccb40035f0 con 0x56025267c140 2024-04-26T01:49:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.637+0000 7fccbcc00640 1 -- 172.21.15.6:0/2669866570 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb4028460 con 0x56025267c140 2024-04-26T01:49:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.638+0000 7fccd1d85740 1 -- 172.21.15.6:0/2669866570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267c140 msgr2=0x56025267e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.638+0000 7fccd1d85740 1 --2- 172.21.15.6:0/2669866570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267c140 0x56025267e1e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fccc806b6e0 tx=0x7fccb4016000 comp rx=0 tx=0).stop 2024-04-26T01:49:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.638+0000 7fccd1d85740 1 -- 172.21.15.6:0/2669866570 shutdown_connections 2024-04-26T01:49:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.638+0000 7fccd1d85740 1 --2- 172.21.15.6:0/2669866570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x560252676d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.638+0000 7fccd1d85740 1 --2- 172.21.15.6:0/2669866570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267c140 0x56025267e1e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.638+0000 7fccd1d85740 1 --2- 172.21.15.6:0/2669866570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267b820 0x5602526924f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.638+0000 7fccd1d85740 1 -- 172.21.15.6:0/2669866570 >> 172.21.15.6:0/2669866570 conn(0x560252677d30 msgr2=0x560252676600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.638+0000 7fccd1d85740 1 -- 172.21.15.6:0/2669866570 shutdown_connections 2024-04-26T01:49:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.638+0000 7fccd1d85740 1 -- 172.21.15.6:0/2669866570 wait complete. 2024-04-26T01:49:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.638+0000 7fccd1d85740 1 -- start start 2024-04-26T01:49:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.639+0000 7fccd1d85740 1 Processor -- start 2024-04-26T01:49:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.639+0000 7fccd1d85740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267b820 0x56025268ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.639+0000 7fccd1d85740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267c140 0x56025268bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.639+0000 7fccd1d85740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x56025268c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.639+0000 7fccd1d85740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fccc81b29c0 con 0x560252686a00 2024-04-26T01:49:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.639+0000 7fccd1d85740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fccc81b3e30 con 0x56025267c140 2024-04-26T01:49:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.639+0000 7fccd1d85740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fccc81b3090 con 0x56025267b820 2024-04-26T01:49:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fccd1d85740 10 client.0 _ll_register_callbacks cb 0x7fccc0003e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fccc7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267b820 0x56025268ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fcccd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x56025268c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fccc7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267b820 0x56025268ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:54748/0 (socket says 172.21.15.6:54748) 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fccc7e00640 1 -- 172.21.15.6:0/3860650883 learned_addr learned my addr 172.21.15.6:0/3860650883 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fccc7400640 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267c140 0x56025268bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fccc7e00640 1 -- 172.21.15.6:0/3860650883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267c140 msgr2=0x56025268bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fccc7e00640 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267c140 0x56025268bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fccc7e00640 1 -- 172.21.15.6:0/3860650883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 msgr2=0x56025268c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fccc7e00640 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x56025268c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fccc7e00640 1 -- 172.21.15.6:0/3860650883 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56025268cb30 con 0x56025267b820 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fcccd000640 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x56025268c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.640+0000 7fccc7e00640 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267b820 0x56025268ec20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fccb80edac0 tx=0x7fccb802a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75763e42760bc774 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.641+0000 7fccb2000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.641+0000 7fccb2000640 1 -- 172.21.15.6:0/3860650883 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb8030450 con 0x56025267b820 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.641+0000 7fccb2000640 1 -- 172.21.15.6:0/3860650883 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fccb80c4a40 con 0x56025267b820 2024-04-26T01:49:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.641+0000 7fccd1d85740 1 -- 172.21.15.6:0/3860650883 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccb40035f0 con 0x56025267b820 2024-04-26T01:49:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.641+0000 7fccb2000640 1 -- 172.21.15.6:0/3860650883 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fccb80bc960 con 0x56025267b820 2024-04-26T01:49:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.641+0000 7fccd1d85740 10 client.25588 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.641+0000 7fccd1d85740 1 -- 172.21.15.6:0/3860650883 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5602526fcf90 con 0x56025267b820 2024-04-26T01:49:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.641+0000 7fccb2000640 1 -- 172.21.15.6:0/3860650883 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccb40035f0 con 0x56025267b820 2024-04-26T01:49:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.641+0000 7fccb2000640 1 -- 172.21.15.6:0/3860650883 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fccb80c0160 con 0x56025267b820 2024-04-26T01:49:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.642+0000 7fccd1d85740 10 client.25588 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.642+0000 7fccd1d85740 10 client.25588 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.642+0000 7fccd1d85740 20 client.25588 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.642+0000 7fccd1d85740 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5602526fd6c0 0x5602526ffae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.642+0000 7fccd1d85740 4 client.25588 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.642+0000 7fccd1d85740 1 -- 172.21.15.6:0/3860650883 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5602525ef320 con 0x5602526fd6c0 2024-04-26T01:49:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.642+0000 7fccc7400640 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5602526fd6c0 0x5602526ffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.642+0000 7fccc7400640 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5602526fd6c0 0x5602526ffae0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c99562f1b6d4529 server_cookie=97f641daf0e481b9 in_seq=0 out_seq=0 2024-04-26T01:49:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.642+0000 7fccb2000640 10 client.25588 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.665+0000 7fccb2000640 1 -- 172.21.15.6:0/3860650883 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5602525ef320 con 0x5602526fd6c0 2024-04-26T01:49:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.665+0000 7fccb2000640 10 client.25588 handle_command_reply: tid=0 2024-04-26T01:49:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.665+0000 7fccd1d85740 1 client.25588 shutdown 2024-04-26T01:49:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.665+0000 7fccd1d85740 1 -- 172.21.15.6:0/3860650883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267b820 msgr2=0x56025268ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.665+0000 7fccd1d85740 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267b820 0x56025268ec20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fccb80edac0 tx=0x7fccb802a340 comp rx=0 tx=0).stop 2024-04-26T01:49:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.665+0000 7fccd1d85740 1 -- 172.21.15.6:0/3860650883 shutdown_connections 2024-04-26T01:49:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.665+0000 7fccd1d85740 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5602526fd6c0 0x5602526ffae0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.665+0000 7fccd1d85740 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560252686a00 0x56025268c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.665+0000 7fccd1d85740 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56025267c140 0x56025268bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.665+0000 7fccd1d85740 1 --2- 172.21.15.6:0/3860650883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56025267b820 0x56025268ec20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.665+0000 7fccb2000640 0 client.25588 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.665+0000 7fccd1d85740 1 -- 172.21.15.6:0/3860650883 >> 172.21.15.6:0/3860650883 conn(0x560252677d30 msgr2=0x560252676e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.666+0000 7fccd1d85740 1 -- 172.21.15.6:0/3860650883 shutdown_connections 2024-04-26T01:49:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.666+0000 7fccd1d85740 1 -- 172.21.15.6:0/3860650883 wait complete. 2024-04-26T01:49:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.666+0000 7fccd1d85740 20 client.25588 trim_cache size 0 max 16384 2024-04-26T01:49:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:49:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:49:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:49:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:49:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.667+0000 7fcccf000640 1 -- 172.21.15.6:0/179028319 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fccac0a3c20 msgr2=0x7fccac0a60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.667+0000 7fcccf000640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fccac0a3c20 0x7fccac0a60e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x560252679150 tx=0x7fccb4009260 comp rx=0 tx=0).stop 2024-04-26T01:49:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.667+0000 7fcccf000640 1 -- 172.21.15.6:0/179028319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccc81043c0 msgr2=0x7fccc8076460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.667+0000 7fcccf000640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccc81043c0 0x7fccc8076460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fccb8035040 tx=0x7fccb8032d50 comp rx=0 tx=0).stop 2024-04-26T01:49:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.667+0000 7fcccf000640 1 -- 172.21.15.6:0/179028319 shutdown_connections 2024-04-26T01:49:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.667+0000 7fcccf000640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fccac0a3c20 0x7fccac0a60e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.667+0000 7fcccf000640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc8076ee0 0x7fccc81aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.667+0000 7fcccf000640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccc8109490 0x7fccc80769a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.667+0000 7fcccf000640 1 --2- 172.21.15.6:0/179028319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccc81043c0 0x7fccc8076460 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.667+0000 7fcccf000640 1 -- 172.21.15.6:0/179028319 >> 172.21.15.6:0/179028319 conn(0x7fccc80ffc40 msgr2=0x7fccc8100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.667+0000 7fcccf000640 1 -- 172.21.15.6:0/179028319 shutdown_connections 2024-04-26T01:49:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:03.667+0000 7fcccf000640 1 -- 172.21.15.6:0/179028319 wait complete. 2024-04-26T01:49:04.015 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:49:04.459 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:04 smithi006 ceph-mon[28419]: pgmap v99: 97 pgs: 97 active+clean; 35 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 988 KiB/s wr, 51 op/s 2024-04-26T01:49:04.561 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.30 MB/sec warmup 88 sec latency 31953.034 ms 2024-04-26T01:49:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:04 smithi188 ceph-mon[32751]: pgmap v99: 97 pgs: 97 active+clean; 35 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 988 KiB/s wr, 51 op/s 2024-04-26T01:49:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:04 smithi029 ceph-mon[32775]: pgmap v99: 97 pgs: 97 active+clean; 35 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 988 KiB/s wr, 51 op/s 2024-04-26T01:49:05.560 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.24 MB/sec warmup 89 sec latency 32953.155 ms 2024-04-26T01:49:06.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.367+0000 7fc157e00700 1 -- 172.21.15.6:0/4232934888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150103460 msgr2=0x7fc15010f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.367+0000 7fc157e00700 1 --2- 172.21.15.6:0/4232934888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150103460 0x7fc15010f940 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc148009f90 tx=0x7fc14800b330 comp rx=0 tx=0).stop 2024-04-26T01:49:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.367+0000 7fc157e00700 1 -- 172.21.15.6:0/4232934888 shutdown_connections 2024-04-26T01:49:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.367+0000 7fc157e00700 1 --2- 172.21.15.6:0/4232934888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150103460 0x7fc15010f940 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.367+0000 7fc157e00700 1 --2- 172.21.15.6:0/4232934888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc150108a70 0x7fc150108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.367+0000 7fc157e00700 1 --2- 172.21.15.6:0/4232934888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc150102ac0 0x7fc150102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.367+0000 7fc157e00700 1 -- 172.21.15.6:0/4232934888 >> 172.21.15.6:0/4232934888 conn(0x7fc150079b10 msgr2=0x7fc150079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.367+0000 7fc157e00700 1 -- 172.21.15.6:0/4232934888 shutdown_connections 2024-04-26T01:49:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.367+0000 7fc157e00700 1 -- 172.21.15.6:0/4232934888 wait complete. 2024-04-26T01:49:06.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.369+0000 7fc157e00700 1 Processor -- start 2024-04-26T01:49:06.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.369+0000 7fc157e00700 1 -- start start 2024-04-26T01:49:06.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.369+0000 7fc157e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc150102ac0 0x7fc1501990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:06.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc157e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150103460 0x7fc150199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:06.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc157e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc150108a70 0x7fc15019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:06.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc157e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc150112c50 con 0x7fc150108a70 2024-04-26T01:49:06.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc157e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc150112ad0 con 0x7fc150103460 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc157e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc150112dd0 con 0x7fc150102ac0 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc155400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc150102ac0 0x7fc1501990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc155400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc150102ac0 0x7fc1501990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:54766/0 (socket says 172.21.15.6:54766) 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc155400700 1 -- 172.21.15.6:0/1740692319 learned_addr learned my addr 172.21.15.6:0/1740692319 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc154a00700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150103460 0x7fc150199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc155e00700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc150108a70 0x7fc15019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc155400700 1 -- 172.21.15.6:0/1740692319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150103460 msgr2=0x7fc150199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc155400700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150103460 0x7fc150199600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc155400700 1 -- 172.21.15.6:0/1740692319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc150108a70 msgr2=0x7fc15019d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc155400700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc150108a70 0x7fc15019d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.370+0000 7fc155400700 1 -- 172.21.15.6:0/1740692319 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15019e0d0 con 0x7fc150102ac0 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.371+0000 7fc155e00700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc150108a70 0x7fc15019d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:49:06.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.371+0000 7fc155400700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc150102ac0 0x7fc1501990c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc14400e9a0 tx=0x7fc14400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:06.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.371+0000 7fc14e000700 1 -- 172.21.15.6:0/1740692319 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14400cbd0 con 0x7fc150102ac0 2024-04-26T01:49:06.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.371+0000 7fc157e00700 1 -- 172.21.15.6:0/1740692319 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc15019e3c0 con 0x7fc150102ac0 2024-04-26T01:49:06.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.371+0000 7fc14e000700 1 -- 172.21.15.6:0/1740692319 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc14400cd70 con 0x7fc150102ac0 2024-04-26T01:49:06.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.371+0000 7fc14e000700 1 -- 172.21.15.6:0/1740692319 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc144018680 con 0x7fc150102ac0 2024-04-26T01:49:06.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.372+0000 7fc157e00700 1 -- 172.21.15.6:0/1740692319 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc150074c70 con 0x7fc150102ac0 2024-04-26T01:49:06.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.373+0000 7fc157e00700 1 -- 172.21.15.6:0/1740692319 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc150069730 con 0x7fc150102ac0 2024-04-26T01:49:06.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.375+0000 7fc14e000700 1 -- 172.21.15.6:0/1740692319 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fc144010430 con 0x7fc150102ac0 2024-04-26T01:49:06.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.376+0000 7fc14e000700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc1380a3e30 0x7fc1380a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:06.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.376+0000 7fc14e000700 1 -- 172.21.15.6:0/1740692319 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fc1440d09b0 con 0x7fc150102ac0 2024-04-26T01:49:06.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.376+0000 7fc154a00700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc1380a3e30 0x7fc1380a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:06.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.377+0000 7fc154a00700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc1380a3e30 0x7fc1380a62f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009750 tx=0x7fc13c009310 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:06.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.380+0000 7fc14e000700 1 -- 172.21.15.6:0/1740692319 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc14409cfd0 con 0x7fc150102ac0 2024-04-26T01:49:06.561 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.18 MB/sec warmup 90 sec latency 33953.252 ms 2024-04-26T01:49:06.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:06 smithi006 ceph-mon[28419]: pgmap v100: 97 pgs: 97 active+clean; 35 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 988 KiB/s wr, 51 op/s 2024-04-26T01:49:06.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:06 smithi188 ceph-mon[32751]: pgmap v100: 97 pgs: 97 active+clean; 35 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 988 KiB/s wr, 51 op/s 2024-04-26T01:49:06.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.665+0000 7fc157e00700 1 -- 172.21.15.6:0/1740692319 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc15019a420 con 0x7fc150102ac0 2024-04-26T01:49:06.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.666+0000 7fc14e000700 1 -- 172.21.15.6:0/1740692319 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11853 (secure 0 0 0) 0x7fc14409c8d0 con 0x7fc150102ac0 2024-04-26T01:49:06.665 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:49:06.665 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:47:59.436110+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25474":{"gid":25474,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1633748308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1633748308},{"type":"v1","addr":"172.21.15.188:6835","nonce":1633748308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:49:06.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.670+0000 7fc157e00700 1 -- 172.21.15.6:0/1740692319 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc1380a3e30 msgr2=0x7fc1380a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:06.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.670+0000 7fc157e00700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc1380a3e30 0x7fc1380a62f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009750 tx=0x7fc13c009310 comp rx=0 tx=0).stop 2024-04-26T01:49:06.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.670+0000 7fc157e00700 1 -- 172.21.15.6:0/1740692319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc150102ac0 msgr2=0x7fc1501990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:06.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.670+0000 7fc157e00700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc150102ac0 0x7fc1501990c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc14400e9a0 tx=0x7fc14400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:49:06.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.670+0000 7fc157e00700 1 -- 172.21.15.6:0/1740692319 shutdown_connections 2024-04-26T01:49:06.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.670+0000 7fc157e00700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc1380a3e30 0x7fc1380a62f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:06.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.670+0000 7fc157e00700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc150108a70 0x7fc15019d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:06.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.670+0000 7fc157e00700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc150102ac0 0x7fc1501990c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:06.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.670+0000 7fc157e00700 1 --2- 172.21.15.6:0/1740692319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150103460 0x7fc150199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:06.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.670+0000 7fc157e00700 1 -- 172.21.15.6:0/1740692319 >> 172.21.15.6:0/1740692319 conn(0x7fc150079b10 msgr2=0x7fc150100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:06.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.671+0000 7fc157e00700 1 -- 172.21.15.6:0/1740692319 shutdown_connections 2024-04-26T01:49:06.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:06.671+0000 7fc157e00700 1 -- 172.21.15.6:0/1740692319 wait complete. 2024-04-26T01:49:06.670 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 31 2024-04-26T01:49:06.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:06 smithi029 ceph-mon[32775]: pgmap v100: 97 pgs: 97 active+clean; 35 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 988 KiB/s wr, 51 op/s 2024-04-26T01:49:07.551 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id c54a4666 --release --await' 2024-04-26T01:49:07.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id c54a4666 --release --await 2024-04-26T01:49:07.561 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.12 MB/sec warmup 91 sec latency 34953.323 ms 2024-04-26T01:49:07.623 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:07 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1740692319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:49:07.623 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:07 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:49:07.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:07 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1740692319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:49:07.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:07 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:49:07.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:07 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1740692319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:49:07.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:07 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:49:08.015 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:49:08.456 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:08 smithi006 ceph-mon[28419]: pgmap v101: 97 pgs: 97 active+clean; 33 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.2 MiB/s wr, 75 op/s 2024-04-26T01:49:08.561 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.07 MB/sec warmup 92 sec latency 35953.396 ms 2024-04-26T01:49:08.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:08 smithi188 ceph-mon[32751]: pgmap v101: 97 pgs: 97 active+clean; 33 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.2 MiB/s wr, 75 op/s 2024-04-26T01:49:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:08 smithi029 ceph-mon[32775]: pgmap v101: 97 pgs: 97 active+clean; 33 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.2 MiB/s wr, 75 op/s 2024-04-26T01:49:09.561 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83234 5.01 MB/sec warmup 93 sec latency 36953.538 ms 2024-04-26T01:49:09.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:09 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:49:09.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:09 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:49:09.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:09 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:49:09.690 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:09 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:49:09.690 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:09 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:49:09.690 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:09 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:49:09.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:09 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:49:09.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:09 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:49:09.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:09 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:49:10.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.189+0000 7fb5d5200700 1 -- 172.21.15.6:0/1318593955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5d010f270 msgr2=0x7fb5d0111660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.189+0000 7fb5d5200700 1 --2- 172.21.15.6:0/1318593955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5d010f270 0x7fb5d0111660 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009990 tx=0x7fb5c8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.190+0000 7fb5d5200700 1 -- 172.21.15.6:0/1318593955 shutdown_connections 2024-04-26T01:49:10.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.190+0000 7fb5d5200700 1 --2- 172.21.15.6:0/1318593955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5d010f270 0x7fb5d0111660 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.190+0000 7fb5d5200700 1 --2- 172.21.15.6:0/1318593955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d01016c0 0x7fb5d0101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.190+0000 7fb5d5200700 1 --2- 172.21.15.6:0/1318593955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5d0102070 0x7fb5d010ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.190+0000 7fb5d5200700 1 -- 172.21.15.6:0/1318593955 >> 172.21.15.6:0/1318593955 conn(0x7fb5d00fd360 msgr2=0x7fb5d00ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:10.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.190+0000 7fb5d5200700 1 -- 172.21.15.6:0/1318593955 shutdown_connections 2024-04-26T01:49:10.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.190+0000 7fb5d5200700 1 -- 172.21.15.6:0/1318593955 wait complete. 2024-04-26T01:49:10.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.191+0000 7fb5d5200700 1 Processor -- start 2024-04-26T01:49:10.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.191+0000 7fb5d5200700 1 -- start start 2024-04-26T01:49:10.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.192+0000 7fb5d5200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5d01016c0 0x7fb5d019b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.192+0000 7fb5d5200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5d0102070 0x7fb5d019b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.192+0000 7fb5d5200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d010f270 0x7fb5d019fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.192+0000 7fb5d5200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d0114db0 con 0x7fb5d0102070 2024-04-26T01:49:10.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.192+0000 7fb5d5200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d0114c30 con 0x7fb5d010f270 2024-04-26T01:49:10.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.192+0000 7fb5d5200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d0114f30 con 0x7fb5d01016c0 2024-04-26T01:49:10.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.192+0000 7fb5cfe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d010f270 0x7fb5d019fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.192+0000 7fb5c6a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5d0102070 0x7fb5d019b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.192+0000 7fb5cfe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d010f270 0x7fb5d019fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53016/0 (socket says 172.21.15.6:53016) 2024-04-26T01:49:10.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.192+0000 7fb5cfe00700 1 -- 172.21.15.6:0/1982043426 learned_addr learned my addr 172.21.15.6:0/1982043426 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:10.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.192+0000 7fb5cf400700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5d01016c0 0x7fb5d019b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.193+0000 7fb5cfe00700 1 -- 172.21.15.6:0/1982043426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5d01016c0 msgr2=0x7fb5d019b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.193+0000 7fb5cfe00700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5d01016c0 0x7fb5d019b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.193+0000 7fb5cfe00700 1 -- 172.21.15.6:0/1982043426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5d0102070 msgr2=0x7fb5d019b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.193+0000 7fb5cfe00700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5d0102070 0x7fb5d019b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.193+0000 7fb5cfe00700 1 -- 172.21.15.6:0/1982043426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d01a0250 con 0x7fb5d010f270 2024-04-26T01:49:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.193+0000 7fb5c6a00700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5d0102070 0x7fb5d019b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.193+0000 7fb5cfe00700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d010f270 0x7fb5d019fb10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80047b0 tx=0x7fb5c8009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.193+0000 7fb5ccc00700 1 -- 172.21.15.6:0/1982043426 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8026070 con 0x7fb5d010f270 2024-04-26T01:49:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.194+0000 7fb5ccc00700 1 -- 172.21.15.6:0/1982043426 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5c80047e0 con 0x7fb5d010f270 2024-04-26T01:49:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.194+0000 7fb5d5200700 1 -- 172.21.15.6:0/1982043426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d01a04e0 con 0x7fb5d010f270 2024-04-26T01:49:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.194+0000 7fb5ccc00700 1 -- 172.21.15.6:0/1982043426 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c80053b0 con 0x7fb5d010f270 2024-04-26T01:49:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.194+0000 7fb5d5200700 1 -- 172.21.15.6:0/1982043426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d01a7dc0 con 0x7fb5d010f270 2024-04-26T01:49:10.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.197+0000 7fb5ccc00700 1 -- 172.21.15.6:0/1982043426 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fb5c8029070 con 0x7fb5d010f270 2024-04-26T01:49:10.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.200+0000 7fb5e1093b80 1 Processor -- start 2024-04-26T01:49:10.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.200+0000 7fb5ccc00700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb5b40a3db0 0x7fb5b40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.200+0000 7fb5ccc00700 1 -- 172.21.15.6:0/1982043426 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fb5c80d1160 con 0x7fb5d010f270 2024-04-26T01:49:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.200+0000 7fb5e1093b80 1 -- start start 2024-04-26T01:49:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.200+0000 7fb5c6a00700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb5b40a3db0 0x7fb5b40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.200+0000 7fb5e1093b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 0x55981e1aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.200+0000 7fb5e1093b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1ad410 0x55981e1b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.200+0000 7fb5e1093b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1b82c0 0x55981e1ba6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.200+0000 7fb5e1093b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55981e0fff90 con 0x55981e1b82c0 2024-04-26T01:49:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5e1093b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55981e0a6830 con 0x55981e1acaf0 2024-04-26T01:49:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5e1093b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55981e0fbf50 con 0x55981e1ad410 2024-04-26T01:49:10.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5cf400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 0x55981e1aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5c6a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1ad410 0x55981e1b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5cfe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1b82c0 0x55981e1ba6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5cf400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 0x55981e1aced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53026/0 (socket says 172.21.15.6:53026) 2024-04-26T01:49:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5cf400700 1 -- 172.21.15.6:0/3384542547 learned_addr learned my addr 172.21.15.6:0/3384542547 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5c6a00700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb5b40a3db0 0x7fb5b40a6270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0103010 tx=0x7fb5b8009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5cf400700 1 -- 172.21.15.6:0/3384542547 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1ad410 msgr2=0x55981e1b7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5cf400700 1 --2- 172.21.15.6:0/3384542547 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1ad410 0x55981e1b7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5cf400700 1 -- 172.21.15.6:0/3384542547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1b82c0 msgr2=0x55981e1ba6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5cf400700 1 --2- 172.21.15.6:0/3384542547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1b82c0 0x55981e1ba6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5cf400700 1 -- 172.21.15.6:0/3384542547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55981e09c780 con 0x55981e1acaf0 2024-04-26T01:49:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.201+0000 7fb5c6a00700 1 --2- 172.21.15.6:0/3384542547 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1ad410 0x55981e1b7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:10.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.202+0000 7fb5cf400700 1 --2- 172.21.15.6:0/3384542547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 0x55981e1aced0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0011650 tx=0x7fb5c0009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38c55107e18141d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:10.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.202+0000 7fb5c4c00700 1 -- 172.21.15.6:0/3384542547 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0019020 con 0x55981e1acaf0 2024-04-26T01:49:10.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.202+0000 7fb5c4c00700 1 -- 172.21.15.6:0/3384542547 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5c00079b0 con 0x55981e1acaf0 2024-04-26T01:49:10.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.202+0000 7fb5c4c00700 1 -- 172.21.15.6:0/3384542547 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0015af0 con 0x55981e1acaf0 2024-04-26T01:49:10.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.203+0000 7fb5e1093b80 1 -- 172.21.15.6:0/3384542547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 msgr2=0x55981e1aced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.203+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/3384542547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 0x55981e1aced0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0011650 tx=0x7fb5c0009920 comp rx=0 tx=0).stop 2024-04-26T01:49:10.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.203+0000 7fb5e1093b80 1 -- 172.21.15.6:0/3384542547 shutdown_connections 2024-04-26T01:49:10.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.203+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/3384542547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1b82c0 0x55981e1ba6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.203+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/3384542547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 0x55981e1aced0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.203+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/3384542547 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1ad410 0x55981e1b7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.203+0000 7fb5e1093b80 1 -- 172.21.15.6:0/3384542547 >> 172.21.15.6:0/3384542547 conn(0x55981e0db6a0 msgr2=0x55981e1a8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:10.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.203+0000 7fb5e1093b80 1 -- 172.21.15.6:0/3384542547 shutdown_connections 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.203+0000 7fb5e1093b80 1 -- 172.21.15.6:0/3384542547 wait complete. 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.204+0000 7fb5e1093b80 1 -- start start 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.204+0000 7fb5e1093b80 1 Processor -- start 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5e1093b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 0x55981e1b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5e1093b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1ad410 0x55981e1b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5e1093b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1b82c0 0x55981e1b5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5e1093b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55981e1bae40 con 0x55981e1acaf0 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5e1093b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55981e0b9960 con 0x55981e1b82c0 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5e1093b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55981e1bafc0 con 0x55981e1ad410 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5e1093b80 10 client.0 _ll_register_callbacks cb 0x55981e098820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5cf400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 0x55981e1b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5cfe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1b82c0 0x55981e1b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5cfe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1b82c0 0x55981e1b5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53038/0 (socket says 172.21.15.6:53038) 2024-04-26T01:49:10.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5cf400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 0x55981e1b79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40278/0 (socket says 172.21.15.6:40278) 2024-04-26T01:49:10.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5cfe00700 1 -- 172.21.15.6:0/949010445 learned_addr learned my addr 172.21.15.6:0/949010445 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:10.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.205+0000 7fb5c6a00700 1 --2- 172.21.15.6:0/949010445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1ad410 0x55981e1b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.206+0000 7fb5cfe00700 1 -- 172.21.15.6:0/949010445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1ad410 msgr2=0x55981e1b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.206+0000 7fb5cfe00700 1 --2- 172.21.15.6:0/949010445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1ad410 0x55981e1b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.206+0000 7fb5cfe00700 1 -- 172.21.15.6:0/949010445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 msgr2=0x55981e1b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.206+0000 7fb5cfe00700 1 --2- 172.21.15.6:0/949010445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 0x55981e1b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.206+0000 7fb5cfe00700 1 -- 172.21.15.6:0/949010445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55981e04d9d0 con 0x55981e1b82c0 2024-04-26T01:49:10.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.206+0000 7fb5cfe00700 1 --2- 172.21.15.6:0/949010445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1b82c0 0x55981e1b5210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009350 tx=0x7fb5c8005c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c8161ab526359be server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:10.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.206+0000 7fb5bd600700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.206+0000 7fb5bd600700 1 -- 172.21.15.6:0/949010445 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c809d110 con 0x55981e1b82c0 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.206+0000 7fb5e1093b80 1 -- 172.21.15.6:0/949010445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5c00079b0 con 0x55981e1b82c0 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.206+0000 7fb5bd600700 1 -- 172.21.15.6:0/949010445 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fb5c8099780 con 0x55981e1b82c0 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.207+0000 7fb5bd600700 1 -- 172.21.15.6:0/949010445 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb5c80a14c0 con 0x55981e1b82c0 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.207+0000 7fb5e1093b80 10 client.25556 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.207+0000 7fb5e1093b80 1 -- 172.21.15.6:0/949010445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55981e096bf0 con 0x55981e1b82c0 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.207+0000 7fb5bd600700 1 -- 172.21.15.6:0/949010445 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5c00079b0 con 0x55981e1b82c0 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.207+0000 7fb5bd600700 1 -- 172.21.15.6:0/949010445 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fb5c80a5860 con 0x55981e1b82c0 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.207+0000 7fb5e1093b80 10 client.25556 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.207+0000 7fb5e1093b80 10 client.25556 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.207+0000 7fb5e1093b80 20 client.25556 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.207+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/949010445 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55981e1c79e0 0x55981e1c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.207+0000 7fb5e1093b80 4 client.25556 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.207+0000 7fb5e1093b80 1 -- 172.21.15.6:0/949010445 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55981e10d780 con 0x55981e1c79e0 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.208+0000 7fb5cf400700 1 --2- 172.21.15.6:0/949010445 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55981e1c79e0 0x55981e1c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.208+0000 7fb5cf400700 1 --2- 172.21.15.6:0/949010445 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55981e1c79e0 0x55981e1c9e60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f671b3bc6bdb38ea server_cookie=bcdf9443818234f0 in_seq=0 out_seq=0 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.208+0000 7fb5bd600700 10 client.25556 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.209+0000 7fb5bd600700 1 -- 172.21.15.6:0/949010445 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55981e10d780 con 0x55981e1c79e0 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.209+0000 7fb5bd600700 10 client.25556 handle_command_reply: tid=0 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.209+0000 7fb5e1093b80 1 client.25556 shutdown 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.210+0000 7fb5e1093b80 1 -- 172.21.15.6:0/949010445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1b82c0 msgr2=0x55981e1b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.210+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/949010445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1b82c0 0x55981e1b5210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009350 tx=0x7fb5c8005c80 comp rx=0 tx=0).stop 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.210+0000 7fb5e1093b80 1 -- 172.21.15.6:0/949010445 shutdown_connections 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.210+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/949010445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1b82c0 0x55981e1b5210 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.210+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/949010445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 0x55981e1b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.210+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/949010445 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55981e1c79e0 0x55981e1c9e60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.210+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/949010445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1ad410 0x55981e1b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.210+0000 7fb5bd600700 0 client.25556 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:10.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.210+0000 7fb5e1093b80 1 -- 172.21.15.6:0/949010445 >> 172.21.15.6:0/949010445 conn(0x55981e0db6a0 msgr2=0x55981e1aeb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:10.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.210+0000 7fb5e1093b80 1 -- 172.21.15.6:0/949010445 shutdown_connections 2024-04-26T01:49:10.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.210+0000 7fb5e1093b80 1 -- 172.21.15.6:0/949010445 wait complete. 2024-04-26T01:49:10.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.210+0000 7fb5e1093b80 20 client.25556 trim_cache size 0 max 16384 2024-04-26T01:49:10.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.232+0000 7fb5e1093b80 1 Processor -- start 2024-04-26T01:49:10.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.232+0000 7fb5e1093b80 1 -- start start 2024-04-26T01:49:10.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.232+0000 7fb5e1093b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 0x55981e1b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.232+0000 7fb5e1093b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1ad410 0x55981e1af370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.232+0000 7fb5e1093b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1b82c0 0x55981e1cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.232+0000 7fb5e1093b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d01b1c10 con 0x55981e1ad410 2024-04-26T01:49:10.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.232+0000 7fb5e1093b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d01af440 con 0x55981e1acaf0 2024-04-26T01:49:10.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.232+0000 7fb5e1093b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d01ad290 con 0x55981e1b82c0 2024-04-26T01:49:10.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.233+0000 7fb5cf400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 0x55981e1b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.233+0000 7fb5cf400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 0x55981e1b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53048/0 (socket says 172.21.15.6:53048) 2024-04-26T01:49:10.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.233+0000 7fb5cf400700 1 -- 172.21.15.6:0/3279304015 learned_addr learned my addr 172.21.15.6:0/3279304015 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:10.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.233+0000 7fb5c6a00700 1 --2- 172.21.15.6:0/3279304015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1ad410 0x55981e1af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.233+0000 7fb5cfe00700 1 --2- 172.21.15.6:0/3279304015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1b82c0 0x55981e1cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.233+0000 7fb5cf400700 1 -- 172.21.15.6:0/3279304015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1b82c0 msgr2=0x55981e1cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.233+0000 7fb5cf400700 1 --2- 172.21.15.6:0/3279304015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1b82c0 0x55981e1cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.233+0000 7fb5cf400700 1 -- 172.21.15.6:0/3279304015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1ad410 msgr2=0x55981e1af370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.233+0000 7fb5cf400700 1 --2- 172.21.15.6:0/3279304015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1ad410 0x55981e1af370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.233+0000 7fb5cf400700 1 -- 172.21.15.6:0/3279304015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55981e1adb10 con 0x55981e1acaf0 2024-04-26T01:49:10.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.233+0000 7fb5cf400700 1 --2- 172.21.15.6:0/3279304015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 0x55981e1b3c90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0011620 tx=0x7fb5c00093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8069b56db667c96a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:10.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.234+0000 7fb5bfe00700 1 -- 172.21.15.6:0/3279304015 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0019040 con 0x55981e1acaf0 2024-04-26T01:49:10.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.234+0000 7fb5bfe00700 1 -- 172.21.15.6:0/3279304015 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5c0029420 con 0x55981e1acaf0 2024-04-26T01:49:10.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.234+0000 7fb5bfe00700 1 -- 172.21.15.6:0/3279304015 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c00158a0 con 0x55981e1acaf0 2024-04-26T01:49:10.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.234+0000 7fb5e1093b80 1 -- 172.21.15.6:0/3279304015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 msgr2=0x55981e1b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.234+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/3279304015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 0x55981e1b3c90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0011620 tx=0x7fb5c00093e0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.234+0000 7fb5e1093b80 1 -- 172.21.15.6:0/3279304015 shutdown_connections 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.234+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/3279304015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1b82c0 0x55981e1cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.234+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/3279304015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1acaf0 0x55981e1b3c90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.234+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/3279304015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1ad410 0x55981e1af370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.234+0000 7fb5e1093b80 1 -- 172.21.15.6:0/3279304015 >> 172.21.15.6:0/3279304015 conn(0x55981e0db6a0 msgr2=0x55981e1a2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.234+0000 7fb5e1093b80 1 -- 172.21.15.6:0/3279304015 shutdown_connections 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.235+0000 7fb5e1093b80 1 -- 172.21.15.6:0/3279304015 wait complete. 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.235+0000 7fb5e1093b80 1 -- start start 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.235+0000 7fb5e1093b80 1 Processor -- start 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.236+0000 7fb5e1093b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 0x55981e230bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.236+0000 7fb5e1093b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1ad410 0x55981e231100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.236+0000 7fb5e1093b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1b82c0 0x55981e1bd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.236+0000 7fb5e1093b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d01ad900 con 0x55981e1acaf0 2024-04-26T01:49:10.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.236+0000 7fb5e1093b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d01aed70 con 0x55981e1ad410 2024-04-26T01:49:10.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.236+0000 7fb5e1093b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d01adfd0 con 0x55981e1b82c0 2024-04-26T01:49:10.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.236+0000 7fb5e1093b80 10 client.0 _ll_register_callbacks cb 0x55981e098820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:10.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.236+0000 7fb5cf400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 0x55981e230bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.236+0000 7fb5cf400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 0x55981e230bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40298/0 (socket says 172.21.15.6:40298) 2024-04-26T01:49:10.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.236+0000 7fb5cf400700 1 -- 172.21.15.6:0/1160293630 learned_addr learned my addr 172.21.15.6:0/1160293630 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:10.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.236+0000 7fb5c6a00700 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1ad410 0x55981e231100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.236+0000 7fb5cfe00700 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1b82c0 0x55981e1bd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.237+0000 7fb5cf400700 1 -- 172.21.15.6:0/1160293630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1b82c0 msgr2=0x55981e1bd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.237+0000 7fb5cf400700 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1b82c0 0x55981e1bd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.237+0000 7fb5cf400700 1 -- 172.21.15.6:0/1160293630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1ad410 msgr2=0x55981e231100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.237+0000 7fb5cf400700 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1ad410 0x55981e231100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.237+0000 7fb5cf400700 1 -- 172.21.15.6:0/1160293630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55981e1bdc50 con 0x55981e1acaf0 2024-04-26T01:49:10.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.237+0000 7fb5cf400700 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 0x55981e230bc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009d10 tx=0x7fb5c0015450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbfa69a6bab0ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:10.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.237+0000 7fb5bd600700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:49:10.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.237+0000 7fb5bd600700 1 -- 172.21.15.6:0/1160293630 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0019030 con 0x55981e1acaf0 2024-04-26T01:49:10.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.237+0000 7fb5bd600700 1 -- 172.21.15.6:0/1160293630 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fb5c0003a00 con 0x55981e1acaf0 2024-04-26T01:49:10.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.237+0000 7fb5e1093b80 1 -- 172.21.15.6:0/1160293630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5c0029420 con 0x55981e1acaf0 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.238+0000 7fb5bd600700 1 -- 172.21.15.6:0/1160293630 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb5c0003c40 con 0x55981e1acaf0 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.238+0000 7fb5e1093b80 10 client.15444 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.238+0000 7fb5e1093b80 1 -- 172.21.15.6:0/1160293630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55981e2383f0 con 0x55981e1acaf0 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.238+0000 7fb5bd600700 1 -- 172.21.15.6:0/1160293630 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5c0029420 con 0x55981e1acaf0 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.238+0000 7fb5bd600700 1 -- 172.21.15.6:0/1160293630 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fb5c0007500 con 0x55981e1acaf0 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.238+0000 7fb5e1093b80 10 client.15444 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.238+0000 7fb5e1093b80 10 client.15444 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.238+0000 7fb5e1093b80 20 client.15444 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.238+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55981e2388d0 0x55981e23acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.238+0000 7fb5e1093b80 4 client.15444 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c54a4666", "release": true, "await": true}] 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.238+0000 7fb5e1093b80 1 -- 172.21.15.6:0/1160293630 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c54a4666", "release": true, "await": true}) -- 0x55981e107820 con 0x55981e2388d0 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.239+0000 7fb5c6a00700 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55981e2388d0 0x55981e23acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.239+0000 7fb5c6a00700 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55981e2388d0 0x55981e23acf0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76593a565ea6e225 server_cookie=40d80bf1cc91516b in_seq=0 out_seq=0 2024-04-26T01:49:10.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.239+0000 7fb5bd600700 10 client.15444 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:10.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5bd600700 1 -- 172.21.15.6:0/1160293630 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55981e107820 con 0x55981e2388d0 2024-04-26T01:49:10.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5bd600700 10 client.15444 handle_command_reply: tid=0 2024-04-26T01:49:10.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5e1093b80 1 client.15444 shutdown 2024-04-26T01:49:10.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5e1093b80 1 -- 172.21.15.6:0/1160293630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 msgr2=0x55981e230bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 0x55981e230bc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009d10 tx=0x7fb5c0015450 comp rx=0 tx=0).stop 2024-04-26T01:49:10.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5e1093b80 1 -- 172.21.15.6:0/1160293630 shutdown_connections 2024-04-26T01:49:10.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55981e2388d0 0x55981e23acf0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55981e1b82c0 0x55981e1bd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55981e1acaf0 0x55981e230bc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5e1093b80 1 --2- 172.21.15.6:0/1160293630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55981e1ad410 0x55981e231100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5e1093b80 1 -- 172.21.15.6:0/1160293630 >> 172.21.15.6:0/1160293630 conn(0x55981e0db6a0 msgr2=0x55981e1af370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:10.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5bd600700 0 client.15444 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:10.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.246+0000 7fb5e1093b80 1 -- 172.21.15.6:0/1160293630 shutdown_connections 2024-04-26T01:49:10.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.247+0000 7fb5e1093b80 1 -- 172.21.15.6:0/1160293630 wait complete. 2024-04-26T01:49:10.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.247+0000 7fb5e1093b80 20 client.15444 trim_cache size 0 max 16384 2024-04-26T01:49:10.246 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 31, 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 8, 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: "c54a4666": { 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 8, 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:49:10.247 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 90.0, 2024-04-26T01:49:10.248 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:49:10.248 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:49:10.248 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:49:10.248 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:49:10.248 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:49:10.248 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:49:10.248 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:10.248 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:10.248 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:10.248 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:10.248 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:10.248 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:49:10.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.248+0000 7fb5d5200700 1 -- 172.21.15.6:0/1982043426 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb5b40a3db0 msgr2=0x7fb5b40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.248+0000 7fb5d5200700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb5b40a3db0 0x7fb5b40a6270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0103010 tx=0x7fb5b8009260 comp rx=0 tx=0).stop 2024-04-26T01:49:10.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.248+0000 7fb5d5200700 1 -- 172.21.15.6:0/1982043426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d010f270 msgr2=0x7fb5d019fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.248+0000 7fb5d5200700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d010f270 0x7fb5d019fb10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80047b0 tx=0x7fb5c8009530 comp rx=0 tx=0).stop 2024-04-26T01:49:10.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.248+0000 7fb5d5200700 1 -- 172.21.15.6:0/1982043426 shutdown_connections 2024-04-26T01:49:10.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.248+0000 7fb5d5200700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb5b40a3db0 0x7fb5b40a6270 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.248+0000 7fb5d5200700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d010f270 0x7fb5d019fb10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.248+0000 7fb5d5200700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5d01016c0 0x7fb5d019b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.248+0000 7fb5d5200700 1 --2- 172.21.15.6:0/1982043426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5d0102070 0x7fb5d019b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.248+0000 7fb5d5200700 1 -- 172.21.15.6:0/1982043426 >> 172.21.15.6:0/1982043426 conn(0x7fb5d00fd360 msgr2=0x7fb5d00ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:10.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.248+0000 7fb5d5200700 1 -- 172.21.15.6:0/1982043426 shutdown_connections 2024-04-26T01:49:10.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:10.248+0000 7fb5d5200700 1 -- 172.21.15.6:0/1982043426 wait complete. 2024-04-26T01:49:10.562 INFO:tasks.workunit.client.0.smithi006.stdout: 1 83672 4.96 MB/sec warmup 94 sec latency 37634.749 ms 2024-04-26T01:49:10.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:10 smithi006 ceph-mon[28419]: pgmap v102: 97 pgs: 97 active+clean; 33 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 784 KiB/s wr, 58 op/s 2024-04-26T01:49:10.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:10 smithi029 ceph-mon[32775]: pgmap v102: 97 pgs: 97 active+clean; 33 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 784 KiB/s wr, 58 op/s 2024-04-26T01:49:10.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:10 smithi188 ceph-mon[32751]: pgmap v102: 97 pgs: 97 active+clean; 33 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 784 KiB/s wr, 58 op/s 2024-04-26T01:49:11.335 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c54a4666', seconds elapsed: 46.4 2024-04-26T01:49:11.335 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-26T01:49:11.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:49:11.562 INFO:tasks.workunit.client.0.smithi006.stdout: 1 84973 5.00 MB/sec warmup 95 sec latency 39.637 ms 2024-04-26T01:49:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:49:12.562 INFO:tasks.workunit.client.0.smithi006.stdout: 1 85769 5.00 MB/sec warmup 96 sec latency 15.790 ms 2024-04-26T01:49:12.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:12 smithi006 ceph-mon[28419]: pgmap v103: 97 pgs: 97 active+clean; 32 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 853 KiB/s wr, 74 op/s 2024-04-26T01:49:12.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:12 smithi029 ceph-mon[32775]: pgmap v103: 97 pgs: 97 active+clean; 32 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 853 KiB/s wr, 74 op/s 2024-04-26T01:49:12.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:12 smithi188 ceph-mon[32751]: pgmap v103: 97 pgs: 97 active+clean; 32 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 853 KiB/s wr, 74 op/s 2024-04-26T01:49:13.562 INFO:tasks.workunit.client.0.smithi006.stdout: 1 87512 5.02 MB/sec warmup 97 sec latency 31.953 ms 2024-04-26T01:49:14.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.164+0000 7fdde0000700 1 -- 172.21.15.6:0/1878382905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd8075590 msgr2=0x7fddd810ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:14.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.164+0000 7fdde0000700 1 --2- 172.21.15.6:0/1878382905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd8075590 0x7fddd810ef60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009990 tx=0x7fddc4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:49:14.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.164+0000 7fdde0000700 1 -- 172.21.15.6:0/1878382905 shutdown_connections 2024-04-26T01:49:14.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.164+0000 7fdde0000700 1 --2- 172.21.15.6:0/1878382905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddd810f4a0 0x7fddd8111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:14.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.164+0000 7fdde0000700 1 --2- 172.21.15.6:0/1878382905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddd8074c70 0x7fddd8075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:14.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.164+0000 7fdde0000700 1 --2- 172.21.15.6:0/1878382905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd8075590 0x7fddd810ef60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:14.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.164+0000 7fdde0000700 1 -- 172.21.15.6:0/1878382905 >> 172.21.15.6:0/1878382905 conn(0x7fddd8079ba0 msgr2=0x7fddd8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:14.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.164+0000 7fdde0000700 1 -- 172.21.15.6:0/1878382905 shutdown_connections 2024-04-26T01:49:14.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.164+0000 7fdde0000700 1 -- 172.21.15.6:0/1878382905 wait complete. 2024-04-26T01:49:14.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.165+0000 7fdde0000700 1 Processor -- start 2024-04-26T01:49:14.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.166+0000 7fdde0000700 1 -- start start 2024-04-26T01:49:14.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.166+0000 7fdde0000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddd8074c70 0x7fddd819b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:14.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.166+0000 7fdde0000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd8075590 0x7fddd819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:14.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.166+0000 7fdde0000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddd810f4a0 0x7fddd819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:14.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.166+0000 7fdde0000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fddd8114ed0 con 0x7fddd8074c70 2024-04-26T01:49:14.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.166+0000 7fdde0000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fddd8114d50 con 0x7fddd8075590 2024-04-26T01:49:14.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.166+0000 7fdde0000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fddd8115050 con 0x7fddd810f4a0 2024-04-26T01:49:14.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.166+0000 7fdddd600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddd8074c70 0x7fddd819b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:14.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.167+0000 7fddde000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddd810f4a0 0x7fddd819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:14.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.167+0000 7fdddcc00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd8075590 0x7fddd819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:14.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.167+0000 7fddde000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddd810f4a0 0x7fddd819fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51548/0 (socket says 172.21.15.6:51548) 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.167+0000 7fddde000700 1 -- 172.21.15.6:0/1938086123 learned_addr learned my addr 172.21.15.6:0/1938086123 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.167+0000 7fddde000700 1 -- 172.21.15.6:0/1938086123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd8075590 msgr2=0x7fddd819b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.167+0000 7fddde000700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd8075590 0x7fddd819b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.167+0000 7fddde000700 1 -- 172.21.15.6:0/1938086123 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddd8074c70 msgr2=0x7fddd819b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.167+0000 7fddde000700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddd8074c70 0x7fddd819b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.167+0000 7fddde000700 1 -- 172.21.15.6:0/1938086123 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd81a0320 con 0x7fddd810f4a0 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.167+0000 7fdddd600700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddd8074c70 0x7fddd819b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.167+0000 7fdddcc00700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd8075590 0x7fddd819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.167+0000 7fddde000700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddd810f4a0 0x7fddd819fbe0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fddd000efc0 tx=0x7fddd000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.168+0000 7fddd6000700 1 -- 172.21.15.6:0/1938086123 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd00092e0 con 0x7fddd810f4a0 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.168+0000 7fddd6000700 1 -- 172.21.15.6:0/1938086123 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddd0004bb0 con 0x7fddd810f4a0 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.168+0000 7fddd6000700 1 -- 172.21.15.6:0/1938086123 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd0004280 con 0x7fddd810f4a0 2024-04-26T01:49:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.168+0000 7fdde0000700 1 -- 172.21.15.6:0/1938086123 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd81a05b0 con 0x7fddd810f4a0 2024-04-26T01:49:14.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.168+0000 7fdde0000700 1 -- 172.21.15.6:0/1938086123 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd81a7e90 con 0x7fddd810f4a0 2024-04-26T01:49:14.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.169+0000 7fdde0000700 1 -- 172.21.15.6:0/1938086123 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd8069730 con 0x7fddd810f4a0 2024-04-26T01:49:14.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.171+0000 7fddd6000700 1 -- 172.21.15.6:0/1938086123 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fddd001b030 con 0x7fddd810f4a0 2024-04-26T01:49:14.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.173+0000 7fddd6000700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fddc00a3e30 0x7fddc00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:14.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.173+0000 7fddd6000700 1 -- 172.21.15.6:0/1938086123 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fddd00d02f0 con 0x7fddd810f4a0 2024-04-26T01:49:14.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.173+0000 7fdddd600700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fddc00a3e30 0x7fddc00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:14.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.174+0000 7fdddd600700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fddc00a3e30 0x7fddc00a62f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009be0 tx=0x7fddcc009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:14.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.177+0000 7fddd6000700 1 -- 172.21.15.6:0/1938086123 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fddd0022020 con 0x7fddd810f4a0 2024-04-26T01:49:14.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.510+0000 7fdde0000700 1 -- 172.21.15.6:0/1938086123 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddd81a8060 con 0x7fddd810f4a0 2024-04-26T01:49:14.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.511+0000 7fddd6000700 1 -- 172.21.15.6:0/1938086123 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11853 (secure 0 0 0) 0x7fddd0010040 con 0x7fddd810f4a0 2024-04-26T01:49:14.510 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:49:14.510 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:47:59.436110+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25474":{"gid":25474,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1633748308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1633748308},{"type":"v1","addr":"172.21.15.188:6835","nonce":1633748308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:49:14.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.514+0000 7fdde0000700 1 -- 172.21.15.6:0/1938086123 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fddc00a3e30 msgr2=0x7fddc00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:14.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.514+0000 7fdde0000700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fddc00a3e30 0x7fddc00a62f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009be0 tx=0x7fddcc009260 comp rx=0 tx=0).stop 2024-04-26T01:49:14.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.514+0000 7fdde0000700 1 -- 172.21.15.6:0/1938086123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddd810f4a0 msgr2=0x7fddd819fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:14.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.514+0000 7fdde0000700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddd810f4a0 0x7fddd819fbe0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fddd000efc0 tx=0x7fddd000c3a0 comp rx=0 tx=0).stop 2024-04-26T01:49:14.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.514+0000 7fdde0000700 1 -- 172.21.15.6:0/1938086123 shutdown_connections 2024-04-26T01:49:14.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.514+0000 7fdde0000700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fddc00a3e30 0x7fddc00a62f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:14.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.514+0000 7fdde0000700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddd810f4a0 0x7fddd819fbe0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:14.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.514+0000 7fdde0000700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fddd8074c70 0x7fddd819b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:14.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.514+0000 7fdde0000700 1 --2- 172.21.15.6:0/1938086123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd8075590 0x7fddd819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:14.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.514+0000 7fdde0000700 1 -- 172.21.15.6:0/1938086123 >> 172.21.15.6:0/1938086123 conn(0x7fddd8079ba0 msgr2=0x7fddd80ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:14.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.514+0000 7fdde0000700 1 -- 172.21.15.6:0/1938086123 shutdown_connections 2024-04-26T01:49:14.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:14.515+0000 7fdde0000700 1 -- 172.21.15.6:0/1938086123 wait complete. 2024-04-26T01:49:14.514 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 31 2024-04-26T01:49:14.562 INFO:tasks.workunit.client.0.smithi006.stdout: 1 88921 5.09 MB/sec warmup 98 sec latency 17.269 ms 2024-04-26T01:49:14.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:14 smithi029 ceph-mon[32775]: pgmap v104: 97 pgs: 97 active+clean; 32 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 341 KiB/s wr, 39 op/s 2024-04-26T01:49:14.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:14 smithi006 ceph-mon[28419]: pgmap v104: 97 pgs: 97 active+clean; 32 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 341 KiB/s wr, 39 op/s 2024-04-26T01:49:14.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:14 smithi188 ceph-mon[32751]: pgmap v104: 97 pgs: 97 active+clean; 32 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 341 KiB/s wr, 39 op/s 2024-04-26T01:49:15.251 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-26T01:49:15.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-26T01:49:15.562 INFO:tasks.workunit.client.0.smithi006.stdout: 1 89633 5.05 MB/sec warmup 99 sec latency 28.384 ms 2024-04-26T01:49:15.735 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:49:15.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:15 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1938086123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:49:15.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:15 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1938086123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:49:15.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:15 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1938086123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:49:16.562 INFO:tasks.workunit.client.0.smithi006.stdout: 1 91302 5.07 MB/sec warmup 100 sec latency 14.135 ms 2024-04-26T01:49:16.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:16 smithi006 ceph-mon[28419]: pgmap v105: 97 pgs: 97 active+clean; 32 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 342 KiB/s wr, 39 op/s 2024-04-26T01:49:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.821+0000 7fd4aa000640 1 -- 172.21.15.6:0/2524069942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a41070a0 msgr2=0x7fd4a410ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.821+0000 7fd4aa000640 1 --2- 172.21.15.6:0/2524069942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a41070a0 0x7fd4a410ec70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009ca0 tx=0x7fd49c02a8b0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.822+0000 7fd4aa000640 1 -- 172.21.15.6:0/2524069942 shutdown_connections 2024-04-26T01:49:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.822+0000 7fd4aa000640 1 --2- 172.21.15.6:0/2524069942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a41070a0 0x7fd4a410ec70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.822+0000 7fd4aa000640 1 --2- 172.21.15.6:0/2524069942 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4a4102980 0x7fd4a41069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.822+0000 7fd4aa000640 1 --2- 172.21.15.6:0/2524069942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4a4101fd0 0x7fd4a41023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.822+0000 7fd4aa000640 1 -- 172.21.15.6:0/2524069942 >> 172.21.15.6:0/2524069942 conn(0x7fd4a40fd890 msgr2=0x7fd4a40ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.822+0000 7fd4aa000640 1 -- 172.21.15.6:0/2524069942 shutdown_connections 2024-04-26T01:49:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.822+0000 7fd4aa000640 1 -- 172.21.15.6:0/2524069942 wait complete. 2024-04-26T01:49:16.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.823+0000 7fd4aa000640 1 Processor -- start 2024-04-26T01:49:16.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.823+0000 7fd4aa000640 1 -- start start 2024-04-26T01:49:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.824+0000 7fd4aa000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4a4101fd0 0x7fd4a4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.824+0000 7fd4aa000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a41070a0 0x7fd4a4198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.824+0000 7fd4aa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4a419cde0 0x7fd4a419d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.824+0000 7fd4aa000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a4112040 con 0x7fd4a419cde0 2024-04-26T01:49:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.824+0000 7fd4aa000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a4111ec0 con 0x7fd4a41070a0 2024-04-26T01:49:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.824+0000 7fd4aa000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41121c0 con 0x7fd4a4101fd0 2024-04-26T01:49:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.824+0000 7fd4a3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a41070a0 0x7fd4a4198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.824+0000 7fd4a3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a41070a0 0x7fd4a4198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53094/0 (socket says 172.21.15.6:53094) 2024-04-26T01:49:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.824+0000 7fd4a3e00640 1 -- 172.21.15.6:0/931211196 learned_addr learned my addr 172.21.15.6:0/931211196 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.824+0000 7fd4a9600640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4a419cde0 0x7fd4a419d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.824+0000 7fd4a8c00640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4a4101fd0 0x7fd4a4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.825+0000 7fd4a3e00640 1 -- 172.21.15.6:0/931211196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4a4101fd0 msgr2=0x7fd4a4198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.825+0000 7fd4a3e00640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4a4101fd0 0x7fd4a4198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.825+0000 7fd4a3e00640 1 -- 172.21.15.6:0/931211196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4a419cde0 msgr2=0x7fd4a419d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.825+0000 7fd4a3e00640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4a419cde0 0x7fd4a419d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.825+0000 7fd4a3e00640 1 -- 172.21.15.6:0/931211196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a419d8d0 con 0x7fd4a41070a0 2024-04-26T01:49:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.825+0000 7fd4a3e00640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a41070a0 0x7fd4a4198a60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd49000e9c0 tx=0x7fd49000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.825+0000 7fd4a1600640 1 -- 172.21.15.6:0/931211196 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49000cd80 con 0x7fd4a41070a0 2024-04-26T01:49:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.825+0000 7fd4a1600640 1 -- 172.21.15.6:0/931211196 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4900054d0 con 0x7fd4a41070a0 2024-04-26T01:49:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.825+0000 7fd4aa000640 1 -- 172.21.15.6:0/931211196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a419dbc0 con 0x7fd4a41070a0 2024-04-26T01:49:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.825+0000 7fd4a1600640 1 -- 172.21.15.6:0/931211196 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490010430 con 0x7fd4a41070a0 2024-04-26T01:49:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.826+0000 7fd4aa000640 1 -- 172.21.15.6:0/931211196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4a41a5400 con 0x7fd4a41070a0 2024-04-26T01:49:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.829+0000 7fd4a1600640 1 -- 172.21.15.6:0/931211196 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fd490004fb0 con 0x7fd4a41070a0 2024-04-26T01:49:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.829+0000 7fd4ae41e740 1 Processor -- start 2024-04-26T01:49:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.829+0000 7fd4ae41e740 1 -- start start 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4ae41e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 0x560e33f6fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4ae41e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f701f0 0x560e33f7ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4ae41e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f7efa0 0x560e33f81390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4ae41e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560e33ed5900 con 0x560e33f7efa0 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4ae41e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560e33e7bef0 con 0x560e33f701f0 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4ae41e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560e33ed18c0 con 0x560e33f6f8d0 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4a9600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 0x560e33f6fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4a8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f701f0 0x560e33f7ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4a9600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 0x560e33f6fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51566/0 (socket says 172.21.15.6:51566) 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4a9600640 1 -- 172.21.15.6:0/38000019 learned_addr learned my addr 172.21.15.6:0/38000019 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4a3e00640 1 --2- 172.21.15.6:0/38000019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f7efa0 0x560e33f81390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4a9600640 1 -- 172.21.15.6:0/38000019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f701f0 msgr2=0x560e33f7ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.830+0000 7fd4a9600640 1 --2- 172.21.15.6:0/38000019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f701f0 0x560e33f7ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.831+0000 7fd4a9600640 1 -- 172.21.15.6:0/38000019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f7efa0 msgr2=0x560e33f81390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.831+0000 7fd4a9600640 1 --2- 172.21.15.6:0/38000019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f7efa0 0x560e33f81390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.831+0000 7fd4a9600640 1 -- 172.21.15.6:0/38000019 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e33be8210 con 0x560e33f6f8d0 2024-04-26T01:49:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.831+0000 7fd4a9600640 1 --2- 172.21.15.6:0/38000019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 0x560e33f6fcb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009ca0 tx=0x7fd49c030fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e9a6c8cde2fe8cd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.831+0000 7fd495c00640 1 -- 172.21.15.6:0/38000019 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49c030800 con 0x560e33f6f8d0 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.831+0000 7fd495c00640 1 -- 172.21.15.6:0/38000019 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd49c036030 con 0x560e33f6f8d0 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.831+0000 7fd495c00640 1 -- 172.21.15.6:0/38000019 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49c004440 con 0x560e33f6f8d0 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.831+0000 7fd4a1600640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd48c0a3d80 0x7fd48c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.832+0000 7fd4a1600640 1 -- 172.21.15.6:0/931211196 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fd4900d0b90 con 0x7fd4a41070a0 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.832+0000 7fd4a8c00640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd48c0a3d80 0x7fd48c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.832+0000 7fd4ae41e740 1 -- 172.21.15.6:0/38000019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 msgr2=0x560e33f6fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.832+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/38000019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 0x560e33f6fcb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009ca0 tx=0x7fd49c030fb0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.832+0000 7fd4ae41e740 1 -- 172.21.15.6:0/38000019 shutdown_connections 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.832+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/38000019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f7efa0 0x560e33f81390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.832+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/38000019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f701f0 0x560e33f7ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.832+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/38000019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 0x560e33f6fcb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.832+0000 7fd4ae41e740 1 -- 172.21.15.6:0/38000019 >> 172.21.15.6:0/38000019 conn(0x560e33f6bec0 msgr2=0x560e33f6c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.832+0000 7fd4ae41e740 1 -- 172.21.15.6:0/38000019 shutdown_connections 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.832+0000 7fd4ae41e740 1 -- 172.21.15.6:0/38000019 wait complete. 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.832+0000 7fd4a8c00640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd48c0a3d80 0x7fd48c0a6240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x560e33f724d0 tx=0x7fd49800b390 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.833+0000 7fd4ae41e740 1 -- start start 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.833+0000 7fd4ae41e740 1 Processor -- start 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.833+0000 7fd4ae41e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 0x560e33f7bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.833+0000 7fd4ae41e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f701f0 0x560e33f7c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.833+0000 7fd4ae41e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f7efa0 0x560e33f7c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.833+0000 7fd4ae41e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560e33f81b70 con 0x560e33f701f0 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4ae41e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560e33e8f310 con 0x560e33f7efa0 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4ae41e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560e33f81cf0 con 0x560e33f6f8d0 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4ae41e740 10 client.0 _ll_register_callbacks cb 0x560e33e5dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4a8c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 0x560e33f7bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4a8c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 0x560e33f7bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51578/0 (socket says 172.21.15.6:51578) 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4a8c00640 1 -- 172.21.15.6:0/46037493 learned_addr learned my addr 172.21.15.6:0/46037493 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4a9600640 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f7efa0 0x560e33f7c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4a3e00640 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f701f0 0x560e33f7c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4a8c00640 1 -- 172.21.15.6:0/46037493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f7efa0 msgr2=0x560e33f7c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4a8c00640 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f7efa0 0x560e33f7c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4a8c00640 1 -- 172.21.15.6:0/46037493 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f701f0 msgr2=0x560e33f7c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4a8c00640 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f701f0 0x560e33f7c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4a8c00640 1 -- 172.21.15.6:0/46037493 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e33cb4620 con 0x560e33f6f8d0 2024-04-26T01:49:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.834+0000 7fd4a3e00640 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f701f0 0x560e33f7c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.835+0000 7fd4a8c00640 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 0x560e33f7bf20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd498003d30 tx=0x7fd4980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65208971e1be7f48 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.835+0000 7fd48a000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.835+0000 7fd48a000640 1 -- 172.21.15.6:0/46037493 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49802c060 con 0x560e33f6f8d0 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.835+0000 7fd4ae41e740 1 -- 172.21.15.6:0/46037493 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd49c036030 con 0x560e33f6f8d0 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.835+0000 7fd48a000640 1 -- 172.21.15.6:0/46037493 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fd498017030 con 0x560e33f6f8d0 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.835+0000 7fd48a000640 1 -- 172.21.15.6:0/46037493 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd498030430 con 0x560e33f6f8d0 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.835+0000 7fd4ae41e740 10 client.25630 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.835+0000 7fd4ae41e740 1 -- 172.21.15.6:0/46037493 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560e33f8de50 con 0x560e33f6f8d0 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.836+0000 7fd48a000640 1 -- 172.21.15.6:0/46037493 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd49c036030 con 0x560e33f6f8d0 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.836+0000 7fd48a000640 1 -- 172.21.15.6:0/46037493 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fd498011460 con 0x560e33f6f8d0 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.836+0000 7fd4ae41e740 10 client.25630 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.836+0000 7fd4ae41e740 10 client.25630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.836+0000 7fd4ae41e740 20 client.25630 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.836+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560e33f8f310 0x560e33f917a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.836+0000 7fd4ae41e740 4 client.25630 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.836+0000 7fd4ae41e740 1 -- 172.21.15.6:0/46037493 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e33ee98b0 con 0x560e33f8f310 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.836+0000 7fd4a3e00640 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560e33f8f310 0x560e33f917a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.837+0000 7fd4a3e00640 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560e33f8f310 0x560e33f917a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91a5ef420369985 server_cookie=80b997ccae55d632 in_seq=0 out_seq=0 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.837+0000 7fd48a000640 10 client.25630 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd48a000640 1 -- 172.21.15.6:0/46037493 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e33ee98b0 con 0x560e33f8f310 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd48a000640 10 client.25630 handle_command_reply: tid=0 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd4ae41e740 1 client.25630 shutdown 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd4ae41e740 1 -- 172.21.15.6:0/46037493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 msgr2=0x560e33f7bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 0x560e33f7bf20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd498003d30 tx=0x7fd4980046c0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd4ae41e740 1 -- 172.21.15.6:0/46037493 shutdown_connections 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560e33f8f310 0x560e33f917a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f701f0 0x560e33f7c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f7efa0 0x560e33f7c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/46037493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f6f8d0 0x560e33f7bf20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd4ae41e740 1 -- 172.21.15.6:0/46037493 >> 172.21.15.6:0/46037493 conn(0x560e33f6bec0 msgr2=0x560e33f71d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd48a000640 0 client.25630 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd4ae41e740 1 -- 172.21.15.6:0/46037493 shutdown_connections 2024-04-26T01:49:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd4ae41e740 1 -- 172.21.15.6:0/46037493 wait complete. 2024-04-26T01:49:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.838+0000 7fd4ae41e740 20 client.25630 trim_cache size 0 max 16384 2024-04-26T01:49:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4ae41e740 1 Processor -- start 2024-04-26T01:49:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4ae41e740 1 -- start start 2024-04-26T01:49:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4ae41e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f6f8d0 0x560e33f8aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4ae41e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f701f0 0x560e33f6aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4ae41e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f7efa0 0x560e33f970b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4ae41e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41ae1f0 con 0x560e33f701f0 2024-04-26T01:49:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4ae41e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41aba60 con 0x560e33f6f8d0 2024-04-26T01:49:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4ae41e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41a98b0 con 0x560e33f7efa0 2024-04-26T01:49:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4a8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f6f8d0 0x560e33f8aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4a8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f6f8d0 0x560e33f8aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53102/0 (socket says 172.21.15.6:53102) 2024-04-26T01:49:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4a8c00640 1 -- 172.21.15.6:0/376375220 learned_addr learned my addr 172.21.15.6:0/376375220 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4a9600640 1 --2- 172.21.15.6:0/376375220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f7efa0 0x560e33f970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.856+0000 7fd4a3e00640 1 --2- 172.21.15.6:0/376375220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f701f0 0x560e33f6aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.857+0000 7fd4a8c00640 1 -- 172.21.15.6:0/376375220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f7efa0 msgr2=0x560e33f970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.857+0000 7fd4a8c00640 1 --2- 172.21.15.6:0/376375220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f7efa0 0x560e33f970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.857+0000 7fd4a8c00640 1 -- 172.21.15.6:0/376375220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f701f0 msgr2=0x560e33f6aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.857+0000 7fd4a8c00640 1 --2- 172.21.15.6:0/376375220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f701f0 0x560e33f6aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.857+0000 7fd4a8c00640 1 -- 172.21.15.6:0/376375220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e33f81ab0 con 0x560e33f6f8d0 2024-04-26T01:49:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.857+0000 7fd4a8c00640 1 --2- 172.21.15.6:0/376375220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f6f8d0 0x560e33f8aa30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd4980059f0 tx=0x7fd498004350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f554b309c21dacad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.857+0000 7fd495200640 1 -- 172.21.15.6:0/376375220 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49801fc90 con 0x560e33f6f8d0 2024-04-26T01:49:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.858+0000 7fd495200640 1 -- 172.21.15.6:0/376375220 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd498011640 con 0x560e33f6f8d0 2024-04-26T01:49:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.858+0000 7fd495200640 1 -- 172.21.15.6:0/376375220 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498004ab0 con 0x560e33f6f8d0 2024-04-26T01:49:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.858+0000 7fd4ae41e740 1 -- 172.21.15.6:0/376375220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f6f8d0 msgr2=0x560e33f8aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.858+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/376375220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f6f8d0 0x560e33f8aa30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd4980059f0 tx=0x7fd498004350 comp rx=0 tx=0).stop 2024-04-26T01:49:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.859+0000 7fd4ae41e740 1 -- 172.21.15.6:0/376375220 shutdown_connections 2024-04-26T01:49:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.859+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/376375220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f701f0 0x560e33f6aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.859+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/376375220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f7efa0 0x560e33f970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.859+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/376375220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f6f8d0 0x560e33f8aa30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.859+0000 7fd4ae41e740 1 -- 172.21.15.6:0/376375220 >> 172.21.15.6:0/376375220 conn(0x560e33f6bec0 msgr2=0x560e33f6a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.859+0000 7fd4ae41e740 1 -- 172.21.15.6:0/376375220 shutdown_connections 2024-04-26T01:49:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.859+0000 7fd4ae41e740 1 -- 172.21.15.6:0/376375220 wait complete. 2024-04-26T01:49:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.859+0000 7fd4ae41e740 1 -- start start 2024-04-26T01:49:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.859+0000 7fd4ae41e740 1 Processor -- start 2024-04-26T01:49:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4ae41e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f6f8d0 0x560e33f84ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4ae41e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f701f0 0x560e33f85010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4ae41e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f7efa0 0x560e33f85550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4ae41e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41a9f20 con 0x560e33f6f8d0 2024-04-26T01:49:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4ae41e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41ab390 con 0x560e33f701f0 2024-04-26T01:49:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4ae41e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41aa5f0 con 0x560e33f7efa0 2024-04-26T01:49:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4ae41e740 10 client.0 _ll_register_callbacks cb 0x7fd498003bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4a3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f701f0 0x560e33f85010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4a9600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f7efa0 0x560e33f85550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4a8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f6f8d0 0x560e33f84ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4a3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f701f0 0x560e33f85010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53114/0 (socket says 172.21.15.6:53114) 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4a3e00640 1 -- 172.21.15.6:0/3904712171 learned_addr learned my addr 172.21.15.6:0/3904712171 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.860+0000 7fd4a9600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f7efa0 0x560e33f85550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51594/0 (socket says 172.21.15.6:51594) 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd4a3e00640 1 -- 172.21.15.6:0/3904712171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f7efa0 msgr2=0x560e33f85550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd4a3e00640 1 --2- 172.21.15.6:0/3904712171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f7efa0 0x560e33f85550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd4a3e00640 1 -- 172.21.15.6:0/3904712171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f6f8d0 msgr2=0x560e33f84ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd4a3e00640 1 --2- 172.21.15.6:0/3904712171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f6f8d0 0x560e33f84ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd4a3e00640 1 -- 172.21.15.6:0/3904712171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e33ff4a30 con 0x560e33f701f0 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd4a3e00640 1 --2- 172.21.15.6:0/3904712171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f701f0 0x560e33f85010 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd4900ce2c0 tx=0x7fd49009c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c90a40e15c2d8464 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd48a000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd48a000640 1 -- 172.21.15.6:0/3904712171 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4900d5040 con 0x560e33f701f0 2024-04-26T01:49:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd48a000640 1 -- 172.21.15.6:0/3904712171 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fd49009ab50 con 0x560e33f701f0 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd4ae41e740 1 -- 172.21.15.6:0/3904712171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560e33ee3320 con 0x560e33f701f0 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd48a000640 1 -- 172.21.15.6:0/3904712171 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd49009cab0 con 0x560e33f701f0 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd4ae41e740 10 client.25586 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.861+0000 7fd4ae41e740 1 -- 172.21.15.6:0/3904712171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560e33ff5da0 con 0x560e33f701f0 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.862+0000 7fd48a000640 1 -- 172.21.15.6:0/3904712171 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560e33ee3320 con 0x560e33f701f0 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.862+0000 7fd48a000640 1 -- 172.21.15.6:0/3904712171 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fd4900ce4e0 con 0x560e33f701f0 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.862+0000 7fd4ae41e740 10 client.25586 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.862+0000 7fd4ae41e740 10 client.25586 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.862+0000 7fd4ae41e740 20 client.25586 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.862+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/3904712171 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560e33ff6530 0x560e33ff8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.862+0000 7fd4ae41e740 4 client.25586 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.862+0000 7fd4ae41e740 1 -- 172.21.15.6:0/3904712171 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560e33edd190 con 0x560e33ff6530 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.862+0000 7fd4a8c00640 1 --2- 172.21.15.6:0/3904712171 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560e33ff6530 0x560e33ff8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.862+0000 7fd4a8c00640 1 --2- 172.21.15.6:0/3904712171 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560e33ff6530 0x560e33ff8950 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24bbc192b25494d3 server_cookie=1d295fde1a04f162 in_seq=0 out_seq=0 2024-04-26T01:49:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.862+0000 7fd48a000640 10 client.25586 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:16.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:16 smithi188 ceph-mon[32751]: pgmap v105: 97 pgs: 97 active+clean; 32 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 342 KiB/s wr, 39 op/s 2024-04-26T01:49:16.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.893+0000 7fd48a000640 1 -- 172.21.15.6:0/3904712171 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560e33edd190 con 0x560e33ff6530 2024-04-26T01:49:16.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.893+0000 7fd48a000640 10 client.25586 handle_command_reply: tid=0 2024-04-26T01:49:16.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.893+0000 7fd4ae41e740 1 client.25586 shutdown 2024-04-26T01:49:16.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.893+0000 7fd4ae41e740 1 -- 172.21.15.6:0/3904712171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f701f0 msgr2=0x560e33f85010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:16.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.893+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/3904712171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f701f0 0x560e33f85010 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd4900ce2c0 tx=0x7fd49009c4d0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.893+0000 7fd4ae41e740 1 -- 172.21.15.6:0/3904712171 shutdown_connections 2024-04-26T01:49:16.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.893+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/3904712171 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560e33ff6530 0x560e33ff8950 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.893+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/3904712171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560e33f7efa0 0x560e33f85550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.893+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/3904712171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560e33f701f0 0x560e33f85010 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.893+0000 7fd48a000640 0 client.25586 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.893+0000 7fd4ae41e740 1 --2- 172.21.15.6:0/3904712171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e33f6f8d0 0x560e33f84ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.893+0000 7fd4ae41e740 1 -- 172.21.15.6:0/3904712171 >> 172.21.15.6:0/3904712171 conn(0x560e33f6bec0 msgr2=0x560e33f6b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.894+0000 7fd4ae41e740 1 -- 172.21.15.6:0/3904712171 shutdown_connections 2024-04-26T01:49:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.894+0000 7fd4ae41e740 1 -- 172.21.15.6:0/3904712171 wait complete. 2024-04-26T01:49:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.894+0000 7fd4ae41e740 20 client.25586 trim_cache size 0 max 16384 2024-04-26T01:49:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:49:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:49:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:49:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:49:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.895+0000 7fd4aa000640 1 -- 172.21.15.6:0/931211196 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd48c0a3d80 msgr2=0x7fd48c0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.895+0000 7fd4aa000640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd48c0a3d80 0x7fd48c0a6240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x560e33f724d0 tx=0x7fd49800b390 comp rx=0 tx=0).stop 2024-04-26T01:49:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.895+0000 7fd4aa000640 1 -- 172.21.15.6:0/931211196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a41070a0 msgr2=0x7fd4a4198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.895+0000 7fd4aa000640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a41070a0 0x7fd4a4198a60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd49000e9c0 tx=0x7fd49000ee80 comp rx=0 tx=0).stop 2024-04-26T01:49:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.895+0000 7fd4aa000640 1 -- 172.21.15.6:0/931211196 shutdown_connections 2024-04-26T01:49:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.895+0000 7fd4aa000640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd48c0a3d80 0x7fd48c0a6240 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.895+0000 7fd4aa000640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4a419cde0 0x7fd4a419d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.895+0000 7fd4aa000640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a41070a0 0x7fd4a4198a60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.895+0000 7fd4aa000640 1 --2- 172.21.15.6:0/931211196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4a4101fd0 0x7fd4a4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.895+0000 7fd4aa000640 1 -- 172.21.15.6:0/931211196 >> 172.21.15.6:0/931211196 conn(0x7fd4a40fd890 msgr2=0x7fd4a40ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.895+0000 7fd4aa000640 1 -- 172.21.15.6:0/931211196 shutdown_connections 2024-04-26T01:49:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:16.895+0000 7fd4aa000640 1 -- 172.21.15.6:0/931211196 wait complete. 2024-04-26T01:49:16.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:16 smithi029 ceph-mon[32775]: pgmap v105: 97 pgs: 97 active+clean; 32 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 342 KiB/s wr, 39 op/s 2024-04-26T01:49:17.562 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92563 5.13 MB/sec warmup 101 sec latency 21.062 ms 2024-04-26T01:49:18.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.037+0000 7fe286600700 1 -- 172.21.15.6:0/3341568752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe28006b100 msgr2=0x7fe28010f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:18.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.037+0000 7fe286600700 1 --2- 172.21.15.6:0/3341568752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe28006b100 0x7fe28010f8c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe2780098f0 tx=0x7fe278009c00 comp rx=0 tx=0).stop 2024-04-26T01:49:18.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.037+0000 7fe286600700 1 -- 172.21.15.6:0/3341568752 shutdown_connections 2024-04-26T01:49:18.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.037+0000 7fe286600700 1 --2- 172.21.15.6:0/3341568752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe28006b100 0x7fe28010f8c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.037+0000 7fe286600700 1 --2- 172.21.15.6:0/3341568752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe280108a60 0x7fe280108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.037+0000 7fe286600700 1 --2- 172.21.15.6:0/3341568752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe28006a740 0x7fe28006abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.037+0000 7fe286600700 1 -- 172.21.15.6:0/3341568752 >> 172.21.15.6:0/3341568752 conn(0x7fe2800788e0 msgr2=0x7fe280078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:18.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.037+0000 7fe286600700 1 -- 172.21.15.6:0/3341568752 shutdown_connections 2024-04-26T01:49:18.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.038+0000 7fe286600700 1 -- 172.21.15.6:0/3341568752 wait complete. 2024-04-26T01:49:18.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.039+0000 7fe286600700 1 Processor -- start 2024-04-26T01:49:18.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.039+0000 7fe286600700 1 -- start start 2024-04-26T01:49:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe286600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe28006a740 0x7fe28019b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe286600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28006b100 0x7fe28019b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe286600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe280108a60 0x7fe28019fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe286600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe280112c60 con 0x7fe280108a60 2024-04-26T01:49:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe286600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe280112ae0 con 0x7fe28006a740 2024-04-26T01:49:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe286600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe280112de0 con 0x7fe28006b100 2024-04-26T01:49:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe27f400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe28006a740 0x7fe28019b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe27ea00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28006b100 0x7fe28019b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe27f400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe28006a740 0x7fe28019b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56322/0 (socket says 172.21.15.6:56322) 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe27f400700 1 -- 172.21.15.6:0/4205166304 learned_addr learned my addr 172.21.15.6:0/4205166304 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe27fe00700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe280108a60 0x7fe28019fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe27f400700 1 -- 172.21.15.6:0/4205166304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28006b100 msgr2=0x7fe28019b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe27f400700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28006b100 0x7fe28019b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe27f400700 1 -- 172.21.15.6:0/4205166304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe280108a60 msgr2=0x7fe28019fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.040+0000 7fe27f400700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe280108a60 0x7fe28019fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.041+0000 7fe27f400700 1 -- 172.21.15.6:0/4205166304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2801a0240 con 0x7fe28006a740 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.041+0000 7fe27fe00700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe280108a60 0x7fe28019fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.041+0000 7fe27f400700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe28006a740 0x7fe28019b230 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe27000c110 tx=0x7fe27000c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.041+0000 7fe277e00700 1 -- 172.21.15.6:0/4205166304 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27000cd60 con 0x7fe28006a740 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.041+0000 7fe277e00700 1 -- 172.21.15.6:0/4205166304 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe270004bb0 con 0x7fe28006a740 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.041+0000 7fe277e00700 1 -- 172.21.15.6:0/4205166304 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270007dc0 con 0x7fe28006a740 2024-04-26T01:49:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.041+0000 7fe286600700 1 -- 172.21.15.6:0/4205166304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2801a04d0 con 0x7fe28006a740 2024-04-26T01:49:18.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.041+0000 7fe286600700 1 -- 172.21.15.6:0/4205166304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2801a7db0 con 0x7fe28006a740 2024-04-26T01:49:18.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.044+0000 7fe277e00700 1 -- 172.21.15.6:0/4205166304 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fe27001a020 con 0x7fe28006a740 2024-04-26T01:49:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.047+0000 7fe277e00700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe2640a3ed0 0x7fe2640a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.047+0000 7fe290f0bb80 1 Processor -- start 2024-04-26T01:49:18.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.047+0000 7fe277e00700 1 -- 172.21.15.6:0/4205166304 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fe2700d0290 con 0x7fe28006a740 2024-04-26T01:49:18.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.047+0000 7fe290f0bb80 1 -- start start 2024-04-26T01:49:18.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.047+0000 7fe27ea00700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe2640a3ed0 0x7fe2640a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe290f0bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d7e8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe290f0bb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e92d0 0x55b72d7f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe290f0bb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7f4180 0x55b72d7f6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe290f0bb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55b72d73c020 con 0x55b72d7f4180 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe290f0bb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b72d6e28c0 con 0x55b72d7e92d0 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe290f0bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b72d737fe0 con 0x55b72d7e89b0 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe27f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d7e8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe27ea00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e92d0 0x55b72d7f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe27f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d7e8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:52900/0 (socket says 172.21.15.6:52900) 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe27f400700 1 -- 172.21.15.6:0/546980042 learned_addr learned my addr 172.21.15.6:0/546980042 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe27ea00700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe2640a3ed0 0x7fe2640a6390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe280108300 tx=0x7fe26c009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe27fe00700 1 --2- 172.21.15.6:0/546980042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7f4180 0x55b72d7f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe27f400700 1 -- 172.21.15.6:0/546980042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e92d0 msgr2=0x55b72d7f3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe27f400700 1 --2- 172.21.15.6:0/546980042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e92d0 0x55b72d7f3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe27f400700 1 -- 172.21.15.6:0/546980042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7f4180 msgr2=0x55b72d7f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe27f400700 1 --2- 172.21.15.6:0/546980042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7f4180 0x55b72d7f6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.048+0000 7fe27f400700 1 -- 172.21.15.6:0/546980042 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b72d6d8810 con 0x55b72d7e89b0 2024-04-26T01:49:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.049+0000 7fe27ea00700 1 --2- 172.21.15.6:0/546980042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e92d0 0x55b72d7f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.049+0000 7fe27f400700 1 --2- 172.21.15.6:0/546980042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d7e8d90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe2700a48f0 tx=0x7fe27009bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edc8be4f756d7786 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.049+0000 7fe274c00700 1 -- 172.21.15.6:0/546980042 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27009c8b0 con 0x55b72d7e89b0 2024-04-26T01:49:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.049+0000 7fe274c00700 1 -- 172.21.15.6:0/546980042 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe27009c1b0 con 0x55b72d7e89b0 2024-04-26T01:49:18.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.049+0000 7fe274c00700 1 -- 172.21.15.6:0/546980042 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2700a0130 con 0x55b72d7e89b0 2024-04-26T01:49:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.050+0000 7fe290f0bb80 1 -- 172.21.15.6:0/546980042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 msgr2=0x55b72d7e8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.050+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/546980042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d7e8d90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe2700a48f0 tx=0x7fe27009bd70 comp rx=0 tx=0).stop 2024-04-26T01:49:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.050+0000 7fe290f0bb80 1 -- 172.21.15.6:0/546980042 shutdown_connections 2024-04-26T01:49:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.050+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/546980042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7f4180 0x55b72d7f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.050+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/546980042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d7e8d90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.050+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/546980042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e92d0 0x55b72d7f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.050+0000 7fe290f0bb80 1 -- 172.21.15.6:0/546980042 >> 172.21.15.6:0/546980042 conn(0x55b72d717730 msgr2=0x55b72d7e4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.050+0000 7fe290f0bb80 1 -- 172.21.15.6:0/546980042 shutdown_connections 2024-04-26T01:49:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.050+0000 7fe290f0bb80 1 -- 172.21.15.6:0/546980042 wait complete. 2024-04-26T01:49:18.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.051+0000 7fe290f0bb80 1 -- start start 2024-04-26T01:49:18.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.051+0000 7fe290f0bb80 1 Processor -- start 2024-04-26T01:49:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.052+0000 7fe290f0bb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e89b0 0x55b72d7f3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.052+0000 7fe290f0bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e92d0 0x55b72d7f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.052+0000 7fe290f0bb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7f4180 0x55b72d7f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.052+0000 7fe290f0bb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55b72d7f6ca0 con 0x55b72d7f4180 2024-04-26T01:49:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.052+0000 7fe290f0bb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b72d6f59f0 con 0x55b72d7e89b0 2024-04-26T01:49:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.052+0000 7fe290f0bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b72d7f6e20 con 0x55b72d7e92d0 2024-04-26T01:49:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.052+0000 7fe290f0bb80 10 client.0 _ll_register_callbacks cb 0x55b72d6d48b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.052+0000 7fe27f400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e89b0 0x55b72d7f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.052+0000 7fe27f400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e89b0 0x55b72d7f3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56330/0 (socket says 172.21.15.6:56330) 2024-04-26T01:49:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.052+0000 7fe27f400700 1 -- 172.21.15.6:0/2917253815 learned_addr learned my addr 172.21.15.6:0/2917253815 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.052+0000 7fe27fe00700 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7f4180 0x55b72d7f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.053+0000 7fe27ea00700 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e92d0 0x55b72d7f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.053+0000 7fe27f400700 1 -- 172.21.15.6:0/2917253815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e92d0 msgr2=0x55b72d7f0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.053+0000 7fe27f400700 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e92d0 0x55b72d7f0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.053+0000 7fe27f400700 1 -- 172.21.15.6:0/2917253815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7f4180 msgr2=0x55b72d7f11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.053+0000 7fe27f400700 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7f4180 0x55b72d7f11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.053+0000 7fe27f400700 1 -- 172.21.15.6:0/2917253815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b72d689a60 con 0x55b72d7e89b0 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.053+0000 7fe27ea00700 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e92d0 0x55b72d7f0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.053+0000 7fe27f400700 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e89b0 0x55b72d7f3980 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe27009c140 tx=0x7fe2700cd900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cf7b9e043ba0c2c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.053+0000 7fe269600700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.053+0000 7fe269600700 1 -- 172.21.15.6:0/2917253815 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2700a0560 con 0x55b72d7e89b0 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.053+0000 7fe269600700 1 -- 172.21.15.6:0/2917253815 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fe27009a4b0 con 0x55b72d7e89b0 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.053+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2917253815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe27009c1b0 con 0x55b72d7e89b0 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.054+0000 7fe269600700 1 -- 172.21.15.6:0/2917253815 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe2700a0130 con 0x55b72d7e89b0 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.054+0000 7fe290f0bb80 10 client.25598 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.054+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2917253815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b72d6d2c80 con 0x55b72d7e89b0 2024-04-26T01:49:18.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.054+0000 7fe269600700 1 -- 172.21.15.6:0/2917253815 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe27009c1b0 con 0x55b72d7e89b0 2024-04-26T01:49:18.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.054+0000 7fe269600700 1 -- 172.21.15.6:0/2917253815 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fe2700a4a90 con 0x55b72d7e89b0 2024-04-26T01:49:18.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.054+0000 7fe290f0bb80 10 client.25598 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:18.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.054+0000 7fe290f0bb80 10 client.25598 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:49:18.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.054+0000 7fe290f0bb80 20 client.25598 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:18.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.054+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b72d8038d0 0x55b72d805cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.054+0000 7fe290f0bb80 4 client.25598 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:18.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.054+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2917253815 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b72d749810 con 0x55b72d8038d0 2024-04-26T01:49:18.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.055+0000 7fe27ea00700 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b72d8038d0 0x55b72d805cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.055+0000 7fe27ea00700 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b72d8038d0 0x55b72d805cc0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a688f73e3a292be server_cookie=c2edf8701eee78d0 in_seq=0 out_seq=0 2024-04-26T01:49:18.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.055+0000 7fe269600700 10 client.25598 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:18.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.056+0000 7fe269600700 1 -- 172.21.15.6:0/2917253815 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b72d749810 con 0x55b72d8038d0 2024-04-26T01:49:18.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.056+0000 7fe269600700 10 client.25598 handle_command_reply: tid=0 2024-04-26T01:49:18.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.056+0000 7fe290f0bb80 1 client.25598 shutdown 2024-04-26T01:49:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.056+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2917253815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e89b0 msgr2=0x55b72d7f3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.056+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e89b0 0x55b72d7f3980 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe27009c140 tx=0x7fe2700cd900 comp rx=0 tx=0).stop 2024-04-26T01:49:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.056+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2917253815 shutdown_connections 2024-04-26T01:49:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.056+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7f4180 0x55b72d7f11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.056+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7e89b0 0x55b72d7f3980 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.056+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b72d8038d0 0x55b72d805cc0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.056+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/2917253815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e92d0 0x55b72d7f0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.056+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2917253815 >> 172.21.15.6:0/2917253815 conn(0x55b72d717730 msgr2=0x55b72d7e4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.057+0000 7fe269600700 0 client.25598 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.057+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2917253815 shutdown_connections 2024-04-26T01:49:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.057+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2917253815 wait complete. 2024-04-26T01:49:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.057+0000 7fe290f0bb80 20 client.25598 trim_cache size 0 max 16384 2024-04-26T01:49:18.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.078+0000 7fe290f0bb80 1 Processor -- start 2024-04-26T01:49:18.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.078+0000 7fe290f0bb80 1 -- start start 2024-04-26T01:49:18.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.079+0000 7fe290f0bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d7f1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.079+0000 7fe290f0bb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7e92d0 0x55b72d808910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.079+0000 7fe290f0bb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7f4180 0x55b72d808e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.079+0000 7fe290f0bb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801b1f10 con 0x55b72d7e92d0 2024-04-26T01:49:18.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.079+0000 7fe290f0bb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801af740 con 0x55b72d7f4180 2024-04-26T01:49:18.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.079+0000 7fe290f0bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801ad590 con 0x55b72d7e89b0 2024-04-26T01:49:18.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.079+0000 7fe27f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d7f1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.079+0000 7fe27f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d7f1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:52926/0 (socket says 172.21.15.6:52926) 2024-04-26T01:49:18.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.079+0000 7fe27f400700 1 -- 172.21.15.6:0/2120042619 learned_addr learned my addr 172.21.15.6:0/2120042619 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:18.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.079+0000 7fe27fe00700 1 --2- 172.21.15.6:0/2120042619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7f4180 0x55b72d808e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.079+0000 7fe27ea00700 1 --2- 172.21.15.6:0/2120042619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7e92d0 0x55b72d808910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.080+0000 7fe27fe00700 1 -- 172.21.15.6:0/2120042619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 msgr2=0x55b72d7f1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:18.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.080+0000 7fe27fe00700 1 --2- 172.21.15.6:0/2120042619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d7f1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.080+0000 7fe27fe00700 1 -- 172.21.15.6:0/2120042619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7e92d0 msgr2=0x55b72d808910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:18.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.080+0000 7fe27fe00700 1 --2- 172.21.15.6:0/2120042619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7e92d0 0x55b72d808910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.080+0000 7fe27fe00700 1 -- 172.21.15.6:0/2120042619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b72d7e96b0 con 0x55b72d7f4180 2024-04-26T01:49:18.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.080+0000 7fe27f400700 1 --2- 172.21.15.6:0/2120042619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d7f1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:18.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.080+0000 7fe27fe00700 1 --2- 172.21.15.6:0/2120042619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7f4180 0x55b72d808e50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe2780098c0 tx=0x7fe27800fbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85a1e06976a32f9d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:18.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.080+0000 7fe26be00700 1 -- 172.21.15.6:0/2120042619 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27800f460 con 0x55b72d7f4180 2024-04-26T01:49:18.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.080+0000 7fe26be00700 1 -- 172.21.15.6:0/2120042619 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe278005040 con 0x55b72d7f4180 2024-04-26T01:49:18.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.080+0000 7fe26be00700 1 -- 172.21.15.6:0/2120042619 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278004a40 con 0x55b72d7f4180 2024-04-26T01:49:18.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.081+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2120042619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7f4180 msgr2=0x55b72d808e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:18.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.081+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/2120042619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7f4180 0x55b72d808e50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe2780098c0 tx=0x7fe27800fbc0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.081+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2120042619 shutdown_connections 2024-04-26T01:49:18.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.081+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/2120042619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7f4180 0x55b72d808e50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.081+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/2120042619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d7f1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.081+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/2120042619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7e92d0 0x55b72d808910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.081+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2120042619 >> 172.21.15.6:0/2120042619 conn(0x55b72d717730 msgr2=0x55b72d7f0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:18.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.081+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2120042619 shutdown_connections 2024-04-26T01:49:18.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.081+0000 7fe290f0bb80 1 -- 172.21.15.6:0/2120042619 wait complete. 2024-04-26T01:49:18.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.082+0000 7fe290f0bb80 1 -- start start 2024-04-26T01:49:18.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.082+0000 7fe290f0bb80 1 Processor -- start 2024-04-26T01:49:18.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.083+0000 7fe290f0bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d85dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.083+0000 7fe290f0bb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7e92d0 0x55b72d85e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.083+0000 7fe290f0bb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7f4180 0x55b72d868ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.083+0000 7fe290f0bb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801adc00 con 0x55b72d7e92d0 2024-04-26T01:49:18.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.083+0000 7fe290f0bb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801af070 con 0x55b72d7f4180 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.083+0000 7fe290f0bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801ae2d0 con 0x55b72d7e89b0 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.083+0000 7fe290f0bb80 10 client.0 _ll_register_callbacks cb 0x55b72d6d48b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.083+0000 7fe27f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d85dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.083+0000 7fe27ea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7e92d0 0x55b72d85e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.083+0000 7fe27f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d85dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:52934/0 (socket says 172.21.15.6:52934) 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.083+0000 7fe27f400700 1 -- 172.21.15.6:0/3619554552 learned_addr learned my addr 172.21.15.6:0/3619554552 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.084+0000 7fe27fe00700 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7f4180 0x55b72d868ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.084+0000 7fe27f400700 1 -- 172.21.15.6:0/3619554552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7f4180 msgr2=0x55b72d868ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.084+0000 7fe27f400700 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7f4180 0x55b72d868ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.084+0000 7fe27f400700 1 -- 172.21.15.6:0/3619554552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7e92d0 msgr2=0x55b72d85e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.084+0000 7fe27f400700 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7e92d0 0x55b72d85e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.084+0000 7fe27f400700 1 -- 172.21.15.6:0/3619554552 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b72d869000 con 0x55b72d7e89b0 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.084+0000 7fe27ea00700 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7e92d0 0x55b72d85e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:18.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.084+0000 7fe27f400700 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d85dc20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe270007fd0 tx=0x7fe27009c030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6514d4a8e0d96222 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:18.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.084+0000 7fe269600700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:49:18.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.085+0000 7fe269600700 1 -- 172.21.15.6:0/3619554552 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2700a0560 con 0x55b72d7e89b0 2024-04-26T01:49:18.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.085+0000 7fe269600700 1 -- 172.21.15.6:0/3619554552 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fe2700853e0 con 0x55b72d7e89b0 2024-04-26T01:49:18.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.085+0000 7fe290f0bb80 1 -- 172.21.15.6:0/3619554552 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe278005040 con 0x55b72d7e89b0 2024-04-26T01:49:18.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.085+0000 7fe269600700 1 -- 172.21.15.6:0/3619554552 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe2700a8020 con 0x55b72d7e89b0 2024-04-26T01:49:18.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.085+0000 7fe290f0bb80 10 client.25654 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:18.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.085+0000 7fe290f0bb80 1 -- 172.21.15.6:0/3619554552 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b72d869000 con 0x55b72d7e89b0 2024-04-26T01:49:18.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.085+0000 7fe269600700 1 -- 172.21.15.6:0/3619554552 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe278005040 con 0x55b72d7e89b0 2024-04-26T01:49:18.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.085+0000 7fe269600700 1 -- 172.21.15.6:0/3619554552 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fe27009c410 con 0x55b72d7e89b0 2024-04-26T01:49:18.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.086+0000 7fe290f0bb80 10 client.25654 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:18.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.086+0000 7fe290f0bb80 10 client.25654 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:49:18.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.086+0000 7fe290f0bb80 20 client.25654 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:18.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.086+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b72d86a520 0x55b72d86c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:18.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.086+0000 7fe290f0bb80 4 client.25654 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-26T01:49:18.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.086+0000 7fe290f0bb80 1 -- 172.21.15.6:0/3619554552 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x55b72d7438b0 con 0x55b72d86a520 2024-04-26T01:49:18.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.086+0000 7fe27ea00700 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b72d86a520 0x55b72d86c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:18.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.086+0000 7fe27ea00700 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b72d86a520 0x55b72d86c910 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6161b582a621eb5b server_cookie=2d9368f81b53f1bc in_seq=0 out_seq=0 2024-04-26T01:49:18.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.086+0000 7fe269600700 10 client.25654 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.150+0000 7fe269600700 1 -- 172.21.15.6:0/3619554552 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55b72d7438b0 con 0x55b72d86a520 2024-04-26T01:49:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.150+0000 7fe269600700 10 client.25654 handle_command_reply: tid=0 2024-04-26T01:49:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.150+0000 7fe290f0bb80 1 client.25654 shutdown 2024-04-26T01:49:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.151+0000 7fe290f0bb80 1 -- 172.21.15.6:0/3619554552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 msgr2=0x55b72d85dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.151+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d85dc20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe270007fd0 tx=0x7fe27009c030 comp rx=0 tx=0).stop 2024-04-26T01:49:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.151+0000 7fe290f0bb80 1 -- 172.21.15.6:0/3619554552 shutdown_connections 2024-04-26T01:49:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.151+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b72d7f4180 0x55b72d868ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.151+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b72d86a520 0x55b72d86c910 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.151+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b72d7e89b0 0x55b72d85dc20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.151+0000 7fe290f0bb80 1 --2- 172.21.15.6:0/3619554552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b72d7e92d0 0x55b72d85e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.151+0000 7fe290f0bb80 1 -- 172.21.15.6:0/3619554552 >> 172.21.15.6:0/3619554552 conn(0x55b72d717730 msgr2=0x55b72d7f24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.151+0000 7fe269600700 0 client.25654 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.152+0000 7fe290f0bb80 1 -- 172.21.15.6:0/3619554552 shutdown_connections 2024-04-26T01:49:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.152+0000 7fe290f0bb80 1 -- 172.21.15.6:0/3619554552 wait complete. 2024-04-26T01:49:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.152+0000 7fe290f0bb80 20 client.25654 trim_cache size 0 max 16384 2024-04-26T01:49:18.152 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:49:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 31, 2024-04-26T01:49:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:49:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 10, 2024-04-26T01:49:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:49:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "dcc36de": { 2024-04-26T01:49:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 10, 2024-04-26T01:49:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:49:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:49:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:49:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:49:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 90.0, 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:18.153 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:49:18.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.153+0000 7fe286600700 1 -- 172.21.15.6:0/4205166304 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe2640a3ed0 msgr2=0x7fe2640a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:18.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.153+0000 7fe286600700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe2640a3ed0 0x7fe2640a6390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe280108300 tx=0x7fe26c009260 comp rx=0 tx=0).stop 2024-04-26T01:49:18.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.154+0000 7fe286600700 1 -- 172.21.15.6:0/4205166304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe28006a740 msgr2=0x7fe28019b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:18.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.154+0000 7fe286600700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe28006a740 0x7fe28019b230 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe27000c110 tx=0x7fe27000c420 comp rx=0 tx=0).stop 2024-04-26T01:49:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.154+0000 7fe286600700 1 -- 172.21.15.6:0/4205166304 shutdown_connections 2024-04-26T01:49:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.154+0000 7fe286600700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe2640a3ed0 0x7fe2640a6390 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.154+0000 7fe286600700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe280108a60 0x7fe28019fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.154+0000 7fe286600700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe28006a740 0x7fe28019b230 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.154+0000 7fe286600700 1 --2- 172.21.15.6:0/4205166304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28006b100 0x7fe28019b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.154+0000 7fe286600700 1 -- 172.21.15.6:0/4205166304 >> 172.21.15.6:0/4205166304 conn(0x7fe2800788e0 msgr2=0x7fe280100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.154+0000 7fe286600700 1 -- 172.21.15.6:0/4205166304 shutdown_connections 2024-04-26T01:49:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:18.154+0000 7fe286600700 1 -- 172.21.15.6:0/4205166304 wait complete. 2024-04-26T01:49:18.562 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 5.08 MB/sec warmup 102 sec latency 411.839 ms 2024-04-26T01:49:18.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:18 smithi006 ceph-mon[28419]: pgmap v106: 97 pgs: 97 active+clean; 36 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 763 KiB/s wr, 111 op/s 2024-04-26T01:49:18.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:18 smithi188 ceph-mon[32751]: pgmap v106: 97 pgs: 97 active+clean; 36 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 763 KiB/s wr, 111 op/s 2024-04-26T01:49:18.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:18 smithi029 ceph-mon[32775]: pgmap v106: 97 pgs: 97 active+clean; 36 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 763 KiB/s wr, 111 op/s 2024-04-26T01:49:19.054 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'dcc36de', quiesce took 7.7 seconds. Will release after: 30.0 2024-04-26T01:49:19.563 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 5.03 MB/sec warmup 103 sec latency 1412.004 ms 2024-04-26T01:49:20.563 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.98 MB/sec warmup 104 sec latency 2412.169 ms 2024-04-26T01:49:20.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:20 smithi006 ceph-mon[28419]: pgmap v107: 97 pgs: 97 active+clean; 36 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 491 KiB/s wr, 87 op/s 2024-04-26T01:49:20.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:20 smithi188 ceph-mon[32751]: pgmap v107: 97 pgs: 97 active+clean; 36 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 491 KiB/s wr, 87 op/s 2024-04-26T01:49:20.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:20 smithi029 ceph-mon[32775]: pgmap v107: 97 pgs: 97 active+clean; 36 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 491 KiB/s wr, 87 op/s 2024-04-26T01:49:21.563 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.94 MB/sec warmup 105 sec latency 3412.328 ms 2024-04-26T01:49:22.563 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.89 MB/sec warmup 106 sec latency 4412.413 ms 2024-04-26T01:49:22.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:22 smithi006 ceph-mon[28419]: pgmap v108: 97 pgs: 97 active+clean; 38 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 689 KiB/s wr, 136 op/s 2024-04-26T01:49:22.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:22 smithi188 ceph-mon[32751]: pgmap v108: 97 pgs: 97 active+clean; 38 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 689 KiB/s wr, 136 op/s 2024-04-26T01:49:22.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:22 smithi029 ceph-mon[32775]: pgmap v108: 97 pgs: 97 active+clean; 38 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 689 KiB/s wr, 136 op/s 2024-04-26T01:49:23.563 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.84 MB/sec warmup 107 sec latency 5412.533 ms 2024-04-26T01:49:24.563 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.80 MB/sec warmup 108 sec latency 6412.690 ms 2024-04-26T01:49:24.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:24 smithi006 ceph-mon[28419]: pgmap v109: 97 pgs: 97 active+clean; 38 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 620 KiB/s wr, 120 op/s 2024-04-26T01:49:24.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:24 smithi188 ceph-mon[32751]: pgmap v109: 97 pgs: 97 active+clean; 38 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 620 KiB/s wr, 120 op/s 2024-04-26T01:49:24.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:24 smithi029 ceph-mon[32775]: pgmap v109: 97 pgs: 97 active+clean; 38 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 620 KiB/s wr, 120 op/s 2024-04-26T01:49:25.563 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.76 MB/sec warmup 109 sec latency 7412.853 ms 2024-04-26T01:49:26.564 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.71 MB/sec warmup 110 sec latency 8413.007 ms 2024-04-26T01:49:26.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:26 smithi006 ceph-mon[28419]: pgmap v110: 97 pgs: 97 active+clean; 38 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 620 KiB/s wr, 120 op/s 2024-04-26T01:49:26.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:26 smithi188 ceph-mon[32751]: pgmap v110: 97 pgs: 97 active+clean; 38 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 620 KiB/s wr, 120 op/s 2024-04-26T01:49:26.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:26 smithi029 ceph-mon[32775]: pgmap v110: 97 pgs: 97 active+clean; 38 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 620 KiB/s wr, 120 op/s 2024-04-26T01:49:27.564 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.67 MB/sec warmup 111 sec latency 9413.167 ms 2024-04-26T01:49:28.564 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.63 MB/sec warmup 112 sec latency 10413.335 ms 2024-04-26T01:49:28.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:28 smithi006 ceph-mon[28419]: pgmap v111: 97 pgs: 97 active+clean; 38 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 707 KiB/s wr, 129 op/s 2024-04-26T01:49:28.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:28 smithi188 ceph-mon[32751]: pgmap v111: 97 pgs: 97 active+clean; 38 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 707 KiB/s wr, 129 op/s 2024-04-26T01:49:28.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:28 smithi029 ceph-mon[32775]: pgmap v111: 97 pgs: 97 active+clean; 38 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 707 KiB/s wr, 129 op/s 2024-04-26T01:49:29.564 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.59 MB/sec warmup 113 sec latency 11413.474 ms 2024-04-26T01:49:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.044+0000 7fa79fe00640 1 -- 172.21.15.6:0/3588984833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7a01092c0 msgr2=0x7fa7a0110e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.044+0000 7fa79fe00640 1 --2- 172.21.15.6:0/3588984833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7a01092c0 0x7fa7a0110e90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa798009860 tx=0x7fa79802a340 comp rx=0 tx=0).stop 2024-04-26T01:49:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.044+0000 7fa79fe00640 1 -- 172.21.15.6:0/3588984833 shutdown_connections 2024-04-26T01:49:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.044+0000 7fa79fe00640 1 --2- 172.21.15.6:0/3588984833 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7a0104ba0 0x7fa7a0108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.044+0000 7fa79fe00640 1 --2- 172.21.15.6:0/3588984833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7a01092c0 0x7fa7a0110e90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.044+0000 7fa79fe00640 1 --2- 172.21.15.6:0/3588984833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7a01041f0 0x7fa7a01045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.044+0000 7fa79fe00640 1 -- 172.21.15.6:0/3588984833 >> 172.21.15.6:0/3588984833 conn(0x7fa7a00ffa50 msgr2=0x7fa7a0101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.044+0000 7fa79fe00640 1 -- 172.21.15.6:0/3588984833 shutdown_connections 2024-04-26T01:49:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.044+0000 7fa79fe00640 1 -- 172.21.15.6:0/3588984833 wait complete. 2024-04-26T01:49:30.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.045+0000 7fa79fe00640 1 Processor -- start 2024-04-26T01:49:30.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.045+0000 7fa79fe00640 1 -- start start 2024-04-26T01:49:30.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7a01041f0 0x7fa7a01a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7a01092c0 0x7fa7a01a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7a019a630 0x7fa7a019aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0114230 con 0x7fa7a019a630 2024-04-26T01:49:30.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79fe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01140b0 con 0x7fa7a01041f0 2024-04-26T01:49:30.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01143b0 con 0x7fa7a01092c0 2024-04-26T01:49:30.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7a019a630 0x7fa7a019aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7a019a630 0x7fa7a019aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54058/0 (socket says 172.21.15.6:54058) 2024-04-26T01:49:30.047 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79f400640 1 -- 172.21.15.6:0/53233238 learned_addr learned my addr 172.21.15.6:0/53233238 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:30.047 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79ea00640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7a01041f0 0x7fa7a01a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.047 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79e000640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7a01092c0 0x7fa7a01a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.047 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79f400640 1 -- 172.21.15.6:0/53233238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7a01092c0 msgr2=0x7fa7a01a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:30.047 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79f400640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7a01092c0 0x7fa7a01a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.047 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79f400640 1 -- 172.21.15.6:0/53233238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7a01041f0 msgr2=0x7fa7a01a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:30.047 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.046+0000 7fa79f400640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7a01041f0 0x7fa7a01a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.047 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.047+0000 7fa79f400640 1 -- 172.21.15.6:0/53233238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a019b150 con 0x7fa7a019a630 2024-04-26T01:49:30.047 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.047+0000 7fa79f400640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7a019a630 0x7fa7a019aae0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa79802a930 tx=0x7fa798032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:30.047 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.047+0000 7fa797400640 1 -- 172.21.15.6:0/53233238 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798040020 con 0x7fa7a019a630 2024-04-26T01:49:30.047 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.047+0000 7fa79fe00640 1 -- 172.21.15.6:0/53233238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a019b3e0 con 0x7fa7a019a630 2024-04-26T01:49:30.047 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.047+0000 7fa797400640 1 -- 172.21.15.6:0/53233238 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa798030d50 con 0x7fa7a019a630 2024-04-26T01:49:30.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.047+0000 7fa797400640 1 -- 172.21.15.6:0/53233238 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798037040 con 0x7fa7a019a630 2024-04-26T01:49:30.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.047+0000 7fa79fe00640 1 -- 172.21.15.6:0/53233238 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a01a7590 con 0x7fa7a019a630 2024-04-26T01:49:30.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.051+0000 7fa7a883a740 1 Processor -- start 2024-04-26T01:49:30.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.051+0000 7fa797400640 1 -- 172.21.15.6:0/53233238 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fa798043020 con 0x7fa7a019a630 2024-04-26T01:49:30.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.051+0000 7fa7a883a740 1 -- start start 2024-04-26T01:49:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.051+0000 7fa7a883a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 0x55fb0799dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.051+0000 7fa7a883a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799e2d0 0x55fb079a8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.051+0000 7fa7a883a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb079a8b90 0x55fb079aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.051+0000 7fa7a883a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55fb07903900 con 0x55fb0799d9b0 2024-04-26T01:49:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.051+0000 7fa7a883a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55fb078a9ef0 con 0x55fb079a8b90 2024-04-26T01:49:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.051+0000 7fa7a883a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fb078ff8c0 con 0x55fb0799e2d0 2024-04-26T01:49:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.052+0000 7fa79e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 0x55fb0799dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.052+0000 7fa79ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799e2d0 0x55fb079a8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.052+0000 7fa79ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799e2d0 0x55fb079a8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:49744/0 (socket says 172.21.15.6:49744) 2024-04-26T01:49:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.052+0000 7fa79e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 0x55fb0799dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54062/0 (socket says 172.21.15.6:54062) 2024-04-26T01:49:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.052+0000 7fa79ea00640 1 -- 172.21.15.6:0/1026022178 learned_addr learned my addr 172.21.15.6:0/1026022178 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.052+0000 7fa79e000640 1 --2- 172.21.15.6:0/1026022178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb079a8b90 0x55fb079aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.052+0000 7fa79ea00640 1 -- 172.21.15.6:0/1026022178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb079a8b90 msgr2=0x55fb079aaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.052+0000 7fa79ea00640 1 --2- 172.21.15.6:0/1026022178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb079a8b90 0x55fb079aaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.052+0000 7fa79ea00640 1 -- 172.21.15.6:0/1026022178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 msgr2=0x55fb0799dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.052+0000 7fa79ea00640 1 --2- 172.21.15.6:0/1026022178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 0x55fb0799dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.052+0000 7fa79ea00640 1 -- 172.21.15.6:0/1026022178 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb07616210 con 0x55fb0799e2d0 2024-04-26T01:49:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.053+0000 7fa79ea00640 1 --2- 172.21.15.6:0/1026022178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799e2d0 0x55fb079a8650 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa7900114f0 tx=0x7fa7900119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2990d97b235dccd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.053+0000 7fa797400640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa7840a3d80 0x7fa7840a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.053+0000 7fa78fe00640 1 -- 172.21.15.6:0/1026022178 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900112f0 con 0x55fb0799e2d0 2024-04-26T01:49:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.053+0000 7fa797400640 1 -- 172.21.15.6:0/53233238 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fa7980f0370 con 0x7fa7a019a630 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.053+0000 7fa78fe00640 1 -- 172.21.15.6:0/1026022178 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa790003970 con 0x55fb0799e2d0 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.053+0000 7fa78fe00640 1 -- 172.21.15.6:0/1026022178 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790004680 con 0x55fb0799e2d0 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.053+0000 7fa79e000640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa7840a3d80 0x7fa7840a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.054+0000 7fa79e000640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa7840a3d80 0x7fa7840a6240 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x55fb0799b050 tx=0x7fa7880097b0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.054+0000 7fa7a883a740 1 -- 172.21.15.6:0/1026022178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799e2d0 msgr2=0x55fb079a8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.054+0000 7fa7a883a740 1 --2- 172.21.15.6:0/1026022178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799e2d0 0x55fb079a8650 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa7900114f0 tx=0x7fa7900119b0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.054+0000 7fa7a883a740 1 -- 172.21.15.6:0/1026022178 shutdown_connections 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.054+0000 7fa7a883a740 1 --2- 172.21.15.6:0/1026022178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb079a8b90 0x55fb079aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.054+0000 7fa7a883a740 1 --2- 172.21.15.6:0/1026022178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799e2d0 0x55fb079a8650 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.054+0000 7fa7a883a740 1 --2- 172.21.15.6:0/1026022178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 0x55fb0799dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.054+0000 7fa7a883a740 1 -- 172.21.15.6:0/1026022178 >> 172.21.15.6:0/1026022178 conn(0x55fb07999ec0 msgr2=0x55fb0799a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.054+0000 7fa7a883a740 1 -- 172.21.15.6:0/1026022178 shutdown_connections 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.054+0000 7fa7a883a740 1 -- 172.21.15.6:0/1026022178 wait complete. 2024-04-26T01:49:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.055+0000 7fa7a883a740 1 -- start start 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.055+0000 7fa7a883a740 1 Processor -- start 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.055+0000 7fa7a883a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799d9b0 0x55fb079a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.055+0000 7fa7a883a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 0x55fb079a6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.055+0000 7fa7a883a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb079a8b90 0x55fb079a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.055+0000 7fa7a883a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55fb079ab7c0 con 0x55fb0799e2d0 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.055+0000 7fa7a883a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55fb078bd310 con 0x55fb079a8b90 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.055+0000 7fa7a883a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fb079ab940 con 0x55fb0799d9b0 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.055+0000 7fa7a883a740 10 client.0 _ll_register_callbacks cb 0x55fb0788bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.056+0000 7fa79e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 0x55fb079a6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.056+0000 7fa79e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 0x55fb079a6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54066/0 (socket says 172.21.15.6:54066) 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.056+0000 7fa79ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799d9b0 0x55fb079a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.056+0000 7fa79e000640 1 -- 172.21.15.6:0/209814485 learned_addr learned my addr 172.21.15.6:0/209814485 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.056+0000 7fa79f400640 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb079a8b90 0x55fb079a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.056+0000 7fa79ea00640 1 -- 172.21.15.6:0/209814485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb079a8b90 msgr2=0x55fb079a65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.056+0000 7fa79ea00640 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb079a8b90 0x55fb079a65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.056+0000 7fa79ea00640 1 -- 172.21.15.6:0/209814485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 msgr2=0x55fb079a6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.056+0000 7fa79ea00640 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 0x55fb079a6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.056+0000 7fa79ea00640 1 -- 172.21.15.6:0/209814485 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb076e2620 con 0x55fb0799d9b0 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.056+0000 7fa79e000640 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 0x55fb079a6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.057+0000 7fa79ea00640 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799d9b0 0x55fb079a5b50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa790011eb0 tx=0x7fa790004280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=644c2d2cf924c522 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.057+0000 7fa78cc00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.057+0000 7fa78cc00640 1 -- 172.21.15.6:0/209814485 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790027020 con 0x55fb0799d9b0 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.057+0000 7fa7a883a740 1 -- 172.21.15.6:0/209814485 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa790003970 con 0x55fb0799d9b0 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.057+0000 7fa78cc00640 1 -- 172.21.15.6:0/209814485 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fa790004a40 con 0x55fb0799d9b0 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.057+0000 7fa78cc00640 1 -- 172.21.15.6:0/209814485 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa790032050 con 0x55fb0799d9b0 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.057+0000 7fa7a883a740 10 client.25666 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.057+0000 7fa7a883a740 1 -- 172.21.15.6:0/209814485 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fb079b7a50 con 0x55fb0799d9b0 2024-04-26T01:49:30.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.058+0000 7fa78cc00640 1 -- 172.21.15.6:0/209814485 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa790003970 con 0x55fb0799d9b0 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.058+0000 7fa78cc00640 1 -- 172.21.15.6:0/209814485 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fa790018070 con 0x55fb0799d9b0 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.058+0000 7fa7a883a740 10 client.25666 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.058+0000 7fa7a883a740 10 client.25666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.058+0000 7fa7a883a740 20 client.25666 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.058+0000 7fa7a883a740 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fb079b8e50 0x55fb079bb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.058+0000 7fa7a883a740 4 client.25666 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.058+0000 7fa7a883a740 1 -- 172.21.15.6:0/209814485 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb079178b0 con 0x55fb079b8e50 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.058+0000 7fa79e000640 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fb079b8e50 0x55fb079bb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.059+0000 7fa79e000640 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fb079b8e50 0x55fb079bb240 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e3358c68147aab2 server_cookie=b8ea3c11d0755e12 in_seq=0 out_seq=0 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.059+0000 7fa78cc00640 10 client.25666 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa78cc00640 1 -- 172.21.15.6:0/209814485 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fb079178b0 con 0x55fb079b8e50 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa78cc00640 10 client.25666 handle_command_reply: tid=0 2024-04-26T01:49:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa7a883a740 1 client.25666 shutdown 2024-04-26T01:49:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa7a883a740 1 -- 172.21.15.6:0/209814485 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799d9b0 msgr2=0x55fb079a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa7a883a740 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799d9b0 0x55fb079a5b50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa790011eb0 tx=0x7fa790004280 comp rx=0 tx=0).stop 2024-04-26T01:49:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa7a883a740 1 -- 172.21.15.6:0/209814485 shutdown_connections 2024-04-26T01:49:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa7a883a740 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fb079b8e50 0x55fb079bb240 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa7a883a740 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 0x55fb079a6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa7a883a740 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb079a8b90 0x55fb079a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa7a883a740 1 --2- 172.21.15.6:0/209814485 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb0799d9b0 0x55fb079a5b50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa7a883a740 1 -- 172.21.15.6:0/209814485 >> 172.21.15.6:0/209814485 conn(0x55fb07999ec0 msgr2=0x55fb0799fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa78cc00640 0 client.25666 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa7a883a740 1 -- 172.21.15.6:0/209814485 shutdown_connections 2024-04-26T01:49:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa7a883a740 1 -- 172.21.15.6:0/209814485 wait complete. 2024-04-26T01:49:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.060+0000 7fa7a883a740 20 client.25666 trim_cache size 0 max 16384 2024-04-26T01:49:30.075 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.077+0000 7fa7a883a740 1 Processor -- start 2024-04-26T01:49:30.075 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.077+0000 7fa7a883a740 1 -- start start 2024-04-26T01:49:30.075 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.077+0000 7fa7a883a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 0x55fb079b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.075 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.077+0000 7fa7a883a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb0799e2d0 0x55fb07998ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.077+0000 7fa7a883a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb079a8b90 0x55fb079c0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.077+0000 7fa7a883a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01b2660 con 0x55fb0799d9b0 2024-04-26T01:49:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.077+0000 7fa7a883a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01afed0 con 0x55fb0799e2d0 2024-04-26T01:49:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.077+0000 7fa7a883a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01add20 con 0x55fb079a8b90 2024-04-26T01:49:30.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.077+0000 7fa79ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 0x55fb079b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa79ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 0x55fb079b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54082/0 (socket says 172.21.15.6:54082) 2024-04-26T01:49:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa79ea00640 1 -- 172.21.15.6:0/1539189102 learned_addr learned my addr 172.21.15.6:0/1539189102 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa79f400640 1 --2- 172.21.15.6:0/1539189102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb079a8b90 0x55fb079c0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa79e000640 1 --2- 172.21.15.6:0/1539189102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb0799e2d0 0x55fb07998ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa79ea00640 1 -- 172.21.15.6:0/1539189102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb079a8b90 msgr2=0x55fb079c0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa79ea00640 1 --2- 172.21.15.6:0/1539189102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb079a8b90 0x55fb079c0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa79ea00640 1 -- 172.21.15.6:0/1539189102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb0799e2d0 msgr2=0x55fb07998ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa79ea00640 1 --2- 172.21.15.6:0/1539189102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb0799e2d0 0x55fb07998ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa79ea00640 1 -- 172.21.15.6:0/1539189102 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb0799ec60 con 0x55fb0799d9b0 2024-04-26T01:49:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa79ea00640 1 --2- 172.21.15.6:0/1539189102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 0x55fb079b4680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa7900119f0 tx=0x7fa790009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95bfa715dbe8f382 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa78f400640 1 -- 172.21.15.6:0/1539189102 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790027030 con 0x55fb0799d9b0 2024-04-26T01:49:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa78f400640 1 -- 172.21.15.6:0/1539189102 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa790003e20 con 0x55fb0799d9b0 2024-04-26T01:49:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.078+0000 7fa78f400640 1 -- 172.21.15.6:0/1539189102 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79001e040 con 0x55fb0799d9b0 2024-04-26T01:49:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.079+0000 7fa7a883a740 1 -- 172.21.15.6:0/1539189102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 msgr2=0x55fb079b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.079+0000 7fa7a883a740 1 --2- 172.21.15.6:0/1539189102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 0x55fb079b4680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa7900119f0 tx=0x7fa790009fd0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.079+0000 7fa7a883a740 1 -- 172.21.15.6:0/1539189102 shutdown_connections 2024-04-26T01:49:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.079+0000 7fa7a883a740 1 --2- 172.21.15.6:0/1539189102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb079a8b90 0x55fb079c0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.079+0000 7fa7a883a740 1 --2- 172.21.15.6:0/1539189102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb0799e2d0 0x55fb07998ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.079+0000 7fa7a883a740 1 --2- 172.21.15.6:0/1539189102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799d9b0 0x55fb079b4680 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.079+0000 7fa7a883a740 1 -- 172.21.15.6:0/1539189102 >> 172.21.15.6:0/1539189102 conn(0x55fb07999ec0 msgr2=0x55fb07998790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.079+0000 7fa7a883a740 1 -- 172.21.15.6:0/1539189102 shutdown_connections 2024-04-26T01:49:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.079+0000 7fa7a883a740 1 -- 172.21.15.6:0/1539189102 wait complete. 2024-04-26T01:49:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.080+0000 7fa7a883a740 1 -- start start 2024-04-26T01:49:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.080+0000 7fa7a883a740 1 Processor -- start 2024-04-26T01:49:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.080+0000 7fa7a883a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb0799d9b0 0x55fb079b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa7a883a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 0x55fb079ae260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa7a883a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb079a8b90 0x55fb079ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa7a883a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01ae390 con 0x55fb0799e2d0 2024-04-26T01:49:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa7a883a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01af800 con 0x55fb0799d9b0 2024-04-26T01:49:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa7a883a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01aea60 con 0x55fb079a8b90 2024-04-26T01:49:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa7a883a740 10 client.0 _ll_register_callbacks cb 0x7fa790009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa79e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 0x55fb079ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa79ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb0799d9b0 0x55fb079b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa79ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb0799d9b0 0x55fb079b0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:60436/0 (socket says 172.21.15.6:60436) 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa79ea00640 1 -- 172.21.15.6:0/148128907 learned_addr learned my addr 172.21.15.6:0/148128907 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa79f400640 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb079a8b90 0x55fb079ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa79e000640 1 -- 172.21.15.6:0/148128907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb079a8b90 msgr2=0x55fb079ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa79e000640 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb079a8b90 0x55fb079ae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa79e000640 1 -- 172.21.15.6:0/148128907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb0799d9b0 msgr2=0x55fb079b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa79e000640 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb0799d9b0 0x55fb079b0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa79e000640 1 -- 172.21.15.6:0/148128907 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb079aeee0 con 0x55fb0799e2d0 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.081+0000 7fa79ea00640 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb0799d9b0 0x55fb079b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.082+0000 7fa79e000640 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 0x55fb079ae260 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55fb079a7120 tx=0x7fa788009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c4ab44ac80edba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.082+0000 7fa78cc00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.082+0000 7fa78cc00640 1 -- 172.21.15.6:0/148128907 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78802f110 con 0x55fb0799e2d0 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.082+0000 7fa78cc00640 1 -- 172.21.15.6:0/148128907 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fa788009040 con 0x55fb0799e2d0 2024-04-26T01:49:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.082+0000 7fa7a883a740 1 -- 172.21.15.6:0/148128907 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa790003e20 con 0x55fb0799e2d0 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.082+0000 7fa78cc00640 1 -- 172.21.15.6:0/148128907 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa78803e050 con 0x55fb0799e2d0 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.082+0000 7fa7a883a740 10 client.15498 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.082+0000 7fa7a883a740 1 -- 172.21.15.6:0/148128907 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fb07a27ac0 con 0x55fb0799e2d0 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.083+0000 7fa78cc00640 1 -- 172.21.15.6:0/148128907 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa790003e20 con 0x55fb0799e2d0 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.083+0000 7fa78cc00640 1 -- 172.21.15.6:0/148128907 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fa7880205e0 con 0x55fb0799e2d0 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.083+0000 7fa7a883a740 10 client.15498 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.083+0000 7fa7a883a740 10 client.15498 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.083+0000 7fa7a883a740 20 client.15498 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.083+0000 7fa7a883a740 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fb07a28270 0x55fb07a2a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.083+0000 7fa7a883a740 4 client.15498 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.083+0000 7fa7a883a740 1 -- 172.21.15.6:0/148128907 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fb07911320 con 0x55fb07a28270 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.083+0000 7fa79ea00640 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fb07a28270 0x55fb07a2a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.084+0000 7fa79ea00640 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fb07a28270 0x55fb07a2a690 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=876934657a3b6c48 server_cookie=5e5ebc4ac1be3614 in_seq=0 out_seq=0 2024-04-26T01:49:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.084+0000 7fa78cc00640 10 client.15498 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa78cc00640 1 -- 172.21.15.6:0/148128907 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fb07911320 con 0x55fb07a28270 2024-04-26T01:49:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa78cc00640 10 client.15498 handle_command_reply: tid=0 2024-04-26T01:49:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa7a883a740 1 client.15498 shutdown 2024-04-26T01:49:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa7a883a740 1 -- 172.21.15.6:0/148128907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 msgr2=0x55fb079ae260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa7a883a740 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 0x55fb079ae260 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55fb079a7120 tx=0x7fa788009260 comp rx=0 tx=0).stop 2024-04-26T01:49:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa7a883a740 1 -- 172.21.15.6:0/148128907 shutdown_connections 2024-04-26T01:49:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa7a883a740 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fb07a28270 0x55fb07a2a690 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa7a883a740 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fb0799e2d0 0x55fb079ae260 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa7a883a740 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fb079a8b90 0x55fb079ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa7a883a740 1 --2- 172.21.15.6:0/148128907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fb0799d9b0 0x55fb079b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa78cc00640 0 client.15498 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa7a883a740 1 -- 172.21.15.6:0/148128907 >> 172.21.15.6:0/148128907 conn(0x55fb07999ec0 msgr2=0x55fb079a7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa7a883a740 1 -- 172.21.15.6:0/148128907 shutdown_connections 2024-04-26T01:49:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa7a883a740 1 -- 172.21.15.6:0/148128907 wait complete. 2024-04-26T01:49:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.112+0000 7fa7a883a740 20 client.15498 trim_cache size 0 max 16384 2024-04-26T01:49:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:49:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:49:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:49:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:49:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.113+0000 7fa79fe00640 1 -- 172.21.15.6:0/53233238 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa7840a3d80 msgr2=0x7fa7840a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.113+0000 7fa79fe00640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa7840a3d80 0x7fa7840a6240 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x55fb0799b050 tx=0x7fa7880097b0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.113+0000 7fa79fe00640 1 -- 172.21.15.6:0/53233238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7a019a630 msgr2=0x7fa7a019aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.113+0000 7fa79fe00640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7a019a630 0x7fa7a019aae0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa79802a930 tx=0x7fa798032d50 comp rx=0 tx=0).stop 2024-04-26T01:49:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.113+0000 7fa79fe00640 1 -- 172.21.15.6:0/53233238 shutdown_connections 2024-04-26T01:49:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.113+0000 7fa79fe00640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa7840a3d80 0x7fa7840a6240 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.113+0000 7fa79fe00640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7a019a630 0x7fa7a019aae0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.113+0000 7fa79fe00640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7a01092c0 0x7fa7a01a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.113+0000 7fa79fe00640 1 --2- 172.21.15.6:0/53233238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7a01041f0 0x7fa7a01a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.114+0000 7fa79fe00640 1 -- 172.21.15.6:0/53233238 >> 172.21.15.6:0/53233238 conn(0x7fa7a00ffa50 msgr2=0x7fa7a0101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.114+0000 7fa79fe00640 1 -- 172.21.15.6:0/53233238 shutdown_connections 2024-04-26T01:49:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:30.114+0000 7fa79fe00640 1 -- 172.21.15.6:0/53233238 wait complete. 2024-04-26T01:49:30.564 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.55 MB/sec warmup 114 sec latency 12413.578 ms 2024-04-26T01:49:30.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:30 smithi006 ceph-mon[28419]: pgmap v112: 97 pgs: 97 active+clean; 38 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 285 KiB/s wr, 57 op/s 2024-04-26T01:49:30.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:30 smithi188 ceph-mon[32751]: pgmap v112: 97 pgs: 97 active+clean; 38 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 285 KiB/s wr, 57 op/s 2024-04-26T01:49:30.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:30 smithi029 ceph-mon[32775]: pgmap v112: 97 pgs: 97 active+clean; 38 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 285 KiB/s wr, 57 op/s 2024-04-26T01:49:31.564 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.51 MB/sec warmup 115 sec latency 13413.681 ms 2024-04-26T01:49:32.564 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.47 MB/sec warmup 116 sec latency 14413.831 ms 2024-04-26T01:49:32.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:32 smithi006 ceph-mon[28419]: pgmap v113: 97 pgs: 97 active+clean; 43 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 999 KiB/s wr, 79 op/s 2024-04-26T01:49:32.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:32 smithi188 ceph-mon[32751]: pgmap v113: 97 pgs: 97 active+clean; 43 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 999 KiB/s wr, 79 op/s 2024-04-26T01:49:32.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:32 smithi029 ceph-mon[32775]: pgmap v113: 97 pgs: 97 active+clean; 43 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 999 KiB/s wr, 79 op/s 2024-04-26T01:49:33.565 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.43 MB/sec warmup 117 sec latency 15413.964 ms 2024-04-26T01:49:34.565 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.39 MB/sec warmup 118 sec latency 16414.098 ms 2024-04-26T01:49:34.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:34 smithi006 ceph-mon[28419]: pgmap v114: 97 pgs: 97 active+clean; 43 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 801 KiB/s wr, 30 op/s 2024-04-26T01:49:34.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:34 smithi188 ceph-mon[32751]: pgmap v114: 97 pgs: 97 active+clean; 43 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 801 KiB/s wr, 30 op/s 2024-04-26T01:49:34.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:34 smithi029 ceph-mon[32775]: pgmap v114: 97 pgs: 97 active+clean; 43 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 801 KiB/s wr, 30 op/s 2024-04-26T01:49:35.565 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 4.36 MB/sec warmup 119 sec latency 17414.265 ms 2024-04-26T01:49:36.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:36 smithi006 ceph-mon[28419]: pgmap v115: 97 pgs: 97 active+clean; 43 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 801 KiB/s wr, 30 op/s 2024-04-26T01:49:36.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:36 smithi188 ceph-mon[32751]: pgmap v115: 97 pgs: 97 active+clean; 43 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 801 KiB/s wr, 30 op/s 2024-04-26T01:49:36.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:36 smithi029 ceph-mon[32775]: pgmap v115: 97 pgs: 97 active+clean; 43 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 801 KiB/s wr, 30 op/s 2024-04-26T01:49:37.565 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 1 sec latency 19414.534 ms 2024-04-26T01:49:38.565 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 2 sec latency 20414.678 ms 2024-04-26T01:49:38.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:38 smithi006 ceph-mon[28419]: pgmap v116: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 837 KiB/s wr, 47 op/s 2024-04-26T01:49:38.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:38 smithi188 ceph-mon[32751]: pgmap v116: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 837 KiB/s wr, 47 op/s 2024-04-26T01:49:38.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:38 smithi029 ceph-mon[32775]: pgmap v116: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 837 KiB/s wr, 47 op/s 2024-04-26T01:49:39.566 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 3 sec latency 21414.811 ms 2024-04-26T01:49:40.566 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 4 sec latency 22414.913 ms 2024-04-26T01:49:40.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:40 smithi006 ceph-mon[28419]: pgmap v117: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 750 KiB/s wr, 38 op/s 2024-04-26T01:49:40.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:40 smithi188 ceph-mon[32751]: pgmap v117: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 750 KiB/s wr, 38 op/s 2024-04-26T01:49:40.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:40 smithi029 ceph-mon[32775]: pgmap v117: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 750 KiB/s wr, 38 op/s 2024-04-26T01:49:41.566 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 5 sec latency 23415.045 ms 2024-04-26T01:49:42.566 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 6 sec latency 24415.205 ms 2024-04-26T01:49:42.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:42 smithi006 ceph-mon[28419]: pgmap v118: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 760 KiB/s wr, 42 op/s 2024-04-26T01:49:42.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:42 smithi188 ceph-mon[32751]: pgmap v118: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 760 KiB/s wr, 42 op/s 2024-04-26T01:49:42.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:42 smithi029 ceph-mon[32775]: pgmap v118: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 760 KiB/s wr, 42 op/s 2024-04-26T01:49:43.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.263+0000 7f448a200640 1 -- 172.21.15.6:0/1088780123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44841092d0 msgr2=0x7f4484110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:43.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.263+0000 7f448a200640 1 --2- 172.21.15.6:0/1088780123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44841092d0 0x7f4484110ea0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f447c009ca0 tx=0x7f447c02a8c0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.263+0000 7f448a200640 1 -- 172.21.15.6:0/1088780123 shutdown_connections 2024-04-26T01:49:43.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.263+0000 7f448a200640 1 --2- 172.21.15.6:0/1088780123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44841092d0 0x7f4484110ea0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.263+0000 7f448a200640 1 --2- 172.21.15.6:0/1088780123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4484104bb0 0x7f4484108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.263+0000 7f448a200640 1 --2- 172.21.15.6:0/1088780123 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4484104200 0x7f44841045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.263+0000 7f448a200640 1 -- 172.21.15.6:0/1088780123 >> 172.21.15.6:0/1088780123 conn(0x7f44840ffa40 msgr2=0x7f4484101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.264+0000 7f448a200640 1 -- 172.21.15.6:0/1088780123 shutdown_connections 2024-04-26T01:49:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.264+0000 7f448a200640 1 -- 172.21.15.6:0/1088780123 wait complete. 2024-04-26T01:49:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.265+0000 7f448a200640 1 Processor -- start 2024-04-26T01:49:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.265+0000 7f448a200640 1 -- start start 2024-04-26T01:49:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.266+0000 7f448a200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4484104200 0x7f44841a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.266+0000 7f448a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44841092d0 0x7f44841a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.266+0000 7f448a200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f448419a890 0x7f448419ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.266+0000 7f448a200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4484114290 con 0x7f44841092d0 2024-04-26T01:49:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.266+0000 7f448a200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4484114110 con 0x7f448419a890 2024-04-26T01:49:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.266+0000 7f448a200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4484114410 con 0x7f4484104200 2024-04-26T01:49:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.266+0000 7f4488e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4484104200 0x7f44841a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.266+0000 7f4488e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4484104200 0x7f44841a06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56432/0 (socket says 172.21.15.6:56432) 2024-04-26T01:49:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.266+0000 7f4488e00640 1 -- 172.21.15.6:0/1896995979 learned_addr learned my addr 172.21.15.6:0/1896995979 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.266+0000 7f4483e00640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44841092d0 0x7f44841a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.266+0000 7f4489800640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f448419a890 0x7f448419ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.267+0000 7f4488e00640 1 -- 172.21.15.6:0/1896995979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f448419a890 msgr2=0x7f448419ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.267+0000 7f4488e00640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f448419a890 0x7f448419ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.267+0000 7f4488e00640 1 -- 172.21.15.6:0/1896995979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44841092d0 msgr2=0x7f44841a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.267+0000 7f4488e00640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44841092d0 0x7f44841a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.267+0000 7f4488e00640 1 -- 172.21.15.6:0/1896995979 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f448419b3b0 con 0x7f4484104200 2024-04-26T01:49:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.267+0000 7f4489800640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f448419a890 0x7f448419ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.267+0000 7f4488e00640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4484104200 0x7f44841a06e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f446c00e9f0 tx=0x7f446c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.267+0000 7f4481600640 1 -- 172.21.15.6:0/1896995979 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c00cd80 con 0x7f4484104200 2024-04-26T01:49:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.267+0000 7f4481600640 1 -- 172.21.15.6:0/1896995979 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f446c0108e0 con 0x7f4484104200 2024-04-26T01:49:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.267+0000 7f448a200640 1 -- 172.21.15.6:0/1896995979 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f448419b6a0 con 0x7f4484104200 2024-04-26T01:49:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.267+0000 7f4481600640 1 -- 172.21.15.6:0/1896995979 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c018980 con 0x7f4484104200 2024-04-26T01:49:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.267+0000 7f448a200640 1 -- 172.21.15.6:0/1896995979 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44841a7940 con 0x7f4484104200 2024-04-26T01:49:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.271+0000 7f448e7b4740 1 Processor -- start 2024-04-26T01:49:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.272+0000 7f448e7b4740 1 -- start start 2024-04-26T01:49:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.272+0000 7f4481600640 1 -- 172.21.15.6:0/1896995979 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f446c010430 con 0x7f4484104200 2024-04-26T01:49:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.272+0000 7f448e7b4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 0x55bd00048cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.272+0000 7f448e7b4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd00049210 0x55bd0005be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.272+0000 7f448e7b4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd0005e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.272+0000 7f448e7b4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55bcfffae900 con 0x55bd000488f0 2024-04-26T01:49:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.272+0000 7f448e7b4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bcfff54ef0 con 0x55bd00049210 2024-04-26T01:49:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.272+0000 7f448e7b4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bcfffaa8c0 con 0x55bd0005c3a0 2024-04-26T01:49:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.273+0000 7f4489800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 0x55bd00048cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.273+0000 7f4483e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd0005e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.273+0000 7f4489800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 0x55bd00048cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36146/0 (socket says 172.21.15.6:36146) 2024-04-26T01:49:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.273+0000 7f4489800640 1 -- 172.21.15.6:0/1888461766 learned_addr learned my addr 172.21.15.6:0/1888461766 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.273+0000 7f4488e00640 1 --2- 172.21.15.6:0/1888461766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd00049210 0x55bd0005be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.273+0000 7f4483e00640 1 -- 172.21.15.6:0/1888461766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd00049210 msgr2=0x55bd0005be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.273+0000 7f4483e00640 1 --2- 172.21.15.6:0/1888461766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd00049210 0x55bd0005be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.273+0000 7f4483e00640 1 -- 172.21.15.6:0/1888461766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 msgr2=0x55bd00048cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.273+0000 7f4483e00640 1 --2- 172.21.15.6:0/1888461766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 0x55bd00048cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.273+0000 7f4483e00640 1 -- 172.21.15.6:0/1888461766 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcffcc1210 con 0x55bd0005c3a0 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.273+0000 7f4489800640 1 --2- 172.21.15.6:0/1888461766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 0x55bd00048cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.273+0000 7f4483e00640 1 --2- 172.21.15.6:0/1888461766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd0005e790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f448419c080 tx=0x7f4474011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1cf4da3daa9ceed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.274+0000 7f447a000640 1 -- 172.21.15.6:0/1888461766 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4474004050 con 0x55bd0005c3a0 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.274+0000 7f447a000640 1 -- 172.21.15.6:0/1888461766 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44740054d0 con 0x55bd0005c3a0 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.274+0000 7f447a000640 1 -- 172.21.15.6:0/1888461766 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4474004c00 con 0x55bd0005c3a0 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.274+0000 7f4481600640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f44680a3d80 0x7f44680a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.274+0000 7f4481600640 1 -- 172.21.15.6:0/1896995979 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f446c0d0c00 con 0x7f4484104200 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.274+0000 7f448e7b4740 1 -- 172.21.15.6:0/1888461766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 msgr2=0x55bd0005e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.274+0000 7f448e7b4740 1 --2- 172.21.15.6:0/1888461766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd0005e790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f448419c080 tx=0x7f4474011810 comp rx=0 tx=0).stop 2024-04-26T01:49:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.274+0000 7f4489800640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f44680a3d80 0x7f44680a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.275+0000 7f448e7b4740 1 -- 172.21.15.6:0/1888461766 shutdown_connections 2024-04-26T01:49:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.275+0000 7f448e7b4740 1 --2- 172.21.15.6:0/1888461766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd0005e790 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.275+0000 7f448e7b4740 1 --2- 172.21.15.6:0/1888461766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd00049210 0x55bd0005be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.275+0000 7f448e7b4740 1 --2- 172.21.15.6:0/1888461766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 0x55bd00048cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.275+0000 7f448e7b4740 1 -- 172.21.15.6:0/1888461766 >> 172.21.15.6:0/1888461766 conn(0x55bd00044ec0 msgr2=0x55bd000452a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.275+0000 7f448e7b4740 1 -- 172.21.15.6:0/1888461766 shutdown_connections 2024-04-26T01:49:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.275+0000 7f448e7b4740 1 -- 172.21.15.6:0/1888461766 wait complete. 2024-04-26T01:49:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.275+0000 7f448e7b4740 1 -- start start 2024-04-26T01:49:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.275+0000 7f4489800640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f44680a3d80 0x7f44680a6240 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f447c02f470 tx=0x7f447c004a40 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.275+0000 7f448e7b4740 1 Processor -- start 2024-04-26T01:49:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.276+0000 7f448e7b4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd000488f0 0x55bd00059330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.276+0000 7f448e7b4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 0x55bd00059870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.276+0000 7f448e7b4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd0005c3a0 0x55bd00059db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.276+0000 7f448e7b4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55bd0005ef70 con 0x55bd00049210 2024-04-26T01:49:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.276+0000 7f448e7b4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bcfff68310 con 0x55bd0005c3a0 2024-04-26T01:49:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.276+0000 7f448e7b4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bd0005f0f0 con 0x55bd000488f0 2024-04-26T01:49:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.276+0000 7f448e7b4740 10 client.0 _ll_register_callbacks cb 0x55bcfff36cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.276+0000 7f4483e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 0x55bd00059870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.276+0000 7f4489800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd0005c3a0 0x55bd00059db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.276+0000 7f4483e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 0x55bd00059870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36162/0 (socket says 172.21.15.6:36162) 2024-04-26T01:49:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.276+0000 7f4483e00640 1 -- 172.21.15.6:0/1803398578 learned_addr learned my addr 172.21.15.6:0/1803398578 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.276+0000 7f4488e00640 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd000488f0 0x55bd00059330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.277+0000 7f4489800640 1 -- 172.21.15.6:0/1803398578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd000488f0 msgr2=0x55bd00059330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.277+0000 7f4489800640 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd000488f0 0x55bd00059330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.277+0000 7f4489800640 1 -- 172.21.15.6:0/1803398578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 msgr2=0x55bd00059870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.277+0000 7f4489800640 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 0x55bd00059870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.277+0000 7f4489800640 1 -- 172.21.15.6:0/1803398578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcffd8d620 con 0x55bd0005c3a0 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.277+0000 7f4488e00640 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd000488f0 0x55bd00059330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.277+0000 7f4489800640 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd0005c3a0 0x55bd00059db0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f447c009ca0 tx=0x7f447c02a8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63883553f8ab149f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.277+0000 7f4472a00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.277+0000 7f4472a00640 1 -- 172.21.15.6:0/1803398578 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c02f610 con 0x55bd0005c3a0 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.277+0000 7f4472a00640 1 -- 172.21.15.6:0/1803398578 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f447c02f7b0 con 0x55bd0005c3a0 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.277+0000 7f448e7b4740 1 -- 172.21.15.6:0/1803398578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44740054d0 con 0x55bd0005c3a0 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.278+0000 7f4472a00640 1 -- 172.21.15.6:0/1803398578 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f447c02f9b0 con 0x55bd0005c3a0 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.278+0000 7f448e7b4740 10 client.25634 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.278+0000 7f448e7b4740 1 -- 172.21.15.6:0/1803398578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bd0006b250 con 0x55bd0005c3a0 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.278+0000 7f4472a00640 1 -- 172.21.15.6:0/1803398578 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44740054d0 con 0x55bd0005c3a0 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.278+0000 7f4472a00640 1 -- 172.21.15.6:0/1803398578 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f447c051070 con 0x55bd0005c3a0 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.278+0000 7f448e7b4740 10 client.25634 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.278+0000 7f448e7b4740 10 client.25634 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.278+0000 7f448e7b4740 20 client.25634 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.278+0000 7f448e7b4740 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd0006c760 0x55bd0006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.278+0000 7f448e7b4740 4 client.25634 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.278+0000 7f448e7b4740 1 -- 172.21.15.6:0/1803398578 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcfffc28b0 con 0x55bd0006c760 2024-04-26T01:49:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.279+0000 7f4488e00640 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd0006c760 0x55bd0006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.279+0000 7f4488e00640 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd0006c760 0x55bd0006ebc0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e68f0023d287222 server_cookie=672ad15cbb848c4e in_seq=0 out_seq=0 2024-04-26T01:49:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.279+0000 7f4472a00640 10 client.25634 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.280+0000 7f4472a00640 1 -- 172.21.15.6:0/1803398578 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bcfffc28b0 con 0x55bd0006c760 2024-04-26T01:49:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.280+0000 7f4472a00640 10 client.25634 handle_command_reply: tid=0 2024-04-26T01:49:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.280+0000 7f448e7b4740 1 client.25634 shutdown 2024-04-26T01:49:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.281+0000 7f448e7b4740 1 -- 172.21.15.6:0/1803398578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd0005c3a0 msgr2=0x55bd00059db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.281+0000 7f448e7b4740 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd0005c3a0 0x55bd00059db0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f447c009ca0 tx=0x7f447c02a8c0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.281+0000 7f448e7b4740 1 -- 172.21.15.6:0/1803398578 shutdown_connections 2024-04-26T01:49:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.281+0000 7f448e7b4740 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd0006c760 0x55bd0006ebc0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.281+0000 7f448e7b4740 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 0x55bd00059870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.281+0000 7f448e7b4740 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd0005c3a0 0x55bd00059db0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.281+0000 7f448e7b4740 1 --2- 172.21.15.6:0/1803398578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd000488f0 0x55bd00059330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.281+0000 7f4472a00640 0 client.25634 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.281+0000 7f448e7b4740 1 -- 172.21.15.6:0/1803398578 >> 172.21.15.6:0/1803398578 conn(0x55bd00044ec0 msgr2=0x55bd0004ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.281+0000 7f448e7b4740 1 -- 172.21.15.6:0/1803398578 shutdown_connections 2024-04-26T01:49:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.281+0000 7f448e7b4740 1 -- 172.21.15.6:0/1803398578 wait complete. 2024-04-26T01:49:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.281+0000 7f448e7b4740 20 client.25634 trim_cache size 0 max 16384 2024-04-26T01:49:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.297+0000 7f448e7b4740 1 Processor -- start 2024-04-26T01:49:43.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.297+0000 7f448e7b4740 1 -- start start 2024-04-26T01:49:43.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f448e7b4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 0x55bd00067e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f448e7b4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd00049210 0x55bd00043ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f448e7b4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd000744d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f448e7b4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f44841ae260 con 0x55bd000488f0 2024-04-26T01:49:43.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f448e7b4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f44841abad0 con 0x55bd00049210 2024-04-26T01:49:43.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f448e7b4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f44841a9920 con 0x55bd0005c3a0 2024-04-26T01:49:43.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f4488e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 0x55bd00067e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f4488e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 0x55bd00067e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36170/0 (socket says 172.21.15.6:36170) 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f4488e00640 1 -- 172.21.15.6:0/2762775306 learned_addr learned my addr 172.21.15.6:0/2762775306 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f4483e00640 1 --2- 172.21.15.6:0/2762775306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd00049210 0x55bd00043ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f4489800640 1 --2- 172.21.15.6:0/2762775306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd000744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f4488e00640 1 -- 172.21.15.6:0/2762775306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 msgr2=0x55bd000744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f4488e00640 1 --2- 172.21.15.6:0/2762775306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd000744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f4488e00640 1 -- 172.21.15.6:0/2762775306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd00049210 msgr2=0x55bd00043ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f4488e00640 1 --2- 172.21.15.6:0/2762775306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd00049210 0x55bd00043ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.298+0000 7f4488e00640 1 -- 172.21.15.6:0/2762775306 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd00049000 con 0x55bd000488f0 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.299+0000 7f4489800640 1 --2- 172.21.15.6:0/2762775306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd000744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.299+0000 7f4488e00640 1 --2- 172.21.15.6:0/2762775306 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 0x55bd00067e70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f446c0994a0 tx=0x7f446c0bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d804cf10eabba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.299+0000 7f4479600640 1 -- 172.21.15.6:0/2762775306 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c010720 con 0x55bd000488f0 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.299+0000 7f4479600640 1 -- 172.21.15.6:0/2762775306 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f446c09de50 con 0x55bd000488f0 2024-04-26T01:49:43.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.299+0000 7f4479600640 1 -- 172.21.15.6:0/2762775306 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c0bd0b0 con 0x55bd000488f0 2024-04-26T01:49:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.300+0000 7f448e7b4740 1 -- 172.21.15.6:0/2762775306 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 msgr2=0x55bd00067e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.300+0000 7f448e7b4740 1 --2- 172.21.15.6:0/2762775306 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 0x55bd00067e70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f446c0994a0 tx=0x7f446c0bd040 comp rx=0 tx=0).stop 2024-04-26T01:49:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.300+0000 7f448e7b4740 1 -- 172.21.15.6:0/2762775306 shutdown_connections 2024-04-26T01:49:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.300+0000 7f448e7b4740 1 --2- 172.21.15.6:0/2762775306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd000744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.300+0000 7f448e7b4740 1 --2- 172.21.15.6:0/2762775306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd00049210 0x55bd00043ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.300+0000 7f448e7b4740 1 --2- 172.21.15.6:0/2762775306 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd000488f0 0x55bd00067e70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.300+0000 7f448e7b4740 1 -- 172.21.15.6:0/2762775306 >> 172.21.15.6:0/2762775306 conn(0x55bd00044ec0 msgr2=0x55bd00043790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.300+0000 7f448e7b4740 1 -- 172.21.15.6:0/2762775306 shutdown_connections 2024-04-26T01:49:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.300+0000 7f448e7b4740 1 -- 172.21.15.6:0/2762775306 wait complete. 2024-04-26T01:49:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.300+0000 7f448e7b4740 1 -- start start 2024-04-26T01:49:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.300+0000 7f448e7b4740 1 Processor -- start 2024-04-26T01:49:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.301+0000 7f448e7b4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd000488f0 0x55bd00061d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.301+0000 7f448e7b4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 0x55bd00062250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.301+0000 7f448e7b4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd00062790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.301+0000 7f448e7b4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f44841a9f90 con 0x55bd00049210 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.301+0000 7f448e7b4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f44841ab400 con 0x55bd000488f0 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.301+0000 7f448e7b4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f44841aa660 con 0x55bd0005c3a0 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.301+0000 7f448e7b4740 10 client.0 _ll_register_callbacks cb 0x7f447c031eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.301+0000 7f4483e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 0x55bd00062250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.301+0000 7f4483e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 0x55bd00062250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36186/0 (socket says 172.21.15.6:36186) 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.301+0000 7f4483e00640 1 -- 172.21.15.6:0/3718155069 learned_addr learned my addr 172.21.15.6:0/3718155069 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.301+0000 7f4488e00640 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd000488f0 0x55bd00061d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.302+0000 7f4489800640 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd00062790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.302+0000 7f4483e00640 1 -- 172.21.15.6:0/3718155069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 msgr2=0x55bd00062790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.302+0000 7f4483e00640 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd00062790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.302+0000 7f4483e00640 1 -- 172.21.15.6:0/3718155069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd000488f0 msgr2=0x55bd00061d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.302+0000 7f4483e00640 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd000488f0 0x55bd00061d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.302+0000 7f4483e00640 1 -- 172.21.15.6:0/3718155069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd000d1d50 con 0x55bd00049210 2024-04-26T01:49:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.302+0000 7f4483e00640 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 0x55bd00062250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4474009250 tx=0x7f4474005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b2efa5a05e630e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.302+0000 7f4472a00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.302+0000 7f4472a00640 1 -- 172.21.15.6:0/3718155069 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447401d040 con 0x55bd00049210 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.302+0000 7f448e7b4740 1 -- 172.21.15.6:0/3718155069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f446c09de50 con 0x55bd00049210 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.302+0000 7f4472a00640 1 -- 172.21.15.6:0/3718155069 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f4474009d50 con 0x55bd00049210 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.303+0000 7f4472a00640 1 -- 172.21.15.6:0/3718155069 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f44740035f0 con 0x55bd00049210 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.303+0000 7f448e7b4740 10 client.15516 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.303+0000 7f448e7b4740 1 -- 172.21.15.6:0/3718155069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bd000d2fa0 con 0x55bd00049210 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.303+0000 7f4472a00640 1 -- 172.21.15.6:0/3718155069 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f446c09de50 con 0x55bd00049210 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.303+0000 7f4472a00640 1 -- 172.21.15.6:0/3718155069 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f447402b020 con 0x55bd00049210 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.303+0000 7f448e7b4740 10 client.15516 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.303+0000 7f448e7b4740 10 client.15516 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.303+0000 7f448e7b4740 20 client.15516 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.303+0000 7f448e7b4740 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd000d34d0 0x55bd000d58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.303+0000 7f448e7b4740 4 client.15516 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.303+0000 7f448e7b4740 1 -- 172.21.15.6:0/3718155069 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bd00054f30 con 0x55bd000d34d0 2024-04-26T01:49:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.304+0000 7f4488e00640 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd000d34d0 0x55bd000d58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.304+0000 7f4488e00640 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd000d34d0 0x55bd000d58f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20a23cacf3f27422 server_cookie=50d4751415546d4a in_seq=0 out_seq=0 2024-04-26T01:49:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.304+0000 7f4472a00640 10 client.15516 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f4472a00640 1 -- 172.21.15.6:0/3718155069 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bd00054f30 con 0x55bd000d34d0 2024-04-26T01:49:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f4472a00640 10 client.15516 handle_command_reply: tid=0 2024-04-26T01:49:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f448e7b4740 1 client.15516 shutdown 2024-04-26T01:49:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f448e7b4740 1 -- 172.21.15.6:0/3718155069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 msgr2=0x55bd00062250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f448e7b4740 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 0x55bd00062250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4474009250 tx=0x7f4474005c50 comp rx=0 tx=0).stop 2024-04-26T01:49:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f448e7b4740 1 -- 172.21.15.6:0/3718155069 shutdown_connections 2024-04-26T01:49:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f448e7b4740 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd000d34d0 0x55bd000d58f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f448e7b4740 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd00049210 0x55bd00062250 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f448e7b4740 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd0005c3a0 0x55bd00062790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f448e7b4740 1 --2- 172.21.15.6:0/3718155069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd000488f0 0x55bd00061d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f4472a00640 0 client.15516 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f448e7b4740 1 -- 172.21.15.6:0/3718155069 >> 172.21.15.6:0/3718155069 conn(0x55bd00044ec0 msgr2=0x55bd00059b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f448e7b4740 1 -- 172.21.15.6:0/3718155069 shutdown_connections 2024-04-26T01:49:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.326+0000 7f448e7b4740 1 -- 172.21.15.6:0/3718155069 wait complete. 2024-04-26T01:49:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.327+0000 7f448e7b4740 20 client.15516 trim_cache size 0 max 16384 2024-04-26T01:49:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:49:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:49:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:49:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:49:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.327+0000 7f448a200640 1 -- 172.21.15.6:0/1896995979 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f44680a3d80 msgr2=0x7f44680a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.327+0000 7f448a200640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f44680a3d80 0x7f44680a6240 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f447c02f470 tx=0x7f447c004a40 comp rx=0 tx=0).stop 2024-04-26T01:49:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.327+0000 7f448a200640 1 -- 172.21.15.6:0/1896995979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4484104200 msgr2=0x7f44841a06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.327+0000 7f448a200640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4484104200 0x7f44841a06e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f446c00e9f0 tx=0x7f446c00eeb0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.328+0000 7f448a200640 1 -- 172.21.15.6:0/1896995979 shutdown_connections 2024-04-26T01:49:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.328+0000 7f448a200640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f44680a3d80 0x7f44680a6240 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.328+0000 7f448a200640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44841092d0 0x7f44841a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.328+0000 7f448a200640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f448419a890 0x7f448419ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.328+0000 7f448a200640 1 --2- 172.21.15.6:0/1896995979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4484104200 0x7f44841a06e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.328+0000 7f448a200640 1 -- 172.21.15.6:0/1896995979 >> 172.21.15.6:0/1896995979 conn(0x7f44840ffa40 msgr2=0x7f4484101da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.328+0000 7f448a200640 1 -- 172.21.15.6:0/1896995979 shutdown_connections 2024-04-26T01:49:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:43.328+0000 7f448a200640 1 -- 172.21.15.6:0/1896995979 wait complete. 2024-04-26T01:49:43.566 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 7 sec latency 25415.356 ms 2024-04-26T01:49:44.566 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 8 sec latency 26415.505 ms 2024-04-26T01:49:44.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:44 smithi006 ceph-mon[28419]: pgmap v119: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 46 KiB/s wr, 20 op/s 2024-04-26T01:49:44.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:44 smithi188 ceph-mon[32751]: pgmap v119: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 46 KiB/s wr, 20 op/s 2024-04-26T01:49:44.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:44 smithi029 ceph-mon[32775]: pgmap v119: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 46 KiB/s wr, 20 op/s 2024-04-26T01:49:45.566 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 9 sec latency 27415.629 ms 2024-04-26T01:49:46.566 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 10 sec latency 28415.795 ms 2024-04-26T01:49:46.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:46 smithi006 ceph-mon[28419]: pgmap v120: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 46 KiB/s wr, 20 op/s 2024-04-26T01:49:46.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:46 smithi188 ceph-mon[32751]: pgmap v120: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 46 KiB/s wr, 20 op/s 2024-04-26T01:49:46.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:46 smithi029 ceph-mon[32775]: pgmap v120: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 46 KiB/s wr, 20 op/s 2024-04-26T01:49:47.566 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 11 sec latency 29415.928 ms 2024-04-26T01:49:48.567 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 12 sec latency 30416.044 ms 2024-04-26T01:49:48.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:48 smithi006 ceph-mon[28419]: pgmap v121: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 56 KiB/s wr, 27 op/s 2024-04-26T01:49:48.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:48 smithi188 ceph-mon[32751]: pgmap v121: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 56 KiB/s wr, 27 op/s 2024-04-26T01:49:48.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:48 smithi029 ceph-mon[32775]: pgmap v121: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 56 KiB/s wr, 27 op/s 2024-04-26T01:49:49.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:49:49.517 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:49:49.566 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 13 sec latency 31416.114 ms 2024-04-26T01:49:50.567 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 14 sec latency 32416.190 ms 2024-04-26T01:49:50.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:50 smithi006 ceph-mon[28419]: pgmap v122: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-04-26T01:49:50.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:50 smithi188 ceph-mon[32751]: pgmap v122: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-04-26T01:49:50.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:50 smithi029 ceph-mon[32775]: pgmap v122: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-04-26T01:49:51.567 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 15 sec latency 33416.392 ms 2024-04-26T01:49:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.927+0000 7f3569800700 1 -- 172.21.15.6:0/4273046685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3564074c70 msgr2=0x7f3564075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.927+0000 7f3569800700 1 --2- 172.21.15.6:0/4273046685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3564074c70 0x7f3564075050 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f35540098f0 tx=0x7f3554009c00 comp rx=0 tx=0).stop 2024-04-26T01:49:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.928+0000 7f3569800700 1 -- 172.21.15.6:0/4273046685 shutdown_connections 2024-04-26T01:49:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.928+0000 7f3569800700 1 --2- 172.21.15.6:0/4273046685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f356410f4a0 0x7f3564111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.928+0000 7f3569800700 1 --2- 172.21.15.6:0/4273046685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3564074c70 0x7f3564075050 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.928+0000 7f3569800700 1 --2- 172.21.15.6:0/4273046685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3564075590 0x7f356410ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.928+0000 7f3569800700 1 -- 172.21.15.6:0/4273046685 >> 172.21.15.6:0/4273046685 conn(0x7f3564079ba0 msgr2=0x7f3564079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.928+0000 7f3569800700 1 -- 172.21.15.6:0/4273046685 shutdown_connections 2024-04-26T01:49:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.928+0000 7f3569800700 1 -- 172.21.15.6:0/4273046685 wait complete. 2024-04-26T01:49:51.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.929+0000 7f3569800700 1 Processor -- start 2024-04-26T01:49:51.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.929+0000 7f3569800700 1 -- start start 2024-04-26T01:49:51.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.930+0000 7f3569800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3564074c70 0x7f3564102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:51.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.930+0000 7f3569800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3564075590 0x7f3564103100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:51.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.930+0000 7f3569800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f356410f4a0 0x7f3564103640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:51.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.930+0000 7f3569800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3564114eb0 con 0x7f356410f4a0 2024-04-26T01:49:51.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.930+0000 7f3569800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3564114d30 con 0x7f3564074c70 2024-04-26T01:49:51.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.930+0000 7f3569800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3564115030 con 0x7f3564075590 2024-04-26T01:49:51.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.930+0000 7f3562a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3564074c70 0x7f3564102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:51.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.930+0000 7f3562a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3564074c70 0x7f3564102bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:48254/0 (socket says 172.21.15.6:48254) 2024-04-26T01:49:51.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.930+0000 7f3562000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3564075590 0x7f3564103100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:51.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.930+0000 7f3562a00700 1 -- 172.21.15.6:0/2376362010 learned_addr learned my addr 172.21.15.6:0/2376362010 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:51.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.931+0000 7f3563400700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f356410f4a0 0x7f3564103640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:51.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.931+0000 7f3562a00700 1 -- 172.21.15.6:0/2376362010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3564075590 msgr2=0x7f3564103100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:51.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.931+0000 7f3562a00700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3564075590 0x7f3564103100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:51.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.931+0000 7f3562a00700 1 -- 172.21.15.6:0/2376362010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f356410f4a0 msgr2=0x7f3564103640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:51.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.931+0000 7f3562a00700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f356410f4a0 0x7f3564103640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:51.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.931+0000 7f3562a00700 1 -- 172.21.15.6:0/2376362010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35641aba10 con 0x7f3564074c70 2024-04-26T01:49:51.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.931+0000 7f3562a00700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3564074c70 0x7f3564102bc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f355400fa70 tx=0x7f355400faa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:51.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.931+0000 7f355b400700 1 -- 172.21.15.6:0/2376362010 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3554025070 con 0x7f3564074c70 2024-04-26T01:49:51.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.932+0000 7f355b400700 1 -- 172.21.15.6:0/2376362010 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f355400b560 con 0x7f3564074c70 2024-04-26T01:49:51.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.932+0000 7f3569800700 1 -- 172.21.15.6:0/2376362010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35641abca0 con 0x7f3564074c70 2024-04-26T01:49:51.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.932+0000 7f355b400700 1 -- 172.21.15.6:0/2376362010 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3554003b20 con 0x7f3564074c70 2024-04-26T01:49:51.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.932+0000 7f3569800700 1 -- 172.21.15.6:0/2376362010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35641ac150 con 0x7f3564074c70 2024-04-26T01:49:51.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.933+0000 7f3569800700 1 -- 172.21.15.6:0/2376362010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3564076350 con 0x7f3564074c70 2024-04-26T01:49:51.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.935+0000 7f355b400700 1 -- 172.21.15.6:0/2376362010 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f3554028070 con 0x7f3564074c70 2024-04-26T01:49:51.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.936+0000 7f355b400700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f35480a3e30 0x7f35480a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:51.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.937+0000 7f355b400700 1 -- 172.21.15.6:0/2376362010 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f35540d0f80 con 0x7f3564074c70 2024-04-26T01:49:51.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.937+0000 7f3562000700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f35480a3e30 0x7f35480a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:51.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.937+0000 7f3562000700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f35480a3e30 0x7f35480a62f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3564104380 tx=0x7f3550009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:51.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:51.941+0000 7f355b400700 1 -- 172.21.15.6:0/2376362010 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3554013030 con 0x7f3564074c70 2024-04-26T01:49:52.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.223+0000 7f3569800700 1 -- 172.21.15.6:0/2376362010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f356402e140 con 0x7f3564074c70 2024-04-26T01:49:52.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.224+0000 7f355b400700 1 -- 172.21.15.6:0/2376362010 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11853 (secure 0 0 0) 0x7f355409d630 con 0x7f3564074c70 2024-04-26T01:49:52.222 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:49:52.222 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:47:59.436110+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25474":{"gid":25474,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1633748308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1633748308},{"type":"v1","addr":"172.21.15.188:6835","nonce":1633748308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:49:52.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.227+0000 7f3569800700 1 -- 172.21.15.6:0/2376362010 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f35480a3e30 msgr2=0x7f35480a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.227+0000 7f3569800700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f35480a3e30 0x7f35480a62f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3564104380 tx=0x7f3550009260 comp rx=0 tx=0).stop 2024-04-26T01:49:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.227+0000 7f3569800700 1 -- 172.21.15.6:0/2376362010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3564074c70 msgr2=0x7f3564102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.227+0000 7f3569800700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3564074c70 0x7f3564102bc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f355400fa70 tx=0x7f355400faa0 comp rx=0 tx=0).stop 2024-04-26T01:49:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.227+0000 7f3569800700 1 -- 172.21.15.6:0/2376362010 shutdown_connections 2024-04-26T01:49:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.227+0000 7f3569800700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f35480a3e30 0x7f35480a62f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.227+0000 7f3569800700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f356410f4a0 0x7f3564103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:52.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.227+0000 7f3569800700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3564074c70 0x7f3564102bc0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:52.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.227+0000 7f3569800700 1 --2- 172.21.15.6:0/2376362010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3564075590 0x7f3564103100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:52.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.227+0000 7f3569800700 1 -- 172.21.15.6:0/2376362010 >> 172.21.15.6:0/2376362010 conn(0x7f3564079ba0 msgr2=0x7f35640ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:52.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.228+0000 7f3569800700 1 -- 172.21.15.6:0/2376362010 shutdown_connections 2024-04-26T01:49:52.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:52.228+0000 7f3569800700 1 -- 172.21.15.6:0/2376362010 wait complete. 2024-04-26T01:49:52.227 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 31 2024-04-26T01:49:52.567 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 16 sec latency 34416.495 ms 2024-04-26T01:49:52.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:52 smithi006 ceph-mon[28419]: pgmap v123: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-04-26T01:49:52.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:52 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2376362010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:49:52.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:52 smithi188 ceph-mon[32751]: pgmap v123: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-04-26T01:49:52.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:52 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2376362010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:49:52.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:52 smithi029 ceph-mon[32775]: pgmap v123: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-04-26T01:49:52.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:52 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2376362010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:49:53.104 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id dcc36de --release --await' 2024-04-26T01:49:53.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id dcc36de --release --await 2024-04-26T01:49:53.541 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:49:53.567 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 17 sec latency 35416.573 ms 2024-04-26T01:49:54.567 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 18 sec latency 36416.656 ms 2024-04-26T01:49:54.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:54 smithi006 ceph-mon[28419]: pgmap v124: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-26T01:49:54.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:54 smithi188 ceph-mon[32751]: pgmap v124: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-26T01:49:54.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:54 smithi029 ceph-mon[32775]: pgmap v124: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-26T01:49:55.567 INFO:tasks.workunit.client.0.smithi006.stdout: 1 92978 0.00 MB/sec execute 19 sec latency 37416.821 ms 2024-04-26T01:49:55.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.776+0000 7f6498c00700 1 -- 172.21.15.6:0/3361351676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6494108a50 msgr2=0x7f6494108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:55.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.776+0000 7f6498c00700 1 --2- 172.21.15.6:0/3361351676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6494108a50 0x7f6494108e30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f64840098f0 tx=0x7f6484009c00 comp rx=0 tx=0).stop 2024-04-26T01:49:55.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.776+0000 7f6498c00700 1 -- 172.21.15.6:0/3361351676 shutdown_connections 2024-04-26T01:49:55.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.776+0000 7f6498c00700 1 --2- 172.21.15.6:0/3361351676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6494103450 0x7f649410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.776+0000 7f6498c00700 1 --2- 172.21.15.6:0/3361351676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6494108a50 0x7f6494108e30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.776+0000 7f6498c00700 1 --2- 172.21.15.6:0/3361351676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6494102a90 0x7f6494102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.776+0000 7f6498c00700 1 -- 172.21.15.6:0/3361351676 >> 172.21.15.6:0/3361351676 conn(0x7f6494079ba0 msgr2=0x7f6494079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:55.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.776+0000 7f6498c00700 1 -- 172.21.15.6:0/3361351676 shutdown_connections 2024-04-26T01:49:55.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.776+0000 7f6498c00700 1 -- 172.21.15.6:0/3361351676 wait complete. 2024-04-26T01:49:55.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.777+0000 7f6498c00700 1 Processor -- start 2024-04-26T01:49:55.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.777+0000 7f6498c00700 1 -- start start 2024-04-26T01:49:55.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.778+0000 7f6498c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6494102a90 0x7f649419b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.778+0000 7f6498c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6494103450 0x7f649419b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.778+0000 7f6498c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6494108a50 0x7f649419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.778+0000 7f6498c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6494112c60 con 0x7f6494103450 2024-04-26T01:49:55.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.778+0000 7f6498c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6494112ae0 con 0x7f6494108a50 2024-04-26T01:49:55.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.778+0000 7f6498c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6494112de0 con 0x7f6494102a90 2024-04-26T01:49:55.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.778+0000 7f6492800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6494108a50 0x7f649419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.778+0000 7f6491400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6494103450 0x7f649419b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.778+0000 7f6492800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6494108a50 0x7f649419fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:48274/0 (socket says 172.21.15.6:48274) 2024-04-26T01:49:55.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.778+0000 7f6492800700 1 -- 172.21.15.6:0/2335455204 learned_addr learned my addr 172.21.15.6:0/2335455204 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:55.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.778+0000 7f6491e00700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6494102a90 0x7f649419b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.779+0000 7f6491e00700 1 -- 172.21.15.6:0/2335455204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6494108a50 msgr2=0x7f649419fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:55.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.779+0000 7f6491e00700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6494108a50 0x7f649419fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.779+0000 7f6491e00700 1 -- 172.21.15.6:0/2335455204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6494103450 msgr2=0x7f649419b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:55.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.779+0000 7f6491e00700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6494103450 0x7f649419b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.779+0000 7f6491e00700 1 -- 172.21.15.6:0/2335455204 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64941a03b0 con 0x7f6494102a90 2024-04-26T01:49:55.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.779+0000 7f6491e00700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6494102a90 0x7f649419b3f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f648400c010 tx=0x7f648400fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:55.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.779+0000 7f648aa00700 1 -- 172.21.15.6:0/2335455204 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64840050c0 con 0x7f6494102a90 2024-04-26T01:49:55.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.779+0000 7f648aa00700 1 -- 172.21.15.6:0/2335455204 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f648400b560 con 0x7f6494102a90 2024-04-26T01:49:55.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.779+0000 7f648aa00700 1 -- 172.21.15.6:0/2335455204 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484003a00 con 0x7f6494102a90 2024-04-26T01:49:55.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.780+0000 7f6498c00700 1 -- 172.21.15.6:0/2335455204 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64941a0640 con 0x7f6494102a90 2024-04-26T01:49:55.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.780+0000 7f6498c00700 1 -- 172.21.15.6:0/2335455204 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64941a7f20 con 0x7f6494102a90 2024-04-26T01:49:55.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.783+0000 7f648aa00700 1 -- 172.21.15.6:0/2335455204 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f648400b710 con 0x7f6494102a90 2024-04-26T01:49:55.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.784+0000 7f648aa00700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f64780a3e30 0x7f64780a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.784+0000 7f64a361bb80 1 Processor -- start 2024-04-26T01:49:55.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.784+0000 7f648aa00700 1 -- 172.21.15.6:0/2335455204 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f64840d1580 con 0x7f6494102a90 2024-04-26T01:49:55.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.785+0000 7f64a361bb80 1 -- start start 2024-04-26T01:49:55.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.785+0000 7f6491400700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f64780a3e30 0x7f64780a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.785+0000 7f64a361bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae8160d40 0x555ae8161120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.785+0000 7f64a361bb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8161660 0x555ae816bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.785+0000 7f64a361bb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae816c4d0 0x555ae816e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.785+0000 7f64a361bb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555ae80b3f90 con 0x555ae816c4d0 2024-04-26T01:49:55.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.785+0000 7f64a361bb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555ae805a830 con 0x555ae8161660 2024-04-26T01:49:55.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.785+0000 7f64a361bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555ae80aff50 con 0x555ae8160d40 2024-04-26T01:49:55.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.786+0000 7f6491e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae8160d40 0x555ae8161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.786+0000 7f6491e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae8160d40 0x555ae8161120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56780/0 (socket says 172.21.15.6:56780) 2024-04-26T01:49:55.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.786+0000 7f6491e00700 1 -- 172.21.15.6:0/937046329 learned_addr learned my addr 172.21.15.6:0/937046329 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:55.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.786+0000 7f6492800700 1 --2- 172.21.15.6:0/937046329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae816c4d0 0x555ae816e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.786+0000 7f6491400700 1 --2- 172.21.15.6:0/937046329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8161660 0x555ae816bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.786+0000 7f6491400700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f64780a3e30 0x7f64780a62f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6480009750 tx=0x7f6480009310 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:55.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.786+0000 7f6491e00700 1 -- 172.21.15.6:0/937046329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8161660 msgr2=0x555ae816bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:55.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.786+0000 7f6491e00700 1 --2- 172.21.15.6:0/937046329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8161660 0x555ae816bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.786+0000 7f6491e00700 1 -- 172.21.15.6:0/937046329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae816c4d0 msgr2=0x555ae816e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:55.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.786+0000 7f6491e00700 1 --2- 172.21.15.6:0/937046329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae816c4d0 0x555ae816e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.786+0000 7f6491e00700 1 -- 172.21.15.6:0/937046329 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ae8050780 con 0x555ae8160d40 2024-04-26T01:49:55.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.786+0000 7f6491e00700 1 --2- 172.21.15.6:0/937046329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae8160d40 0x555ae8161120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f64840098c0 tx=0x7f648409d0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51225ca38957cb95 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:55.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.787+0000 7f647f400700 1 -- 172.21.15.6:0/937046329 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64840b6070 con 0x555ae8160d40 2024-04-26T01:49:55.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.787+0000 7f647f400700 1 -- 172.21.15.6:0/937046329 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f648409dc30 con 0x555ae8160d40 2024-04-26T01:49:55.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.787+0000 7f647f400700 1 -- 172.21.15.6:0/937046329 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64840ceed0 con 0x555ae8160d40 2024-04-26T01:49:55.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.788+0000 7f64a361bb80 1 -- 172.21.15.6:0/937046329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae8160d40 msgr2=0x555ae8161120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:55.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.788+0000 7f64a361bb80 1 --2- 172.21.15.6:0/937046329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae8160d40 0x555ae8161120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f64840098c0 tx=0x7f648409d0f0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.788+0000 7f64a361bb80 1 -- 172.21.15.6:0/937046329 shutdown_connections 2024-04-26T01:49:55.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.788+0000 7f64a361bb80 1 --2- 172.21.15.6:0/937046329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae816c4d0 0x555ae816e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.788+0000 7f64a361bb80 1 --2- 172.21.15.6:0/937046329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae8160d40 0x555ae8161120 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.788+0000 7f64a361bb80 1 --2- 172.21.15.6:0/937046329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8161660 0x555ae816bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.788+0000 7f64a361bb80 1 -- 172.21.15.6:0/937046329 >> 172.21.15.6:0/937046329 conn(0x555ae8090050 msgr2=0x555ae815d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:55.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.788+0000 7f64a361bb80 1 -- 172.21.15.6:0/937046329 shutdown_connections 2024-04-26T01:49:55.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.788+0000 7f64a361bb80 1 -- 172.21.15.6:0/937046329 wait complete. 2024-04-26T01:49:55.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.788+0000 7f64a361bb80 1 -- start start 2024-04-26T01:49:55.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.789+0000 7f64a361bb80 1 Processor -- start 2024-04-26T01:49:55.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.789+0000 7f64a361bb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8160d40 0x555ae816bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.789+0000 7f64a361bb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 0x555ae8168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.789+0000 7f64a361bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae8169420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.789+0000 7f64a361bb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555ae816ee00 con 0x555ae8161660 2024-04-26T01:49:55.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.789+0000 7f64a361bb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555ae806d960 con 0x555ae8160d40 2024-04-26T01:49:55.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.789+0000 7f64a361bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555ae816ef80 con 0x555ae816c4d0 2024-04-26T01:49:55.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.789+0000 7f64a361bb80 10 client.0 _ll_register_callbacks cb 0x555ae804c820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:55.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.790+0000 7f6491400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 0x555ae8168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.790+0000 7f6491400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 0x555ae8168ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55584/0 (socket says 172.21.15.6:55584) 2024-04-26T01:49:55.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.790+0000 7f6491400700 1 -- 172.21.15.6:0/1722466949 learned_addr learned my addr 172.21.15.6:0/1722466949 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:55.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.790+0000 7f6492800700 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae8169420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.790+0000 7f6491e00700 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8160d40 0x555ae816bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.790+0000 7f6491400700 1 -- 172.21.15.6:0/1722466949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 msgr2=0x555ae8169420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:55.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.790+0000 7f6491400700 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae8169420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.790+0000 7f6491400700 1 -- 172.21.15.6:0/1722466949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8160d40 msgr2=0x555ae816bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:55.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.790+0000 7f6491400700 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8160d40 0x555ae816bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.790+0000 7f6491400700 1 -- 172.21.15.6:0/1722466949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ae80019d0 con 0x555ae8161660 2024-04-26T01:49:55.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.791+0000 7f6491400700 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 0x555ae8168ee0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f648001dc00 tx=0x7f64800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4851f2d9d6f6d026 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:55.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.791+0000 7f6473e00700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:49:55.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.791+0000 7f6473e00700 1 -- 172.21.15.6:0/1722466949 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f648002b060 con 0x555ae8161660 2024-04-26T01:49:55.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.791+0000 7f6473e00700 1 -- 172.21.15.6:0/1722466949 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f648001cae0 con 0x555ae8161660 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.791+0000 7f64a361bb80 1 -- 172.21.15.6:0/1722466949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f648409dc30 con 0x555ae8161660 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.791+0000 7f6473e00700 1 -- 172.21.15.6:0/1722466949 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f648001cce0 con 0x555ae8161660 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.791+0000 7f64a361bb80 10 client.15534 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.791+0000 7f64a361bb80 1 -- 172.21.15.6:0/1722466949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555ae804abf0 con 0x555ae8161660 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.792+0000 7f6473e00700 1 -- 172.21.15.6:0/1722466949 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f648409dc30 con 0x555ae8161660 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.792+0000 7f6473e00700 1 -- 172.21.15.6:0/1722466949 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f6480003b00 con 0x555ae8161660 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.792+0000 7f64a361bb80 10 client.15534 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.792+0000 7f64a361bb80 10 client.15534 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.792+0000 7f64a361bb80 20 client.15534 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.792+0000 7f64a361bb80 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ae817ba60 0x555ae817de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.792+0000 7f64a361bb80 4 client.15534 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.792+0000 7f64a361bb80 1 -- 172.21.15.6:0/1722466949 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ae80c1780 con 0x555ae817ba60 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.792+0000 7f6491e00700 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ae817ba60 0x555ae817de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.793+0000 7f6491e00700 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ae817ba60 0x555ae817de50 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5245caa17ee0e11 server_cookie=328f5c5b79b43c1c in_seq=0 out_seq=0 2024-04-26T01:49:55.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.793+0000 7f6473e00700 10 client.15534 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:55.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.794+0000 7f6473e00700 1 -- 172.21.15.6:0/1722466949 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555ae80c1780 con 0x555ae817ba60 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.794+0000 7f6473e00700 10 client.15534 handle_command_reply: tid=0 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.794+0000 7f64a361bb80 1 client.15534 shutdown 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.794+0000 7f64a361bb80 1 -- 172.21.15.6:0/1722466949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 msgr2=0x555ae8168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.794+0000 7f64a361bb80 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 0x555ae8168ee0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f648001dc00 tx=0x7f64800047c0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.794+0000 7f64a361bb80 1 -- 172.21.15.6:0/1722466949 shutdown_connections 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.794+0000 7f64a361bb80 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ae817ba60 0x555ae817de50 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.794+0000 7f64a361bb80 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae8169420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.794+0000 7f64a361bb80 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8160d40 0x555ae816bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.794+0000 7f64a361bb80 1 --2- 172.21.15.6:0/1722466949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 0x555ae8168ee0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.794+0000 7f64a361bb80 1 -- 172.21.15.6:0/1722466949 >> 172.21.15.6:0/1722466949 conn(0x555ae8090050 msgr2=0x555ae815ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.794+0000 7f6473e00700 0 client.15534 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.795+0000 7f64a361bb80 1 -- 172.21.15.6:0/1722466949 shutdown_connections 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.795+0000 7f64a361bb80 1 -- 172.21.15.6:0/1722466949 wait complete. 2024-04-26T01:49:55.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.795+0000 7f64a361bb80 20 client.15534 trim_cache size 0 max 16384 2024-04-26T01:49:55.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.816+0000 7f64a361bb80 1 Processor -- start 2024-04-26T01:49:55.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.817+0000 7f64a361bb80 1 -- start start 2024-04-26T01:49:55.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.817+0000 7f64a361bb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8160d40 0x555ae81836e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.817+0000 7f64a361bb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8161660 0x555ae8183c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.817+0000 7f64a361bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae81740b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.817+0000 7f64a361bb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f64941b1f50 con 0x555ae8160d40 2024-04-26T01:49:55.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.817+0000 7f64a361bb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f64941af780 con 0x555ae8161660 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.817+0000 7f64a361bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f64941ad5d0 con 0x555ae816c4d0 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.817+0000 7f6491e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8160d40 0x555ae81836e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.817+0000 7f6491e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8160d40 0x555ae81836e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55600/0 (socket says 172.21.15.6:55600) 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.817+0000 7f6492800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae81740b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.817+0000 7f6491e00700 1 -- 172.21.15.6:0/1623270766 learned_addr learned my addr 172.21.15.6:0/1623270766 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.818+0000 7f6491400700 1 --2- 172.21.15.6:0/1623270766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8161660 0x555ae8183c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.818+0000 7f6492800700 1 -- 172.21.15.6:0/1623270766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8161660 msgr2=0x555ae8183c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.818+0000 7f6492800700 1 --2- 172.21.15.6:0/1623270766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8161660 0x555ae8183c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.818+0000 7f6492800700 1 -- 172.21.15.6:0/1623270766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8160d40 msgr2=0x555ae81836e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.818+0000 7f6492800700 1 --2- 172.21.15.6:0/1623270766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8160d40 0x555ae81836e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.818+0000 7f6492800700 1 -- 172.21.15.6:0/1623270766 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ae8162bb0 con 0x555ae816c4d0 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.818+0000 7f6491400700 1 --2- 172.21.15.6:0/1623270766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8161660 0x555ae8183c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.818+0000 7f6491e00700 1 --2- 172.21.15.6:0/1623270766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8160d40 0x555ae81836e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:55.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.818+0000 7f6492800700 1 --2- 172.21.15.6:0/1623270766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae81740b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f648c009fd0 tx=0x7f648c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d57e91c27b301534 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:55.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.819+0000 7f647ea00700 1 -- 172.21.15.6:0/1623270766 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f648c004db0 con 0x555ae816c4d0 2024-04-26T01:49:55.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.819+0000 7f647ea00700 1 -- 172.21.15.6:0/1623270766 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f648c0093a0 con 0x555ae816c4d0 2024-04-26T01:49:55.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.819+0000 7f647ea00700 1 -- 172.21.15.6:0/1623270766 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f648c017530 con 0x555ae816c4d0 2024-04-26T01:49:55.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.819+0000 7f64a361bb80 1 -- 172.21.15.6:0/1623270766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 msgr2=0x555ae81740b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:55.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.819+0000 7f64a361bb80 1 --2- 172.21.15.6:0/1623270766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae81740b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f648c009fd0 tx=0x7f648c009920 comp rx=0 tx=0).stop 2024-04-26T01:49:55.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.819+0000 7f64a361bb80 1 -- 172.21.15.6:0/1623270766 shutdown_connections 2024-04-26T01:49:55.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.819+0000 7f64a361bb80 1 --2- 172.21.15.6:0/1623270766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae81740b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.819+0000 7f64a361bb80 1 --2- 172.21.15.6:0/1623270766 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8160d40 0x555ae81836e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.819+0000 7f64a361bb80 1 --2- 172.21.15.6:0/1623270766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8161660 0x555ae8183c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.819+0000 7f64a361bb80 1 -- 172.21.15.6:0/1623270766 >> 172.21.15.6:0/1623270766 conn(0x555ae8090050 msgr2=0x555ae816afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:55.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.819+0000 7f64a361bb80 1 -- 172.21.15.6:0/1623270766 shutdown_connections 2024-04-26T01:49:55.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.820+0000 7f64a361bb80 1 -- 172.21.15.6:0/1623270766 wait complete. 2024-04-26T01:49:55.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.820+0000 7f64a361bb80 1 -- start start 2024-04-26T01:49:55.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.820+0000 7f64a361bb80 1 Processor -- start 2024-04-26T01:49:55.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.821+0000 7f64a361bb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8160d40 0x555ae8171030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.821+0000 7f64a361bb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 0x555ae8171570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.821+0000 7f64a361bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae8171ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.821+0000 7f64a361bb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f64941adc40 con 0x555ae8161660 2024-04-26T01:49:55.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.821+0000 7f64a361bb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f64941af0b0 con 0x555ae8160d40 2024-04-26T01:49:55.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.821+0000 7f64a361bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f64941ae310 con 0x555ae816c4d0 2024-04-26T01:49:55.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.821+0000 7f64a361bb80 10 client.0 _ll_register_callbacks cb 0x555ae804c820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:55.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.821+0000 7f6491400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 0x555ae8171570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.821+0000 7f6491e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8160d40 0x555ae8171030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.821+0000 7f6491e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8160d40 0x555ae8171030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:48318/0 (socket says 172.21.15.6:48318) 2024-04-26T01:49:55.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.821+0000 7f6491e00700 1 -- 172.21.15.6:0/3381715265 learned_addr learned my addr 172.21.15.6:0/3381715265 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.821+0000 7f6492800700 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae8171ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f6491e00700 1 -- 172.21.15.6:0/3381715265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 msgr2=0x555ae8171ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f6491e00700 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae8171ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f6491e00700 1 -- 172.21.15.6:0/3381715265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 msgr2=0x555ae8171570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f6491e00700 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 0x555ae8171570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f6491e00700 1 -- 172.21.15.6:0/3381715265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ae8161ce0 con 0x555ae8160d40 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f6491400700 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 0x555ae8171570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f6491e00700 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8160d40 0x555ae8171030 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6484099990 tx=0x7f6484009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=659d04223882484d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f6473e00700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f6473e00700 1 -- 172.21.15.6:0/3381715265 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484082e60 con 0x555ae8160d40 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f6473e00700 1 -- 172.21.15.6:0/3381715265 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f64840bc330 con 0x555ae8160d40 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f64a361bb80 1 -- 172.21.15.6:0/3381715265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f648c0093a0 con 0x555ae8160d40 2024-04-26T01:49:55.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f6473e00700 1 -- 172.21.15.6:0/3381715265 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f64840b6040 con 0x555ae8160d40 2024-04-26T01:49:55.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f64a361bb80 10 client.25664 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:55.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.822+0000 7f64a361bb80 1 -- 172.21.15.6:0/3381715265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555ae81e32f0 con 0x555ae8160d40 2024-04-26T01:49:55.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.823+0000 7f6473e00700 1 -- 172.21.15.6:0/3381715265 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f648c0093a0 con 0x555ae8160d40 2024-04-26T01:49:55.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.823+0000 7f6473e00700 1 -- 172.21.15.6:0/3381715265 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f64840ceed0 con 0x555ae8160d40 2024-04-26T01:49:55.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.823+0000 7f64a361bb80 10 client.25664 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:55.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.823+0000 7f64a361bb80 10 client.25664 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:49:55.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.823+0000 7f64a361bb80 20 client.25664 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:55.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.823+0000 7f64a361bb80 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ae81e3860 0x555ae81e5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:55.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.823+0000 7f64a361bb80 4 client.25664 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "dcc36de", "release": true, "await": true}] 2024-04-26T01:49:55.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.823+0000 7f64a361bb80 1 -- 172.21.15.6:0/3381715265 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "dcc36de", "release": true, "await": true}) -- 0x555ae81698e0 con 0x555ae81e3860 2024-04-26T01:49:55.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.823+0000 7f6491400700 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ae81e3860 0x555ae81e5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:55.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.824+0000 7f6491400700 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ae81e3860 0x555ae81e5c80 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6f3a992523868d3 server_cookie=49e702e81344e177 in_seq=0 out_seq=0 2024-04-26T01:49:55.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.824+0000 7f6473e00700 10 client.25664 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:55.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f6473e00700 1 -- 172.21.15.6:0/3381715265 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x555ae81698e0 con 0x555ae81e3860 2024-04-26T01:49:55.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f6473e00700 10 client.25664 handle_command_reply: tid=0 2024-04-26T01:49:55.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f64a361bb80 1 client.25664 shutdown 2024-04-26T01:49:55.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f64a361bb80 1 -- 172.21.15.6:0/3381715265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8160d40 msgr2=0x555ae8171030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:55.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f64a361bb80 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8160d40 0x555ae8171030 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6484099990 tx=0x7f6484009290 comp rx=0 tx=0).stop 2024-04-26T01:49:55.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f64a361bb80 1 -- 172.21.15.6:0/3381715265 shutdown_connections 2024-04-26T01:49:55.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f64a361bb80 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ae81e3860 0x555ae81e5c80 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f64a361bb80 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ae816c4d0 0x555ae8171ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f64a361bb80 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ae8160d40 0x555ae8171030 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f6473e00700 0 client.25664 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:55.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f64a361bb80 1 --2- 172.21.15.6:0/3381715265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ae8161660 0x555ae8171570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f64a361bb80 1 -- 172.21.15.6:0/3381715265 >> 172.21.15.6:0/3381715265 conn(0x555ae8090050 msgr2=0x555ae815c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:55.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f64a361bb80 1 -- 172.21.15.6:0/3381715265 shutdown_connections 2024-04-26T01:49:55.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.832+0000 7f64a361bb80 1 -- 172.21.15.6:0/3381715265 wait complete. 2024-04-26T01:49:55.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.833+0000 7f64a361bb80 20 client.25664 trim_cache size 0 max 16384 2024-04-26T01:49:55.831 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 31, 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 12, 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "dcc36de": { 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 12, 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 90.0, 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:49:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:49:55.833 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:49:55.833 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:49:55.833 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:49:55.833 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:49:55.833 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:55.833 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:55.833 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:55.833 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:55.833 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:49:55.833 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:49:55.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.833+0000 7f6498c00700 1 -- 172.21.15.6:0/2335455204 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f64780a3e30 msgr2=0x7f64780a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:55.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.833+0000 7f6498c00700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f64780a3e30 0x7f64780a62f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6480009750 tx=0x7f6480009310 comp rx=0 tx=0).stop 2024-04-26T01:49:55.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.833+0000 7f6498c00700 1 -- 172.21.15.6:0/2335455204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6494102a90 msgr2=0x7f649419b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:55.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.833+0000 7f6498c00700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6494102a90 0x7f649419b3f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f648400c010 tx=0x7f648400fa00 comp rx=0 tx=0).stop 2024-04-26T01:49:55.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.833+0000 7f6498c00700 1 -- 172.21.15.6:0/2335455204 shutdown_connections 2024-04-26T01:49:55.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.833+0000 7f6498c00700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f64780a3e30 0x7f64780a62f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.833+0000 7f6498c00700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6494108a50 0x7f649419fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.833+0000 7f6498c00700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6494102a90 0x7f649419b3f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.833+0000 7f6498c00700 1 --2- 172.21.15.6:0/2335455204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6494103450 0x7f649419b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:55.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.833+0000 7f6498c00700 1 -- 172.21.15.6:0/2335455204 >> 172.21.15.6:0/2335455204 conn(0x7f6494079ba0 msgr2=0x7f6494101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:55.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.834+0000 7f6498c00700 1 -- 172.21.15.6:0/2335455204 shutdown_connections 2024-04-26T01:49:55.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:49:55.834+0000 7f6498c00700 1 -- 172.21.15.6:0/2335455204 wait complete. 2024-04-26T01:49:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.485+0000 7f7b73e00640 1 -- 172.21.15.6:0/321145594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b74103850 msgr2=0x7f7b7410bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.485+0000 7f7b73e00640 1 --2- 172.21.15.6:0/321145594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b74103850 0x7f7b7410bd30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009900 tx=0x7f7b6002a480 comp rx=0 tx=0).stop 2024-04-26T01:49:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.485+0000 7f7b73e00640 1 -- 172.21.15.6:0/321145594 shutdown_connections 2024-04-26T01:49:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.485+0000 7f7b73e00640 1 --2- 172.21.15.6:0/321145594 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b7410c350 0x7f7b7410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.485+0000 7f7b73e00640 1 --2- 172.21.15.6:0/321145594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b74103850 0x7f7b7410bd30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.485+0000 7f7b73e00640 1 --2- 172.21.15.6:0/321145594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b74102f30 0x7f7b74103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.485+0000 7f7b73e00640 1 -- 172.21.15.6:0/321145594 >> 172.21.15.6:0/321145594 conn(0x7f7b740fc730 msgr2=0x7f7b740feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.485+0000 7f7b73e00640 1 -- 172.21.15.6:0/321145594 shutdown_connections 2024-04-26T01:49:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.485+0000 7f7b73e00640 1 -- 172.21.15.6:0/321145594 wait complete. 2024-04-26T01:49:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.486+0000 7f7b73e00640 1 Processor -- start 2024-04-26T01:49:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.487+0000 7f7b73e00640 1 -- start start 2024-04-26T01:49:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.487+0000 7f7b73e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b74102f30 0x7f7b7419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.487+0000 7f7b73e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b74103850 0x7f7b7419ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.487+0000 7f7b73e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b7410c350 0x7f7b7419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.487+0000 7f7b73e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7b74112010 con 0x7f7b74103850 2024-04-26T01:49:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.487+0000 7f7b73e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7b74111e90 con 0x7f7b74102f30 2024-04-26T01:49:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.487+0000 7f7b73e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7b74112190 con 0x7f7b7410c350 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.488+0000 7f7b72a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b74102f30 0x7f7b7419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.488+0000 7f7b72a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b74102f30 0x7f7b7419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:48344/0 (socket says 172.21.15.6:48344) 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.488+0000 7f7b72a00640 1 -- 172.21.15.6:0/960807431 learned_addr learned my addr 172.21.15.6:0/960807431 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.488+0000 7f7b72000640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b74103850 0x7f7b7419ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.488+0000 7f7b73400640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b7410c350 0x7f7b7419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.488+0000 7f7b72a00640 1 -- 172.21.15.6:0/960807431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b7410c350 msgr2=0x7f7b7419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.488+0000 7f7b72a00640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b7410c350 0x7f7b7419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.488+0000 7f7b72a00640 1 -- 172.21.15.6:0/960807431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b74103850 msgr2=0x7f7b7419ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.488+0000 7f7b72a00640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b74103850 0x7f7b7419ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.488+0000 7f7b72a00640 1 -- 172.21.15.6:0/960807431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7419f740 con 0x7f7b74102f30 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.488+0000 7f7b72000640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b74103850 0x7f7b7419ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.488+0000 7f7b72a00640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b74102f30 0x7f7b7419a750 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b6400ebc0 tx=0x7f7b6400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.489+0000 7f7b6b400640 1 -- 172.21.15.6:0/960807431 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64010630 con 0x7f7b74102f30 2024-04-26T01:49:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.489+0000 7f7b73e00640 1 -- 172.21.15.6:0/960807431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7419fa30 con 0x7f7b74102f30 2024-04-26T01:49:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.489+0000 7f7b6b400640 1 -- 172.21.15.6:0/960807431 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b64010c80 con 0x7f7b74102f30 2024-04-26T01:49:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.489+0000 7f7b6b400640 1 -- 172.21.15.6:0/960807431 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64018b60 con 0x7f7b74102f30 2024-04-26T01:49:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.489+0000 7f7b73e00640 1 -- 172.21.15.6:0/960807431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b741a72c0 con 0x7f7b74102f30 2024-04-26T01:49:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.492+0000 7f7b6b400640 1 -- 172.21.15.6:0/960807431 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f7b64018d00 con 0x7f7b74102f30 2024-04-26T01:49:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.492+0000 7f7b7c775740 1 Processor -- start 2024-04-26T01:49:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.492+0000 7f7b7c775740 1 -- start start 2024-04-26T01:49:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.493+0000 7f7b7c775740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275ddd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.493+0000 7f7b7c775740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275e8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.493+0000 7f7b7c775740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x5613275eaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.493+0000 7f7b7c775740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x561327543900 con 0x5613275de2d0 2024-04-26T01:49:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.493+0000 7f7b7c775740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5613274e9ef0 con 0x5613275dd9b0 2024-04-26T01:49:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.493+0000 7f7b7c775740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56132753f8c0 con 0x5613275e8b90 2024-04-26T01:49:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.493+0000 7f7b73400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275ddd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.493+0000 7f7b73400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275ddd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:48358/0 (socket says 172.21.15.6:48358) 2024-04-26T01:49:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.493+0000 7f7b73400640 1 -- 172.21.15.6:0/1676087364 learned_addr learned my addr 172.21.15.6:0/1676087364 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.493+0000 7f7b72a00640 1 --2- 172.21.15.6:0/1676087364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275e8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.493+0000 7f7b72000640 1 --2- 172.21.15.6:0/1676087364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x5613275eaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b73400640 1 -- 172.21.15.6:0/1676087364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 msgr2=0x5613275eaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b73400640 1 --2- 172.21.15.6:0/1676087364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x5613275eaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b73400640 1 -- 172.21.15.6:0/1676087364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 msgr2=0x5613275e8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b73400640 1 --2- 172.21.15.6:0/1676087364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275e8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b73400640 1 -- 172.21.15.6:0/1676087364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561327256210 con 0x5613275dd9b0 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b72a00640 1 --2- 172.21.15.6:0/1676087364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275e8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b73400640 1 --2- 172.21.15.6:0/1676087364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275ddd90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009b90 tx=0x7f7b6c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a3e0b4cd5b90e6c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b6b400640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7b580a3d00 0x7f7b580a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b5fe00640 1 -- 172.21.15.6:0/1676087364 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c012450 con 0x5613275dd9b0 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b5fe00640 1 -- 172.21.15.6:0/1676087364 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b6c012aa0 con 0x5613275dd9b0 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b5fe00640 1 -- 172.21.15.6:0/1676087364 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c00fa30 con 0x5613275dd9b0 2024-04-26T01:49:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b6b400640 1 -- 172.21.15.6:0/960807431 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f7b640d0fc0 con 0x7f7b74102f30 2024-04-26T01:49:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.494+0000 7f7b72a00640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7b580a3d00 0x7f7b580a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.495+0000 7f7b7c775740 1 -- 172.21.15.6:0/1676087364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 msgr2=0x5613275ddd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.495+0000 7f7b7c775740 1 --2- 172.21.15.6:0/1676087364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275ddd90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009b90 tx=0x7f7b6c0117b0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.495+0000 7f7b72a00640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7b580a3d00 0x7f7b580a61c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x5613275def50 tx=0x7f7b64000f40 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.495+0000 7f7b7c775740 1 -- 172.21.15.6:0/1676087364 shutdown_connections 2024-04-26T01:49:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.495+0000 7f7b7c775740 1 --2- 172.21.15.6:0/1676087364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275e8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.495+0000 7f7b7c775740 1 --2- 172.21.15.6:0/1676087364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x5613275eaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.495+0000 7f7b7c775740 1 --2- 172.21.15.6:0/1676087364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275ddd90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.495+0000 7f7b7c775740 1 -- 172.21.15.6:0/1676087364 >> 172.21.15.6:0/1676087364 conn(0x5613275d9ec0 msgr2=0x5613275da2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.495+0000 7f7b7c775740 1 -- 172.21.15.6:0/1676087364 shutdown_connections 2024-04-26T01:49:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.495+0000 7f7b7c775740 1 -- 172.21.15.6:0/1676087364 wait complete. 2024-04-26T01:49:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.496+0000 7f7b7c775740 1 -- start start 2024-04-26T01:49:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.496+0000 7f7b7c775740 1 Processor -- start 2024-04-26T01:49:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.496+0000 7f7b7c775740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275e5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b7c775740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275e6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b7c775740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x5613275e6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b7c775740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5613275eb7c0 con 0x5613275de2d0 2024-04-26T01:49:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b7c775740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5613274fd310 con 0x5613275dd9b0 2024-04-26T01:49:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b7c775740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5613275eb940 con 0x5613275e8b90 2024-04-26T01:49:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b7c775740 10 client.0 _ll_register_callbacks cb 0x5613274cbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b73400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275e5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b72000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275e6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b73400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275e5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:48368/0 (socket says 172.21.15.6:48368) 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b73400640 1 -- 172.21.15.6:0/265792818 learned_addr learned my addr 172.21.15.6:0/265792818 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b73400640 1 --2- 172.21.15.6:0/265792818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x5613275e6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b72000640 1 -- 172.21.15.6:0/265792818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 msgr2=0x5613275e6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b72000640 1 --2- 172.21.15.6:0/265792818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x5613275e6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b72000640 1 -- 172.21.15.6:0/265792818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 msgr2=0x5613275e5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b72000640 1 --2- 172.21.15.6:0/265792818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275e5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.497+0000 7f7b72000640 1 -- 172.21.15.6:0/265792818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561327322620 con 0x5613275de2d0 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.498+0000 7f7b72000640 1 --2- 172.21.15.6:0/265792818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275e6050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7b6002aa70 tx=0x7f7b60005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1b99513f727853 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.498+0000 7f7b5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.498+0000 7f7b5cc00640 1 -- 172.21.15.6:0/265792818 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60035040 con 0x5613275de2d0 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.498+0000 7f7b7c775740 1 -- 172.21.15.6:0/265792818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b6c012aa0 con 0x5613275de2d0 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.498+0000 7f7b5cc00640 1 -- 172.21.15.6:0/265792818 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f7b600048c0 con 0x5613275de2d0 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.498+0000 7f7b5cc00640 1 -- 172.21.15.6:0/265792818 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7b60004ac0 con 0x5613275de2d0 2024-04-26T01:49:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.498+0000 7f7b7c775740 10 client.15552 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.498+0000 7f7b7c775740 1 -- 172.21.15.6:0/265792818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5613275f7a50 con 0x5613275de2d0 2024-04-26T01:49:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.499+0000 7f7b5cc00640 1 -- 172.21.15.6:0/265792818 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b6c012aa0 con 0x5613275de2d0 2024-04-26T01:49:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.499+0000 7f7b5cc00640 1 -- 172.21.15.6:0/265792818 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f7b600314a0 con 0x5613275de2d0 2024-04-26T01:49:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.499+0000 7f7b7c775740 10 client.15552 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.499+0000 7f7b7c775740 10 client.15552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.499+0000 7f7b7c775740 20 client.15552 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.499+0000 7f7b7c775740 1 --2- 172.21.15.6:0/265792818 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5613275f8ec0 0x5613275fb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.499+0000 7f7b7c775740 4 client.15552 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.499+0000 7f7b7c775740 1 -- 172.21.15.6:0/265792818 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613275578b0 con 0x5613275f8ec0 2024-04-26T01:49:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.499+0000 7f7b73400640 1 --2- 172.21.15.6:0/265792818 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5613275f8ec0 0x5613275fb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.500+0000 7f7b73400640 1 --2- 172.21.15.6:0/265792818 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5613275f8ec0 0x5613275fb2b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=859a995dbb2d821d server_cookie=27e3fe9ab3315460 in_seq=0 out_seq=0 2024-04-26T01:49:56.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.500+0000 7f7b5cc00640 10 client.15552 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:56.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.501+0000 7f7b5cc00640 1 -- 172.21.15.6:0/265792818 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613275578b0 con 0x5613275f8ec0 2024-04-26T01:49:56.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.501+0000 7f7b5cc00640 10 client.15552 handle_command_reply: tid=0 2024-04-26T01:49:56.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.501+0000 7f7b7c775740 1 client.15552 shutdown 2024-04-26T01:49:56.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.501+0000 7f7b7c775740 1 -- 172.21.15.6:0/265792818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 msgr2=0x5613275e6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:56.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.501+0000 7f7b7c775740 1 --2- 172.21.15.6:0/265792818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275e6050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7b6002aa70 tx=0x7f7b60005270 comp rx=0 tx=0).stop 2024-04-26T01:49:56.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.501+0000 7f7b7c775740 1 -- 172.21.15.6:0/265792818 shutdown_connections 2024-04-26T01:49:56.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.501+0000 7f7b7c775740 1 --2- 172.21.15.6:0/265792818 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5613275f8ec0 0x5613275fb2b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.501+0000 7f7b7c775740 1 --2- 172.21.15.6:0/265792818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275e6050 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.501+0000 7f7b7c775740 1 --2- 172.21.15.6:0/265792818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x5613275e6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.501+0000 7f7b7c775740 1 --2- 172.21.15.6:0/265792818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275e5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.501+0000 7f7b5cc00640 0 client.15552 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:56.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.501+0000 7f7b7c775740 1 -- 172.21.15.6:0/265792818 >> 172.21.15.6:0/265792818 conn(0x5613275d9ec0 msgr2=0x5613275dfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:56.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.502+0000 7f7b7c775740 1 -- 172.21.15.6:0/265792818 shutdown_connections 2024-04-26T01:49:56.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.502+0000 7f7b7c775740 1 -- 172.21.15.6:0/265792818 wait complete. 2024-04-26T01:49:56.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.502+0000 7f7b7c775740 20 client.15552 trim_cache size 0 max 16384 2024-04-26T01:49:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.519+0000 7f7b7c775740 1 Processor -- start 2024-04-26T01:49:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.520+0000 7f7b7c775740 1 -- start start 2024-04-26T01:49:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.520+0000 7f7b7c775740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275f4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.520+0000 7f7b7c775740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x561327600bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.520+0000 7f7b7c775740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x561327601100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.520+0000 7f7b7c775740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7b741b2680 con 0x5613275de2d0 2024-04-26T01:49:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.520+0000 7f7b7c775740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7b741afef0 con 0x5613275dd9b0 2024-04-26T01:49:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.520+0000 7f7b7c775740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7b741add40 con 0x5613275e8b90 2024-04-26T01:49:56.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.520+0000 7f7b72000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x561327600bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.520+0000 7f7b73400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.520+0000 7f7b73400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275f4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:48374/0 (socket says 172.21.15.6:48374) 2024-04-26T01:49:56.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.520+0000 7f7b73400640 1 -- 172.21.15.6:0/2943781716 learned_addr learned my addr 172.21.15.6:0/2943781716 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:56.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.520+0000 7f7b73400640 1 --2- 172.21.15.6:0/2943781716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x561327601100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.521+0000 7f7b72000640 1 -- 172.21.15.6:0/2943781716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 msgr2=0x561327601100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:56.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.521+0000 7f7b72000640 1 --2- 172.21.15.6:0/2943781716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x561327601100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.521+0000 7f7b72000640 1 -- 172.21.15.6:0/2943781716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 msgr2=0x5613275f4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:56.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.521+0000 7f7b72000640 1 --2- 172.21.15.6:0/2943781716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275f4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.521+0000 7f7b72000640 1 -- 172.21.15.6:0/2943781716 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613275de100 con 0x5613275de2d0 2024-04-26T01:49:56.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.521+0000 7f7b73400640 1 --2- 172.21.15.6:0/2943781716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.521+0000 7f7b72000640 1 --2- 172.21.15.6:0/2943781716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x561327600bc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7b600318a0 tx=0x7f7b60005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345c479fdb544d42 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.521+0000 7f7b5f400640 1 -- 172.21.15.6:0/2943781716 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60052070 con 0x5613275de2d0 2024-04-26T01:49:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.521+0000 7f7b5f400640 1 -- 172.21.15.6:0/2943781716 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b60040070 con 0x5613275de2d0 2024-04-26T01:49:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.521+0000 7f7b5f400640 1 -- 172.21.15.6:0/2943781716 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60035040 con 0x5613275de2d0 2024-04-26T01:49:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.522+0000 7f7b7c775740 1 -- 172.21.15.6:0/2943781716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 msgr2=0x561327600bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.522+0000 7f7b7c775740 1 --2- 172.21.15.6:0/2943781716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x561327600bc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7b600318a0 tx=0x7f7b60005a20 comp rx=0 tx=0).stop 2024-04-26T01:49:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.522+0000 7f7b7c775740 1 -- 172.21.15.6:0/2943781716 shutdown_connections 2024-04-26T01:49:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.522+0000 7f7b7c775740 1 --2- 172.21.15.6:0/2943781716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x561327600bc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.522+0000 7f7b7c775740 1 --2- 172.21.15.6:0/2943781716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x561327601100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.522+0000 7f7b7c775740 1 --2- 172.21.15.6:0/2943781716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.522+0000 7f7b7c775740 1 -- 172.21.15.6:0/2943781716 >> 172.21.15.6:0/2943781716 conn(0x5613275d9ec0 msgr2=0x5613275e0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.522+0000 7f7b7c775740 1 -- 172.21.15.6:0/2943781716 shutdown_connections 2024-04-26T01:49:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.522+0000 7f7b7c775740 1 -- 172.21.15.6:0/2943781716 wait complete. 2024-04-26T01:49:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.523+0000 7f7b7c775740 1 -- start start 2024-04-26T01:49:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.523+0000 7f7b7c775740 1 Processor -- start 2024-04-26T01:49:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.523+0000 7f7b7c775740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275edd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b7c775740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275ee290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b7c775740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x56132765d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b7c775740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7b741ae3b0 con 0x5613275de2d0 2024-04-26T01:49:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b7c775740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7b741af820 con 0x5613275dd9b0 2024-04-26T01:49:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b7c775740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7b741aea80 con 0x5613275e8b90 2024-04-26T01:49:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b7c775740 10 client.0 _ll_register_callbacks cb 0x7f7b60009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b72000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275ee290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b73400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275edd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b72000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275ee290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55660/0 (socket says 172.21.15.6:55660) 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b72000640 1 -- 172.21.15.6:0/3700569241 learned_addr learned my addr 172.21.15.6:0/3700569241 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b73400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275edd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:48382/0 (socket says 172.21.15.6:48382) 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b73400640 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x56132765d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b72000640 1 -- 172.21.15.6:0/3700569241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 msgr2=0x56132765d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b72000640 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x56132765d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b72000640 1 -- 172.21.15.6:0/3700569241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 msgr2=0x5613275edd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b72000640 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275edd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b72000640 1 -- 172.21.15.6:0/3700569241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56132765d970 con 0x5613275de2d0 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.524+0000 7f7b73400640 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275edd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.525+0000 7f7b72000640 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275ee290 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7b60031990 tx=0x7f7b60031cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92540d40cf16796e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.525+0000 7f7b5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.525+0000 7f7b5cc00640 1 -- 172.21.15.6:0/3700569241 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6005a070 con 0x5613275de2d0 2024-04-26T01:49:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.525+0000 7f7b5cc00640 1 -- 172.21.15.6:0/3700569241 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f7b600046c0 con 0x5613275de2d0 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.525+0000 7f7b7c775740 1 -- 172.21.15.6:0/3700569241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b60040070 con 0x5613275de2d0 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.525+0000 7f7b5cc00640 1 -- 172.21.15.6:0/3700569241 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7b60052050 con 0x5613275de2d0 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.525+0000 7f7b7c775740 10 client.15558 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.525+0000 7f7b7c775740 1 -- 172.21.15.6:0/3700569241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56132765ec60 con 0x5613275de2d0 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.525+0000 7f7b5cc00640 1 -- 172.21.15.6:0/3700569241 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b60040070 con 0x5613275de2d0 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.526+0000 7f7b5cc00640 1 -- 172.21.15.6:0/3700569241 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f7b6005d020 con 0x5613275de2d0 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.526+0000 7f7b7c775740 10 client.15558 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.526+0000 7f7b7c775740 10 client.15558 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.526+0000 7f7b7c775740 20 client.15558 populate_metadata read hostname 'smithi006' 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.526+0000 7f7b7c775740 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56132765f390 0x5613276617b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.526+0000 7f7b7c775740 4 client.15558 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.526+0000 7f7b7c775740 1 -- 172.21.15.6:0/3700569241 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561327551320 con 0x56132765f390 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.526+0000 7f7b73400640 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56132765f390 0x5613276617b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.527+0000 7f7b73400640 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56132765f390 0x5613276617b0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfb919ad43a89a37 server_cookie=2ea00896dea873c6 in_seq=0 out_seq=0 2024-04-26T01:49:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.527+0000 7f7b5cc00640 10 client.15558 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b5cc00640 1 -- 172.21.15.6:0/3700569241 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561327551320 con 0x56132765f390 2024-04-26T01:49:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b5cc00640 10 client.15558 handle_command_reply: tid=0 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b7c775740 1 client.15558 shutdown 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b7c775740 1 -- 172.21.15.6:0/3700569241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 msgr2=0x5613275ee290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b7c775740 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275ee290 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7b60031990 tx=0x7f7b60031cc0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b7c775740 1 -- 172.21.15.6:0/3700569241 shutdown_connections 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b7c775740 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56132765f390 0x5613276617b0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b7c775740 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613275de2d0 0x5613275ee290 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b7c775740 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613275e8b90 0x56132765d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b7c775740 1 --2- 172.21.15.6:0/3700569241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613275dd9b0 0x5613275edd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b5cc00640 0 client.15558 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b7c775740 1 -- 172.21.15.6:0/3700569241 >> 172.21.15.6:0/3700569241 conn(0x5613275d9ec0 msgr2=0x5613275e7340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b7c775740 1 -- 172.21.15.6:0/3700569241 shutdown_connections 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b7c775740 1 -- 172.21.15.6:0/3700569241 wait complete. 2024-04-26T01:49:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.551+0000 7f7b7c775740 20 client.15558 trim_cache size 0 max 16384 2024-04-26T01:49:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:49:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:49:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:49:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:49:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.552+0000 7f7b73e00640 1 -- 172.21.15.6:0/960807431 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7b580a3d00 msgr2=0x7f7b580a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.552+0000 7f7b73e00640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7b580a3d00 0x7f7b580a61c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x5613275def50 tx=0x7f7b64000f40 comp rx=0 tx=0).stop 2024-04-26T01:49:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.552+0000 7f7b73e00640 1 -- 172.21.15.6:0/960807431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b74102f30 msgr2=0x7f7b7419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.552+0000 7f7b73e00640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b74102f30 0x7f7b7419a750 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b6400ebc0 tx=0x7f7b6400c6a0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.552+0000 7f7b73e00640 1 -- 172.21.15.6:0/960807431 shutdown_connections 2024-04-26T01:49:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.552+0000 7f7b73e00640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7b580a3d00 0x7f7b580a61c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.552+0000 7f7b73e00640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b74103850 0x7f7b7419ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.552+0000 7f7b73e00640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b7410c350 0x7f7b7419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.552+0000 7f7b73e00640 1 --2- 172.21.15.6:0/960807431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b74102f30 0x7f7b7419a750 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.552+0000 7f7b73e00640 1 -- 172.21.15.6:0/960807431 >> 172.21.15.6:0/960807431 conn(0x7f7b740fc730 msgr2=0x7f7b74100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.553+0000 7f7b73e00640 1 -- 172.21.15.6:0/960807431 shutdown_connections 2024-04-26T01:49:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:49:56.553+0000 7f7b73e00640 1 -- 172.21.15.6:0/960807431 wait complete. 2024-04-26T01:49:56.567 INFO:tasks.workunit.client.0.smithi006.stdout: 1 93964 0.25 MB/sec execute 20 sec latency 37678.270 ms 2024-04-26T01:49:56.807 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'dcc36de', seconds elapsed: 45.5 2024-04-26T01:49:56.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:56 smithi006 ceph-mon[28419]: pgmap v125: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-26T01:49:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:56 smithi188 ceph-mon[32751]: pgmap v125: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-26T01:49:56.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:56 smithi029 ceph-mon[32775]: pgmap v125: 97 pgs: 97 active+clean; 34 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-26T01:49:57.568 INFO:tasks.workunit.client.0.smithi006.stdout: 1 95477 0.62 MB/sec execute 21 sec latency 14.133 ms 2024-04-26T01:49:58.568 INFO:tasks.workunit.client.0.smithi006.stdout: 1 96286 0.86 MB/sec execute 22 sec latency 14.915 ms 2024-04-26T01:49:58.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:49:58 smithi006 ceph-mon[28419]: pgmap v126: 97 pgs: 97 active+clean; 34 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 58 KiB/s wr, 16 op/s 2024-04-26T01:49:58.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:49:58 smithi188 ceph-mon[32751]: pgmap v126: 97 pgs: 97 active+clean; 34 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 58 KiB/s wr, 16 op/s 2024-04-26T01:49:58.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:49:58 smithi029 ceph-mon[32775]: pgmap v126: 97 pgs: 97 active+clean; 34 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 58 KiB/s wr, 16 op/s 2024-04-26T01:49:59.568 INFO:tasks.workunit.client.0.smithi006.stdout: 1 97749 1.06 MB/sec execute 23 sec latency 14.831 ms 2024-04-26T01:50:00.568 INFO:tasks.workunit.client.0.smithi006.stdout: 1 99057 1.35 MB/sec execute 24 sec latency 53.763 ms 2024-04-26T01:50:00.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:00 smithi006 ceph-mon[28419]: pgmap v127: 97 pgs: 97 active+clean; 34 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 48 KiB/s wr, 9 op/s 2024-04-26T01:50:00.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:00 smithi006 ceph-mon[28419]: overall HEALTH_OK 2024-04-26T01:50:00.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:00 smithi188 ceph-mon[32751]: pgmap v127: 97 pgs: 97 active+clean; 34 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 48 KiB/s wr, 9 op/s 2024-04-26T01:50:00.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:00 smithi188 ceph-mon[32751]: overall HEALTH_OK 2024-04-26T01:50:00.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:00 smithi029 ceph-mon[32775]: pgmap v127: 97 pgs: 97 active+clean; 34 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 48 KiB/s wr, 9 op/s 2024-04-26T01:50:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:00 smithi029 ceph-mon[32775]: overall HEALTH_OK 2024-04-26T01:50:01.571 INFO:tasks.workunit.client.0.smithi006.stdout: 1 99673 1.51 MB/sec execute 25 sec latency 59.166 ms 2024-04-26T01:50:02.568 INFO:tasks.workunit.client.0.smithi006.stdout: 1 100613 1.54 MB/sec execute 26 sec latency 14.860 ms 2024-04-26T01:50:02.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:02 smithi188 ceph-mon[32751]: pgmap v128: 97 pgs: 97 active+clean; 38 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 505 KiB/s wr, 115 op/s 2024-04-26T01:50:02.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:02 smithi029 ceph-mon[32775]: pgmap v128: 97 pgs: 97 active+clean; 38 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 505 KiB/s wr, 115 op/s 2024-04-26T01:50:03.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:02 smithi006 ceph-mon[28419]: pgmap v128: 97 pgs: 97 active+clean; 38 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 505 KiB/s wr, 115 op/s 2024-04-26T01:50:03.569 INFO:tasks.workunit.client.0.smithi006.stdout: 1 102259 1.85 MB/sec execute 27 sec latency 20.600 ms 2024-04-26T01:50:03.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:03 smithi006 ceph-mon[28419]: pgmap v129: 97 pgs: 97 active+clean; 38 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 503 KiB/s wr, 111 op/s 2024-04-26T01:50:03.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:03 smithi188 ceph-mon[32751]: pgmap v129: 97 pgs: 97 active+clean; 38 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 503 KiB/s wr, 111 op/s 2024-04-26T01:50:03.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:03 smithi029 ceph-mon[32775]: pgmap v129: 97 pgs: 97 active+clean; 38 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 503 KiB/s wr, 111 op/s 2024-04-26T01:50:04.569 INFO:tasks.workunit.client.0.smithi006.stdout: 1 103280 2.04 MB/sec execute 28 sec latency 16.325 ms 2024-04-26T01:50:05.569 INFO:tasks.workunit.client.0.smithi006.stdout: 1 104403 2.15 MB/sec execute 29 sec latency 22.998 ms 2024-04-26T01:50:06.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:06 smithi188 ceph-mon[32751]: pgmap v130: 97 pgs: 97 active+clean; 38 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 503 KiB/s wr, 111 op/s 2024-04-26T01:50:06.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:06 smithi029 ceph-mon[32775]: pgmap v130: 97 pgs: 97 active+clean; 38 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 503 KiB/s wr, 111 op/s 2024-04-26T01:50:06.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:06 smithi006 ceph-mon[28419]: pgmap v130: 97 pgs: 97 active+clean; 38 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 503 KiB/s wr, 111 op/s 2024-04-26T01:50:06.569 INFO:tasks.workunit.client.0.smithi006.stdout: 1 105891 2.32 MB/sec execute 30 sec latency 18.618 ms 2024-04-26T01:50:07.569 INFO:tasks.workunit.client.0.smithi006.stdout: 1 106866 2.46 MB/sec execute 31 sec latency 18.053 ms 2024-04-26T01:50:08.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:08 smithi188 ceph-mon[32751]: pgmap v131: 97 pgs: 97 active+clean; 42 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 949 KiB/s wr, 182 op/s 2024-04-26T01:50:08.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:08 smithi029 ceph-mon[32775]: pgmap v131: 97 pgs: 97 active+clean; 42 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 949 KiB/s wr, 182 op/s 2024-04-26T01:50:08.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:08 smithi006 ceph-mon[28419]: pgmap v131: 97 pgs: 97 active+clean; 42 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 949 KiB/s wr, 182 op/s 2024-04-26T01:50:08.569 INFO:tasks.workunit.client.0.smithi006.stdout: 1 108088 2.55 MB/sec execute 32 sec latency 21.205 ms 2024-04-26T01:50:09.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:09 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:50:09.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:09 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:50:09.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:09 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:50:09.569 INFO:tasks.workunit.client.0.smithi006.stdout: 1 109008 2.55 MB/sec execute 33 sec latency 437.447 ms 2024-04-26T01:50:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.694+0000 7fc33f000640 1 -- 172.21.15.6:0/4013054025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc338109490 msgr2=0x7fc338111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.694+0000 7fc33f000640 1 --2- 172.21.15.6:0/4013054025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc338109490 0x7fc338111060 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc330009ca0 tx=0x7fc33002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.695+0000 7fc33f000640 1 -- 172.21.15.6:0/4013054025 shutdown_connections 2024-04-26T01:50:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.695+0000 7fc33f000640 1 --2- 172.21.15.6:0/4013054025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc338104d70 0x7fc338108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.695+0000 7fc33f000640 1 --2- 172.21.15.6:0/4013054025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc338109490 0x7fc338111060 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.695+0000 7fc33f000640 1 --2- 172.21.15.6:0/4013054025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3381043c0 0x7fc3381047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.695+0000 7fc33f000640 1 -- 172.21.15.6:0/4013054025 >> 172.21.15.6:0/4013054025 conn(0x7fc3380ffc40 msgr2=0x7fc338102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:09.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.695+0000 7fc33f000640 1 -- 172.21.15.6:0/4013054025 shutdown_connections 2024-04-26T01:50:09.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.695+0000 7fc33f000640 1 -- 172.21.15.6:0/4013054025 wait complete. 2024-04-26T01:50:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.696+0000 7fc33f000640 1 Processor -- start 2024-04-26T01:50:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.696+0000 7fc33f000640 1 -- start start 2024-04-26T01:50:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.696+0000 7fc33f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3381043c0 0x7fc3381a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.696+0000 7fc33f000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc338109490 0x7fc3381a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.696+0000 7fc33f000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc33819a7d0 0x7fc33819ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.696+0000 7fc33f000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc338114430 con 0x7fc3381043c0 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.696+0000 7fc33f000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381142b0 con 0x7fc338109490 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.696+0000 7fc33f000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381145b0 con 0x7fc33819a7d0 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.697+0000 7fc32f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc338109490 0x7fc3381a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.697+0000 7fc337e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3381043c0 0x7fc3381a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.697+0000 7fc32f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc338109490 0x7fc3381a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34766/0 (socket says 172.21.15.6:34766) 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.697+0000 7fc32f400640 1 -- 172.21.15.6:0/3224140372 learned_addr learned my addr 172.21.15.6:0/3224140372 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.697+0000 7fc33d000640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc33819a7d0 0x7fc33819ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.697+0000 7fc32f400640 1 -- 172.21.15.6:0/3224140372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc33819a7d0 msgr2=0x7fc33819ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.697+0000 7fc32f400640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc33819a7d0 0x7fc33819ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.697+0000 7fc32f400640 1 -- 172.21.15.6:0/3224140372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3381043c0 msgr2=0x7fc3381a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.697+0000 7fc32f400640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3381043c0 0x7fc3381a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.697+0000 7fc32f400640 1 -- 172.21.15.6:0/3224140372 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc33819b3d0 con 0x7fc338109490 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.697+0000 7fc337e00640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3381043c0 0x7fc3381a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.697+0000 7fc32f400640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc338109490 0x7fc3381a0bb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc32000e9c0 tx=0x7fc32000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.698+0000 7fc335600640 1 -- 172.21.15.6:0/3224140372 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32000cd80 con 0x7fc338109490 2024-04-26T01:50:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.698+0000 7fc335600640 1 -- 172.21.15.6:0/3224140372 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3200054d0 con 0x7fc338109490 2024-04-26T01:50:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.698+0000 7fc33f000640 1 -- 172.21.15.6:0/3224140372 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3381a7410 con 0x7fc338109490 2024-04-26T01:50:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.698+0000 7fc335600640 1 -- 172.21.15.6:0/3224140372 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320010430 con 0x7fc338109490 2024-04-26T01:50:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.698+0000 7fc33f000640 1 -- 172.21.15.6:0/3224140372 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3381a7810 con 0x7fc338109490 2024-04-26T01:50:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.701+0000 7fc341e4e740 1 Processor -- start 2024-04-26T01:50:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.701+0000 7fc335600640 1 -- 172.21.15.6:0/3224140372 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fc320004fd0 con 0x7fc338109490 2024-04-26T01:50:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.701+0000 7fc341e4e740 1 -- start start 2024-04-26T01:50:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.701+0000 7fc341e4e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381b820 0x55741381bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.701+0000 7fc341e4e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 0x5574138264c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.701+0000 7fc341e4e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557413826a00 0x557413828df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc341e4e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x557413781900 con 0x55741381b820 2024-04-26T01:50:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc341e4e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557413727ef0 con 0x557413826a00 2024-04-26T01:50:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc341e4e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55741377d8c0 con 0x55741381c140 2024-04-26T01:50:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc33d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381b820 0x55741381bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc337e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 0x5574138264c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc32f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557413826a00 0x557413828df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc337e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 0x5574138264c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50026/0 (socket says 172.21.15.6:50026) 2024-04-26T01:50:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc337e00640 1 -- 172.21.15.6:0/1726304993 learned_addr learned my addr 172.21.15.6:0/1726304993 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc337e00640 1 -- 172.21.15.6:0/1726304993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557413826a00 msgr2=0x557413828df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc337e00640 1 --2- 172.21.15.6:0/1726304993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557413826a00 0x557413828df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc337e00640 1 -- 172.21.15.6:0/1726304993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381b820 msgr2=0x55741381bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc337e00640 1 --2- 172.21.15.6:0/1726304993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381b820 0x55741381bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.702+0000 7fc337e00640 1 -- 172.21.15.6:0/1726304993 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557413494210 con 0x55741381c140 2024-04-26T01:50:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.703+0000 7fc337e00640 1 --2- 172.21.15.6:0/1726304993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 0x5574138264c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc3280115f0 tx=0x7fc328011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bfa28faf94aa4ae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.703+0000 7fc32d600640 1 -- 172.21.15.6:0/1726304993 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32801e070 con 0x55741381c140 2024-04-26T01:50:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.703+0000 7fc32d600640 1 -- 172.21.15.6:0/1726304993 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc32800faf0 con 0x55741381c140 2024-04-26T01:50:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.703+0000 7fc32d600640 1 -- 172.21.15.6:0/1726304993 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328019ac0 con 0x55741381c140 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.703+0000 7fc335600640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc31c0a3d10 0x7fc31c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.703+0000 7fc335600640 1 -- 172.21.15.6:0/3224140372 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fc3200d0ca0 con 0x7fc338109490 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.704+0000 7fc341e4e740 1 -- 172.21.15.6:0/1726304993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 msgr2=0x5574138264c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.704+0000 7fc341e4e740 1 --2- 172.21.15.6:0/1726304993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 0x5574138264c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc3280115f0 tx=0x7fc328011ab0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.704+0000 7fc33d000640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc31c0a3d10 0x7fc31c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.704+0000 7fc341e4e740 1 -- 172.21.15.6:0/1726304993 shutdown_connections 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.704+0000 7fc341e4e740 1 --2- 172.21.15.6:0/1726304993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557413826a00 0x557413828df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.704+0000 7fc341e4e740 1 --2- 172.21.15.6:0/1726304993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 0x5574138264c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.704+0000 7fc341e4e740 1 --2- 172.21.15.6:0/1726304993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381b820 0x55741381bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.704+0000 7fc341e4e740 1 -- 172.21.15.6:0/1726304993 >> 172.21.15.6:0/1726304993 conn(0x557413817d30 msgr2=0x557413818110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.704+0000 7fc341e4e740 1 -- 172.21.15.6:0/1726304993 shutdown_connections 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.704+0000 7fc341e4e740 1 -- 172.21.15.6:0/1726304993 wait complete. 2024-04-26T01:50:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.704+0000 7fc341e4e740 1 -- start start 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.705+0000 7fc33d000640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc31c0a3d10 0x7fc31c0a61d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc330009430 tx=0x7fc330009940 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.705+0000 7fc341e4e740 1 Processor -- start 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.705+0000 7fc341e4e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381b820 0x557413823930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.705+0000 7fc341e4e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381c140 0x557413823e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.705+0000 7fc341e4e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557413826a00 0x5574138243b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.705+0000 7fc341e4e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x557413829630 con 0x55741381c140 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.705+0000 7fc341e4e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55741373b310 con 0x55741381b820 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.705+0000 7fc341e4e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5574138297b0 con 0x557413826a00 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.705+0000 7fc341e4e740 10 client.0 _ll_register_callbacks cb 0x557413709cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.706+0000 7fc32f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381c140 0x557413823e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.706+0000 7fc32f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381c140 0x557413823e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53592/0 (socket says 172.21.15.6:53592) 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.706+0000 7fc32f400640 1 -- 172.21.15.6:0/2276194295 learned_addr learned my addr 172.21.15.6:0/2276194295 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.706+0000 7fc33d000640 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557413826a00 0x5574138243b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.706+0000 7fc337e00640 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381b820 0x557413823930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.706+0000 7fc32f400640 1 -- 172.21.15.6:0/2276194295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557413826a00 msgr2=0x5574138243b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.706+0000 7fc32f400640 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557413826a00 0x5574138243b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.706+0000 7fc32f400640 1 -- 172.21.15.6:0/2276194295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381b820 msgr2=0x557413823930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.706+0000 7fc32f400640 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381b820 0x557413823930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.706+0000 7fc32f400640 1 -- 172.21.15.6:0/2276194295 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557413560620 con 0x55741381c140 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.706+0000 7fc32f400640 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381c140 0x557413823e70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc32009cb50 tx=0x7fc3200d4280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e7948f15f7df85 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.706+0000 7fc326000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.707+0000 7fc326000640 1 -- 172.21.15.6:0/2276194295 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32009c990 con 0x55741381c140 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.707+0000 7fc326000640 1 -- 172.21.15.6:0/2276194295 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fc3200ce5f0 con 0x55741381c140 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.707+0000 7fc341e4e740 1 -- 172.21.15.6:0/2276194295 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc32800faf0 con 0x55741381c140 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.707+0000 7fc326000640 1 -- 172.21.15.6:0/2276194295 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc32009c520 con 0x55741381c140 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.707+0000 7fc341e4e740 10 client.15570 fetch_fsmap learned FSMap version 32 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.707+0000 7fc341e4e740 1 -- 172.21.15.6:0/2276194295 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557413835890 con 0x55741381c140 2024-04-26T01:50:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.707+0000 7fc326000640 1 -- 172.21.15.6:0/2276194295 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc32800faf0 con 0x55741381c140 2024-04-26T01:50:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.708+0000 7fc326000640 1 -- 172.21.15.6:0/2276194295 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fc3200d6070 con 0x55741381c140 2024-04-26T01:50:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.708+0000 7fc341e4e740 10 client.15570 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T01:50:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.708+0000 7fc341e4e740 10 client.15570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.708+0000 7fc341e4e740 20 client.15570 populate_metadata read hostname 'smithi006' 2024-04-26T01:50:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.708+0000 7fc341e4e740 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x557413836df0 0x557413839280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.708+0000 7fc341e4e740 4 client.15570 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.708+0000 7fc341e4e740 1 -- 172.21.15.6:0/2276194295 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574137958b0 con 0x557413836df0 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.708+0000 7fc337e00640 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x557413836df0 0x557413839280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.709+0000 7fc337e00640 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x557413836df0 0x557413839280 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cf5f7c21ce44036 server_cookie=9350570831c4a2f6 in_seq=0 out_seq=0 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.709+0000 7fc326000640 10 client.15570 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.709+0000 7fc326000640 1 -- 172.21.15.6:0/2276194295 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5574137958b0 con 0x557413836df0 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.709+0000 7fc326000640 10 client.15570 handle_command_reply: tid=0 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.709+0000 7fc341e4e740 1 client.15570 shutdown 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.710+0000 7fc341e4e740 1 -- 172.21.15.6:0/2276194295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381c140 msgr2=0x557413823e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.710+0000 7fc341e4e740 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381c140 0x557413823e70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc32009cb50 tx=0x7fc3200d4280 comp rx=0 tx=0).stop 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.710+0000 7fc341e4e740 1 -- 172.21.15.6:0/2276194295 shutdown_connections 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.710+0000 7fc341e4e740 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x557413836df0 0x557413839280 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.710+0000 7fc341e4e740 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381c140 0x557413823e70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.710+0000 7fc341e4e740 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557413826a00 0x5574138243b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.710+0000 7fc341e4e740 1 --2- 172.21.15.6:0/2276194295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381b820 0x557413823930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.710+0000 7fc326000640 0 client.15570 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.710+0000 7fc341e4e740 1 -- 172.21.15.6:0/2276194295 >> 172.21.15.6:0/2276194295 conn(0x557413817d30 msgr2=0x55741381dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.710+0000 7fc341e4e740 1 -- 172.21.15.6:0/2276194295 shutdown_connections 2024-04-26T01:50:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.710+0000 7fc341e4e740 1 -- 172.21.15.6:0/2276194295 wait complete. 2024-04-26T01:50:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.710+0000 7fc341e4e740 20 client.15570 trim_cache size 0 max 16384 2024-04-26T01:50:09.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.727+0000 7fc341e4e740 1 Processor -- start 2024-04-26T01:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.727+0000 7fc341e4e740 1 -- start start 2024-04-26T01:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.727+0000 7fc341e4e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381b820 0x557413817610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.727+0000 7fc341e4e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 0x5574138324f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.727+0000 7fc341e4e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557413826a00 0x55741383eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.727+0000 7fc341e4e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381b27f0 con 0x557413826a00 2024-04-26T01:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.727+0000 7fc341e4e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381b0060 con 0x55741381b820 2024-04-26T01:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.727+0000 7fc341e4e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381adeb0 con 0x55741381c140 2024-04-26T01:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.727+0000 7fc32f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 0x5574138324f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.727+0000 7fc33d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557413826a00 0x55741383eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.727+0000 7fc32f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 0x5574138324f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50048/0 (socket says 172.21.15.6:50048) 2024-04-26T01:50:09.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.727+0000 7fc32f400640 1 -- 172.21.15.6:0/2401615954 learned_addr learned my addr 172.21.15.6:0/2401615954 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.728+0000 7fc337e00640 1 --2- 172.21.15.6:0/2401615954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381b820 0x557413817610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.728+0000 7fc32f400640 1 -- 172.21.15.6:0/2401615954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381b820 msgr2=0x557413817610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.728+0000 7fc32f400640 1 --2- 172.21.15.6:0/2401615954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381b820 0x557413817610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.728+0000 7fc32f400640 1 -- 172.21.15.6:0/2401615954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557413826a00 msgr2=0x55741383eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.728+0000 7fc32f400640 1 --2- 172.21.15.6:0/2401615954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557413826a00 0x55741383eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.728+0000 7fc32f400640 1 -- 172.21.15.6:0/2401615954 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574138122a0 con 0x55741381c140 2024-04-26T01:50:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.728+0000 7fc32f400640 1 --2- 172.21.15.6:0/2401615954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 0x5574138324f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc320099590 tx=0x7fc32009b0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37309731d16b065d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.728+0000 7fc32cc00640 1 -- 172.21.15.6:0/2401615954 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32009c700 con 0x55741381c140 2024-04-26T01:50:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.728+0000 7fc32cc00640 1 -- 172.21.15.6:0/2401615954 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3200046c0 con 0x55741381c140 2024-04-26T01:50:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.728+0000 7fc32cc00640 1 -- 172.21.15.6:0/2401615954 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3200bd0b0 con 0x55741381c140 2024-04-26T01:50:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.729+0000 7fc341e4e740 1 -- 172.21.15.6:0/2401615954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 msgr2=0x5574138324f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.729+0000 7fc341e4e740 1 --2- 172.21.15.6:0/2401615954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 0x5574138324f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc320099590 tx=0x7fc32009b0d0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.729+0000 7fc341e4e740 1 -- 172.21.15.6:0/2401615954 shutdown_connections 2024-04-26T01:50:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.729+0000 7fc341e4e740 1 --2- 172.21.15.6:0/2401615954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557413826a00 0x55741383eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.729+0000 7fc341e4e740 1 --2- 172.21.15.6:0/2401615954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55741381c140 0x5574138324f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.729+0000 7fc341e4e740 1 --2- 172.21.15.6:0/2401615954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381b820 0x557413817610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.729+0000 7fc341e4e740 1 -- 172.21.15.6:0/2401615954 >> 172.21.15.6:0/2401615954 conn(0x557413817d30 msgr2=0x557413824180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.729+0000 7fc341e4e740 1 -- 172.21.15.6:0/2401615954 shutdown_connections 2024-04-26T01:50:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.729+0000 7fc341e4e740 1 -- 172.21.15.6:0/2401615954 wait complete. 2024-04-26T01:50:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.730+0000 7fc341e4e740 1 -- start start 2024-04-26T01:50:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.730+0000 7fc341e4e740 1 Processor -- start 2024-04-26T01:50:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.730+0000 7fc341e4e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381b820 0x55741382caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.730+0000 7fc341e4e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381c140 0x55741382cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.730+0000 7fc341e4e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557413826a00 0x55741389c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.730+0000 7fc341e4e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381ae520 con 0x55741381b820 2024-04-26T01:50:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.730+0000 7fc341e4e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381af990 con 0x55741381c140 2024-04-26T01:50:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.730+0000 7fc341e4e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381aebf0 con 0x557413826a00 2024-04-26T01:50:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.730+0000 7fc341e4e740 10 client.0 _ll_register_callbacks cb 0x7fc3200048b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc32f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381c140 0x55741382cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc32f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381c140 0x55741382cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34784/0 (socket says 172.21.15.6:34784) 2024-04-26T01:50:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc337e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381b820 0x55741382caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc32f400640 1 -- 172.21.15.6:0/3101151355 learned_addr learned my addr 172.21.15.6:0/3101151355 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc33d000640 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557413826a00 0x55741389c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc32f400640 1 -- 172.21.15.6:0/3101151355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557413826a00 msgr2=0x55741389c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc32f400640 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557413826a00 0x55741389c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc32f400640 1 -- 172.21.15.6:0/3101151355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381b820 msgr2=0x55741382caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc32f400640 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381b820 0x55741382caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc32f400640 1 -- 172.21.15.6:0/3101151355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55741389c910 con 0x55741381c140 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc33d000640 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557413826a00 0x55741389c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc32f400640 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381c140 0x55741382cfe0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc320009360 tx=0x7fc320005760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9ff238232ee8524 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc326000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc326000640 1 -- 172.21.15.6:0/3101151355 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32009c700 con 0x55741381c140 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc326000640 1 -- 172.21.15.6:0/3101151355 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fc3200ce5f0 con 0x55741381c140 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.731+0000 7fc341e4e740 1 -- 172.21.15.6:0/3101151355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55741378f320 con 0x55741381c140 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.732+0000 7fc326000640 1 -- 172.21.15.6:0/3101151355 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc32009c990 con 0x55741381c140 2024-04-26T01:50:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.732+0000 7fc341e4e740 10 client.25700 fetch_fsmap learned FSMap version 32 2024-04-26T01:50:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.732+0000 7fc341e4e740 1 -- 172.21.15.6:0/3101151355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55741389db50 con 0x55741381c140 2024-04-26T01:50:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.732+0000 7fc326000640 1 -- 172.21.15.6:0/3101151355 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55741378f320 con 0x55741381c140 2024-04-26T01:50:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.732+0000 7fc326000640 1 -- 172.21.15.6:0/3101151355 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fc3200d6050 con 0x55741381c140 2024-04-26T01:50:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.732+0000 7fc341e4e740 10 client.25700 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T01:50:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.732+0000 7fc341e4e740 10 client.25700 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.732+0000 7fc341e4e740 20 client.25700 populate_metadata read hostname 'smithi006' 2024-04-26T01:50:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.732+0000 7fc341e4e740 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55741389e060 0x5574138a0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.732+0000 7fc341e4e740 4 client.25700 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.732+0000 7fc341e4e740 1 -- 172.21.15.6:0/3101151355 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557413789190 con 0x55741389e060 2024-04-26T01:50:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.733+0000 7fc337e00640 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55741389e060 0x5574138a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.733+0000 7fc337e00640 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55741389e060 0x5574138a0480 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb06c6fa87cb45f server_cookie=d21da7cc69738ea1 in_seq=0 out_seq=0 2024-04-26T01:50:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.733+0000 7fc326000640 10 client.25700 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.765+0000 7fc326000640 1 -- 172.21.15.6:0/3101151355 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557413789190 con 0x55741389e060 2024-04-26T01:50:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.765+0000 7fc326000640 10 client.25700 handle_command_reply: tid=0 2024-04-26T01:50:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.765+0000 7fc341e4e740 1 client.25700 shutdown 2024-04-26T01:50:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.766+0000 7fc341e4e740 1 -- 172.21.15.6:0/3101151355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381c140 msgr2=0x55741382cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.766+0000 7fc341e4e740 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381c140 0x55741382cfe0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc320009360 tx=0x7fc320005760 comp rx=0 tx=0).stop 2024-04-26T01:50:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.766+0000 7fc341e4e740 1 -- 172.21.15.6:0/3101151355 shutdown_connections 2024-04-26T01:50:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.766+0000 7fc341e4e740 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55741389e060 0x5574138a0480 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.766+0000 7fc341e4e740 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557413826a00 0x55741389c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.766+0000 7fc341e4e740 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741381c140 0x55741382cfe0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.766+0000 7fc341e4e740 1 --2- 172.21.15.6:0/3101151355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55741381b820 0x55741382caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.766+0000 7fc341e4e740 1 -- 172.21.15.6:0/3101151355 >> 172.21.15.6:0/3101151355 conn(0x557413817d30 msgr2=0x55741381d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:09.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.766+0000 7fc326000640 0 client.25700 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:09.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.766+0000 7fc341e4e740 1 -- 172.21.15.6:0/3101151355 shutdown_connections 2024-04-26T01:50:09.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.766+0000 7fc341e4e740 1 -- 172.21.15.6:0/3101151355 wait complete. 2024-04-26T01:50:09.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.766+0000 7fc341e4e740 20 client.25700 trim_cache size 0 max 16384 2024-04-26T01:50:09.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:50:09.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:50:09.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:50:09.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:50:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.767+0000 7fc33f000640 1 -- 172.21.15.6:0/3224140372 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc31c0a3d10 msgr2=0x7fc31c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.767+0000 7fc33f000640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc31c0a3d10 0x7fc31c0a61d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc330009430 tx=0x7fc330009940 comp rx=0 tx=0).stop 2024-04-26T01:50:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.767+0000 7fc33f000640 1 -- 172.21.15.6:0/3224140372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc338109490 msgr2=0x7fc3381a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.767+0000 7fc33f000640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc338109490 0x7fc3381a0bb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc32000e9c0 tx=0x7fc32000ee80 comp rx=0 tx=0).stop 2024-04-26T01:50:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.767+0000 7fc33f000640 1 -- 172.21.15.6:0/3224140372 shutdown_connections 2024-04-26T01:50:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.767+0000 7fc33f000640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc31c0a3d10 0x7fc31c0a61d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.767+0000 7fc33f000640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc33819a7d0 0x7fc33819ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.767+0000 7fc33f000640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc338109490 0x7fc3381a0bb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.767+0000 7fc33f000640 1 --2- 172.21.15.6:0/3224140372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3381043c0 0x7fc3381a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:09.769 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.767+0000 7fc33f000640 1 -- 172.21.15.6:0/3224140372 >> 172.21.15.6:0/3224140372 conn(0x7fc3380ffc40 msgr2=0x7fc338101010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:09.769 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.767+0000 7fc33f000640 1 -- 172.21.15.6:0/3224140372 shutdown_connections 2024-04-26T01:50:09.769 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:09.767+0000 7fc33f000640 1 -- 172.21.15.6:0/3224140372 wait complete. 2024-04-26T01:50:10.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:10 smithi188 ceph-mon[32751]: pgmap v132: 97 pgs: 97 active+clean; 42 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 904 KiB/s wr, 175 op/s 2024-04-26T01:50:10.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:10 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:10.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:10 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:50:10.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:10 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:50:10.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:10 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:50:10.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:10 smithi029 ceph-mon[32775]: pgmap v132: 97 pgs: 97 active+clean; 42 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 904 KiB/s wr, 175 op/s 2024-04-26T01:50:10.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:10 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:10.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:10 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:50:10.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:10 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:50:10.441 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:10 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:50:10.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:10 smithi006 ceph-mon[28419]: pgmap v132: 97 pgs: 97 active+clean; 42 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 904 KiB/s wr, 175 op/s 2024-04-26T01:50:10.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:10 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:10.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:10 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:50:10.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:10 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:50:10.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:10 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:50:10.569 INFO:tasks.workunit.client.0.smithi006.stdout: 1 109008 2.48 MB/sec execute 34 sec latency 1437.559 ms 2024-04-26T01:50:11.570 INFO:tasks.workunit.client.0.smithi006.stdout: 1 109008 2.41 MB/sec execute 35 sec latency 2437.683 ms 2024-04-26T01:50:12.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:12 smithi188 ceph-mon[32751]: pgmap v133: 97 pgs: 97 active+clean; 41 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.2 MiB/s wr, 233 op/s 2024-04-26T01:50:12.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:12 smithi029 ceph-mon[32775]: pgmap v133: 97 pgs: 97 active+clean; 41 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.2 MiB/s wr, 233 op/s 2024-04-26T01:50:12.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:12 smithi006 ceph-mon[28419]: pgmap v133: 97 pgs: 97 active+clean; 41 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.2 MiB/s wr, 233 op/s 2024-04-26T01:50:12.570 INFO:tasks.workunit.client.0.smithi006.stdout: 1 109008 2.34 MB/sec execute 36 sec latency 3437.849 ms 2024-04-26T01:50:13.570 INFO:tasks.workunit.client.0.smithi006.stdout: 1 109008 2.28 MB/sec execute 37 sec latency 4437.948 ms 2024-04-26T01:50:14.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:14 smithi029 ceph-mon[32775]: pgmap v134: 97 pgs: 97 active+clean; 41 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 745 KiB/s wr, 128 op/s 2024-04-26T01:50:14.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:14 smithi006 ceph-mon[28419]: pgmap v134: 97 pgs: 97 active+clean; 41 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 745 KiB/s wr, 128 op/s 2024-04-26T01:50:14.570 INFO:tasks.workunit.client.0.smithi006.stdout: 1 109008 2.22 MB/sec execute 38 sec latency 5438.080 ms 2024-04-26T01:50:14.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:14 smithi188 ceph-mon[32751]: pgmap v134: 97 pgs: 97 active+clean; 41 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 745 KiB/s wr, 128 op/s 2024-04-26T01:50:15.570 INFO:tasks.workunit.client.0.smithi006.stdout: 1 109008 2.16 MB/sec execute 39 sec latency 6438.184 ms 2024-04-26T01:50:16.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:16 smithi029 ceph-mon[32775]: pgmap v135: 97 pgs: 97 active+clean; 41 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 745 KiB/s wr, 128 op/s 2024-04-26T01:50:16.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:16 smithi006 ceph-mon[28419]: pgmap v135: 97 pgs: 97 active+clean; 41 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 745 KiB/s wr, 128 op/s 2024-04-26T01:50:16.570 INFO:tasks.workunit.client.0.smithi006.stdout: 1 109008 2.11 MB/sec execute 40 sec latency 7438.364 ms 2024-04-26T01:50:16.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:16 smithi188 ceph-mon[32751]: pgmap v135: 97 pgs: 97 active+clean; 41 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 745 KiB/s wr, 128 op/s 2024-04-26T01:50:17.570 INFO:tasks.workunit.client.0.smithi006.stdout: 1 109008 2.05 MB/sec execute 41 sec latency 8438.494 ms 2024-04-26T01:50:18.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:18 smithi029 ceph-mon[32775]: pgmap v136: 97 pgs: 97 active+clean; 43 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 926 KiB/s wr, 137 op/s 2024-04-26T01:50:18.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:18 smithi006 ceph-mon[28419]: pgmap v136: 97 pgs: 97 active+clean; 43 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 926 KiB/s wr, 137 op/s 2024-04-26T01:50:18.570 INFO:tasks.workunit.client.0.smithi006.stdout: 1 109008 2.00 MB/sec execute 42 sec latency 9438.654 ms 2024-04-26T01:50:18.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:18 smithi188 ceph-mon[32751]: pgmap v136: 97 pgs: 97 active+clean; 43 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 926 KiB/s wr, 137 op/s 2024-04-26T01:50:19.571 INFO:tasks.workunit.client.0.smithi006.stdout: 1 109962 2.18 MB/sec execute 43 sec latency 9856.706 ms 2024-04-26T01:50:20.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:20 smithi029 ceph-mon[32775]: pgmap v137: 97 pgs: 97 active+clean; 43 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 480 KiB/s wr, 67 op/s 2024-04-26T01:50:20.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:20 smithi006 ceph-mon[28419]: pgmap v137: 97 pgs: 97 active+clean; 43 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 480 KiB/s wr, 67 op/s 2024-04-26T01:50:20.571 INFO:tasks.workunit.client.0.smithi006.stdout: 1 110699 2.17 MB/sec execute 44 sec latency 13.218 ms 2024-04-26T01:50:20.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:20 smithi188 ceph-mon[32751]: pgmap v137: 97 pgs: 97 active+clean; 43 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 480 KiB/s wr, 67 op/s 2024-04-26T01:50:21.571 INFO:tasks.workunit.client.0.smithi006.stdout: 1 112377 2.27 MB/sec execute 45 sec latency 15.509 ms 2024-04-26T01:50:22.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:22 smithi029 ceph-mon[32775]: pgmap v138: 97 pgs: 97 active+clean; 45 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 663 KiB/s wr, 99 op/s 2024-04-26T01:50:22.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:22 smithi006 ceph-mon[28419]: pgmap v138: 97 pgs: 97 active+clean; 45 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 663 KiB/s wr, 99 op/s 2024-04-26T01:50:22.571 INFO:tasks.workunit.client.0.smithi006.stdout: 1 113820 2.48 MB/sec execute 46 sec latency 15.679 ms 2024-04-26T01:50:22.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:22 smithi188 ceph-mon[32751]: pgmap v138: 97 pgs: 97 active+clean; 45 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 663 KiB/s wr, 99 op/s 2024-04-26T01:50:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.912+0000 7f7637a00640 1 -- 172.21.15.6:0/1508355462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f763010c590 msgr2=0x7f763010e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.913+0000 7f7637a00640 1 --2- 172.21.15.6:0/1508355462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f763010c590 0x7f763010e9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7628009ca0 tx=0x7f762802a950 comp rx=0 tx=0).stop 2024-04-26T01:50:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.913+0000 7f7637a00640 1 -- 172.21.15.6:0/1508355462 shutdown_connections 2024-04-26T01:50:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.913+0000 7f7637a00640 1 --2- 172.21.15.6:0/1508355462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f763010c590 0x7f763010e9c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.913+0000 7f7637a00640 1 --2- 172.21.15.6:0/1508355462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7630103ae0 0x7f763010bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.913+0000 7f7637a00640 1 --2- 172.21.15.6:0/1508355462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76301031c0 0x7f76301035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.913+0000 7f7637a00640 1 -- 172.21.15.6:0/1508355462 >> 172.21.15.6:0/1508355462 conn(0x7f7630077f70 msgr2=0x7f7630078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.913+0000 7f7637a00640 1 -- 172.21.15.6:0/1508355462 shutdown_connections 2024-04-26T01:50:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.913+0000 7f7637a00640 1 -- 172.21.15.6:0/1508355462 wait complete. 2024-04-26T01:50:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.914+0000 7f7637a00640 1 Processor -- start 2024-04-26T01:50:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.914+0000 7f7637a00640 1 -- start start 2024-04-26T01:50:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7637a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76301031c0 0x7f76301a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7637a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7630103ae0 0x7f76301a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7637a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f763010c590 0x7f763019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7637a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f76301121b0 con 0x7f7630103ae0 2024-04-26T01:50:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7637a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7630112030 con 0x7f76301031c0 2024-04-26T01:50:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7637a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7630112330 con 0x7f763010c590 2024-04-26T01:50:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7635000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76301031c0 0x7f76301a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7635000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76301031c0 0x7f76301a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:33340/0 (socket says 172.21.15.6:33340) 2024-04-26T01:50:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7635000640 1 -- 172.21.15.6:0/200881804 learned_addr learned my addr 172.21.15.6:0/200881804 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7635a00640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f763010c590 0x7f763019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f762fe00640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7630103ae0 0x7f76301a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7635000640 1 -- 172.21.15.6:0/200881804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f763010c590 msgr2=0x7f763019a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7635000640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f763010c590 0x7f763019a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7635000640 1 -- 172.21.15.6:0/200881804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7630103ae0 msgr2=0x7f76301a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.915+0000 7f7635000640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7630103ae0 0x7f76301a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.916+0000 7f7635000640 1 -- 172.21.15.6:0/200881804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763019afc0 con 0x7f76301031c0 2024-04-26T01:50:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.916+0000 7f7635000640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76301031c0 0x7f76301a0630 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f762400d950 tx=0x7f762400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.916+0000 7f762d600640 1 -- 172.21.15.6:0/200881804 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7624017030 con 0x7f76301031c0 2024-04-26T01:50:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.916+0000 7f7637a00640 1 -- 172.21.15.6:0/200881804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f763019b2b0 con 0x7f76301031c0 2024-04-26T01:50:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.916+0000 7f762d600640 1 -- 172.21.15.6:0/200881804 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7624004020 con 0x7f76301031c0 2024-04-26T01:50:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.916+0000 7f762d600640 1 -- 172.21.15.6:0/200881804 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76240047a0 con 0x7f76301031c0 2024-04-26T01:50:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.916+0000 7f7637a00640 1 -- 172.21.15.6:0/200881804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76301a7480 con 0x7f76301031c0 2024-04-26T01:50:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.921+0000 7f763a9eb740 1 Processor -- start 2024-04-26T01:50:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.921+0000 7f763a9eb740 1 -- start start 2024-04-26T01:50:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.921+0000 7f763a9eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 0x556e04074c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.921+0000 7f763a9eb740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 0x556e0407f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.921+0000 7f763a9eb740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e04081df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.921+0000 7f763a9eb740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x556e03fda900 con 0x556e0407fa00 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.921+0000 7f763a9eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556e03f80ef0 con 0x556e04074820 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.921+0000 7f763a9eb740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556e03fd68c0 con 0x556e04075140 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.922+0000 7f7635a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 0x556e04074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.922+0000 7f7635a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 0x556e04074c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:33348/0 (socket says 172.21.15.6:33348) 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.922+0000 7f7635a00640 1 -- 172.21.15.6:0/109019139 learned_addr learned my addr 172.21.15.6:0/109019139 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.922+0000 7f762fe00640 1 --2- 172.21.15.6:0/109019139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e04081df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.922+0000 7f762d600640 1 -- 172.21.15.6:0/200881804 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f7624009a30 con 0x7f76301031c0 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.922+0000 7f7635a00640 1 -- 172.21.15.6:0/109019139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 msgr2=0x556e0407f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.922+0000 7f7635a00640 1 --2- 172.21.15.6:0/109019139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 0x556e0407f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.922+0000 7f7635a00640 1 -- 172.21.15.6:0/109019139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 msgr2=0x556e04081df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.922+0000 7f7635a00640 1 --2- 172.21.15.6:0/109019139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e04081df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.922+0000 7f7635a00640 1 -- 172.21.15.6:0/109019139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e03ced210 con 0x556e04074820 2024-04-26T01:50:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.923+0000 7f7635a00640 1 --2- 172.21.15.6:0/109019139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 0x556e04074c00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7628009ca0 tx=0x7f7628005fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be574585add07b2b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.923+0000 7f761e000640 1 -- 172.21.15.6:0/109019139 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7628048070 con 0x556e04074820 2024-04-26T01:50:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.923+0000 7f761e000640 1 -- 172.21.15.6:0/109019139 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7628003e20 con 0x556e04074820 2024-04-26T01:50:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.923+0000 7f761e000640 1 -- 172.21.15.6:0/109019139 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7628034040 con 0x556e04074820 2024-04-26T01:50:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.924+0000 7f763a9eb740 1 -- 172.21.15.6:0/109019139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 msgr2=0x556e04074c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.924+0000 7f763a9eb740 1 --2- 172.21.15.6:0/109019139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 0x556e04074c00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7628009ca0 tx=0x7f7628005fa0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.924+0000 7f763a9eb740 1 -- 172.21.15.6:0/109019139 shutdown_connections 2024-04-26T01:50:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.924+0000 7f763a9eb740 1 --2- 172.21.15.6:0/109019139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e04081df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.924+0000 7f763a9eb740 1 --2- 172.21.15.6:0/109019139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 0x556e0407f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.924+0000 7f763a9eb740 1 --2- 172.21.15.6:0/109019139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 0x556e04074c00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.924+0000 7f763a9eb740 1 -- 172.21.15.6:0/109019139 >> 172.21.15.6:0/109019139 conn(0x556e04070d30 msgr2=0x556e04071110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.924+0000 7f763a9eb740 1 -- 172.21.15.6:0/109019139 shutdown_connections 2024-04-26T01:50:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.924+0000 7f763a9eb740 1 -- 172.21.15.6:0/109019139 wait complete. 2024-04-26T01:50:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.924+0000 7f762d600640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f76180a3ca0 0x7f76180a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.925+0000 7f762d600640 1 -- 172.21.15.6:0/200881804 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f76240d01c0 con 0x7f76301031c0 2024-04-26T01:50:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.925+0000 7f763a9eb740 1 -- start start 2024-04-26T01:50:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.925+0000 7f762fe00640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f76180a3ca0 0x7f76180a6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.925+0000 7f763a9eb740 1 Processor -- start 2024-04-26T01:50:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.925+0000 7f763a9eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 0x556e0407c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f763a9eb740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 0x556e0407cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f763a9eb740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e0407d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f763a9eb740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x556e040825d0 con 0x556e0407fa00 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f763a9eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556e03f94310 con 0x556e04074820 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f763a9eb740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556e04082750 con 0x556e04075140 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f762fe00640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f76180a3ca0 0x7f76180a6160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x556e04072150 tx=0x7f7620005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f763a9eb740 10 client.0 _ll_register_callbacks cb 0x556e03f62cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f7635000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 0x556e0407c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f7635000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 0x556e0407c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:33354/0 (socket says 172.21.15.6:33354) 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f7635000640 1 -- 172.21.15.6:0/4019001705 learned_addr learned my addr 172.21.15.6:0/4019001705 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f762fe00640 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 0x556e0407cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f7635a00640 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e0407d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f7635000640 1 -- 172.21.15.6:0/4019001705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 msgr2=0x556e0407cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f7635000640 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 0x556e0407cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f7635000640 1 -- 172.21.15.6:0/4019001705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 msgr2=0x556e0407d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f7635000640 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e0407d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.926+0000 7f7635000640 1 -- 172.21.15.6:0/4019001705 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e03db9620 con 0x556e04074820 2024-04-26T01:50:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.927+0000 7f7635000640 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 0x556e0407c9b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7624098540 tx=0x7f762409bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cc19d0da35651ab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.927+0000 7f7612a00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.927+0000 7f7612a00640 1 -- 172.21.15.6:0/4019001705 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76240a0490 con 0x556e04074820 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.927+0000 7f7612a00640 1 -- 172.21.15.6:0/4019001705 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f76240a0130 con 0x556e04074820 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.927+0000 7f763a9eb740 1 -- 172.21.15.6:0/4019001705 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7628003e20 con 0x556e04074820 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.927+0000 7f7612a00640 1 -- 172.21.15.6:0/4019001705 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f76240ba050 con 0x556e04074820 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.927+0000 7f763a9eb740 10 client.25712 fetch_fsmap learned FSMap version 32 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.927+0000 7f763a9eb740 1 -- 172.21.15.6:0/4019001705 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556e0408e940 con 0x556e04074820 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.927+0000 7f7612a00640 1 -- 172.21.15.6:0/4019001705 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7628003e20 con 0x556e04074820 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.928+0000 7f7612a00640 1 -- 172.21.15.6:0/4019001705 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f76240cdb10 con 0x556e04074820 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.928+0000 7f763a9eb740 10 client.25712 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.928+0000 7f763a9eb740 10 client.25712 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.928+0000 7f763a9eb740 20 client.25712 populate_metadata read hostname 'smithi006' 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.928+0000 7f763a9eb740 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556e0408fe50 0x556e04092240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.928+0000 7f763a9eb740 4 client.25712 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.928+0000 7f763a9eb740 1 -- 172.21.15.6:0/4019001705 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e03fee8b0 con 0x556e0408fe50 2024-04-26T01:50:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.928+0000 7f762fe00640 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556e0408fe50 0x556e04092240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.929+0000 7f762fe00640 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556e0408fe50 0x556e04092240 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae616187bd1147ba server_cookie=2d6a429480c9ee4f in_seq=0 out_seq=0 2024-04-26T01:50:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.929+0000 7f7612a00640 10 client.25712 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f7612a00640 1 -- 172.21.15.6:0/4019001705 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e03fee8b0 con 0x556e0408fe50 2024-04-26T01:50:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f7612a00640 10 client.25712 handle_command_reply: tid=0 2024-04-26T01:50:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f763a9eb740 1 client.25712 shutdown 2024-04-26T01:50:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f763a9eb740 1 -- 172.21.15.6:0/4019001705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 msgr2=0x556e0407c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f763a9eb740 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 0x556e0407c9b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7624098540 tx=0x7f762409bb70 comp rx=0 tx=0).stop 2024-04-26T01:50:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f763a9eb740 1 -- 172.21.15.6:0/4019001705 shutdown_connections 2024-04-26T01:50:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f763a9eb740 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556e0408fe50 0x556e04092240 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f763a9eb740 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e0407d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f763a9eb740 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 0x556e0407cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f763a9eb740 1 --2- 172.21.15.6:0/4019001705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04074820 0x556e0407c9b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f7612a00640 0 client.25712 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f763a9eb740 1 -- 172.21.15.6:0/4019001705 >> 172.21.15.6:0/4019001705 conn(0x556e04070d30 msgr2=0x556e04076a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f763a9eb740 1 -- 172.21.15.6:0/4019001705 shutdown_connections 2024-04-26T01:50:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f763a9eb740 1 -- 172.21.15.6:0/4019001705 wait complete. 2024-04-26T01:50:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.930+0000 7f763a9eb740 20 client.25712 trim_cache size 0 max 16384 2024-04-26T01:50:22.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.952+0000 7f763a9eb740 1 Processor -- start 2024-04-26T01:50:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.952+0000 7f763a9eb740 1 -- start start 2024-04-26T01:50:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f763a9eb740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04074820 0x556e0408b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f763a9eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04075140 0x556e040771e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f763a9eb740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e04097b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f763a9eb740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f76301b26f0 con 0x556e0407fa00 2024-04-26T01:50:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f763a9eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f76301aff60 con 0x556e04075140 2024-04-26T01:50:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f763a9eb740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f76301addb0 con 0x556e04074820 2024-04-26T01:50:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f762fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04075140 0x556e040771e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f7635000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04074820 0x556e0408b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f762fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04075140 0x556e040771e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:33360/0 (socket says 172.21.15.6:33360) 2024-04-26T01:50:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f7635000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04074820 0x556e0408b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:57614/0 (socket says 172.21.15.6:57614) 2024-04-26T01:50:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f762fe00640 1 -- 172.21.15.6:0/1151350438 learned_addr learned my addr 172.21.15.6:0/1151350438 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f7635a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e04097b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f762fe00640 1 -- 172.21.15.6:0/1151350438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04074820 msgr2=0x556e0408b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f762fe00640 1 --2- 172.21.15.6:0/1151350438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04074820 0x556e0408b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f762fe00640 1 -- 172.21.15.6:0/1151350438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 msgr2=0x556e04097b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f762fe00640 1 --2- 172.21.15.6:0/1151350438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e04097b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.953+0000 7f762fe00640 1 -- 172.21.15.6:0/1151350438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e04092870 con 0x556e04075140 2024-04-26T01:50:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.954+0000 7f7635000640 1 --2- 172.21.15.6:0/1151350438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04074820 0x556e0408b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.954+0000 7f7635a00640 1 --2- 172.21.15.6:0/1151350438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e04097b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.954+0000 7f762fe00640 1 --2- 172.21.15.6:0/1151350438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04075140 0x556e040771e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7620013010 tx=0x7f7620017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a98a3f0cc041399 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.954+0000 7f761d600640 1 -- 172.21.15.6:0/1151350438 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76200043c0 con 0x556e04075140 2024-04-26T01:50:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.954+0000 7f761d600640 1 -- 172.21.15.6:0/1151350438 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f762002b8d0 con 0x556e04075140 2024-04-26T01:50:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.954+0000 7f761d600640 1 -- 172.21.15.6:0/1151350438 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76200047a0 con 0x556e04075140 2024-04-26T01:50:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.955+0000 7f763a9eb740 1 -- 172.21.15.6:0/1151350438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04075140 msgr2=0x556e040771e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.955+0000 7f763a9eb740 1 --2- 172.21.15.6:0/1151350438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04075140 0x556e040771e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7620013010 tx=0x7f7620017000 comp rx=0 tx=0).stop 2024-04-26T01:50:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.955+0000 7f763a9eb740 1 -- 172.21.15.6:0/1151350438 shutdown_connections 2024-04-26T01:50:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.955+0000 7f763a9eb740 1 --2- 172.21.15.6:0/1151350438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e0407fa00 0x556e04097b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.955+0000 7f763a9eb740 1 --2- 172.21.15.6:0/1151350438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e04075140 0x556e040771e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.955+0000 7f763a9eb740 1 --2- 172.21.15.6:0/1151350438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04074820 0x556e0408b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.955+0000 7f763a9eb740 1 -- 172.21.15.6:0/1151350438 >> 172.21.15.6:0/1151350438 conn(0x556e04070d30 msgr2=0x556e0407d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.955+0000 7f763a9eb740 1 -- 172.21.15.6:0/1151350438 shutdown_connections 2024-04-26T01:50:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.955+0000 7f763a9eb740 1 -- 172.21.15.6:0/1151350438 wait complete. 2024-04-26T01:50:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.956+0000 7f763a9eb740 1 -- start start 2024-04-26T01:50:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.956+0000 7f763a9eb740 1 Processor -- start 2024-04-26T01:50:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.956+0000 7f763a9eb740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e04074820 0x556e04087b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.956+0000 7f763a9eb740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 0x556e04084f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f763a9eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e0407fa00 0x556e040854d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f763a9eb740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f76301ae420 con 0x556e04074820 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f763a9eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f76301af890 con 0x556e0407fa00 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f763a9eb740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f76301aeaf0 con 0x556e04075140 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f763a9eb740 10 client.0 _ll_register_callbacks cb 0x7f762409bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f7635000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e04074820 0x556e04087b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f762fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 0x556e04084f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f7635000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e04074820 0x556e04087b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60518/0 (socket says 172.21.15.6:60518) 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f7635000640 1 -- 172.21.15.6:0/805618554 learned_addr learned my addr 172.21.15.6:0/805618554 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f7635a00640 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e0407fa00 0x556e040854d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f762fe00640 1 -- 172.21.15.6:0/805618554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e0407fa00 msgr2=0x556e040854d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f762fe00640 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e0407fa00 0x556e040854d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f762fe00640 1 -- 172.21.15.6:0/805618554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e04074820 msgr2=0x556e04087b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f762fe00640 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e04074820 0x556e04087b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f762fe00640 1 -- 172.21.15.6:0/805618554 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e04085ac0 con 0x556e04075140 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.957+0000 7f7635a00640 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e0407fa00 0x556e040854d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.958+0000 7f762fe00640 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 0x556e04084f90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x556e0406c960 tx=0x7f7620012ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=448b12827aad6e21 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.958+0000 7f7612a00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.958+0000 7f7612a00640 1 -- 172.21.15.6:0/805618554 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620038020 con 0x556e04075140 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.958+0000 7f763a9eb740 1 -- 172.21.15.6:0/805618554 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f762002b8d0 con 0x556e04075140 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.958+0000 7f7612a00640 1 -- 172.21.15.6:0/805618554 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f7620018040 con 0x556e04075140 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.958+0000 7f7612a00640 1 -- 172.21.15.6:0/805618554 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7620043050 con 0x556e04075140 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.958+0000 7f763a9eb740 10 client.25765 fetch_fsmap learned FSMap version 32 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.958+0000 7f763a9eb740 1 -- 172.21.15.6:0/805618554 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556e040f6190 con 0x556e04075140 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.959+0000 7f7612a00640 1 -- 172.21.15.6:0/805618554 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f762002b8d0 con 0x556e04075140 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.959+0000 7f7612a00640 1 -- 172.21.15.6:0/805618554 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f7620034930 con 0x556e04075140 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.959+0000 7f763a9eb740 10 client.25765 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.959+0000 7f763a9eb740 10 client.25765 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:22.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.959+0000 7f763a9eb740 20 client.25765 populate_metadata read hostname 'smithi006' 2024-04-26T01:50:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.959+0000 7f763a9eb740 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556e040f69d0 0x556e040f8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.959+0000 7f763a9eb740 4 client.25765 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.959+0000 7f763a9eb740 1 -- 172.21.15.6:0/805618554 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556e03fe8320 con 0x556e040f69d0 2024-04-26T01:50:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.959+0000 7f7635000640 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556e040f69d0 0x556e040f8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.960+0000 7f7612a00640 10 client.25765 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.960+0000 7f7635000640 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556e040f69d0 0x556e040f8df0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b045f70cc1b2980 server_cookie=aefe3fe3b032a2f3 in_seq=0 out_seq=0 2024-04-26T01:50:22.989 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.990+0000 7f7612a00640 1 -- 172.21.15.6:0/805618554 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556e03fe8320 con 0x556e040f69d0 2024-04-26T01:50:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.990+0000 7f7612a00640 10 client.25765 handle_command_reply: tid=0 2024-04-26T01:50:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.990+0000 7f763a9eb740 1 client.25765 shutdown 2024-04-26T01:50:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.991+0000 7f763a9eb740 1 -- 172.21.15.6:0/805618554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 msgr2=0x556e04084f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.991+0000 7f763a9eb740 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 0x556e04084f90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x556e0406c960 tx=0x7f7620012ad0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.991+0000 7f763a9eb740 1 -- 172.21.15.6:0/805618554 shutdown_connections 2024-04-26T01:50:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.991+0000 7f763a9eb740 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556e040f69d0 0x556e040f8df0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.991+0000 7f763a9eb740 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556e0407fa00 0x556e040854d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.991+0000 7f763a9eb740 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556e04075140 0x556e04084f90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.991+0000 7f7612a00640 0 client.25765 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.991+0000 7f763a9eb740 1 --2- 172.21.15.6:0/805618554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556e04074820 0x556e04087b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.991+0000 7f763a9eb740 1 -- 172.21.15.6:0/805618554 >> 172.21.15.6:0/805618554 conn(0x556e04070d30 msgr2=0x556e0407d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:22.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.991+0000 7f763a9eb740 1 -- 172.21.15.6:0/805618554 shutdown_connections 2024-04-26T01:50:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.991+0000 7f763a9eb740 1 -- 172.21.15.6:0/805618554 wait complete. 2024-04-26T01:50:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.991+0000 7f763a9eb740 20 client.25765 trim_cache size 0 max 16384 2024-04-26T01:50:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:50:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:50:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:50:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:50:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.992+0000 7f7637a00640 1 -- 172.21.15.6:0/200881804 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f76180a3ca0 msgr2=0x7f76180a6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.992+0000 7f7637a00640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f76180a3ca0 0x7f76180a6160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x556e04072150 tx=0x7f7620005f90 comp rx=0 tx=0).stop 2024-04-26T01:50:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.992+0000 7f7637a00640 1 -- 172.21.15.6:0/200881804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76301031c0 msgr2=0x7f76301a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.992+0000 7f7637a00640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76301031c0 0x7f76301a0630 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f762400d950 tx=0x7f762400de10 comp rx=0 tx=0).stop 2024-04-26T01:50:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.992+0000 7f7637a00640 1 -- 172.21.15.6:0/200881804 shutdown_connections 2024-04-26T01:50:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.992+0000 7f7637a00640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f76180a3ca0 0x7f76180a6160 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.992+0000 7f7637a00640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7630103ae0 0x7f76301a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.992+0000 7f7637a00640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f763010c590 0x7f763019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.992+0000 7f7637a00640 1 --2- 172.21.15.6:0/200881804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76301031c0 0x7f76301a0630 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.992+0000 7f7637a00640 1 -- 172.21.15.6:0/200881804 >> 172.21.15.6:0/200881804 conn(0x7f7630077f70 msgr2=0x7f76301084f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.992+0000 7f7637a00640 1 -- 172.21.15.6:0/200881804 shutdown_connections 2024-04-26T01:50:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:22.992+0000 7f7637a00640 1 -- 172.21.15.6:0/200881804 wait complete. 2024-04-26T01:50:23.571 INFO:tasks.workunit.client.0.smithi006.stdout: 1 114776 2.48 MB/sec execute 47 sec latency 23.948 ms 2024-04-26T01:50:24.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:24 smithi029 ceph-mon[32775]: pgmap v139: 97 pgs: 97 active+clean; 45 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 365 KiB/s wr, 41 op/s 2024-04-26T01:50:24.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:24 smithi006 ceph-mon[28419]: pgmap v139: 97 pgs: 97 active+clean; 45 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 365 KiB/s wr, 41 op/s 2024-04-26T01:50:24.571 INFO:tasks.workunit.client.0.smithi006.stdout: 1 116474 2.65 MB/sec execute 48 sec latency 17.226 ms 2024-04-26T01:50:24.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:24 smithi188 ceph-mon[32751]: pgmap v139: 97 pgs: 97 active+clean; 45 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 365 KiB/s wr, 41 op/s 2024-04-26T01:50:25.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:25 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:25.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:25 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:25.571 INFO:tasks.workunit.client.0.smithi006.stdout: 1 117483 2.73 MB/sec execute 49 sec latency 18.507 ms 2024-04-26T01:50:25.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:25 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:26.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:26 smithi006 ceph-mon[28419]: pgmap v140: 97 pgs: 97 active+clean; 45 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 365 KiB/s wr, 41 op/s 2024-04-26T01:50:26.571 INFO:tasks.workunit.client.0.smithi006.stdout: 1 118538 2.77 MB/sec execute 50 sec latency 19.473 ms 2024-04-26T01:50:26.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:26 smithi188 ceph-mon[32751]: pgmap v140: 97 pgs: 97 active+clean; 45 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 365 KiB/s wr, 41 op/s 2024-04-26T01:50:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:26 smithi029 ceph-mon[32775]: pgmap v140: 97 pgs: 97 active+clean; 45 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 365 KiB/s wr, 41 op/s 2024-04-26T01:50:27.572 INFO:tasks.workunit.client.0.smithi006.stdout: 1 120067 2.87 MB/sec execute 51 sec latency 18.788 ms 2024-04-26T01:50:28.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:28 smithi006 ceph-mon[28419]: pgmap v141: 97 pgs: 97 active+clean; 49 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 802 KiB/s wr, 121 op/s 2024-04-26T01:50:28.572 INFO:tasks.workunit.client.0.smithi006.stdout: 1 121052 2.94 MB/sec execute 52 sec latency 15.259 ms 2024-04-26T01:50:28.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:28 smithi188 ceph-mon[32751]: pgmap v141: 97 pgs: 97 active+clean; 49 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 802 KiB/s wr, 121 op/s 2024-04-26T01:50:28.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:28 smithi029 ceph-mon[32775]: pgmap v141: 97 pgs: 97 active+clean; 49 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 802 KiB/s wr, 121 op/s 2024-04-26T01:50:29.572 INFO:tasks.workunit.client.0.smithi006.stdout: 1 122129 2.98 MB/sec execute 53 sec latency 19.071 ms 2024-04-26T01:50:30.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:30 smithi006 ceph-mon[28419]: pgmap v142: 97 pgs: 97 active+clean; 49 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 621 KiB/s wr, 112 op/s 2024-04-26T01:50:30.572 INFO:tasks.workunit.client.0.smithi006.stdout: 1 123654 3.06 MB/sec execute 54 sec latency 16.434 ms 2024-04-26T01:50:30.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:30 smithi188 ceph-mon[32751]: pgmap v142: 97 pgs: 97 active+clean; 49 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 621 KiB/s wr, 112 op/s 2024-04-26T01:50:30.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:30 smithi029 ceph-mon[32775]: pgmap v142: 97 pgs: 97 active+clean; 49 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 621 KiB/s wr, 112 op/s 2024-04-26T01:50:31.572 INFO:tasks.workunit.client.0.smithi006.stdout: 1 124604 3.12 MB/sec execute 55 sec latency 15.778 ms 2024-04-26T01:50:32.532 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:32 smithi006 ceph-mon[28419]: pgmap v143: 97 pgs: 97 active+clean; 54 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-04-26T01:50:32.572 INFO:tasks.workunit.client.0.smithi006.stdout: 1 125671 3.16 MB/sec execute 56 sec latency 27.019 ms 2024-04-26T01:50:32.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:32 smithi188 ceph-mon[32751]: pgmap v143: 97 pgs: 97 active+clean; 54 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-04-26T01:50:32.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:32 smithi029 ceph-mon[32775]: pgmap v143: 97 pgs: 97 active+clean; 54 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-04-26T01:50:33.573 INFO:tasks.workunit.client.0.smithi006.stdout: 1 127156 3.23 MB/sec execute 57 sec latency 15.483 ms 2024-04-26T01:50:34.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:34 smithi006 ceph-mon[28419]: pgmap v144: 97 pgs: 97 active+clean; 54 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 971 KiB/s wr, 151 op/s 2024-04-26T01:50:34.572 INFO:tasks.workunit.client.0.smithi006.stdout: 1 128119 3.29 MB/sec execute 58 sec latency 17.039 ms 2024-04-26T01:50:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:34 smithi188 ceph-mon[32751]: pgmap v144: 97 pgs: 97 active+clean; 54 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 971 KiB/s wr, 151 op/s 2024-04-26T01:50:34.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:34 smithi029 ceph-mon[32775]: pgmap v144: 97 pgs: 97 active+clean; 54 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 971 KiB/s wr, 151 op/s 2024-04-26T01:50:35.573 INFO:tasks.workunit.client.0.smithi006.stdout: 1 129132 3.32 MB/sec execute 59 sec latency 15.550 ms 2024-04-26T01:50:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.149+0000 7f2803200640 1 -- 172.21.15.6:0/3286534772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804101630 msgr2=0x7f2804105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.149+0000 7f2803200640 1 --2- 172.21.15.6:0/3286534772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804101630 0x7f2804105af0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f27f0009900 tx=0x7f27f002a480 comp rx=0 tx=0).stop 2024-04-26T01:50:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.149+0000 7f2803200640 1 -- 172.21.15.6:0/3286534772 shutdown_connections 2024-04-26T01:50:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.149+0000 7f2803200640 1 --2- 172.21.15.6:0/3286534772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2804106110 0x7f28041065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.149+0000 7f2803200640 1 --2- 172.21.15.6:0/3286534772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804101630 0x7f2804105af0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.149+0000 7f2803200640 1 --2- 172.21.15.6:0/3286534772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2804100d10 0x7f28041010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.149+0000 7f2803200640 1 -- 172.21.15.6:0/3286534772 >> 172.21.15.6:0/3286534772 conn(0x7f28040fc760 msgr2=0x7f28040feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.149+0000 7f2803200640 1 -- 172.21.15.6:0/3286534772 shutdown_connections 2024-04-26T01:50:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.149+0000 7f2803200640 1 -- 172.21.15.6:0/3286534772 wait complete. 2024-04-26T01:50:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.150+0000 7f2803200640 1 Processor -- start 2024-04-26T01:50:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.150+0000 7f2803200640 1 -- start start 2024-04-26T01:50:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2803200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2804100d10 0x7f28041a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2803200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804101630 0x7f28041a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2803200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2804106110 0x7f280419ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2803200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f28041185b0 con 0x7f2804100d10 2024-04-26T01:50:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2803200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2804118430 con 0x7f2804101630 2024-04-26T01:50:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2803200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2804118730 con 0x7f2804106110 2024-04-26T01:50:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2801400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804101630 0x7f28041a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2802800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2804106110 0x7f280419ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2801400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804101630 0x7f28041a4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59122/0 (socket says 172.21.15.6:59122) 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2801400640 1 -- 172.21.15.6:0/1457566121 learned_addr learned my addr 172.21.15.6:0/1457566121 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2801e00640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2804100d10 0x7f28041a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2801400640 1 -- 172.21.15.6:0/1457566121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2804106110 msgr2=0x7f280419ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2801400640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2804106110 0x7f280419ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2801400640 1 -- 172.21.15.6:0/1457566121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2804100d10 msgr2=0x7f28041a4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2801400640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2804100d10 0x7f28041a4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2801400640 1 -- 172.21.15.6:0/1457566121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280419f1b0 con 0x7f2804101630 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.151+0000 7f2801e00640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2804100d10 0x7f28041a4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.152+0000 7f2801400640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804101630 0x7f28041a4dc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f27f0005ab0 tx=0x7f27f0032b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.152+0000 7f27faa00640 1 -- 172.21.15.6:0/1457566121 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f003f030 con 0x7f2804101630 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.152+0000 7f27faa00640 1 -- 172.21.15.6:0/1457566121 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27f0003aa0 con 0x7f2804101630 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.152+0000 7f2803200640 1 -- 172.21.15.6:0/1457566121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f280419f440 con 0x7f2804101630 2024-04-26T01:50:36.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.152+0000 7f27faa00640 1 -- 172.21.15.6:0/1457566121 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f0035070 con 0x7f2804101630 2024-04-26T01:50:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.152+0000 7f2803200640 1 -- 172.21.15.6:0/1457566121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28041ab740 con 0x7f2804101630 2024-04-26T01:50:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.156+0000 7f27faa00640 1 -- 172.21.15.6:0/1457566121 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f27f0042030 con 0x7f2804101630 2024-04-26T01:50:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.156+0000 7f280bb8a740 1 Processor -- start 2024-04-26T01:50:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.157+0000 7f280bb8a740 1 -- start start 2024-04-26T01:50:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.157+0000 7f280bb8a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c5427f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.157+0000 7f280bb8a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c5427f62d0 0x55c542800650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.157+0000 7f280bb8a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c542800b90 0x55c542802f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.157+0000 7f280bb8a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55c54275b900 con 0x55c5427f62d0 2024-04-26T01:50:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.157+0000 7f280bb8a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c542701ef0 con 0x55c542800b90 2024-04-26T01:50:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.157+0000 7f280bb8a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c5427578c0 con 0x55c5427f59b0 2024-04-26T01:50:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.157+0000 7f2802800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c5427f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.157+0000 7f2802800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c5427f5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37736/0 (socket says 172.21.15.6:37736) 2024-04-26T01:50:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.157+0000 7f2801e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c5427f62d0 0x55c542800650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.157+0000 7f2802800640 1 -- 172.21.15.6:0/1154253645 learned_addr learned my addr 172.21.15.6:0/1154253645 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.158+0000 7f2801400640 1 --2- 172.21.15.6:0/1154253645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c542800b90 0x55c542802f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.158+0000 7f2802800640 1 -- 172.21.15.6:0/1154253645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c542800b90 msgr2=0x55c542802f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.158+0000 7f2802800640 1 --2- 172.21.15.6:0/1154253645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c542800b90 0x55c542802f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.158+0000 7f2802800640 1 -- 172.21.15.6:0/1154253645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c5427f62d0 msgr2=0x55c542800650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.158+0000 7f2802800640 1 --2- 172.21.15.6:0/1154253645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c5427f62d0 0x55c542800650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.158+0000 7f2802800640 1 -- 172.21.15.6:0/1154253645 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c54246e210 con 0x55c5427f59b0 2024-04-26T01:50:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.158+0000 7f27faa00640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27e80a3d80 0x7f27e80a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.159+0000 7f2802800640 1 --2- 172.21.15.6:0/1154253645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c5427f5d90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009f50 tx=0x7f27fc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71591849db309fe7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.159+0000 7f2801e00640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27e80a3d80 0x7f27e80a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.159+0000 7f27faa00640 1 -- 172.21.15.6:0/1457566121 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f27f00f0e50 con 0x7f2804101630 2024-04-26T01:50:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.159+0000 7f27ef400640 1 -- 172.21.15.6:0/1154253645 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc01e070 con 0x55c5427f59b0 2024-04-26T01:50:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.159+0000 7f27ef400640 1 -- 172.21.15.6:0/1154253645 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27fc003c30 con 0x55c5427f59b0 2024-04-26T01:50:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.159+0000 7f27ef400640 1 -- 172.21.15.6:0/1154253645 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc00f8d0 con 0x55c5427f59b0 2024-04-26T01:50:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.160+0000 7f2801e00640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27e80a3d80 0x7f27e80a6240 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55c5427f6f50 tx=0x7f27f4005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.160+0000 7f280bb8a740 1 -- 172.21.15.6:0/1154253645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 msgr2=0x55c5427f5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.160+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1154253645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c5427f5d90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009f50 tx=0x7f27fc0118d0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.160+0000 7f280bb8a740 1 -- 172.21.15.6:0/1154253645 shutdown_connections 2024-04-26T01:50:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.160+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1154253645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c5427f62d0 0x55c542800650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.160+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1154253645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c542800b90 0x55c542802f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.160+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1154253645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c5427f5d90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.160+0000 7f280bb8a740 1 -- 172.21.15.6:0/1154253645 >> 172.21.15.6:0/1154253645 conn(0x55c5427f1ec0 msgr2=0x55c5427f22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.160+0000 7f280bb8a740 1 -- 172.21.15.6:0/1154253645 shutdown_connections 2024-04-26T01:50:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.160+0000 7f280bb8a740 1 -- 172.21.15.6:0/1154253645 wait complete. 2024-04-26T01:50:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.161+0000 7f280bb8a740 1 -- start start 2024-04-26T01:50:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.161+0000 7f280bb8a740 1 Processor -- start 2024-04-26T01:50:36.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.161+0000 7f280bb8a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c5427f59b0 0x55c5427fda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.161+0000 7f280bb8a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c5427fdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.161+0000 7f280bb8a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c542800b90 0x55c5427fe510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.161+0000 7f280bb8a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55c5428037c0 con 0x55c5427f59b0 2024-04-26T01:50:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.161+0000 7f280bb8a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c542715310 con 0x55c5427f62d0 2024-04-26T01:50:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.162+0000 7f280bb8a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c542803940 con 0x55c542800b90 2024-04-26T01:50:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.162+0000 7f280bb8a740 10 client.0 _ll_register_callbacks cb 0x55c5426e3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.162+0000 7f2802800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c542800b90 0x55c5427fe510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.162+0000 7f2801e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c5427f59b0 0x55c5427fda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.162+0000 7f2801400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c5427fdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.162+0000 7f2802800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c542800b90 0x55c5427fe510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37742/0 (socket says 172.21.15.6:37742) 2024-04-26T01:50:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.162+0000 7f2802800640 1 -- 172.21.15.6:0/1603186736 learned_addr learned my addr 172.21.15.6:0/1603186736 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.162+0000 7f2802800640 1 -- 172.21.15.6:0/1603186736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 msgr2=0x55c5427fdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.162+0000 7f2802800640 1 --2- 172.21.15.6:0/1603186736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c5427fdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.162+0000 7f2802800640 1 -- 172.21.15.6:0/1603186736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c5427f59b0 msgr2=0x55c5427fda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.162+0000 7f2802800640 1 --2- 172.21.15.6:0/1603186736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c5427f59b0 0x55c5427fda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.162+0000 7f2802800640 1 -- 172.21.15.6:0/1603186736 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c54253a620 con 0x55c542800b90 2024-04-26T01:50:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.163+0000 7f2802800640 1 --2- 172.21.15.6:0/1603186736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c542800b90 0x55c5427fe510 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f27fc012fd0 tx=0x7f27fc005a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54a1faf6da404dcf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.163+0000 7f27e3e00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:50:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.163+0000 7f27e3e00640 1 -- 172.21.15.6:0/1603186736 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc01e050 con 0x55c542800b90 2024-04-26T01:50:36.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.163+0000 7f27e3e00640 1 -- 172.21.15.6:0/1603186736 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f27fc00f5d0 con 0x55c542800b90 2024-04-26T01:50:36.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.163+0000 7f280bb8a740 1 -- 172.21.15.6:0/1603186736 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27fc003c30 con 0x55c542800b90 2024-04-26T01:50:36.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.163+0000 7f27e3e00640 1 -- 172.21.15.6:0/1603186736 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f27fc033050 con 0x55c542800b90 2024-04-26T01:50:36.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.163+0000 7f280bb8a740 10 client.25777 fetch_fsmap learned FSMap version 33 2024-04-26T01:50:36.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.163+0000 7f280bb8a740 1 -- 172.21.15.6:0/1603186736 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c54280f9c0 con 0x55c542800b90 2024-04-26T01:50:36.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.164+0000 7f27e3e00640 1 -- 172.21.15.6:0/1603186736 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27fc003c30 con 0x55c542800b90 2024-04-26T01:50:36.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.164+0000 7f27e3e00640 1 -- 172.21.15.6:0/1603186736 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f27fc025960 con 0x55c542800b90 2024-04-26T01:50:36.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.164+0000 7f280bb8a740 10 client.25777 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T01:50:36.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.164+0000 7f280bb8a740 10 client.25777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.164+0000 7f280bb8a740 20 client.25777 populate_metadata read hostname 'smithi006' 2024-04-26T01:50:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.164+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1603186736 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c542810dc0 0x55c5428131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.164+0000 7f280bb8a740 4 client.25777 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.164+0000 7f280bb8a740 1 -- 172.21.15.6:0/1603186736 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c54276f8b0 con 0x55c542810dc0 2024-04-26T01:50:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.164+0000 7f2801e00640 1 --2- 172.21.15.6:0/1603186736 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c542810dc0 0x55c5428131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.165+0000 7f2801e00640 1 --2- 172.21.15.6:0/1603186736 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c542810dc0 0x55c5428131b0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=833ff1a129238db6 server_cookie=6b2a00f478a31a58 in_seq=0 out_seq=0 2024-04-26T01:50:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.165+0000 7f27e3e00640 10 client.25777 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f27e3e00640 1 -- 172.21.15.6:0/1603186736 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c54276f8b0 con 0x55c542810dc0 2024-04-26T01:50:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f27e3e00640 10 client.25777 handle_command_reply: tid=0 2024-04-26T01:50:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f280bb8a740 1 client.25777 shutdown 2024-04-26T01:50:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f280bb8a740 1 -- 172.21.15.6:0/1603186736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c542800b90 msgr2=0x55c5427fe510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1603186736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c542800b90 0x55c5427fe510 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f27fc012fd0 tx=0x7f27fc005a50 comp rx=0 tx=0).stop 2024-04-26T01:50:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f280bb8a740 1 -- 172.21.15.6:0/1603186736 shutdown_connections 2024-04-26T01:50:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1603186736 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c542810dc0 0x55c5428131b0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1603186736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c542800b90 0x55c5427fe510 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f27e3e00640 0 client.25777 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1603186736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c5427fdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1603186736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c5427f59b0 0x55c5427fda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f280bb8a740 1 -- 172.21.15.6:0/1603186736 >> 172.21.15.6:0/1603186736 conn(0x55c5427f1ec0 msgr2=0x55c5427f7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f280bb8a740 1 -- 172.21.15.6:0/1603186736 shutdown_connections 2024-04-26T01:50:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f280bb8a740 1 -- 172.21.15.6:0/1603186736 wait complete. 2024-04-26T01:50:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.166+0000 7f280bb8a740 20 client.25777 trim_cache size 0 max 16384 2024-04-26T01:50:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.184+0000 7f280bb8a740 1 Processor -- start 2024-04-26T01:50:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.184+0000 7f280bb8a740 1 -- start start 2024-04-26T01:50:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.184+0000 7f280bb8a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c54280c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f280bb8a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c5427f0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f280bb8a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c542800b90 0x55c542818920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f280bb8a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f28041b6a80 con 0x55c542800b90 2024-04-26T01:50:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f280bb8a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f28041b42f0 con 0x55c5427f62d0 2024-04-26T01:50:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f280bb8a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f28041b2140 con 0x55c5427f59b0 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f2801e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c54280c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f2801e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c54280c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37752/0 (socket says 172.21.15.6:37752) 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f2801e00640 1 -- 172.21.15.6:0/1846844189 learned_addr learned my addr 172.21.15.6:0/1846844189 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f2802800640 1 --2- 172.21.15.6:0/1846844189 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c542800b90 0x55c542818920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f2801400640 1 --2- 172.21.15.6:0/1846844189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c5427f0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f2801e00640 1 -- 172.21.15.6:0/1846844189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 msgr2=0x55c5427f0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f2801e00640 1 --2- 172.21.15.6:0/1846844189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c5427f0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f2801e00640 1 -- 172.21.15.6:0/1846844189 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c542800b90 msgr2=0x55c542818920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f2801e00640 1 --2- 172.21.15.6:0/1846844189 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c542800b90 0x55c542818920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f2801e00640 1 -- 172.21.15.6:0/1846844189 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5427fbfe0 con 0x55c5427f59b0 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.185+0000 7f2801400640 1 --2- 172.21.15.6:0/1846844189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c5427f0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.186+0000 7f2801e00640 1 --2- 172.21.15.6:0/1846844189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c54280c680 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f280419fb70 tx=0x7f27f401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4252e1b31575215f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.186+0000 7f27eea00640 1 -- 172.21.15.6:0/1846844189 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f401ee60 con 0x55c5427f59b0 2024-04-26T01:50:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.186+0000 7f27eea00640 1 -- 172.21.15.6:0/1846844189 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27f402a8f0 con 0x55c5427f59b0 2024-04-26T01:50:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.186+0000 7f27eea00640 1 -- 172.21.15.6:0/1846844189 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f4011c70 con 0x55c5427f59b0 2024-04-26T01:50:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.187+0000 7f280bb8a740 1 -- 172.21.15.6:0/1846844189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 msgr2=0x55c54280c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.187+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1846844189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c54280c680 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f280419fb70 tx=0x7f27f401a000 comp rx=0 tx=0).stop 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.187+0000 7f280bb8a740 1 -- 172.21.15.6:0/1846844189 shutdown_connections 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.187+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1846844189 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c542800b90 0x55c542818920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.187+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1846844189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c5427f0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.187+0000 7f280bb8a740 1 --2- 172.21.15.6:0/1846844189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c54280c680 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.187+0000 7f280bb8a740 1 -- 172.21.15.6:0/1846844189 >> 172.21.15.6:0/1846844189 conn(0x55c5427f1ec0 msgr2=0x55c5427f0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.187+0000 7f280bb8a740 1 -- 172.21.15.6:0/1846844189 shutdown_connections 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.187+0000 7f280bb8a740 1 -- 172.21.15.6:0/1846844189 wait complete. 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.187+0000 7f280bb8a740 1 -- start start 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.188+0000 7f280bb8a740 1 Processor -- start 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.188+0000 7f280bb8a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c542809080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.188+0000 7f280bb8a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c542806410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.188+0000 7f280bb8a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c542800b90 0x55c542806a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.188+0000 7f280bb8a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f28041b27b0 con 0x55c542800b90 2024-04-26T01:50:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.188+0000 7f280bb8a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f28041b3c20 con 0x55c5427f62d0 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.188+0000 7f280bb8a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f28041b2e80 con 0x55c5427f59b0 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f280bb8a740 10 client.0 _ll_register_callbacks cb 0x7f27f00f47d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f2801e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c542809080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f2801e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c542809080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37764/0 (socket says 172.21.15.6:37764) 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f2801e00640 1 -- 172.21.15.6:0/4213704790 learned_addr learned my addr 172.21.15.6:0/4213704790 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f2802800640 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c542800b90 0x55c542806a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f2801400640 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c542806410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f2801e00640 1 -- 172.21.15.6:0/4213704790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 msgr2=0x55c542806410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f2801e00640 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c542806410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f2801e00640 1 -- 172.21.15.6:0/4213704790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c542800b90 msgr2=0x55c542806a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f2801e00640 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c542800b90 0x55c542806a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f2801e00640 1 -- 172.21.15.6:0/4213704790 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c542807250 con 0x55c5427f59b0 2024-04-26T01:50:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f2801e00640 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c542809080 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f27f4009b90 tx=0x7f27f4009db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7766c1b018d2cf98 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.189+0000 7f27e3e00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:50:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.190+0000 7f27e3e00640 1 -- 172.21.15.6:0/4213704790 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f4039020 con 0x55c5427f59b0 2024-04-26T01:50:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.190+0000 7f27e3e00640 1 -- 172.21.15.6:0/4213704790 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f27f4011bb0 con 0x55c5427f59b0 2024-04-26T01:50:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.190+0000 7f280bb8a740 1 -- 172.21.15.6:0/4213704790 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27f402a8f0 con 0x55c5427f59b0 2024-04-26T01:50:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.190+0000 7f27e3e00640 1 -- 172.21.15.6:0/4213704790 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f27f4011db0 con 0x55c5427f59b0 2024-04-26T01:50:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.190+0000 7f280bb8a740 10 client.25783 fetch_fsmap learned FSMap version 33 2024-04-26T01:50:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.190+0000 7f280bb8a740 1 -- 172.21.15.6:0/4213704790 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c5428772e0 con 0x55c5427f59b0 2024-04-26T01:50:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.190+0000 7f27e3e00640 1 -- 172.21.15.6:0/4213704790 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27f402a8f0 con 0x55c5427f59b0 2024-04-26T01:50:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.190+0000 7f27e3e00640 1 -- 172.21.15.6:0/4213704790 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f27f4033670 con 0x55c5427f59b0 2024-04-26T01:50:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.191+0000 7f280bb8a740 10 client.25783 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T01:50:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.191+0000 7f280bb8a740 10 client.25783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.191+0000 7f280bb8a740 20 client.25783 populate_metadata read hostname 'smithi006' 2024-04-26T01:50:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.191+0000 7f280bb8a740 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c542877a50 0x55c542879e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.191+0000 7f280bb8a740 4 client.25783 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.191+0000 7f280bb8a740 1 -- 172.21.15.6:0/4213704790 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c542769320 con 0x55c542877a50 2024-04-26T01:50:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.191+0000 7f2801400640 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c542877a50 0x55c542879e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.191+0000 7f2801400640 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c542877a50 0x55c542879e70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fc5bbe3fe4fb6cd server_cookie=b663fea364380e0a in_seq=0 out_seq=0 2024-04-26T01:50:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.191+0000 7f27e3e00640 10 client.25783 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.237+0000 7f27e3e00640 1 -- 172.21.15.6:0/4213704790 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c542769320 con 0x55c542877a50 2024-04-26T01:50:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.237+0000 7f27e3e00640 10 client.25783 handle_command_reply: tid=0 2024-04-26T01:50:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.237+0000 7f280bb8a740 1 client.25783 shutdown 2024-04-26T01:50:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.237+0000 7f280bb8a740 1 -- 172.21.15.6:0/4213704790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 msgr2=0x55c542809080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.237+0000 7f280bb8a740 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c542809080 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f27f4009b90 tx=0x7f27f4009db0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.237+0000 7f280bb8a740 1 -- 172.21.15.6:0/4213704790 shutdown_connections 2024-04-26T01:50:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.237+0000 7f280bb8a740 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c542877a50 0x55c542879e70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.237+0000 7f280bb8a740 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c542800b90 0x55c542806a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.237+0000 7f280bb8a740 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5427f62d0 0x55c542806410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.237+0000 7f280bb8a740 1 --2- 172.21.15.6:0/4213704790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5427f59b0 0x55c542809080 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.237+0000 7f280bb8a740 1 -- 172.21.15.6:0/4213704790 >> 172.21.15.6:0/4213704790 conn(0x55c5427f1ec0 msgr2=0x55c5427f1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.237+0000 7f27e3e00640 0 client.25783 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.238+0000 7f280bb8a740 1 -- 172.21.15.6:0/4213704790 shutdown_connections 2024-04-26T01:50:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.238+0000 7f280bb8a740 1 -- 172.21.15.6:0/4213704790 wait complete. 2024-04-26T01:50:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.238+0000 7f280bb8a740 20 client.25783 trim_cache size 0 max 16384 2024-04-26T01:50:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:50:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:50:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:50:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:50:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.239+0000 7f2803200640 1 -- 172.21.15.6:0/1457566121 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27e80a3d80 msgr2=0x7f27e80a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.239+0000 7f2803200640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27e80a3d80 0x7f27e80a6240 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55c5427f6f50 tx=0x7f27f4005f90 comp rx=0 tx=0).stop 2024-04-26T01:50:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.239+0000 7f2803200640 1 -- 172.21.15.6:0/1457566121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804101630 msgr2=0x7f28041a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.239+0000 7f2803200640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804101630 0x7f28041a4dc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f27f0005ab0 tx=0x7f27f0032b50 comp rx=0 tx=0).stop 2024-04-26T01:50:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.239+0000 7f2803200640 1 -- 172.21.15.6:0/1457566121 shutdown_connections 2024-04-26T01:50:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.239+0000 7f2803200640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27e80a3d80 0x7f27e80a6240 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.239+0000 7f2803200640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2804106110 0x7f280419ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.239+0000 7f2803200640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804101630 0x7f28041a4dc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.239+0000 7f2803200640 1 --2- 172.21.15.6:0/1457566121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2804100d10 0x7f28041a4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.239+0000 7f2803200640 1 -- 172.21.15.6:0/1457566121 >> 172.21.15.6:0/1457566121 conn(0x7f28040fc760 msgr2=0x7f280410f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.239+0000 7f2803200640 1 -- 172.21.15.6:0/1457566121 shutdown_connections 2024-04-26T01:50:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:36.239+0000 7f2803200640 1 -- 172.21.15.6:0/1457566121 wait complete. 2024-04-26T01:50:36.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:36 smithi006 ceph-mon[28419]: pgmap v145: 97 pgs: 97 active+clean; 54 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 971 KiB/s wr, 151 op/s 2024-04-26T01:50:36.573 INFO:tasks.workunit.client.0.smithi006.stdout: 1 130708 3.39 MB/sec execute 60 sec latency 22.911 ms 2024-04-26T01:50:36.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:36 smithi188 ceph-mon[32751]: pgmap v145: 97 pgs: 97 active+clean; 54 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 971 KiB/s wr, 151 op/s 2024-04-26T01:50:36.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:36 smithi029 ceph-mon[32775]: pgmap v145: 97 pgs: 97 active+clean; 54 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 971 KiB/s wr, 151 op/s 2024-04-26T01:50:37.573 INFO:tasks.workunit.client.0.smithi006.stdout: 1 131665 3.44 MB/sec execute 61 sec latency 19.754 ms 2024-04-26T01:50:38.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:38 smithi006 ceph-mon[28419]: pgmap v146: 97 pgs: 97 active+clean; 57 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 227 op/s 2024-04-26T01:50:38.573 INFO:tasks.workunit.client.0.smithi006.stdout: 1 132686 3.47 MB/sec execute 62 sec latency 18.037 ms 2024-04-26T01:50:38.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:38 smithi188 ceph-mon[32751]: pgmap v146: 97 pgs: 97 active+clean; 57 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 227 op/s 2024-04-26T01:50:38.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:38 smithi029 ceph-mon[32775]: pgmap v146: 97 pgs: 97 active+clean; 57 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.7 MiB/s wr, 227 op/s 2024-04-26T01:50:39.573 INFO:tasks.workunit.client.0.smithi006.stdout: 1 134205 3.53 MB/sec execute 63 sec latency 19.802 ms 2024-04-26T01:50:40.493 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:40 smithi029 ceph-mon[32775]: pgmap v147: 97 pgs: 97 active+clean; 57 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-26T01:50:40.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:40 smithi006 ceph-mon[28419]: pgmap v147: 97 pgs: 97 active+clean; 57 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-26T01:50:40.573 INFO:tasks.workunit.client.0.smithi006.stdout: 1 135201 3.58 MB/sec execute 64 sec latency 18.014 ms 2024-04-26T01:50:40.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:40 smithi188 ceph-mon[32751]: pgmap v147: 97 pgs: 97 active+clean; 57 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-26T01:50:41.574 INFO:tasks.workunit.client.0.smithi006.stdout: 1 136041 3.56 MB/sec execute 65 sec latency 17.389 ms 2024-04-26T01:50:42.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:42 smithi006 ceph-mon[28419]: pgmap v148: 97 pgs: 97 active+clean; 52 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 245 op/s 2024-04-26T01:50:42.573 INFO:tasks.workunit.client.0.smithi006.stdout: 1 137594 3.64 MB/sec execute 66 sec latency 22.380 ms 2024-04-26T01:50:42.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:42 smithi188 ceph-mon[32751]: pgmap v148: 97 pgs: 97 active+clean; 52 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 245 op/s 2024-04-26T01:50:42.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:42 smithi029 ceph-mon[32775]: pgmap v148: 97 pgs: 97 active+clean; 52 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 245 op/s 2024-04-26T01:50:43.574 INFO:tasks.workunit.client.0.smithi006.stdout: 1 138728 3.71 MB/sec execute 67 sec latency 18.194 ms 2024-04-26T01:50:44.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:44 smithi006 ceph-mon[28419]: pgmap v149: 97 pgs: 97 active+clean; 52 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-04-26T01:50:44.574 INFO:tasks.workunit.client.0.smithi006.stdout: 1 139767 3.73 MB/sec execute 68 sec latency 13.202 ms 2024-04-26T01:50:44.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:44 smithi188 ceph-mon[32751]: pgmap v149: 97 pgs: 97 active+clean; 52 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-04-26T01:50:44.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:44 smithi029 ceph-mon[32775]: pgmap v149: 97 pgs: 97 active+clean; 52 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-04-26T01:50:45.574 INFO:tasks.workunit.client.0.smithi006.stdout: 1 141378 3.78 MB/sec execute 69 sec latency 16.347 ms 2024-04-26T01:50:46.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:46 smithi006 ceph-mon[28419]: pgmap v150: 97 pgs: 97 active+clean; 52 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-04-26T01:50:46.574 INFO:tasks.workunit.client.0.smithi006.stdout: 1 142353 3.82 MB/sec execute 70 sec latency 15.275 ms 2024-04-26T01:50:46.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:46 smithi188 ceph-mon[32751]: pgmap v150: 97 pgs: 97 active+clean; 52 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-04-26T01:50:46.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:46 smithi029 ceph-mon[32775]: pgmap v150: 97 pgs: 97 active+clean; 52 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-04-26T01:50:47.574 INFO:tasks.workunit.client.0.smithi006.stdout: 1 143579 3.84 MB/sec execute 71 sec latency 18.654 ms 2024-04-26T01:50:48.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:48 smithi006 ceph-mon[28419]: pgmap v151: 97 pgs: 97 active+clean; 56 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 248 op/s 2024-04-26T01:50:48.574 INFO:tasks.workunit.client.0.smithi006.stdout: 1 145455 3.96 MB/sec execute 72 sec latency 21.571 ms 2024-04-26T01:50:48.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:48 smithi188 ceph-mon[32751]: pgmap v151: 97 pgs: 97 active+clean; 56 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 248 op/s 2024-04-26T01:50:48.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:48 smithi029 ceph-mon[32775]: pgmap v151: 97 pgs: 97 active+clean; 56 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 248 op/s 2024-04-26T01:50:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.392+0000 7f9f96e00640 1 -- 172.21.15.6:0/4125314024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f901083b0 msgr2=0x7f9f90108810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.392+0000 7f9f96e00640 1 --2- 172.21.15.6:0/4125314024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f901083b0 0x7f9f90108810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009900 tx=0x7f9f7c02a480 comp rx=0 tx=0).stop 2024-04-26T01:50:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.392+0000 7f9f96e00640 1 -- 172.21.15.6:0/4125314024 shutdown_connections 2024-04-26T01:50:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.392+0000 7f9f96e00640 1 --2- 172.21.15.6:0/4125314024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f90108d50 0x7f9f90115210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.392+0000 7f9f96e00640 1 --2- 172.21.15.6:0/4125314024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f901083b0 0x7f9f90108810 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.392+0000 7f9f96e00640 1 --2- 172.21.15.6:0/4125314024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f9010e470 0x7f9f9010e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.392+0000 7f9f96e00640 1 -- 172.21.15.6:0/4125314024 >> 172.21.15.6:0/4125314024 conn(0x7f9f90077b00 msgr2=0x7f9f90079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.392+0000 7f9f96e00640 1 -- 172.21.15.6:0/4125314024 shutdown_connections 2024-04-26T01:50:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.392+0000 7f9f96e00640 1 -- 172.21.15.6:0/4125314024 wait complete. 2024-04-26T01:50:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.393+0000 7f9f96e00640 1 Processor -- start 2024-04-26T01:50:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.393+0000 7f9f96e00640 1 -- start start 2024-04-26T01:50:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.393+0000 7f9f96e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f901083b0 0x7f9f901a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.393+0000 7f9f96e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f90108d50 0x7f9f901a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.393+0000 7f9f96e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f9010e470 0x7f9f9019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f96e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9f90118650 con 0x7f9f9010e470 2024-04-26T01:50:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f96e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f901184d0 con 0x7f9f901083b0 2024-04-26T01:50:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f96e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9f901187d0 con 0x7f9f90108d50 2024-04-26T01:50:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f95a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f901083b0 0x7f9f901a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f95000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f90108d50 0x7f9f901a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f95a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f901083b0 0x7f9f901a4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53574/0 (socket says 172.21.15.6:53574) 2024-04-26T01:50:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f95a00640 1 -- 172.21.15.6:0/462495947 learned_addr learned my addr 172.21.15.6:0/462495947 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f96400640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f9010e470 0x7f9f9019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f95a00640 1 -- 172.21.15.6:0/462495947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f90108d50 msgr2=0x7f9f901a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f95a00640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f90108d50 0x7f9f901a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f95a00640 1 -- 172.21.15.6:0/462495947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f9010e470 msgr2=0x7f9f9019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f95a00640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f9010e470 0x7f9f9019e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.394+0000 7f9f95a00640 1 -- 172.21.15.6:0/462495947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f9019f0b0 con 0x7f9f901083b0 2024-04-26T01:50:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.395+0000 7f9f95a00640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f901083b0 0x7f9f901a4780 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400e9f0 tx=0x7f9f8400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.395+0000 7f9f8e000640 1 -- 172.21.15.6:0/462495947 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8400cd80 con 0x7f9f901083b0 2024-04-26T01:50:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.395+0000 7f9f8e000640 1 -- 172.21.15.6:0/462495947 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f840054d0 con 0x7f9f901083b0 2024-04-26T01:50:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.395+0000 7f9f96e00640 1 -- 172.21.15.6:0/462495947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f9019f3a0 con 0x7f9f901083b0 2024-04-26T01:50:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.395+0000 7f9f8e000640 1 -- 172.21.15.6:0/462495947 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84010690 con 0x7f9f901083b0 2024-04-26T01:50:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.395+0000 7f9f96e00640 1 -- 172.21.15.6:0/462495947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f901ab640 con 0x7f9f901083b0 2024-04-26T01:50:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.398+0000 7f9f8e000640 1 -- 172.21.15.6:0/462495947 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f9f84004e90 con 0x7f9f901083b0 2024-04-26T01:50:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.398+0000 7f9f9b280740 1 Processor -- start 2024-04-26T01:50:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.399+0000 7f9f9b280740 1 -- start start 2024-04-26T01:50:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.399+0000 7f9f9b280740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 0x563cfa828d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.399+0000 7f9f9b280740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8292d0 0x563cfa833650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.399+0000 7f9f9b280740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa833b90 0x563cfa835f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.399+0000 7f9f9b280740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x563cfa78e900 con 0x563cfa833b90 2024-04-26T01:50:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.399+0000 7f9f9b280740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563cfa734ef0 con 0x563cfa8292d0 2024-04-26T01:50:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.399+0000 7f9f9b280740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563cfa78a8c0 con 0x563cfa8289b0 2024-04-26T01:50:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.399+0000 7f9f96400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 0x563cfa828d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.399+0000 7f9f95000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa833b90 0x563cfa835f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.399+0000 7f9f95a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8292d0 0x563cfa833650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.399+0000 7f9f96400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 0x563cfa828d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:40288/0 (socket says 172.21.15.6:40288) 2024-04-26T01:50:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.399+0000 7f9f96400640 1 -- 172.21.15.6:0/2380619110 learned_addr learned my addr 172.21.15.6:0/2380619110 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.400+0000 7f9f96400640 1 -- 172.21.15.6:0/2380619110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8292d0 msgr2=0x563cfa833650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.400+0000 7f9f96400640 1 --2- 172.21.15.6:0/2380619110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8292d0 0x563cfa833650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.400+0000 7f9f96400640 1 -- 172.21.15.6:0/2380619110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa833b90 msgr2=0x563cfa835f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.400+0000 7f9f96400640 1 --2- 172.21.15.6:0/2380619110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa833b90 0x563cfa835f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.400+0000 7f9f96400640 1 -- 172.21.15.6:0/2380619110 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cfa4a1210 con 0x563cfa8289b0 2024-04-26T01:50:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.400+0000 7f9f95000640 1 --2- 172.21.15.6:0/2380619110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa833b90 0x563cfa835f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.401+0000 7f9f96400640 1 --2- 172.21.15.6:0/2380619110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 0x563cfa828d90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009fd0 tx=0x7f9f880118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8364bb948753e64 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.401+0000 7f9f8e000640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9f780a3d00 0x7f9f780a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.401+0000 7f9f82a00640 1 -- 172.21.15.6:0/2380619110 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f88004d20 con 0x563cfa8289b0 2024-04-26T01:50:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.401+0000 7f9f82a00640 1 -- 172.21.15.6:0/2380619110 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f88004050 con 0x563cfa8289b0 2024-04-26T01:50:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.401+0000 7f9f82a00640 1 -- 172.21.15.6:0/2380619110 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f88005280 con 0x563cfa8289b0 2024-04-26T01:50:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.401+0000 7f9f8e000640 1 -- 172.21.15.6:0/462495947 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f9f840d0c80 con 0x7f9f901083b0 2024-04-26T01:50:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.401+0000 7f9f95a00640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9f780a3d00 0x7f9f780a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.402+0000 7f9f9b280740 1 -- 172.21.15.6:0/2380619110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 msgr2=0x563cfa828d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.402+0000 7f9f95a00640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9f780a3d00 0x7f9f780a61c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x563cfa829f50 tx=0x7f9f840d4730 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.402+0000 7f9f9b280740 1 --2- 172.21.15.6:0/2380619110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 0x563cfa828d90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009fd0 tx=0x7f9f880118d0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.402+0000 7f9f9b280740 1 -- 172.21.15.6:0/2380619110 shutdown_connections 2024-04-26T01:50:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.402+0000 7f9f9b280740 1 --2- 172.21.15.6:0/2380619110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa833b90 0x563cfa835f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.402+0000 7f9f9b280740 1 --2- 172.21.15.6:0/2380619110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8292d0 0x563cfa833650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.402+0000 7f9f9b280740 1 --2- 172.21.15.6:0/2380619110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 0x563cfa828d90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.402+0000 7f9f9b280740 1 -- 172.21.15.6:0/2380619110 >> 172.21.15.6:0/2380619110 conn(0x563cfa824ec0 msgr2=0x563cfa8252a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.402+0000 7f9f9b280740 1 -- 172.21.15.6:0/2380619110 shutdown_connections 2024-04-26T01:50:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.402+0000 7f9f9b280740 1 -- 172.21.15.6:0/2380619110 wait complete. 2024-04-26T01:50:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.403+0000 7f9f9b280740 1 -- start start 2024-04-26T01:50:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.403+0000 7f9f9b280740 1 Processor -- start 2024-04-26T01:50:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.403+0000 7f9f9b280740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 0x563cfa830b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.403+0000 7f9f9b280740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 0x563cfa831090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.403+0000 7f9f9b280740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa833b90 0x563cfa8315d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.403+0000 7f9f9b280740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x563cfa8367c0 con 0x563cfa8292d0 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.403+0000 7f9f9b280740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563cfa748310 con 0x563cfa833b90 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.403+0000 7f9f9b280740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563cfa836940 con 0x563cfa8289b0 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.403+0000 7f9f9b280740 10 client.0 _ll_register_callbacks cb 0x563cfa716cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.404+0000 7f9f95000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 0x563cfa831090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.404+0000 7f9f96400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 0x563cfa830b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.404+0000 7f9f95000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 0x563cfa831090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44798/0 (socket says 172.21.15.6:44798) 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.404+0000 7f9f95000640 1 -- 172.21.15.6:0/742956320 learned_addr learned my addr 172.21.15.6:0/742956320 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.404+0000 7f9f96400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 0x563cfa830b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:40302/0 (socket says 172.21.15.6:40302) 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.404+0000 7f9f96400640 1 --2- 172.21.15.6:0/742956320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa833b90 0x563cfa8315d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.404+0000 7f9f95000640 1 -- 172.21.15.6:0/742956320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 msgr2=0x563cfa830b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.404+0000 7f9f95000640 1 --2- 172.21.15.6:0/742956320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 0x563cfa830b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.404+0000 7f9f95000640 1 -- 172.21.15.6:0/742956320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa833b90 msgr2=0x563cfa8315d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.404+0000 7f9f95000640 1 --2- 172.21.15.6:0/742956320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa833b90 0x563cfa8315d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.404+0000 7f9f95000640 1 -- 172.21.15.6:0/742956320 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cfa56d620 con 0x563cfa8292d0 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.405+0000 7f9f95000640 1 --2- 172.21.15.6:0/742956320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 0x563cfa831090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009230 tx=0x7f9f7c004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54ced208597982d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.405+0000 7f9f77400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.405+0000 7f9f77400640 1 -- 172.21.15.6:0/742956320 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c035040 con 0x563cfa8292d0 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.405+0000 7f9f77400640 1 -- 172.21.15.6:0/742956320 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f9f7c005510 con 0x563cfa8292d0 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.405+0000 7f9f9b280740 1 -- 172.21.15.6:0/742956320 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f88004050 con 0x563cfa8292d0 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.405+0000 7f9f77400640 1 -- 172.21.15.6:0/742956320 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9f7c004770 con 0x563cfa8292d0 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.405+0000 7f9f9b280740 10 client.15624 fetch_fsmap learned FSMap version 33 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.405+0000 7f9f9b280740 1 -- 172.21.15.6:0/742956320 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563cfa842a50 con 0x563cfa8292d0 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.406+0000 7f9f77400640 1 -- 172.21.15.6:0/742956320 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f88004050 con 0x563cfa8292d0 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.406+0000 7f9f77400640 1 -- 172.21.15.6:0/742956320 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f9f7c003b10 con 0x563cfa8292d0 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.406+0000 7f9f9b280740 10 client.15624 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.406+0000 7f9f9b280740 10 client.15624 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.406+0000 7f9f9b280740 20 client.15624 populate_metadata read hostname 'smithi006' 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.406+0000 7f9f9b280740 1 --2- 172.21.15.6:0/742956320 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563cfa843f60 0x563cfa846350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.406+0000 7f9f9b280740 4 client.15624 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.406+0000 7f9f9b280740 1 -- 172.21.15.6:0/742956320 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cfa7a28b0 con 0x563cfa843f60 2024-04-26T01:50:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.406+0000 7f9f96400640 1 --2- 172.21.15.6:0/742956320 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563cfa843f60 0x563cfa846350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.407+0000 7f9f96400640 1 --2- 172.21.15.6:0/742956320 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563cfa843f60 0x563cfa846350 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d08e2967085a77 server_cookie=2d689e3aa9365e49 in_seq=0 out_seq=0 2024-04-26T01:50:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.407+0000 7f9f77400640 10 client.15624 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.407+0000 7f9f77400640 1 -- 172.21.15.6:0/742956320 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563cfa7a28b0 con 0x563cfa843f60 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.407+0000 7f9f77400640 10 client.15624 handle_command_reply: tid=0 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f9b280740 1 client.15624 shutdown 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f9b280740 1 -- 172.21.15.6:0/742956320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 msgr2=0x563cfa831090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f9b280740 1 --2- 172.21.15.6:0/742956320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 0x563cfa831090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009230 tx=0x7f9f7c004440 comp rx=0 tx=0).stop 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f9b280740 1 -- 172.21.15.6:0/742956320 shutdown_connections 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f9b280740 1 --2- 172.21.15.6:0/742956320 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563cfa843f60 0x563cfa846350 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f9b280740 1 --2- 172.21.15.6:0/742956320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 0x563cfa831090 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f9b280740 1 --2- 172.21.15.6:0/742956320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa833b90 0x563cfa8315d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f9b280740 1 --2- 172.21.15.6:0/742956320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa8289b0 0x563cfa830b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f77400640 0 client.15624 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f9b280740 1 -- 172.21.15.6:0/742956320 >> 172.21.15.6:0/742956320 conn(0x563cfa824ec0 msgr2=0x563cfa82abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f9b280740 1 -- 172.21.15.6:0/742956320 shutdown_connections 2024-04-26T01:50:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f9b280740 1 -- 172.21.15.6:0/742956320 wait complete. 2024-04-26T01:50:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.408+0000 7f9f9b280740 20 client.15624 trim_cache size 0 max 16384 2024-04-26T01:50:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f9b280740 1 Processor -- start 2024-04-26T01:50:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f9b280740 1 -- start start 2024-04-26T01:50:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f9b280740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8289b0 0x563cfa8247a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f9b280740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 0x563cfa8313f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f9b280740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa833b90 0x563cfa84bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f9b280740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9f901b69b0 con 0x563cfa8292d0 2024-04-26T01:50:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f9b280740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f901b4220 con 0x563cfa8289b0 2024-04-26T01:50:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f9b280740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9f901b2070 con 0x563cfa833b90 2024-04-26T01:50:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f96400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8289b0 0x563cfa8247a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f95000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 0x563cfa8313f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f96400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8289b0 0x563cfa8247a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53594/0 (socket says 172.21.15.6:53594) 2024-04-26T01:50:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f96400640 1 -- 172.21.15.6:0/707394929 learned_addr learned my addr 172.21.15.6:0/707394929 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.426+0000 7f9f96400640 1 --2- 172.21.15.6:0/707394929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa833b90 0x563cfa84bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.427+0000 7f9f95000640 1 -- 172.21.15.6:0/707394929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa833b90 msgr2=0x563cfa84bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.427+0000 7f9f95000640 1 --2- 172.21.15.6:0/707394929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa833b90 0x563cfa84bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.427+0000 7f9f95000640 1 -- 172.21.15.6:0/707394929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8289b0 msgr2=0x563cfa8247a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.427+0000 7f9f95000640 1 --2- 172.21.15.6:0/707394929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8289b0 0x563cfa8247a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.427+0000 7f9f95000640 1 -- 172.21.15.6:0/707394929 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cfa8361b0 con 0x563cfa8292d0 2024-04-26T01:50:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.427+0000 7f9f95000640 1 --2- 172.21.15.6:0/707394929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 0x563cfa8313f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c005ab0 tx=0x7f9f7c02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda074566becb434 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.427+0000 7f9f82000640 1 -- 172.21.15.6:0/707394929 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c035070 con 0x563cfa8292d0 2024-04-26T01:50:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.428+0000 7f9f82000640 1 -- 172.21.15.6:0/707394929 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f7c041070 con 0x563cfa8292d0 2024-04-26T01:50:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.428+0000 7f9f82000640 1 -- 172.21.15.6:0/707394929 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c053070 con 0x563cfa8292d0 2024-04-26T01:50:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.428+0000 7f9f9b280740 1 -- 172.21.15.6:0/707394929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 msgr2=0x563cfa8313f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.428+0000 7f9f9b280740 1 --2- 172.21.15.6:0/707394929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 0x563cfa8313f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c005ab0 tx=0x7f9f7c02a480 comp rx=0 tx=0).stop 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.428+0000 7f9f9b280740 1 -- 172.21.15.6:0/707394929 shutdown_connections 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.428+0000 7f9f9b280740 1 --2- 172.21.15.6:0/707394929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8292d0 0x563cfa8313f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.428+0000 7f9f9b280740 1 --2- 172.21.15.6:0/707394929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa833b90 0x563cfa84bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.428+0000 7f9f9b280740 1 --2- 172.21.15.6:0/707394929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8289b0 0x563cfa8247a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.428+0000 7f9f9b280740 1 -- 172.21.15.6:0/707394929 >> 172.21.15.6:0/707394929 conn(0x563cfa824ec0 msgr2=0x563cfa82abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.428+0000 7f9f9b280740 1 -- 172.21.15.6:0/707394929 shutdown_connections 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.428+0000 7f9f9b280740 1 -- 172.21.15.6:0/707394929 wait complete. 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.429+0000 7f9f9b280740 1 -- start start 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.429+0000 7f9f9b280740 1 Processor -- start 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.429+0000 7f9f9b280740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8289b0 0x563cfa8397f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.429+0000 7f9f9b280740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8292d0 0x563cfa839d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.429+0000 7f9f9b280740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa833b90 0x563cfa83a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.429+0000 7f9f9b280740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9f901b26e0 con 0x563cfa8289b0 2024-04-26T01:50:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.429+0000 7f9f9b280740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f901b3b50 con 0x563cfa8292d0 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.429+0000 7f9f9b280740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9f901b2db0 con 0x563cfa833b90 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.430+0000 7f9f9b280740 10 client.0 _ll_register_callbacks cb 0x7f9f7c0040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.430+0000 7f9f96400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8289b0 0x563cfa8397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.430+0000 7f9f96400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8289b0 0x563cfa8397f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44828/0 (socket says 172.21.15.6:44828) 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.430+0000 7f9f96400640 1 -- 172.21.15.6:0/1902894219 learned_addr learned my addr 172.21.15.6:0/1902894219 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.430+0000 7f9f95000640 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8292d0 0x563cfa839d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.430+0000 7f9f96400640 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa833b90 0x563cfa83a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.430+0000 7f9f96400640 1 -- 172.21.15.6:0/1902894219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa833b90 msgr2=0x563cfa83a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.430+0000 7f9f96400640 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa833b90 0x563cfa83a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.430+0000 7f9f96400640 1 -- 172.21.15.6:0/1902894219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8292d0 msgr2=0x563cfa839d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.430+0000 7f9f96400640 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8292d0 0x563cfa839d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.430+0000 7f9f96400640 1 -- 172.21.15.6:0/1902894219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cfa8a96e0 con 0x563cfa8289b0 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.430+0000 7f9f95000640 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8292d0 0x563cfa839d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.431+0000 7f9f96400640 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8289b0 0x563cfa8397f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9f88006010 tx=0x7f9f88009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96670e36b612a6b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.431+0000 7f9f77400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:50:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.431+0000 7f9f77400640 1 -- 172.21.15.6:0/1902894219 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f880044a0 con 0x563cfa8289b0 2024-04-26T01:50:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.431+0000 7f9f77400640 1 -- 172.21.15.6:0/1902894219 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7f9f88004640 con 0x563cfa8289b0 2024-04-26T01:50:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.431+0000 7f9f9b280740 1 -- 172.21.15.6:0/1902894219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f7c041070 con 0x563cfa8289b0 2024-04-26T01:50:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.431+0000 7f9f77400640 1 -- 172.21.15.6:0/1902894219 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9f8800f4e0 con 0x563cfa8289b0 2024-04-26T01:50:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.431+0000 7f9f9b280740 10 client.15630 fetch_fsmap learned FSMap version 33 2024-04-26T01:50:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.431+0000 7f9f9b280740 1 -- 172.21.15.6:0/1902894219 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563cfa8aa9b0 con 0x563cfa8289b0 2024-04-26T01:50:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.431+0000 7f9f77400640 1 -- 172.21.15.6:0/1902894219 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f7c041070 con 0x563cfa8289b0 2024-04-26T01:50:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.432+0000 7f9f77400640 1 -- 172.21.15.6:0/1902894219 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f9f88039030 con 0x563cfa8289b0 2024-04-26T01:50:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.432+0000 7f9f9b280740 10 client.15630 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T01:50:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.432+0000 7f9f9b280740 10 client.15630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:50:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.432+0000 7f9f9b280740 20 client.15630 populate_metadata read hostname 'smithi006' 2024-04-26T01:50:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.432+0000 7f9f9b280740 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563cfa8aaeb0 0x563cfa8ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.432+0000 7f9f9b280740 4 client.15630 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.432+0000 7f9f9b280740 1 -- 172.21.15.6:0/1902894219 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563cfa79c320 con 0x563cfa8aaeb0 2024-04-26T01:50:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.432+0000 7f9f95000640 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563cfa8aaeb0 0x563cfa8ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:49.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.433+0000 7f9f95000640 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563cfa8aaeb0 0x563cfa8ad2d0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d21438c492e85e47 server_cookie=7d199ec845acd89 in_seq=0 out_seq=0 2024-04-26T01:50:49.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.433+0000 7f9f77400640 10 client.15630 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.486+0000 7f9f77400640 1 -- 172.21.15.6:0/1902894219 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563cfa79c320 con 0x563cfa8aaeb0 2024-04-26T01:50:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.486+0000 7f9f77400640 10 client.15630 handle_command_reply: tid=0 2024-04-26T01:50:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.486+0000 7f9f9b280740 1 client.15630 shutdown 2024-04-26T01:50:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.487+0000 7f9f9b280740 1 -- 172.21.15.6:0/1902894219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8289b0 msgr2=0x563cfa8397f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.487+0000 7f9f9b280740 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8289b0 0x563cfa8397f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9f88006010 tx=0x7f9f88009500 comp rx=0 tx=0).stop 2024-04-26T01:50:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.487+0000 7f9f9b280740 1 -- 172.21.15.6:0/1902894219 shutdown_connections 2024-04-26T01:50:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.487+0000 7f9f9b280740 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563cfa8aaeb0 0x563cfa8ad2d0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.487+0000 7f9f9b280740 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563cfa833b90 0x563cfa83a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.487+0000 7f9f9b280740 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfa8292d0 0x563cfa839d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.487+0000 7f9f9b280740 1 --2- 172.21.15.6:0/1902894219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cfa8289b0 0x563cfa8397f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.487+0000 7f9f77400640 0 client.15630 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:50:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.487+0000 7f9f9b280740 1 -- 172.21.15.6:0/1902894219 >> 172.21.15.6:0/1902894219 conn(0x563cfa824ec0 msgr2=0x563cfa832380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.487+0000 7f9f9b280740 1 -- 172.21.15.6:0/1902894219 shutdown_connections 2024-04-26T01:50:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.487+0000 7f9f9b280740 1 -- 172.21.15.6:0/1902894219 wait complete. 2024-04-26T01:50:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.487+0000 7f9f9b280740 20 client.15630 trim_cache size 0 max 16384 2024-04-26T01:50:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:50:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:50:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:50:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:50:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.488+0000 7f9f96e00640 1 -- 172.21.15.6:0/462495947 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9f780a3d00 msgr2=0x7f9f780a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.488+0000 7f9f96e00640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9f780a3d00 0x7f9f780a61c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x563cfa829f50 tx=0x7f9f840d4730 comp rx=0 tx=0).stop 2024-04-26T01:50:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.488+0000 7f9f96e00640 1 -- 172.21.15.6:0/462495947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f901083b0 msgr2=0x7f9f901a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.488+0000 7f9f96e00640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f901083b0 0x7f9f901a4780 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400e9f0 tx=0x7f9f8400edf0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.488+0000 7f9f96e00640 1 -- 172.21.15.6:0/462495947 shutdown_connections 2024-04-26T01:50:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.488+0000 7f9f96e00640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9f780a3d00 0x7f9f780a61c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.488+0000 7f9f96e00640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f9010e470 0x7f9f9019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.488+0000 7f9f96e00640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9f90108d50 0x7f9f901a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.488+0000 7f9f96e00640 1 --2- 172.21.15.6:0/462495947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f901083b0 0x7f9f901a4780 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.488+0000 7f9f96e00640 1 -- 172.21.15.6:0/462495947 >> 172.21.15.6:0/462495947 conn(0x7f9f90077b00 msgr2=0x7f9f90079300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.488+0000 7f9f96e00640 1 -- 172.21.15.6:0/462495947 shutdown_connections 2024-04-26T01:50:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:50:49.488+0000 7f9f96e00640 1 -- 172.21.15.6:0/462495947 wait complete. 2024-04-26T01:50:49.574 INFO:tasks.workunit.client.0.smithi006.stdout: 1 146176 3.93 MB/sec execute 73 sec latency 21.307 ms 2024-04-26T01:50:50.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:50 smithi006 ceph-mon[28419]: pgmap v152: 97 pgs: 97 active+clean; 56 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-04-26T01:50:50.574 INFO:tasks.workunit.client.0.smithi006.stdout: 1 147807 3.97 MB/sec execute 74 sec latency 16.300 ms 2024-04-26T01:50:50.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:50 smithi188 ceph-mon[32751]: pgmap v152: 97 pgs: 97 active+clean; 56 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-04-26T01:50:50.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:50 smithi029 ceph-mon[32775]: pgmap v152: 97 pgs: 97 active+clean; 56 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-04-26T01:50:51.575 INFO:tasks.workunit.client.0.smithi006.stdout: 1 149237 4.07 MB/sec execute 75 sec latency 17.184 ms 2024-04-26T01:50:52.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:52 smithi006 ceph-mon[28419]: pgmap v153: 97 pgs: 97 active+clean; 57 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 263 op/s 2024-04-26T01:50:52.574 INFO:tasks.workunit.client.0.smithi006.stdout: 1 149915 4.03 MB/sec execute 76 sec latency 15.900 ms 2024-04-26T01:50:52.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:52 smithi188 ceph-mon[32751]: pgmap v153: 97 pgs: 97 active+clean; 57 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 263 op/s 2024-04-26T01:50:52.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:52 smithi029 ceph-mon[32775]: pgmap v153: 97 pgs: 97 active+clean; 57 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 263 op/s 2024-04-26T01:50:53.575 INFO:tasks.workunit.client.0.smithi006.stdout: 1 151455 4.06 MB/sec execute 77 sec latency 16.102 ms 2024-04-26T01:50:54.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:54 smithi006 ceph-mon[28419]: pgmap v154: 97 pgs: 97 active+clean; 57 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 165 op/s 2024-04-26T01:50:54.575 INFO:tasks.workunit.client.0.smithi006.stdout: 1 152779 4.16 MB/sec execute 78 sec latency 23.610 ms 2024-04-26T01:50:54.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:54 smithi188 ceph-mon[32751]: pgmap v154: 97 pgs: 97 active+clean; 57 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 165 op/s 2024-04-26T01:50:54.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:54 smithi029 ceph-mon[32775]: pgmap v154: 97 pgs: 97 active+clean; 57 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 165 op/s 2024-04-26T01:50:55.575 INFO:tasks.workunit.client.0.smithi006.stdout: 1 153425 4.12 MB/sec execute 79 sec latency 15.820 ms 2024-04-26T01:50:56.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:56 smithi006 ceph-mon[28419]: pgmap v155: 97 pgs: 97 active+clean; 57 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 165 op/s 2024-04-26T01:50:56.575 INFO:tasks.workunit.client.0.smithi006.stdout: 1 154951 4.15 MB/sec execute 80 sec latency 18.290 ms 2024-04-26T01:50:56.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:56 smithi188 ceph-mon[32751]: pgmap v155: 97 pgs: 97 active+clean; 57 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 165 op/s 2024-04-26T01:50:56.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:56 smithi029 ceph-mon[32775]: pgmap v155: 97 pgs: 97 active+clean; 57 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 165 op/s 2024-04-26T01:50:57.575 INFO:tasks.workunit.client.0.smithi006.stdout: 1 156224 4.23 MB/sec execute 81 sec latency 60.069 ms 2024-04-26T01:50:58.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:50:58 smithi006 ceph-mon[28419]: pgmap v156: 97 pgs: 97 active+clean; 60 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 239 op/s 2024-04-26T01:50:58.575 INFO:tasks.workunit.client.0.smithi006.stdout: 1 156815 4.20 MB/sec execute 82 sec latency 33.978 ms 2024-04-26T01:50:58.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:50:58 smithi188 ceph-mon[32751]: pgmap v156: 97 pgs: 97 active+clean; 60 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 239 op/s 2024-04-26T01:50:58.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:50:58 smithi029 ceph-mon[32775]: pgmap v156: 97 pgs: 97 active+clean; 60 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 239 op/s 2024-04-26T01:50:59.575 INFO:tasks.workunit.client.0.smithi006.stdout: 1 158157 4.21 MB/sec execute 83 sec latency 18.786 ms 2024-04-26T01:51:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:00 smithi006 ceph-mon[28419]: pgmap v157: 97 pgs: 97 active+clean; 60 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-04-26T01:51:00.575 INFO:tasks.workunit.client.0.smithi006.stdout: 1 159643 4.30 MB/sec execute 84 sec latency 17.596 ms 2024-04-26T01:51:00.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:00 smithi188 ceph-mon[32751]: pgmap v157: 97 pgs: 97 active+clean; 60 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-04-26T01:51:00.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:00 smithi029 ceph-mon[32775]: pgmap v157: 97 pgs: 97 active+clean; 60 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-04-26T01:51:01.576 INFO:tasks.workunit.client.0.smithi006.stdout: 1 159949 4.27 MB/sec execute 85 sec latency 53.848 ms 2024-04-26T01:51:02.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:02 smithi006 ceph-mon[28419]: pgmap v158: 97 pgs: 97 active+clean; 63 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 262 op/s 2024-04-26T01:51:02.576 INFO:tasks.workunit.client.0.smithi006.stdout: 1 160608 4.23 MB/sec execute 86 sec latency 46.571 ms 2024-04-26T01:51:02.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.630+0000 7fa79cc00640 1 -- 172.21.15.6:0/3842879051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa79810c590 msgr2=0x7fa79810e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:02.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.630+0000 7fa79cc00640 1 --2- 172.21.15.6:0/3842879051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa79810c590 0x7fa79810e9c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa790009ca0 tx=0x7fa79002ea20 comp rx=0 tx=0).stop 2024-04-26T01:51:02.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.631+0000 7fa79cc00640 1 -- 172.21.15.6:0/3842879051 shutdown_connections 2024-04-26T01:51:02.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.631+0000 7fa79cc00640 1 --2- 172.21.15.6:0/3842879051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa798103ae0 0x7fa79810bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.631+0000 7fa79cc00640 1 --2- 172.21.15.6:0/3842879051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa79810c590 0x7fa79810e9c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.631+0000 7fa79cc00640 1 --2- 172.21.15.6:0/3842879051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7981031c0 0x7fa7981035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.631+0000 7fa79cc00640 1 -- 172.21.15.6:0/3842879051 >> 172.21.15.6:0/3842879051 conn(0x7fa798077f70 msgr2=0x7fa798078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:02.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.631+0000 7fa79cc00640 1 -- 172.21.15.6:0/3842879051 shutdown_connections 2024-04-26T01:51:02.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.631+0000 7fa79cc00640 1 -- 172.21.15.6:0/3842879051 wait complete. 2024-04-26T01:51:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.632+0000 7fa79cc00640 1 Processor -- start 2024-04-26T01:51:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.632+0000 7fa79cc00640 1 -- start start 2024-04-26T01:51:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.632+0000 7fa79cc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7981031c0 0x7fa7981964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.632+0000 7fa79cc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa798103ae0 0x7fa798196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.632+0000 7fa79cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79810c590 0x7fa79819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.632+0000 7fa79cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa798110010 con 0x7fa79810c590 2024-04-26T01:51:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.632+0000 7fa79cc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa79810fe90 con 0x7fa7981031c0 2024-04-26T01:51:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.632+0000 7fa79cc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa798110190 con 0x7fa798103ae0 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.632+0000 7fa795e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7981031c0 0x7fa7981964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.632+0000 7fa795400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa798103ae0 0x7fa798196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.632+0000 7fa795e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7981031c0 0x7fa7981964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:54194/0 (socket says 172.21.15.6:54194) 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.632+0000 7fa795e00640 1 -- 172.21.15.6:0/2524882389 learned_addr learned my addr 172.21.15.6:0/2524882389 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.633+0000 7fa796800640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79810c590 0x7fa79819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.633+0000 7fa795e00640 1 -- 172.21.15.6:0/2524882389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa798103ae0 msgr2=0x7fa798196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.633+0000 7fa795e00640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa798103ae0 0x7fa798196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.633+0000 7fa795e00640 1 -- 172.21.15.6:0/2524882389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79810c590 msgr2=0x7fa79819ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.633+0000 7fa795e00640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79810c590 0x7fa79819ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.633+0000 7fa795e00640 1 -- 172.21.15.6:0/2524882389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79819b4c0 con 0x7fa7981031c0 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.633+0000 7fa796800640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79810c590 0x7fa79819ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.633+0000 7fa795e00640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7981031c0 0x7fa7981964d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa78800e9f0 tx=0x7fa78800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.633+0000 7fa78ea00640 1 -- 172.21.15.6:0/2524882389 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78800cd80 con 0x7fa7981031c0 2024-04-26T01:51:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.634+0000 7fa78ea00640 1 -- 172.21.15.6:0/2524882389 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7880108e0 con 0x7fa7981031c0 2024-04-26T01:51:02.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.634+0000 7fa79cc00640 1 -- 172.21.15.6:0/2524882389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa79819b7b0 con 0x7fa7981031c0 2024-04-26T01:51:02.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.634+0000 7fa78ea00640 1 -- 172.21.15.6:0/2524882389 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa788018980 con 0x7fa7981031c0 2024-04-26T01:51:02.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.634+0000 7fa79cc00640 1 -- 172.21.15.6:0/2524882389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7981a3040 con 0x7fa7981031c0 2024-04-26T01:51:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.637+0000 7fa78ea00640 1 -- 172.21.15.6:0/2524882389 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fa788010430 con 0x7fa7981031c0 2024-04-26T01:51:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.637+0000 7fa79f9e8740 1 Processor -- start 2024-04-26T01:51:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.637+0000 7fa79f9e8740 1 -- start start 2024-04-26T01:51:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa79f9e8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 0x55b3c860fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa79f9e8740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 0x55b3c861a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa79f9e8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c861aa00 0x55b3c861cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa79f9e8740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55b3c8575900 con 0x55b3c8610140 2024-04-26T01:51:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa79f9e8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b3c851bef0 con 0x55b3c861aa00 2024-04-26T01:51:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa79f9e8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b3c85718c0 con 0x55b3c860f820 2024-04-26T01:51:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa796800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 0x55b3c860fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa796800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 0x55b3c860fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45554/0 (socket says 172.21.15.6:45554) 2024-04-26T01:51:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa796800640 1 -- 172.21.15.6:0/1263339533 learned_addr learned my addr 172.21.15.6:0/1263339533 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa795400640 1 --2- 172.21.15.6:0/1263339533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c861aa00 0x55b3c861cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa795e00640 1 --2- 172.21.15.6:0/1263339533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 0x55b3c861a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa796800640 1 -- 172.21.15.6:0/1263339533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c861aa00 msgr2=0x55b3c861cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa796800640 1 --2- 172.21.15.6:0/1263339533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c861aa00 0x55b3c861cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa796800640 1 -- 172.21.15.6:0/1263339533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 msgr2=0x55b3c861a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa796800640 1 --2- 172.21.15.6:0/1263339533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 0x55b3c861a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa796800640 1 -- 172.21.15.6:0/1263339533 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3c8288210 con 0x55b3c860f820 2024-04-26T01:51:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.638+0000 7fa795400640 1 --2- 172.21.15.6:0/1263339533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c861aa00 0x55b3c861cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.639+0000 7fa78ea00640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa77c0a3d90 0x7fa77c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.639+0000 7fa796800640 1 --2- 172.21.15.6:0/1263339533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 0x55b3c860fc00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa790009ca0 tx=0x7fa790009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66185c947a6c7165 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.639+0000 7fa78ea00640 1 -- 172.21.15.6:0/2524882389 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fa7880d0ba0 con 0x7fa7981031c0 2024-04-26T01:51:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.639+0000 7fa783000640 1 -- 172.21.15.6:0/1263339533 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790037050 con 0x55b3c860f820 2024-04-26T01:51:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.639+0000 7fa795e00640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa77c0a3d90 0x7fa77c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.639+0000 7fa783000640 1 -- 172.21.15.6:0/1263339533 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa790003e20 con 0x55b3c860f820 2024-04-26T01:51:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.639+0000 7fa783000640 1 -- 172.21.15.6:0/1263339533 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900338e0 con 0x55b3c860f820 2024-04-26T01:51:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.640+0000 7fa79f9e8740 1 -- 172.21.15.6:0/1263339533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 msgr2=0x55b3c860fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.640+0000 7fa795e00640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa77c0a3d90 0x7fa77c0a6250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55b3c8610dc0 tx=0x7fa7880d4650 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.640+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/1263339533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 0x55b3c860fc00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa790009ca0 tx=0x7fa790009a20 comp rx=0 tx=0).stop 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.640+0000 7fa79f9e8740 1 -- 172.21.15.6:0/1263339533 shutdown_connections 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.640+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/1263339533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 0x55b3c861a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.640+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/1263339533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c861aa00 0x55b3c861cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.640+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/1263339533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 0x55b3c860fc00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.640+0000 7fa79f9e8740 1 -- 172.21.15.6:0/1263339533 >> 172.21.15.6:0/1263339533 conn(0x55b3c860bd30 msgr2=0x55b3c860c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.640+0000 7fa79f9e8740 1 -- 172.21.15.6:0/1263339533 shutdown_connections 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.640+0000 7fa79f9e8740 1 -- 172.21.15.6:0/1263339533 wait complete. 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.641+0000 7fa79f9e8740 1 -- start start 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.641+0000 7fa79f9e8740 1 Processor -- start 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.641+0000 7fa79f9e8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c860f820 0x55b3c86179a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.641+0000 7fa79f9e8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c8610140 0x55b3c8617ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.641+0000 7fa79f9e8740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c861aa00 0x55b3c8618420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.641+0000 7fa79f9e8740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55b3c861d5d0 con 0x55b3c861aa00 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.641+0000 7fa79f9e8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b3c852f310 con 0x55b3c860f820 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.641+0000 7fa79f9e8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b3c861d750 con 0x55b3c8610140 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.641+0000 7fa79f9e8740 10 client.0 _ll_register_callbacks cb 0x55b3c84fdcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.642+0000 7fa795400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c8610140 0x55b3c8617ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.642+0000 7fa796800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c860f820 0x55b3c86179a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.642+0000 7fa795400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c8610140 0x55b3c8617ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45556/0 (socket says 172.21.15.6:45556) 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.642+0000 7fa795400640 1 -- 172.21.15.6:0/2782523133 learned_addr learned my addr 172.21.15.6:0/2782523133 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.642+0000 7fa796800640 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c861aa00 0x55b3c8618420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.642+0000 7fa795400640 1 -- 172.21.15.6:0/2782523133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c860f820 msgr2=0x55b3c86179a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.642+0000 7fa795400640 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c860f820 0x55b3c86179a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.642+0000 7fa795400640 1 -- 172.21.15.6:0/2782523133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c861aa00 msgr2=0x55b3c8618420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.642+0000 7fa795400640 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c861aa00 0x55b3c8618420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.642+0000 7fa795400640 1 -- 172.21.15.6:0/2782523133 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3c8354620 con 0x55b3c8610140 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.642+0000 7fa796800640 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c861aa00 0x55b3c8618420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.642+0000 7fa795400640 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c8610140 0x55b3c8617ee0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa784012730 tx=0x7fa784012bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2590c58f70f233d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.643+0000 7fa777400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:51:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.643+0000 7fa777400640 1 -- 172.21.15.6:0/2782523133 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa784004050 con 0x55b3c8610140 2024-04-26T01:51:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.643+0000 7fa777400640 1 -- 172.21.15.6:0/2782523133 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fa7840041f0 con 0x55b3c8610140 2024-04-26T01:51:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.643+0000 7fa79f9e8740 1 -- 172.21.15.6:0/2782523133 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa790003e20 con 0x55b3c8610140 2024-04-26T01:51:02.644 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:02 smithi188 ceph-mon[32751]: pgmap v158: 97 pgs: 97 active+clean; 63 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 262 op/s 2024-04-26T01:51:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.643+0000 7fa777400640 1 -- 172.21.15.6:0/2782523133 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa7840043f0 con 0x55b3c8610140 2024-04-26T01:51:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.643+0000 7fa79f9e8740 10 client.25813 fetch_fsmap learned FSMap version 33 2024-04-26T01:51:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.643+0000 7fa79f9e8740 1 -- 172.21.15.6:0/2782523133 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b3c8629940 con 0x55b3c8610140 2024-04-26T01:51:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.643+0000 7fa777400640 1 -- 172.21.15.6:0/2782523133 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa790003e20 con 0x55b3c8610140 2024-04-26T01:51:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.644+0000 7fa777400640 1 -- 172.21.15.6:0/2782523133 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fa784003b60 con 0x55b3c8610140 2024-04-26T01:51:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.644+0000 7fa79f9e8740 10 client.25813 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.644+0000 7fa79f9e8740 10 client.25813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.644+0000 7fa79f9e8740 20 client.25813 populate_metadata read hostname 'smithi006' 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.644+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b3c862ae70 0x55b3c862d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.644+0000 7fa79f9e8740 4 client.25813 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.644+0000 7fa79f9e8740 1 -- 172.21.15.6:0/2782523133 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3c85898b0 con 0x55b3c862ae70 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.644+0000 7fa796800640 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b3c862ae70 0x55b3c862d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.645+0000 7fa796800640 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b3c862ae70 0x55b3c862d280 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47c5b7dbec3e09d0 server_cookie=626a218cc1da454e in_seq=0 out_seq=0 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.645+0000 7fa777400640 10 client.25813 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.645+0000 7fa777400640 1 -- 172.21.15.6:0/2782523133 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3c85898b0 con 0x55b3c862ae70 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.645+0000 7fa777400640 10 client.25813 handle_command_reply: tid=0 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.645+0000 7fa79f9e8740 1 client.25813 shutdown 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.646+0000 7fa79f9e8740 1 -- 172.21.15.6:0/2782523133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c8610140 msgr2=0x55b3c8617ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.646+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c8610140 0x55b3c8617ee0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa784012730 tx=0x7fa784012bf0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.646+0000 7fa79f9e8740 1 -- 172.21.15.6:0/2782523133 shutdown_connections 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.646+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b3c862ae70 0x55b3c862d280 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.646+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c861aa00 0x55b3c8618420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.646+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c8610140 0x55b3c8617ee0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.646+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/2782523133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c860f820 0x55b3c86179a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.646+0000 7fa777400640 0 client.25813 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.646+0000 7fa79f9e8740 1 -- 172.21.15.6:0/2782523133 >> 172.21.15.6:0/2782523133 conn(0x55b3c860bd30 msgr2=0x55b3c8611ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.646+0000 7fa79f9e8740 1 -- 172.21.15.6:0/2782523133 shutdown_connections 2024-04-26T01:51:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.646+0000 7fa79f9e8740 1 -- 172.21.15.6:0/2782523133 wait complete. 2024-04-26T01:51:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.646+0000 7fa79f9e8740 20 client.25813 trim_cache size 0 max 16384 2024-04-26T01:51:02.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.664+0000 7fa79f9e8740 1 Processor -- start 2024-04-26T01:51:02.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.664+0000 7fa79f9e8740 1 -- start start 2024-04-26T01:51:02.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.664+0000 7fa79f9e8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c860f820 0x55b3c8626490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.664+0000 7fa79f9e8740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 0x55b3c860b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.664+0000 7fa79f9e8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c861aa00 0x55b3c8632be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.664+0000 7fa79f9e8740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa7981ae360 con 0x55b3c8610140 2024-04-26T01:51:02.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.664+0000 7fa79f9e8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7981abbd0 con 0x55b3c860f820 2024-04-26T01:51:02.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.664+0000 7fa79f9e8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa7981a9a20 con 0x55b3c861aa00 2024-04-26T01:51:02.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.664+0000 7fa796800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c860f820 0x55b3c8626490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.664+0000 7fa796800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c860f820 0x55b3c8626490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:54226/0 (socket says 172.21.15.6:54226) 2024-04-26T01:51:02.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.664+0000 7fa796800640 1 -- 172.21.15.6:0/4025443608 learned_addr learned my addr 172.21.15.6:0/4025443608 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:02.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.664+0000 7fa796800640 1 --2- 172.21.15.6:0/4025443608 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c861aa00 0x55b3c8632be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.665+0000 7fa795400640 1 --2- 172.21.15.6:0/4025443608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 0x55b3c860b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.665+0000 7fa796800640 1 -- 172.21.15.6:0/4025443608 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c861aa00 msgr2=0x55b3c8632be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:02.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.665+0000 7fa796800640 1 --2- 172.21.15.6:0/4025443608 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c861aa00 0x55b3c8632be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.665+0000 7fa796800640 1 -- 172.21.15.6:0/4025443608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 msgr2=0x55b3c860b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:02.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.665+0000 7fa796800640 1 --2- 172.21.15.6:0/4025443608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 0x55b3c860b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.665+0000 7fa796800640 1 -- 172.21.15.6:0/4025443608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3c8610ad0 con 0x55b3c860f820 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.665+0000 7fa796800640 1 --2- 172.21.15.6:0/4025443608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c860f820 0x55b3c8626490 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa790006010 tx=0x7fa79002ea20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a5815b5309490d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.665+0000 7fa782600640 1 -- 172.21.15.6:0/4025443608 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790037030 con 0x55b3c860f820 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.665+0000 7fa782600640 1 -- 172.21.15.6:0/4025443608 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa79005a030 con 0x55b3c860f820 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.665+0000 7fa782600640 1 -- 172.21.15.6:0/4025443608 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790032630 con 0x55b3c860f820 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.666+0000 7fa79f9e8740 1 -- 172.21.15.6:0/4025443608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c860f820 msgr2=0x55b3c8626490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.666+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/4025443608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c860f820 0x55b3c8626490 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa790006010 tx=0x7fa79002ea20 comp rx=0 tx=0).stop 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.666+0000 7fa79f9e8740 1 -- 172.21.15.6:0/4025443608 shutdown_connections 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.666+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/4025443608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 0x55b3c860b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.666+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/4025443608 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c861aa00 0x55b3c8632be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.666+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/4025443608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c860f820 0x55b3c8626490 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.666+0000 7fa79f9e8740 1 -- 172.21.15.6:0/4025443608 >> 172.21.15.6:0/4025443608 conn(0x55b3c860bd30 msgr2=0x55b3c86181f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.666+0000 7fa79f9e8740 1 -- 172.21.15.6:0/4025443608 shutdown_connections 2024-04-26T01:51:02.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.666+0000 7fa79f9e8740 1 -- 172.21.15.6:0/4025443608 wait complete. 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.667+0000 7fa79f9e8740 1 -- start start 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.667+0000 7fa79f9e8740 1 Processor -- start 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.667+0000 7fa79f9e8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 0x55b3c86205b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.667+0000 7fa79f9e8740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 0x55b3c8620af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.667+0000 7fa79f9e8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c861aa00 0x55b3c8621030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa79f9e8740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa7981aa090 con 0x55b3c8610140 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa79f9e8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7981ab500 con 0x55b3c861aa00 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa79f9e8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa7981aa760 con 0x55b3c860f820 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa79f9e8740 10 client.0 _ll_register_callbacks cb 0x7fa784009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa796800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 0x55b3c86205b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa795400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 0x55b3c8620af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa796800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 0x55b3c86205b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45574/0 (socket says 172.21.15.6:45574) 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa796800640 1 -- 172.21.15.6:0/3571833079 learned_addr learned my addr 172.21.15.6:0/3571833079 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa796800640 1 --2- 172.21.15.6:0/3571833079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c861aa00 0x55b3c8621030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa796800640 1 -- 172.21.15.6:0/3571833079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c861aa00 msgr2=0x55b3c8621030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa796800640 1 --2- 172.21.15.6:0/3571833079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c861aa00 0x55b3c8621030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa796800640 1 -- 172.21.15.6:0/3571833079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 msgr2=0x55b3c8620af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa796800640 1 --2- 172.21.15.6:0/3571833079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 0x55b3c8620af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa796800640 1 -- 172.21.15.6:0/3571833079 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3c8690660 con 0x55b3c860f820 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa796800640 1 --2- 172.21.15.6:0/3571833079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 0x55b3c86205b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa790009620 tx=0x7fa790009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3349b54b8b14c23b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.668+0000 7fa777400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa777400640 1 -- 172.21.15.6:0/3571833079 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790037040 con 0x55b3c860f820 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa777400640 1 -- 172.21.15.6:0/3571833079 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8556+0+0 (secure 0 0 0) 0x7fa790004b40 con 0x55b3c860f820 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa79f9e8740 1 -- 172.21.15.6:0/3571833079 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa79005a030 con 0x55b3c860f820 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa777400640 1 -- 172.21.15.6:0/3571833079 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa790004d40 con 0x55b3c860f820 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa79f9e8740 10 client.25819 fetch_fsmap learned FSMap version 33 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa79f9e8740 1 -- 172.21.15.6:0/3571833079 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b3c8691930 con 0x55b3c860f820 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa777400640 1 -- 172.21.15.6:0/3571833079 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa79005a030 con 0x55b3c860f820 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa777400640 1 -- 172.21.15.6:0/3571833079 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fa79005d720 con 0x55b3c860f820 2024-04-26T01:51:02.670 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa79f9e8740 10 client.25819 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T01:51:02.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa79f9e8740 10 client.25819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:02.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa79f9e8740 20 client.25819 populate_metadata read hostname 'smithi006' 2024-04-26T01:51:02.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/3571833079 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b3c8691b50 0x55b3c8693f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:02.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa79f9e8740 4 client.25819 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:02.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.669+0000 7fa79f9e8740 1 -- 172.21.15.6:0/3571833079 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b3c8583320 con 0x55b3c8691b50 2024-04-26T01:51:02.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.670+0000 7fa795400640 1 --2- 172.21.15.6:0/3571833079 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b3c8691b50 0x55b3c8693f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:02.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.670+0000 7fa795400640 1 --2- 172.21.15.6:0/3571833079 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b3c8691b50 0x55b3c8693f40 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58f414171cd73723 server_cookie=f4afdf233a0977e3 in_seq=0 out_seq=0 2024-04-26T01:51:02.671 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.670+0000 7fa777400640 10 client.25819 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:02.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:02 smithi029 ceph-mon[32775]: pgmap v158: 97 pgs: 97 active+clean; 63 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 262 op/s 2024-04-26T01:51:02.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.710+0000 7fa777400640 1 -- 172.21.15.6:0/3571833079 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b3c8583320 con 0x55b3c8691b50 2024-04-26T01:51:02.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.710+0000 7fa777400640 10 client.25819 handle_command_reply: tid=0 2024-04-26T01:51:02.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.710+0000 7fa79f9e8740 1 client.25819 shutdown 2024-04-26T01:51:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.710+0000 7fa79f9e8740 1 -- 172.21.15.6:0/3571833079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 msgr2=0x55b3c86205b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.710+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/3571833079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 0x55b3c86205b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa790009620 tx=0x7fa790009a20 comp rx=0 tx=0).stop 2024-04-26T01:51:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.710+0000 7fa79f9e8740 1 -- 172.21.15.6:0/3571833079 shutdown_connections 2024-04-26T01:51:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.710+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/3571833079 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b3c8691b50 0x55b3c8693f40 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.710+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/3571833079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b3c8610140 0x55b3c8620af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.710+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/3571833079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3c861aa00 0x55b3c8621030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.710+0000 7fa79f9e8740 1 --2- 172.21.15.6:0/3571833079 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b3c860f820 0x55b3c86205b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.710+0000 7fa79f9e8740 1 -- 172.21.15.6:0/3571833079 >> 172.21.15.6:0/3571833079 conn(0x55b3c860bd30 msgr2=0x55b3c86111d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.710+0000 7fa777400640 0 client.25819 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.711+0000 7fa79f9e8740 1 -- 172.21.15.6:0/3571833079 shutdown_connections 2024-04-26T01:51:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.711+0000 7fa79f9e8740 1 -- 172.21.15.6:0/3571833079 wait complete. 2024-04-26T01:51:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.711+0000 7fa79f9e8740 20 client.25819 trim_cache size 0 max 16384 2024-04-26T01:51:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:51:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:51:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:51:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:51:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.712+0000 7fa79cc00640 1 -- 172.21.15.6:0/2524882389 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa77c0a3d90 msgr2=0x7fa77c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.712+0000 7fa79cc00640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa77c0a3d90 0x7fa77c0a6250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55b3c8610dc0 tx=0x7fa7880d4650 comp rx=0 tx=0).stop 2024-04-26T01:51:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.712+0000 7fa79cc00640 1 -- 172.21.15.6:0/2524882389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7981031c0 msgr2=0x7fa7981964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.712+0000 7fa79cc00640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7981031c0 0x7fa7981964d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa78800e9f0 tx=0x7fa78800eeb0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.712+0000 7fa79cc00640 1 -- 172.21.15.6:0/2524882389 shutdown_connections 2024-04-26T01:51:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.712+0000 7fa79cc00640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa77c0a3d90 0x7fa77c0a6250 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.712+0000 7fa79cc00640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa79810c590 0x7fa79819ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.712+0000 7fa79cc00640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa798103ae0 0x7fa798196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.712+0000 7fa79cc00640 1 --2- 172.21.15.6:0/2524882389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7981031c0 0x7fa7981964d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.712+0000 7fa79cc00640 1 -- 172.21.15.6:0/2524882389 >> 172.21.15.6:0/2524882389 conn(0x7fa798077f70 msgr2=0x7fa7980fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.712+0000 7fa79cc00640 1 -- 172.21.15.6:0/2524882389 shutdown_connections 2024-04-26T01:51:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:02.712+0000 7fa79cc00640 1 -- 172.21.15.6:0/2524882389 wait complete. 2024-04-26T01:51:03.140 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:51:02 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: ignoring --setuser ceph since I am not root 2024-04-26T01:51:03.140 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:51:02 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: ignoring --setgroup ceph since I am not root 2024-04-26T01:51:03.140 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:51:02 smithi188 ceph-mds[68153]: -- 172.21.15.188:0/3023418378 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557a645d7a40 con 0x557a645aac00 2024-04-26T01:51:03.140 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:51:02 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: starting mds.f at 2024-04-26T01:51:03.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:03 smithi006 ceph-mon[28419]: daemon mds.f restarted 2024-04-26T01:51:03.576 INFO:tasks.workunit.client.0.smithi006.stdout: 1 162057 4.25 MB/sec execute 87 sec latency 86.636 ms 2024-04-26T01:51:03.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:03 smithi188 ceph-mon[32751]: daemon mds.f restarted 2024-04-26T01:51:03.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:03 smithi029 ceph-mon[32775]: daemon mds.f restarted 2024-04-26T01:51:04.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:04 smithi006 ceph-mon[28419]: pgmap v159: 97 pgs: 97 active+clean; 63 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 171 op/s 2024-04-26T01:51:04.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:04 smithi006 ceph-mon[28419]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T01:51:04.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:04 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:51:04.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:04 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6834/1129799058,v1:172.21.15.188:6835/1129799058] up:boot 2024-04-26T01:51:04.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:04 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:51:04.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:04 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:04.576 INFO:tasks.workunit.client.0.smithi006.stdout: 1 163435 4.34 MB/sec execute 88 sec latency 38.180 ms 2024-04-26T01:51:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:04 smithi188 ceph-mon[32751]: pgmap v159: 97 pgs: 97 active+clean; 63 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 171 op/s 2024-04-26T01:51:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:04 smithi188 ceph-mon[32751]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T01:51:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:04 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:51:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:04 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6834/1129799058,v1:172.21.15.188:6835/1129799058] up:boot 2024-04-26T01:51:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:04 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:51:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:04 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:04 smithi029 ceph-mon[32775]: pgmap v159: 97 pgs: 97 active+clean; 63 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 171 op/s 2024-04-26T01:51:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:04 smithi029 ceph-mon[32775]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T01:51:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:04 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:51:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:04 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6834/1129799058,v1:172.21.15.188:6835/1129799058] up:boot 2024-04-26T01:51:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:04 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:51:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:04 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:05.576 INFO:tasks.workunit.client.0.smithi006.stdout: 1 164277 4.32 MB/sec execute 89 sec latency 18.472 ms 2024-04-26T01:51:06.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:06 smithi006 ceph-mon[28419]: pgmap v161: 97 pgs: 97 active+clean; 63 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 205 op/s 2024-04-26T01:51:06.576 INFO:tasks.workunit.client.0.smithi006.stdout: 1 165956 4.37 MB/sec execute 90 sec latency 17.609 ms 2024-04-26T01:51:06.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:06 smithi188 ceph-mon[32751]: pgmap v161: 97 pgs: 97 active+clean; 63 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 205 op/s 2024-04-26T01:51:06.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:06 smithi029 ceph-mon[32775]: pgmap v161: 97 pgs: 97 active+clean; 63 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 205 op/s 2024-04-26T01:51:07.577 INFO:tasks.workunit.client.0.smithi006.stdout: 1 167115 4.41 MB/sec execute 91 sec latency 21.669 ms 2024-04-26T01:51:08.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:08 smithi006 ceph-mon[28419]: pgmap v162: 97 pgs: 97 active+clean; 59 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 253 op/s 2024-04-26T01:51:08.577 INFO:tasks.workunit.client.0.smithi006.stdout: 1 167327 4.36 MB/sec execute 92 sec latency 627.365 ms 2024-04-26T01:51:08.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:08 smithi188 ceph-mon[32751]: pgmap v162: 97 pgs: 97 active+clean; 59 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 253 op/s 2024-04-26T01:51:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:08 smithi029 ceph-mon[32775]: pgmap v162: 97 pgs: 97 active+clean; 59 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 253 op/s 2024-04-26T01:51:09.577 INFO:tasks.workunit.client.0.smithi006.stdout: 1 167327 4.31 MB/sec execute 93 sec latency 1627.524 ms 2024-04-26T01:51:10.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:10 smithi006 ceph-mon[28419]: pgmap v163: 97 pgs: 97 active+clean; 59 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 253 op/s 2024-04-26T01:51:10.577 INFO:tasks.workunit.client.0.smithi006.stdout: 1 167327 4.27 MB/sec execute 94 sec latency 2627.618 ms 2024-04-26T01:51:10.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:10 smithi188 ceph-mon[32751]: pgmap v163: 97 pgs: 97 active+clean; 59 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 253 op/s 2024-04-26T01:51:10.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:10 smithi029 ceph-mon[32775]: pgmap v163: 97 pgs: 97 active+clean; 59 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 253 op/s 2024-04-26T01:51:11.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:11 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:51:11.577 INFO:tasks.workunit.client.0.smithi006.stdout: 1 167327 4.22 MB/sec execute 95 sec latency 3627.746 ms 2024-04-26T01:51:11.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:11 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:51:11.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:11 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:51:12.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:12 smithi006 ceph-mon[28419]: pgmap v164: 97 pgs: 97 active+clean; 60 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 194 op/s 2024-04-26T01:51:12.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:12 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:51:12.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:12 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:51:12.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:12 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:51:12.578 INFO:tasks.workunit.client.0.smithi006.stdout: 1 167327 4.18 MB/sec execute 96 sec latency 4627.890 ms 2024-04-26T01:51:12.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:12 smithi188 ceph-mon[32751]: pgmap v164: 97 pgs: 97 active+clean; 60 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 194 op/s 2024-04-26T01:51:12.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:12 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:51:12.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:12 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:51:12.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:12 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:51:12.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:12 smithi029 ceph-mon[32775]: pgmap v164: 97 pgs: 97 active+clean; 60 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 194 op/s 2024-04-26T01:51:12.692 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:12 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:51:12.692 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:12 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:51:12.692 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:12 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:51:13.578 INFO:tasks.workunit.client.0.smithi006.stdout: 1 167327 4.14 MB/sec execute 97 sec latency 5628.029 ms 2024-04-26T01:51:14.578 INFO:tasks.workunit.client.0.smithi006.stdout: 1 167327 4.09 MB/sec execute 98 sec latency 6628.190 ms 2024-04-26T01:51:14.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:14 smithi188 ceph-mon[32751]: pgmap v165: 97 pgs: 97 active+clean; 60 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 194 op/s 2024-04-26T01:51:14.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:14 smithi029 ceph-mon[32775]: pgmap v165: 97 pgs: 97 active+clean; 60 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 194 op/s 2024-04-26T01:51:14.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:14 smithi006 ceph-mon[28419]: pgmap v165: 97 pgs: 97 active+clean; 60 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 194 op/s 2024-04-26T01:51:15.578 INFO:tasks.workunit.client.0.smithi006.stdout: 1 167327 4.05 MB/sec execute 99 sec latency 7628.294 ms 2024-04-26T01:51:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.858+0000 7f385be00640 1 -- 172.21.15.6:0/2852614929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f385c104b80 msgr2=0x7f385c108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.858+0000 7f385be00640 1 --2- 172.21.15.6:0/2852614929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f385c104b80 0x7f385c108bc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3848009900 tx=0x7f384802a3f0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.859+0000 7f385be00640 1 -- 172.21.15.6:0/2852614929 shutdown_connections 2024-04-26T01:51:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.859+0000 7f385be00640 1 --2- 172.21.15.6:0/2852614929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f385c1092a0 0x7f385c110e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.859+0000 7f385be00640 1 --2- 172.21.15.6:0/2852614929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f385c104b80 0x7f385c108bc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.859+0000 7f385be00640 1 --2- 172.21.15.6:0/2852614929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f385c1041d0 0x7f385c1045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.859+0000 7f385be00640 1 -- 172.21.15.6:0/2852614929 >> 172.21.15.6:0/2852614929 conn(0x7f385c0ffa30 msgr2=0x7f385c101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.859+0000 7f385be00640 1 -- 172.21.15.6:0/2852614929 shutdown_connections 2024-04-26T01:51:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.859+0000 7f385be00640 1 -- 172.21.15.6:0/2852614929 wait complete. 2024-04-26T01:51:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.860+0000 7f385be00640 1 Processor -- start 2024-04-26T01:51:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.860+0000 7f385be00640 1 -- start start 2024-04-26T01:51:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.860+0000 7f385be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f385c1041d0 0x7f385c1a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.860+0000 7f385be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f385c1092a0 0x7f385c1a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.860+0000 7f385be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f385c19a5f0 0x7f385c19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.860+0000 7f385be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f385c114230 con 0x7f385c1092a0 2024-04-26T01:51:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.860+0000 7f385be00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1140b0 con 0x7f385c19a5f0 2024-04-26T01:51:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.860+0000 7f385be00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1143b0 con 0x7f385c1041d0 2024-04-26T01:51:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.860+0000 7f385a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f385c1092a0 0x7f385c1a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.861+0000 7f385a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f385c1092a0 0x7f385c1a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36792/0 (socket says 172.21.15.6:36792) 2024-04-26T01:51:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.861+0000 7f385a000640 1 -- 172.21.15.6:0/1569454920 learned_addr learned my addr 172.21.15.6:0/1569454920 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.861+0000 7f385aa00640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f385c1041d0 0x7f385c1a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.861+0000 7f385b400640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f385c19a5f0 0x7f385c19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.861+0000 7f385a000640 1 -- 172.21.15.6:0/1569454920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f385c1041d0 msgr2=0x7f385c1a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.861+0000 7f385a000640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f385c1041d0 0x7f385c1a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.861+0000 7f385a000640 1 -- 172.21.15.6:0/1569454920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f385c19a5f0 msgr2=0x7f385c19aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.861+0000 7f385a000640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f385c19a5f0 0x7f385c19aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.861+0000 7f385a000640 1 -- 172.21.15.6:0/1569454920 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385c19b110 con 0x7f385c1092a0 2024-04-26T01:51:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.861+0000 7f385a000640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f385c1092a0 0x7f385c1a0930 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3848005ab0 tx=0x7f3848003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.862+0000 7f3853400640 1 -- 172.21.15.6:0/1569454920 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384803f070 con 0x7f385c1092a0 2024-04-26T01:51:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.862+0000 7f3853400640 1 -- 172.21.15.6:0/1569454920 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3848031b50 con 0x7f385c1092a0 2024-04-26T01:51:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.862+0000 7f3853400640 1 -- 172.21.15.6:0/1569454920 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3848035030 con 0x7f385c1092a0 2024-04-26T01:51:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.862+0000 7f385be00640 1 -- 172.21.15.6:0/1569454920 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f385c19b3a0 con 0x7f385c1092a0 2024-04-26T01:51:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.862+0000 7f385be00640 1 -- 172.21.15.6:0/1569454920 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f385c1a7540 con 0x7f385c1092a0 2024-04-26T01:51:15.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.865+0000 7f38647e1740 1 Processor -- start 2024-04-26T01:51:15.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.865+0000 7f3853400640 1 -- 172.21.15.6:0/1569454920 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f3848042070 con 0x7f385c1092a0 2024-04-26T01:51:15.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.865+0000 7f38647e1740 1 -- start start 2024-04-26T01:51:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.866+0000 7f38647e1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdea9b0 0x55780fdead90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.866+0000 7f38647e1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 0x55780fdf5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.866+0000 7f38647e1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdf5b90 0x55780fdf7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.866+0000 7f38647e1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55780fd50900 con 0x55780fdea9b0 2024-04-26T01:51:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.866+0000 7f38647e1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55780fcf6ef0 con 0x55780fdf5b90 2024-04-26T01:51:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.866+0000 7f38647e1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55780fd4c8c0 con 0x55780fdeb2d0 2024-04-26T01:51:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.866+0000 7f385a000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdf5b90 0x55780fdf7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.866+0000 7f385aa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 0x55780fdf5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.866+0000 7f385b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdea9b0 0x55780fdead90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.866+0000 7f385aa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 0x55780fdf5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38048/0 (socket says 172.21.15.6:38048) 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.866+0000 7f385aa00640 1 -- 172.21.15.6:0/705514348 learned_addr learned my addr 172.21.15.6:0/705514348 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f385aa00640 1 -- 172.21.15.6:0/705514348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdf5b90 msgr2=0x55780fdf7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f385aa00640 1 --2- 172.21.15.6:0/705514348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdf5b90 0x55780fdf7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f385aa00640 1 -- 172.21.15.6:0/705514348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdea9b0 msgr2=0x55780fdead90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f385aa00640 1 --2- 172.21.15.6:0/705514348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdea9b0 0x55780fdead90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f385aa00640 1 -- 172.21.15.6:0/705514348 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55780fa63210 con 0x55780fdeb2d0 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f385a000640 1 --2- 172.21.15.6:0/705514348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdf5b90 0x55780fdf7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f3853400640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38400a3d00 0x7f38400a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f385aa00640 1 --2- 172.21.15.6:0/705514348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 0x55780fdf5650 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f384c009cf0 tx=0x7f384c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ee0271d00d07ce3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f3853400640 1 -- 172.21.15.6:0/1569454920 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7f38480f0e50 con 0x7f385c1092a0 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f3847e00640 1 -- 172.21.15.6:0/705514348 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c004230 con 0x55780fdeb2d0 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f3847e00640 1 -- 172.21.15.6:0/705514348 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f384c0043d0 con 0x55780fdeb2d0 2024-04-26T01:51:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f385b400640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38400a3d00 0x7f38400a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.867+0000 7f3847e00640 1 -- 172.21.15.6:0/705514348 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c019820 con 0x55780fdeb2d0 2024-04-26T01:51:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.868+0000 7f385b400640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38400a3d00 0x7f38400a61c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3854005fd0 tx=0x7f3854009500 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.868+0000 7f38647e1740 1 -- 172.21.15.6:0/705514348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 msgr2=0x55780fdf5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.868+0000 7f38647e1740 1 --2- 172.21.15.6:0/705514348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 0x55780fdf5650 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f384c009cf0 tx=0x7f384c011810 comp rx=0 tx=0).stop 2024-04-26T01:51:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.868+0000 7f38647e1740 1 -- 172.21.15.6:0/705514348 shutdown_connections 2024-04-26T01:51:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.868+0000 7f38647e1740 1 --2- 172.21.15.6:0/705514348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdf5b90 0x55780fdf7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.868+0000 7f38647e1740 1 --2- 172.21.15.6:0/705514348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 0x55780fdf5650 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.868+0000 7f38647e1740 1 --2- 172.21.15.6:0/705514348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdea9b0 0x55780fdead90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.868+0000 7f38647e1740 1 -- 172.21.15.6:0/705514348 >> 172.21.15.6:0/705514348 conn(0x55780fde6ec0 msgr2=0x55780fde72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.868+0000 7f38647e1740 1 -- 172.21.15.6:0/705514348 shutdown_connections 2024-04-26T01:51:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.868+0000 7f38647e1740 1 -- 172.21.15.6:0/705514348 wait complete. 2024-04-26T01:51:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.869+0000 7f38647e1740 1 -- start start 2024-04-26T01:51:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.869+0000 7f38647e1740 1 Processor -- start 2024-04-26T01:51:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.869+0000 7f38647e1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdea9b0 0x55780fdf2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.869+0000 7f38647e1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdeb2d0 0x55780fdf2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.869+0000 7f38647e1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdf5b90 0x55780fdf34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.869+0000 7f38647e1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55780fdf87c0 con 0x55780fdea9b0 2024-04-26T01:51:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.869+0000 7f38647e1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55780fd0a310 con 0x55780fdeb2d0 2024-04-26T01:51:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.869+0000 7f38647e1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55780fdf8940 con 0x55780fdf5b90 2024-04-26T01:51:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.869+0000 7f38647e1740 10 client.0 _ll_register_callbacks cb 0x55780fcd8cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.870+0000 7f385aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdea9b0 0x55780fdf2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.870+0000 7f385aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdea9b0 0x55780fdf2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36812/0 (socket says 172.21.15.6:36812) 2024-04-26T01:51:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.870+0000 7f385aa00640 1 -- 172.21.15.6:0/1503510995 learned_addr learned my addr 172.21.15.6:0/1503510995 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.870+0000 7f385b400640 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdf5b90 0x55780fdf34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.870+0000 7f385a000640 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdeb2d0 0x55780fdf2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.870+0000 7f385aa00640 1 -- 172.21.15.6:0/1503510995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdf5b90 msgr2=0x55780fdf34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.870+0000 7f385aa00640 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdf5b90 0x55780fdf34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.870+0000 7f385aa00640 1 -- 172.21.15.6:0/1503510995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdeb2d0 msgr2=0x55780fdf2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.870+0000 7f385aa00640 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdeb2d0 0x55780fdf2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.870+0000 7f385aa00640 1 -- 172.21.15.6:0/1503510995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55780fb2f620 con 0x55780fdea9b0 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.871+0000 7f385aa00640 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdea9b0 0x55780fdf2a70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f384c0038d0 tx=0x7f384c0045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820da535137e3144 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.871+0000 7f3844c00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.871+0000 7f3844c00640 1 -- 172.21.15.6:0/1503510995 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c027070 con 0x55780fdea9b0 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.871+0000 7f3844c00640 1 -- 172.21.15.6:0/1503510995 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7f384c005240 con 0x55780fdea9b0 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.871+0000 7f38647e1740 1 -- 172.21.15.6:0/1503510995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f384c0043d0 con 0x55780fdea9b0 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.871+0000 7f3844c00640 1 -- 172.21.15.6:0/1503510995 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f384c033050 con 0x55780fdea9b0 2024-04-26T01:51:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.871+0000 7f38647e1740 10 client.15666 fetch_fsmap learned FSMap version 35 2024-04-26T01:51:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.871+0000 7f38647e1740 1 -- 172.21.15.6:0/1503510995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55780fe04960 con 0x55780fdea9b0 2024-04-26T01:51:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.871+0000 7f3844c00640 1 -- 172.21.15.6:0/1503510995 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f384c0043d0 con 0x55780fdea9b0 2024-04-26T01:51:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.872+0000 7f3844c00640 1 -- 172.21.15.6:0/1503510995 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f384c004b70 con 0x55780fdea9b0 2024-04-26T01:51:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.872+0000 7f38647e1740 10 client.15666 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T01:51:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.872+0000 7f38647e1740 10 client.15666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.872+0000 7f38647e1740 20 client.15666 populate_metadata read hostname 'smithi006' 2024-04-26T01:51:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.872+0000 7f38647e1740 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55780fe05d70 0x55780fe08160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.872+0000 7f38647e1740 4 client.15666 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.872+0000 7f38647e1740 1 -- 172.21.15.6:0/1503510995 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55780fd648b0 con 0x55780fe05d70 2024-04-26T01:51:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.872+0000 7f385a000640 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55780fe05d70 0x55780fe08160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.873+0000 7f385a000640 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55780fe05d70 0x55780fe08160 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c92f23e74726db7 server_cookie=f235baf60237b7b1 in_seq=0 out_seq=0 2024-04-26T01:51:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.873+0000 7f3844c00640 10 client.15666 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f3844c00640 1 -- 172.21.15.6:0/1503510995 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55780fd648b0 con 0x55780fe05d70 2024-04-26T01:51:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f3844c00640 10 client.15666 handle_command_reply: tid=0 2024-04-26T01:51:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f38647e1740 1 client.15666 shutdown 2024-04-26T01:51:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f38647e1740 1 -- 172.21.15.6:0/1503510995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdea9b0 msgr2=0x55780fdf2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f38647e1740 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdea9b0 0x55780fdf2a70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f384c0038d0 tx=0x7f384c0045f0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f38647e1740 1 -- 172.21.15.6:0/1503510995 shutdown_connections 2024-04-26T01:51:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f38647e1740 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55780fe05d70 0x55780fe08160 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f38647e1740 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdf5b90 0x55780fdf34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f38647e1740 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdeb2d0 0x55780fdf2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f38647e1740 1 --2- 172.21.15.6:0/1503510995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdea9b0 0x55780fdf2a70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f3844c00640 0 client.15666 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f38647e1740 1 -- 172.21.15.6:0/1503510995 >> 172.21.15.6:0/1503510995 conn(0x55780fde6ec0 msgr2=0x55780fdecc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f38647e1740 1 -- 172.21.15.6:0/1503510995 shutdown_connections 2024-04-26T01:51:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.874+0000 7f38647e1740 1 -- 172.21.15.6:0/1503510995 wait complete. 2024-04-26T01:51:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.875+0000 7f38647e1740 20 client.15666 trim_cache size 0 max 16384 2024-04-26T01:51:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.891+0000 7f38647e1740 1 Processor -- start 2024-04-26T01:51:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.891+0000 7f38647e1740 1 -- start start 2024-04-26T01:51:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.891+0000 7f38647e1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdea9b0 0x55780fe01680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.891+0000 7f38647e1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 0x55780fe0d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.891+0000 7f38647e1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdf5b90 0x55780fe0dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.891+0000 7f38647e1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1b2660 con 0x55780fdf5b90 2024-04-26T01:51:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.891+0000 7f38647e1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1afed0 con 0x55780fdea9b0 2024-04-26T01:51:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.891+0000 7f38647e1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1add20 con 0x55780fdeb2d0 2024-04-26T01:51:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.892+0000 7f385aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdea9b0 0x55780fe01680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.892+0000 7f385a000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 0x55780fe0d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.892+0000 7f385aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdea9b0 0x55780fe01680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:45938/0 (socket says 172.21.15.6:45938) 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.892+0000 7f385aa00640 1 -- 172.21.15.6:0/2721115313 learned_addr learned my addr 172.21.15.6:0/2721115313 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.892+0000 7f385b400640 1 --2- 172.21.15.6:0/2721115313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdf5b90 0x55780fe0dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.892+0000 7f385aa00640 1 -- 172.21.15.6:0/2721115313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 msgr2=0x55780fe0d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.892+0000 7f385aa00640 1 --2- 172.21.15.6:0/2721115313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 0x55780fe0d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.892+0000 7f385aa00640 1 -- 172.21.15.6:0/2721115313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdf5b90 msgr2=0x55780fe0dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.892+0000 7f385aa00640 1 --2- 172.21.15.6:0/2721115313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdf5b90 0x55780fe0dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.892+0000 7f385aa00640 1 -- 172.21.15.6:0/2721115313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55780fded820 con 0x55780fdea9b0 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.892+0000 7f385a000640 1 --2- 172.21.15.6:0/2721115313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 0x55780fe0d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.893+0000 7f385aa00640 1 --2- 172.21.15.6:0/2721115313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdea9b0 0x55780fe01680 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f384c009560 tx=0x7f384c005970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a32d4c1d590d8bf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.893+0000 7f3847400640 1 -- 172.21.15.6:0/2721115313 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c03c070 con 0x55780fdea9b0 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.893+0000 7f3847400640 1 -- 172.21.15.6:0/2721115313 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f384c015030 con 0x55780fdea9b0 2024-04-26T01:51:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.893+0000 7f3847400640 1 -- 172.21.15.6:0/2721115313 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c027070 con 0x55780fdea9b0 2024-04-26T01:51:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.894+0000 7f38647e1740 1 -- 172.21.15.6:0/2721115313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdea9b0 msgr2=0x55780fe01680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.894+0000 7f38647e1740 1 --2- 172.21.15.6:0/2721115313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdea9b0 0x55780fe01680 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f384c009560 tx=0x7f384c005970 comp rx=0 tx=0).stop 2024-04-26T01:51:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.894+0000 7f38647e1740 1 -- 172.21.15.6:0/2721115313 shutdown_connections 2024-04-26T01:51:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.894+0000 7f38647e1740 1 --2- 172.21.15.6:0/2721115313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdf5b90 0x55780fe0dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.894+0000 7f38647e1740 1 --2- 172.21.15.6:0/2721115313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdeb2d0 0x55780fe0d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.894+0000 7f38647e1740 1 --2- 172.21.15.6:0/2721115313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdea9b0 0x55780fe01680 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.894+0000 7f38647e1740 1 -- 172.21.15.6:0/2721115313 >> 172.21.15.6:0/2721115313 conn(0x55780fde6ec0 msgr2=0x55780fdf1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.894+0000 7f38647e1740 1 -- 172.21.15.6:0/2721115313 shutdown_connections 2024-04-26T01:51:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.894+0000 7f38647e1740 1 -- 172.21.15.6:0/2721115313 wait complete. 2024-04-26T01:51:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.894+0000 7f38647e1740 1 -- start start 2024-04-26T01:51:15.896 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.895+0000 7f38647e1740 1 Processor -- start 2024-04-26T01:51:15.896 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.895+0000 7f38647e1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdea9b0 0x55780fdfab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.896 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.895+0000 7f38647e1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdeb2d0 0x55780fdfb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.896 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.895+0000 7f38647e1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdf5b90 0x55780fdfb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.895+0000 7f38647e1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1ae390 con 0x55780fdeb2d0 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.895+0000 7f38647e1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1af800 con 0x55780fdf5b90 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.895+0000 7f38647e1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1aea60 con 0x55780fdea9b0 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.895+0000 7f38647e1740 10 client.0 _ll_register_callbacks cb 0x7f384c009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.895+0000 7f385a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdeb2d0 0x55780fdfb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.895+0000 7f385a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdeb2d0 0x55780fdfb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36834/0 (socket says 172.21.15.6:36834) 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.895+0000 7f385a000640 1 -- 172.21.15.6:0/1838012378 learned_addr learned my addr 172.21.15.6:0/1838012378 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f385aa00640 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdea9b0 0x55780fdfab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f385b400640 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdf5b90 0x55780fdfb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f385a000640 1 -- 172.21.15.6:0/1838012378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdea9b0 msgr2=0x55780fdfab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f385a000640 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdea9b0 0x55780fdfab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f385a000640 1 -- 172.21.15.6:0/1838012378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdf5b90 msgr2=0x55780fdfb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f385a000640 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdf5b90 0x55780fdfb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f385a000640 1 -- 172.21.15.6:0/1838012378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55780fe6a2f0 con 0x55780fdeb2d0 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f385b400640 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdf5b90 0x55780fdfb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f385a000640 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdeb2d0 0x55780fdfb0a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3848031440 tx=0x7f38480f4900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff1b8cf2d7365f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f3844c00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f3844c00640 1 -- 172.21.15.6:0/1838012378 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3848049040 con 0x55780fdeb2d0 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f3844c00640 1 -- 172.21.15.6:0/1838012378 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7f384803b460 con 0x55780fdeb2d0 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.896+0000 7f38647e1740 1 -- 172.21.15.6:0/1838012378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f384c015030 con 0x55780fdeb2d0 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.897+0000 7f3844c00640 1 -- 172.21.15.6:0/1838012378 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f38480f4750 con 0x55780fdeb2d0 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.897+0000 7f38647e1740 10 client.15672 fetch_fsmap learned FSMap version 35 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.897+0000 7f38647e1740 1 -- 172.21.15.6:0/1838012378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55780fe6b660 con 0x55780fdeb2d0 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.897+0000 7f3844c00640 1 -- 172.21.15.6:0/1838012378 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f384c015030 con 0x55780fdeb2d0 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.897+0000 7f3844c00640 1 -- 172.21.15.6:0/1838012378 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f38480b9490 con 0x55780fdeb2d0 2024-04-26T01:51:15.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.897+0000 7f38647e1740 10 client.15672 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T01:51:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.897+0000 7f38647e1740 10 client.15672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.897+0000 7f38647e1740 20 client.15672 populate_metadata read hostname 'smithi006' 2024-04-26T01:51:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.897+0000 7f38647e1740 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55780fe6be20 0x55780fe6e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.897+0000 7f38647e1740 4 client.15672 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.897+0000 7f38647e1740 1 -- 172.21.15.6:0/1838012378 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55780fd5e320 con 0x55780fe6be20 2024-04-26T01:51:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.898+0000 7f385aa00640 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55780fe6be20 0x55780fe6e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.898+0000 7f385aa00640 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55780fe6be20 0x55780fe6e240 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6b8397056a898c2 server_cookie=59f8d54ed71678b9 in_seq=0 out_seq=0 2024-04-26T01:51:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.898+0000 7f3844c00640 10 client.15672 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.937+0000 7f3844c00640 1 -- 172.21.15.6:0/1838012378 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55780fd5e320 con 0x55780fe6be20 2024-04-26T01:51:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.937+0000 7f3844c00640 10 client.15672 handle_command_reply: tid=0 2024-04-26T01:51:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.937+0000 7f38647e1740 1 client.15672 shutdown 2024-04-26T01:51:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.937+0000 7f38647e1740 1 -- 172.21.15.6:0/1838012378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdeb2d0 msgr2=0x55780fdfb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.937+0000 7f38647e1740 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdeb2d0 0x55780fdfb0a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3848031440 tx=0x7f38480f4900 comp rx=0 tx=0).stop 2024-04-26T01:51:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.937+0000 7f38647e1740 1 -- 172.21.15.6:0/1838012378 shutdown_connections 2024-04-26T01:51:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.937+0000 7f38647e1740 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55780fe6be20 0x55780fe6e240 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.937+0000 7f38647e1740 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55780fdeb2d0 0x55780fdfb0a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.937+0000 7f38647e1740 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55780fdf5b90 0x55780fdfb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.937+0000 7f38647e1740 1 --2- 172.21.15.6:0/1838012378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55780fdea9b0 0x55780fdfab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.937+0000 7f3844c00640 0 client.15672 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.937+0000 7f38647e1740 1 -- 172.21.15.6:0/1838012378 >> 172.21.15.6:0/1838012378 conn(0x55780fde6ec0 msgr2=0x55780fdec360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.938+0000 7f38647e1740 1 -- 172.21.15.6:0/1838012378 shutdown_connections 2024-04-26T01:51:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.938+0000 7f38647e1740 1 -- 172.21.15.6:0/1838012378 wait complete. 2024-04-26T01:51:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.938+0000 7f38647e1740 20 client.15672 trim_cache size 0 max 16384 2024-04-26T01:51:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:51:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:51:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:51:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:51:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.939+0000 7f385be00640 1 -- 172.21.15.6:0/1569454920 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38400a3d00 msgr2=0x7f38400a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.939+0000 7f385be00640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38400a3d00 0x7f38400a61c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3854005fd0 tx=0x7f3854009500 comp rx=0 tx=0).stop 2024-04-26T01:51:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.939+0000 7f385be00640 1 -- 172.21.15.6:0/1569454920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f385c1092a0 msgr2=0x7f385c1a0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.939+0000 7f385be00640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f385c1092a0 0x7f385c1a0930 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3848005ab0 tx=0x7f3848003b90 comp rx=0 tx=0).stop 2024-04-26T01:51:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.939+0000 7f385be00640 1 -- 172.21.15.6:0/1569454920 shutdown_connections 2024-04-26T01:51:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.939+0000 7f385be00640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38400a3d00 0x7f38400a61c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.939+0000 7f385be00640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f385c1092a0 0x7f385c1a0930 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.939+0000 7f385be00640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f385c19a5f0 0x7f385c19aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.939+0000 7f385be00640 1 --2- 172.21.15.6:0/1569454920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f385c1041d0 0x7f385c1a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.939+0000 7f385be00640 1 -- 172.21.15.6:0/1569454920 >> 172.21.15.6:0/1569454920 conn(0x7f385c0ffa30 msgr2=0x7f385c100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.939+0000 7f385be00640 1 -- 172.21.15.6:0/1569454920 shutdown_connections 2024-04-26T01:51:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:15.939+0000 7f385be00640 1 -- 172.21.15.6:0/1569454920 wait complete. 2024-04-26T01:51:16.578 INFO:tasks.workunit.client.0.smithi006.stdout: 1 167327 4.01 MB/sec execute 100 sec latency 8628.459 ms 2024-04-26T01:51:16.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:16 smithi188 ceph-mon[32751]: pgmap v166: 97 pgs: 97 active+clean; 60 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 164 op/s 2024-04-26T01:51:16.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:16 smithi029 ceph-mon[32775]: pgmap v166: 97 pgs: 97 active+clean; 60 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 164 op/s 2024-04-26T01:51:16.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:16 smithi006 ceph-mon[28419]: pgmap v166: 97 pgs: 97 active+clean; 60 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 164 op/s 2024-04-26T01:51:17.578 INFO:tasks.workunit.client.0.smithi006.stdout: 1 167327 3.97 MB/sec execute 101 sec latency 9628.593 ms 2024-04-26T01:51:18.578 INFO:tasks.workunit.client.0.smithi006.stdout: 1 167327 3.93 MB/sec execute 102 sec latency 10628.709 ms 2024-04-26T01:51:18.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:18 smithi188 ceph-mon[32751]: pgmap v167: 97 pgs: 97 active+clean; 62 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 197 op/s 2024-04-26T01:51:18.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:18 smithi029 ceph-mon[32775]: pgmap v167: 97 pgs: 97 active+clean; 62 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 197 op/s 2024-04-26T01:51:18.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:18 smithi006 ceph-mon[28419]: pgmap v167: 97 pgs: 97 active+clean; 62 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 197 op/s 2024-04-26T01:51:19.578 INFO:tasks.workunit.client.0.smithi006.stdout: 1 168117 3.94 MB/sec execute 103 sec latency 10997.449 ms 2024-04-26T01:51:20.579 INFO:tasks.workunit.client.0.smithi006.stdout: 1 169329 3.94 MB/sec execute 104 sec latency 37.008 ms 2024-04-26T01:51:20.619 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:20 smithi006 ceph-mon[28419]: pgmap v168: 97 pgs: 97 active+clean; 62 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.3 MiB/s wr, 82 op/s 2024-04-26T01:51:20.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:20 smithi188 ceph-mon[32751]: pgmap v168: 97 pgs: 97 active+clean; 62 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.3 MiB/s wr, 82 op/s 2024-04-26T01:51:20.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:20 smithi029 ceph-mon[32775]: pgmap v168: 97 pgs: 97 active+clean; 62 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.3 MiB/s wr, 82 op/s 2024-04-26T01:51:21.579 INFO:tasks.workunit.client.0.smithi006.stdout: 1 170553 4.00 MB/sec execute 105 sec latency 15.447 ms 2024-04-26T01:51:21.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:21 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:21.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:21 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:21.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:21 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:22.579 INFO:tasks.workunit.client.0.smithi006.stdout: 1 171251 3.97 MB/sec execute 106 sec latency 26.032 ms 2024-04-26T01:51:22.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:22 smithi188 ceph-mon[32751]: pgmap v169: 97 pgs: 97 active+clean; 63 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 1.8 MiB/s wr, 128 op/s 2024-04-26T01:51:22.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:22 smithi029 ceph-mon[32775]: pgmap v169: 97 pgs: 97 active+clean; 63 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 1.8 MiB/s wr, 128 op/s 2024-04-26T01:51:22.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:22 smithi006 ceph-mon[28419]: pgmap v169: 97 pgs: 97 active+clean; 63 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 1.8 MiB/s wr, 128 op/s 2024-04-26T01:51:23.579 INFO:tasks.workunit.client.0.smithi006.stdout: 1 172364 3.98 MB/sec execute 107 sec latency 73.381 ms 2024-04-26T01:51:24.579 INFO:tasks.workunit.client.0.smithi006.stdout: 1 173827 4.05 MB/sec execute 108 sec latency 19.076 ms 2024-04-26T01:51:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:24 smithi188 ceph-mon[32751]: pgmap v170: 97 pgs: 97 active+clean; 63 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-26T01:51:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:24 smithi029 ceph-mon[32775]: pgmap v170: 97 pgs: 97 active+clean; 63 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-26T01:51:24.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:24 smithi006 ceph-mon[28419]: pgmap v170: 97 pgs: 97 active+clean; 63 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-26T01:51:25.579 INFO:tasks.workunit.client.0.smithi006.stdout: 1 174414 4.03 MB/sec execute 109 sec latency 18.667 ms 2024-04-26T01:51:26.579 INFO:tasks.workunit.client.0.smithi006.stdout: 1 175462 4.04 MB/sec execute 110 sec latency 23.184 ms 2024-04-26T01:51:26.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:26 smithi188 ceph-mon[32751]: pgmap v171: 97 pgs: 97 active+clean; 63 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-26T01:51:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:26 smithi029 ceph-mon[32775]: pgmap v171: 97 pgs: 97 active+clean; 63 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-26T01:51:26.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:26 smithi006 ceph-mon[28419]: pgmap v171: 97 pgs: 97 active+clean; 63 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-26T01:51:27.580 INFO:tasks.workunit.client.0.smithi006.stdout: 1 176842 4.07 MB/sec execute 111 sec latency 22.691 ms 2024-04-26T01:51:28.580 INFO:tasks.workunit.client.0.smithi006.stdout: 1 177774 4.09 MB/sec execute 112 sec latency 43.982 ms 2024-04-26T01:51:28.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:28 smithi188 ceph-mon[32751]: pgmap v172: 97 pgs: 97 active+clean; 67 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 156 op/s 2024-04-26T01:51:28.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:28 smithi029 ceph-mon[32775]: pgmap v172: 97 pgs: 97 active+clean; 67 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 156 op/s 2024-04-26T01:51:28.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:28 smithi006 ceph-mon[28419]: pgmap v172: 97 pgs: 97 active+clean; 67 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 156 op/s 2024-04-26T01:51:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.098+0000 7f172da00640 1 -- 172.21.15.6:0/2670088981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17281041d0 msgr2=0x7f17281045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.098+0000 7f172da00640 1 --2- 172.21.15.6:0/2670088981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17281041d0 0x7f17281045b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1718009ca0 tx=0x7f171802a8c0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.099+0000 7f172da00640 1 -- 172.21.15.6:0/2670088981 shutdown_connections 2024-04-26T01:51:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.099+0000 7f172da00640 1 --2- 172.21.15.6:0/2670088981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17281092a0 0x7f1728110e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.099+0000 7f172da00640 1 --2- 172.21.15.6:0/2670088981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1728104b80 0x7f1728108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.099+0000 7f172da00640 1 --2- 172.21.15.6:0/2670088981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17281041d0 0x7f17281045b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.099+0000 7f172da00640 1 -- 172.21.15.6:0/2670088981 >> 172.21.15.6:0/2670088981 conn(0x7f17280ffa50 msgr2=0x7f1728101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.099+0000 7f172da00640 1 -- 172.21.15.6:0/2670088981 shutdown_connections 2024-04-26T01:51:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.099+0000 7f172da00640 1 -- 172.21.15.6:0/2670088981 wait complete. 2024-04-26T01:51:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.100+0000 7f172da00640 1 Processor -- start 2024-04-26T01:51:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.100+0000 7f172da00640 1 -- start start 2024-04-26T01:51:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172da00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17281041d0 0x7f172819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17281092a0 0x7f172819acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172da00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f172819f020 0x7f172819f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1728114230 con 0x7f17281092a0 2024-04-26T01:51:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172da00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17281140b0 con 0x7f172819f020 2024-04-26T01:51:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172da00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f17281143b0 con 0x7f17281041d0 2024-04-26T01:51:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f1727e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17281041d0 0x7f172819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172d000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f172819f020 0x7f172819f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f1727e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17281041d0 0x7f172819a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56262/0 (socket says 172.21.15.6:56262) 2024-04-26T01:51:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172d000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f172819f020 0x7f172819f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41486/0 (socket says 172.21.15.6:41486) 2024-04-26T01:51:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f1727e00640 1 -- 172.21.15.6:0/4078559305 learned_addr learned my addr 172.21.15.6:0/4078559305 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f1727400640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17281092a0 0x7f172819acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172d000640 1 -- 172.21.15.6:0/4078559305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17281041d0 msgr2=0x7f172819a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:29.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172d000640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17281041d0 0x7f172819a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172d000640 1 -- 172.21.15.6:0/4078559305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17281092a0 msgr2=0x7f172819acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172d000640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17281092a0 0x7f172819acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.101+0000 7f172d000640 1 -- 172.21.15.6:0/4078559305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f172819fb10 con 0x7f172819f020 2024-04-26T01:51:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.102+0000 7f172d000640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f172819f020 0x7f172819f4d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f172000e970 tx=0x7f172000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.102+0000 7f1724c00640 1 -- 172.21.15.6:0/4078559305 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172000ccc0 con 0x7f172819f020 2024-04-26T01:51:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.102+0000 7f172da00640 1 -- 172.21.15.6:0/4078559305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f172819fe00 con 0x7f172819f020 2024-04-26T01:51:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.102+0000 7f1724c00640 1 -- 172.21.15.6:0/4078559305 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1720005020 con 0x7f172819f020 2024-04-26T01:51:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.102+0000 7f1724c00640 1 -- 172.21.15.6:0/4078559305 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720004a00 con 0x7f172819f020 2024-04-26T01:51:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.103+0000 7f172da00640 1 -- 172.21.15.6:0/4078559305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17281a75f0 con 0x7f172819f020 2024-04-26T01:51:29.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.106+0000 7f1731fb5740 1 Processor -- start 2024-04-26T01:51:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.107+0000 7f1724c00640 1 -- 172.21.15.6:0/4078559305 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f1720004ba0 con 0x7f172819f020 2024-04-26T01:51:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.107+0000 7f1731fb5740 1 -- start start 2024-04-26T01:51:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.107+0000 7f1731fb5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 0x563caa351d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.107+0000 7f1731fb5740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3522d0 0x563caa35c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.107+0000 7f1731fb5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa35cb90 0x563caa35ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.107+0000 7f1731fb5740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x563caa2b7900 con 0x563caa3522d0 2024-04-26T01:51:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.107+0000 7f1731fb5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563caa25def0 con 0x563caa35cb90 2024-04-26T01:51:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.107+0000 7f1731fb5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563caa2b38c0 con 0x563caa3519b0 2024-04-26T01:51:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.108+0000 7f1727400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 0x563caa351d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.108+0000 7f1727400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 0x563caa351d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56274/0 (socket says 172.21.15.6:56274) 2024-04-26T01:51:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.108+0000 7f1727400640 1 -- 172.21.15.6:0/560710792 learned_addr learned my addr 172.21.15.6:0/560710792 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:29.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.108+0000 7f1727e00640 1 --2- 172.21.15.6:0/560710792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3522d0 0x563caa35c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.108+0000 7f1727400640 1 --2- 172.21.15.6:0/560710792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa35cb90 0x563caa35ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.108+0000 7f1727400640 1 -- 172.21.15.6:0/560710792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa35cb90 msgr2=0x563caa35ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.108+0000 7f1727400640 1 --2- 172.21.15.6:0/560710792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa35cb90 0x563caa35ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.108+0000 7f1727400640 1 -- 172.21.15.6:0/560710792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3522d0 msgr2=0x563caa35c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.108+0000 7f1727400640 1 --2- 172.21.15.6:0/560710792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3522d0 0x563caa35c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.108+0000 7f1727400640 1 -- 172.21.15.6:0/560710792 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ca9fca210 con 0x563caa3519b0 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.108+0000 7f1724c00640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f170c0a3d80 0x7f170c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.108+0000 7f1727400640 1 --2- 172.21.15.6:0/560710792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 0x563caa351d90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x563caa34f050 tx=0x7f1710004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d59100b2ef5fc96 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.109+0000 7f1724c00640 1 -- 172.21.15.6:0/4078559305 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7f17200d0700 con 0x7f172819f020 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.109+0000 7f1727e00640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f170c0a3d80 0x7f170c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.109+0000 7f171d600640 1 -- 172.21.15.6:0/560710792 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1710012ca0 con 0x563caa3519b0 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.109+0000 7f171d600640 1 -- 172.21.15.6:0/560710792 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1710012e40 con 0x563caa3519b0 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.109+0000 7f171d600640 1 -- 172.21.15.6:0/560710792 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17100049a0 con 0x563caa3519b0 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.110+0000 7f1731fb5740 1 -- 172.21.15.6:0/560710792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 msgr2=0x563caa351d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.110+0000 7f1731fb5740 1 --2- 172.21.15.6:0/560710792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 0x563caa351d90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x563caa34f050 tx=0x7f1710004730 comp rx=0 tx=0).stop 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.110+0000 7f1727e00640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f170c0a3d80 0x7f170c0a6240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x563caa352f50 tx=0x7f1718005d80 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.110+0000 7f1731fb5740 1 -- 172.21.15.6:0/560710792 shutdown_connections 2024-04-26T01:51:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.110+0000 7f1731fb5740 1 --2- 172.21.15.6:0/560710792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3522d0 0x563caa35c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.110+0000 7f1731fb5740 1 --2- 172.21.15.6:0/560710792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa35cb90 0x563caa35ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.110+0000 7f1731fb5740 1 --2- 172.21.15.6:0/560710792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 0x563caa351d90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.110+0000 7f1731fb5740 1 -- 172.21.15.6:0/560710792 >> 172.21.15.6:0/560710792 conn(0x563caa34dec0 msgr2=0x563caa34e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.110+0000 7f1731fb5740 1 -- 172.21.15.6:0/560710792 shutdown_connections 2024-04-26T01:51:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.110+0000 7f1731fb5740 1 -- 172.21.15.6:0/560710792 wait complete. 2024-04-26T01:51:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.110+0000 7f1731fb5740 1 -- start start 2024-04-26T01:51:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.110+0000 7f1731fb5740 1 Processor -- start 2024-04-26T01:51:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.111+0000 7f1731fb5740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3519b0 0x563caa359b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.111+0000 7f1731fb5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3522d0 0x563caa35a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.111+0000 7f1731fb5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa35cb90 0x563caa35a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.111+0000 7f1731fb5740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x563caa35f7c0 con 0x563caa3519b0 2024-04-26T01:51:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.111+0000 7f1731fb5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563caa271310 con 0x563caa3522d0 2024-04-26T01:51:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.111+0000 7f1731fb5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563caa35f940 con 0x563caa35cb90 2024-04-26T01:51:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.111+0000 7f1731fb5740 10 client.0 _ll_register_callbacks cb 0x563caa23fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.111+0000 7f1727e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3519b0 0x563caa359b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.111+0000 7f1727400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3522d0 0x563caa35a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.112+0000 7f1727400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3522d0 0x563caa35a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41506/0 (socket says 172.21.15.6:41506) 2024-04-26T01:51:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.112+0000 7f1727400640 1 -- 172.21.15.6:0/196149102 learned_addr learned my addr 172.21.15.6:0/196149102 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.112+0000 7f172d000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa35cb90 0x563caa35a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.112+0000 7f1727400640 1 -- 172.21.15.6:0/196149102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa35cb90 msgr2=0x563caa35a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.112+0000 7f1727400640 1 --2- 172.21.15.6:0/196149102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa35cb90 0x563caa35a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.112+0000 7f1727400640 1 -- 172.21.15.6:0/196149102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3519b0 msgr2=0x563caa359b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.112+0000 7f1727400640 1 --2- 172.21.15.6:0/196149102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3519b0 0x563caa359b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.112+0000 7f1727400640 1 -- 172.21.15.6:0/196149102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563caa096620 con 0x563caa3522d0 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.112+0000 7f1727400640 1 --2- 172.21.15.6:0/196149102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3522d0 0x563caa35a070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1710005630 tx=0x7f1710005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7782601ec2e168cb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.112+0000 7f1716000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1716000640 1 -- 172.21.15.6:0/196149102 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1710012ca0 con 0x563caa3522d0 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1731fb5740 1 -- 172.21.15.6:0/196149102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563caa2cb8b0 con 0x563caa3522d0 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1716000640 1 -- 172.21.15.6:0/196149102 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7f1710010970 con 0x563caa3522d0 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1716000640 1 -- 172.21.15.6:0/196149102 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1710010b70 con 0x563caa3522d0 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1731fb5740 10 client.25808 fetch_fsmap learned FSMap version 36 2024-04-26T01:51:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1731fb5740 1 -- 172.21.15.6:0/196149102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563caa374250 con 0x563caa3522d0 2024-04-26T01:51:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1716000640 1 -- 172.21.15.6:0/196149102 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563caa2cb8b0 con 0x563caa3522d0 2024-04-26T01:51:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1716000640 1 -- 172.21.15.6:0/196149102 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f1710026070 con 0x563caa3522d0 2024-04-26T01:51:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1731fb5740 10 client.25808 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T01:51:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1731fb5740 10 client.25808 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1731fb5740 20 client.25808 populate_metadata read hostname 'smithi006' 2024-04-26T01:51:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1731fb5740 1 --2- 172.21.15.6:0/196149102 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563caa3755a0 0x563caa377990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1731fb5740 4 client.25808 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.113+0000 7f1731fb5740 1 -- 172.21.15.6:0/196149102 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563caa2c5320 con 0x563caa3755a0 2024-04-26T01:51:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.114+0000 7f1727e00640 1 --2- 172.21.15.6:0/196149102 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563caa3755a0 0x563caa377990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.114+0000 7f1727e00640 1 --2- 172.21.15.6:0/196149102 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563caa3755a0 0x563caa377990 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13f871ea681c436c server_cookie=1421f8e5342fb925 in_seq=0 out_seq=0 2024-04-26T01:51:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.114+0000 7f1716000640 10 client.25808 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.115+0000 7f1716000640 1 -- 172.21.15.6:0/196149102 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563caa2c5320 con 0x563caa3755a0 2024-04-26T01:51:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.115+0000 7f1716000640 10 client.25808 handle_command_reply: tid=0 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.115+0000 7f1731fb5740 1 client.25808 shutdown 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.115+0000 7f1731fb5740 1 -- 172.21.15.6:0/196149102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3522d0 msgr2=0x563caa35a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.115+0000 7f1731fb5740 1 --2- 172.21.15.6:0/196149102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3522d0 0x563caa35a070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1710005630 tx=0x7f1710005780 comp rx=0 tx=0).stop 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.115+0000 7f1731fb5740 1 -- 172.21.15.6:0/196149102 shutdown_connections 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.115+0000 7f1731fb5740 1 --2- 172.21.15.6:0/196149102 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563caa3755a0 0x563caa377990 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.115+0000 7f1731fb5740 1 --2- 172.21.15.6:0/196149102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa35cb90 0x563caa35a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.115+0000 7f1731fb5740 1 --2- 172.21.15.6:0/196149102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3522d0 0x563caa35a070 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.115+0000 7f1731fb5740 1 --2- 172.21.15.6:0/196149102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3519b0 0x563caa359b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.115+0000 7f1716000640 0 client.25808 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.115+0000 7f1731fb5740 1 -- 172.21.15.6:0/196149102 >> 172.21.15.6:0/196149102 conn(0x563caa34dec0 msgr2=0x563caa353c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.116+0000 7f1731fb5740 1 -- 172.21.15.6:0/196149102 shutdown_connections 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.116+0000 7f1731fb5740 1 -- 172.21.15.6:0/196149102 wait complete. 2024-04-26T01:51:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.116+0000 7f1731fb5740 20 client.25808 trim_cache size 0 max 16384 2024-04-26T01:51:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.134+0000 7f1731fb5740 1 Processor -- start 2024-04-26T01:51:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.134+0000 7f1731fb5740 1 -- start start 2024-04-26T01:51:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.134+0000 7f1731fb5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 0x563caa370ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.134+0000 7f1731fb5740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3522d0 0x563caa354370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.134+0000 7f1731fb5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa35cb90 0x563caa37d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1731fb5740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f17281ae100 con 0x563caa3522d0 2024-04-26T01:51:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1731fb5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17281ab9d0 con 0x563caa35cb90 2024-04-26T01:51:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1731fb5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f17281a9840 con 0x563caa3519b0 2024-04-26T01:51:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1727e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 0x563caa370ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1727400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3522d0 0x563caa354370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1727400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3522d0 0x563caa354370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40002/0 (socket says 172.21.15.6:40002) 2024-04-26T01:51:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1727400640 1 -- 172.21.15.6:0/3419024294 learned_addr learned my addr 172.21.15.6:0/3419024294 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1727e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 0x563caa370ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56294/0 (socket says 172.21.15.6:56294) 2024-04-26T01:51:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f172d000640 1 --2- 172.21.15.6:0/3419024294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa35cb90 0x563caa37d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1727400640 1 -- 172.21.15.6:0/3419024294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 msgr2=0x563caa370ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1727400640 1 --2- 172.21.15.6:0/3419024294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 0x563caa370ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1727400640 1 -- 172.21.15.6:0/3419024294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa35cb90 msgr2=0x563caa37d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1727400640 1 --2- 172.21.15.6:0/3419024294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa35cb90 0x563caa37d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.135+0000 7f1727400640 1 -- 172.21.15.6:0/3419024294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563caa3548b0 con 0x563caa3522d0 2024-04-26T01:51:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.136+0000 7f1727400640 1 --2- 172.21.15.6:0/3419024294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3522d0 0x563caa354370 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f171000bbf0 tx=0x7f171000ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ab8c8d8a86ca33 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.136+0000 7f171cc00640 1 -- 172.21.15.6:0/3419024294 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171000b410 con 0x563caa3522d0 2024-04-26T01:51:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.136+0000 7f171cc00640 1 -- 172.21.15.6:0/3419024294 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1710016040 con 0x563caa3522d0 2024-04-26T01:51:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.136+0000 7f171cc00640 1 -- 172.21.15.6:0/3419024294 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1710010740 con 0x563caa3522d0 2024-04-26T01:51:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.137+0000 7f1731fb5740 1 -- 172.21.15.6:0/3419024294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3522d0 msgr2=0x563caa354370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.137+0000 7f1731fb5740 1 --2- 172.21.15.6:0/3419024294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3522d0 0x563caa354370 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f171000bbf0 tx=0x7f171000ba60 comp rx=0 tx=0).stop 2024-04-26T01:51:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.137+0000 7f1731fb5740 1 -- 172.21.15.6:0/3419024294 shutdown_connections 2024-04-26T01:51:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.137+0000 7f1731fb5740 1 --2- 172.21.15.6:0/3419024294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa3522d0 0x563caa354370 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.137+0000 7f1731fb5740 1 --2- 172.21.15.6:0/3419024294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa35cb90 0x563caa37d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.137+0000 7f1731fb5740 1 --2- 172.21.15.6:0/3419024294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3519b0 0x563caa370ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.137+0000 7f1731fb5740 1 -- 172.21.15.6:0/3419024294 >> 172.21.15.6:0/3419024294 conn(0x563caa34dec0 msgr2=0x563caa34d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.137+0000 7f1731fb5740 1 -- 172.21.15.6:0/3419024294 shutdown_connections 2024-04-26T01:51:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.137+0000 7f1731fb5740 1 -- 172.21.15.6:0/3419024294 wait complete. 2024-04-26T01:51:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.137+0000 7f1731fb5740 1 -- start start 2024-04-26T01:51:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.138+0000 7f1731fb5740 1 Processor -- start 2024-04-26T01:51:29.139 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.138+0000 7f1731fb5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3519b0 0x563caa36d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.139 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.138+0000 7f1731fb5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3522d0 0x563caa36aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.139 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.138+0000 7f1731fb5740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa35cb90 0x563caa36b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.139 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.138+0000 7f1731fb5740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f17281a9e90 con 0x563caa35cb90 2024-04-26T01:51:29.139 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.138+0000 7f1731fb5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17281ab300 con 0x563caa3519b0 2024-04-26T01:51:29.139 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.138+0000 7f1731fb5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f17281aa560 con 0x563caa3522d0 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.138+0000 7f1731fb5740 10 client.0 _ll_register_callbacks cb 0x7f1710012650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.138+0000 7f1727e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3519b0 0x563caa36d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1727400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3522d0 0x563caa36aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1727e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3519b0 0x563caa36d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41526/0 (socket says 172.21.15.6:41526) 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1727e00640 1 -- 172.21.15.6:0/1989506542 learned_addr learned my addr 172.21.15.6:0/1989506542 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f172d000640 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa35cb90 0x563caa36b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1727e00640 1 -- 172.21.15.6:0/1989506542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3522d0 msgr2=0x563caa36aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1727e00640 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3522d0 0x563caa36aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1727e00640 1 -- 172.21.15.6:0/1989506542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa35cb90 msgr2=0x563caa36b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1727e00640 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa35cb90 0x563caa36b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1727e00640 1 -- 172.21.15.6:0/1989506542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563caa36b7e0 con 0x563caa3519b0 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f172d000640 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa35cb90 0x563caa36b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1727400640 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3522d0 0x563caa36aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1727e00640 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3519b0 0x563caa36d740 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1718005e30 tx=0x7f1718031d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec01cd9f24c1912e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1716000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1716000640 1 -- 172.21.15.6:0/1989506542 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1718058070 con 0x563caa3519b0 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.139+0000 7f1716000640 1 -- 172.21.15.6:0/1989506542 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7f17180315f0 con 0x563caa3519b0 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.140+0000 7f1731fb5740 1 -- 172.21.15.6:0/1989506542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1710016040 con 0x563caa3519b0 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.140+0000 7f1716000640 1 -- 172.21.15.6:0/1989506542 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1718061050 con 0x563caa3519b0 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.140+0000 7f1731fb5740 10 client.25814 fetch_fsmap learned FSMap version 36 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.140+0000 7f1731fb5740 1 -- 172.21.15.6:0/1989506542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563caa3db9b0 con 0x563caa3519b0 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.140+0000 7f1716000640 1 -- 172.21.15.6:0/1989506542 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1710016040 con 0x563caa3519b0 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.140+0000 7f1716000640 1 -- 172.21.15.6:0/1989506542 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f171805b020 con 0x563caa3519b0 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.140+0000 7f1731fb5740 10 client.25814 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.140+0000 7f1731fb5740 10 client.25814 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.140+0000 7f1731fb5740 20 client.25814 populate_metadata read hostname 'smithi006' 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.140+0000 7f1731fb5740 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563caa3dc130 0x563caa3de550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.140+0000 7f1731fb5740 4 client.25814 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.140+0000 7f1731fb5740 1 -- 172.21.15.6:0/1989506542 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563caa2bf190 con 0x563caa3dc130 2024-04-26T01:51:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.141+0000 7f1727400640 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563caa3dc130 0x563caa3de550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.141+0000 7f1727400640 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563caa3dc130 0x563caa3de550 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=667134838a9a29d6 server_cookie=5fa6e1610b937366 in_seq=0 out_seq=0 2024-04-26T01:51:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.141+0000 7f1716000640 10 client.25814 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1716000640 1 -- 172.21.15.6:0/1989506542 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563caa2bf190 con 0x563caa3dc130 2024-04-26T01:51:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1716000640 10 client.25814 handle_command_reply: tid=0 2024-04-26T01:51:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1731fb5740 1 client.25814 shutdown 2024-04-26T01:51:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1731fb5740 1 -- 172.21.15.6:0/1989506542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3519b0 msgr2=0x563caa36d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1731fb5740 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3519b0 0x563caa36d740 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1718005e30 tx=0x7f1718031d90 comp rx=0 tx=0).stop 2024-04-26T01:51:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1731fb5740 1 -- 172.21.15.6:0/1989506542 shutdown_connections 2024-04-26T01:51:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1731fb5740 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563caa3dc130 0x563caa3de550 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1731fb5740 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563caa35cb90 0x563caa36b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1716000640 0 client.25814 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1731fb5740 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563caa3522d0 0x563caa36aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1731fb5740 1 --2- 172.21.15.6:0/1989506542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563caa3519b0 0x563caa36d740 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1731fb5740 1 -- 172.21.15.6:0/1989506542 >> 172.21.15.6:0/1989506542 conn(0x563caa34dec0 msgr2=0x563caa359ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1731fb5740 1 -- 172.21.15.6:0/1989506542 shutdown_connections 2024-04-26T01:51:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.181+0000 7f1731fb5740 1 -- 172.21.15.6:0/1989506542 wait complete. 2024-04-26T01:51:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.182+0000 7f1731fb5740 20 client.25814 trim_cache size 0 max 16384 2024-04-26T01:51:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:51:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:51:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:51:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:51:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.182+0000 7f172da00640 1 -- 172.21.15.6:0/4078559305 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f170c0a3d80 msgr2=0x7f170c0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.182+0000 7f172da00640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f170c0a3d80 0x7f170c0a6240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x563caa352f50 tx=0x7f1718005d80 comp rx=0 tx=0).stop 2024-04-26T01:51:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.182+0000 7f172da00640 1 -- 172.21.15.6:0/4078559305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f172819f020 msgr2=0x7f172819f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.183+0000 7f172da00640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f172819f020 0x7f172819f4d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f172000e970 tx=0x7f172000ee30 comp rx=0 tx=0).stop 2024-04-26T01:51:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.183+0000 7f172da00640 1 -- 172.21.15.6:0/4078559305 shutdown_connections 2024-04-26T01:51:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.183+0000 7f172da00640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f170c0a3d80 0x7f170c0a6240 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.183+0000 7f172da00640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17281092a0 0x7f172819acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.183+0000 7f172da00640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f172819f020 0x7f172819f4d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.183+0000 7f172da00640 1 --2- 172.21.15.6:0/4078559305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17281041d0 0x7f172819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.183+0000 7f172da00640 1 -- 172.21.15.6:0/4078559305 >> 172.21.15.6:0/4078559305 conn(0x7f17280ffa50 msgr2=0x7f1728100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.183+0000 7f172da00640 1 -- 172.21.15.6:0/4078559305 shutdown_connections 2024-04-26T01:51:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:29.183+0000 7f172da00640 1 -- 172.21.15.6:0/4078559305 wait complete. 2024-04-26T01:51:29.580 INFO:tasks.workunit.client.0.smithi006.stdout: 1 178768 4.10 MB/sec execute 113 sec latency 32.897 ms 2024-04-26T01:51:29.890 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:51:29 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: ignoring --setuser ceph since I am not root 2024-04-26T01:51:29.890 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:51:29 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: ignoring --setgroup ceph since I am not root 2024-04-26T01:51:29.890 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:51:29 smithi188 ceph-mds[68153]: -- 172.21.15.188:0/3255506884 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ffc41c1a40 con 0x55ffc4195000 2024-04-26T01:51:29.890 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 01:51:29 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: starting mds.f at 2024-04-26T01:51:30.580 INFO:tasks.workunit.client.0.smithi006.stdout: 1 179972 4.09 MB/sec execute 114 sec latency 27.901 ms 2024-04-26T01:51:30.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:30 smithi188 ceph-mon[32751]: pgmap v173: 97 pgs: 97 active+clean; 67 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 120 op/s 2024-04-26T01:51:30.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:30 smithi188 ceph-mon[32751]: daemon mds.f restarted 2024-04-26T01:51:30.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:30 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:51:30.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:30 smithi188 ceph-mon[32751]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T01:51:30.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:30 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6834/1414952991,v1:172.21.15.188:6835/1414952991] up:boot 2024-04-26T01:51:30.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:30 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:30.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:30 smithi029 ceph-mon[32775]: pgmap v173: 97 pgs: 97 active+clean; 67 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 120 op/s 2024-04-26T01:51:30.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:30 smithi029 ceph-mon[32775]: daemon mds.f restarted 2024-04-26T01:51:30.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:30 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:51:30.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:30 smithi029 ceph-mon[32775]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T01:51:30.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:30 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6834/1414952991,v1:172.21.15.188:6835/1414952991] up:boot 2024-04-26T01:51:30.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:30 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:30.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:30 smithi006 ceph-mon[28419]: pgmap v173: 97 pgs: 97 active+clean; 67 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 120 op/s 2024-04-26T01:51:30.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:30 smithi006 ceph-mon[28419]: daemon mds.f restarted 2024-04-26T01:51:30.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:30 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:51:30.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:30 smithi006 ceph-mon[28419]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T01:51:30.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:30 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6834/1414952991,v1:172.21.15.188:6835/1414952991] up:boot 2024-04-26T01:51:30.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:30 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:31.581 INFO:tasks.workunit.client.0.smithi006.stdout: 1 181202 4.15 MB/sec execute 115 sec latency 15.738 ms 2024-04-26T01:51:32.580 INFO:tasks.workunit.client.0.smithi006.stdout: 1 181839 4.13 MB/sec execute 116 sec latency 32.301 ms 2024-04-26T01:51:32.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:32 smithi188 ceph-mon[32751]: pgmap v175: 97 pgs: 97 active+clean; 63 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-26T01:51:32.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:32 smithi029 ceph-mon[32775]: pgmap v175: 97 pgs: 97 active+clean; 63 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-26T01:51:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:32 smithi006 ceph-mon[28419]: pgmap v175: 97 pgs: 97 active+clean; 63 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-26T01:51:33.580 INFO:tasks.workunit.client.0.smithi006.stdout: 1 182775 4.13 MB/sec execute 117 sec latency 49.833 ms 2024-04-26T01:51:34.580 INFO:tasks.workunit.client.0.smithi006.stdout: 1 184305 4.17 MB/sec execute 118 sec latency 28.184 ms 2024-04-26T01:51:34.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:34 smithi188 ceph-mon[32751]: pgmap v176: 97 pgs: 97 active+clean; 63 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-26T01:51:34.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:34 smithi029 ceph-mon[32775]: pgmap v176: 97 pgs: 97 active+clean; 63 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-26T01:51:34.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:34 smithi006 ceph-mon[28419]: pgmap v176: 97 pgs: 97 active+clean; 63 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-26T01:51:35.580 INFO:tasks.workunit.client.0.smithi006.stdout: 1 184960 4.18 MB/sec execute 119 sec latency 16.474 ms 2024-04-26T01:51:36.581 INFO:tasks.workunit.client.0.smithi006.stdout: 1 185975 4.18 MB/sec execute 120 sec latency 24.169 ms 2024-04-26T01:51:36.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:36 smithi029 ceph-mon[32775]: pgmap v177: 97 pgs: 97 active+clean; 63 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-26T01:51:36.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:36 smithi006 ceph-mon[28419]: pgmap v177: 97 pgs: 97 active+clean; 63 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-26T01:51:36.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:36 smithi188 ceph-mon[32751]: pgmap v177: 97 pgs: 97 active+clean; 63 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-26T01:51:37.581 INFO:tasks.workunit.client.0.smithi006.stdout: 1 187354 4.20 MB/sec execute 121 sec latency 25.615 ms 2024-04-26T01:51:38.581 INFO:tasks.workunit.client.0.smithi006.stdout: 1 188325 4.23 MB/sec execute 122 sec latency 36.174 ms 2024-04-26T01:51:38.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:38 smithi029 ceph-mon[32775]: pgmap v178: 97 pgs: 97 active+clean; 66 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-04-26T01:51:38.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:38 smithi006 ceph-mon[28419]: pgmap v178: 97 pgs: 97 active+clean; 66 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-04-26T01:51:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:38 smithi188 ceph-mon[32751]: pgmap v178: 97 pgs: 97 active+clean; 66 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-04-26T01:51:39.581 INFO:tasks.workunit.client.0.smithi006.stdout: 1 189078 4.21 MB/sec execute 123 sec latency 20.475 ms 2024-04-26T01:51:40.581 INFO:tasks.workunit.client.0.smithi006.stdout: 1 190366 4.22 MB/sec execute 124 sec latency 26.217 ms 2024-04-26T01:51:40.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:40 smithi029 ceph-mon[32775]: pgmap v179: 97 pgs: 97 active+clean; 66 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-04-26T01:51:40.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:40 smithi006 ceph-mon[28419]: pgmap v179: 97 pgs: 97 active+clean; 66 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-04-26T01:51:40.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:40 smithi188 ceph-mon[32751]: pgmap v179: 97 pgs: 97 active+clean; 66 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-04-26T01:51:41.582 INFO:tasks.workunit.client.0.smithi006.stdout: 1 191646 4.27 MB/sec execute 125 sec latency 22.254 ms 2024-04-26T01:51:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.335+0000 7feee0e00640 1 -- 172.21.15.6:0/533672380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feedc102b50 msgr2=0x7feedc106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.335+0000 7feee0e00640 1 --2- 172.21.15.6:0/533672380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feedc102b50 0x7feedc106b90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feec8009860 tx=0x7feec802a340 comp rx=0 tx=0).stop 2024-04-26T01:51:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.336+0000 7feee0e00640 1 -- 172.21.15.6:0/533672380 shutdown_connections 2024-04-26T01:51:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.336+0000 7feee0e00640 1 --2- 172.21.15.6:0/533672380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feedc107270 0x7feedc10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.336+0000 7feee0e00640 1 --2- 172.21.15.6:0/533672380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feedc102b50 0x7feedc106b90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.336+0000 7feee0e00640 1 --2- 172.21.15.6:0/533672380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feedc1021a0 0x7feedc102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.336+0000 7feee0e00640 1 -- 172.21.15.6:0/533672380 >> 172.21.15.6:0/533672380 conn(0x7feedc0fd9e0 msgr2=0x7feedc0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.336+0000 7feee0e00640 1 -- 172.21.15.6:0/533672380 shutdown_connections 2024-04-26T01:51:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.336+0000 7feee0e00640 1 -- 172.21.15.6:0/533672380 wait complete. 2024-04-26T01:51:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.337+0000 7feee0e00640 1 Processor -- start 2024-04-26T01:51:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.338+0000 7feee0e00640 1 -- start start 2024-04-26T01:51:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.338+0000 7feee0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feedc1021a0 0x7feedc19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.338+0000 7feee0e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feedc107270 0x7feedc19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.338+0000 7feee0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feedc1985b0 0x7feedc198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.338+0000 7feee0e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1121b0 con 0x7feedc1021a0 2024-04-26T01:51:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.338+0000 7feee0e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feedc112030 con 0x7feedc107270 2024-04-26T01:51:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.338+0000 7feee0e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7feedc112330 con 0x7feedc1985b0 2024-04-26T01:51:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.338+0000 7feed9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feedc1021a0 0x7feedc19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.338+0000 7feed9400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feedc107270 0x7feedc19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.338+0000 7feed9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feedc1021a0 0x7feedc19e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59350/0 (socket says 172.21.15.6:59350) 2024-04-26T01:51:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.338+0000 7feed9e00640 1 -- 172.21.15.6:0/124587760 learned_addr learned my addr 172.21.15.6:0/124587760 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.339+0000 7feeda800640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feedc1985b0 0x7feedc198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.339+0000 7feed9400640 1 -- 172.21.15.6:0/124587760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feedc1985b0 msgr2=0x7feedc198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.339+0000 7feed9400640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feedc1985b0 0x7feedc198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.339+0000 7feed9400640 1 -- 172.21.15.6:0/124587760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feedc1021a0 msgr2=0x7feedc19e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:42.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.339+0000 7feed9400640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feedc1021a0 0x7feedc19e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.339+0000 7feed9400640 1 -- 172.21.15.6:0/124587760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feedc1990d0 con 0x7feedc107270 2024-04-26T01:51:42.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.339+0000 7feeda800640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feedc1985b0 0x7feedc198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:42.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.339+0000 7feed9400640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feedc107270 0x7feedc19e940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feec802a860 tx=0x7feec8032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:42.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.340+0000 7feed2a00640 1 -- 172.21.15.6:0/124587760 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec8037040 con 0x7feedc107270 2024-04-26T01:51:42.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.340+0000 7feed2a00640 1 -- 172.21.15.6:0/124587760 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feec8030c10 con 0x7feedc107270 2024-04-26T01:51:42.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.340+0000 7feee0e00640 1 -- 172.21.15.6:0/124587760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feedc199360 con 0x7feedc107270 2024-04-26T01:51:42.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.340+0000 7feed2a00640 1 -- 172.21.15.6:0/124587760 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec80035f0 con 0x7feedc107270 2024-04-26T01:51:42.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.340+0000 7feee0e00640 1 -- 172.21.15.6:0/124587760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feedc1a54f0 con 0x7feedc107270 2024-04-26T01:51:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.343+0000 7feed2a00640 1 -- 172.21.15.6:0/124587760 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7feec8043020 con 0x7feedc107270 2024-04-26T01:51:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.343+0000 7feee3db9740 1 Processor -- start 2024-04-26T01:51:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.344+0000 7feee3db9740 1 -- start start 2024-04-26T01:51:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.344+0000 7feee3db9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f8820 0x563b8c4f8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.344+0000 7feee3db9740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f9140 0x563b8c5034c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.344+0000 7feee3db9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c503a00 0x563b8c505df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.344+0000 7feee3db9740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x563b8c45e900 con 0x563b8c4f9140 2024-04-26T01:51:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.344+0000 7feee3db9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563b8c404ef0 con 0x563b8c503a00 2024-04-26T01:51:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.344+0000 7feee3db9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563b8c45a8c0 con 0x563b8c4f8820 2024-04-26T01:51:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.344+0000 7feeda800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f8820 0x563b8c4f8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.344+0000 7feeda800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f8820 0x563b8c4f8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36394/0 (socket says 172.21.15.6:36394) 2024-04-26T01:51:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.344+0000 7feeda800640 1 -- 172.21.15.6:0/144949319 learned_addr learned my addr 172.21.15.6:0/144949319 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.344+0000 7feed9400640 1 --2- 172.21.15.6:0/144949319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c503a00 0x563b8c505df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.344+0000 7feed9e00640 1 --2- 172.21.15.6:0/144949319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f9140 0x563b8c5034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.345+0000 7feeda800640 1 -- 172.21.15.6:0/144949319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c503a00 msgr2=0x563b8c505df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.345+0000 7feeda800640 1 --2- 172.21.15.6:0/144949319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c503a00 0x563b8c505df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.345+0000 7feeda800640 1 -- 172.21.15.6:0/144949319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f9140 msgr2=0x563b8c5034c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.345+0000 7feeda800640 1 --2- 172.21.15.6:0/144949319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f9140 0x563b8c5034c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.345+0000 7feeda800640 1 -- 172.21.15.6:0/144949319 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b8c171210 con 0x563b8c4f8820 2024-04-26T01:51:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.345+0000 7feed9e00640 1 --2- 172.21.15.6:0/144949319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f9140 0x563b8c5034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.345+0000 7feeda800640 1 --2- 172.21.15.6:0/144949319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f8820 0x563b8c4f8c00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feed4011790 tx=0x7feed4011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1365009d4a831b93 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.345+0000 7feec7400640 1 -- 172.21.15.6:0/144949319 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed4003c30 con 0x563b8c4f8820 2024-04-26T01:51:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.345+0000 7feec7400640 1 -- 172.21.15.6:0/144949319 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feed400fe50 con 0x563b8c4f8820 2024-04-26T01:51:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.345+0000 7feec7400640 1 -- 172.21.15.6:0/144949319 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed40193e0 con 0x563b8c4f8820 2024-04-26T01:51:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.346+0000 7feed2a00640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7feec00a3d10 0x7feec00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.349 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.346+0000 7feed2a00640 1 -- 172.21.15.6:0/124587760 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8556+0+0 (secure 0 0 0) 0x7feec80ef920 con 0x7feedc107270 2024-04-26T01:51:42.349 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.346+0000 7feed9e00640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7feec00a3d10 0x7feec00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.349 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.346+0000 7feee3db9740 1 -- 172.21.15.6:0/144949319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f8820 msgr2=0x563b8c4f8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.346+0000 7feee3db9740 1 --2- 172.21.15.6:0/144949319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f8820 0x563b8c4f8c00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feed4011790 tx=0x7feed4011c50 comp rx=0 tx=0).stop 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.346+0000 7feee3db9740 1 -- 172.21.15.6:0/144949319 shutdown_connections 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.346+0000 7feee3db9740 1 --2- 172.21.15.6:0/144949319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f9140 0x563b8c5034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.346+0000 7feee3db9740 1 --2- 172.21.15.6:0/144949319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c503a00 0x563b8c505df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.346+0000 7feee3db9740 1 --2- 172.21.15.6:0/144949319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f8820 0x563b8c4f8c00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.346+0000 7feee3db9740 1 -- 172.21.15.6:0/144949319 >> 172.21.15.6:0/144949319 conn(0x563b8c4f4d30 msgr2=0x563b8c4f5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.346+0000 7feee3db9740 1 -- 172.21.15.6:0/144949319 shutdown_connections 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.347+0000 7feee3db9740 1 -- 172.21.15.6:0/144949319 wait complete. 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.347+0000 7feed9e00640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7feec00a3d10 0x7feec00a61d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x563b8c4f9dc0 tx=0x7feecc009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.347+0000 7feee3db9740 1 -- start start 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.347+0000 7feee3db9740 1 Processor -- start 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feee3db9740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f8820 0x563b8c500950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feee3db9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f9140 0x563b8c500e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feee3db9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c503a00 0x563b8c5013d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feee3db9740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x563b8c506630 con 0x563b8c4f8820 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feee3db9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563b8c418310 con 0x563b8c4f9140 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feee3db9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563b8c5067b0 con 0x563b8c503a00 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feee3db9740 10 client.0 _ll_register_callbacks cb 0x563b8c3e6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feed9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f8820 0x563b8c500950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feeda800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c503a00 0x563b8c5013d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feed9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f8820 0x563b8c500950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59368/0 (socket says 172.21.15.6:59368) 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feed9e00640 1 -- 172.21.15.6:0/802589826 learned_addr learned my addr 172.21.15.6:0/802589826 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feed9400640 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f9140 0x563b8c500e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.348+0000 7feed9e00640 1 -- 172.21.15.6:0/802589826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c503a00 msgr2=0x563b8c5013d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.349+0000 7feed9e00640 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c503a00 0x563b8c5013d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.349+0000 7feed9e00640 1 -- 172.21.15.6:0/802589826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f9140 msgr2=0x563b8c500e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.349+0000 7feed9e00640 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f9140 0x563b8c500e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.349+0000 7feed9e00640 1 -- 172.21.15.6:0/802589826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b8c23d620 con 0x563b8c4f8820 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.349+0000 7feeda800640 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c503a00 0x563b8c5013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.349+0000 7feed9400640 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f9140 0x563b8c500e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.349+0000 7feed9e00640 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f8820 0x563b8c500950 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feecc01fa00 tx=0x7feecc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367a3566c8a7dff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.349+0000 7feebbe00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.349+0000 7feebbe00640 1 -- 172.21.15.6:0/802589826 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc01d620 con 0x563b8c4f8820 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.349+0000 7feee3db9740 1 -- 172.21.15.6:0/802589826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feed400fe50 con 0x563b8c4f8820 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.349+0000 7feebbe00640 1 -- 172.21.15.6:0/802589826 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8556+0+0 (secure 0 0 0) 0x7feecc01d7c0 con 0x563b8c4f8820 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.350+0000 7feebbe00640 1 -- 172.21.15.6:0/802589826 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7feecc01d9c0 con 0x563b8c4f8820 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.350+0000 7feee3db9740 10 client.15702 fetch_fsmap learned FSMap version 37 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.350+0000 7feee3db9740 1 -- 172.21.15.6:0/802589826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563b8c516d00 con 0x563b8c4f8820 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.350+0000 7feebbe00640 1 -- 172.21.15.6:0/802589826 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feed400fe50 con 0x563b8c4f8820 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.350+0000 7feebbe00640 1 -- 172.21.15.6:0/802589826 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7feecc032cd0 con 0x563b8c4f8820 2024-04-26T01:51:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.350+0000 7feee3db9740 10 client.15702 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:51:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.350+0000 7feee3db9740 10 client.15702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.350+0000 7feee3db9740 20 client.15702 populate_metadata read hostname 'smithi006' 2024-04-26T01:51:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.350+0000 7feee3db9740 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563b8c5181c0 0x563b8c51a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.350+0000 7feee3db9740 4 client.15702 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.350+0000 7feee3db9740 1 -- 172.21.15.6:0/802589826 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b8c4728b0 con 0x563b8c5181c0 2024-04-26T01:51:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.351+0000 7feed9400640 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563b8c5181c0 0x563b8c51a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.351+0000 7feed9400640 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563b8c5181c0 0x563b8c51a5f0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ea17093f748b50a server_cookie=2a4ad2220328d282 in_seq=0 out_seq=0 2024-04-26T01:51:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.351+0000 7feebbe00640 10 client.15702 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.352+0000 7feebbe00640 1 -- 172.21.15.6:0/802589826 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563b8c4728b0 con 0x563b8c5181c0 2024-04-26T01:51:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.352+0000 7feebbe00640 10 client.15702 handle_command_reply: tid=0 2024-04-26T01:51:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.352+0000 7feee3db9740 1 client.15702 shutdown 2024-04-26T01:51:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.352+0000 7feee3db9740 1 -- 172.21.15.6:0/802589826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f8820 msgr2=0x563b8c500950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.352+0000 7feee3db9740 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f8820 0x563b8c500950 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feecc01fa00 tx=0x7feecc004730 comp rx=0 tx=0).stop 2024-04-26T01:51:42.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.352+0000 7feee3db9740 1 -- 172.21.15.6:0/802589826 shutdown_connections 2024-04-26T01:51:42.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.352+0000 7feee3db9740 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563b8c5181c0 0x563b8c51a5f0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.352+0000 7feee3db9740 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c503a00 0x563b8c5013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.352+0000 7feee3db9740 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f9140 0x563b8c500e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.352+0000 7feee3db9740 1 --2- 172.21.15.6:0/802589826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c4f8820 0x563b8c500950 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.352+0000 7feebbe00640 0 client.15702 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:42.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.352+0000 7feee3db9740 1 -- 172.21.15.6:0/802589826 >> 172.21.15.6:0/802589826 conn(0x563b8c4f4d30 msgr2=0x563b8c4faab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:42.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.353+0000 7feee3db9740 1 -- 172.21.15.6:0/802589826 shutdown_connections 2024-04-26T01:51:42.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.353+0000 7feee3db9740 1 -- 172.21.15.6:0/802589826 wait complete. 2024-04-26T01:51:42.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.353+0000 7feee3db9740 20 client.15702 trim_cache size 0 max 16384 2024-04-26T01:51:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.370+0000 7feee3db9740 1 Processor -- start 2024-04-26T01:51:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.371+0000 7feee3db9740 1 -- start start 2024-04-26T01:51:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.371+0000 7feee3db9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 0x563b8c513940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.371+0000 7feee3db9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f9140 0x563b8c51ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.371+0000 7feee3db9740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 0x563b8c520440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.371+0000 7feee3db9740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1ae2c0 con 0x563b8c503a00 2024-04-26T01:51:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.371+0000 7feee3db9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1abb90 con 0x563b8c4f8820 2024-04-26T01:51:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.371+0000 7feee3db9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1a99e0 con 0x563b8c4f9140 2024-04-26T01:51:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.371+0000 7feed9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 0x563b8c513940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.371+0000 7feed9400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f9140 0x563b8c51ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.371+0000 7feed9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 0x563b8c513940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53514/0 (socket says 172.21.15.6:53514) 2024-04-26T01:51:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.371+0000 7feeda800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 0x563b8c520440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.371+0000 7feed9e00640 1 -- 172.21.15.6:0/3240837233 learned_addr learned my addr 172.21.15.6:0/3240837233 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.372+0000 7feed9e00640 1 -- 172.21.15.6:0/3240837233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f9140 msgr2=0x563b8c51ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.372+0000 7feed9e00640 1 --2- 172.21.15.6:0/3240837233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f9140 0x563b8c51ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.372+0000 7feed9e00640 1 -- 172.21.15.6:0/3240837233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 msgr2=0x563b8c520440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.372+0000 7feed9e00640 1 --2- 172.21.15.6:0/3240837233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 0x563b8c520440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.372+0000 7feed9e00640 1 -- 172.21.15.6:0/3240837233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b8c4f0690 con 0x563b8c4f8820 2024-04-26T01:51:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.372+0000 7feeda800640 1 --2- 172.21.15.6:0/3240837233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 0x563b8c520440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.372+0000 7feed9400640 1 --2- 172.21.15.6:0/3240837233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f9140 0x563b8c51ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.372+0000 7feed9e00640 1 --2- 172.21.15.6:0/3240837233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 0x563b8c513940 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feecc012cc0 tx=0x7feecc01ff00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d55442cfc890b56 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.372+0000 7feec6a00640 1 -- 172.21.15.6:0/3240837233 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc01d820 con 0x563b8c4f8820 2024-04-26T01:51:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.373+0000 7feec6a00640 1 -- 172.21.15.6:0/3240837233 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feecc0054d0 con 0x563b8c4f8820 2024-04-26T01:51:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.373+0000 7feec6a00640 1 -- 172.21.15.6:0/3240837233 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc004a00 con 0x563b8c4f8820 2024-04-26T01:51:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.373+0000 7feee3db9740 1 -- 172.21.15.6:0/3240837233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 msgr2=0x563b8c513940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.373+0000 7feee3db9740 1 --2- 172.21.15.6:0/3240837233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 0x563b8c513940 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feecc012cc0 tx=0x7feecc01ff00 comp rx=0 tx=0).stop 2024-04-26T01:51:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.373+0000 7feee3db9740 1 -- 172.21.15.6:0/3240837233 shutdown_connections 2024-04-26T01:51:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.373+0000 7feee3db9740 1 --2- 172.21.15.6:0/3240837233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 0x563b8c520440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.373+0000 7feee3db9740 1 --2- 172.21.15.6:0/3240837233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f9140 0x563b8c51ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.373+0000 7feee3db9740 1 --2- 172.21.15.6:0/3240837233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 0x563b8c513940 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.373+0000 7feee3db9740 1 -- 172.21.15.6:0/3240837233 >> 172.21.15.6:0/3240837233 conn(0x563b8c4f4d30 msgr2=0x563b8c4fbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.374+0000 7feee3db9740 1 -- 172.21.15.6:0/3240837233 shutdown_connections 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.374+0000 7feee3db9740 1 -- 172.21.15.6:0/3240837233 wait complete. 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.374+0000 7feee3db9740 1 -- start start 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.374+0000 7feee3db9740 1 Processor -- start 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.375+0000 7feee3db9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 0x563b8c510170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.375+0000 7feee3db9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f9140 0x563b8c50d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.375+0000 7feee3db9740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 0x563b8c50da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.375+0000 7feee3db9740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1aa050 con 0x563b8c503a00 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.375+0000 7feee3db9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1ab4c0 con 0x563b8c4f8820 2024-04-26T01:51:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.375+0000 7feee3db9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1aa720 con 0x563b8c4f9140 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.375+0000 7feee3db9740 10 client.0 _ll_register_callbacks cb 0x7feecc01f890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.375+0000 7feeda800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 0x563b8c50da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.375+0000 7feeda800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 0x563b8c50da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59378/0 (socket says 172.21.15.6:59378) 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.375+0000 7feed9400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f9140 0x563b8c50d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.375+0000 7feeda800640 1 -- 172.21.15.6:0/2290445399 learned_addr learned my addr 172.21.15.6:0/2290445399 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.375+0000 7feed9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 0x563b8c510170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feed9e00640 1 -- 172.21.15.6:0/2290445399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f9140 msgr2=0x563b8c50d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feed9e00640 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f9140 0x563b8c50d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feed9e00640 1 -- 172.21.15.6:0/2290445399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 msgr2=0x563b8c50da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feed9e00640 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 0x563b8c50da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feed9e00640 1 -- 172.21.15.6:0/2290445399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b8c50e180 con 0x563b8c4f8820 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feeda800640 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 0x563b8c50da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feed9400640 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f9140 0x563b8c50d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feed9e00640 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 0x563b8c510170 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7feecc009c80 tx=0x7feecc01d8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7d292bc63f8b054 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feebbe00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:51:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feebbe00640 1 -- 172.21.15.6:0/2290445399 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc0052f0 con 0x563b8c4f8820 2024-04-26T01:51:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feebbe00640 1 -- 172.21.15.6:0/2290445399 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8556+0+0 (secure 0 0 0) 0x7feecc004730 con 0x563b8c4f8820 2024-04-26T01:51:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feee3db9740 1 -- 172.21.15.6:0/2290445399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feecc0054d0 con 0x563b8c4f8820 2024-04-26T01:51:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feebbe00640 1 -- 172.21.15.6:0/2290445399 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7feecc048050 con 0x563b8c4f8820 2024-04-26T01:51:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feee3db9740 10 client.25838 fetch_fsmap learned FSMap version 37 2024-04-26T01:51:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.376+0000 7feee3db9740 1 -- 172.21.15.6:0/2290445399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563b8c57e6c0 con 0x563b8c4f8820 2024-04-26T01:51:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.377+0000 7feebbe00640 1 -- 172.21.15.6:0/2290445399 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feecc0054d0 con 0x563b8c4f8820 2024-04-26T01:51:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.377+0000 7feebbe00640 1 -- 172.21.15.6:0/2290445399 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7feecc029be0 con 0x563b8c4f8820 2024-04-26T01:51:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.377+0000 7feee3db9740 10 client.25838 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:51:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.377+0000 7feee3db9740 10 client.25838 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.377+0000 7feee3db9740 20 client.25838 populate_metadata read hostname 'smithi006' 2024-04-26T01:51:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.377+0000 7feee3db9740 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563b8c57ecf0 0x563b8c581110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.377+0000 7feee3db9740 4 client.25838 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.377+0000 7feee3db9740 1 -- 172.21.15.6:0/2290445399 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563b8c46c320 con 0x563b8c57ecf0 2024-04-26T01:51:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.377+0000 7feed9400640 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563b8c57ecf0 0x563b8c581110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.378+0000 7feed9400640 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563b8c57ecf0 0x563b8c581110 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=587ca0d85a56a972 server_cookie=959571553633e832 in_seq=0 out_seq=0 2024-04-26T01:51:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.378+0000 7feebbe00640 10 client.25838 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.419+0000 7feebbe00640 1 -- 172.21.15.6:0/2290445399 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563b8c46c320 con 0x563b8c57ecf0 2024-04-26T01:51:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.419+0000 7feebbe00640 10 client.25838 handle_command_reply: tid=0 2024-04-26T01:51:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.419+0000 7feee3db9740 1 client.25838 shutdown 2024-04-26T01:51:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.419+0000 7feee3db9740 1 -- 172.21.15.6:0/2290445399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 msgr2=0x563b8c510170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.419+0000 7feee3db9740 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 0x563b8c510170 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7feecc009c80 tx=0x7feecc01d8d0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.419+0000 7feee3db9740 1 -- 172.21.15.6:0/2290445399 shutdown_connections 2024-04-26T01:51:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.419+0000 7feee3db9740 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563b8c57ecf0 0x563b8c581110 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.419+0000 7feee3db9740 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563b8c503a00 0x563b8c50da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.419+0000 7feee3db9740 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b8c4f9140 0x563b8c50d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.419+0000 7feee3db9740 1 --2- 172.21.15.6:0/2290445399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563b8c4f8820 0x563b8c510170 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.419+0000 7feebbe00640 0 client.25838 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.419+0000 7feee3db9740 1 -- 172.21.15.6:0/2290445399 >> 172.21.15.6:0/2290445399 conn(0x563b8c4f4d30 msgr2=0x563b8c4f3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.420+0000 7feee3db9740 1 -- 172.21.15.6:0/2290445399 shutdown_connections 2024-04-26T01:51:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.420+0000 7feee3db9740 1 -- 172.21.15.6:0/2290445399 wait complete. 2024-04-26T01:51:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.420+0000 7feee3db9740 20 client.25838 trim_cache size 0 max 16384 2024-04-26T01:51:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:51:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:51:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:51:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:51:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.421+0000 7feee0e00640 1 -- 172.21.15.6:0/124587760 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7feec00a3d10 msgr2=0x7feec00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.421+0000 7feee0e00640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7feec00a3d10 0x7feec00a61d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x563b8c4f9dc0 tx=0x7feecc009260 comp rx=0 tx=0).stop 2024-04-26T01:51:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.421+0000 7feee0e00640 1 -- 172.21.15.6:0/124587760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feedc107270 msgr2=0x7feedc19e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.421+0000 7feee0e00640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feedc107270 0x7feedc19e940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feec802a860 tx=0x7feec8032d50 comp rx=0 tx=0).stop 2024-04-26T01:51:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.421+0000 7feee0e00640 1 -- 172.21.15.6:0/124587760 shutdown_connections 2024-04-26T01:51:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.421+0000 7feee0e00640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7feec00a3d10 0x7feec00a61d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.421+0000 7feee0e00640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feedc1985b0 0x7feedc198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.421+0000 7feee0e00640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feedc107270 0x7feedc19e940 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.421+0000 7feee0e00640 1 --2- 172.21.15.6:0/124587760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feedc1021a0 0x7feedc19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.421+0000 7feee0e00640 1 -- 172.21.15.6:0/124587760 >> 172.21.15.6:0/124587760 conn(0x7feedc0fd9e0 msgr2=0x7feedc0feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.421+0000 7feee0e00640 1 -- 172.21.15.6:0/124587760 shutdown_connections 2024-04-26T01:51:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:42.421+0000 7feee0e00640 1 -- 172.21.15.6:0/124587760 wait complete. 2024-04-26T01:51:42.581 INFO:tasks.workunit.client.0.smithi006.stdout: 1 192223 4.25 MB/sec execute 126 sec latency 24.585 ms 2024-04-26T01:51:42.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:42 smithi029 ceph-mon[32775]: pgmap v180: 97 pgs: 97 active+clean; 70 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 252 op/s 2024-04-26T01:51:42.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:42 smithi006 ceph-mon[28419]: pgmap v180: 97 pgs: 97 active+clean; 70 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 252 op/s 2024-04-26T01:51:42.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:42 smithi188 ceph-mon[32751]: pgmap v180: 97 pgs: 97 active+clean; 70 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 252 op/s 2024-04-26T01:51:43.581 INFO:tasks.workunit.client.0.smithi006.stdout: 1 193215 4.25 MB/sec execute 127 sec latency 23.598 ms 2024-04-26T01:51:44.582 INFO:tasks.workunit.client.0.smithi006.stdout: 1 194569 4.28 MB/sec execute 128 sec latency 28.008 ms 2024-04-26T01:51:44.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:44 smithi029 ceph-mon[32775]: pgmap v181: 97 pgs: 97 active+clean; 70 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-26T01:51:44.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:44 smithi006 ceph-mon[28419]: pgmap v181: 97 pgs: 97 active+clean; 70 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-26T01:51:44.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:44 smithi188 ceph-mon[32751]: pgmap v181: 97 pgs: 97 active+clean; 70 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-26T01:51:45.582 INFO:tasks.workunit.client.0.smithi006.stdout: 1 195495 4.30 MB/sec execute 129 sec latency 18.101 ms 2024-04-26T01:51:46.582 INFO:tasks.workunit.client.0.smithi006.stdout: 1 196317 4.28 MB/sec execute 130 sec latency 16.157 ms 2024-04-26T01:51:46.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:46 smithi029 ceph-mon[32775]: pgmap v182: 97 pgs: 97 active+clean; 70 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-26T01:51:46.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:46 smithi006 ceph-mon[28419]: pgmap v182: 97 pgs: 97 active+clean; 70 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-26T01:51:46.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:46 smithi188 ceph-mon[32751]: pgmap v182: 97 pgs: 97 active+clean; 70 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-26T01:51:47.583 INFO:tasks.workunit.client.0.smithi006.stdout: 1 197551 4.29 MB/sec execute 131 sec latency 24.272 ms 2024-04-26T01:51:48.236 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-04-26T01:51:48.237 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:51:48.582 INFO:tasks.workunit.client.0.smithi006.stdout: 1 198937 4.34 MB/sec execute 132 sec latency 25.132 ms 2024-04-26T01:51:48.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:48 smithi029 ceph-mon[32775]: pgmap v183: 97 pgs: 97 active+clean; 70 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-04-26T01:51:48.717 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:51:48.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:48 smithi006 ceph-mon[28419]: pgmap v183: 97 pgs: 97 active+clean; 70 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-04-26T01:51:48.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:48 smithi188 ceph-mon[32751]: pgmap v183: 97 pgs: 97 active+clean; 70 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-04-26T01:51:49.582 INFO:tasks.workunit.client.0.smithi006.stdout: 1 199871 4.33 MB/sec execute 133 sec latency 21.294 ms 2024-04-26T01:51:50.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:50 smithi006 ceph-mon[28419]: pgmap v184: 97 pgs: 97 active+clean; 70 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-04-26T01:51:50.582 INFO:tasks.workunit.client.0.smithi006.stdout: 1 201041 4.33 MB/sec execute 134 sec latency 28.166 ms 2024-04-26T01:51:50.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:50 smithi029 ceph-mon[32775]: pgmap v184: 97 pgs: 97 active+clean; 70 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-04-26T01:51:50.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:50 smithi188 ceph-mon[32751]: pgmap v184: 97 pgs: 97 active+clean; 70 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-04-26T01:51:51.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.228+0000 7fd0eb200700 1 -- 172.21.15.6:0/1264117875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0e406b0f0 msgr2=0x7fd0e410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:51.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.228+0000 7fd0eb200700 1 --2- 172.21.15.6:0/1264117875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0e406b0f0 0x7fd0e410f900 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009990 tx=0x7fd0dc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.228+0000 7fd0eb200700 1 -- 172.21.15.6:0/1264117875 shutdown_connections 2024-04-26T01:51:51.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.228+0000 7fd0eb200700 1 --2- 172.21.15.6:0/1264117875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0e406b0f0 0x7fd0e410f900 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.228+0000 7fd0eb200700 1 --2- 172.21.15.6:0/1264117875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0e4108a30 0x7fd0e4108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.228+0000 7fd0eb200700 1 --2- 172.21.15.6:0/1264117875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0e406a750 0x7fd0e406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.228+0000 7fd0eb200700 1 -- 172.21.15.6:0/1264117875 >> 172.21.15.6:0/1264117875 conn(0x7fd0e40788f0 msgr2=0x7fd0e4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:51.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.228+0000 7fd0eb200700 1 -- 172.21.15.6:0/1264117875 shutdown_connections 2024-04-26T01:51:51.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.229+0000 7fd0eb200700 1 -- 172.21.15.6:0/1264117875 wait complete. 2024-04-26T01:51:51.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.230+0000 7fd0eb200700 1 Processor -- start 2024-04-26T01:51:51.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.230+0000 7fd0eb200700 1 -- start start 2024-04-26T01:51:51.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.230+0000 7fd0eb200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0e406a750 0x7fd0e419b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:51.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.230+0000 7fd0eb200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0e406b0f0 0x7fd0e419b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:51.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.230+0000 7fd0eb200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0e4108a30 0x7fd0e419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:51.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.230+0000 7fd0eb200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e4112c60 con 0x7fd0e406a750 2024-04-26T01:51:51.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.230+0000 7fd0eb200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e4112ae0 con 0x7fd0e406b0f0 2024-04-26T01:51:51.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.230+0000 7fd0eb200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e4112de0 con 0x7fd0e4108a30 2024-04-26T01:51:51.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.231+0000 7fd0db400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0e406b0f0 0x7fd0e419b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:51.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.231+0000 7fd0db400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0e406b0f0 0x7fd0e419b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52224/0 (socket says 172.21.15.6:52224) 2024-04-26T01:51:51.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.231+0000 7fd0db400700 1 -- 172.21.15.6:0/1928002036 learned_addr learned my addr 172.21.15.6:0/1928002036 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:51.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.231+0000 7fd0e3e00700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0e406a750 0x7fd0e419b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.231+0000 7fd0db400700 1 -- 172.21.15.6:0/1928002036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0e4108a30 msgr2=0x7fd0e419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.231+0000 7fd0db400700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0e4108a30 0x7fd0e419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.231+0000 7fd0db400700 1 -- 172.21.15.6:0/1928002036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0e406a750 msgr2=0x7fd0e419b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.231+0000 7fd0db400700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0e406a750 0x7fd0e419b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.231+0000 7fd0db400700 1 -- 172.21.15.6:0/1928002036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e41a0310 con 0x7fd0e406b0f0 2024-04-26T01:51:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.231+0000 7fd0db400700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0e406b0f0 0x7fd0e419b930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00e9c0 tx=0x7fd0cc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.232+0000 7fd0e1600700 1 -- 172.21.15.6:0/1928002036 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc00cbd0 con 0x7fd0e406b0f0 2024-04-26T01:51:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.232+0000 7fd0e1600700 1 -- 172.21.15.6:0/1928002036 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0cc00cd70 con 0x7fd0e406b0f0 2024-04-26T01:51:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.232+0000 7fd0e1600700 1 -- 172.21.15.6:0/1928002036 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc005610 con 0x7fd0e406b0f0 2024-04-26T01:51:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.232+0000 7fd0eb200700 1 -- 172.21.15.6:0/1928002036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e41a0600 con 0x7fd0e406b0f0 2024-04-26T01:51:51.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.232+0000 7fd0eb200700 1 -- 172.21.15.6:0/1928002036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e41a7e80 con 0x7fd0e406b0f0 2024-04-26T01:51:51.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.233+0000 7fd0eb200700 1 -- 172.21.15.6:0/1928002036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0e4069730 con 0x7fd0e406b0f0 2024-04-26T01:51:51.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.235+0000 7fd0e1600700 1 -- 172.21.15.6:0/1928002036 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fd0cc004750 con 0x7fd0e406b0f0 2024-04-26T01:51:51.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.237+0000 7fd0e1600700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0c80a3db0 0x7fd0c80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:51.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.237+0000 7fd0e1600700 1 -- 172.21.15.6:0/1928002036 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8556+0+0 (secure 0 0 0) 0x7fd0cc0d0be0 con 0x7fd0e406b0f0 2024-04-26T01:51:51.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.237+0000 7fd0e3e00700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0c80a3db0 0x7fd0c80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:51.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.238+0000 7fd0e3e00700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0c80a3db0 0x7fd0c80a6270 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400f540 tx=0x7fd0d400b040 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:51.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.241+0000 7fd0e1600700 1 -- 172.21.15.6:0/1928002036 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0cc09d200 con 0x7fd0e406b0f0 2024-04-26T01:51:51.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.556+0000 7fd0eb200700 1 -- 172.21.15.6:0/1928002036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0e402df80 con 0x7fd0e406b0f0 2024-04-26T01:51:51.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.557+0000 7fd0e1600700 1 -- 172.21.15.6:0/1928002036 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11853 (secure 0 0 0) 0x7fd0cc09cb00 con 0x7fd0e406b0f0 2024-04-26T01:51:51.556 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:51:51.556 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:51:30.200605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:51:51.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.560+0000 7fd0eb200700 1 -- 172.21.15.6:0/1928002036 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0c80a3db0 msgr2=0x7fd0c80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:51.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.560+0000 7fd0eb200700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0c80a3db0 0x7fd0c80a6270 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400f540 tx=0x7fd0d400b040 comp rx=0 tx=0).stop 2024-04-26T01:51:51.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.560+0000 7fd0eb200700 1 -- 172.21.15.6:0/1928002036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0e406b0f0 msgr2=0x7fd0e419b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:51.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.560+0000 7fd0eb200700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0e406b0f0 0x7fd0e419b930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00e9c0 tx=0x7fd0cc00ecd0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.560+0000 7fd0eb200700 1 -- 172.21.15.6:0/1928002036 shutdown_connections 2024-04-26T01:51:51.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.560+0000 7fd0eb200700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0c80a3db0 0x7fd0c80a6270 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.560+0000 7fd0eb200700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0e4108a30 0x7fd0e419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.560+0000 7fd0eb200700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0e406a750 0x7fd0e419b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.560+0000 7fd0eb200700 1 --2- 172.21.15.6:0/1928002036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0e406b0f0 0x7fd0e419b930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:51.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.560+0000 7fd0eb200700 1 -- 172.21.15.6:0/1928002036 >> 172.21.15.6:0/1928002036 conn(0x7fd0e40788f0 msgr2=0x7fd0e410df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:51.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.560+0000 7fd0eb200700 1 -- 172.21.15.6:0/1928002036 shutdown_connections 2024-04-26T01:51:51.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:51.560+0000 7fd0eb200700 1 -- 172.21.15.6:0/1928002036 wait complete. 2024-04-26T01:51:51.560 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 37 2024-04-26T01:51:51.582 INFO:tasks.workunit.client.0.smithi006.stdout: 1 202463 4.39 MB/sec execute 135 sec latency 21.866 ms 2024-04-26T01:51:52.348 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 70 --await-for 42.0' 2024-04-26T01:51:52.349 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 70 --await-for 42.0 2024-04-26T01:51:52.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:52 smithi006 ceph-mon[28419]: pgmap v185: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 205 op/s 2024-04-26T01:51:52.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:52 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1928002036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:51:52.582 INFO:tasks.workunit.client.0.smithi006.stdout: 1 203406 4.37 MB/sec execute 136 sec latency 29.587 ms 2024-04-26T01:51:52.846 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:51:52.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:52 smithi188 ceph-mon[32751]: pgmap v185: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 205 op/s 2024-04-26T01:51:52.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:52 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1928002036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:51:52.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:52 smithi029 ceph-mon[32775]: pgmap v185: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 205 op/s 2024-04-26T01:51:52.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:52 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1928002036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:51:53.583 INFO:tasks.workunit.client.0.smithi006.stdout: 1 205083 4.41 MB/sec execute 137 sec latency 39.187 ms 2024-04-26T01:51:54.583 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206121 4.43 MB/sec execute 138 sec latency 37.265 ms 2024-04-26T01:51:54.749 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:54 smithi006 ceph-mon[28419]: pgmap v186: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 868 KiB/s wr, 136 op/s 2024-04-26T01:51:54.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:54 smithi188 ceph-mon[32751]: pgmap v186: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 868 KiB/s wr, 136 op/s 2024-04-26T01:51:54.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:54 smithi029 ceph-mon[32775]: pgmap v186: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 868 KiB/s wr, 136 op/s 2024-04-26T01:51:55.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.094+0000 7fb4d2000700 1 -- 172.21.15.6:0/1528534655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4cc1092d0 msgr2=0x7fb4cc1096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.094+0000 7fb4d2000700 1 --2- 172.21.15.6:0/1528534655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4cc1092d0 0x7fb4cc1096b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009990 tx=0x7fb4bc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.094+0000 7fb4d2000700 1 -- 172.21.15.6:0/1528534655 shutdown_connections 2024-04-26T01:51:55.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.094+0000 7fb4d2000700 1 --2- 172.21.15.6:0/1528534655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4cc10e3b0 0x7fb4cc115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.094+0000 7fb4d2000700 1 --2- 172.21.15.6:0/1528534655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4cc1092d0 0x7fb4cc1096b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.094+0000 7fb4d2000700 1 --2- 172.21.15.6:0/1528534655 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4cc109c80 0x7fb4cc10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.094+0000 7fb4d2000700 1 -- 172.21.15.6:0/1528534655 >> 172.21.15.6:0/1528534655 conn(0x7fb4cc076210 msgr2=0x7fb4cc107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:55.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.094+0000 7fb4d2000700 1 -- 172.21.15.6:0/1528534655 shutdown_connections 2024-04-26T01:51:55.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.094+0000 7fb4d2000700 1 -- 172.21.15.6:0/1528534655 wait complete. 2024-04-26T01:51:55.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.096+0000 7fb4d2000700 1 Processor -- start 2024-04-26T01:51:55.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.096+0000 7fb4d2000700 1 -- start start 2024-04-26T01:51:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.096+0000 7fb4d2000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4cc1092d0 0x7fb4cc19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.096+0000 7fb4d2000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4cc109c80 0x7fb4cc19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.096+0000 7fb4d2000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4cc10e3b0 0x7fb4cc1a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.096+0000 7fb4d2000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc119290 con 0x7fb4cc1092d0 2024-04-26T01:51:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.096+0000 7fb4d2000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc119110 con 0x7fb4cc10e3b0 2024-04-26T01:51:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.096+0000 7fb4d2000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc119410 con 0x7fb4cc109c80 2024-04-26T01:51:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.097+0000 7fb4caa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4cc109c80 0x7fb4cc19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.097+0000 7fb4caa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4cc109c80 0x7fb4cc19fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:41884/0 (socket says 172.21.15.6:41884) 2024-04-26T01:51:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.097+0000 7fb4caa00700 1 -- 172.21.15.6:0/3553397758 learned_addr learned my addr 172.21.15.6:0/3553397758 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.097+0000 7fb4cb400700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4cc1092d0 0x7fb4cc19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.097+0000 7fb4cbe00700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4cc10e3b0 0x7fb4cc1a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.097+0000 7fb4caa00700 1 -- 172.21.15.6:0/3553397758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4cc10e3b0 msgr2=0x7fb4cc1a3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.097+0000 7fb4caa00700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4cc10e3b0 0x7fb4cc1a3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.097+0000 7fb4caa00700 1 -- 172.21.15.6:0/3553397758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4cc1092d0 msgr2=0x7fb4cc19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.097+0000 7fb4caa00700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4cc1092d0 0x7fb4cc19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.097+0000 7fb4caa00700 1 -- 172.21.15.6:0/3553397758 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4cc1a4720 con 0x7fb4cc109c80 2024-04-26T01:51:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.097+0000 7fb4caa00700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4cc109c80 0x7fb4cc19fca0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800e970 tx=0x7fb4b800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.098+0000 7fb4c3e00700 1 -- 172.21.15.6:0/3553397758 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8004a30 con 0x7fb4cc109c80 2024-04-26T01:51:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.098+0000 7fb4d2000700 1 -- 172.21.15.6:0/3553397758 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4cc1a4a10 con 0x7fb4cc109c80 2024-04-26T01:51:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.098+0000 7fb4c3e00700 1 -- 172.21.15.6:0/3553397758 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4b800cc90 con 0x7fb4cc109c80 2024-04-26T01:51:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.098+0000 7fb4c3e00700 1 -- 172.21.15.6:0/3553397758 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8005610 con 0x7fb4cc109c80 2024-04-26T01:51:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.098+0000 7fb4d2000700 1 -- 172.21.15.6:0/3553397758 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4cc1ac290 con 0x7fb4cc109c80 2024-04-26T01:51:55.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.102+0000 7fb4c3e00700 1 -- 172.21.15.6:0/3553397758 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fb4b800c7e0 con 0x7fb4cc109c80 2024-04-26T01:51:55.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.104+0000 7fb4c3e00700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb4b00a3e50 0x7fb4b00a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.104+0000 7fb4dc9bcb80 1 Processor -- start 2024-04-26T01:51:55.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.104+0000 7fb4c3e00700 1 -- 172.21.15.6:0/3553397758 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8556+0+0 (secure 0 0 0) 0x7fb4b80d0a20 con 0x7fb4cc109c80 2024-04-26T01:51:55.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.104+0000 7fb4cb400700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb4b00a3e50 0x7fb4b00a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.104+0000 7fb4dc9bcb80 1 -- start start 2024-04-26T01:51:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.105+0000 7fb4dc9bcb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 0x55d7422441b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.105+0000 7fb4dc9bcb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d7422446f0 0x55d74224f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.105+0000 7fb4dc9bcb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d74224f560 0x55d742251950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.105+0000 7fb4dc9bcb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d742197020 con 0x55d74224f560 2024-04-26T01:51:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.105+0000 7fb4dc9bcb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d74213d8c0 con 0x55d742243dd0 2024-04-26T01:51:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.105+0000 7fb4dc9bcb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d742192fe0 con 0x55d7422446f0 2024-04-26T01:51:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.105+0000 7fb4caa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d7422446f0 0x55d74224f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.105+0000 7fb4cb400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 0x55d7422441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.105+0000 7fb4cbe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d74224f560 0x55d742251950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.105+0000 7fb4cb400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 0x55d7422441b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52240/0 (socket says 172.21.15.6:52240) 2024-04-26T01:51:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.105+0000 7fb4cb400700 1 -- 172.21.15.6:0/3340135787 learned_addr learned my addr 172.21.15.6:0/3340135787 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:55.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.106+0000 7fb4cb400700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb4b00a3e50 0x7fb4b00a6310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00b790 tx=0x7fb4bc005fd0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.106+0000 7fb4cb400700 1 -- 172.21.15.6:0/3340135787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d7422446f0 msgr2=0x55d74224f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.106+0000 7fb4cb400700 1 --2- 172.21.15.6:0/3340135787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d7422446f0 0x55d74224f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.106+0000 7fb4cb400700 1 -- 172.21.15.6:0/3340135787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d74224f560 msgr2=0x55d742251950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.106+0000 7fb4cb400700 1 --2- 172.21.15.6:0/3340135787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d74224f560 0x55d742251950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.106+0000 7fb4cb400700 1 -- 172.21.15.6:0/3340135787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d742133810 con 0x55d742243dd0 2024-04-26T01:51:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.106+0000 7fb4cbe00700 1 --2- 172.21.15.6:0/3340135787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d74224f560 0x55d742251950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.106+0000 7fb4cb400700 1 --2- 172.21.15.6:0/3340135787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 0x55d7422441b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc015000 tx=0x7fb4bc01f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39e2e314d5ca21bf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.107+0000 7fb4c0c00700 1 -- 172.21.15.6:0/3340135787 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc0056b0 con 0x55d742243dd0 2024-04-26T01:51:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.107+0000 7fb4c0c00700 1 -- 172.21.15.6:0/3340135787 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4bc00bc00 con 0x55d742243dd0 2024-04-26T01:51:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.107+0000 7fb4c0c00700 1 -- 172.21.15.6:0/3340135787 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc00f6c0 con 0x55d742243dd0 2024-04-26T01:51:55.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.108+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/3340135787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 msgr2=0x55d7422441b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.108+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/3340135787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 0x55d7422441b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc015000 tx=0x7fb4bc01f000 comp rx=0 tx=0).stop 2024-04-26T01:51:55.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.109+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/3340135787 shutdown_connections 2024-04-26T01:51:55.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.109+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/3340135787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d74224f560 0x55d742251950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.109+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/3340135787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 0x55d7422441b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.109+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/3340135787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d7422446f0 0x55d74224f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.109+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/3340135787 >> 172.21.15.6:0/3340135787 conn(0x55d7421730e0 msgr2=0x55d7422402e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:55.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.109+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/3340135787 shutdown_connections 2024-04-26T01:51:55.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.109+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/3340135787 wait complete. 2024-04-26T01:51:55.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.110+0000 7fb4dc9bcb80 1 -- start start 2024-04-26T01:51:55.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.111+0000 7fb4dc9bcb80 1 Processor -- start 2024-04-26T01:51:55.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.111+0000 7fb4dc9bcb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d742243dd0 0x55d74224ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.111+0000 7fb4dc9bcb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 0x55d74224bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.111+0000 7fb4dc9bcb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d74224f560 0x55d74224c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.111+0000 7fb4dc9bcb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d742251f30 con 0x55d742243dd0 2024-04-26T01:51:55.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.111+0000 7fb4dc9bcb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d7421509f0 con 0x55d7422446f0 2024-04-26T01:51:55.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.111+0000 7fb4dc9bcb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d7422520b0 con 0x55d74224f560 2024-04-26T01:51:55.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.112+0000 7fb4dc9bcb80 10 client.0 _ll_register_callbacks cb 0x55d74212f8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.112+0000 7fb4caa00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 0x55d74224bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.112+0000 7fb4caa00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 0x55d74224bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52254/0 (socket says 172.21.15.6:52254) 2024-04-26T01:51:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.112+0000 7fb4caa00700 1 -- 172.21.15.6:0/2528052204 learned_addr learned my addr 172.21.15.6:0/2528052204 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.112+0000 7fb4cbe00700 1 --2- 172.21.15.6:0/2528052204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d74224f560 0x55d74224c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.112+0000 7fb4caa00700 1 -- 172.21.15.6:0/2528052204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d74224f560 msgr2=0x55d74224c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.112+0000 7fb4caa00700 1 --2- 172.21.15.6:0/2528052204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d74224f560 0x55d74224c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.112+0000 7fb4caa00700 1 -- 172.21.15.6:0/2528052204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d742243dd0 msgr2=0x55d74224ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.112+0000 7fb4caa00700 1 --2- 172.21.15.6:0/2528052204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d742243dd0 0x55d74224ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.112+0000 7fb4caa00700 1 -- 172.21.15.6:0/2528052204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7420e4a60 con 0x55d7422446f0 2024-04-26T01:51:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.113+0000 7fb4caa00700 1 --2- 172.21.15.6:0/2528052204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 0x55d74224bf80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8099310 tx=0x7fb4b809c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a857d8541d1a3e26 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:55.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.113+0000 7fb4b5600700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:51:55.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.113+0000 7fb4b5600700 1 -- 172.21.15.6:0/2528052204 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b809d040 con 0x55d7422446f0 2024-04-26T01:51:55.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.113+0000 7fb4b5600700 1 -- 172.21.15.6:0/2528052204 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8556+0+0 (secure 0 0 0) 0x7fb4b809cba0 con 0x55d7422446f0 2024-04-26T01:51:55.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.113+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/2528052204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4bc00bc00 con 0x55d7422446f0 2024-04-26T01:51:55.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.114+0000 7fb4b5600700 1 -- 172.21.15.6:0/2528052204 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb4b809c940 con 0x55d7422446f0 2024-04-26T01:51:55.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.114+0000 7fb4dc9bcb80 10 client.25856 fetch_fsmap learned FSMap version 37 2024-04-26T01:51:55.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.114+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/2528052204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d74212dc80 con 0x55d7422446f0 2024-04-26T01:51:55.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.114+0000 7fb4b5600700 1 -- 172.21.15.6:0/2528052204 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4bc00bc00 con 0x55d7422446f0 2024-04-26T01:51:55.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.115+0000 7fb4b5600700 1 -- 172.21.15.6:0/2528052204 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fb4b80672a0 con 0x55d7422446f0 2024-04-26T01:51:55.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.115+0000 7fb4dc9bcb80 10 client.25856 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:51:55.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.115+0000 7fb4dc9bcb80 10 client.25856 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:51:55.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.115+0000 7fb4dc9bcb80 20 client.25856 populate_metadata read hostname 'smithi006' 2024-04-26T01:51:55.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.115+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/2528052204 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d74225ec60 0x55d742261050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.115+0000 7fb4dc9bcb80 4 client.25856 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:55.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.115+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/2528052204 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7421a4810 con 0x55d74225ec60 2024-04-26T01:51:55.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.115+0000 7fb4cb400700 1 --2- 172.21.15.6:0/2528052204 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d74225ec60 0x55d742261050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.116+0000 7fb4cb400700 1 --2- 172.21.15.6:0/2528052204 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d74225ec60 0x55d742261050 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=825c807e047fefaf server_cookie=4b7a9b188ecfb29a in_seq=0 out_seq=0 2024-04-26T01:51:55.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.116+0000 7fb4b5600700 10 client.25856 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:55.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.117+0000 7fb4b5600700 1 -- 172.21.15.6:0/2528052204 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7421a4810 con 0x55d74225ec60 2024-04-26T01:51:55.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.117+0000 7fb4b5600700 10 client.25856 handle_command_reply: tid=0 2024-04-26T01:51:55.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.117+0000 7fb4dc9bcb80 1 client.25856 shutdown 2024-04-26T01:51:55.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.117+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/2528052204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 msgr2=0x55d74224bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.117+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/2528052204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 0x55d74224bf80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8099310 tx=0x7fb4b809c320 comp rx=0 tx=0).stop 2024-04-26T01:51:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.117+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/2528052204 shutdown_connections 2024-04-26T01:51:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.117+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/2528052204 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d74225ec60 0x55d742261050 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.117+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/2528052204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d74224f560 0x55d74224c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.117+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/2528052204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d742243dd0 0x55d74224ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.117+0000 7fb4b5600700 0 client.25856 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.117+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/2528052204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 0x55d74224bf80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.117+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/2528052204 >> 172.21.15.6:0/2528052204 conn(0x55d7421730e0 msgr2=0x55d74223fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.118+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/2528052204 shutdown_connections 2024-04-26T01:51:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.118+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/2528052204 wait complete. 2024-04-26T01:51:55.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.118+0000 7fb4dc9bcb80 20 client.25856 trim_cache size 0 max 16384 2024-04-26T01:51:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4dc9bcb80 1 Processor -- start 2024-04-26T01:51:55.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4dc9bcb80 1 -- start start 2024-04-26T01:51:55.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4dc9bcb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d742243dd0 0x55d74224b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4dc9bcb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 0x55d7422573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4dc9bcb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d74224f560 0x55d7422578e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4dc9bcb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1b62d0 con 0x55d742243dd0 2024-04-26T01:51:55.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4dc9bcb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1b3b60 con 0x55d7422446f0 2024-04-26T01:51:55.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4dc9bcb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1b19b0 con 0x55d74224f560 2024-04-26T01:51:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4cb400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d742243dd0 0x55d74224b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4caa00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 0x55d7422573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4cb400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d742243dd0 0x55d74224b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53642/0 (socket says 172.21.15.6:53642) 2024-04-26T01:51:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4cb400700 1 -- 172.21.15.6:0/1852135720 learned_addr learned my addr 172.21.15.6:0/1852135720 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.143+0000 7fb4cbe00700 1 --2- 172.21.15.6:0/1852135720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d74224f560 0x55d7422578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.144+0000 7fb4caa00700 1 -- 172.21.15.6:0/1852135720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d74224f560 msgr2=0x55d7422578e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.144+0000 7fb4caa00700 1 --2- 172.21.15.6:0/1852135720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d74224f560 0x55d7422578e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.144+0000 7fb4caa00700 1 -- 172.21.15.6:0/1852135720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d742243dd0 msgr2=0x55d74224b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.144+0000 7fb4caa00700 1 --2- 172.21.15.6:0/1852135720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d742243dd0 0x55d74224b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.144+0000 7fb4caa00700 1 -- 172.21.15.6:0/1852135720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d742244d70 con 0x55d7422446f0 2024-04-26T01:51:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.144+0000 7fb4cb400700 1 --2- 172.21.15.6:0/1852135720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d742243dd0 0x55d74224b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:55.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.144+0000 7fb4caa00700 1 --2- 172.21.15.6:0/1852135720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 0x55d7422573a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009880 tx=0x7fb4b8098d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9c82fe70464cfb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:55.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.144+0000 7fb4b7e00700 1 -- 172.21.15.6:0/1852135720 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b809d040 con 0x55d7422446f0 2024-04-26T01:51:55.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.144+0000 7fb4b7e00700 1 -- 172.21.15.6:0/1852135720 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4b80a0cf0 con 0x55d7422446f0 2024-04-26T01:51:55.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.144+0000 7fb4b7e00700 1 -- 172.21.15.6:0/1852135720 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8087150 con 0x55d7422446f0 2024-04-26T01:51:55.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.145+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/1852135720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 msgr2=0x55d7422573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.145+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/1852135720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 0x55d7422573a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009880 tx=0x7fb4b8098d10 comp rx=0 tx=0).stop 2024-04-26T01:51:55.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.145+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/1852135720 shutdown_connections 2024-04-26T01:51:55.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.145+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/1852135720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d74224f560 0x55d7422578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.145+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/1852135720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d742243dd0 0x55d74224b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.145+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/1852135720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d7422446f0 0x55d7422573a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.145+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/1852135720 >> 172.21.15.6:0/1852135720 conn(0x55d7421730e0 msgr2=0x55d742259e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.145+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/1852135720 shutdown_connections 2024-04-26T01:51:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.145+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/1852135720 wait complete. 2024-04-26T01:51:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.146+0000 7fb4dc9bcb80 1 -- start start 2024-04-26T01:51:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.146+0000 7fb4dc9bcb80 1 Processor -- start 2024-04-26T01:51:55.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4dc9bcb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 0x55d7422bf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4dc9bcb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d7422446f0 0x55d7422bfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4dc9bcb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d74224f560 0x55d7422569f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4dc9bcb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1b2020 con 0x55d74224f560 2024-04-26T01:51:55.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4dc9bcb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1b3490 con 0x55d742243dd0 2024-04-26T01:51:55.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4dc9bcb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1b26f0 con 0x55d7422446f0 2024-04-26T01:51:55.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4dc9bcb80 10 client.0 _ll_register_callbacks cb 0x55d74212f8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:55.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4cb400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 0x55d7422bf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4caa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d7422446f0 0x55d7422bfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4caa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d7422446f0 0x55d7422bfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:41932/0 (socket says 172.21.15.6:41932) 2024-04-26T01:51:55.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4cb400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 0x55d7422bf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52274/0 (socket says 172.21.15.6:52274) 2024-04-26T01:51:55.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4caa00700 1 -- 172.21.15.6:0/4135834278 learned_addr learned my addr 172.21.15.6:0/4135834278 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4cbe00700 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d74224f560 0x55d7422569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4cb400700 1 -- 172.21.15.6:0/4135834278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d7422446f0 msgr2=0x55d7422bfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4cb400700 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d7422446f0 0x55d7422bfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4cb400700 1 -- 172.21.15.6:0/4135834278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d74224f560 msgr2=0x55d7422569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4cb400700 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d74224f560 0x55d7422569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4cb400700 1 -- 172.21.15.6:0/4135834278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d742257120 con 0x55d742243dd0 2024-04-26T01:51:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4caa00700 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d7422446f0 0x55d7422bfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.147+0000 7fb4cbe00700 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d74224f560 0x55d7422569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.148+0000 7fb4cb400700 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 0x55d7422bf800 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc030d00 tx=0x7fb4bc009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae47ba75152884c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.148+0000 7fb4b5600700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:51:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.148+0000 7fb4b5600700 1 -- 172.21.15.6:0/4135834278 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc03c060 con 0x55d742243dd0 2024-04-26T01:51:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.148+0000 7fb4b5600700 1 -- 172.21.15.6:0/4135834278 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8556+0+0 (secure 0 0 0) 0x7fb4bc00bd90 con 0x55d742243dd0 2024-04-26T01:51:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.148+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/4135834278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4b80a0cf0 con 0x55d742243dd0 2024-04-26T01:51:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.148+0000 7fb4b5600700 1 -- 172.21.15.6:0/4135834278 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb4bc030460 con 0x55d742243dd0 2024-04-26T01:51:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.148+0000 7fb4dc9bcb80 10 client.25862 fetch_fsmap learned FSMap version 37 2024-04-26T01:51:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.148+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/4135834278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d7422c7030 con 0x55d742243dd0 2024-04-26T01:51:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.149+0000 7fb4b5600700 1 -- 172.21.15.6:0/4135834278 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4b80a0cf0 con 0x55d742243dd0 2024-04-26T01:51:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.149+0000 7fb4b5600700 1 -- 172.21.15.6:0/4135834278 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fb4bc00f460 con 0x55d742243dd0 2024-04-26T01:51:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.149+0000 7fb4dc9bcb80 10 client.25862 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:51:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.149+0000 7fb4dc9bcb80 10 client.25862 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:51:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.149+0000 7fb4dc9bcb80 20 client.25862 populate_metadata read hostname 'smithi006' 2024-04-26T01:51:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.149+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d7422c7260 0x55d7422c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.149+0000 7fb4dc9bcb80 4 client.25862 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}] 2024-04-26T01:51:55.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.149+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/4135834278 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}) -- 0x55d74219e8b0 con 0x55d7422c7260 2024-04-26T01:51:55.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.149+0000 7fb4caa00700 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d7422c7260 0x55d7422c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.150+0000 7fb4caa00700 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d7422c7260 0x55d7422c9650 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df02537ea76ba23 server_cookie=e41d0afa4d06d663 in_seq=0 out_seq=0 2024-04-26T01:51:55.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.150+0000 7fb4b5600700 10 client.25862 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:55.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.211+0000 7fb4b5600700 1 -- 172.21.15.6:0/4135834278 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55d74219e8b0 con 0x55d7422c7260 2024-04-26T01:51:55.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.211+0000 7fb4b5600700 10 client.25862 handle_command_reply: tid=0 2024-04-26T01:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.211+0000 7fb4dc9bcb80 1 client.25862 shutdown 2024-04-26T01:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.211+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/4135834278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 msgr2=0x55d7422bf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.211+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 0x55d7422bf800 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc030d00 tx=0x7fb4bc009310 comp rx=0 tx=0).stop 2024-04-26T01:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.211+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/4135834278 shutdown_connections 2024-04-26T01:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.211+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d74224f560 0x55d7422569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.211+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d742243dd0 0x55d7422bf800 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.211+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d7422c7260 0x55d7422c9650 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.211+0000 7fb4dc9bcb80 1 --2- 172.21.15.6:0/4135834278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d7422446f0 0x55d7422bfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.211+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/4135834278 >> 172.21.15.6:0/4135834278 conn(0x55d7421730e0 msgr2=0x55d742240210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.211+0000 7fb4b5600700 0 client.25862 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.212+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/4135834278 shutdown_connections 2024-04-26T01:51:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.212+0000 7fb4dc9bcb80 1 -- 172.21.15.6:0/4135834278 wait complete. 2024-04-26T01:51:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.212+0000 7fb4dc9bcb80 20 client.25862 trim_cache size 0 max 16384 2024-04-26T01:51:55.212 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:51:55.212 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 37, 2024-04-26T01:51:55.212 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 14, 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "3f39552a": { 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 14, 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 70.0, 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:51:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:51:55.214 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:51:55.214 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:51:55.214 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:51:55.214 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:51:55.214 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:51:55.214 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:51:55.214 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:51:55.214 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:51:55.214 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:51:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.213+0000 7fb4d2000700 1 -- 172.21.15.6:0/3553397758 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb4b00a3e50 msgr2=0x7fb4b00a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.213+0000 7fb4d2000700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb4b00a3e50 0x7fb4b00a6310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00b790 tx=0x7fb4bc005fd0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.213+0000 7fb4d2000700 1 -- 172.21.15.6:0/3553397758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4cc109c80 msgr2=0x7fb4cc19fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.213+0000 7fb4d2000700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4cc109c80 0x7fb4cc19fca0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800e970 tx=0x7fb4b800ec80 comp rx=0 tx=0).stop 2024-04-26T01:51:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.213+0000 7fb4d2000700 1 -- 172.21.15.6:0/3553397758 shutdown_connections 2024-04-26T01:51:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.213+0000 7fb4d2000700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb4b00a3e50 0x7fb4b00a6310 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.213+0000 7fb4d2000700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4cc10e3b0 0x7fb4cc1a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.213+0000 7fb4d2000700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4cc1092d0 0x7fb4cc19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.213+0000 7fb4d2000700 1 --2- 172.21.15.6:0/3553397758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4cc109c80 0x7fb4cc19fca0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.213+0000 7fb4d2000700 1 -- 172.21.15.6:0/3553397758 >> 172.21.15.6:0/3553397758 conn(0x7fb4cc076210 msgr2=0x7fb4cc0767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.214+0000 7fb4d2000700 1 -- 172.21.15.6:0/3553397758 shutdown_connections 2024-04-26T01:51:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:51:55.214+0000 7fb4d2000700 1 -- 172.21.15.6:0/3553397758 wait complete. 2024-04-26T01:51:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.573+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3707939625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef0104d90 msgr2=0x7f3ef0108dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.573+0000 7f3ef6c00640 1 --2- 172.21.15.6:0/3707939625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef0104d90 0x7f3ef0108dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3edc009850 tx=0x7f3edc02a340 comp rx=0 tx=0).stop 2024-04-26T01:51:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.573+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3707939625 shutdown_connections 2024-04-26T01:51:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.573+0000 7f3ef6c00640 1 --2- 172.21.15.6:0/3707939625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef01094b0 0x7f3ef0111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.573+0000 7f3ef6c00640 1 --2- 172.21.15.6:0/3707939625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef0104d90 0x7f3ef0108dd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.573+0000 7f3ef6c00640 1 --2- 172.21.15.6:0/3707939625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ef01043e0 0x7f3ef01047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.573+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3707939625 >> 172.21.15.6:0/3707939625 conn(0x7f3ef00ffc00 msgr2=0x7f3ef0102020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.573+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3707939625 shutdown_connections 2024-04-26T01:51:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.573+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3707939625 wait complete. 2024-04-26T01:51:55.575 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.574+0000 7f3ef6c00640 1 Processor -- start 2024-04-26T01:51:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.574+0000 7f3ef6c00640 1 -- start start 2024-04-26T01:51:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3ef6c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef01043e0 0x7f3ef01a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3ef6c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ef01094b0 0x7f3ef01a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3ef6c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef019a7f0 0x7f3ef019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3ef6c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01143d0 con 0x7f3ef01094b0 2024-04-26T01:51:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3ef6c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef0114250 con 0x7f3ef019a7f0 2024-04-26T01:51:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3ef6c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef0114550 con 0x7f3ef01043e0 2024-04-26T01:51:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3eefe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef01043e0 0x7f3ef01a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3eef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ef01094b0 0x7f3ef01a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3eef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ef01094b0 0x7f3ef01a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53660/0 (socket says 172.21.15.6:53660) 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3eefe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef01043e0 0x7f3ef01a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:41948/0 (socket says 172.21.15.6:41948) 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3eef400640 1 -- 172.21.15.6:0/3208821071 learned_addr learned my addr 172.21.15.6:0/3208821071 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef019a7f0 0x7f3ef019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3eefe00640 1 -- 172.21.15.6:0/3208821071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef019a7f0 msgr2=0x7f3ef019aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3eefe00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef019a7f0 0x7f3ef019aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3eefe00640 1 -- 172.21.15.6:0/3208821071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ef01094b0 msgr2=0x7f3ef01a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3eefe00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ef01094b0 0x7f3ef01a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3eefe00640 1 -- 172.21.15.6:0/3208821071 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef019b380 con 0x7f3ef01043e0 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.575+0000 7f3eef400640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ef01094b0 0x7f3ef01a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.576+0000 7f3eefe00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef01043e0 0x7f3ef01a0680 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3ee000e790 tx=0x7f3ee000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.576+0000 7f3eecc00640 1 -- 172.21.15.6:0/3208821071 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee000eea0 con 0x7f3ef01043e0 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.576+0000 7f3eecc00640 1 -- 172.21.15.6:0/3208821071 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ee000ccc0 con 0x7f3ef01043e0 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.576+0000 7f3eecc00640 1 -- 172.21.15.6:0/3208821071 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0018680 con 0x7f3ef01043e0 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.576+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3208821071 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef019b670 con 0x7f3ef01043e0 2024-04-26T01:51:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.576+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3208821071 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef01a7850 con 0x7f3ef01043e0 2024-04-26T01:51:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.581+0000 7f3ef9a3c740 1 Processor -- start 2024-04-26T01:51:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.581+0000 7f3ef9a3c740 1 -- start start 2024-04-26T01:51:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.581+0000 7f3eecc00640 1 -- 172.21.15.6:0/3208821071 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f3ee0010430 con 0x7f3ef01043e0 2024-04-26T01:51:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.581+0000 7f3ef9a3c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x56519988ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.581+0000 7f3ef9a3c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x5651998994c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.581+0000 7f3ef9a3c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x56519989bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.581+0000 7f3ef9a3c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5651997f4900 con 0x56519988f140 2024-04-26T01:51:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.581+0000 7f3ef9a3c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56519979aef0 con 0x56519988e820 2024-04-26T01:51:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.581+0000 7f3ef9a3c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5651997f08c0 con 0x565199899a00 2024-04-26T01:51:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.582+0000 7f3eefe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x5651998994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.582+0000 7f3ef4c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x56519988ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.582+0000 7f3eefe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x5651998994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53668/0 (socket says 172.21.15.6:53668) 2024-04-26T01:51:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.582+0000 7f3eefe00640 1 -- 172.21.15.6:0/1076977295 learned_addr learned my addr 172.21.15.6:0/1076977295 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.582+0000 7f3eef400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x56519989bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.582+0000 7f3ef4c00640 1 -- 172.21.15.6:0/1076977295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 msgr2=0x56519989bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.582+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/1076977295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x56519989bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.582+0000 7f3ef4c00640 1 -- 172.21.15.6:0/1076977295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 msgr2=0x5651998994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.582+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/1076977295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x5651998994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.582+0000 7f3ef4c00640 1 -- 172.21.15.6:0/1076977295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565199507210 con 0x56519988e820 2024-04-26T01:51:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.583+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/1076977295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x56519988ec00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8009f00 tx=0x7f3ee8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5c282d893326a4e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.583+0000 7f3ee5200640 1 -- 172.21.15.6:0/1076977295 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8004d20 con 0x56519988e820 2024-04-26T01:51:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.583+0000 7f3ee5200640 1 -- 172.21.15.6:0/1076977295 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ee8004050 con 0x56519988e820 2024-04-26T01:51:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.583+0000 7f3ee5200640 1 -- 172.21.15.6:0/1076977295 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8005280 con 0x56519988e820 2024-04-26T01:51:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.583+0000 7f3eecc00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3ed40a3d10 0x7f3ed40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.584+0000 7f3eecc00640 1 -- 172.21.15.6:0/3208821071 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8556+0+0 (secure 0 0 0) 0x7f3ee00d0a40 con 0x7f3ef01043e0 2024-04-26T01:51:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.584+0000 7f3eefe00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3ed40a3d10 0x7f3ed40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.584+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/1076977295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 msgr2=0x56519988ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.584+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/1076977295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x56519988ec00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8009f00 tx=0x7f3ee8011810 comp rx=0 tx=0).stop 2024-04-26T01:51:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.584+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/1076977295 shutdown_connections 2024-04-26T01:51:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.584+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/1076977295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x5651998994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.584+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/1076977295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x56519989bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.584+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/1076977295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x56519988ec00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.584+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/1076977295 >> 172.21.15.6:0/1076977295 conn(0x56519988ad30 msgr2=0x56519988b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.584+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/1076977295 shutdown_connections 2024-04-26T01:51:55.585 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.40 MB/sec execute 139 sec latency 408.344 ms 2024-04-26T01:51:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.584+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/1076977295 wait complete. 2024-04-26T01:51:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.584+0000 7f3eefe00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3ed40a3d10 0x7f3ed40a61d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0000c00 tx=0x7f3ee00d44f0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.585+0000 7f3ef9a3c740 1 -- start start 2024-04-26T01:51:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.585+0000 7f3ef9a3c740 1 Processor -- start 2024-04-26T01:51:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3ef9a3c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988e820 0x5651998969d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3ef9a3c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56519988f140 0x565199896f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3ef9a3c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565199899a00 0x565199897450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3ef9a3c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56519989c630 con 0x56519988e820 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3ef9a3c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5651997ae310 con 0x565199899a00 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3ef9a3c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56519989c7b0 con 0x56519988f140 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3ef9a3c740 10 client.0 _ll_register_callbacks cb 0x56519977ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3ef4c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988e820 0x5651998969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3ef4c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988e820 0x5651998969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53676/0 (socket says 172.21.15.6:53676) 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3ef4c00640 1 -- 172.21.15.6:0/921607920 learned_addr learned my addr 172.21.15.6:0/921607920 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3eef400640 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56519988f140 0x565199896f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565199899a00 0x565199897450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3eef400640 1 -- 172.21.15.6:0/921607920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565199899a00 msgr2=0x565199897450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3eef400640 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565199899a00 0x565199897450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3eef400640 1 -- 172.21.15.6:0/921607920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988e820 msgr2=0x5651998969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3eef400640 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988e820 0x5651998969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3eef400640 1 -- 172.21.15.6:0/921607920 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651995d3620 con 0x56519988f140 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.586+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565199899a00 0x565199897450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.587+0000 7f3eef400640 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56519988f140 0x565199896f10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3edc02a930 tx=0x7f3edc0045f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=482dd7efd2c14a5d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.587+0000 7f3ed9600640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.587+0000 7f3ed9600640 1 -- 172.21.15.6:0/921607920 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3edc037040 con 0x56519988f140 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.587+0000 7f3ed9600640 1 -- 172.21.15.6:0/921607920 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8556+0+0 (secure 0 0 0) 0x7f3edc0308d0 con 0x56519988f140 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.587+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/921607920 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ee8004050 con 0x56519988f140 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.587+0000 7f3ed9600640 1 -- 172.21.15.6:0/921607920 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3edc04c050 con 0x56519988f140 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.587+0000 7f3ef9a3c740 10 client.25918 fetch_fsmap learned FSMap version 37 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.587+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/921607920 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5651998a88e0 con 0x56519988f140 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.587+0000 7f3ed9600640 1 -- 172.21.15.6:0/921607920 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ee8004050 con 0x56519988f140 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.588+0000 7f3ed9600640 1 -- 172.21.15.6:0/921607920 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f3edc03d6c0 con 0x56519988f140 2024-04-26T01:51:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.588+0000 7f3ef9a3c740 10 client.25918 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.588+0000 7f3ef9a3c740 10 client.25918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.588+0000 7f3ef9a3c740 20 client.25918 populate_metadata read hostname 'smithi006' 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.588+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5651998a9da0 0x5651998ac190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.588+0000 7f3ef9a3c740 4 client.25918 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.588+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/921607920 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651998088b0 con 0x5651998a9da0 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.588+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5651998a9da0 0x5651998ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.589+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5651998a9da0 0x5651998ac190 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b26aff38d38ab5ae server_cookie=b9039a5e530bc5c9 in_seq=0 out_seq=0 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.589+0000 7f3ed9600640 10 client.25918 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.590+0000 7f3ed9600640 1 -- 172.21.15.6:0/921607920 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5651998088b0 con 0x5651998a9da0 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.590+0000 7f3ed9600640 10 client.25918 handle_command_reply: tid=0 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.590+0000 7f3ef9a3c740 1 client.25918 shutdown 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.590+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/921607920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56519988f140 msgr2=0x565199896f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.590+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56519988f140 0x565199896f10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3edc02a930 tx=0x7f3edc0045f0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.590+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/921607920 shutdown_connections 2024-04-26T01:51:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.590+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5651998a9da0 0x5651998ac190 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.590+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565199899a00 0x565199897450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.590+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56519988f140 0x565199896f10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.590+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/921607920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988e820 0x5651998969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.590+0000 7f3ed9600640 0 client.25918 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.590+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/921607920 >> 172.21.15.6:0/921607920 conn(0x56519988ad30 msgr2=0x565199890ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.591+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/921607920 shutdown_connections 2024-04-26T01:51:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.591+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/921607920 wait complete. 2024-04-26T01:51:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.591+0000 7f3ef9a3c740 20 client.25918 trim_cache size 0 max 16384 2024-04-26T01:51:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.613+0000 7f3ef9a3c740 1 Processor -- start 2024-04-26T01:51:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.614+0000 7f3ef9a3c740 1 -- start start 2024-04-26T01:51:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.614+0000 7f3ef9a3c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x5651998911e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.614+0000 7f3ef9a3c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x5651998970a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.614+0000 7f3ef9a3c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x5651998b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.614+0000 7f3ef9a3c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01b2850 con 0x56519988f140 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.614+0000 7f3ef9a3c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01b0120 con 0x56519988e820 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.614+0000 7f3ef9a3c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01adf70 con 0x565199899a00 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.614+0000 7f3eef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x5651998970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.614+0000 7f3eef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x5651998970a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53680/0 (socket says 172.21.15.6:53680) 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.614+0000 7f3eef400640 1 -- 172.21.15.6:0/2184926887 learned_addr learned my addr 172.21.15.6:0/2184926887 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.614+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/2184926887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x5651998b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.615+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/2184926887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x5651998911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.615+0000 7f3eef400640 1 -- 172.21.15.6:0/2184926887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 msgr2=0x5651998b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.615+0000 7f3eef400640 1 --2- 172.21.15.6:0/2184926887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x5651998b1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.615+0000 7f3eef400640 1 -- 172.21.15.6:0/2184926887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 msgr2=0x5651998911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.615+0000 7f3eef400640 1 --2- 172.21.15.6:0/2184926887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x5651998911e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.615+0000 7f3eef400640 1 -- 172.21.15.6:0/2184926887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56519988ef70 con 0x56519988f140 2024-04-26T01:51:55.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.615+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/2184926887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x5651998b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:55.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.615+0000 7f3eef400640 1 --2- 172.21.15.6:0/2184926887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x5651998970a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3edc035040 tx=0x7f3edc02a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d6d58bc76275c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:55.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.615+0000 7f3edbe00640 1 -- 172.21.15.6:0/2184926887 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3edc037040 con 0x56519988f140 2024-04-26T01:51:55.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.615+0000 7f3edbe00640 1 -- 172.21.15.6:0/2184926887 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3edc0057e0 con 0x56519988f140 2024-04-26T01:51:55.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.615+0000 7f3edbe00640 1 -- 172.21.15.6:0/2184926887 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3edc005460 con 0x56519988f140 2024-04-26T01:51:55.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.616+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/2184926887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 msgr2=0x5651998970a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.616+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/2184926887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x5651998970a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3edc035040 tx=0x7f3edc02a340 comp rx=0 tx=0).stop 2024-04-26T01:51:55.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.616+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/2184926887 shutdown_connections 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.616+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/2184926887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x5651998970a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.616+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/2184926887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x5651998b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.616+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/2184926887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x5651998911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.616+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/2184926887 >> 172.21.15.6:0/2184926887 conn(0x56519988ad30 msgr2=0x565199896990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.616+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/2184926887 shutdown_connections 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.616+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/2184926887 wait complete. 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.617+0000 7f3ef9a3c740 1 -- start start 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.617+0000 7f3ef9a3c740 1 Processor -- start 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3ef9a3c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x56519989fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3ef9a3c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x56519989ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3ef9a3c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x56519990f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3ef9a3c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01ae5e0 con 0x56519988f140 2024-04-26T01:51:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3ef9a3c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01afa50 con 0x56519988e820 2024-04-26T01:51:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3ef9a3c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01aecb0 con 0x565199899a00 2024-04-26T01:51:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3ef9a3c740 10 client.0 _ll_register_callbacks cb 0x7f3edc004490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3eef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x56519989ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3eef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x56519989ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53688/0 (socket says 172.21.15.6:53688) 2024-04-26T01:51:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3eef400640 1 -- 172.21.15.6:0/4011870398 learned_addr learned my addr 172.21.15.6:0/4011870398 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:51:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x56519990f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x56519989fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3eef400640 1 -- 172.21.15.6:0/4011870398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 msgr2=0x56519990f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3eef400640 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x56519990f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3eef400640 1 -- 172.21.15.6:0/4011870398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 msgr2=0x56519989fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3eef400640 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x56519989fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3eef400640 1 -- 172.21.15.6:0/4011870398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56519990f840 con 0x56519988f140 2024-04-26T01:51:55.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.618+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x56519989fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:55.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.619+0000 7f3eef400640 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x56519989ffb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3edc03dcd0 tx=0x7f3edc030900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127e77b40fd1d64 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:55.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.619+0000 7f3ed9600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:51:55.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.619+0000 7f3ed9600640 1 -- 172.21.15.6:0/4011870398 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3edc037070 con 0x56519988f140 2024-04-26T01:51:55.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.619+0000 7f3ed9600640 1 -- 172.21.15.6:0/4011870398 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8556+0+0 (secure 0 0 0) 0x7f3edc03d860 con 0x56519988f140 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.619+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/4011870398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x565199802320 con 0x56519988f140 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.619+0000 7f3ed9600640 1 -- 172.21.15.6:0/4011870398 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3edc060050 con 0x56519988f140 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.619+0000 7f3ef9a3c740 10 client.15747 fetch_fsmap learned FSMap version 37 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.619+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/4011870398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x565199910a50 con 0x56519988f140 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.619+0000 7f3ed9600640 1 -- 172.21.15.6:0/4011870398 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x565199802320 con 0x56519988f140 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.620+0000 7f3ed9600640 1 -- 172.21.15.6:0/4011870398 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f3edc0035f0 con 0x56519988f140 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.620+0000 7f3ef9a3c740 10 client.15747 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.620+0000 7f3ef9a3c740 10 client.15747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.620+0000 7f3ef9a3c740 20 client.15747 populate_metadata read hostname 'smithi006' 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.620+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x565199911050 0x565199913440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.620+0000 7f3ef9a3c740 4 client.15747 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.620+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/4011870398 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5651997fc190 con 0x565199911050 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.620+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x565199911050 0x565199913440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.621+0000 7f3ef4c00640 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x565199911050 0x565199913440 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51d2885b62a3977a server_cookie=17fdbc45c6b0987f in_seq=0 out_seq=0 2024-04-26T01:51:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.621+0000 7f3ed9600640 10 client.15747 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:55.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.665+0000 7f3ed9600640 1 -- 172.21.15.6:0/4011870398 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5651997fc190 con 0x565199911050 2024-04-26T01:51:55.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.665+0000 7f3ed9600640 10 client.15747 handle_command_reply: tid=0 2024-04-26T01:51:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.665+0000 7f3ef9a3c740 1 client.15747 shutdown 2024-04-26T01:51:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.665+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/4011870398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 msgr2=0x56519989ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.665+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x56519989ffb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3edc03dcd0 tx=0x7f3edc030900 comp rx=0 tx=0).stop 2024-04-26T01:51:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.665+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/4011870398 shutdown_connections 2024-04-26T01:51:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.665+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x565199911050 0x565199913440 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.665+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56519988f140 0x56519989ffb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.665+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x565199899a00 0x56519990f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.665+0000 7f3ef9a3c740 1 --2- 172.21.15.6:0/4011870398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56519988e820 0x56519989fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.665+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/4011870398 >> 172.21.15.6:0/4011870398 conn(0x56519988ad30 msgr2=0x565199897220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.665+0000 7f3ed9600640 0 client.15747 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:51:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.666+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/4011870398 shutdown_connections 2024-04-26T01:51:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.666+0000 7f3ef9a3c740 1 -- 172.21.15.6:0/4011870398 wait complete. 2024-04-26T01:51:55.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.666+0000 7f3ef9a3c740 20 client.15747 trim_cache size 0 max 16384 2024-04-26T01:51:55.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:51:55.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:51:55.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:51:55.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:51:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.667+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3208821071 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3ed40a3d10 msgr2=0x7f3ed40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.667+0000 7f3ef6c00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3ed40a3d10 0x7f3ed40a61d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0000c00 tx=0x7f3ee00d44f0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.667+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3208821071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef01043e0 msgr2=0x7f3ef01a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.667+0000 7f3ef6c00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef01043e0 0x7f3ef01a0680 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3ee000e790 tx=0x7f3ee000ec50 comp rx=0 tx=0).stop 2024-04-26T01:51:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.667+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3208821071 shutdown_connections 2024-04-26T01:51:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.667+0000 7f3ef6c00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3ed40a3d10 0x7f3ed40a61d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.667+0000 7f3ef6c00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ef01094b0 0x7f3ef01a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.667+0000 7f3ef6c00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef019a7f0 0x7f3ef019aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.667+0000 7f3ef6c00640 1 --2- 172.21.15.6:0/3208821071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef01043e0 0x7f3ef01a0680 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.667+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3208821071 >> 172.21.15.6:0/3208821071 conn(0x7f3ef00ffc00 msgr2=0x7f3ef0100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.667+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3208821071 shutdown_connections 2024-04-26T01:51:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:51:55.667+0000 7f3ef6c00640 1 -- 172.21.15.6:0/3208821071 wait complete. 2024-04-26T01:51:56.048 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '3f39552a', quiesce took 7.8 seconds. Will release after: 10 2024-04-26T01:51:56.583 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.37 MB/sec execute 140 sec latency 1408.463 ms 2024-04-26T01:51:56.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:56 smithi006 ceph-mon[28419]: pgmap v187: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 868 KiB/s wr, 136 op/s 2024-04-26T01:51:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:56 smithi188 ceph-mon[32751]: pgmap v187: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 868 KiB/s wr, 136 op/s 2024-04-26T01:51:56.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:56 smithi029 ceph-mon[32775]: pgmap v187: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 868 KiB/s wr, 136 op/s 2024-04-26T01:51:57.583 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.34 MB/sec execute 141 sec latency 2408.630 ms 2024-04-26T01:51:58.583 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.31 MB/sec execute 142 sec latency 3408.776 ms 2024-04-26T01:51:58.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:51:58 smithi006 ceph-mon[28419]: pgmap v188: 97 pgs: 97 active+clean; 70 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 235 op/s 2024-04-26T01:51:58.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:51:58 smithi188 ceph-mon[32751]: pgmap v188: 97 pgs: 97 active+clean; 70 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 235 op/s 2024-04-26T01:51:58.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:51:58 smithi029 ceph-mon[32775]: pgmap v188: 97 pgs: 97 active+clean; 70 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 235 op/s 2024-04-26T01:51:59.586 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.28 MB/sec execute 143 sec latency 4408.922 ms 2024-04-26T01:52:00.584 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.25 MB/sec execute 144 sec latency 5409.060 ms 2024-04-26T01:52:00.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:00 smithi006 ceph-mon[28419]: pgmap v189: 97 pgs: 97 active+clean; 70 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-26T01:52:00.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:00 smithi188 ceph-mon[32751]: pgmap v189: 97 pgs: 97 active+clean; 70 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-26T01:52:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:00 smithi029 ceph-mon[32775]: pgmap v189: 97 pgs: 97 active+clean; 70 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-26T01:52:01.584 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.22 MB/sec execute 145 sec latency 6409.201 ms 2024-04-26T01:52:02.584 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.19 MB/sec execute 146 sec latency 7409.366 ms 2024-04-26T01:52:02.766 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:02 smithi006 ceph-mon[28419]: pgmap v190: 97 pgs: 97 active+clean; 69 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-04-26T01:52:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:02 smithi188 ceph-mon[32751]: pgmap v190: 97 pgs: 97 active+clean; 69 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-04-26T01:52:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:02 smithi029 ceph-mon[32775]: pgmap v190: 97 pgs: 97 active+clean; 69 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-04-26T01:52:03.584 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.17 MB/sec execute 147 sec latency 8409.513 ms 2024-04-26T01:52:04.584 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.14 MB/sec execute 148 sec latency 9409.714 ms 2024-04-26T01:52:04.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:04 smithi006 ceph-mon[28419]: pgmap v191: 97 pgs: 97 active+clean; 69 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-04-26T01:52:04.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:04 smithi188 ceph-mon[32751]: pgmap v191: 97 pgs: 97 active+clean; 69 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-04-26T01:52:04.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:04 smithi029 ceph-mon[32775]: pgmap v191: 97 pgs: 97 active+clean; 69 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-04-26T01:52:05.584 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.11 MB/sec execute 149 sec latency 10409.865 ms 2024-04-26T01:52:05.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:05 smithi006 ceph-mon[28419]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T01:52:05.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:05 smithi188 ceph-mon[32751]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T01:52:05.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:05 smithi029 ceph-mon[32775]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T01:52:06.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:52:06.540 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:52:06.584 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.08 MB/sec execute 150 sec latency 11409.938 ms 2024-04-26T01:52:06.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:06 smithi006 ceph-mon[28419]: pgmap v193: 97 pgs: 97 active+clean; 69 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-04-26T01:52:06.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:06 smithi188 ceph-mon[32751]: pgmap v193: 97 pgs: 97 active+clean; 69 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-04-26T01:52:06.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:06 smithi029 ceph-mon[32775]: pgmap v193: 97 pgs: 97 active+clean; 69 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-04-26T01:52:07.585 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.05 MB/sec execute 151 sec latency 12410.017 ms 2024-04-26T01:52:08.585 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.03 MB/sec execute 152 sec latency 13410.569 ms 2024-04-26T01:52:08.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.637+0000 7efd9d800700 1 -- 172.21.15.6:0/3801150719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9810e3b0 msgr2=0x7efd98115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.637+0000 7efd9d800700 1 --2- 172.21.15.6:0/3801150719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9810e3b0 0x7efd98115f80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efd9000b0a0 tx=0x7efd9000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.638+0000 7efd9d800700 1 -- 172.21.15.6:0/3801150719 shutdown_connections 2024-04-26T01:52:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.638+0000 7efd9d800700 1 --2- 172.21.15.6:0/3801150719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9810e3b0 0x7efd98115f80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.638+0000 7efd9d800700 1 --2- 172.21.15.6:0/3801150719 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd981092d0 0x7efd981096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.638+0000 7efd9d800700 1 --2- 172.21.15.6:0/3801150719 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd98109c80 0x7efd9810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.638+0000 7efd9d800700 1 -- 172.21.15.6:0/3801150719 >> 172.21.15.6:0/3801150719 conn(0x7efd98076210 msgr2=0x7efd98107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.638+0000 7efd9d800700 1 -- 172.21.15.6:0/3801150719 shutdown_connections 2024-04-26T01:52:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.638+0000 7efd9d800700 1 -- 172.21.15.6:0/3801150719 wait complete. 2024-04-26T01:52:08.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.639+0000 7efd9d800700 1 Processor -- start 2024-04-26T01:52:08.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.639+0000 7efd9d800700 1 -- start start 2024-04-26T01:52:08.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.639+0000 7efd9d800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd981092d0 0x7efd9819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.639+0000 7efd9d800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd98109c80 0x7efd9819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.639+0000 7efd9d800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9810e3b0 0x7efd981a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.639+0000 7efd9d800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efd981192a0 con 0x7efd98109c80 2024-04-26T01:52:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.639+0000 7efd9d800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efd98119120 con 0x7efd9810e3b0 2024-04-26T01:52:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.639+0000 7efd9d800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efd98119420 con 0x7efd981092d0 2024-04-26T01:52:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.640+0000 7efd96a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd981092d0 0x7efd9819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.640+0000 7efd96000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd98109c80 0x7efd9819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.640+0000 7efd96a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd981092d0 0x7efd9819f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:57272/0 (socket says 172.21.15.6:57272) 2024-04-26T01:52:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.640+0000 7efd96a00700 1 -- 172.21.15.6:0/756574188 learned_addr learned my addr 172.21.15.6:0/756574188 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.640+0000 7efd97400700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9810e3b0 0x7efd981a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.640+0000 7efd96a00700 1 -- 172.21.15.6:0/756574188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9810e3b0 msgr2=0x7efd981a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.640+0000 7efd96a00700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9810e3b0 0x7efd981a3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.640+0000 7efd96a00700 1 -- 172.21.15.6:0/756574188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd98109c80 msgr2=0x7efd9819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.640+0000 7efd96a00700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd98109c80 0x7efd9819fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.640+0000 7efd96a00700 1 -- 172.21.15.6:0/756574188 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd981a4710 con 0x7efd981092d0 2024-04-26T01:52:08.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.640+0000 7efd97400700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9810e3b0 0x7efd981a3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:08.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.640+0000 7efd96000700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd98109c80 0x7efd9819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:08.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.641+0000 7efd96a00700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd981092d0 0x7efd9819f750 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efd8800e9a0 tx=0x7efd8800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:08.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.641+0000 7efd8f400700 1 -- 172.21.15.6:0/756574188 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8800ca10 con 0x7efd981092d0 2024-04-26T01:52:08.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.641+0000 7efd8f400700 1 -- 172.21.15.6:0/756574188 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd8800cbb0 con 0x7efd981092d0 2024-04-26T01:52:08.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.641+0000 7efd9d800700 1 -- 172.21.15.6:0/756574188 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd981a4a00 con 0x7efd981092d0 2024-04-26T01:52:08.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.641+0000 7efd8f400700 1 -- 172.21.15.6:0/756574188 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd88018680 con 0x7efd981092d0 2024-04-26T01:52:08.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.641+0000 7efd9d800700 1 -- 172.21.15.6:0/756574188 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd981ac280 con 0x7efd981092d0 2024-04-26T01:52:08.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.642+0000 7efd9d800700 1 -- 172.21.15.6:0/756574188 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd98069730 con 0x7efd981092d0 2024-04-26T01:52:08.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.644+0000 7efd8f400700 1 -- 172.21.15.6:0/756574188 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7efd88010430 con 0x7efd981092d0 2024-04-26T01:52:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.646+0000 7efd8f400700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7efd7c0a3e30 0x7efd7c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.646+0000 7efd8f400700 1 -- 172.21.15.6:0/756574188 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8644+0+0 (secure 0 0 0) 0x7efd880d0a20 con 0x7efd981092d0 2024-04-26T01:52:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.646+0000 7efd96000700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7efd7c0a3e30 0x7efd7c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.647+0000 7efd96000700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7efd7c0a3e30 0x7efd7c0a62f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd84009680 tx=0x7efd84009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:08.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.650+0000 7efd8f400700 1 -- 172.21.15.6:0/756574188 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd8809cfe0 con 0x7efd981092d0 2024-04-26T01:52:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.803+0000 7f710fe00640 1 -- 172.21.15.6:0/132326364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7110102f30 msgr2=0x7f7110103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.803+0000 7f710fe00640 1 --2- 172.21.15.6:0/132326364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7110102f30 0x7f7110103310 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7100009ca0 tx=0x7f710002a950 comp rx=0 tx=0).stop 2024-04-26T01:52:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.803+0000 7f710fe00640 1 -- 172.21.15.6:0/132326364 shutdown_connections 2024-04-26T01:52:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.803+0000 7f710fe00640 1 --2- 172.21.15.6:0/132326364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7110103850 0x7f711010bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.803+0000 7f710fe00640 1 --2- 172.21.15.6:0/132326364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711010c350 0x7f711010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.803+0000 7f710fe00640 1 --2- 172.21.15.6:0/132326364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7110102f30 0x7f7110103310 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.803+0000 7f710fe00640 1 -- 172.21.15.6:0/132326364 >> 172.21.15.6:0/132326364 conn(0x7f71100fc730 msgr2=0x7f71100feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.803+0000 7f710fe00640 1 -- 172.21.15.6:0/132326364 shutdown_connections 2024-04-26T01:52:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.803+0000 7f710fe00640 1 -- 172.21.15.6:0/132326364 wait complete. 2024-04-26T01:52:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.805+0000 7f710fe00640 1 Processor -- start 2024-04-26T01:52:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.805+0000 7f710fe00640 1 -- start start 2024-04-26T01:52:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.806+0000 7f710fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7110102f30 0x7f7110100b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.806+0000 7f710fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7110103850 0x7f71101010c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.806+0000 7f710fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711010c350 0x7f7110101600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.806+0000 7f710fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7110111fc0 con 0x7f7110102f30 2024-04-26T01:52:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.806+0000 7f710fe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7110111e40 con 0x7f711010c350 2024-04-26T01:52:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.806+0000 7f710fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7110112140 con 0x7f7110103850 2024-04-26T01:52:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.806+0000 7f710ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7110102f30 0x7f7110100b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.806+0000 7f710f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711010c350 0x7f7110101600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.806+0000 7f710ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7110102f30 0x7f7110100b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55240/0 (socket says 172.21.15.6:55240) 2024-04-26T01:52:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.806+0000 7f710ea00640 1 -- 172.21.15.6:0/224775723 learned_addr learned my addr 172.21.15.6:0/224775723 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.806+0000 7f7106000640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7110103850 0x7f71101010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.807+0000 7f710ea00640 1 -- 172.21.15.6:0/224775723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7110103850 msgr2=0x7f71101010c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.807+0000 7f710ea00640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7110103850 0x7f71101010c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.807+0000 7f710ea00640 1 -- 172.21.15.6:0/224775723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711010c350 msgr2=0x7f7110101600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.807+0000 7f710ea00640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711010c350 0x7f7110101600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.807+0000 7f710ea00640 1 -- 172.21.15.6:0/224775723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71101a6a60 con 0x7f7110102f30 2024-04-26T01:52:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.807+0000 7f710ea00640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7110102f30 0x7f7110100b80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7100005cc0 tx=0x7f710002f3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.807+0000 7f7107e00640 1 -- 172.21.15.6:0/224775723 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7100034040 con 0x7f7110102f30 2024-04-26T01:52:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.808+0000 7f7107e00640 1 -- 172.21.15.6:0/224775723 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71000308f0 con 0x7f7110102f30 2024-04-26T01:52:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.808+0000 7f710fe00640 1 -- 172.21.15.6:0/224775723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71101a6cf0 con 0x7f7110102f30 2024-04-26T01:52:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.808+0000 7f7107e00640 1 -- 172.21.15.6:0/224775723 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71000045b0 con 0x7f7110102f30 2024-04-26T01:52:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.808+0000 7f710fe00640 1 -- 172.21.15.6:0/224775723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71101a71d0 con 0x7f7110102f30 2024-04-26T01:52:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.812+0000 7f71183c3740 1 Processor -- start 2024-04-26T01:52:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.812+0000 7f71183c3740 1 -- start start 2024-04-26T01:52:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f71183c3740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d909b0 0x55cc07d90d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f71183c3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07d9b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f71183c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d9bb90 0x55cc07d9df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f71183c3740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55cc07cf6900 con 0x55cc07d909b0 2024-04-26T01:52:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f71183c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cc07c9cef0 con 0x55cc07d9bb90 2024-04-26T01:52:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f71183c3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55cc07cf28c0 con 0x55cc07d912d0 2024-04-26T01:52:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f7107e00640 1 -- 172.21.15.6:0/224775723 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f7100037020 con 0x7f7110102f30 2024-04-26T01:52:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f710f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d909b0 0x55cc07d90d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f7106000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d9bb90 0x55cc07d9df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f710f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d909b0 0x55cc07d90d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55248/0 (socket says 172.21.15.6:55248) 2024-04-26T01:52:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f710f400640 1 -- 172.21.15.6:0/3770218624 learned_addr learned my addr 172.21.15.6:0/3770218624 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f710ea00640 1 --2- 172.21.15.6:0/3770218624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07d9b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f7106000640 1 -- 172.21.15.6:0/3770218624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 msgr2=0x55cc07d9b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f7106000640 1 --2- 172.21.15.6:0/3770218624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07d9b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f7106000640 1 -- 172.21.15.6:0/3770218624 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d909b0 msgr2=0x55cc07d90d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.813+0000 7f7106000640 1 --2- 172.21.15.6:0/3770218624 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d909b0 0x55cc07d90d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.814+0000 7f7106000640 1 -- 172.21.15.6:0/3770218624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc07a09210 con 0x55cc07d9bb90 2024-04-26T01:52:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.814+0000 7f7106000640 1 --2- 172.21.15.6:0/3770218624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d9bb90 0x55cc07d9df80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009be0 tx=0x7f70f8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=148e7d555b5351f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.814+0000 7f70ff400640 1 -- 172.21.15.6:0/3770218624 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f8004050 con 0x55cc07d9bb90 2024-04-26T01:52:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.814+0000 7f70ff400640 1 -- 172.21.15.6:0/3770218624 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70f80054d0 con 0x55cc07d9bb90 2024-04-26T01:52:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.814+0000 7f70ff400640 1 -- 172.21.15.6:0/3770218624 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f8004ca0 con 0x55cc07d9bb90 2024-04-26T01:52:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.815+0000 7f7107e00640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f70f40a3d80 0x7f70f40a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.815+0000 7f71183c3740 1 -- 172.21.15.6:0/3770218624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d9bb90 msgr2=0x55cc07d9df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.815+0000 7f71183c3740 1 --2- 172.21.15.6:0/3770218624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d9bb90 0x55cc07d9df80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009be0 tx=0x7f70f8011810 comp rx=0 tx=0).stop 2024-04-26T01:52:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.815+0000 7f7107e00640 1 -- 172.21.15.6:0/224775723 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8644+0+0 (secure 0 0 0) 0x7f71000f14c0 con 0x7f7110102f30 2024-04-26T01:52:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.815+0000 7f710f400640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f70f40a3d80 0x7f70f40a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.815+0000 7f71183c3740 1 -- 172.21.15.6:0/3770218624 shutdown_connections 2024-04-26T01:52:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.815+0000 7f71183c3740 1 --2- 172.21.15.6:0/3770218624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d9bb90 0x55cc07d9df80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.815+0000 7f71183c3740 1 --2- 172.21.15.6:0/3770218624 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07d9b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.815+0000 7f71183c3740 1 --2- 172.21.15.6:0/3770218624 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d909b0 0x55cc07d90d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.815+0000 7f71183c3740 1 -- 172.21.15.6:0/3770218624 >> 172.21.15.6:0/3770218624 conn(0x55cc07d8cec0 msgr2=0x55cc07d8d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.815+0000 7f71183c3740 1 -- 172.21.15.6:0/3770218624 shutdown_connections 2024-04-26T01:52:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.815+0000 7f71183c3740 1 -- 172.21.15.6:0/3770218624 wait complete. 2024-04-26T01:52:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.816+0000 7f71183c3740 1 -- start start 2024-04-26T01:52:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.816+0000 7f710f400640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f70f40a3d80 0x7f70f40a6240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7108005b90 tx=0x7f7108005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.816+0000 7f71183c3740 1 Processor -- start 2024-04-26T01:52:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f71183c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 0x55cc07d98a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f71183c3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07d98fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f71183c3740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07d994f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f71183c3740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55cc07d9e7c0 con 0x55cc07d9bb90 2024-04-26T01:52:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f71183c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cc07cb0310 con 0x55cc07d909b0 2024-04-26T01:52:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f71183c3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55cc07d9e940 con 0x55cc07d912d0 2024-04-26T01:52:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f71183c3740 10 client.0 _ll_register_callbacks cb 0x55cc07c7ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f710ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 0x55cc07d98a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f710ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 0x55cc07d98a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:46758/0 (socket says 172.21.15.6:46758) 2024-04-26T01:52:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f710ea00640 1 -- 172.21.15.6:0/3772874081 learned_addr learned my addr 172.21.15.6:0/3772874081 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f710f400640 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07d994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f7106000640 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07d98fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f710ea00640 1 -- 172.21.15.6:0/3772874081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 msgr2=0x55cc07d98fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f710ea00640 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07d98fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f710ea00640 1 -- 172.21.15.6:0/3772874081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 msgr2=0x55cc07d994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f710ea00640 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07d994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.817+0000 7f710ea00640 1 -- 172.21.15.6:0/3772874081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc07ad5620 con 0x55cc07d909b0 2024-04-26T01:52:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.818+0000 7f710ea00640 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 0x55cc07d98a70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7100030040 tx=0x7f710002a950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=690b7930bc568d83 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.818+0000 7f70f3e00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:52:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.818+0000 7f70f3e00640 1 -- 172.21.15.6:0/3772874081 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71000bd490 con 0x55cc07d909b0 2024-04-26T01:52:08.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:08 smithi006 ceph-mon[28419]: pgmap v194: 97 pgs: 97 active+clean; 69 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.0 MiB/s wr, 63 op/s 2024-04-26T01:52:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.818+0000 7f70f3e00640 1 -- 172.21.15.6:0/3772874081 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8644+0+0 (secure 0 0 0) 0x7f71000f4aa0 con 0x55cc07d909b0 2024-04-26T01:52:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.818+0000 7f71183c3740 1 -- 172.21.15.6:0/3772874081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70f80054d0 con 0x55cc07d909b0 2024-04-26T01:52:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.818+0000 7f70f3e00640 1 -- 172.21.15.6:0/3772874081 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f71000d6070 con 0x55cc07d909b0 2024-04-26T01:52:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.818+0000 7f71183c3740 10 client.25895 fetch_fsmap learned FSMap version 37 2024-04-26T01:52:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.818+0000 7f71183c3740 1 -- 172.21.15.6:0/3772874081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cc07d9e7c0 con 0x55cc07d909b0 2024-04-26T01:52:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.819+0000 7f70f3e00640 1 -- 172.21.15.6:0/3772874081 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70f80054d0 con 0x55cc07d909b0 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.819+0000 7f70f3e00640 1 -- 172.21.15.6:0/3772874081 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f710004d030 con 0x55cc07d909b0 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.819+0000 7f71183c3740 10 client.25895 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.819+0000 7f71183c3740 10 client.25895 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.819+0000 7f71183c3740 20 client.25895 populate_metadata read hostname 'smithi006' 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.819+0000 7f71183c3740 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cc07dabe10 0x55cc07dae240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.819+0000 7f71183c3740 4 client.25895 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.819+0000 7f71183c3740 1 -- 172.21.15.6:0/3772874081 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc07d0a8b0 con 0x55cc07dabe10 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.820+0000 7f7106000640 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cc07dabe10 0x55cc07dae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.820+0000 7f7106000640 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cc07dabe10 0x55cc07dae240 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2916c6528845c43 server_cookie=a91a301b48327373 in_seq=0 out_seq=0 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.820+0000 7f70f3e00640 10 client.25895 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.821+0000 7f70f3e00640 1 -- 172.21.15.6:0/3772874081 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cc07d0a8b0 con 0x55cc07dabe10 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.821+0000 7f70f3e00640 10 client.25895 handle_command_reply: tid=0 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.821+0000 7f71183c3740 1 client.25895 shutdown 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.821+0000 7f71183c3740 1 -- 172.21.15.6:0/3772874081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 msgr2=0x55cc07d98a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.821+0000 7f71183c3740 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 0x55cc07d98a70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7100030040 tx=0x7f710002a950 comp rx=0 tx=0).stop 2024-04-26T01:52:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.822+0000 7f71183c3740 1 -- 172.21.15.6:0/3772874081 shutdown_connections 2024-04-26T01:52:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.822+0000 7f71183c3740 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cc07dabe10 0x55cc07dae240 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.822+0000 7f71183c3740 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07d994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.822+0000 7f71183c3740 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07d98fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.822+0000 7f71183c3740 1 --2- 172.21.15.6:0/3772874081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 0x55cc07d98a70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.822+0000 7f70f3e00640 0 client.25895 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.822+0000 7f71183c3740 1 -- 172.21.15.6:0/3772874081 >> 172.21.15.6:0/3772874081 conn(0x55cc07d8cec0 msgr2=0x55cc07d92c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.822+0000 7f71183c3740 1 -- 172.21.15.6:0/3772874081 shutdown_connections 2024-04-26T01:52:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.822+0000 7f71183c3740 1 -- 172.21.15.6:0/3772874081 wait complete. 2024-04-26T01:52:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.822+0000 7f71183c3740 20 client.25895 trim_cache size 0 max 16384 2024-04-26T01:52:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.856+0000 7f71183c3740 1 Processor -- start 2024-04-26T01:52:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.857+0000 7f71183c3740 1 -- start start 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.857+0000 7f71183c3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d909b0 0x55cc07da7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.857+0000 7f71183c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d912d0 0x55cc07d93370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.857+0000 7f71183c3740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07d8bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.857+0000 7f71183c3740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f71101b2570 con 0x55cc07d9bb90 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.857+0000 7f71183c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f71101afe40 con 0x55cc07d912d0 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.857+0000 7f71183c3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f71101adcb0 con 0x55cc07d909b0 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.858+0000 7f710f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07d8bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.858+0000 7f710ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d909b0 0x55cc07da7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.858+0000 7f710f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07d8bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55264/0 (socket says 172.21.15.6:55264) 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.858+0000 7f710f400640 1 -- 172.21.15.6:0/1139137676 learned_addr learned my addr 172.21.15.6:0/1139137676 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.858+0000 7f7106000640 1 --2- 172.21.15.6:0/1139137676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d912d0 0x55cc07d93370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.858+0000 7f710ea00640 1 -- 172.21.15.6:0/1139137676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d912d0 msgr2=0x55cc07d93370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.858+0000 7f710ea00640 1 --2- 172.21.15.6:0/1139137676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d912d0 0x55cc07d93370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.858+0000 7f710ea00640 1 -- 172.21.15.6:0/1139137676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 msgr2=0x55cc07d8bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.858+0000 7f710ea00640 1 --2- 172.21.15.6:0/1139137676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07d8bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.858+0000 7f710ea00640 1 -- 172.21.15.6:0/1139137676 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc07d938b0 con 0x55cc07d909b0 2024-04-26T01:52:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.858+0000 7f710ea00640 1 --2- 172.21.15.6:0/1139137676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d909b0 0x55cc07da7680 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f71000b9170 tx=0x7f71000bb9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=185e406fc6384b06 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.859+0000 7f70fea00640 1 -- 172.21.15.6:0/1139137676 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71000bcdf0 con 0x55cc07d909b0 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.859+0000 7f70fea00640 1 -- 172.21.15.6:0/1139137676 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71000c6100 con 0x55cc07d909b0 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.859+0000 7f70fea00640 1 -- 172.21.15.6:0/1139137676 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71000c5ed0 con 0x55cc07d909b0 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.860+0000 7f71183c3740 1 -- 172.21.15.6:0/1139137676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d909b0 msgr2=0x55cc07da7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.860+0000 7f71183c3740 1 --2- 172.21.15.6:0/1139137676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d909b0 0x55cc07da7680 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f71000b9170 tx=0x7f71000bb9a0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.860+0000 7f71183c3740 1 -- 172.21.15.6:0/1139137676 shutdown_connections 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.860+0000 7f71183c3740 1 --2- 172.21.15.6:0/1139137676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07d8bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.860+0000 7f71183c3740 1 --2- 172.21.15.6:0/1139137676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d912d0 0x55cc07d93370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.860+0000 7f71183c3740 1 --2- 172.21.15.6:0/1139137676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d909b0 0x55cc07da7680 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.860+0000 7f71183c3740 1 -- 172.21.15.6:0/1139137676 >> 172.21.15.6:0/1139137676 conn(0x55cc07d8cec0 msgr2=0x55cc07d8b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.860+0000 7f71183c3740 1 -- 172.21.15.6:0/1139137676 shutdown_connections 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.860+0000 7f71183c3740 1 -- 172.21.15.6:0/1139137676 wait complete. 2024-04-26T01:52:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.860+0000 7f71183c3740 1 -- start start 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f71183c3740 1 Processor -- start 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f71183c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 0x55cc07da3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f71183c3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07da3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f71183c3740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07da0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f71183c3740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f71101ae300 con 0x55cc07d9bb90 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f71183c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f71101af770 con 0x55cc07d909b0 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f71183c3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f71101ae9d0 con 0x55cc07d912d0 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f71183c3740 10 client.0 _ll_register_callbacks cb 0x7f71000b9800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f710f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07da0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f7106000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07da3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f710ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 0x55cc07da3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f7106000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07da3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:57326/0 (socket says 172.21.15.6:57326) 2024-04-26T01:52:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.861+0000 7f7106000640 1 -- 172.21.15.6:0/2453345778 learned_addr learned my addr 172.21.15.6:0/2453345778 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.862+0000 7f7106000640 1 -- 172.21.15.6:0/2453345778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 msgr2=0x55cc07da3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.862+0000 7f7106000640 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 0x55cc07da3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.862+0000 7f7106000640 1 -- 172.21.15.6:0/2453345778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 msgr2=0x55cc07da0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.862+0000 7f7106000640 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07da0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.862+0000 7f7106000640 1 -- 172.21.15.6:0/2453345778 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc07da1600 con 0x55cc07d912d0 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.862+0000 7f710f400640 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07da0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.862+0000 7f710ea00640 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 0x55cc07da3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.862+0000 7f7106000640 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07da3da0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009250 tx=0x7f70f8009790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c956a59489ca3970 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.862+0000 7f70f3e00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.862+0000 7f70f3e00640 1 -- 172.21.15.6:0/2453345778 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f8032020 con 0x55cc07d912d0 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.863+0000 7f71183c3740 1 -- 172.21.15.6:0/2453345778 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71000c6100 con 0x55cc07d912d0 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.863+0000 7f70f3e00640 1 -- 172.21.15.6:0/2453345778 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8644+0+0 (secure 0 0 0) 0x7f70f802edd0 con 0x55cc07d912d0 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.863+0000 7f70f3e00640 1 -- 172.21.15.6:0/2453345778 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f70f803d050 con 0x55cc07d912d0 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.863+0000 7f71183c3740 10 client.25948 fetch_fsmap learned FSMap version 37 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.863+0000 7f71183c3740 1 -- 172.21.15.6:0/2453345778 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cc07e11e20 con 0x55cc07d912d0 2024-04-26T01:52:08.864 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.863+0000 7f70f3e00640 1 -- 172.21.15.6:0/2453345778 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71000c6100 con 0x55cc07d912d0 2024-04-26T01:52:08.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.863+0000 7f70f3e00640 1 -- 172.21.15.6:0/2453345778 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f70f8004490 con 0x55cc07d912d0 2024-04-26T01:52:08.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.864+0000 7f71183c3740 10 client.25948 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:52:08.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.864+0000 7f71183c3740 10 client.25948 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:08.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.864+0000 7f71183c3740 20 client.25948 populate_metadata read hostname 'smithi006' 2024-04-26T01:52:08.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.864+0000 7f71183c3740 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cc07e12460 0x55cc07e14880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:08.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.864+0000 7f71183c3740 4 client.25948 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:08.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.864+0000 7f71183c3740 1 -- 172.21.15.6:0/2453345778 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cc07d04320 con 0x55cc07e12460 2024-04-26T01:52:08.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.864+0000 7f710ea00640 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cc07e12460 0x55cc07e14880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:08.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.864+0000 7f710ea00640 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cc07e12460 0x55cc07e14880 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=328c035dc232ca52 server_cookie=b576ea244e332361 in_seq=0 out_seq=0 2024-04-26T01:52:08.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.864+0000 7f70f3e00640 10 client.25948 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:08.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:08 smithi188 ceph-mon[32751]: pgmap v194: 97 pgs: 97 active+clean; 69 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.0 MiB/s wr, 63 op/s 2024-04-26T01:52:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.892+0000 7f70f3e00640 1 -- 172.21.15.6:0/2453345778 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cc07d04320 con 0x55cc07e12460 2024-04-26T01:52:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.892+0000 7f70f3e00640 10 client.25948 handle_command_reply: tid=0 2024-04-26T01:52:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.892+0000 7f71183c3740 1 client.25948 shutdown 2024-04-26T01:52:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.892+0000 7f71183c3740 1 -- 172.21.15.6:0/2453345778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 msgr2=0x55cc07da3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.892+0000 7f71183c3740 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07da3da0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009250 tx=0x7f70f8009790 comp rx=0 tx=0).stop 2024-04-26T01:52:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.893+0000 7f71183c3740 1 -- 172.21.15.6:0/2453345778 shutdown_connections 2024-04-26T01:52:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.893+0000 7f71183c3740 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cc07e12460 0x55cc07e14880 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.893+0000 7f71183c3740 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cc07d9bb90 0x55cc07da0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.893+0000 7f71183c3740 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cc07d912d0 0x55cc07da3da0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.893+0000 7f71183c3740 1 --2- 172.21.15.6:0/2453345778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc07d909b0 0x55cc07da3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.893+0000 7f71183c3740 1 -- 172.21.15.6:0/2453345778 >> 172.21.15.6:0/2453345778 conn(0x55cc07d8cec0 msgr2=0x55cc07d8c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.893+0000 7f70f3e00640 0 client.25948 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.893+0000 7f71183c3740 1 -- 172.21.15.6:0/2453345778 shutdown_connections 2024-04-26T01:52:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.893+0000 7f71183c3740 1 -- 172.21.15.6:0/2453345778 wait complete. 2024-04-26T01:52:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.893+0000 7f71183c3740 20 client.25948 trim_cache size 0 max 16384 2024-04-26T01:52:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:52:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:52:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:52:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:52:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.894+0000 7f710fe00640 1 -- 172.21.15.6:0/224775723 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f70f40a3d80 msgr2=0x7f70f40a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.894+0000 7f710fe00640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f70f40a3d80 0x7f70f40a6240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7108005b90 tx=0x7f7108005f90 comp rx=0 tx=0).stop 2024-04-26T01:52:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.894+0000 7f710fe00640 1 -- 172.21.15.6:0/224775723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7110102f30 msgr2=0x7f7110100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.894+0000 7f710fe00640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7110102f30 0x7f7110100b80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7100005cc0 tx=0x7f710002f3a0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.894+0000 7f710fe00640 1 -- 172.21.15.6:0/224775723 shutdown_connections 2024-04-26T01:52:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.894+0000 7f710fe00640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f70f40a3d80 0x7f70f40a6240 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.894+0000 7f710fe00640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711010c350 0x7f7110101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.894+0000 7f710fe00640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7110103850 0x7f71101010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.894+0000 7f710fe00640 1 --2- 172.21.15.6:0/224775723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7110102f30 0x7f7110100b80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.894+0000 7f710fe00640 1 -- 172.21.15.6:0/224775723 >> 172.21.15.6:0/224775723 conn(0x7f71100fc730 msgr2=0x7f71100fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.895+0000 7f710fe00640 1 -- 172.21.15.6:0/224775723 shutdown_connections 2024-04-26T01:52:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:08.895+0000 7f710fe00640 1 -- 172.21.15.6:0/224775723 wait complete. 2024-04-26T01:52:08.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:08 smithi029 ceph-mon[32775]: pgmap v194: 97 pgs: 97 active+clean; 69 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.0 MiB/s wr, 63 op/s 2024-04-26T01:52:08.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.980+0000 7efd9d800700 1 -- 172.21.15.6:0/756574188 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd9802df80 con 0x7efd981092d0 2024-04-26T01:52:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.982+0000 7efd8f400700 1 -- 172.21.15.6:0/756574188 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11853 (secure 0 0 0) 0x7efd8809c8e0 con 0x7efd981092d0 2024-04-26T01:52:08.981 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:52:08.981 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:51:30.200605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:52:08.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.985+0000 7efd9d800700 1 -- 172.21.15.6:0/756574188 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7efd7c0a3e30 msgr2=0x7efd7c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.985+0000 7efd9d800700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7efd7c0a3e30 0x7efd7c0a62f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd84009680 tx=0x7efd84009260 comp rx=0 tx=0).stop 2024-04-26T01:52:08.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.985+0000 7efd9d800700 1 -- 172.21.15.6:0/756574188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd981092d0 msgr2=0x7efd9819f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:08.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.985+0000 7efd9d800700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd981092d0 0x7efd9819f750 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efd8800e9a0 tx=0x7efd8800ecb0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.985+0000 7efd9d800700 1 -- 172.21.15.6:0/756574188 shutdown_connections 2024-04-26T01:52:08.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.985+0000 7efd9d800700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7efd7c0a3e30 0x7efd7c0a62f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.985+0000 7efd9d800700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9810e3b0 0x7efd981a3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.985+0000 7efd9d800700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd981092d0 0x7efd9819f750 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.985+0000 7efd9d800700 1 --2- 172.21.15.6:0/756574188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd98109c80 0x7efd9819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:08.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.985+0000 7efd9d800700 1 -- 172.21.15.6:0/756574188 >> 172.21.15.6:0/756574188 conn(0x7efd98076210 msgr2=0x7efd980767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:08.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.985+0000 7efd9d800700 1 -- 172.21.15.6:0/756574188 shutdown_connections 2024-04-26T01:52:08.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:08.985+0000 7efd9d800700 1 -- 172.21.15.6:0/756574188 wait complete. 2024-04-26T01:52:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 37 2024-04-26T01:52:09.585 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 4.00 MB/sec execute 153 sec latency 14410.731 ms 2024-04-26T01:52:09.718 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:09 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/756574188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:09.778 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id 3f39552a --release --await' 2024-04-26T01:52:09.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id 3f39552a --release --await 2024-04-26T01:52:09.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:09 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/756574188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:09.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:09 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/756574188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:10.232 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:52:10.586 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 3.98 MB/sec execute 154 sec latency 15410.826 ms 2024-04-26T01:52:10.722 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:10 smithi006 ceph-mon[28419]: pgmap v195: 97 pgs: 97 active+clean; 69 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.0 MiB/s wr, 63 op/s 2024-04-26T01:52:10.723 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:10 smithi006 ceph-mon[28419]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T01:52:10.842 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:10 smithi029 ceph-mon[32775]: pgmap v195: 97 pgs: 97 active+clean; 69 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.0 MiB/s wr, 63 op/s 2024-04-26T01:52:10.842 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:10 smithi029 ceph-mon[32775]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T01:52:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:10 smithi188 ceph-mon[32751]: pgmap v195: 97 pgs: 97 active+clean; 69 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.0 MiB/s wr, 63 op/s 2024-04-26T01:52:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:10 smithi188 ceph-mon[32751]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T01:52:11.585 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206637 3.95 MB/sec execute 155 sec latency 16410.956 ms 2024-04-26T01:52:12.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.415+0000 7fed66c00700 1 -- 172.21.15.6:0/2141742530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed60108a20 msgr2=0x7fed60108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.415+0000 7fed66c00700 1 --2- 172.21.15.6:0/2141742530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed60108a20 0x7fed60108e00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fed50009990 tx=0x7fed50009ca0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.415+0000 7fed66c00700 1 -- 172.21.15.6:0/2141742530 shutdown_connections 2024-04-26T01:52:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.415+0000 7fed66c00700 1 --2- 172.21.15.6:0/2141742530 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6006b980 0x7fed6010f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.415+0000 7fed66c00700 1 --2- 172.21.15.6:0/2141742530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed60108a20 0x7fed60108e00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.415+0000 7fed66c00700 1 --2- 172.21.15.6:0/2141742530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed6006afe0 0x7fed6006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.415+0000 7fed66c00700 1 -- 172.21.15.6:0/2141742530 >> 172.21.15.6:0/2141742530 conn(0x7fed600788b0 msgr2=0x7fed60078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.416+0000 7fed66c00700 1 -- 172.21.15.6:0/2141742530 shutdown_connections 2024-04-26T01:52:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.416+0000 7fed66c00700 1 -- 172.21.15.6:0/2141742530 wait complete. 2024-04-26T01:52:12.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.417+0000 7fed66c00700 1 Processor -- start 2024-04-26T01:52:12.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.417+0000 7fed66c00700 1 -- start start 2024-04-26T01:52:12.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.417+0000 7fed66c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6006afe0 0x7fed6019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.417+0000 7fed66c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6006b980 0x7fed6019b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.417+0000 7fed66c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed60108a20 0x7fed6019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed66c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fed60112bf0 con 0x7fed6006afe0 2024-04-26T01:52:12.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed66c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fed60112a70 con 0x7fed6006b980 2024-04-26T01:52:12.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed66c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fed60112d70 con 0x7fed60108a20 2024-04-26T01:52:12.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed5fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6006afe0 0x7fed6019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed5fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6006afe0 0x7fed6019b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55284/0 (socket says 172.21.15.6:55284) 2024-04-26T01:52:12.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed5fe00700 1 -- 172.21.15.6:0/226292787 learned_addr learned my addr 172.21.15.6:0/226292787 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:12.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed64c00700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed60108a20 0x7fed6019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed5f400700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6006b980 0x7fed6019b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed64c00700 1 -- 172.21.15.6:0/226292787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6006b980 msgr2=0x7fed6019b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:12.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed64c00700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6006b980 0x7fed6019b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed64c00700 1 -- 172.21.15.6:0/226292787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6006afe0 msgr2=0x7fed6019b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:12.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed64c00700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6006afe0 0x7fed6019b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed64c00700 1 -- 172.21.15.6:0/226292787 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed601a0350 con 0x7fed60108a20 2024-04-26T01:52:12.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.418+0000 7fed5fe00700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6006afe0 0x7fed6019b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:12.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.419+0000 7fed64c00700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed60108a20 0x7fed6019fc10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fed5800efc0 tx=0x7fed5800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:12.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.419+0000 7fed5cc00700 1 -- 172.21.15.6:0/226292787 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed580092e0 con 0x7fed60108a20 2024-04-26T01:52:12.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.419+0000 7fed5cc00700 1 -- 172.21.15.6:0/226292787 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed58004bb0 con 0x7fed60108a20 2024-04-26T01:52:12.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.419+0000 7fed66c00700 1 -- 172.21.15.6:0/226292787 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed601a05e0 con 0x7fed60108a20 2024-04-26T01:52:12.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.419+0000 7fed5cc00700 1 -- 172.21.15.6:0/226292787 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed58004080 con 0x7fed60108a20 2024-04-26T01:52:12.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.419+0000 7fed66c00700 1 -- 172.21.15.6:0/226292787 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed601a7ec0 con 0x7fed60108a20 2024-04-26T01:52:12.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.424+0000 7fed71619b80 1 Processor -- start 2024-04-26T01:52:12.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.425+0000 7fed5cc00700 1 -- 172.21.15.6:0/226292787 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fed5801b030 con 0x7fed60108a20 2024-04-26T01:52:12.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.425+0000 7fed71619b80 1 -- start start 2024-04-26T01:52:12.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.425+0000 7fed71619b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fc4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.425+0000 7fed71619b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 0x5558e3fcef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.425+0000 7fed71619b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fcf4d0 0x5558e3fd18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.425+0000 7fed71619b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5558e3f16f90 con 0x5558e3fc3d40 2024-04-26T01:52:12.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.425+0000 7fed71619b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5558e3ebd830 con 0x5558e3fcf4d0 2024-04-26T01:52:12.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.425+0000 7fed71619b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5558e3f12f50 con 0x5558e3fc4660 2024-04-26T01:52:12.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.426+0000 7fed5fe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 0x5558e3fcef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.426+0000 7fed5fe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 0x5558e3fcef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:57358/0 (socket says 172.21.15.6:57358) 2024-04-26T01:52:12.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.426+0000 7fed5fe00700 1 -- 172.21.15.6:0/3907967117 learned_addr learned my addr 172.21.15.6:0/3907967117 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:12.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.426+0000 7fed5f400700 1 --2- 172.21.15.6:0/3907967117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fc4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.426+0000 7fed5f400700 1 --2- 172.21.15.6:0/3907967117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fcf4d0 0x5558e3fd18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.426+0000 7fed5fe00700 1 -- 172.21.15.6:0/3907967117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fcf4d0 msgr2=0x5558e3fd18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:12.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.426+0000 7fed5fe00700 1 --2- 172.21.15.6:0/3907967117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fcf4d0 0x5558e3fd18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.426+0000 7fed5fe00700 1 -- 172.21.15.6:0/3907967117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 msgr2=0x5558e3fc4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:12.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.426+0000 7fed5fe00700 1 --2- 172.21.15.6:0/3907967117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fc4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.426+0000 7fed5fe00700 1 -- 172.21.15.6:0/3907967117 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558e3eb3780 con 0x5558e3fc4660 2024-04-26T01:52:12.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.427+0000 7fed5fe00700 1 --2- 172.21.15.6:0/3907967117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 0x5558e3fcef90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fed500096d0 tx=0x7fed50009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c1f5fa32f03b4db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:12.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.427+0000 7fed5cc00700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fed440a3db0 0x7fed440a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.427+0000 7fed5cc00700 1 -- 172.21.15.6:0/226292787 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7fed580d0350 con 0x7fed60108a20 2024-04-26T01:52:12.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.427+0000 7fed55600700 1 -- 172.21.15.6:0/3907967117 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50027070 con 0x5558e3fc4660 2024-04-26T01:52:12.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.427+0000 7fed55600700 1 -- 172.21.15.6:0/3907967117 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed50003680 con 0x5558e3fc4660 2024-04-26T01:52:12.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.427+0000 7fed55600700 1 -- 172.21.15.6:0/3907967117 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5000f6c0 con 0x5558e3fc4660 2024-04-26T01:52:12.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.428+0000 7fed71619b80 1 -- 172.21.15.6:0/3907967117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 msgr2=0x5558e3fcef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:12.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.428+0000 7fed71619b80 1 --2- 172.21.15.6:0/3907967117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 0x5558e3fcef90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fed500096d0 tx=0x7fed50009700 comp rx=0 tx=0).stop 2024-04-26T01:52:12.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.428+0000 7fed5f400700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fed440a3db0 0x7fed440a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.428+0000 7fed71619b80 1 -- 172.21.15.6:0/3907967117 shutdown_connections 2024-04-26T01:52:12.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.428+0000 7fed71619b80 1 --2- 172.21.15.6:0/3907967117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fcf4d0 0x5558e3fd18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.428+0000 7fed71619b80 1 --2- 172.21.15.6:0/3907967117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fc4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.428+0000 7fed71619b80 1 --2- 172.21.15.6:0/3907967117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 0x5558e3fcef90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.428+0000 7fed71619b80 1 -- 172.21.15.6:0/3907967117 >> 172.21.15.6:0/3907967117 conn(0x5558e3ef3050 msgr2=0x5558e3fc0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:12.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.429+0000 7fed71619b80 1 -- 172.21.15.6:0/3907967117 shutdown_connections 2024-04-26T01:52:12.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.429+0000 7fed71619b80 1 -- 172.21.15.6:0/3907967117 wait complete. 2024-04-26T01:52:12.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.429+0000 7fed5f400700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fed440a3db0 0x7fed440a6270 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x5558e3fc5710 tx=0x7fed4c0047c0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:12.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.429+0000 7fed71619b80 1 -- start start 2024-04-26T01:52:12.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.430+0000 7fed71619b80 1 Processor -- start 2024-04-26T01:52:12.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.430+0000 7fed71619b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fcebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.430+0000 7fed71619b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 0x5558e3fcbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.430+0000 7fed71619b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fcf4d0 0x5558e3fcc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.430+0000 7fed71619b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5558e3fd1f60 con 0x5558e3fc3d40 2024-04-26T01:52:12.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.430+0000 7fed71619b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5558e3ed0960 con 0x5558e3fcf4d0 2024-04-26T01:52:12.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.430+0000 7fed71619b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5558e3fd20e0 con 0x5558e3fc4660 2024-04-26T01:52:12.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.430+0000 7fed71619b80 10 client.0 _ll_register_callbacks cb 0x5558e3eaf820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:12.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.431+0000 7fed5f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 0x5558e3fcbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.431+0000 7fed64c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fcf4d0 0x5558e3fcc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.431+0000 7fed5f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 0x5558e3fcbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:57368/0 (socket says 172.21.15.6:57368) 2024-04-26T01:52:12.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.431+0000 7fed5f400700 1 -- 172.21.15.6:0/2270254148 learned_addr learned my addr 172.21.15.6:0/2270254148 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.431+0000 7fed5fe00700 1 --2- 172.21.15.6:0/2270254148 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fcebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.431+0000 7fed64c00700 1 -- 172.21.15.6:0/2270254148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 msgr2=0x5558e3fcbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.431+0000 7fed64c00700 1 --2- 172.21.15.6:0/2270254148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 0x5558e3fcbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.431+0000 7fed64c00700 1 -- 172.21.15.6:0/2270254148 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 msgr2=0x5558e3fcebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.431+0000 7fed64c00700 1 --2- 172.21.15.6:0/2270254148 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fcebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.431+0000 7fed64c00700 1 -- 172.21.15.6:0/2270254148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558e3e649d0 con 0x5558e3fcf4d0 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.431+0000 7fed64c00700 1 --2- 172.21.15.6:0/2270254148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fcf4d0 0x5558e3fcc430 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fed580cda50 tx=0x7fed5809bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20902c5fb06256ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.431+0000 7fed4a000700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.432+0000 7fed4a000700 1 -- 172.21.15.6:0/2270254148 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5809c330 con 0x5558e3fcf4d0 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.432+0000 7fed71619b80 1 -- 172.21.15.6:0/2270254148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed50003680 con 0x5558e3fcf4d0 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.432+0000 7fed4a000700 1 -- 172.21.15.6:0/2270254148 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7fed5809b780 con 0x5558e3fcf4d0 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.432+0000 7fed4a000700 1 -- 172.21.15.6:0/2270254148 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fed580a06e0 con 0x5558e3fcf4d0 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.432+0000 7fed71619b80 10 client.25913 fetch_fsmap learned FSMap version 37 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.432+0000 7fed71619b80 1 -- 172.21.15.6:0/2270254148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5558e3eadbf0 con 0x5558e3fcf4d0 2024-04-26T01:52:12.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.432+0000 7fed4a000700 1 -- 172.21.15.6:0/2270254148 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed50003680 con 0x5558e3fcf4d0 2024-04-26T01:52:12.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.432+0000 7fed4a000700 1 -- 172.21.15.6:0/2270254148 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fed580981a0 con 0x5558e3fcf4d0 2024-04-26T01:52:12.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.433+0000 7fed71619b80 10 client.25913 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:52:12.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.433+0000 7fed71619b80 10 client.25913 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:52:12.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.433+0000 7fed71619b80 20 client.25913 populate_metadata read hostname 'smithi006' 2024-04-26T01:52:12.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.433+0000 7fed71619b80 1 --2- 172.21.15.6:0/2270254148 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558e3fdebd0 0x5558e3fe1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.433+0000 7fed71619b80 4 client.25913 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:12.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.433+0000 7fed71619b80 1 -- 172.21.15.6:0/2270254148 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558e3f24780 con 0x5558e3fdebd0 2024-04-26T01:52:12.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.433+0000 7fed5fe00700 1 --2- 172.21.15.6:0/2270254148 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558e3fdebd0 0x5558e3fe1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.434+0000 7fed5fe00700 1 --2- 172.21.15.6:0/2270254148 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558e3fdebd0 0x5558e3fe1000 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc02741f5a2521c5 server_cookie=9330c6fd81efa04d in_seq=0 out_seq=0 2024-04-26T01:52:12.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.434+0000 7fed4a000700 10 client.25913 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:12.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed4a000700 1 -- 172.21.15.6:0/2270254148 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5558e3f24780 con 0x5558e3fdebd0 2024-04-26T01:52:12.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed4a000700 10 client.25913 handle_command_reply: tid=0 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed71619b80 1 client.25913 shutdown 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed71619b80 1 -- 172.21.15.6:0/2270254148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fcf4d0 msgr2=0x5558e3fcc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed71619b80 1 --2- 172.21.15.6:0/2270254148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fcf4d0 0x5558e3fcc430 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fed580cda50 tx=0x7fed5809bd10 comp rx=0 tx=0).stop 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed71619b80 1 -- 172.21.15.6:0/2270254148 shutdown_connections 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed71619b80 1 --2- 172.21.15.6:0/2270254148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fcf4d0 0x5558e3fcc430 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed71619b80 1 --2- 172.21.15.6:0/2270254148 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fcebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed71619b80 1 --2- 172.21.15.6:0/2270254148 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558e3fdebd0 0x5558e3fe1000 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed71619b80 1 --2- 172.21.15.6:0/2270254148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fc4660 0x5558e3fcbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed4a000700 0 client.25913 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed71619b80 1 -- 172.21.15.6:0/2270254148 >> 172.21.15.6:0/2270254148 conn(0x5558e3ef3050 msgr2=0x5558e3fbfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed71619b80 1 -- 172.21.15.6:0/2270254148 shutdown_connections 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.435+0000 7fed71619b80 1 -- 172.21.15.6:0/2270254148 wait complete. 2024-04-26T01:52:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.436+0000 7fed71619b80 20 client.25913 trim_cache size 0 max 16384 2024-04-26T01:52:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.458+0000 7fed71619b80 1 Processor -- start 2024-04-26T01:52:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.458+0000 7fed71619b80 1 -- start start 2024-04-26T01:52:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.458+0000 7fed71619b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fc3d40 0x5558e3fdae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed71619b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc4660 0x5558e3fc6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed71619b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fcf4d0 0x5558e3fc7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed71619b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fed601b1e80 con 0x5558e3fc4660 2024-04-26T01:52:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed71619b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fed601af710 con 0x5558e3fc3d40 2024-04-26T01:52:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed71619b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fed601ad580 con 0x5558e3fcf4d0 2024-04-26T01:52:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed64c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fcf4d0 0x5558e3fc7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed5fe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fc3d40 0x5558e3fdae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed64c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fcf4d0 0x5558e3fc7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:57382/0 (socket says 172.21.15.6:57382) 2024-04-26T01:52:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed64c00700 1 -- 172.21.15.6:0/4210939133 learned_addr learned my addr 172.21.15.6:0/4210939133 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed5f400700 1 --2- 172.21.15.6:0/4210939133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc4660 0x5558e3fc6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed64c00700 1 -- 172.21.15.6:0/4210939133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fc3d40 msgr2=0x5558e3fdae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed64c00700 1 --2- 172.21.15.6:0/4210939133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fc3d40 0x5558e3fdae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed64c00700 1 -- 172.21.15.6:0/4210939133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc4660 msgr2=0x5558e3fc6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.459+0000 7fed64c00700 1 --2- 172.21.15.6:0/4210939133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc4660 0x5558e3fc6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.460+0000 7fed64c00700 1 -- 172.21.15.6:0/4210939133 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558e3fc5bb0 con 0x5558e3fcf4d0 2024-04-26T01:52:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.460+0000 7fed5fe00700 1 --2- 172.21.15.6:0/4210939133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fc3d40 0x5558e3fdae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.460+0000 7fed64c00700 1 --2- 172.21.15.6:0/4210939133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fcf4d0 0x5558e3fc7330 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fed58098d00 tx=0x7fed5801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54090e876d0ee677 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.460+0000 7fed54c00700 1 -- 172.21.15.6:0/4210939133 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5809c330 con 0x5558e3fcf4d0 2024-04-26T01:52:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.460+0000 7fed54c00700 1 -- 172.21.15.6:0/4210939133 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed58098080 con 0x5558e3fcf4d0 2024-04-26T01:52:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.460+0000 7fed54c00700 1 -- 172.21.15.6:0/4210939133 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed580a03a0 con 0x5558e3fcf4d0 2024-04-26T01:52:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.461+0000 7fed71619b80 1 -- 172.21.15.6:0/4210939133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fcf4d0 msgr2=0x5558e3fc7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.461+0000 7fed71619b80 1 --2- 172.21.15.6:0/4210939133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fcf4d0 0x5558e3fc7330 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fed58098d00 tx=0x7fed5801c000 comp rx=0 tx=0).stop 2024-04-26T01:52:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.461+0000 7fed71619b80 1 -- 172.21.15.6:0/4210939133 shutdown_connections 2024-04-26T01:52:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.461+0000 7fed71619b80 1 --2- 172.21.15.6:0/4210939133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fcf4d0 0x5558e3fc7330 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.461+0000 7fed71619b80 1 --2- 172.21.15.6:0/4210939133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fc3d40 0x5558e3fdae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.461+0000 7fed71619b80 1 --2- 172.21.15.6:0/4210939133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc4660 0x5558e3fc6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.461+0000 7fed71619b80 1 -- 172.21.15.6:0/4210939133 >> 172.21.15.6:0/4210939133 conn(0x5558e3ef3050 msgr2=0x5558e3fbfc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.461+0000 7fed71619b80 1 -- 172.21.15.6:0/4210939133 shutdown_connections 2024-04-26T01:52:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.461+0000 7fed71619b80 1 -- 172.21.15.6:0/4210939133 wait complete. 2024-04-26T01:52:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.462+0000 7fed71619b80 1 -- start start 2024-04-26T01:52:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.462+0000 7fed71619b80 1 Processor -- start 2024-04-26T01:52:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed71619b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fd6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed71619b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fc4660 0x5558e3fd7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed71619b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fcf4d0 0x5558e3fd44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed71619b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fed601adbd0 con 0x5558e3fc3d40 2024-04-26T01:52:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed71619b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fed601af040 con 0x5558e3fc4660 2024-04-26T01:52:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed71619b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fed601ae2a0 con 0x5558e3fcf4d0 2024-04-26T01:52:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed71619b80 10 client.0 _ll_register_callbacks cb 0x5558e3eaf820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed5f400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fc4660 0x5558e3fd7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed5fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fd6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed5f400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fc4660 0x5558e3fd7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:46842/0 (socket says 172.21.15.6:46842) 2024-04-26T01:52:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed5f400700 1 -- 172.21.15.6:0/2816892077 learned_addr learned my addr 172.21.15.6:0/2816892077 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed5fe00700 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fd6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55316/0 (socket says 172.21.15.6:55316) 2024-04-26T01:52:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.463+0000 7fed64c00700 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fcf4d0 0x5558e3fd44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.464+0000 7fed5fe00700 1 -- 172.21.15.6:0/2816892077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fcf4d0 msgr2=0x5558e3fd44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.464+0000 7fed5fe00700 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fcf4d0 0x5558e3fd44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.464+0000 7fed5fe00700 1 -- 172.21.15.6:0/2816892077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fc4660 msgr2=0x5558e3fd7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:12.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.464+0000 7fed5fe00700 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fc4660 0x5558e3fd7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.464+0000 7fed5fe00700 1 -- 172.21.15.6:0/2816892077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558e3fd4c70 con 0x5558e3fc3d40 2024-04-26T01:52:12.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.464+0000 7fed5fe00700 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fd6fc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fed50005a80 tx=0x7fed50009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f103cf6c9e6731a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:12.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.464+0000 7fed4a000700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:52:12.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.464+0000 7fed4a000700 1 -- 172.21.15.6:0/2816892077 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50027040 con 0x5558e3fc3d40 2024-04-26T01:52:12.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.464+0000 7fed4a000700 1 -- 172.21.15.6:0/2816892077 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7fed50003a50 con 0x5558e3fc3d40 2024-04-26T01:52:12.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.464+0000 7fed71619b80 1 -- 172.21.15.6:0/2816892077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed58098080 con 0x5558e3fc3d40 2024-04-26T01:52:12.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.465+0000 7fed4a000700 1 -- 172.21.15.6:0/2816892077 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fed5003e050 con 0x5558e3fc3d40 2024-04-26T01:52:12.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.465+0000 7fed71619b80 10 client.15789 fetch_fsmap learned FSMap version 37 2024-04-26T01:52:12.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.465+0000 7fed71619b80 1 -- 172.21.15.6:0/2816892077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5558e4046c90 con 0x5558e3fc3d40 2024-04-26T01:52:12.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.465+0000 7fed4a000700 1 -- 172.21.15.6:0/2816892077 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed58098080 con 0x5558e3fc3d40 2024-04-26T01:52:12.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.465+0000 7fed4a000700 1 -- 172.21.15.6:0/2816892077 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fed50021030 con 0x5558e3fc3d40 2024-04-26T01:52:12.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.465+0000 7fed71619b80 10 client.15789 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:52:12.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.465+0000 7fed71619b80 10 client.15789 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:52:12.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.465+0000 7fed71619b80 20 client.15789 populate_metadata read hostname 'smithi006' 2024-04-26T01:52:12.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.465+0000 7fed71619b80 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558e4046ec0 0x5558e40492b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:12.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.465+0000 7fed71619b80 4 client.15789 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3f39552a", "release": true, "await": true}] 2024-04-26T01:52:12.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.465+0000 7fed71619b80 1 -- 172.21.15.6:0/2816892077 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3f39552a", "release": true, "await": true}) -- 0x5558e3f1e820 con 0x5558e4046ec0 2024-04-26T01:52:12.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.466+0000 7fed5f400700 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558e4046ec0 0x5558e40492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:12.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.466+0000 7fed5f400700 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558e4046ec0 0x5558e40492b0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f37cb8d9b4a8c8e9 server_cookie=9afc279555275c3c in_seq=0 out_seq=0 2024-04-26T01:52:12.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.466+0000 7fed4a000700 10 client.15789 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:12.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.471+0000 7fed4a000700 1 -- 172.21.15.6:0/2816892077 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5558e3f1e820 con 0x5558e4046ec0 2024-04-26T01:52:12.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.471+0000 7fed4a000700 10 client.15789 handle_command_reply: tid=0 2024-04-26T01:52:12.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.471+0000 7fed71619b80 1 client.15789 shutdown 2024-04-26T01:52:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.471+0000 7fed71619b80 1 -- 172.21.15.6:0/2816892077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 msgr2=0x5558e3fd6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.471+0000 7fed71619b80 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fd6fc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fed50005a80 tx=0x7fed50009310 comp rx=0 tx=0).stop 2024-04-26T01:52:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.472+0000 7fed71619b80 1 -- 172.21.15.6:0/2816892077 shutdown_connections 2024-04-26T01:52:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.472+0000 7fed71619b80 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558e4046ec0 0x5558e40492b0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.472+0000 7fed71619b80 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558e3fcf4d0 0x5558e3fd44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.472+0000 7fed71619b80 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558e3fc3d40 0x5558e3fd6fc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.472+0000 7fed71619b80 1 --2- 172.21.15.6:0/2816892077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558e3fc4660 0x5558e3fd7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.472+0000 7fed71619b80 1 -- 172.21.15.6:0/2816892077 >> 172.21.15.6:0/2816892077 conn(0x5558e3ef3050 msgr2=0x5558e3fd9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.472+0000 7fed4a000700 0 client.15789 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.472+0000 7fed71619b80 1 -- 172.21.15.6:0/2816892077 shutdown_connections 2024-04-26T01:52:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.472+0000 7fed71619b80 1 -- 172.21.15.6:0/2816892077 wait complete. 2024-04-26T01:52:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.472+0000 7fed71619b80 20 client.15789 trim_cache size 0 max 16384 2024-04-26T01:52:12.472 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:52:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 37, 2024-04-26T01:52:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:52:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 16, 2024-04-26T01:52:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:52:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "3f39552a": { 2024-04-26T01:52:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 16, 2024-04-26T01:52:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:52:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:52:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:52:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 70.0, 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:52:12.473 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:52:12.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.473+0000 7fed66c00700 1 -- 172.21.15.6:0/226292787 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fed440a3db0 msgr2=0x7fed440a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:12.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.473+0000 7fed66c00700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fed440a3db0 0x7fed440a6270 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x5558e3fc5710 tx=0x7fed4c0047c0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.473+0000 7fed66c00700 1 -- 172.21.15.6:0/226292787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed60108a20 msgr2=0x7fed6019fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:12.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.473+0000 7fed66c00700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed60108a20 0x7fed6019fc10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fed5800efc0 tx=0x7fed5800c3a0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.473+0000 7fed66c00700 1 -- 172.21.15.6:0/226292787 shutdown_connections 2024-04-26T01:52:12.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.473+0000 7fed66c00700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fed440a3db0 0x7fed440a6270 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.473+0000 7fed66c00700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed60108a20 0x7fed6019fc10 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.473+0000 7fed66c00700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6006afe0 0x7fed6019b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.473+0000 7fed66c00700 1 --2- 172.21.15.6:0/226292787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6006b980 0x7fed6019b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:12.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.473+0000 7fed66c00700 1 -- 172.21.15.6:0/226292787 >> 172.21.15.6:0/226292787 conn(0x7fed600788b0 msgr2=0x7fed6010dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:12.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.473+0000 7fed66c00700 1 -- 172.21.15.6:0/226292787 shutdown_connections 2024-04-26T01:52:12.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:12.473+0000 7fed66c00700 1 -- 172.21.15.6:0/226292787 wait complete. 2024-04-26T01:52:12.586 INFO:tasks.workunit.client.0.smithi006.stdout: 1 206749 3.93 MB/sec execute 156 sec latency 17302.270 ms 2024-04-26T01:52:12.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:12 smithi006 ceph-mon[28419]: pgmap v197: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 1.1 MiB/s wr, 58 op/s 2024-04-26T01:52:12.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:12 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:52:12.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:12 smithi188 ceph-mon[32751]: pgmap v197: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 1.1 MiB/s wr, 58 op/s 2024-04-26T01:52:12.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:12 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:52:12.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:12 smithi029 ceph-mon[32775]: pgmap v197: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 1.1 MiB/s wr, 58 op/s 2024-04-26T01:52:12.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:12 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:52:13.468 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '3f39552a', seconds elapsed: 25.2 2024-04-26T01:52:13.585 INFO:tasks.workunit.client.0.smithi006.stdout: 1 207933 3.93 MB/sec execute 157 sec latency 50.223 ms 2024-04-26T01:52:13.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:13 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:52:13.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:13 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:52:13.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:13 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:52:13.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:13 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:52:13.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:13 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:52:13.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:13 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:52:13.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:13 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:52:13.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:13 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:52:13.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:13 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:52:14.586 INFO:tasks.workunit.client.0.smithi006.stdout: 1 209346 3.98 MB/sec execute 158 sec latency 19.619 ms 2024-04-26T01:52:14.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:14 smithi006 ceph-mon[28419]: pgmap v198: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 1.0 MiB/s wr, 54 op/s 2024-04-26T01:52:14.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:14 smithi188 ceph-mon[32751]: pgmap v198: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 1.0 MiB/s wr, 54 op/s 2024-04-26T01:52:14.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:14 smithi029 ceph-mon[32775]: pgmap v198: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 1.0 MiB/s wr, 54 op/s 2024-04-26T01:52:15.586 INFO:tasks.workunit.client.0.smithi006.stdout: 1 209872 3.97 MB/sec execute 159 sec latency 37.813 ms 2024-04-26T01:52:15.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:15 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:15.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:15 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:15.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:15 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:16.586 INFO:tasks.workunit.client.0.smithi006.stdout: 1 210797 3.97 MB/sec execute 160 sec latency 32.744 ms 2024-04-26T01:52:16.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:16 smithi006 ceph-mon[28419]: pgmap v199: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 922 KiB/s wr, 46 op/s 2024-04-26T01:52:16.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:16 smithi188 ceph-mon[32751]: pgmap v199: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 922 KiB/s wr, 46 op/s 2024-04-26T01:52:16.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:16 smithi029 ceph-mon[32775]: pgmap v199: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 922 KiB/s wr, 46 op/s 2024-04-26T01:52:17.587 INFO:tasks.workunit.client.0.smithi006.stdout: 1 212037 3.99 MB/sec execute 161 sec latency 37.220 ms 2024-04-26T01:52:18.587 INFO:tasks.workunit.client.0.smithi006.stdout: 1 213138 4.01 MB/sec execute 162 sec latency 19.447 ms 2024-04-26T01:52:18.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:18 smithi006 ceph-mon[28419]: pgmap v200: 97 pgs: 97 active+clean; 71 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 93 op/s 2024-04-26T01:52:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:18 smithi188 ceph-mon[32751]: pgmap v200: 97 pgs: 97 active+clean; 71 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 93 op/s 2024-04-26T01:52:18.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:18 smithi029 ceph-mon[32775]: pgmap v200: 97 pgs: 97 active+clean; 71 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 93 op/s 2024-04-26T01:52:19.587 INFO:tasks.workunit.client.0.smithi006.stdout: 1 213619 3.99 MB/sec execute 163 sec latency 209.036 ms 2024-04-26T01:52:20.587 INFO:tasks.workunit.client.0.smithi006.stdout: 1 214761 4.00 MB/sec execute 164 sec latency 24.826 ms 2024-04-26T01:52:20.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:20 smithi006 ceph-mon[28419]: pgmap v201: 97 pgs: 97 active+clean; 71 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 93 op/s 2024-04-26T01:52:20.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:20 smithi188 ceph-mon[32751]: pgmap v201: 97 pgs: 97 active+clean; 71 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 93 op/s 2024-04-26T01:52:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:20 smithi029 ceph-mon[32775]: pgmap v201: 97 pgs: 97 active+clean; 71 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.3 MiB/s wr, 93 op/s 2024-04-26T01:52:21.587 INFO:tasks.workunit.client.0.smithi006.stdout: 1 216558 4.05 MB/sec execute 165 sec latency 20.261 ms 2024-04-26T01:52:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.049+0000 7f652e200640 1 -- 172.21.15.6:0/3438548859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65281021b0 msgr2=0x7f6528102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.049+0000 7f652e200640 1 --2- 172.21.15.6:0/3438548859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65281021b0 0x7f6528102590 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6510009ca0 tx=0x7f651002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.050+0000 7f652e200640 1 -- 172.21.15.6:0/3438548859 shutdown_connections 2024-04-26T01:52:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.050+0000 7f652e200640 1 --2- 172.21.15.6:0/3438548859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6528102b60 0x7f6528106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.050+0000 7f652e200640 1 --2- 172.21.15.6:0/3438548859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6528107280 0x7f652810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.050+0000 7f652e200640 1 --2- 172.21.15.6:0/3438548859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65281021b0 0x7f6528102590 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.050+0000 7f652e200640 1 -- 172.21.15.6:0/3438548859 >> 172.21.15.6:0/3438548859 conn(0x7f65280fda10 msgr2=0x7f65280ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.050+0000 7f652e200640 1 -- 172.21.15.6:0/3438548859 shutdown_connections 2024-04-26T01:52:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.050+0000 7f652e200640 1 -- 172.21.15.6:0/3438548859 wait complete. 2024-04-26T01:52:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.051+0000 7f652e200640 1 Processor -- start 2024-04-26T01:52:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.051+0000 7f652e200640 1 -- start start 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.051+0000 7f652e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65281021b0 0x7f65281a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.051+0000 7f652e200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6528107280 0x7f65281a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f652e200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652819a840 0x7f652819acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f652e200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6528112240 con 0x7f65281021b0 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f652e200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f65281120c0 con 0x7f6528107280 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f652e200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f65281123c0 con 0x7f652819a840 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f6527400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65281021b0 0x7f65281a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f6527e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652819a840 0x7f652819acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f6527e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652819a840 0x7f652819acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:35646/0 (socket says 172.21.15.6:35646) 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f6527e00640 1 -- 172.21.15.6:0/4100801210 learned_addr learned my addr 172.21.15.6:0/4100801210 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f6526a00640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6528107280 0x7f65281a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f6527e00640 1 -- 172.21.15.6:0/4100801210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6528107280 msgr2=0x7f65281a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f6527e00640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6528107280 0x7f65281a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f6527e00640 1 -- 172.21.15.6:0/4100801210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65281021b0 msgr2=0x7f65281a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f6527e00640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65281021b0 0x7f65281a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.052+0000 7f6527e00640 1 -- 172.21.15.6:0/4100801210 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652819b3d0 con 0x7f652819a840 2024-04-26T01:52:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.053+0000 7f6527e00640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652819a840 0x7f652819acf0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f652000e970 tx=0x7f652000ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.053+0000 7f651fe00640 1 -- 172.21.15.6:0/4100801210 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652000ccc0 con 0x7f652819a840 2024-04-26T01:52:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.053+0000 7f651fe00640 1 -- 172.21.15.6:0/4100801210 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f652000ce60 con 0x7f652819a840 2024-04-26T01:52:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.053+0000 7f652e200640 1 -- 172.21.15.6:0/4100801210 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f652819b6c0 con 0x7f652819a840 2024-04-26T01:52:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.053+0000 7f651fe00640 1 -- 172.21.15.6:0/4100801210 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520010430 con 0x7f652819a840 2024-04-26T01:52:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.053+0000 7f652e200640 1 -- 172.21.15.6:0/4100801210 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65281a78f0 con 0x7f652819a840 2024-04-26T01:52:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.056+0000 7f651fe00640 1 -- 172.21.15.6:0/4100801210 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f65200046c0 con 0x7f652819a840 2024-04-26T01:52:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.058+0000 7f651fe00640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f65080a3d90 0x7f65080a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.058+0000 7f651fe00640 1 -- 172.21.15.6:0/4100801210 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7f65200d0ae0 con 0x7f652819a840 2024-04-26T01:52:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.058+0000 7f6530fbf740 1 Processor -- start 2024-04-26T01:52:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.058+0000 7f6527400640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f65080a3d90 0x7f65080a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.058+0000 7f6530fbf740 1 -- start start 2024-04-26T01:52:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.059+0000 7f6530fbf740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539be820 0x5565539bec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.059+0000 7f6530fbf740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539bf140 0x5565539c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.059+0000 7f6530fbf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539c9a00 0x5565539cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.059+0000 7f6530fbf740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x556553924900 con 0x5565539be820 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.059+0000 7f6530fbf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5565538caef0 con 0x5565539c9a00 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.059+0000 7f6530fbf740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5565539208c0 con 0x5565539bf140 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.059+0000 7f6527400640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f65080a3d90 0x7f65080a6250 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6510006010 tx=0x7f6510009820 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.059+0000 7f6526a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539bf140 0x5565539c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.059+0000 7f6527e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539c9a00 0x5565539cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.059+0000 7f6526a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539bf140 0x5565539c94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:35654/0 (socket says 172.21.15.6:35654) 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.059+0000 7f6526a00640 1 -- 172.21.15.6:0/1504280706 learned_addr learned my addr 172.21.15.6:0/1504280706 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.059+0000 7f6527400640 1 --2- 172.21.15.6:0/1504280706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539be820 0x5565539bec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.060+0000 7f6527e00640 1 -- 172.21.15.6:0/1504280706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539bf140 msgr2=0x5565539c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.060+0000 7f6527e00640 1 --2- 172.21.15.6:0/1504280706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539bf140 0x5565539c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.060+0000 7f6527e00640 1 -- 172.21.15.6:0/1504280706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539be820 msgr2=0x5565539bec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.060+0000 7f6527e00640 1 --2- 172.21.15.6:0/1504280706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539be820 0x5565539bec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.060+0000 7f6527e00640 1 -- 172.21.15.6:0/1504280706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556553637210 con 0x5565539c9a00 2024-04-26T01:52:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.060+0000 7f6527400640 1 --2- 172.21.15.6:0/1504280706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539be820 0x5565539bec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.060+0000 7f6526a00640 1 --2- 172.21.15.6:0/1504280706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539bf140 0x5565539c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.060+0000 7f6527e00640 1 --2- 172.21.15.6:0/1504280706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539c9a00 0x5565539cbdf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f65200ce180 tx=0x7f652009c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3039b3e977bf32c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.060+0000 7f6517e00640 1 -- 172.21.15.6:0/1504280706 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65200a0d80 con 0x5565539c9a00 2024-04-26T01:52:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.060+0000 7f6517e00640 1 -- 172.21.15.6:0/1504280706 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65200a5640 con 0x5565539c9a00 2024-04-26T01:52:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.060+0000 7f6517e00640 1 -- 172.21.15.6:0/1504280706 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65200bbb90 con 0x5565539c9a00 2024-04-26T01:52:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.061+0000 7f6530fbf740 1 -- 172.21.15.6:0/1504280706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539c9a00 msgr2=0x5565539cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.061+0000 7f6530fbf740 1 --2- 172.21.15.6:0/1504280706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539c9a00 0x5565539cbdf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f65200ce180 tx=0x7f652009c3f0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.061+0000 7f6530fbf740 1 -- 172.21.15.6:0/1504280706 shutdown_connections 2024-04-26T01:52:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.061+0000 7f6530fbf740 1 --2- 172.21.15.6:0/1504280706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539c9a00 0x5565539cbdf0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.061+0000 7f6530fbf740 1 --2- 172.21.15.6:0/1504280706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539bf140 0x5565539c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.061+0000 7f6530fbf740 1 --2- 172.21.15.6:0/1504280706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539be820 0x5565539bec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.061+0000 7f6530fbf740 1 -- 172.21.15.6:0/1504280706 >> 172.21.15.6:0/1504280706 conn(0x5565539bad30 msgr2=0x5565539bb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.061+0000 7f6530fbf740 1 -- 172.21.15.6:0/1504280706 shutdown_connections 2024-04-26T01:52:22.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.061+0000 7f6530fbf740 1 -- 172.21.15.6:0/1504280706 wait complete. 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.062+0000 7f6530fbf740 1 -- start start 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.062+0000 7f6530fbf740 1 Processor -- start 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.063+0000 7f6530fbf740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539be820 0x5565539c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.063+0000 7f6530fbf740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.063+0000 7f6530fbf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539c9a00 0x5565539c7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.063+0000 7f6530fbf740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5565539cc630 con 0x5565539bf140 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.063+0000 7f6530fbf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5565538de310 con 0x5565539c9a00 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.063+0000 7f6530fbf740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5565539cc7b0 con 0x5565539be820 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.063+0000 7f6530fbf740 10 client.0 _ll_register_callbacks cb 0x5565538accb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.063+0000 7f6527400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539be820 0x5565539c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.063+0000 7f6527400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539be820 0x5565539c69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:35668/0 (socket says 172.21.15.6:35668) 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.063+0000 7f6527400640 1 -- 172.21.15.6:0/418377541 learned_addr learned my addr 172.21.15.6:0/418377541 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.063+0000 7f6527e00640 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539c9a00 0x5565539c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.063+0000 7f6526a00640 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6527e00640 1 -- 172.21.15.6:0/418377541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539be820 msgr2=0x5565539c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6527e00640 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539be820 0x5565539c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6527e00640 1 -- 172.21.15.6:0/418377541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 msgr2=0x5565539c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6527e00640 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6527e00640 1 -- 172.21.15.6:0/418377541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556553703620 con 0x5565539c9a00 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6526a00640 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6527400640 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539be820 0x5565539c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6527e00640 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539c9a00 0x5565539c7470 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x5565539bbec0 tx=0x7f6520000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=785874ae0d4887fc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6514c00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6514c00640 1 -- 172.21.15.6:0/418377541 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65200a5430 con 0x5565539c9a00 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6514c00640 1 -- 172.21.15.6:0/418377541 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7f65200bbb90 con 0x5565539c9a00 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6530fbf740 1 -- 172.21.15.6:0/418377541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65200a5640 con 0x5565539c9a00 2024-04-26T01:52:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6514c00640 1 -- 172.21.15.6:0/418377541 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f65200bbd90 con 0x5565539c9a00 2024-04-26T01:52:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.064+0000 7f6530fbf740 10 client.25931 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.065+0000 7f6530fbf740 1 -- 172.21.15.6:0/418377541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5565539d88f0 con 0x5565539c9a00 2024-04-26T01:52:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.065+0000 7f6514c00640 1 -- 172.21.15.6:0/418377541 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65200a5640 con 0x5565539c9a00 2024-04-26T01:52:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.065+0000 7f6514c00640 1 -- 172.21.15.6:0/418377541 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f6520098890 con 0x5565539c9a00 2024-04-26T01:52:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.065+0000 7f6530fbf740 10 client.25931 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.065+0000 7f6530fbf740 10 client.25931 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.065+0000 7f6530fbf740 20 client.25931 populate_metadata read hostname 'smithi006' 2024-04-26T01:52:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.065+0000 7f6530fbf740 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5565539d9c40 0x5565539dc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.065+0000 7f6530fbf740 4 client.25931 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.065+0000 7f6530fbf740 1 -- 172.21.15.6:0/418377541 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565539388b0 con 0x5565539d9c40 2024-04-26T01:52:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.065+0000 7f6527400640 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5565539d9c40 0x5565539dc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.066+0000 7f6527400640 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5565539d9c40 0x5565539dc030 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b99d868c436595b3 server_cookie=2dd2e02da7acc7f4 in_seq=0 out_seq=0 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.066+0000 7f6514c00640 10 client.25931 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6514c00640 1 -- 172.21.15.6:0/418377541 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565539388b0 con 0x5565539d9c40 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6514c00640 10 client.25931 handle_command_reply: tid=0 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6530fbf740 1 client.25931 shutdown 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6530fbf740 1 -- 172.21.15.6:0/418377541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539c9a00 msgr2=0x5565539c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6530fbf740 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539c9a00 0x5565539c7470 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x5565539bbec0 tx=0x7f6520000bc0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6530fbf740 1 -- 172.21.15.6:0/418377541 shutdown_connections 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6530fbf740 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5565539d9c40 0x5565539dc030 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6530fbf740 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6530fbf740 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539c9a00 0x5565539c7470 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6530fbf740 1 --2- 172.21.15.6:0/418377541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539be820 0x5565539c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6514c00640 0 client.25931 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6530fbf740 1 -- 172.21.15.6:0/418377541 >> 172.21.15.6:0/418377541 conn(0x5565539bad30 msgr2=0x5565539c0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6530fbf740 1 -- 172.21.15.6:0/418377541 shutdown_connections 2024-04-26T01:52:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.067+0000 7f6530fbf740 1 -- 172.21.15.6:0/418377541 wait complete. 2024-04-26T01:52:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.068+0000 7f6530fbf740 20 client.25931 trim_cache size 0 max 16384 2024-04-26T01:52:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.085+0000 7f6530fbf740 1 Processor -- start 2024-04-26T01:52:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.085+0000 7f6530fbf740 1 -- start start 2024-04-26T01:52:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.085+0000 7f6530fbf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 0x5565539d5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.085+0000 7f6530fbf740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539c01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.085+0000 7f6530fbf740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539c9a00 0x5565539c0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.085+0000 7f6530fbf740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f65281b2870 con 0x5565539bf140 2024-04-26T01:52:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.085+0000 7f6530fbf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f65281b0140 con 0x5565539be820 2024-04-26T01:52:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.085+0000 7f6530fbf740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f65281adfb0 con 0x5565539c9a00 2024-04-26T01:52:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.085+0000 7f6527400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 0x5565539d5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.085+0000 7f6527400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 0x5565539d5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43782/0 (socket says 172.21.15.6:43782) 2024-04-26T01:52:22.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.086+0000 7f6527e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539c9a00 0x5565539c0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.086+0000 7f6527400640 1 -- 172.21.15.6:0/829345706 learned_addr learned my addr 172.21.15.6:0/829345706 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.086+0000 7f6526a00640 1 --2- 172.21.15.6:0/829345706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539c01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.086+0000 7f6527400640 1 -- 172.21.15.6:0/829345706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539c9a00 msgr2=0x5565539c0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.086+0000 7f6527400640 1 --2- 172.21.15.6:0/829345706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539c9a00 0x5565539c0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.086+0000 7f6527400640 1 -- 172.21.15.6:0/829345706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 msgr2=0x5565539c01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.086+0000 7f6527400640 1 --2- 172.21.15.6:0/829345706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539c01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.086+0000 7f6527400640 1 -- 172.21.15.6:0/829345706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565539bf8b0 con 0x5565539be820 2024-04-26T01:52:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.086+0000 7f6527400640 1 --2- 172.21.15.6:0/829345706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 0x5565539d5110 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f651002aeb0 tx=0x7f651003f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85af93f3fd78f534 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.086+0000 7f6517400640 1 -- 172.21.15.6:0/829345706 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651005a070 con 0x5565539be820 2024-04-26T01:52:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.087+0000 7f6517400640 1 -- 172.21.15.6:0/829345706 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6510003e40 con 0x5565539be820 2024-04-26T01:52:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.087+0000 7f6517400640 1 -- 172.21.15.6:0/829345706 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510004820 con 0x5565539be820 2024-04-26T01:52:22.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.087+0000 7f6530fbf740 1 -- 172.21.15.6:0/829345706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 msgr2=0x5565539d5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.087+0000 7f6530fbf740 1 --2- 172.21.15.6:0/829345706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 0x5565539d5110 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f651002aeb0 tx=0x7f651003f000 comp rx=0 tx=0).stop 2024-04-26T01:52:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.087+0000 7f6530fbf740 1 -- 172.21.15.6:0/829345706 shutdown_connections 2024-04-26T01:52:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.087+0000 7f6530fbf740 1 --2- 172.21.15.6:0/829345706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539c01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.087+0000 7f6530fbf740 1 --2- 172.21.15.6:0/829345706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539c9a00 0x5565539c0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.087+0000 7f6530fbf740 1 --2- 172.21.15.6:0/829345706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 0x5565539d5110 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.087+0000 7f6530fbf740 1 -- 172.21.15.6:0/829345706 >> 172.21.15.6:0/829345706 conn(0x5565539bad30 msgr2=0x5565539d4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.088+0000 7f6530fbf740 1 -- 172.21.15.6:0/829345706 shutdown_connections 2024-04-26T01:52:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.088+0000 7f6530fbf740 1 -- 172.21.15.6:0/829345706 wait complete. 2024-04-26T01:52:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.088+0000 7f6530fbf740 1 -- start start 2024-04-26T01:52:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.088+0000 7f6530fbf740 1 Processor -- start 2024-04-26T01:52:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6530fbf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 0x5565539ddc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6530fbf740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539de150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6530fbf740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539c9a00 0x5565539de690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6530fbf740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f65281ae600 con 0x5565539bf140 2024-04-26T01:52:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6530fbf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f65281afa70 con 0x5565539be820 2024-04-26T01:52:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6530fbf740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f65281aecd0 con 0x5565539c9a00 2024-04-26T01:52:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6530fbf740 10 client.0 _ll_register_callbacks cb 0x7f652009c510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6526a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539de150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6527e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539c9a00 0x5565539de690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6527e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539c9a00 0x5565539de690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:35692/0 (socket says 172.21.15.6:35692) 2024-04-26T01:52:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6527e00640 1 -- 172.21.15.6:0/271401855 learned_addr learned my addr 172.21.15.6:0/271401855 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6526a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539de150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42082/0 (socket says 172.21.15.6:42082) 2024-04-26T01:52:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6527400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 0x5565539ddc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6527400640 1 -- 172.21.15.6:0/271401855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539c9a00 msgr2=0x5565539de690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6527400640 1 --2- 172.21.15.6:0/271401855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539c9a00 0x5565539de690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6527400640 1 -- 172.21.15.6:0/271401855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 msgr2=0x5565539de150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6527400640 1 --2- 172.21.15.6:0/271401855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539de150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6527400640 1 -- 172.21.15.6:0/271401855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556553a3dfd0 con 0x5565539be820 2024-04-26T01:52:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.089+0000 7f6527e00640 1 --2- 172.21.15.6:0/271401855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539c9a00 0x5565539de690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.090+0000 7f6527400640 1 --2- 172.21.15.6:0/271401855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 0x5565539ddc10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f651002ade0 tx=0x7f6510031e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec8f3103277c78a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.090+0000 7f6514c00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:52:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.090+0000 7f6514c00640 1 -- 172.21.15.6:0/271401855 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651005a070 con 0x5565539be820 2024-04-26T01:52:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.090+0000 7f6514c00640 1 -- 172.21.15.6:0/271401855 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7f6510004720 con 0x5565539be820 2024-04-26T01:52:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.090+0000 7f6530fbf740 1 -- 172.21.15.6:0/271401855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556553932320 con 0x5565539be820 2024-04-26T01:52:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.090+0000 7f6514c00640 1 -- 172.21.15.6:0/271401855 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f65100318e0 con 0x5565539be820 2024-04-26T01:52:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.090+0000 7f6530fbf740 10 client.25937 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.090+0000 7f6530fbf740 1 -- 172.21.15.6:0/271401855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556553a3f2e0 con 0x5565539be820 2024-04-26T01:52:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.091+0000 7f6514c00640 1 -- 172.21.15.6:0/271401855 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556553932320 con 0x5565539be820 2024-04-26T01:52:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.091+0000 7f6514c00640 1 -- 172.21.15.6:0/271401855 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f6510043040 con 0x5565539be820 2024-04-26T01:52:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.091+0000 7f6530fbf740 10 client.25937 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.091+0000 7f6530fbf740 10 client.25937 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.091+0000 7f6530fbf740 20 client.25937 populate_metadata read hostname 'smithi006' 2024-04-26T01:52:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.091+0000 7f6530fbf740 1 --2- 172.21.15.6:0/271401855 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556553a3f950 0x556553a41d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.091+0000 7f6530fbf740 4 client.25937 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.091+0000 7f6530fbf740 1 -- 172.21.15.6:0/271401855 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55655392c190 con 0x556553a3f950 2024-04-26T01:52:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.091+0000 7f6526a00640 1 --2- 172.21.15.6:0/271401855 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556553a3f950 0x556553a41d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.092+0000 7f6526a00640 1 --2- 172.21.15.6:0/271401855 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556553a3f950 0x556553a41d70 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2b19df4eb2e7a3f server_cookie=1ea768a086ef02a3 in_seq=0 out_seq=0 2024-04-26T01:52:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.092+0000 7f6514c00640 10 client.25937 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:22.129 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.131+0000 7f6514c00640 1 -- 172.21.15.6:0/271401855 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55655392c190 con 0x556553a3f950 2024-04-26T01:52:22.130 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.131+0000 7f6514c00640 10 client.25937 handle_command_reply: tid=0 2024-04-26T01:52:22.130 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.131+0000 7f6530fbf740 1 client.25937 shutdown 2024-04-26T01:52:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.131+0000 7f6530fbf740 1 -- 172.21.15.6:0/271401855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 msgr2=0x5565539ddc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.131+0000 7f6530fbf740 1 --2- 172.21.15.6:0/271401855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 0x5565539ddc10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f651002ade0 tx=0x7f6510031e00 comp rx=0 tx=0).stop 2024-04-26T01:52:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.132+0000 7f6530fbf740 1 -- 172.21.15.6:0/271401855 shutdown_connections 2024-04-26T01:52:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.132+0000 7f6530fbf740 1 --2- 172.21.15.6:0/271401855 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556553a3f950 0x556553a41d70 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.132+0000 7f6530fbf740 1 --2- 172.21.15.6:0/271401855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5565539bf140 0x5565539de150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.132+0000 7f6530fbf740 1 --2- 172.21.15.6:0/271401855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565539c9a00 0x5565539de690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.132+0000 7f6530fbf740 1 --2- 172.21.15.6:0/271401855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565539be820 0x5565539ddc10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.132+0000 7f6530fbf740 1 -- 172.21.15.6:0/271401855 >> 172.21.15.6:0/271401855 conn(0x5565539bad30 msgr2=0x5565539c0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.132+0000 7f6514c00640 0 client.25937 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.132+0000 7f6530fbf740 1 -- 172.21.15.6:0/271401855 shutdown_connections 2024-04-26T01:52:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.132+0000 7f6530fbf740 1 -- 172.21.15.6:0/271401855 wait complete. 2024-04-26T01:52:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.132+0000 7f6530fbf740 20 client.25937 trim_cache size 0 max 16384 2024-04-26T01:52:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:52:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:52:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:52:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:52:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.133+0000 7f652e200640 1 -- 172.21.15.6:0/4100801210 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f65080a3d90 msgr2=0x7f65080a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.133+0000 7f652e200640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f65080a3d90 0x7f65080a6250 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6510006010 tx=0x7f6510009820 comp rx=0 tx=0).stop 2024-04-26T01:52:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.133+0000 7f652e200640 1 -- 172.21.15.6:0/4100801210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652819a840 msgr2=0x7f652819acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.133+0000 7f652e200640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652819a840 0x7f652819acf0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f652000e970 tx=0x7f652000ed70 comp rx=0 tx=0).stop 2024-04-26T01:52:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.133+0000 7f652e200640 1 -- 172.21.15.6:0/4100801210 shutdown_connections 2024-04-26T01:52:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.133+0000 7f652e200640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f65080a3d90 0x7f65080a6250 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.133+0000 7f652e200640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652819a840 0x7f652819acf0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.133+0000 7f652e200640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6528107280 0x7f65281a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.133+0000 7f652e200640 1 --2- 172.21.15.6:0/4100801210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65281021b0 0x7f65281a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.133+0000 7f652e200640 1 -- 172.21.15.6:0/4100801210 >> 172.21.15.6:0/4100801210 conn(0x7f65280fda10 msgr2=0x7f65280ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.134+0000 7f652e200640 1 -- 172.21.15.6:0/4100801210 shutdown_connections 2024-04-26T01:52:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:22.134+0000 7f652e200640 1 -- 172.21.15.6:0/4100801210 wait complete. 2024-04-26T01:52:22.587 INFO:tasks.workunit.client.0.smithi006.stdout: 1 217336 4.03 MB/sec execute 166 sec latency 25.404 ms 2024-04-26T01:52:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:22 smithi006 ceph-mon[28419]: pgmap v202: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-04-26T01:52:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:22 smithi188 ceph-mon[32751]: pgmap v202: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-04-26T01:52:22.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:22 smithi029 ceph-mon[32775]: pgmap v202: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-04-26T01:52:23.587 INFO:tasks.workunit.client.0.smithi006.stdout: 1 218114 4.04 MB/sec execute 167 sec latency 24.879 ms 2024-04-26T01:52:24.587 INFO:tasks.workunit.client.0.smithi006.stdout: 1 219296 4.05 MB/sec execute 168 sec latency 70.506 ms 2024-04-26T01:52:24.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:24 smithi188 ceph-mon[32751]: pgmap v203: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-26T01:52:24.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:24 smithi029 ceph-mon[32775]: pgmap v203: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-26T01:52:25.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:24 smithi006 ceph-mon[28419]: pgmap v203: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-26T01:52:25.587 INFO:tasks.workunit.client.0.smithi006.stdout: 1 220399 4.07 MB/sec execute 169 sec latency 23.753 ms 2024-04-26T01:52:26.588 INFO:tasks.workunit.client.0.smithi006.stdout: 1 221465 4.08 MB/sec execute 170 sec latency 27.232 ms 2024-04-26T01:52:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:26 smithi188 ceph-mon[32751]: pgmap v204: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-26T01:52:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:26 smithi029 ceph-mon[32775]: pgmap v204: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-26T01:52:27.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:26 smithi006 ceph-mon[28419]: pgmap v204: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-26T01:52:27.588 INFO:tasks.workunit.client.0.smithi006.stdout: 1 222951 4.10 MB/sec execute 171 sec latency 34.016 ms 2024-04-26T01:52:28.588 INFO:tasks.workunit.client.0.smithi006.stdout: 1 223977 4.11 MB/sec execute 172 sec latency 24.033 ms 2024-04-26T01:52:28.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:28 smithi188 ceph-mon[32751]: pgmap v205: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 219 op/s 2024-04-26T01:52:28.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:28 smithi029 ceph-mon[32775]: pgmap v205: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 219 op/s 2024-04-26T01:52:29.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:28 smithi006 ceph-mon[28419]: pgmap v205: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 219 op/s 2024-04-26T01:52:29.588 INFO:tasks.workunit.client.0.smithi006.stdout: 1 224982 4.12 MB/sec execute 173 sec latency 28.516 ms 2024-04-26T01:52:30.588 INFO:tasks.workunit.client.0.smithi006.stdout: 1 225848 4.10 MB/sec execute 174 sec latency 23.342 ms 2024-04-26T01:52:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:30 smithi188 ceph-mon[32751]: pgmap v206: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 161 op/s 2024-04-26T01:52:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:30 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:30.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:30 smithi029 ceph-mon[32775]: pgmap v206: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 161 op/s 2024-04-26T01:52:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:30 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:31.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:30 smithi006 ceph-mon[28419]: pgmap v206: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 161 op/s 2024-04-26T01:52:31.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:30 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:31.588 INFO:tasks.workunit.client.0.smithi006.stdout: 1 227326 4.15 MB/sec execute 175 sec latency 14.522 ms 2024-04-26T01:52:31.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:31 smithi188 ceph-mon[32751]: pgmap v207: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 221 op/s 2024-04-26T01:52:31.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:31 smithi029 ceph-mon[32775]: pgmap v207: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 221 op/s 2024-04-26T01:52:32.040 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 15.9 2024-04-26T01:52:32.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:52:32.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:31 smithi006 ceph-mon[28419]: pgmap v207: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 221 op/s 2024-04-26T01:52:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:52:32.588 INFO:tasks.workunit.client.0.smithi006.stdout: 1 228306 4.14 MB/sec execute 176 sec latency 12.620 ms 2024-04-26T01:52:33.589 INFO:tasks.workunit.client.0.smithi006.stdout: 1 229984 4.17 MB/sec execute 177 sec latency 19.298 ms 2024-04-26T01:52:34.438 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:34 smithi006 ceph-mon[28419]: pgmap v208: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-04-26T01:52:34.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:34 smithi029 ceph-mon[32775]: pgmap v208: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-04-26T01:52:34.589 INFO:tasks.workunit.client.0.smithi006.stdout: 1 231072 4.19 MB/sec execute 178 sec latency 15.240 ms 2024-04-26T01:52:34.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:34 smithi188 ceph-mon[32751]: pgmap v208: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-04-26T01:52:34.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.777+0000 7fd021400700 1 -- 172.21.15.6:0/3238968201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd01c108a30 msgr2=0x7fd01c108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:34.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.777+0000 7fd021400700 1 --2- 172.21.15.6:0/3238968201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd01c108a30 0x7fd01c108e10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd00c009990 tx=0x7fd00c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:52:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.777+0000 7fd021400700 1 -- 172.21.15.6:0/3238968201 shutdown_connections 2024-04-26T01:52:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.777+0000 7fd021400700 1 --2- 172.21.15.6:0/3238968201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd01c06b0f0 0x7fd01c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.777+0000 7fd021400700 1 --2- 172.21.15.6:0/3238968201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd01c108a30 0x7fd01c108e10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.777+0000 7fd021400700 1 --2- 172.21.15.6:0/3238968201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd01c06a750 0x7fd01c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.777+0000 7fd021400700 1 -- 172.21.15.6:0/3238968201 >> 172.21.15.6:0/3238968201 conn(0x7fd01c0788f0 msgr2=0x7fd01c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.777+0000 7fd021400700 1 -- 172.21.15.6:0/3238968201 shutdown_connections 2024-04-26T01:52:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.778+0000 7fd021400700 1 -- 172.21.15.6:0/3238968201 wait complete. 2024-04-26T01:52:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.779+0000 7fd021400700 1 Processor -- start 2024-04-26T01:52:34.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.779+0000 7fd021400700 1 -- start start 2024-04-26T01:52:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.779+0000 7fd021400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd01c06a750 0x7fd01c19b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.779+0000 7fd021400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd01c06b0f0 0x7fd01c19b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.779+0000 7fd021400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd01c108a30 0x7fd01c19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.779+0000 7fd021400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd01c112c00 con 0x7fd01c108a30 2024-04-26T01:52:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.779+0000 7fd021400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd01c112a80 con 0x7fd01c06b0f0 2024-04-26T01:52:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.779+0000 7fd021400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd01c112d80 con 0x7fd01c06a750 2024-04-26T01:52:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.780+0000 7fd01a800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd01c06a750 0x7fd01c19b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.780+0000 7fd01a800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd01c06a750 0x7fd01c19b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58058/0 (socket says 172.21.15.6:58058) 2024-04-26T01:52:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.780+0000 7fd01a800700 1 -- 172.21.15.6:0/3266591235 learned_addr learned my addr 172.21.15.6:0/3266591235 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.780+0000 7fd019e00700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd01c06b0f0 0x7fd01c19b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.780+0000 7fd01b200700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd01c108a30 0x7fd01c19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.780+0000 7fd01a800700 1 -- 172.21.15.6:0/3266591235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd01c06b0f0 msgr2=0x7fd01c19b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:34.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.780+0000 7fd01a800700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd01c06b0f0 0x7fd01c19b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:34.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.780+0000 7fd01a800700 1 -- 172.21.15.6:0/3266591235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd01c108a30 msgr2=0x7fd01c19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:34.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.780+0000 7fd01a800700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd01c108a30 0x7fd01c19fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:34.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.780+0000 7fd01a800700 1 -- 172.21.15.6:0/3266591235 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd01c1a0340 con 0x7fd01c06a750 2024-04-26T01:52:34.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.781+0000 7fd01a800700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd01c06a750 0x7fd01c19b330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000c00 tx=0x7fd00c0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:34.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.781+0000 7fd013400700 1 -- 172.21.15.6:0/3266591235 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c027070 con 0x7fd01c06a750 2024-04-26T01:52:34.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.781+0000 7fd013400700 1 -- 172.21.15.6:0/3266591235 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd00c022470 con 0x7fd01c06a750 2024-04-26T01:52:34.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.781+0000 7fd013400700 1 -- 172.21.15.6:0/3266591235 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c017650 con 0x7fd01c06a750 2024-04-26T01:52:34.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.781+0000 7fd021400700 1 -- 172.21.15.6:0/3266591235 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd01c1a05d0 con 0x7fd01c06a750 2024-04-26T01:52:34.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.781+0000 7fd021400700 1 -- 172.21.15.6:0/3266591235 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd01c1a7eb0 con 0x7fd01c06a750 2024-04-26T01:52:34.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.783+0000 7fd021400700 1 -- 172.21.15.6:0/3266591235 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd01c104170 con 0x7fd01c06a750 2024-04-26T01:52:34.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.785+0000 7fd013400700 1 -- 172.21.15.6:0/3266591235 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fd00c02a070 con 0x7fd01c06a750 2024-04-26T01:52:34.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.786+0000 7fd013400700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0000a3db0 0x7fd0000a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:34.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.786+0000 7fd013400700 1 -- 172.21.15.6:0/3266591235 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7fd00c0d1710 con 0x7fd01c06a750 2024-04-26T01:52:34.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.786+0000 7fd019e00700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0000a3db0 0x7fd0000a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:34.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.787+0000 7fd019e00700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0000a3db0 0x7fd0000a6270 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd01c078f60 tx=0x7fd008009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:34.790+0000 7fd013400700 1 -- 172.21.15.6:0/3266591235 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd00c013070 con 0x7fd01c06a750 2024-04-26T01:52:35.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.113+0000 7fd021400700 1 -- 172.21.15.6:0/3266591235 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd01c19c690 con 0x7fd01c06a750 2024-04-26T01:52:35.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.114+0000 7fd013400700 1 -- 172.21.15.6:0/3266591235 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11850 (secure 0 0 0) 0x7fd00c09deb0 con 0x7fd01c06a750 2024-04-26T01:52:35.113 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:52:35.113 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:52:28.987258+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:52:35.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.117+0000 7fd021400700 1 -- 172.21.15.6:0/3266591235 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0000a3db0 msgr2=0x7fd0000a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:35.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.117+0000 7fd021400700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0000a3db0 0x7fd0000a6270 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd01c078f60 tx=0x7fd008009260 comp rx=0 tx=0).stop 2024-04-26T01:52:35.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.117+0000 7fd021400700 1 -- 172.21.15.6:0/3266591235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd01c06a750 msgr2=0x7fd01c19b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:35.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.117+0000 7fd021400700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd01c06a750 0x7fd01c19b330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000c00 tx=0x7fd00c0057c0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.117+0000 7fd021400700 1 -- 172.21.15.6:0/3266591235 shutdown_connections 2024-04-26T01:52:35.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.117+0000 7fd021400700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0000a3db0 0x7fd0000a6270 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.117+0000 7fd021400700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd01c108a30 0x7fd01c19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.117+0000 7fd021400700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd01c06a750 0x7fd01c19b330 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.117+0000 7fd021400700 1 --2- 172.21.15.6:0/3266591235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd01c06b0f0 0x7fd01c19b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.117+0000 7fd021400700 1 -- 172.21.15.6:0/3266591235 >> 172.21.15.6:0/3266591235 conn(0x7fd01c0788f0 msgr2=0x7fd01c10de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:35.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.117+0000 7fd021400700 1 -- 172.21.15.6:0/3266591235 shutdown_connections 2024-04-26T01:52:35.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:35.117+0000 7fd021400700 1 -- 172.21.15.6:0/3266591235 wait complete. 2024-04-26T01:52:35.117 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 39 2024-04-26T01:52:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.296+0000 7f4d17e00640 1 -- 172.21.15.6:0/3155192789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d18102f30 msgr2=0x7f4d18103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.296+0000 7f4d17e00640 1 --2- 172.21.15.6:0/3155192789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d18102f30 0x7f4d18103310 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009900 tx=0x7f4d0802a480 comp rx=0 tx=0).stop 2024-04-26T01:52:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.296+0000 7f4d17e00640 1 -- 172.21.15.6:0/3155192789 shutdown_connections 2024-04-26T01:52:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.296+0000 7f4d17e00640 1 --2- 172.21.15.6:0/3155192789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d1810c350 0x7f4d1810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.296+0000 7f4d17e00640 1 --2- 172.21.15.6:0/3155192789 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d18103850 0x7f4d1810bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.296+0000 7f4d17e00640 1 --2- 172.21.15.6:0/3155192789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d18102f30 0x7f4d18103310 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.296+0000 7f4d17e00640 1 -- 172.21.15.6:0/3155192789 >> 172.21.15.6:0/3155192789 conn(0x7f4d180fc730 msgr2=0x7f4d180feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.296+0000 7f4d17e00640 1 -- 172.21.15.6:0/3155192789 shutdown_connections 2024-04-26T01:52:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.296+0000 7f4d17e00640 1 -- 172.21.15.6:0/3155192789 wait complete. 2024-04-26T01:52:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.297+0000 7f4d17e00640 1 Processor -- start 2024-04-26T01:52:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.297+0000 7f4d17e00640 1 -- start start 2024-04-26T01:52:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.297+0000 7f4d17e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d18102f30 0x7f4d181a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.297+0000 7f4d17e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d18103850 0x7f4d181a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.297+0000 7f4d17e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d1810c350 0x7f4d1819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.297+0000 7f4d17e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4d18112030 con 0x7f4d18102f30 2024-04-26T01:52:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.297+0000 7f4d17e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d18111eb0 con 0x7f4d1810c350 2024-04-26T01:52:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.297+0000 7f4d17e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181121b0 con 0x7f4d18103850 2024-04-26T01:52:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.297+0000 7f4d16a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d18102f30 0x7f4d181a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.298+0000 7f4d16000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d18103850 0x7f4d181a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.298+0000 7f4d17400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d1810c350 0x7f4d1819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.298+0000 7f4d17400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d1810c350 0x7f4d1819a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43818/0 (socket says 172.21.15.6:43818) 2024-04-26T01:52:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.298+0000 7f4d17400640 1 -- 172.21.15.6:0/1305953423 learned_addr learned my addr 172.21.15.6:0/1305953423 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.298+0000 7f4d16000640 1 -- 172.21.15.6:0/1305953423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d1810c350 msgr2=0x7f4d1819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.298+0000 7f4d16000640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d1810c350 0x7f4d1819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.298+0000 7f4d16000640 1 -- 172.21.15.6:0/1305953423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d18102f30 msgr2=0x7f4d181a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.298+0000 7f4d16000640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d18102f30 0x7f4d181a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.298+0000 7f4d16000640 1 -- 172.21.15.6:0/1305953423 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d1819ad70 con 0x7f4d18103850 2024-04-26T01:52:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.298+0000 7f4d17400640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d1810c350 0x7f4d1819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:35.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.298+0000 7f4d16000640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d18103850 0x7f4d181a0930 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4d0400e7c0 tx=0x7f4d0400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.298+0000 7f4d1ca00640 1 -- 172.21.15.6:0/1305953423 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d04009530 con 0x7f4d18103850 2024-04-26T01:52:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.299+0000 7f4d1ca00640 1 -- 172.21.15.6:0/1305953423 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d0400ccc0 con 0x7f4d18103850 2024-04-26T01:52:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.299+0000 7f4d1ca00640 1 -- 172.21.15.6:0/1305953423 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d04004a00 con 0x7f4d18103850 2024-04-26T01:52:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.299+0000 7f4d17e00640 1 -- 172.21.15.6:0/1305953423 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d1819b060 con 0x7f4d18103850 2024-04-26T01:52:35.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.299+0000 7f4d17e00640 1 -- 172.21.15.6:0/1305953423 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d18103cd0 con 0x7f4d18103850 2024-04-26T01:52:35.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.303+0000 7f4d20ddd740 1 Processor -- start 2024-04-26T01:52:35.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.303+0000 7f4d20ddd740 1 -- start start 2024-04-26T01:52:35.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d1ca00640 1 -- 172.21.15.6:0/1305953423 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f4d04004ba0 con 0x7f4d18103850 2024-04-26T01:52:35.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d20ddd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d75d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d20ddd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 0x55b142d80650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d20ddd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d80b90 0x55b142d82f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d20ddd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55b142cdb900 con 0x55b142d80b90 2024-04-26T01:52:35.303 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d20ddd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b142c81ef0 con 0x55b142d759b0 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d20ddd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b142cd78c0 con 0x55b142d762d0 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d16a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 0x55b142d80650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d17400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d75d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d16000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d80b90 0x55b142d82f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d16a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 0x55b142d80650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58080/0 (socket says 172.21.15.6:58080) 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d17400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d75d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43830/0 (socket says 172.21.15.6:43830) 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.304+0000 7f4d16a00640 1 -- 172.21.15.6:0/3611436244 learned_addr learned my addr 172.21.15.6:0/3611436244 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.305+0000 7f4d16a00640 1 -- 172.21.15.6:0/3611436244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 msgr2=0x55b142d75d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.305+0000 7f4d16a00640 1 --2- 172.21.15.6:0/3611436244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d75d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.305+0000 7f4d16a00640 1 -- 172.21.15.6:0/3611436244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d80b90 msgr2=0x55b142d82f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.305+0000 7f4d16a00640 1 --2- 172.21.15.6:0/3611436244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d80b90 0x55b142d82f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.305+0000 7f4d16a00640 1 -- 172.21.15.6:0/3611436244 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1429ee210 con 0x55b142d762d0 2024-04-26T01:52:35.304 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.305+0000 7f4d17400640 1 --2- 172.21.15.6:0/3611436244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d75d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.305+0000 7f4d16a00640 1 --2- 172.21.15.6:0/3611436244 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 0x55b142d80650 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4d08000c00 tx=0x7f4d08004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8033c0088a3c2b01 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.305+0000 7f4d03e00640 1 -- 172.21.15.6:0/3611436244 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08035040 con 0x55b142d762d0 2024-04-26T01:52:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.305+0000 7f4d03e00640 1 -- 172.21.15.6:0/3611436244 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d08004430 con 0x55b142d762d0 2024-04-26T01:52:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.305+0000 7f4d03e00640 1 -- 172.21.15.6:0/3611436244 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08004e60 con 0x55b142d762d0 2024-04-26T01:52:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.306+0000 7f4d1ca00640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4cfc0ac590 0x7f4cfc0aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.306+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3611436244 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 msgr2=0x55b142d80650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.306+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/3611436244 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 0x55b142d80650 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4d08000c00 tx=0x7f4d08004040 comp rx=0 tx=0).stop 2024-04-26T01:52:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.306+0000 7f4d1ca00640 1 -- 172.21.15.6:0/1305953423 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7f4d040d0670 con 0x7f4d18103850 2024-04-26T01:52:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.306+0000 7f4d17400640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4cfc0ac590 0x7f4cfc0aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.306+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3611436244 shutdown_connections 2024-04-26T01:52:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.306+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/3611436244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d80b90 0x55b142d82f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.306+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/3611436244 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 0x55b142d80650 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.306+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/3611436244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d75d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.306+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3611436244 >> 172.21.15.6:0/3611436244 conn(0x55b142d71ec0 msgr2=0x55b142d722a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.306+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3611436244 shutdown_connections 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.307+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3611436244 wait complete. 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.307+0000 7f4d20ddd740 1 -- start start 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.307+0000 7f4d17400640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4cfc0ac590 0x7f4cfc0aea50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4d100049d0 tx=0x7f4d10009500 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.307+0000 7f4d20ddd740 1 Processor -- start 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d20ddd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d7dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d20ddd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 0x55b142d7e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d20ddd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d80b90 0x55b142d7e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d20ddd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55b142d83760 con 0x55b142d80b90 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d20ddd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b142c95310 con 0x55b142d759b0 2024-04-26T01:52:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d20ddd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b142d838e0 con 0x55b142d762d0 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d20ddd740 10 client.0 _ll_register_callbacks cb 0x55b142c63cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d16000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 0x55b142d7e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d16000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 0x55b142d7e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58092/0 (socket says 172.21.15.6:58092) 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d16000640 1 -- 172.21.15.6:0/3151699983 learned_addr learned my addr 172.21.15.6:0/3151699983 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d17400640 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d80b90 0x55b142d7e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d16a00640 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d7dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d16000640 1 -- 172.21.15.6:0/3151699983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 msgr2=0x55b142d7dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.308+0000 7f4d16000640 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d7dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d16000640 1 -- 172.21.15.6:0/3151699983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d80b90 msgr2=0x55b142d7e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d16000640 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d80b90 0x55b142d7e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d16000640 1 -- 172.21.15.6:0/3151699983 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b142aba620 con 0x55b142d762d0 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d17400640 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d80b90 0x55b142d7e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d16000640 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 0x55b142d7e120 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4d0400e790 tx=0x7f4d04005210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=957aca350af9f731 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d00c00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:52:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d00c00640 1 -- 172.21.15.6:0/3151699983 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0401c070 con 0x55b142d762d0 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3151699983 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d08004430 con 0x55b142d762d0 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d00c00640 1 -- 172.21.15.6:0/3151699983 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7f4d040a4fa0 con 0x55b142d762d0 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d00c00640 1 -- 172.21.15.6:0/3151699983 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4d04098860 con 0x55b142d762d0 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d20ddd740 10 client.26002 fetch_fsmap learned FSMap version 39 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3151699983 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b142d8fb30 con 0x55b142d762d0 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.309+0000 7f4d00c00640 1 -- 172.21.15.6:0/3151699983 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d08004430 con 0x55b142d762d0 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.310+0000 7f4d00c00640 1 -- 172.21.15.6:0/3151699983 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 39) ==== 8338+0+0 (secure 0 0 0) 0x7f4d040d0b20 con 0x55b142d762d0 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.310+0000 7f4d20ddd740 10 client.26002 fetch_fsmap finished waiting for FSMap version 39 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.310+0000 7f4d20ddd740 10 client.26002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.310+0000 7f4d20ddd740 20 client.26002 populate_metadata read hostname 'smithi006' 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.310+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b142d90fa0 0x55b142d93390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.310+0000 7f4d20ddd740 4 client.26002 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.310+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3151699983 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b142cef8b0 con 0x55b142d90fa0 2024-04-26T01:52:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.310+0000 7f4d16a00640 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b142d90fa0 0x55b142d93390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.311+0000 7f4d16a00640 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b142d90fa0 0x55b142d93390 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=665270ed439f9cb5 server_cookie=a8d03e165bf9eeb5 in_seq=0 out_seq=0 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.311+0000 7f4d00c00640 10 client.26002 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.311+0000 7f4d00c00640 1 -- 172.21.15.6:0/3151699983 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b142cef8b0 con 0x55b142d90fa0 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.311+0000 7f4d00c00640 10 client.26002 handle_command_reply: tid=0 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d20ddd740 1 client.26002 shutdown 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3151699983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 msgr2=0x55b142d7e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 0x55b142d7e120 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4d0400e790 tx=0x7f4d04005210 comp rx=0 tx=0).stop 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3151699983 shutdown_connections 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b142d90fa0 0x55b142d93390 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d80b90 0x55b142d7e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d762d0 0x55b142d7e120 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/3151699983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d7dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3151699983 >> 172.21.15.6:0/3151699983 conn(0x55b142d71ec0 msgr2=0x55b142d77bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d00c00640 0 client.26002 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3151699983 shutdown_connections 2024-04-26T01:52:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d20ddd740 1 -- 172.21.15.6:0/3151699983 wait complete. 2024-04-26T01:52:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.312+0000 7f4d20ddd740 20 client.26002 trim_cache size 0 max 16384 2024-04-26T01:52:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.329+0000 7f4d20ddd740 1 Processor -- start 2024-04-26T01:52:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.330+0000 7f4d20ddd740 1 -- start start 2024-04-26T01:52:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.330+0000 7f4d20ddd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d759b0 0x55b142d8c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.330+0000 7f4d20ddd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d762d0 0x55b142d717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.330+0000 7f4d20ddd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d80b90 0x55b142d98c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.330+0000 7f4d20ddd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181ae210 con 0x55b142d762d0 2024-04-26T01:52:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.330+0000 7f4d20ddd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181abae0 con 0x55b142d80b90 2024-04-26T01:52:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.330+0000 7f4d20ddd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181a99b0 con 0x55b142d759b0 2024-04-26T01:52:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.330+0000 7f4d17400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d80b90 0x55b142d98c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.330+0000 7f4d16000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d762d0 0x55b142d717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.330+0000 7f4d17400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d80b90 0x55b142d98c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43842/0 (socket says 172.21.15.6:43842) 2024-04-26T01:52:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.330+0000 7f4d17400640 1 -- 172.21.15.6:0/1729018986 learned_addr learned my addr 172.21.15.6:0/1729018986 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.330+0000 7f4d16a00640 1 --2- 172.21.15.6:0/1729018986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d759b0 0x55b142d8c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.331+0000 7f4d17400640 1 -- 172.21.15.6:0/1729018986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d759b0 msgr2=0x55b142d8c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.331+0000 7f4d17400640 1 --2- 172.21.15.6:0/1729018986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d759b0 0x55b142d8c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.331+0000 7f4d17400640 1 -- 172.21.15.6:0/1729018986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d762d0 msgr2=0x55b142d717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.331+0000 7f4d17400640 1 --2- 172.21.15.6:0/1729018986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d762d0 0x55b142d717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.331+0000 7f4d17400640 1 -- 172.21.15.6:0/1729018986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b142d93700 con 0x55b142d80b90 2024-04-26T01:52:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.331+0000 7f4d16000640 1 --2- 172.21.15.6:0/1729018986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d762d0 0x55b142d717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.331+0000 7f4d17400640 1 --2- 172.21.15.6:0/1729018986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d80b90 0x55b142d98c80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000f970 tx=0x7f4d10017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4d65ef798937470 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.331+0000 7f4d03400640 1 -- 172.21.15.6:0/1729018986 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d100104f0 con 0x55b142d80b90 2024-04-26T01:52:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.331+0000 7f4d03400640 1 -- 172.21.15.6:0/1729018986 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d10010b40 con 0x55b142d80b90 2024-04-26T01:52:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.331+0000 7f4d03400640 1 -- 172.21.15.6:0/1729018986 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1001bc00 con 0x55b142d80b90 2024-04-26T01:52:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.332+0000 7f4d20ddd740 1 -- 172.21.15.6:0/1729018986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d80b90 msgr2=0x55b142d98c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.332+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/1729018986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d80b90 0x55b142d98c80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000f970 tx=0x7f4d10017000 comp rx=0 tx=0).stop 2024-04-26T01:52:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.332+0000 7f4d20ddd740 1 -- 172.21.15.6:0/1729018986 shutdown_connections 2024-04-26T01:52:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.332+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/1729018986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d762d0 0x55b142d717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.332+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/1729018986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d80b90 0x55b142d98c80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.332+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/1729018986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d759b0 0x55b142d8c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.332+0000 7f4d20ddd740 1 -- 172.21.15.6:0/1729018986 >> 172.21.15.6:0/1729018986 conn(0x55b142d71ec0 msgr2=0x55b142d78f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.332+0000 7f4d20ddd740 1 -- 172.21.15.6:0/1729018986 shutdown_connections 2024-04-26T01:52:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.332+0000 7f4d20ddd740 1 -- 172.21.15.6:0/1729018986 wait complete. 2024-04-26T01:52:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.333+0000 7f4d20ddd740 1 -- start start 2024-04-26T01:52:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.333+0000 7f4d20ddd740 1 Processor -- start 2024-04-26T01:52:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.333+0000 7f4d20ddd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d88c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.333+0000 7f4d20ddd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d762d0 0x55b142d877d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.333+0000 7f4d20ddd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d80b90 0x55b142d87d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.333+0000 7f4d20ddd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181aa000 con 0x55b142d762d0 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.333+0000 7f4d20ddd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181ab410 con 0x55b142d759b0 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.333+0000 7f4d20ddd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181aa670 con 0x55b142d80b90 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.334+0000 7f4d20ddd740 10 client.0 _ll_register_callbacks cb 0x7f4d040046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.334+0000 7f4d16a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d88c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.334+0000 7f4d16a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d88c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43846/0 (socket says 172.21.15.6:43846) 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.334+0000 7f4d16a00640 1 -- 172.21.15.6:0/900622731 learned_addr learned my addr 172.21.15.6:0/900622731 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.334+0000 7f4d17400640 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d80b90 0x55b142d87d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.334+0000 7f4d16000640 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d762d0 0x55b142d877d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.334+0000 7f4d16a00640 1 -- 172.21.15.6:0/900622731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d80b90 msgr2=0x55b142d87d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.334+0000 7f4d16a00640 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d80b90 0x55b142d87d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.334+0000 7f4d16a00640 1 -- 172.21.15.6:0/900622731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d762d0 msgr2=0x55b142d877d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.334+0000 7f4d16a00640 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d762d0 0x55b142d877d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.334+0000 7f4d16a00640 1 -- 172.21.15.6:0/900622731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b142d882d0 con 0x55b142d759b0 2024-04-26T01:52:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.334+0000 7f4d16a00640 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d88c70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4d0802aa70 tx=0x7f4d08004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d827d1535ba3aad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.335+0000 7f4d00c00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:52:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.335+0000 7f4d00c00640 1 -- 172.21.15.6:0/900622731 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08058060 con 0x55b142d759b0 2024-04-26T01:52:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.335+0000 7f4d20ddd740 1 -- 172.21.15.6:0/900622731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d10010b40 con 0x55b142d759b0 2024-04-26T01:52:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.335+0000 7f4d00c00640 1 -- 172.21.15.6:0/900622731 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7f4d080533b0 con 0x55b142d759b0 2024-04-26T01:52:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.335+0000 7f4d00c00640 1 -- 172.21.15.6:0/900622731 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4d08035040 con 0x55b142d759b0 2024-04-26T01:52:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.335+0000 7f4d20ddd740 10 client.25961 fetch_fsmap learned FSMap version 39 2024-04-26T01:52:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.335+0000 7f4d20ddd740 1 -- 172.21.15.6:0/900622731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b142df72c0 con 0x55b142d759b0 2024-04-26T01:52:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.335+0000 7f4d00c00640 1 -- 172.21.15.6:0/900622731 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d10010b40 con 0x55b142d759b0 2024-04-26T01:52:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.335+0000 7f4d00c00640 1 -- 172.21.15.6:0/900622731 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8338+0+0 (secure 0 0 0) 0x7f4d08003ab0 con 0x55b142d759b0 2024-04-26T01:52:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.336+0000 7f4d20ddd740 10 client.25961 fetch_fsmap finished waiting for FSMap version 39 2024-04-26T01:52:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.336+0000 7f4d20ddd740 10 client.25961 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.336+0000 7f4d20ddd740 20 client.25961 populate_metadata read hostname 'smithi006' 2024-04-26T01:52:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.336+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b142df75f0 0x55b142df99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.336+0000 7f4d20ddd740 4 client.25961 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.336+0000 7f4d20ddd740 1 -- 172.21.15.6:0/900622731 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b142ce9320 con 0x55b142df75f0 2024-04-26T01:52:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.336+0000 7f4d16000640 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b142df75f0 0x55b142df99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.336+0000 7f4d16000640 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b142df75f0 0x55b142df99e0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af157b129a32cba1 server_cookie=7b0e51ebe61138b0 in_seq=0 out_seq=0 2024-04-26T01:52:35.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.336+0000 7f4d00c00640 10 client.25961 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:35.379 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.380+0000 7f4d00c00640 1 -- 172.21.15.6:0/900622731 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b142ce9320 con 0x55b142df75f0 2024-04-26T01:52:35.379 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.380+0000 7f4d00c00640 10 client.25961 handle_command_reply: tid=0 2024-04-26T01:52:35.379 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.380+0000 7f4d20ddd740 1 client.25961 shutdown 2024-04-26T01:52:35.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.381+0000 7f4d20ddd740 1 -- 172.21.15.6:0/900622731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 msgr2=0x55b142d88c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:35.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.381+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d88c70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4d0802aa70 tx=0x7f4d08004a40 comp rx=0 tx=0).stop 2024-04-26T01:52:35.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.381+0000 7f4d20ddd740 1 -- 172.21.15.6:0/900622731 shutdown_connections 2024-04-26T01:52:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.381+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b142df75f0 0x55b142df99e0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.381+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b142d762d0 0x55b142d877d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.381+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b142d80b90 0x55b142d87d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.381+0000 7f4d20ddd740 1 --2- 172.21.15.6:0/900622731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b142d759b0 0x55b142d88c70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.381+0000 7f4d00c00640 0 client.25961 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.381+0000 7f4d20ddd740 1 -- 172.21.15.6:0/900622731 >> 172.21.15.6:0/900622731 conn(0x55b142d71ec0 msgr2=0x55b142d77360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.381+0000 7f4d20ddd740 1 -- 172.21.15.6:0/900622731 shutdown_connections 2024-04-26T01:52:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.382+0000 7f4d20ddd740 1 -- 172.21.15.6:0/900622731 wait complete. 2024-04-26T01:52:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.382+0000 7f4d20ddd740 20 client.25961 trim_cache size 0 max 16384 2024-04-26T01:52:35.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:52:35.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:52:35.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:52:35.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:52:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.383+0000 7f4d17e00640 1 -- 172.21.15.6:0/1305953423 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4cfc0ac590 msgr2=0x7f4cfc0aea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.383+0000 7f4d17e00640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4cfc0ac590 0x7f4cfc0aea50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4d100049d0 tx=0x7f4d10009500 comp rx=0 tx=0).stop 2024-04-26T01:52:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.383+0000 7f4d17e00640 1 -- 172.21.15.6:0/1305953423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d18103850 msgr2=0x7f4d181a0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.383+0000 7f4d17e00640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d18103850 0x7f4d181a0930 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4d0400e7c0 tx=0x7f4d0400ec80 comp rx=0 tx=0).stop 2024-04-26T01:52:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.383+0000 7f4d17e00640 1 -- 172.21.15.6:0/1305953423 shutdown_connections 2024-04-26T01:52:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.383+0000 7f4d17e00640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4cfc0ac590 0x7f4cfc0aea50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.383+0000 7f4d17e00640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d1810c350 0x7f4d1819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.383+0000 7f4d17e00640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d18103850 0x7f4d181a0930 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.383+0000 7f4d17e00640 1 --2- 172.21.15.6:0/1305953423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d18102f30 0x7f4d181a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.383+0000 7f4d17e00640 1 -- 172.21.15.6:0/1305953423 >> 172.21.15.6:0/1305953423 conn(0x7f4d180fc730 msgr2=0x7f4d18100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.383+0000 7f4d17e00640 1 -- 172.21.15.6:0/1305953423 shutdown_connections 2024-04-26T01:52:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:35.383+0000 7f4d17e00640 1 -- 172.21.15.6:0/1305953423 wait complete. 2024-04-26T01:52:35.440 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:35 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3266591235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:35.481 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:35 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3266591235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:35.589 INFO:tasks.workunit.client.0.smithi006.stdout: 1 232098 4.19 MB/sec execute 179 sec latency 113.720 ms 2024-04-26T01:52:35.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:35 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3266591235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:35.984 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 75.9 --await-for 42.0' 2024-04-26T01:52:35.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 75.9 --await-for 42.0 2024-04-26T01:52:36.498 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:52:36.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:36 smithi006 ceph-mon[28419]: pgmap v209: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-04-26T01:52:36.588 INFO:tasks.workunit.client.0.smithi006.stdout: 1 233379 4.20 MB/sec execute 180 sec latency 17.701 ms 2024-04-26T01:52:36.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:36 smithi188 ceph-mon[32751]: pgmap v209: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-04-26T01:52:36.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:36 smithi029 ceph-mon[32775]: pgmap v209: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-04-26T01:52:37.589 INFO:tasks.workunit.client.0.smithi006.stdout: 1 234553 4.22 MB/sec execute 181 sec latency 58.551 ms 2024-04-26T01:52:38.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:38 smithi006 ceph-mon[28419]: pgmap v210: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 243 op/s 2024-04-26T01:52:38.590 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235583 4.23 MB/sec execute 182 sec latency 35.647 ms 2024-04-26T01:52:38.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:38 smithi188 ceph-mon[32751]: pgmap v210: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 243 op/s 2024-04-26T01:52:38.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:38 smithi029 ceph-mon[32775]: pgmap v210: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 243 op/s 2024-04-26T01:52:38.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.936+0000 7f3803400700 1 -- 172.21.15.6:0/3201099200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc103480 msgr2=0x7f37fc10f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:38.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.936+0000 7f3803400700 1 --2- 172.21.15.6:0/3201099200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc103480 0x7f37fc10f910 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f37f400b0a0 tx=0x7f37f400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:52:38.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.937+0000 7f3803400700 1 -- 172.21.15.6:0/3201099200 shutdown_connections 2024-04-26T01:52:38.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.937+0000 7f3803400700 1 --2- 172.21.15.6:0/3201099200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc103480 0x7f37fc10f910 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:38.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.937+0000 7f3803400700 1 --2- 172.21.15.6:0/3201099200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37fc108ab0 0x7f37fc108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:38.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.937+0000 7f3803400700 1 --2- 172.21.15.6:0/3201099200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37fc102ac0 0x7f37fc102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:38.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.937+0000 7f3803400700 1 -- 172.21.15.6:0/3201099200 >> 172.21.15.6:0/3201099200 conn(0x7f37fc079bf0 msgr2=0x7f37fc07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:38.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.937+0000 7f3803400700 1 -- 172.21.15.6:0/3201099200 shutdown_connections 2024-04-26T01:52:38.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.937+0000 7f3803400700 1 -- 172.21.15.6:0/3201099200 wait complete. 2024-04-26T01:52:38.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.938+0000 7f3803400700 1 Processor -- start 2024-04-26T01:52:38.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.938+0000 7f3803400700 1 -- start start 2024-04-26T01:52:38.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.938+0000 7f3803400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37fc102ac0 0x7f37fc199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:38.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.938+0000 7f3803400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37fc103480 0x7f37fc1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:38.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f3803400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc108ab0 0x7f37fc19d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:38.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f3803400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc112c90 con 0x7f37fc102ac0 2024-04-26T01:52:38.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f3803400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc112b10 con 0x7f37fc108ab0 2024-04-26T01:52:38.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f3803400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc112e10 con 0x7f37fc103480 2024-04-26T01:52:38.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f37f3e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37fc103480 0x7f37fc1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:38.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f37f3e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37fc103480 0x7f37fc1995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37548/0 (socket says 172.21.15.6:37548) 2024-04-26T01:52:38.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f37f3e00700 1 -- 172.21.15.6:0/2801559827 learned_addr learned my addr 172.21.15.6:0/2801559827 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:38.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f3800a00700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37fc102ac0 0x7f37fc199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:38.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f3801400700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc108ab0 0x7f37fc19d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:38.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f37f3e00700 1 -- 172.21.15.6:0/2801559827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc108ab0 msgr2=0x7f37fc19d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:38.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f37f3e00700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc108ab0 0x7f37fc19d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:38.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f37f3e00700 1 -- 172.21.15.6:0/2801559827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37fc102ac0 msgr2=0x7f37fc199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:38.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f37f3e00700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37fc102ac0 0x7f37fc199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:38.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.939+0000 7f37f3e00700 1 -- 172.21.15.6:0/2801559827 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37fc19e020 con 0x7f37fc103480 2024-04-26T01:52:38.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.940+0000 7f37f3e00700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37fc103480 0x7f37fc1995a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f37e400e9a0 tx=0x7f37e400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:38.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.940+0000 7f37fa000700 1 -- 172.21.15.6:0/2801559827 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e400ca10 con 0x7f37fc103480 2024-04-26T01:52:38.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.940+0000 7f3803400700 1 -- 172.21.15.6:0/2801559827 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37fc19e310 con 0x7f37fc103480 2024-04-26T01:52:38.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.940+0000 7f37fa000700 1 -- 172.21.15.6:0/2801559827 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e400cbb0 con 0x7f37fc103480 2024-04-26T01:52:38.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.940+0000 7f37fa000700 1 -- 172.21.15.6:0/2801559827 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4018680 con 0x7f37fc103480 2024-04-26T01:52:38.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.940+0000 7f3803400700 1 -- 172.21.15.6:0/2801559827 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37fc1a5b90 con 0x7f37fc103480 2024-04-26T01:52:38.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.944+0000 7f37fa000700 1 -- 172.21.15.6:0/2801559827 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f37e4010430 con 0x7f37fc103480 2024-04-26T01:52:38.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.945+0000 7f37fa000700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f37e00a3ed0 0x7f37e00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:38.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.946+0000 7f37fa000700 1 -- 172.21.15.6:0/2801559827 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7f37e40d09f0 con 0x7f37fc103480 2024-04-26T01:52:38.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.946+0000 7f3800a00700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f37e00a3ed0 0x7f37e00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:38.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.946+0000 7f380de82b80 1 Processor -- start 2024-04-26T01:52:38.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.946+0000 7f380de82b80 1 -- start start 2024-04-26T01:52:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f380de82b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939d99b0 0x55da939d9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f3800a00700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f37e00a3ed0 0x7f37e00a6390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009be0 tx=0x7f37ec009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f380de82b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939e4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f380de82b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 0x55da939e7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f380de82b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55da9392d020 con 0x55da939d99b0 2024-04-26T01:52:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f380de82b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55da938d38c0 con 0x55da939da2d0 2024-04-26T01:52:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f380de82b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55da93928fe0 con 0x55da939e5180 2024-04-26T01:52:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f37f3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939e4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f3800a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939d99b0 0x55da939d9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f37f3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939e4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41118/0 (socket says 172.21.15.6:41118) 2024-04-26T01:52:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f37f3e00700 1 -- 172.21.15.6:0/1196339846 learned_addr learned my addr 172.21.15.6:0/1196339846 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f3801400700 1 --2- 172.21.15.6:0/1196339846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 0x55da939e7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:38.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f37f3e00700 1 -- 172.21.15.6:0/1196339846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 msgr2=0x55da939e7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:38.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f37f3e00700 1 --2- 172.21.15.6:0/1196339846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 0x55da939e7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:38.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f37f3e00700 1 -- 172.21.15.6:0/1196339846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939d99b0 msgr2=0x55da939d9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:38.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.947+0000 7f37f3e00700 1 --2- 172.21.15.6:0/1196339846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939d99b0 0x55da939d9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:38.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.948+0000 7f37f3e00700 1 -- 172.21.15.6:0/1196339846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da938c9810 con 0x55da939da2d0 2024-04-26T01:52:38.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.948+0000 7f3801400700 1 --2- 172.21.15.6:0/1196339846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 0x55da939e7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:38.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.948+0000 7f37f3e00700 1 --2- 172.21.15.6:0/1196339846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939e4c40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f37e40992b0 tx=0x7f37e409c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c54c47c310df735 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:38.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.948+0000 7f37f2000700 1 -- 172.21.15.6:0/1196339846 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e40a5550 con 0x55da939da2d0 2024-04-26T01:52:38.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.948+0000 7f37f2000700 1 -- 172.21.15.6:0/1196339846 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e409abe0 con 0x55da939da2d0 2024-04-26T01:52:38.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.948+0000 7f37f2000700 1 -- 172.21.15.6:0/1196339846 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e40a5320 con 0x55da939da2d0 2024-04-26T01:52:38.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.949+0000 7f380de82b80 1 -- 172.21.15.6:0/1196339846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 msgr2=0x55da939e4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:38.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.949+0000 7f380de82b80 1 --2- 172.21.15.6:0/1196339846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939e4c40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f37e40992b0 tx=0x7f37e409c8e0 comp rx=0 tx=0).stop 2024-04-26T01:52:38.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.949+0000 7f380de82b80 1 -- 172.21.15.6:0/1196339846 shutdown_connections 2024-04-26T01:52:38.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.949+0000 7f380de82b80 1 --2- 172.21.15.6:0/1196339846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 0x55da939e7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:38.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.949+0000 7f380de82b80 1 --2- 172.21.15.6:0/1196339846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939d99b0 0x55da939d9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:38.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.949+0000 7f380de82b80 1 --2- 172.21.15.6:0/1196339846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939e4c40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:38.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.949+0000 7f380de82b80 1 -- 172.21.15.6:0/1196339846 >> 172.21.15.6:0/1196339846 conn(0x55da93908730 msgr2=0x55da939d5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:38.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.949+0000 7f380de82b80 1 -- 172.21.15.6:0/1196339846 shutdown_connections 2024-04-26T01:52:38.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.949+0000 7f380de82b80 1 -- 172.21.15.6:0/1196339846 wait complete. 2024-04-26T01:52:39.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.950+0000 7f380de82b80 1 -- start start 2024-04-26T01:52:39.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.950+0000 7f380de82b80 1 Processor -- start 2024-04-26T01:52:39.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.951+0000 7f380de82b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939d99b0 0x55da939e4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:39.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.951+0000 7f380de82b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939e1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:39.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.951+0000 7f380de82b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939e5180 0x55da939e21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:39.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.951+0000 7f380de82b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55da939e7ca0 con 0x55da939e5180 2024-04-26T01:52:39.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.951+0000 7f380de82b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55da938e69f0 con 0x55da939da2d0 2024-04-26T01:52:39.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.951+0000 7f380de82b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55da939e7e20 con 0x55da939d99b0 2024-04-26T01:52:39.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.951+0000 7f380de82b80 10 client.0 _ll_register_callbacks cb 0x55da938c58b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:39.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.951+0000 7f37f3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939e1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.951+0000 7f37f3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939e1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41122/0 (socket says 172.21.15.6:41122) 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.951+0000 7f37f3e00700 1 -- 172.21.15.6:0/29119510 learned_addr learned my addr 172.21.15.6:0/29119510 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.951+0000 7f3800a00700 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939d99b0 0x55da939e4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.951+0000 7f3801400700 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939e5180 0x55da939e21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f37f3e00700 1 -- 172.21.15.6:0/29119510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939d99b0 msgr2=0x55da939e4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f37f3e00700 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939d99b0 0x55da939e4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f37f3e00700 1 -- 172.21.15.6:0/29119510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939e5180 msgr2=0x55da939e21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f37f3e00700 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939e5180 0x55da939e21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f37f3e00700 1 -- 172.21.15.6:0/29119510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da9387aa60 con 0x55da939da2d0 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f37f3e00700 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939e1c70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f37e40a5020 tx=0x7f37e40ce030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a3c93f21537f684 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f37eaa00700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f37eaa00700 1 -- 172.21.15.6:0/29119510 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4081060 con 0x55da939da2d0 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f37eaa00700 1 -- 172.21.15.6:0/29119510 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7f37e40a50c0 con 0x55da939da2d0 2024-04-26T01:52:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f380de82b80 1 -- 172.21.15.6:0/29119510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37e409abe0 con 0x55da939da2d0 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f37eaa00700 1 -- 172.21.15.6:0/29119510 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f37e40bb070 con 0x55da939da2d0 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f380de82b80 10 client.25973 fetch_fsmap learned FSMap version 39 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.952+0000 7f380de82b80 1 -- 172.21.15.6:0/29119510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55da938c3c80 con 0x55da939da2d0 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.953+0000 7f37eaa00700 1 -- 172.21.15.6:0/29119510 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e409abe0 con 0x55da939da2d0 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.953+0000 7f37eaa00700 1 -- 172.21.15.6:0/29119510 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8338+0+0 (secure 0 0 0) 0x7f37e4098630 con 0x55da939da2d0 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.953+0000 7f380de82b80 10 client.25973 fetch_fsmap finished waiting for FSMap version 39 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.953+0000 7f380de82b80 10 client.25973 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.953+0000 7f380de82b80 20 client.25973 populate_metadata read hostname 'smithi006' 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.953+0000 7f380de82b80 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55da939f8ce0 0x55da939fb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.953+0000 7f380de82b80 4 client.25973 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.953+0000 7f380de82b80 1 -- 172.21.15.6:0/29119510 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da9393a810 con 0x55da939f8ce0 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.953+0000 7f3800a00700 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55da939f8ce0 0x55da939fb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:39.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.954+0000 7f3800a00700 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55da939f8ce0 0x55da939fb0d0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9968c67bf5dd3cbf server_cookie=ca8834263e1475c in_seq=0 out_seq=0 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.954+0000 7f37eaa00700 10 client.25973 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f37eaa00700 1 -- 172.21.15.6:0/29119510 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da9393a810 con 0x55da939f8ce0 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f37eaa00700 10 client.25973 handle_command_reply: tid=0 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f380de82b80 1 client.25973 shutdown 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f380de82b80 1 -- 172.21.15.6:0/29119510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 msgr2=0x55da939e1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f380de82b80 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939e1c70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f37e40a5020 tx=0x7f37e40ce030 comp rx=0 tx=0).stop 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f380de82b80 1 -- 172.21.15.6:0/29119510 shutdown_connections 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f380de82b80 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939e5180 0x55da939e21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f380de82b80 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55da939f8ce0 0x55da939fb0d0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f380de82b80 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939d99b0 0x55da939e4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f380de82b80 1 --2- 172.21.15.6:0/29119510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939e1c70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f37eaa00700 0 client.25973 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f380de82b80 1 -- 172.21.15.6:0/29119510 >> 172.21.15.6:0/29119510 conn(0x55da93908730 msgr2=0x55da939d5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f380de82b80 1 -- 172.21.15.6:0/29119510 shutdown_connections 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.955+0000 7f380de82b80 1 -- 172.21.15.6:0/29119510 wait complete. 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.956+0000 7f380de82b80 20 client.25973 trim_cache size 0 max 16384 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.978+0000 7f380de82b80 1 Processor -- start 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.978+0000 7f380de82b80 1 -- start start 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.978+0000 7f380de82b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939d99b0 0x55da939fe100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.978+0000 7f380de82b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939fe640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.978+0000 7f380de82b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 0x55da939f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.978+0000 7f380de82b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc1adac0 con 0x55da939d99b0 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.978+0000 7f380de82b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc1ab350 con 0x55da939da2d0 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.978+0000 7f380de82b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc1a91c0 con 0x55da939e5180 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.978+0000 7f3801400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 0x55da939f4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.978+0000 7f3800a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939d99b0 0x55da939fe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.978+0000 7f3801400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 0x55da939f4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37582/0 (socket says 172.21.15.6:37582) 2024-04-26T01:52:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.978+0000 7f3801400700 1 -- 172.21.15.6:0/337663859 learned_addr learned my addr 172.21.15.6:0/337663859 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.979+0000 7f37f3e00700 1 --2- 172.21.15.6:0/337663859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.979+0000 7f3801400700 1 -- 172.21.15.6:0/337663859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 msgr2=0x55da939fe640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.979+0000 7f3801400700 1 --2- 172.21.15.6:0/337663859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939fe640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.979+0000 7f3801400700 1 -- 172.21.15.6:0/337663859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939d99b0 msgr2=0x55da939fe100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.979+0000 7f3801400700 1 --2- 172.21.15.6:0/337663859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939d99b0 0x55da939fe100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.979+0000 7f3801400700 1 -- 172.21.15.6:0/337663859 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da939e2120 con 0x55da939e5180 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.979+0000 7f3800a00700 1 --2- 172.21.15.6:0/337663859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939d99b0 0x55da939fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.979+0000 7f3801400700 1 --2- 172.21.15.6:0/337663859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 0x55da939f4c50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f37f400b0a0 tx=0x7f37f40095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32bffde26a6b0254 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.980+0000 7f37f1600700 1 -- 172.21.15.6:0/337663859 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f400bbb0 con 0x55da939e5180 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.980+0000 7f37f1600700 1 -- 172.21.15.6:0/337663859 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37f400bd50 con 0x55da939e5180 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.980+0000 7f37f1600700 1 -- 172.21.15.6:0/337663859 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4005040 con 0x55da939e5180 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.981+0000 7f380de82b80 1 -- 172.21.15.6:0/337663859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 msgr2=0x55da939f4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.981+0000 7f380de82b80 1 --2- 172.21.15.6:0/337663859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 0x55da939f4c50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f37f400b0a0 tx=0x7f37f40095b0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.981+0000 7f380de82b80 1 -- 172.21.15.6:0/337663859 shutdown_connections 2024-04-26T01:52:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.981+0000 7f380de82b80 1 --2- 172.21.15.6:0/337663859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939e5180 0x55da939f4c50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.981+0000 7f380de82b80 1 --2- 172.21.15.6:0/337663859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939d99b0 0x55da939fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.981+0000 7f380de82b80 1 --2- 172.21.15.6:0/337663859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da939fe640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.981+0000 7f380de82b80 1 -- 172.21.15.6:0/337663859 >> 172.21.15.6:0/337663859 conn(0x55da93908730 msgr2=0x55da939f4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.981+0000 7f380de82b80 1 -- 172.21.15.6:0/337663859 shutdown_connections 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.981+0000 7f380de82b80 1 -- 172.21.15.6:0/337663859 wait complete. 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.981+0000 7f380de82b80 1 -- start start 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.982+0000 7f380de82b80 1 Processor -- start 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.982+0000 7f380de82b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939d99b0 0x55da93a58c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.982+0000 7f380de82b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da93a591b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.982+0000 7f380de82b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939e5180 0x55da93a53070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.982+0000 7f380de82b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc1a9810 con 0x55da939e5180 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.982+0000 7f380de82b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc1aac80 con 0x55da939da2d0 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.982+0000 7f380de82b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc1a9ee0 con 0x55da939d99b0 2024-04-26T01:52:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f380de82b80 10 client.0 _ll_register_callbacks cb 0x55da938c58b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f37f3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da93a591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f3800a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939d99b0 0x55da93a58c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f37f3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da93a591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41154/0 (socket says 172.21.15.6:41154) 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f3800a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939d99b0 0x55da93a58c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37592/0 (socket says 172.21.15.6:37592) 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f37f3e00700 1 -- 172.21.15.6:0/2470559859 learned_addr learned my addr 172.21.15.6:0/2470559859 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f3801400700 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939e5180 0x55da93a53070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f37f3e00700 1 -- 172.21.15.6:0/2470559859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939d99b0 msgr2=0x55da93a58c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f37f3e00700 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939d99b0 0x55da93a58c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f37f3e00700 1 -- 172.21.15.6:0/2470559859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939e5180 msgr2=0x55da93a53070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f37f3e00700 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939e5180 0x55da93a53070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f37f3e00700 1 -- 172.21.15.6:0/2470559859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da93a53780 con 0x55da939da2d0 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f3801400700 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939e5180 0x55da93a53070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f37f3e00700 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da93a591b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f37e4098d40 tx=0x7f37e4087350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35064a0e0bb74bbc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.983+0000 7f37eaa00700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:52:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.984+0000 7f37eaa00700 1 -- 172.21.15.6:0/2470559859 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4081070 con 0x55da939da2d0 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.984+0000 7f37eaa00700 1 -- 172.21.15.6:0/2470559859 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7f37e409b7f0 con 0x55da939da2d0 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.984+0000 7f380de82b80 1 -- 172.21.15.6:0/2470559859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37f400bd50 con 0x55da939da2d0 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.984+0000 7f37eaa00700 1 -- 172.21.15.6:0/2470559859 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f37e40a5550 con 0x55da939da2d0 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.984+0000 7f380de82b80 10 client.25979 fetch_fsmap learned FSMap version 39 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.984+0000 7f380de82b80 1 -- 172.21.15.6:0/2470559859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55da93a60440 con 0x55da939da2d0 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.984+0000 7f37eaa00700 1 -- 172.21.15.6:0/2470559859 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37f400bd50 con 0x55da939da2d0 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.984+0000 7f37eaa00700 1 -- 172.21.15.6:0/2470559859 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8338+0+0 (secure 0 0 0) 0x7f37e40bbbc0 con 0x55da939da2d0 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.984+0000 7f380de82b80 10 client.25979 fetch_fsmap finished waiting for FSMap version 39 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.984+0000 7f380de82b80 10 client.25979 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.984+0000 7f380de82b80 20 client.25979 populate_metadata read hostname 'smithi006' 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.985+0000 7f380de82b80 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55da93a60950 0x55da93a62d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.985+0000 7f380de82b80 4 client.25979 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 75.9, "await_for": 42.0}] 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.985+0000 7f380de82b80 1 -- 172.21.15.6:0/2470559859 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 75.9, "await_for": 42.0}) -- 0x55da939348b0 con 0x55da93a60950 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.985+0000 7f3800a00700 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55da93a60950 0x55da93a62d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.985+0000 7f3800a00700 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55da93a60950 0x55da93a62d70 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52fae4ae5f8c801 server_cookie=317dd43fb1fc8eeb in_seq=0 out_seq=0 2024-04-26T01:52:39.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:38.985+0000 7f37eaa00700 10 client.25979 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.060+0000 7f37eaa00700 1 -- 172.21.15.6:0/2470559859 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55da939348b0 con 0x55da93a60950 2024-04-26T01:52:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.060+0000 7f37eaa00700 10 client.25979 handle_command_reply: tid=0 2024-04-26T01:52:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.060+0000 7f380de82b80 1 client.25979 shutdown 2024-04-26T01:52:39.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.061+0000 7f380de82b80 1 -- 172.21.15.6:0/2470559859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 msgr2=0x55da93a591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:39.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.061+0000 7f380de82b80 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da93a591b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f37e4098d40 tx=0x7f37e4087350 comp rx=0 tx=0).stop 2024-04-26T01:52:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.061+0000 7f380de82b80 1 -- 172.21.15.6:0/2470559859 shutdown_connections 2024-04-26T01:52:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.061+0000 7f380de82b80 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55da939e5180 0x55da93a53070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.061+0000 7f380de82b80 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55da93a60950 0x55da93a62d70 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.061+0000 7f380de82b80 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55da939d99b0 0x55da93a58c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.061+0000 7f380de82b80 1 --2- 172.21.15.6:0/2470559859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da939da2d0 0x55da93a591b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.061+0000 7f380de82b80 1 -- 172.21.15.6:0/2470559859 >> 172.21.15.6:0/2470559859 conn(0x55da93908730 msgr2=0x55da939f4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.061+0000 7f37eaa00700 0 client.25979 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.061+0000 7f380de82b80 1 -- 172.21.15.6:0/2470559859 shutdown_connections 2024-04-26T01:52:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.062+0000 7f380de82b80 1 -- 172.21.15.6:0/2470559859 wait complete. 2024-04-26T01:52:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.062+0000 7f380de82b80 20 client.25979 trim_cache size 0 max 16384 2024-04-26T01:52:39.061 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:52:39.061 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 39, 2024-04-26T01:52:39.061 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:52:39.061 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 18, 2024-04-26T01:52:39.061 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:52:39.061 INFO:teuthology.orchestra.run.smithi006.stdout: "5f38731d": { 2024-04-26T01:52:39.062 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 18, 2024-04-26T01:52:39.062 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:52:39.062 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:52:39.062 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:52:39.062 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:52:39.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:52:39.062 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:52:39.062 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 75.9, 2024-04-26T01:52:39.062 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:52:39.062 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:52:39.062 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:52:39.062 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:52:39.063 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:52:39.063 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:52:39.063 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:52:39.063 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:52:39.063 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:52:39.063 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:52:39.063 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:52:39.063 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:52:39.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.063+0000 7f3803400700 1 -- 172.21.15.6:0/2801559827 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f37e00a3ed0 msgr2=0x7f37e00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:39.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.063+0000 7f3803400700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f37e00a3ed0 0x7f37e00a6390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009be0 tx=0x7f37ec009260 comp rx=0 tx=0).stop 2024-04-26T01:52:39.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.063+0000 7f3803400700 1 -- 172.21.15.6:0/2801559827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37fc103480 msgr2=0x7f37fc1995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:39.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.063+0000 7f3803400700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37fc103480 0x7f37fc1995a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f37e400e9a0 tx=0x7f37e400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.063+0000 7f3803400700 1 -- 172.21.15.6:0/2801559827 shutdown_connections 2024-04-26T01:52:39.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.063+0000 7f3803400700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f37e00a3ed0 0x7f37e00a6390 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.063+0000 7f3803400700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc108ab0 0x7f37fc19d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.063+0000 7f3803400700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37fc102ac0 0x7f37fc199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.063+0000 7f3803400700 1 --2- 172.21.15.6:0/2801559827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37fc103480 0x7f37fc1995a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:39.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.064+0000 7f3803400700 1 -- 172.21.15.6:0/2801559827 >> 172.21.15.6:0/2801559827 conn(0x7f37fc079bf0 msgr2=0x7f37fc101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:39.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.064+0000 7f3803400700 1 -- 172.21.15.6:0/2801559827 shutdown_connections 2024-04-26T01:52:39.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:39.064+0000 7f3803400700 1 -- 172.21.15.6:0/2801559827 wait complete. 2024-04-26T01:52:39.589 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 4.21 MB/sec execute 183 sec latency 605.878 ms 2024-04-26T01:52:39.896 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5f38731d', quiesce took 7.9 seconds. Will release after: 15.9 2024-04-26T01:52:40.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:40 smithi006 ceph-mon[28419]: pgmap v211: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1023 KiB/s wr, 150 op/s 2024-04-26T01:52:40.589 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 4.18 MB/sec execute 184 sec latency 1606.038 ms 2024-04-26T01:52:40.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:40 smithi188 ceph-mon[32751]: pgmap v211: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1023 KiB/s wr, 150 op/s 2024-04-26T01:52:40.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:40 smithi029 ceph-mon[32775]: pgmap v211: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1023 KiB/s wr, 150 op/s 2024-04-26T01:52:41.589 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 4.16 MB/sec execute 185 sec latency 2606.188 ms 2024-04-26T01:52:42.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:42 smithi006 ceph-mon[28419]: pgmap v212: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 194 op/s 2024-04-26T01:52:42.589 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 4.14 MB/sec execute 186 sec latency 3606.308 ms 2024-04-26T01:52:42.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:42 smithi188 ceph-mon[32751]: pgmap v212: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 194 op/s 2024-04-26T01:52:42.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:42 smithi029 ceph-mon[32775]: pgmap v212: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 194 op/s 2024-04-26T01:52:43.590 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 4.12 MB/sec execute 187 sec latency 4606.457 ms 2024-04-26T01:52:44.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:44 smithi006 ceph-mon[28419]: pgmap v213: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 840 KiB/s wr, 134 op/s 2024-04-26T01:52:44.590 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 4.09 MB/sec execute 188 sec latency 5606.615 ms 2024-04-26T01:52:44.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:44 smithi188 ceph-mon[32751]: pgmap v213: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 840 KiB/s wr, 134 op/s 2024-04-26T01:52:44.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:44 smithi029 ceph-mon[32775]: pgmap v213: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 840 KiB/s wr, 134 op/s 2024-04-26T01:52:45.590 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 4.07 MB/sec execute 189 sec latency 6606.775 ms 2024-04-26T01:52:46.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:46 smithi006 ceph-mon[28419]: pgmap v214: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 840 KiB/s wr, 134 op/s 2024-04-26T01:52:46.590 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 4.05 MB/sec execute 190 sec latency 7606.913 ms 2024-04-26T01:52:46.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:46 smithi188 ceph-mon[32751]: pgmap v214: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 840 KiB/s wr, 134 op/s 2024-04-26T01:52:46.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:46 smithi029 ceph-mon[32775]: pgmap v214: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 840 KiB/s wr, 134 op/s 2024-04-26T01:52:47.590 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 4.03 MB/sec execute 191 sec latency 8607.073 ms 2024-04-26T01:52:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.533+0000 7f1d75a00640 1 -- 172.21.15.6:0/63670054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d701043c0 msgr2=0x7f1d701047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.533+0000 7f1d75a00640 1 --2- 172.21.15.6:0/63670054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d701043c0 0x7f1d701047a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009900 tx=0x7f1d6002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.534+0000 7f1d75a00640 1 -- 172.21.15.6:0/63670054 shutdown_connections 2024-04-26T01:52:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.534+0000 7f1d75a00640 1 --2- 172.21.15.6:0/63670054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d70109490 0x7f1d70111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.534+0000 7f1d75a00640 1 --2- 172.21.15.6:0/63670054 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d70104d70 0x7f1d70108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.534+0000 7f1d75a00640 1 --2- 172.21.15.6:0/63670054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d701043c0 0x7f1d701047a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.534+0000 7f1d75a00640 1 -- 172.21.15.6:0/63670054 >> 172.21.15.6:0/63670054 conn(0x7f1d700ffc20 msgr2=0x7f1d70102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.534+0000 7f1d75a00640 1 -- 172.21.15.6:0/63670054 shutdown_connections 2024-04-26T01:52:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.534+0000 7f1d75a00640 1 -- 172.21.15.6:0/63670054 wait complete. 2024-04-26T01:52:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.535+0000 7f1d75a00640 1 Processor -- start 2024-04-26T01:52:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.535+0000 7f1d75a00640 1 -- start start 2024-04-26T01:52:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.535+0000 7f1d75a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d701043c0 0x7f1d701a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.535+0000 7f1d75a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d70109490 0x7f1d701a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.535+0000 7f1d75a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d7019a7e0 0x7f1d7019ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.535+0000 7f1d75a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1d70114420 con 0x7f1d7019a7e0 2024-04-26T01:52:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d75a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1d701142a0 con 0x7f1d70109490 2024-04-26T01:52:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d75a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1d701145a0 con 0x7f1d701043c0 2024-04-26T01:52:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d6f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d7019a7e0 0x7f1d7019ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d6ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d701043c0 0x7f1d701a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d6f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d7019a7e0 0x7f1d7019ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53166/0 (socket says 172.21.15.6:53166) 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d6f400640 1 -- 172.21.15.6:0/237923889 learned_addr learned my addr 172.21.15.6:0/237923889 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d67e00640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d70109490 0x7f1d701a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d6ea00640 1 -- 172.21.15.6:0/237923889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d70109490 msgr2=0x7f1d701a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d6ea00640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d70109490 0x7f1d701a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d6ea00640 1 -- 172.21.15.6:0/237923889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d7019a7e0 msgr2=0x7f1d7019ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d6ea00640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d7019a7e0 0x7f1d7019ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d6ea00640 1 -- 172.21.15.6:0/237923889 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7019b370 con 0x7f1d701043c0 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.536+0000 7f1d67e00640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d70109490 0x7f1d701a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.537+0000 7f1d6ea00640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d701043c0 0x7f1d701a0670 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1d6002a9e0 tx=0x7f1d60003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.537+0000 7f1d67400640 1 -- 172.21.15.6:0/237923889 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d6003f070 con 0x7f1d701043c0 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.537+0000 7f1d67400640 1 -- 172.21.15.6:0/237923889 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d60031b50 con 0x7f1d701043c0 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.537+0000 7f1d75a00640 1 -- 172.21.15.6:0/237923889 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7019b600 con 0x7f1d701043c0 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.537+0000 7f1d67400640 1 -- 172.21.15.6:0/237923889 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d60035030 con 0x7f1d701043c0 2024-04-26T01:52:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.537+0000 7f1d75a00640 1 -- 172.21.15.6:0/237923889 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d701a77b0 con 0x7f1d701043c0 2024-04-26T01:52:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.540+0000 7f1d787f5740 1 Processor -- start 2024-04-26T01:52:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d787f5740 1 -- start start 2024-04-26T01:52:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d67400640 1 -- 172.21.15.6:0/237923889 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f1d60042070 con 0x7f1d701043c0 2024-04-26T01:52:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d787f5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2e5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d787f5740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2f04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d787f5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf2f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d787f5740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55e7cf24b900 con 0x55e7cf2e6140 2024-04-26T01:52:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d787f5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e7cf1f1ef0 con 0x55e7cf2f0a00 2024-04-26T01:52:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d787f5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e7cf2478c0 con 0x55e7cf2e5820 2024-04-26T01:52:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d6ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2f04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d67e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf2f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d67e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf2f2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:57300/0 (socket says 172.21.15.6:57300) 2024-04-26T01:52:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d6ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2f04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53172/0 (socket says 172.21.15.6:53172) 2024-04-26T01:52:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d67e00640 1 -- 172.21.15.6:0/2572386697 learned_addr learned my addr 172.21.15.6:0/2572386697 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.541+0000 7f1d6f400640 1 --2- 172.21.15.6:0/2572386697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2e5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.542+0000 7f1d67e00640 1 -- 172.21.15.6:0/2572386697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 msgr2=0x55e7cf2e5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.542+0000 7f1d67e00640 1 --2- 172.21.15.6:0/2572386697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2e5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.542+0000 7f1d67e00640 1 -- 172.21.15.6:0/2572386697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 msgr2=0x55e7cf2f04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.542+0000 7f1d67e00640 1 --2- 172.21.15.6:0/2572386697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2f04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.542+0000 7f1d67e00640 1 -- 172.21.15.6:0/2572386697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7cef5e210 con 0x55e7cf2f0a00 2024-04-26T01:52:48.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.542+0000 7f1d67e00640 1 --2- 172.21.15.6:0/2572386697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf2f2df0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1d58011470 tx=0x7f1d58011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f5a7befbd95c051 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.542+0000 7f1d5fe00640 1 -- 172.21.15.6:0/2572386697 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5801e070 con 0x55e7cf2f0a00 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.542+0000 7f1d5fe00640 1 -- 172.21.15.6:0/2572386697 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d580035f0 con 0x55e7cf2f0a00 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.542+0000 7f1d5fe00640 1 -- 172.21.15.6:0/2572386697 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5800f860 con 0x55e7cf2f0a00 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.542+0000 7f1d67400640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1d540a3d90 0x7f1d540a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.543+0000 7f1d67400640 1 -- 172.21.15.6:0/237923889 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7f1d600f0dd0 con 0x7f1d701043c0 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.543+0000 7f1d6f400640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1d540a3d90 0x7f1d540a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.543+0000 7f1d787f5740 1 -- 172.21.15.6:0/2572386697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 msgr2=0x55e7cf2f2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.543+0000 7f1d787f5740 1 --2- 172.21.15.6:0/2572386697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf2f2df0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1d58011470 tx=0x7f1d58011930 comp rx=0 tx=0).stop 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.543+0000 7f1d787f5740 1 -- 172.21.15.6:0/2572386697 shutdown_connections 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.543+0000 7f1d787f5740 1 --2- 172.21.15.6:0/2572386697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2f04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.543+0000 7f1d787f5740 1 --2- 172.21.15.6:0/2572386697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf2f2df0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.543+0000 7f1d787f5740 1 --2- 172.21.15.6:0/2572386697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2e5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.543+0000 7f1d787f5740 1 -- 172.21.15.6:0/2572386697 >> 172.21.15.6:0/2572386697 conn(0x55e7cf2e1d30 msgr2=0x55e7cf2e2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.543+0000 7f1d787f5740 1 -- 172.21.15.6:0/2572386697 shutdown_connections 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.543+0000 7f1d787f5740 1 -- 172.21.15.6:0/2572386697 wait complete. 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.544+0000 7f1d6f400640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1d540a3d90 0x7f1d540a6250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009270 tx=0x7f1d68009680 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.544+0000 7f1d787f5740 1 -- start start 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.544+0000 7f1d787f5740 1 Processor -- start 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d787f5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2ed930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d787f5740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2ede70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d787f5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf2ee3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d787f5740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55e7cf2f3630 con 0x55e7cf2e6140 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d787f5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e7cf205310 con 0x55e7cf2f0a00 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d787f5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e7cf2f37b0 con 0x55e7cf2e5820 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d787f5740 10 client.0 _ll_register_callbacks cb 0x55e7cf1d3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d6ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2ed930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d67e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2ede70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d6ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2ed930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:53348/0 (socket says 172.21.15.6:53348) 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d6ea00640 1 -- 172.21.15.6:0/898566873 learned_addr learned my addr 172.21.15.6:0/898566873 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d6f400640 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf2ee3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d6f400640 1 -- 172.21.15.6:0/898566873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 msgr2=0x55e7cf2ed930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d6f400640 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2ed930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d6f400640 1 -- 172.21.15.6:0/898566873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 msgr2=0x55e7cf2ede70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d6f400640 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2ede70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d6f400640 1 -- 172.21.15.6:0/898566873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7cf02a620 con 0x55e7cf2f0a00 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d6ea00640 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2ed930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.545+0000 7f1d67e00640 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2ede70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.546+0000 7f1d6f400640 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf2ee3b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1d680117a0 tx=0x7f1d68004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c5fb8698f0a9fef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.546+0000 7f1d5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.546+0000 7f1d5cc00640 1 -- 172.21.15.6:0/898566873 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d68004b20 con 0x55e7cf2f0a00 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.546+0000 7f1d787f5740 1 -- 172.21.15.6:0/898566873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d580035f0 con 0x55e7cf2f0a00 2024-04-26T01:52:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.546+0000 7f1d5cc00640 1 -- 172.21.15.6:0/898566873 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7f1d68004cc0 con 0x55e7cf2f0a00 2024-04-26T01:52:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.546+0000 7f1d5cc00640 1 -- 172.21.15.6:0/898566873 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1d680209a0 con 0x55e7cf2f0a00 2024-04-26T01:52:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.547+0000 7f1d787f5740 10 client.25991 fetch_fsmap learned FSMap version 39 2024-04-26T01:52:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.547+0000 7f1d787f5740 1 -- 172.21.15.6:0/898566873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e7cf2ff950 con 0x55e7cf2f0a00 2024-04-26T01:52:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.547+0000 7f1d5cc00640 1 -- 172.21.15.6:0/898566873 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d580035f0 con 0x55e7cf2f0a00 2024-04-26T01:52:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.547+0000 7f1d5cc00640 1 -- 172.21.15.6:0/898566873 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8338+0+0 (secure 0 0 0) 0x7f1d68027070 con 0x55e7cf2f0a00 2024-04-26T01:52:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.547+0000 7f1d787f5740 10 client.25991 fetch_fsmap finished waiting for FSMap version 39 2024-04-26T01:52:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.547+0000 7f1d787f5740 10 client.25991 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.547+0000 7f1d787f5740 20 client.25991 populate_metadata read hostname 'smithi006' 2024-04-26T01:52:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.547+0000 7f1d787f5740 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e7cf300e10 0x55e7cf303200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.547+0000 7f1d787f5740 4 client.25991 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.547+0000 7f1d787f5740 1 -- 172.21.15.6:0/898566873 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7cf25f8b0 con 0x55e7cf300e10 2024-04-26T01:52:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.548+0000 7f1d6ea00640 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e7cf300e10 0x55e7cf303200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.548+0000 7f1d6ea00640 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e7cf300e10 0x55e7cf303200 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c8390ce7122dc46 server_cookie=4dc52494dcb62871 in_seq=0 out_seq=0 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.548+0000 7f1d5cc00640 10 client.25991 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.549+0000 7f1d5cc00640 1 -- 172.21.15.6:0/898566873 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e7cf25f8b0 con 0x55e7cf300e10 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.549+0000 7f1d5cc00640 10 client.25991 handle_command_reply: tid=0 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.549+0000 7f1d787f5740 1 client.25991 shutdown 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.549+0000 7f1d787f5740 1 -- 172.21.15.6:0/898566873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 msgr2=0x55e7cf2ee3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.549+0000 7f1d787f5740 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf2ee3b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1d680117a0 tx=0x7f1d68004a40 comp rx=0 tx=0).stop 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.549+0000 7f1d787f5740 1 -- 172.21.15.6:0/898566873 shutdown_connections 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.549+0000 7f1d787f5740 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e7cf300e10 0x55e7cf303200 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.549+0000 7f1d787f5740 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2ede70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.549+0000 7f1d787f5740 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf2ee3b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.549+0000 7f1d787f5740 1 --2- 172.21.15.6:0/898566873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2ed930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.549+0000 7f1d787f5740 1 -- 172.21.15.6:0/898566873 >> 172.21.15.6:0/898566873 conn(0x55e7cf2e1d30 msgr2=0x55e7cf2e7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.549+0000 7f1d5cc00640 0 client.25991 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.550+0000 7f1d787f5740 1 -- 172.21.15.6:0/898566873 shutdown_connections 2024-04-26T01:52:48.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.550+0000 7f1d787f5740 1 -- 172.21.15.6:0/898566873 wait complete. 2024-04-26T01:52:48.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.550+0000 7f1d787f5740 20 client.25991 trim_cache size 0 max 16384 2024-04-26T01:52:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.566+0000 7f1d787f5740 1 Processor -- start 2024-04-26T01:52:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.566+0000 7f1d787f5740 1 -- start start 2024-04-26T01:52:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.566+0000 7f1d787f5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2fc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.566+0000 7f1d787f5740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.566+0000 7f1d787f5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf308b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.566+0000 7f1d787f5740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1d701b27d0 con 0x55e7cf2e6140 2024-04-26T01:52:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.566+0000 7f1d787f5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1d701b00a0 con 0x55e7cf2f0a00 2024-04-26T01:52:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.566+0000 7f1d787f5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1d701adf10 con 0x55e7cf2e5820 2024-04-26T01:52:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.567+0000 7f1d67e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.567+0000 7f1d67e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2e81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53204/0 (socket says 172.21.15.6:53204) 2024-04-26T01:52:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.567+0000 7f1d67e00640 1 -- 172.21.15.6:0/66347113 learned_addr learned my addr 172.21.15.6:0/66347113 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.567+0000 7f1d6f400640 1 --2- 172.21.15.6:0/66347113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf308b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.567+0000 7f1d6ea00640 1 --2- 172.21.15.6:0/66347113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2fc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.567+0000 7f1d67e00640 1 -- 172.21.15.6:0/66347113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 msgr2=0x55e7cf2fc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.567+0000 7f1d67e00640 1 --2- 172.21.15.6:0/66347113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2fc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.567+0000 7f1d67e00640 1 -- 172.21.15.6:0/66347113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 msgr2=0x55e7cf308b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.567+0000 7f1d67e00640 1 --2- 172.21.15.6:0/66347113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf308b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.567+0000 7f1d67e00640 1 -- 172.21.15.6:0/66347113 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7cf303570 con 0x55e7cf2e6140 2024-04-26T01:52:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.567+0000 7f1d6f400640 1 --2- 172.21.15.6:0/66347113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf308b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.568+0000 7f1d67e00640 1 --2- 172.21.15.6:0/66347113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2e81e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1d580058c0 tx=0x7f1d58005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7a339423b8f9d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.568+0000 7f1d5f400640 1 -- 172.21.15.6:0/66347113 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d58029070 con 0x55e7cf2e6140 2024-04-26T01:52:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.568+0000 7f1d5f400640 1 -- 172.21.15.6:0/66347113 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d580055b0 con 0x55e7cf2e6140 2024-04-26T01:52:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.568+0000 7f1d5f400640 1 -- 172.21.15.6:0/66347113 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5801e040 con 0x55e7cf2e6140 2024-04-26T01:52:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.568+0000 7f1d787f5740 1 -- 172.21.15.6:0/66347113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 msgr2=0x55e7cf2e81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.568+0000 7f1d787f5740 1 --2- 172.21.15.6:0/66347113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2e81e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1d580058c0 tx=0x7f1d58005850 comp rx=0 tx=0).stop 2024-04-26T01:52:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.569+0000 7f1d787f5740 1 -- 172.21.15.6:0/66347113 shutdown_connections 2024-04-26T01:52:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.569+0000 7f1d787f5740 1 --2- 172.21.15.6:0/66347113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2e6140 0x55e7cf2e81e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.569+0000 7f1d787f5740 1 --2- 172.21.15.6:0/66347113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2f0a00 0x55e7cf308b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.569+0000 7f1d787f5740 1 --2- 172.21.15.6:0/66347113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2fc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.569+0000 7f1d787f5740 1 -- 172.21.15.6:0/66347113 >> 172.21.15.6:0/66347113 conn(0x55e7cf2e1d30 msgr2=0x55e7cf2e8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:48.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:48 smithi006 ceph-mon[28419]: pgmap v215: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.1 MiB/s wr, 144 op/s 2024-04-26T01:52:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.569+0000 7f1d787f5740 1 -- 172.21.15.6:0/66347113 shutdown_connections 2024-04-26T01:52:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.569+0000 7f1d787f5740 1 -- 172.21.15.6:0/66347113 wait complete. 2024-04-26T01:52:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.569+0000 7f1d787f5740 1 -- start start 2024-04-26T01:52:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.569+0000 7f1d787f5740 1 Processor -- start 2024-04-26T01:52:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.570+0000 7f1d787f5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.570+0000 7f1d787f5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2e6140 0x55e7cf2f6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.570+0000 7f1d787f5740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2f0a00 0x55e7cf365380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.570+0000 7f1d787f5740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1d701ae560 con 0x55e7cf2f0a00 2024-04-26T01:52:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.570+0000 7f1d787f5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1d701af9d0 con 0x55e7cf2e6140 2024-04-26T01:52:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.570+0000 7f1d787f5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1d701aec30 con 0x55e7cf2e5820 2024-04-26T01:52:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.570+0000 7f1d787f5740 10 client.0 _ll_register_callbacks cb 0x7f1d68011660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.570+0000 7f1d6ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.570+0000 7f1d6f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2f0a00 0x55e7cf365380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.570+0000 7f1d6ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2f5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:53366/0 (socket says 172.21.15.6:53366) 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.570+0000 7f1d6ea00640 1 -- 172.21.15.6:0/2542329943 learned_addr learned my addr 172.21.15.6:0/2542329943 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.570+0000 7f1d67e00640 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2e6140 0x55e7cf2f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.571+0000 7f1d67e00640 1 -- 172.21.15.6:0/2542329943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 msgr2=0x55e7cf2f5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.571+0000 7f1d67e00640 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2f5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.571+0000 7f1d67e00640 1 -- 172.21.15.6:0/2542329943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2f0a00 msgr2=0x55e7cf365380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.571+0000 7f1d67e00640 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2f0a00 0x55e7cf365380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.571+0000 7f1d67e00640 1 -- 172.21.15.6:0/2542329943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7cf3658c0 con 0x55e7cf2e6140 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.571+0000 7f1d6ea00640 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2f5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.571+0000 7f1d6f400640 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2f0a00 0x55e7cf365380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.571+0000 7f1d67e00640 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2e6140 0x55e7cf2f6130 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55e7cf2e28a0 tx=0x7f1d58005540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a2868125cf141fc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.571+0000 7f1d5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.571+0000 7f1d5cc00640 1 -- 172.21.15.6:0/2542329943 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d58031030 con 0x55e7cf2e6140 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.571+0000 7f1d5cc00640 1 -- 172.21.15.6:0/2542329943 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7f1d58016040 con 0x55e7cf2e6140 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.571+0000 7f1d787f5740 1 -- 172.21.15.6:0/2542329943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d580055b0 con 0x55e7cf2e6140 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.572+0000 7f1d5cc00640 1 -- 172.21.15.6:0/2542329943 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1d5803b050 con 0x55e7cf2e6140 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.572+0000 7f1d787f5740 10 client.25997 fetch_fsmap learned FSMap version 39 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.572+0000 7f1d787f5740 1 -- 172.21.15.6:0/2542329943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e7cf366b80 con 0x55e7cf2e6140 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.572+0000 7f1d5cc00640 1 -- 172.21.15.6:0/2542329943 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d580055b0 con 0x55e7cf2e6140 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.572+0000 7f1d5cc00640 1 -- 172.21.15.6:0/2542329943 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8338+0+0 (secure 0 0 0) 0x7f1d58025460 con 0x55e7cf2e6140 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.572+0000 7f1d787f5740 10 client.25997 fetch_fsmap finished waiting for FSMap version 39 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.572+0000 7f1d787f5740 10 client.25997 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.572+0000 7f1d787f5740 20 client.25997 populate_metadata read hostname 'smithi006' 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.572+0000 7f1d787f5740 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e7cf3670a0 0x55e7cf369490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.572+0000 7f1d787f5740 4 client.25997 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.572+0000 7f1d787f5740 1 -- 172.21.15.6:0/2542329943 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e7cf259320 con 0x55e7cf3670a0 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.573+0000 7f1d6ea00640 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e7cf3670a0 0x55e7cf369490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.573+0000 7f1d6ea00640 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e7cf3670a0 0x55e7cf369490 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=874f1a049730293 server_cookie=e365ec48e36321d8 in_seq=0 out_seq=0 2024-04-26T01:52:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.573+0000 7f1d5cc00640 10 client.25997 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:48.590 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 4.01 MB/sec execute 192 sec latency 9607.153 ms 2024-04-26T01:52:48.608 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.610+0000 7f1d5cc00640 1 -- 172.21.15.6:0/2542329943 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e7cf259320 con 0x55e7cf3670a0 2024-04-26T01:52:48.608 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.610+0000 7f1d5cc00640 10 client.25997 handle_command_reply: tid=0 2024-04-26T01:52:48.608 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.610+0000 7f1d787f5740 1 client.25997 shutdown 2024-04-26T01:52:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.611+0000 7f1d787f5740 1 -- 172.21.15.6:0/2542329943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2e6140 msgr2=0x55e7cf2f6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.611+0000 7f1d787f5740 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2e6140 0x55e7cf2f6130 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55e7cf2e28a0 tx=0x7f1d58005540 comp rx=0 tx=0).stop 2024-04-26T01:52:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.611+0000 7f1d787f5740 1 -- 172.21.15.6:0/2542329943 shutdown_connections 2024-04-26T01:52:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.611+0000 7f1d787f5740 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e7cf3670a0 0x55e7cf369490 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.611+0000 7f1d787f5740 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e7cf2f0a00 0x55e7cf365380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.611+0000 7f1d787f5740 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e7cf2e6140 0x55e7cf2f6130 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.611+0000 7f1d787f5740 1 --2- 172.21.15.6:0/2542329943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e7cf2e5820 0x55e7cf2f5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.611+0000 7f1d5cc00640 0 client.25997 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:52:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.611+0000 7f1d787f5740 1 -- 172.21.15.6:0/2542329943 >> 172.21.15.6:0/2542329943 conn(0x55e7cf2e1d30 msgr2=0x55e7cf2e0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.611+0000 7f1d787f5740 1 -- 172.21.15.6:0/2542329943 shutdown_connections 2024-04-26T01:52:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.611+0000 7f1d787f5740 1 -- 172.21.15.6:0/2542329943 wait complete. 2024-04-26T01:52:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.611+0000 7f1d787f5740 20 client.25997 trim_cache size 0 max 16384 2024-04-26T01:52:48.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:52:48.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:52:48.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:52:48.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:52:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.612+0000 7f1d75a00640 1 -- 172.21.15.6:0/237923889 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1d540a3d90 msgr2=0x7f1d540a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.612+0000 7f1d75a00640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1d540a3d90 0x7f1d540a6250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009270 tx=0x7f1d68009680 comp rx=0 tx=0).stop 2024-04-26T01:52:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.612+0000 7f1d75a00640 1 -- 172.21.15.6:0/237923889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d701043c0 msgr2=0x7f1d701a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.612+0000 7f1d75a00640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d701043c0 0x7f1d701a0670 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1d6002a9e0 tx=0x7f1d60003b90 comp rx=0 tx=0).stop 2024-04-26T01:52:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.612+0000 7f1d75a00640 1 -- 172.21.15.6:0/237923889 shutdown_connections 2024-04-26T01:52:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.612+0000 7f1d75a00640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1d540a3d90 0x7f1d540a6250 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.612+0000 7f1d75a00640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d7019a7e0 0x7f1d7019ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.612+0000 7f1d75a00640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d70109490 0x7f1d701a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.612+0000 7f1d75a00640 1 --2- 172.21.15.6:0/237923889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d701043c0 0x7f1d701a0670 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.612+0000 7f1d75a00640 1 -- 172.21.15.6:0/237923889 >> 172.21.15.6:0/237923889 conn(0x7f1d700ffc20 msgr2=0x7f1d70100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.612+0000 7f1d75a00640 1 -- 172.21.15.6:0/237923889 shutdown_connections 2024-04-26T01:52:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:52:48.613+0000 7f1d75a00640 1 -- 172.21.15.6:0/237923889 wait complete. 2024-04-26T01:52:48.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:48 smithi188 ceph-mon[32751]: pgmap v215: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.1 MiB/s wr, 144 op/s 2024-04-26T01:52:48.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:48 smithi029 ceph-mon[32775]: pgmap v215: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.1 MiB/s wr, 144 op/s 2024-04-26T01:52:49.591 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.99 MB/sec execute 193 sec latency 10607.297 ms 2024-04-26T01:52:50.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:50 smithi006 ceph-mon[28419]: pgmap v216: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 611 KiB/s wr, 53 op/s 2024-04-26T01:52:50.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:50 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:50.591 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.97 MB/sec execute 194 sec latency 11607.427 ms 2024-04-26T01:52:50.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:50 smithi188 ceph-mon[32751]: pgmap v216: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 611 KiB/s wr, 53 op/s 2024-04-26T01:52:50.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:50 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:50.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:50 smithi029 ceph-mon[32775]: pgmap v216: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 611 KiB/s wr, 53 op/s 2024-04-26T01:52:50.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:50 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:52:51.591 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.95 MB/sec execute 195 sec latency 12607.551 ms 2024-04-26T01:52:52.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:52 smithi006 ceph-mon[28419]: pgmap v217: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 919 KiB/s wr, 77 op/s 2024-04-26T01:52:52.591 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.93 MB/sec execute 196 sec latency 13607.705 ms 2024-04-26T01:52:52.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:52 smithi188 ceph-mon[32751]: pgmap v217: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 919 KiB/s wr, 77 op/s 2024-04-26T01:52:52.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:52 smithi029 ceph-mon[32775]: pgmap v217: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 919 KiB/s wr, 77 op/s 2024-04-26T01:52:53.591 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.91 MB/sec execute 197 sec latency 14607.885 ms 2024-04-26T01:52:54.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:54 smithi006 ceph-mon[28419]: pgmap v218: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 610 KiB/s wr, 34 op/s 2024-04-26T01:52:54.591 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.89 MB/sec execute 198 sec latency 15608.038 ms 2024-04-26T01:52:54.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:54 smithi188 ceph-mon[32751]: pgmap v218: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 610 KiB/s wr, 34 op/s 2024-04-26T01:52:54.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:54 smithi029 ceph-mon[32775]: pgmap v218: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 610 KiB/s wr, 34 op/s 2024-04-26T01:52:55.592 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.87 MB/sec execute 199 sec latency 16608.201 ms 2024-04-26T01:52:55.797 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:52:56.275 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:52:56.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:56 smithi006 ceph-mon[28419]: pgmap v219: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 610 KiB/s wr, 34 op/s 2024-04-26T01:52:56.591 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.85 MB/sec execute 200 sec latency 17608.293 ms 2024-04-26T01:52:56.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:56 smithi188 ceph-mon[32751]: pgmap v219: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 610 KiB/s wr, 34 op/s 2024-04-26T01:52:56.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:56 smithi029 ceph-mon[32775]: pgmap v219: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 610 KiB/s wr, 34 op/s 2024-04-26T01:52:57.592 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.83 MB/sec execute 201 sec latency 18608.391 ms 2024-04-26T01:52:58.408 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:58 smithi006 ceph-mon[28419]: pgmap v220: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 1.1 MiB/s wr, 52 op/s 2024-04-26T01:52:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.584+0000 7fd51aa00700 1 -- 172.21.15.6:0/1792593945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd514108970 msgr2=0x7fd514108d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.584+0000 7fd51aa00700 1 --2- 172.21.15.6:0/1792593945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd514108970 0x7fd514108d50 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd508009990 tx=0x7fd508009ca0 comp rx=0 tx=0).stop 2024-04-26T01:52:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.584+0000 7fd51aa00700 1 -- 172.21.15.6:0/1792593945 shutdown_connections 2024-04-26T01:52:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.584+0000 7fd51aa00700 1 --2- 172.21.15.6:0/1792593945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd514103330 0x7fd51410f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.584+0000 7fd51aa00700 1 --2- 172.21.15.6:0/1792593945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd514108970 0x7fd514108d50 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.584+0000 7fd51aa00700 1 --2- 172.21.15.6:0/1792593945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd514102970 0x7fd514102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.584+0000 7fd51aa00700 1 -- 172.21.15.6:0/1792593945 >> 172.21.15.6:0/1792593945 conn(0x7fd5140fe4c0 msgr2=0x7fd5141008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.584+0000 7fd51aa00700 1 -- 172.21.15.6:0/1792593945 shutdown_connections 2024-04-26T01:52:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.584+0000 7fd51aa00700 1 -- 172.21.15.6:0/1792593945 wait complete. 2024-04-26T01:52:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.586+0000 7fd51aa00700 1 Processor -- start 2024-04-26T01:52:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.586+0000 7fd51aa00700 1 -- start start 2024-04-26T01:52:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.586+0000 7fd51aa00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd514102970 0x7fd51419b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.586+0000 7fd51aa00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd514103330 0x7fd51419b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.586+0000 7fd51aa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd514108970 0x7fd51419fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.586+0000 7fd51aa00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd514112c00 con 0x7fd514102970 2024-04-26T01:52:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.586+0000 7fd51aa00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd514112a80 con 0x7fd514103330 2024-04-26T01:52:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.586+0000 7fd51aa00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd514112d80 con 0x7fd514108970 2024-04-26T01:52:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.586+0000 7fd519600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd514102970 0x7fd51419b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.587+0000 7fd519600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd514102970 0x7fd51419b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36996/0 (socket says 172.21.15.6:36996) 2024-04-26T01:52:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.587+0000 7fd519600700 1 -- 172.21.15.6:0/3481039542 learned_addr learned my addr 172.21.15.6:0/3481039542 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.587+0000 7fd51a000700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd514108970 0x7fd51419fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.587+0000 7fd518c00700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd514103330 0x7fd51419b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.587+0000 7fd519600700 1 -- 172.21.15.6:0/3481039542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd514108970 msgr2=0x7fd51419fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.587+0000 7fd519600700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd514108970 0x7fd51419fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.587+0000 7fd519600700 1 -- 172.21.15.6:0/3481039542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd514103330 msgr2=0x7fd51419b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.587+0000 7fd519600700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd514103330 0x7fd51419b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.587+0000 7fd519600700 1 -- 172.21.15.6:0/3481039542 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5141a0280 con 0x7fd514102970 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.587+0000 7fd519600700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd514102970 0x7fd51419b270 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd508005910 tx=0x7fd50800b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.588+0000 7fd512000700 1 -- 172.21.15.6:0/3481039542 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508027070 con 0x7fd514102970 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.588+0000 7fd512000700 1 -- 172.21.15.6:0/3481039542 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd508022470 con 0x7fd514102970 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.588+0000 7fd512000700 1 -- 172.21.15.6:0/3481039542 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508017650 con 0x7fd514102970 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.588+0000 7fd51aa00700 1 -- 172.21.15.6:0/3481039542 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5141a0510 con 0x7fd514102970 2024-04-26T01:52:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.588+0000 7fd51aa00700 1 -- 172.21.15.6:0/3481039542 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5141a7df0 con 0x7fd514102970 2024-04-26T01:52:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.589+0000 7fd51aa00700 1 -- 172.21.15.6:0/3481039542 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd514069730 con 0x7fd514102970 2024-04-26T01:52:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.591+0000 7fd512000700 1 -- 172.21.15.6:0/3481039542 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fd50802a070 con 0x7fd514102970 2024-04-26T01:52:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.593+0000 7fd512000700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd4fc0a3db0 0x7fd4fc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:58.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.593+0000 7fd512000700 1 -- 172.21.15.6:0/3481039542 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7fd5080d1760 con 0x7fd514102970 2024-04-26T01:52:58.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.593+0000 7fd518c00700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd4fc0a3db0 0x7fd4fc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:58.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.594+0000 7fd518c00700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd4fc0a3db0 0x7fd4fc0a6270 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd51410fbb0 tx=0x7fd500009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:58.592 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.81 MB/sec execute 202 sec latency 19608.461 ms 2024-04-26T01:52:58.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.596+0000 7fd512000700 1 -- 172.21.15.6:0/3481039542 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd508013070 con 0x7fd514102970 2024-04-26T01:52:58.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:58 smithi188 ceph-mon[32751]: pgmap v220: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 1.1 MiB/s wr, 52 op/s 2024-04-26T01:52:58.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:58 smithi029 ceph-mon[32775]: pgmap v220: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 1.1 MiB/s wr, 52 op/s 2024-04-26T01:52:58.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.877+0000 7fd51aa00700 1 -- 172.21.15.6:0/3481039542 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5141a7fc0 con 0x7fd514102970 2024-04-26T01:52:58.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.879+0000 7fd512000700 1 -- 172.21.15.6:0/3481039542 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11851 (secure 0 0 0) 0x7fd50809df00 con 0x7fd514102970 2024-04-26T01:52:58.878 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:52:58.878 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:52:48.909773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24512,24505]},"id":1}]} 2024-04-26T01:52:58.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.882+0000 7fd51aa00700 1 -- 172.21.15.6:0/3481039542 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd4fc0a3db0 msgr2=0x7fd4fc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:58.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.882+0000 7fd51aa00700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd4fc0a3db0 0x7fd4fc0a6270 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd51410fbb0 tx=0x7fd500009260 comp rx=0 tx=0).stop 2024-04-26T01:52:58.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.882+0000 7fd51aa00700 1 -- 172.21.15.6:0/3481039542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd514102970 msgr2=0x7fd51419b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:58.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.882+0000 7fd51aa00700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd514102970 0x7fd51419b270 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd508005910 tx=0x7fd50800b670 comp rx=0 tx=0).stop 2024-04-26T01:52:58.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.882+0000 7fd51aa00700 1 -- 172.21.15.6:0/3481039542 shutdown_connections 2024-04-26T01:52:58.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.882+0000 7fd51aa00700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd4fc0a3db0 0x7fd4fc0a6270 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:58.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.882+0000 7fd51aa00700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd514108970 0x7fd51419fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:58.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.882+0000 7fd51aa00700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd514102970 0x7fd51419b270 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:58.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.882+0000 7fd51aa00700 1 --2- 172.21.15.6:0/3481039542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd514103330 0x7fd51419b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:58.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.882+0000 7fd51aa00700 1 -- 172.21.15.6:0/3481039542 >> 172.21.15.6:0/3481039542 conn(0x7fd5140fe4c0 msgr2=0x7fd51410e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:58.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.883+0000 7fd51aa00700 1 -- 172.21.15.6:0/3481039542 shutdown_connections 2024-04-26T01:52:58.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:52:58.883+0000 7fd51aa00700 1 -- 172.21.15.6:0/3481039542 wait complete. 2024-04-26T01:52:58.881 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 40 2024-04-26T01:52:59.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:52:59 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3481039542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:59.592 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.79 MB/sec execute 203 sec latency 20608.553 ms 2024-04-26T01:52:59.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:52:59 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3481039542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:59.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:52:59 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3481039542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:59.733 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id 5f38731d --release --await' 2024-04-26T01:52:59.734 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id 5f38731d --release --await 2024-04-26T01:53:00.245 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:53:00.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:00 smithi006 ceph-mon[28419]: pgmap v221: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 775 KiB/s wr, 42 op/s 2024-04-26T01:53:00.592 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.77 MB/sec execute 204 sec latency 21608.645 ms 2024-04-26T01:53:00.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:00 smithi188 ceph-mon[32751]: pgmap v221: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 775 KiB/s wr, 42 op/s 2024-04-26T01:53:00.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:00 smithi029 ceph-mon[32775]: pgmap v221: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 775 KiB/s wr, 42 op/s 2024-04-26T01:53:01.592 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235839 3.76 MB/sec execute 205 sec latency 22608.775 ms 2024-04-26T01:53:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.765+0000 7fbaeec00640 1 -- 172.21.15.6:0/55133117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbae81041f0 msgr2=0x7fbae81045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.765+0000 7fbaeec00640 1 --2- 172.21.15.6:0/55133117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbae81041f0 0x7fbae81045d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbadc009900 tx=0x7fbadc02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.765+0000 7fbaeec00640 1 -- 172.21.15.6:0/55133117 shutdown_connections 2024-04-26T01:53:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.765+0000 7fbaeec00640 1 --2- 172.21.15.6:0/55133117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbae81092c0 0x7fbae8110e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.765+0000 7fbaeec00640 1 --2- 172.21.15.6:0/55133117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae8104ba0 0x7fbae8108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.765+0000 7fbaeec00640 1 --2- 172.21.15.6:0/55133117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbae81041f0 0x7fbae81045d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.765+0000 7fbaeec00640 1 -- 172.21.15.6:0/55133117 >> 172.21.15.6:0/55133117 conn(0x7fbae80ffa50 msgr2=0x7fbae8101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.765+0000 7fbaeec00640 1 -- 172.21.15.6:0/55133117 shutdown_connections 2024-04-26T01:53:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.765+0000 7fbaeec00640 1 -- 172.21.15.6:0/55133117 wait complete. 2024-04-26T01:53:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.767+0000 7fbaeec00640 1 Processor -- start 2024-04-26T01:53:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.767+0000 7fbaeec00640 1 -- start start 2024-04-26T01:53:01.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.768+0000 7fbaeec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbae81041f0 0x7fbae81a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.768+0000 7fbaeec00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbae81092c0 0x7fbae81a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.768+0000 7fbaeec00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae819a610 0x7fbae819aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.768+0000 7fbaeec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbae8114250 con 0x7fbae81041f0 2024-04-26T01:53:01.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.768+0000 7fbaeec00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81140d0 con 0x7fbae819a610 2024-04-26T01:53:01.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.768+0000 7fbaeec00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81143d0 con 0x7fbae81092c0 2024-04-26T01:53:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.768+0000 7fbaed800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbae81041f0 0x7fbae81a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.768+0000 7fbaed800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbae81041f0 0x7fbae81a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37008/0 (socket says 172.21.15.6:37008) 2024-04-26T01:53:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.768+0000 7fbaed800640 1 -- 172.21.15.6:0/264550271 learned_addr learned my addr 172.21.15.6:0/264550271 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.768+0000 7fbaece00640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbae81092c0 0x7fbae81a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.768+0000 7fbaee200640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae819a610 0x7fbae819aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.769+0000 7fbaed800640 1 -- 172.21.15.6:0/264550271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbae81092c0 msgr2=0x7fbae81a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.769+0000 7fbaed800640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbae81092c0 0x7fbae81a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.769+0000 7fbaed800640 1 -- 172.21.15.6:0/264550271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae819a610 msgr2=0x7fbae819aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.769+0000 7fbaed800640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae819a610 0x7fbae819aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.769+0000 7fbaed800640 1 -- 172.21.15.6:0/264550271 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae819b130 con 0x7fbae81041f0 2024-04-26T01:53:01.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.769+0000 7fbaed800640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbae81041f0 0x7fbae81a0460 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbadc005ab0 tx=0x7fbadc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:01.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.769+0000 7fbae6000640 1 -- 172.21.15.6:0/264550271 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc03f070 con 0x7fbae81041f0 2024-04-26T01:53:01.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.769+0000 7fbae6000640 1 -- 172.21.15.6:0/264550271 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbadc031b60 con 0x7fbae81041f0 2024-04-26T01:53:01.769 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.769+0000 7fbae6000640 1 -- 172.21.15.6:0/264550271 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc035030 con 0x7fbae81041f0 2024-04-26T01:53:01.769 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.770+0000 7fbaeec00640 1 -- 172.21.15.6:0/264550271 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae819b3c0 con 0x7fbae81041f0 2024-04-26T01:53:01.769 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.770+0000 7fbaeec00640 1 -- 172.21.15.6:0/264550271 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae81a7560 con 0x7fbae81041f0 2024-04-26T01:53:01.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.774+0000 7fbaf317e740 1 Processor -- start 2024-04-26T01:53:01.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.774+0000 7fbaf317e740 1 -- start start 2024-04-26T01:53:01.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.774+0000 7fbae6000640 1 -- 172.21.15.6:0/264550271 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fbadc042070 con 0x7fbae81041f0 2024-04-26T01:53:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.775+0000 7fbaf317e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d889b0 0x564324d88d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.775+0000 7fbaf317e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d892d0 0x564324d93650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.775+0000 7fbaf317e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d93b90 0x564324d95f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.775+0000 7fbaf317e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564324cee900 con 0x564324d93b90 2024-04-26T01:53:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.775+0000 7fbaf317e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564324c94ef0 con 0x564324d889b0 2024-04-26T01:53:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.775+0000 7fbaf317e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564324cea8c0 con 0x564324d892d0 2024-04-26T01:53:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.775+0000 7fbaed800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d892d0 0x564324d93650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.775+0000 7fbaee200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d889b0 0x564324d88d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.775+0000 7fbaee200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d889b0 0x564324d88d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:48040/0 (socket says 172.21.15.6:48040) 2024-04-26T01:53:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.775+0000 7fbaee200640 1 -- 172.21.15.6:0/2858075009 learned_addr learned my addr 172.21.15.6:0/2858075009 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.775+0000 7fbaece00640 1 --2- 172.21.15.6:0/2858075009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d93b90 0x564324d95f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.775+0000 7fbaee200640 1 -- 172.21.15.6:0/2858075009 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d892d0 msgr2=0x564324d93650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:01.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.776+0000 7fbaee200640 1 --2- 172.21.15.6:0/2858075009 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d892d0 0x564324d93650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.776+0000 7fbaee200640 1 -- 172.21.15.6:0/2858075009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d93b90 msgr2=0x564324d95f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.776+0000 7fbaee200640 1 --2- 172.21.15.6:0/2858075009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d93b90 0x564324d95f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.776+0000 7fbaee200640 1 -- 172.21.15.6:0/2858075009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564324a01210 con 0x564324d889b0 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.776+0000 7fbaee200640 1 --2- 172.21.15.6:0/2858075009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d889b0 0x564324d88d90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009f60 tx=0x7fbae0009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21d7b5da2a69c507 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.776+0000 7fbadaa00640 1 -- 172.21.15.6:0/2858075009 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae001e070 con 0x564324d889b0 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.776+0000 7fbadaa00640 1 -- 172.21.15.6:0/2858075009 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae0003c30 con 0x564324d889b0 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.776+0000 7fbadaa00640 1 -- 172.21.15.6:0/2858075009 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae000f840 con 0x564324d889b0 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.777+0000 7fbae6000640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbad00a3d00 0x7fbad00a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.777+0000 7fbaf317e740 1 -- 172.21.15.6:0/2858075009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d889b0 msgr2=0x564324d88d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.777+0000 7fbaf317e740 1 --2- 172.21.15.6:0/2858075009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d889b0 0x564324d88d90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009f60 tx=0x7fbae0009f90 comp rx=0 tx=0).stop 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.777+0000 7fbaf317e740 1 -- 172.21.15.6:0/2858075009 shutdown_connections 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.777+0000 7fbaf317e740 1 --2- 172.21.15.6:0/2858075009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d93b90 0x564324d95f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.777+0000 7fbaf317e740 1 --2- 172.21.15.6:0/2858075009 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d892d0 0x564324d93650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.777 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.777+0000 7fbaf317e740 1 --2- 172.21.15.6:0/2858075009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d889b0 0x564324d88d90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.777 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.777+0000 7fbaf317e740 1 -- 172.21.15.6:0/2858075009 >> 172.21.15.6:0/2858075009 conn(0x564324d84ec0 msgr2=0x564324d852a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:01.777 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.777+0000 7fbae6000640 1 -- 172.21.15.6:0/264550271 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7fbadc0f0ea0 con 0x7fbae81041f0 2024-04-26T01:53:01.777 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.777+0000 7fbaed800640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbad00a3d00 0x7fbad00a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.777 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.777+0000 7fbaf317e740 1 -- 172.21.15.6:0/2858075009 shutdown_connections 2024-04-26T01:53:01.777 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.777+0000 7fbaf317e740 1 -- 172.21.15.6:0/2858075009 wait complete. 2024-04-26T01:53:01.777 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.778+0000 7fbaf317e740 1 -- start start 2024-04-26T01:53:01.777 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.778+0000 7fbaed800640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbad00a3d00 0x7fbad00a61c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0ee650 tx=0x7fbadc0f49f0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:01.777 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.778+0000 7fbaf317e740 1 Processor -- start 2024-04-26T01:53:01.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaf317e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 0x564324d90b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaf317e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d892d0 0x564324d910b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaf317e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d93b90 0x564324d915f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaf317e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564324d967c0 con 0x564324d889b0 2024-04-26T01:53:01.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaf317e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564324ca8310 con 0x564324d892d0 2024-04-26T01:53:01.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaf317e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564324d96940 con 0x564324d93b90 2024-04-26T01:53:01.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaf317e740 10 client.0 _ll_register_callbacks cb 0x564324c76cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:01.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaee200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 0x564324d90b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaece00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d892d0 0x564324d910b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaee200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 0x564324d90b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37034/0 (socket says 172.21.15.6:37034) 2024-04-26T01:53:01.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaee200640 1 -- 172.21.15.6:0/2872076598 learned_addr learned my addr 172.21.15.6:0/2872076598 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaee200640 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d93b90 0x564324d915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaece00640 1 -- 172.21.15.6:0/2872076598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d93b90 msgr2=0x564324d915f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.779+0000 7fbaece00640 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d93b90 0x564324d915f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.780+0000 7fbaece00640 1 -- 172.21.15.6:0/2872076598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 msgr2=0x564324d90b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.780+0000 7fbaece00640 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 0x564324d90b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.780+0000 7fbaece00640 1 -- 172.21.15.6:0/2872076598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564324acd620 con 0x564324d892d0 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.780+0000 7fbaee200640 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 0x564324d90b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.780+0000 7fbaece00640 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d892d0 0x564324d910b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbae80fffe0 tx=0x7fbad4012ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6026908087193184 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.780+0000 7fbacf400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.780+0000 7fbacf400640 1 -- 172.21.15.6:0/2872076598 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad400f8f0 con 0x564324d892d0 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.780+0000 7fbaf317e740 1 -- 172.21.15.6:0/2872076598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbae0003c30 con 0x564324d892d0 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.780+0000 7fbacf400640 1 -- 172.21.15.6:0/2872076598 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7fbad400fa90 con 0x564324d892d0 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.780+0000 7fbacf400640 1 -- 172.21.15.6:0/2872076598 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fbad400fc90 con 0x564324d892d0 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.780+0000 7fbaf317e740 10 client.26015 fetch_fsmap learned FSMap version 40 2024-04-26T01:53:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.780+0000 7fbaf317e740 1 -- 172.21.15.6:0/2872076598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x564324da2b20 con 0x564324d892d0 2024-04-26T01:53:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.781+0000 7fbacf400640 1 -- 172.21.15.6:0/2872076598 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae0003c30 con 0x564324d892d0 2024-04-26T01:53:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.781+0000 7fbacf400640 1 -- 172.21.15.6:0/2872076598 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7fbad401c420 con 0x564324d892d0 2024-04-26T01:53:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.781+0000 7fbaf317e740 10 client.26015 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:53:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.781+0000 7fbaf317e740 10 client.26015 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.781+0000 7fbaf317e740 20 client.26015 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.781+0000 7fbaf317e740 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564324da3f90 0x564324da63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.781+0000 7fbaf317e740 4 client.26015 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.781+0000 7fbaf317e740 1 -- 172.21.15.6:0/2872076598 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564324d028b0 con 0x564324da3f90 2024-04-26T01:53:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.782+0000 7fbaee200640 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564324da3f90 0x564324da63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.782+0000 7fbaee200640 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564324da3f90 0x564324da63a0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf4b9f037e3d92ec server_cookie=d3be761dab82a6b6 in_seq=0 out_seq=0 2024-04-26T01:53:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.782+0000 7fbacf400640 10 client.26015 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbacf400640 1 -- 172.21.15.6:0/2872076598 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564324d028b0 con 0x564324da3f90 2024-04-26T01:53:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbacf400640 10 client.26015 handle_command_reply: tid=0 2024-04-26T01:53:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbaf317e740 1 client.26015 shutdown 2024-04-26T01:53:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbaf317e740 1 -- 172.21.15.6:0/2872076598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d892d0 msgr2=0x564324d910b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbaf317e740 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d892d0 0x564324d910b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbae80fffe0 tx=0x7fbad4012ed0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbaf317e740 1 -- 172.21.15.6:0/2872076598 shutdown_connections 2024-04-26T01:53:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbaf317e740 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564324da3f90 0x564324da63a0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbaf317e740 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d93b90 0x564324d915f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbaf317e740 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d892d0 0x564324d910b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbaf317e740 1 --2- 172.21.15.6:0/2872076598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 0x564324d90b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbaf317e740 1 -- 172.21.15.6:0/2872076598 >> 172.21.15.6:0/2872076598 conn(0x564324d84ec0 msgr2=0x564324d8abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbacf400640 0 client.26015 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbaf317e740 1 -- 172.21.15.6:0/2872076598 shutdown_connections 2024-04-26T01:53:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.783+0000 7fbaf317e740 1 -- 172.21.15.6:0/2872076598 wait complete. 2024-04-26T01:53:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.784+0000 7fbaf317e740 20 client.26015 trim_cache size 0 max 16384 2024-04-26T01:53:01.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.800+0000 7fbaf317e740 1 Processor -- start 2024-04-26T01:53:01.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.800+0000 7fbaf317e740 1 -- start start 2024-04-26T01:53:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.801+0000 7fbaf317e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 0x564324d9f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.801+0000 7fbaf317e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d892d0 0x564324d83ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.801+0000 7fbaf317e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d93b90 0x564324dabcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.801+0000 7fbaf317e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81b2710 con 0x564324d889b0 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.801+0000 7fbaf317e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81affb0 con 0x564324d93b90 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.801+0000 7fbaf317e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81ade20 con 0x564324d892d0 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.801+0000 7fbaee200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 0x564324d9f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.801+0000 7fbaece00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d892d0 0x564324d83ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.801+0000 7fbaee200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 0x564324d9f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37044/0 (socket says 172.21.15.6:37044) 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.801+0000 7fbaee200640 1 -- 172.21.15.6:0/4104630814 learned_addr learned my addr 172.21.15.6:0/4104630814 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.801+0000 7fbaee200640 1 --2- 172.21.15.6:0/4104630814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d93b90 0x564324dabcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.802+0000 7fbaece00640 1 -- 172.21.15.6:0/4104630814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d93b90 msgr2=0x564324dabcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.802+0000 7fbaece00640 1 --2- 172.21.15.6:0/4104630814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d93b90 0x564324dabcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.802+0000 7fbaece00640 1 -- 172.21.15.6:0/4104630814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 msgr2=0x564324d9f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.802+0000 7fbaece00640 1 --2- 172.21.15.6:0/4104630814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 0x564324d9f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.802+0000 7fbaece00640 1 -- 172.21.15.6:0/4104630814 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564324d80820 con 0x564324d892d0 2024-04-26T01:53:01.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.802+0000 7fbaee200640 1 --2- 172.21.15.6:0/4104630814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d93b90 0x564324dabcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:01.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.802+0000 7fbaece00640 1 --2- 172.21.15.6:0/4104630814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d892d0 0x564324d83ea0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbad4000c00 tx=0x7fbad400f810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41500a19564effd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:01.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.802+0000 7fbada000640 1 -- 172.21.15.6:0/4104630814 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad400f8f0 con 0x564324d892d0 2024-04-26T01:53:01.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.802+0000 7fbada000640 1 -- 172.21.15.6:0/4104630814 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbad400fa90 con 0x564324d892d0 2024-04-26T01:53:01.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.802+0000 7fbada000640 1 -- 172.21.15.6:0/4104630814 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad40053e0 con 0x564324d892d0 2024-04-26T01:53:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.803+0000 7fbaf317e740 1 -- 172.21.15.6:0/4104630814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d892d0 msgr2=0x564324d83ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.803+0000 7fbaf317e740 1 --2- 172.21.15.6:0/4104630814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d892d0 0x564324d83ea0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbad4000c00 tx=0x7fbad400f810 comp rx=0 tx=0).stop 2024-04-26T01:53:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.803+0000 7fbaf317e740 1 -- 172.21.15.6:0/4104630814 shutdown_connections 2024-04-26T01:53:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.803+0000 7fbaf317e740 1 --2- 172.21.15.6:0/4104630814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d93b90 0x564324dabcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.803+0000 7fbaf317e740 1 --2- 172.21.15.6:0/4104630814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d892d0 0x564324d83ea0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.803+0000 7fbaf317e740 1 --2- 172.21.15.6:0/4104630814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d889b0 0x564324d9f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.803+0000 7fbaf317e740 1 -- 172.21.15.6:0/4104630814 >> 172.21.15.6:0/4104630814 conn(0x564324d84ec0 msgr2=0x564324d83790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.803+0000 7fbaf317e740 1 -- 172.21.15.6:0/4104630814 shutdown_connections 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.803+0000 7fbaf317e740 1 -- 172.21.15.6:0/4104630814 wait complete. 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.804+0000 7fbaf317e740 1 -- start start 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.804+0000 7fbaf317e740 1 Processor -- start 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.804+0000 7fbaf317e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d889b0 0x564324d998a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.804+0000 7fbaf317e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d892d0 0x564324d99de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.804+0000 7fbaf317e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d93b90 0x564324d9a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.804+0000 7fbaf317e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81ae470 con 0x564324d93b90 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.804+0000 7fbaf317e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81af8e0 con 0x564324d892d0 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.804+0000 7fbaf317e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81aeb40 con 0x564324d889b0 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.804+0000 7fbaf317e740 10 client.0 _ll_register_callbacks cb 0x7fbad4012680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbaee200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d93b90 0x564324d9a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbaee200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d93b90 0x564324d9a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37060/0 (socket says 172.21.15.6:37060) 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbaee200640 1 -- 172.21.15.6:0/939127975 learned_addr learned my addr 172.21.15.6:0/939127975 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbaee200640 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d889b0 0x564324d998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbaece00640 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d892d0 0x564324d99de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbaee200640 1 -- 172.21.15.6:0/939127975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d889b0 msgr2=0x564324d998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbaee200640 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d889b0 0x564324d998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbaee200640 1 -- 172.21.15.6:0/939127975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d892d0 msgr2=0x564324d99de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbaee200640 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d892d0 0x564324d99de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbaee200640 1 -- 172.21.15.6:0/939127975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564324e09800 con 0x564324d93b90 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbaece00640 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d892d0 0x564324d99de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbaee200640 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d93b90 0x564324d9a340 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009f60 tx=0x7fbae000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd94dc7c19de9335 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.805+0000 7fbacf400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.806+0000 7fbacf400640 1 -- 172.21.15.6:0/939127975 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae001e040 con 0x564324d93b90 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.806+0000 7fbacf400640 1 -- 172.21.15.6:0/939127975 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7fbae0004250 con 0x564324d93b90 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.806+0000 7fbaf317e740 1 -- 172.21.15.6:0/939127975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbad400fa90 con 0x564324d93b90 2024-04-26T01:53:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.806+0000 7fbacf400640 1 -- 172.21.15.6:0/939127975 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fbae003a050 con 0x564324d93b90 2024-04-26T01:53:01.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.806+0000 7fbaf317e740 10 client.15885 fetch_fsmap learned FSMap version 40 2024-04-26T01:53:01.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.806+0000 7fbaf317e740 1 -- 172.21.15.6:0/939127975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x564324e0ab10 con 0x564324d93b90 2024-04-26T01:53:01.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.806+0000 7fbacf400640 1 -- 172.21.15.6:0/939127975 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbad400fa90 con 0x564324d93b90 2024-04-26T01:53:01.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.807+0000 7fbacf400640 1 -- 172.21.15.6:0/939127975 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7fbae002c5e0 con 0x564324d93b90 2024-04-26T01:53:01.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.807+0000 7fbaf317e740 10 client.15885 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:53:01.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.807+0000 7fbaf317e740 10 client.15885 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:01.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.807+0000 7fbaf317e740 20 client.15885 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:01.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.807+0000 7fbaf317e740 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564324e0b040 0x564324e0d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:01.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.807+0000 7fbaf317e740 4 client.15885 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:01.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.807+0000 7fbaf317e740 1 -- 172.21.15.6:0/939127975 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564324cfc320 con 0x564324e0b040 2024-04-26T01:53:01.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.807+0000 7fbaece00640 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564324e0b040 0x564324e0d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:01.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.808+0000 7fbaece00640 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564324e0b040 0x564324e0d430 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e497516f827506d server_cookie=36798213ecb090f5 in_seq=0 out_seq=0 2024-04-26T01:53:01.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.808+0000 7fbacf400640 10 client.15885 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:01.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.844+0000 7fbacf400640 1 -- 172.21.15.6:0/939127975 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564324cfc320 con 0x564324e0b040 2024-04-26T01:53:01.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.844+0000 7fbacf400640 10 client.15885 handle_command_reply: tid=0 2024-04-26T01:53:01.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.844+0000 7fbaf317e740 1 client.15885 shutdown 2024-04-26T01:53:01.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.844+0000 7fbaf317e740 1 -- 172.21.15.6:0/939127975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d93b90 msgr2=0x564324d9a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:01.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.844+0000 7fbaf317e740 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d93b90 0x564324d9a340 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009f60 tx=0x7fbae000f520 comp rx=0 tx=0).stop 2024-04-26T01:53:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.845+0000 7fbaf317e740 1 -- 172.21.15.6:0/939127975 shutdown_connections 2024-04-26T01:53:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.845+0000 7fbaf317e740 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564324e0b040 0x564324e0d430 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.845+0000 7fbaf317e740 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564324d93b90 0x564324d9a340 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.845+0000 7fbaf317e740 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564324d892d0 0x564324d99de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.845+0000 7fbaf317e740 1 --2- 172.21.15.6:0/939127975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564324d889b0 0x564324d998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.845+0000 7fbaf317e740 1 -- 172.21.15.6:0/939127975 >> 172.21.15.6:0/939127975 conn(0x564324d84ec0 msgr2=0x564324d8abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.845+0000 7fbacf400640 0 client.15885 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.845+0000 7fbaf317e740 1 -- 172.21.15.6:0/939127975 shutdown_connections 2024-04-26T01:53:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.845+0000 7fbaf317e740 1 -- 172.21.15.6:0/939127975 wait complete. 2024-04-26T01:53:01.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.845+0000 7fbaf317e740 20 client.15885 trim_cache size 0 max 16384 2024-04-26T01:53:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:53:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:53:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:53:01.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:53:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.846+0000 7fbaeec00640 1 -- 172.21.15.6:0/264550271 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbad00a3d00 msgr2=0x7fbad00a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.846+0000 7fbaeec00640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbad00a3d00 0x7fbad00a61c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0ee650 tx=0x7fbadc0f49f0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.846+0000 7fbaeec00640 1 -- 172.21.15.6:0/264550271 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbae81041f0 msgr2=0x7fbae81a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.846+0000 7fbaeec00640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbae81041f0 0x7fbae81a0460 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbadc005ab0 tx=0x7fbadc003c10 comp rx=0 tx=0).stop 2024-04-26T01:53:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.846+0000 7fbaeec00640 1 -- 172.21.15.6:0/264550271 shutdown_connections 2024-04-26T01:53:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.846+0000 7fbaeec00640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbad00a3d00 0x7fbad00a61c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.846+0000 7fbaeec00640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae819a610 0x7fbae819aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.846+0000 7fbaeec00640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbae81092c0 0x7fbae81a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.846+0000 7fbaeec00640 1 --2- 172.21.15.6:0/264550271 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbae81041f0 0x7fbae81a0460 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.846+0000 7fbaeec00640 1 -- 172.21.15.6:0/264550271 >> 172.21.15.6:0/264550271 conn(0x7fbae80ffa50 msgr2=0x7fbae8100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.846+0000 7fbaeec00640 1 -- 172.21.15.6:0/264550271 shutdown_connections 2024-04-26T01:53:01.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:01.846+0000 7fbaeec00640 1 -- 172.21.15.6:0/264550271 wait complete. 2024-04-26T01:53:02.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.506+0000 7fd0d3e00700 1 -- 172.21.15.6:0/3209256350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0cc1092d0 msgr2=0x7fd0cc1096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:02.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.506+0000 7fd0d3e00700 1 --2- 172.21.15.6:0/3209256350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0cc1092d0 0x7fd0cc1096b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00099c0 tx=0x7fd0c0009cd0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.506+0000 7fd0d3e00700 1 -- 172.21.15.6:0/3209256350 shutdown_connections 2024-04-26T01:53:02.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.506+0000 7fd0d3e00700 1 --2- 172.21.15.6:0/3209256350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc10e3b0 0x7fd0cc115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.506+0000 7fd0d3e00700 1 --2- 172.21.15.6:0/3209256350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0cc1092d0 0x7fd0cc1096b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.506+0000 7fd0d3e00700 1 --2- 172.21.15.6:0/3209256350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0cc109c80 0x7fd0cc10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.506+0000 7fd0d3e00700 1 -- 172.21.15.6:0/3209256350 >> 172.21.15.6:0/3209256350 conn(0x7fd0cc076210 msgr2=0x7fd0cc107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:02.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.506+0000 7fd0d3e00700 1 -- 172.21.15.6:0/3209256350 shutdown_connections 2024-04-26T01:53:02.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.506+0000 7fd0d3e00700 1 -- 172.21.15.6:0/3209256350 wait complete. 2024-04-26T01:53:02.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.508+0000 7fd0d3e00700 1 Processor -- start 2024-04-26T01:53:02.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.508+0000 7fd0d3e00700 1 -- start start 2024-04-26T01:53:02.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.508+0000 7fd0d3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0cc1092d0 0x7fd0cc1a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.508+0000 7fd0d3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc109c80 0x7fd0cc1a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.508+0000 7fd0d3e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0cc10e3b0 0x7fd0cc19f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.508+0000 7fd0d3e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc119290 con 0x7fd0cc1092d0 2024-04-26T01:53:02.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.508+0000 7fd0d3e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc119110 con 0x7fd0cc109c80 2024-04-26T01:53:02.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.508+0000 7fd0d3e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc119410 con 0x7fd0cc10e3b0 2024-04-26T01:53:02.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.509+0000 7fd0d1400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0cc1092d0 0x7fd0cc1a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.509+0000 7fd0d1400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0cc1092d0 0x7fd0cc1a53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37072/0 (socket says 172.21.15.6:37072) 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.509+0000 7fd0d1400700 1 -- 172.21.15.6:0/126635114 learned_addr learned my addr 172.21.15.6:0/126635114 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.509+0000 7fd0d0a00700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc109c80 0x7fd0cc1a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.509+0000 7fd0d1e00700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0cc10e3b0 0x7fd0cc19f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.509+0000 7fd0d1400700 1 -- 172.21.15.6:0/126635114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0cc10e3b0 msgr2=0x7fd0cc19f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.509+0000 7fd0d1400700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0cc10e3b0 0x7fd0cc19f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.509+0000 7fd0d1400700 1 -- 172.21.15.6:0/126635114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc109c80 msgr2=0x7fd0cc1a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.509+0000 7fd0d1400700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc109c80 0x7fd0cc1a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.509+0000 7fd0d1400700 1 -- 172.21.15.6:0/126635114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0cc19fd10 con 0x7fd0cc1092d0 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.509+0000 7fd0d1400700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0cc1092d0 0x7fd0cc1a53f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0009990 tx=0x7fd0c00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.510+0000 7fd0ca000700 1 -- 172.21.15.6:0/126635114 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0026070 con 0x7fd0cc1092d0 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.510+0000 7fd0ca000700 1 -- 172.21.15.6:0/126635114 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0c0003f20 con 0x7fd0cc1092d0 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.510+0000 7fd0d3e00700 1 -- 172.21.15.6:0/126635114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0cc19ffa0 con 0x7fd0cc1092d0 2024-04-26T01:53:02.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.510+0000 7fd0ca000700 1 -- 172.21.15.6:0/126635114 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c000fc00 con 0x7fd0cc1092d0 2024-04-26T01:53:02.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.510+0000 7fd0d3e00700 1 -- 172.21.15.6:0/126635114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0cc1ac2c0 con 0x7fd0cc1092d0 2024-04-26T01:53:02.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.514+0000 7fd0ca000700 1 -- 172.21.15.6:0/126635114 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fd0c0029070 con 0x7fd0cc1092d0 2024-04-26T01:53:02.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.515+0000 7fd0de779b80 1 Processor -- start 2024-04-26T01:53:02.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.515+0000 7fd0de779b80 1 -- start start 2024-04-26T01:53:02.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.516+0000 7fd0de779b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60be9920 0x555a60be9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.516+0000 7fd0ca000700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0b40a3db0 0x7fd0b40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.516+0000 7fd0de779b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60bea240 0x555a60bf4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.516+0000 7fd0de779b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bf50f0 0x555a60bf74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.516+0000 7fd0de779b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555a60b3cf90 con 0x555a60be9920 2024-04-26T01:53:02.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.516+0000 7fd0de779b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555a60ae3830 con 0x555a60bf50f0 2024-04-26T01:53:02.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.516+0000 7fd0de779b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555a60b38f50 con 0x555a60bea240 2024-04-26T01:53:02.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.516+0000 7fd0ca000700 1 -- 172.21.15.6:0/126635114 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7fd0c00d11c0 con 0x7fd0cc1092d0 2024-04-26T01:53:02.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.516+0000 7fd0d1e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60be9920 0x555a60be9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.516+0000 7fd0d0a00700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0b40a3db0 0x7fd0b40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.516+0000 7fd0d1400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60bea240 0x555a60bf4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.517+0000 7fd0d1400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60bea240 0x555a60bf4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38108/0 (socket says 172.21.15.6:38108) 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.517+0000 7fd0d1400700 1 -- 172.21.15.6:0/2160211700 learned_addr learned my addr 172.21.15.6:0/2160211700 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.517+0000 7fd0d1e00700 1 --2- 172.21.15.6:0/2160211700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bf50f0 0x555a60bf74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.517+0000 7fd0d1400700 1 -- 172.21.15.6:0/2160211700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bf50f0 msgr2=0x555a60bf74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.517+0000 7fd0d1400700 1 --2- 172.21.15.6:0/2160211700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bf50f0 0x555a60bf74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.517+0000 7fd0d1400700 1 -- 172.21.15.6:0/2160211700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60be9920 msgr2=0x555a60be9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.517+0000 7fd0d1400700 1 --2- 172.21.15.6:0/2160211700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60be9920 0x555a60be9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.517+0000 7fd0d1400700 1 -- 172.21.15.6:0/2160211700 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a60ad9780 con 0x555a60bea240 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.517+0000 7fd0d1e00700 1 --2- 172.21.15.6:0/2160211700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bf50f0 0x555a60bf74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.517+0000 7fd0d1400700 1 --2- 172.21.15.6:0/2160211700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60bea240 0x555a60bf4bb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00a57f0 tx=0x7fd0c009cdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abe4c148bf0ab5f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.518+0000 7fd0d0a00700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0b40a3db0 0x7fd0b40a6270 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc108910 tx=0x7fd0b800b410 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.518+0000 7fd0bea00700 1 -- 172.21.15.6:0/2160211700 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c00aab10 con 0x555a60bea240 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.518+0000 7fd0bea00700 1 -- 172.21.15.6:0/2160211700 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0c00a6be0 con 0x555a60bea240 2024-04-26T01:53:02.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.518+0000 7fd0bea00700 1 -- 172.21.15.6:0/2160211700 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c00ad4a0 con 0x555a60bea240 2024-04-26T01:53:02.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.519+0000 7fd0de779b80 1 -- 172.21.15.6:0/2160211700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60bea240 msgr2=0x555a60bf4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:02.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.519+0000 7fd0de779b80 1 --2- 172.21.15.6:0/2160211700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60bea240 0x555a60bf4bb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00a57f0 tx=0x7fd0c009cdb0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.519+0000 7fd0de779b80 1 -- 172.21.15.6:0/2160211700 shutdown_connections 2024-04-26T01:53:02.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.519+0000 7fd0de779b80 1 --2- 172.21.15.6:0/2160211700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bf50f0 0x555a60bf74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.519+0000 7fd0de779b80 1 --2- 172.21.15.6:0/2160211700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60be9920 0x555a60be9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.519+0000 7fd0de779b80 1 --2- 172.21.15.6:0/2160211700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60bea240 0x555a60bf4bb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.519+0000 7fd0de779b80 1 -- 172.21.15.6:0/2160211700 >> 172.21.15.6:0/2160211700 conn(0x555a60b186a0 msgr2=0x555a60be5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:02.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.519+0000 7fd0de779b80 1 -- 172.21.15.6:0/2160211700 shutdown_connections 2024-04-26T01:53:02.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.519+0000 7fd0de779b80 1 -- 172.21.15.6:0/2160211700 wait complete. 2024-04-26T01:53:02.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.519+0000 7fd0de779b80 1 -- start start 2024-04-26T01:53:02.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.520+0000 7fd0de779b80 1 Processor -- start 2024-04-26T01:53:02.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.520+0000 7fd0de779b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60bf47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.520+0000 7fd0de779b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60bf1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.520+0000 7fd0de779b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60bf20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.520+0000 7fd0de779b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555a60bf7a20 con 0x555a60bf50f0 2024-04-26T01:53:02.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.520+0000 7fd0de779b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555a60af6960 con 0x555a60bea240 2024-04-26T01:53:02.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.520+0000 7fd0de779b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555a60bf7ba0 con 0x555a60be9920 2024-04-26T01:53:02.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.520+0000 7fd0de779b80 10 client.0 _ll_register_callbacks cb 0x555a60ad5820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0d1400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60bf47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0d0a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60bf1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0d1400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60bf47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38114/0 (socket says 172.21.15.6:38114) 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0d1400700 1 -- 172.21.15.6:0/116712573 learned_addr learned my addr 172.21.15.6:0/116712573 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0d1e00700 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60bf20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0d1400700 1 -- 172.21.15.6:0/116712573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 msgr2=0x555a60bf1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0d1400700 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60bf1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0d1400700 1 -- 172.21.15.6:0/116712573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 msgr2=0x555a60bf20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0d1400700 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60bf20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0d1400700 1 -- 172.21.15.6:0/116712573 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a60a8a9d0 con 0x555a60be9920 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0d1e00700 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60bf20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0d1400700 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60bf47d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000c010 tx=0x7fd0c009b650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28e340f31bedcc7b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.521+0000 7fd0b3400700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.522+0000 7fd0b3400700 1 -- 172.21.15.6:0/116712573 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c00a14f0 con 0x555a60be9920 2024-04-26T01:53:02.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.522+0000 7fd0de779b80 1 -- 172.21.15.6:0/116712573 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0c00a6be0 con 0x555a60be9920 2024-04-26T01:53:02.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.522+0000 7fd0b3400700 1 -- 172.21.15.6:0/116712573 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7fd0c00a5890 con 0x555a60be9920 2024-04-26T01:53:02.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.522+0000 7fd0b3400700 1 -- 172.21.15.6:0/116712573 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd0c00a5a90 con 0x555a60be9920 2024-04-26T01:53:02.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.522+0000 7fd0de779b80 10 client.26080 fetch_fsmap learned FSMap version 40 2024-04-26T01:53:02.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.522+0000 7fd0de779b80 1 -- 172.21.15.6:0/116712573 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555a60ad3bf0 con 0x555a60be9920 2024-04-26T01:53:02.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.522+0000 7fd0b3400700 1 -- 172.21.15.6:0/116712573 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0c00a6be0 con 0x555a60be9920 2024-04-26T01:53:02.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.522+0000 7fd0b3400700 1 -- 172.21.15.6:0/116712573 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7fd0c00ad3b0 con 0x555a60be9920 2024-04-26T01:53:02.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.523+0000 7fd0de779b80 10 client.26080 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:53:02.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.523+0000 7fd0de779b80 10 client.26080 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:53:02.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.523+0000 7fd0de779b80 20 client.26080 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:02.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.523+0000 7fd0de779b80 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a60c04500 0x555a60c068f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.523+0000 7fd0de779b80 4 client.26080 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:02.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.523+0000 7fd0de779b80 1 -- 172.21.15.6:0/116712573 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a60b4a780 con 0x555a60c04500 2024-04-26T01:53:02.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.523+0000 7fd0d0a00700 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a60c04500 0x555a60c068f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.524+0000 7fd0d0a00700 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a60c04500 0x555a60c068f0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbed47a6be43f93b server_cookie=9691aca011742c3d in_seq=0 out_seq=0 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.524+0000 7fd0b3400700 10 client.26080 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.524+0000 7fd0b3400700 1 -- 172.21.15.6:0/116712573 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a60b4a780 con 0x555a60c04500 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.524+0000 7fd0b3400700 10 client.26080 handle_command_reply: tid=0 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0de779b80 1 client.26080 shutdown 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0de779b80 1 -- 172.21.15.6:0/116712573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 msgr2=0x555a60bf47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0de779b80 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60bf47d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000c010 tx=0x7fd0c009b650 comp rx=0 tx=0).stop 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0de779b80 1 -- 172.21.15.6:0/116712573 shutdown_connections 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0de779b80 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60bf20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0de779b80 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a60c04500 0x555a60c068f0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0de779b80 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60bf47d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0de779b80 1 --2- 172.21.15.6:0/116712573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60bf1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0b3400700 0 client.26080 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0de779b80 1 -- 172.21.15.6:0/116712573 >> 172.21.15.6:0/116712573 conn(0x555a60b186a0 msgr2=0x555a60be59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:02.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0de779b80 1 -- 172.21.15.6:0/116712573 shutdown_connections 2024-04-26T01:53:02.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0de779b80 1 -- 172.21.15.6:0/116712573 wait complete. 2024-04-26T01:53:02.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.525+0000 7fd0de779b80 20 client.26080 trim_cache size 0 max 16384 2024-04-26T01:53:02.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.547+0000 7fd0de779b80 1 Processor -- start 2024-04-26T01:53:02.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.547+0000 7fd0de779b80 1 -- start start 2024-04-26T01:53:02.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.548+0000 7fd0de779b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60be5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.548+0000 7fd0de779b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60c008e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.548+0000 7fd0de779b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60bfcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.548+0000 7fd0de779b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc1b6290 con 0x555a60bf50f0 2024-04-26T01:53:02.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.548+0000 7fd0de779b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc1b3af0 con 0x555a60bea240 2024-04-26T01:53:02.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.548+0000 7fd0de779b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc1b1960 con 0x555a60be9920 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.548+0000 7fd0d1e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60bfcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.548+0000 7fd0d1400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60be5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.548+0000 7fd0d1e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60bfcdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37096/0 (socket says 172.21.15.6:37096) 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.548+0000 7fd0d1400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60be5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38122/0 (socket says 172.21.15.6:38122) 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.548+0000 7fd0d1e00700 1 -- 172.21.15.6:0/1749481265 learned_addr learned my addr 172.21.15.6:0/1749481265 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.548+0000 7fd0d0a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60c008e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.549+0000 7fd0d1e00700 1 -- 172.21.15.6:0/1749481265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 msgr2=0x555a60be5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.549+0000 7fd0d1e00700 1 --2- 172.21.15.6:0/1749481265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60be5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.549+0000 7fd0d1e00700 1 -- 172.21.15.6:0/1749481265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 msgr2=0x555a60c008e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.549+0000 7fd0d1e00700 1 --2- 172.21.15.6:0/1749481265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60c008e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.549+0000 7fd0d1e00700 1 -- 172.21.15.6:0/1749481265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a60bf78a0 con 0x555a60bf50f0 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.549+0000 7fd0d0a00700 1 --2- 172.21.15.6:0/1749481265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60c008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.549+0000 7fd0d1400700 1 --2- 172.21.15.6:0/1749481265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60be5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.549+0000 7fd0d1e00700 1 --2- 172.21.15.6:0/1749481265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60bfcdf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4009c00 tx=0x7fd0c4009130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0ee93f33abaabc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:02.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.549+0000 7fd0be000700 1 -- 172.21.15.6:0/1749481265 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4004bb0 con 0x555a60bf50f0 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.550+0000 7fd0be000700 1 -- 172.21.15.6:0/1749481265 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0c4004d50 con 0x555a60bf50f0 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.550+0000 7fd0be000700 1 -- 172.21.15.6:0/1749481265 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4007e40 con 0x555a60bf50f0 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.550+0000 7fd0de779b80 1 -- 172.21.15.6:0/1749481265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 msgr2=0x555a60bfcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.550+0000 7fd0de779b80 1 --2- 172.21.15.6:0/1749481265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60bfcdf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4009c00 tx=0x7fd0c4009130 comp rx=0 tx=0).stop 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.550+0000 7fd0de779b80 1 -- 172.21.15.6:0/1749481265 shutdown_connections 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.550+0000 7fd0de779b80 1 --2- 172.21.15.6:0/1749481265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60bfcdf0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.550+0000 7fd0de779b80 1 --2- 172.21.15.6:0/1749481265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60be5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.550+0000 7fd0de779b80 1 --2- 172.21.15.6:0/1749481265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60c008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.550+0000 7fd0de779b80 1 -- 172.21.15.6:0/1749481265 >> 172.21.15.6:0/1749481265 conn(0x555a60b186a0 msgr2=0x555a60be5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.550+0000 7fd0de779b80 1 -- 172.21.15.6:0/1749481265 shutdown_connections 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.550+0000 7fd0de779b80 1 -- 172.21.15.6:0/1749481265 wait complete. 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.551+0000 7fd0de779b80 1 -- start start 2024-04-26T01:53:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.551+0000 7fd0de779b80 1 Processor -- start 2024-04-26T01:53:02.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0de779b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60bfc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0de779b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60bfc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0de779b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60c649f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0de779b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc1b1fb0 con 0x555a60bf50f0 2024-04-26T01:53:02.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0de779b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc1b3420 con 0x555a60bea240 2024-04-26T01:53:02.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0de779b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc1b2680 con 0x555a60be9920 2024-04-26T01:53:02.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0de779b80 10 client.0 _ll_register_callbacks cb 0x555a60ad5820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:02.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0d1400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60bfc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0d1400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60bfc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38124/0 (socket says 172.21.15.6:38124) 2024-04-26T01:53:02.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0d1400700 1 -- 172.21.15.6:0/3393924978 learned_addr learned my addr 172.21.15.6:0/3393924978 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:02.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0d1e00700 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60c649f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0d0a00700 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60bfc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0d1400700 1 -- 172.21.15.6:0/3393924978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 msgr2=0x555a60bfc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0d1400700 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60bfc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0d1400700 1 -- 172.21.15.6:0/3393924978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 msgr2=0x555a60c649f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0d1400700 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60c649f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.552+0000 7fd0d1400700 1 -- 172.21.15.6:0/3393924978 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a60c64fb0 con 0x555a60be9920 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.553+0000 7fd0d1400700 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60bfc120 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd0c009b3f0 tx=0x7fd0c0004300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e25a78a5b9e9db3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.553+0000 7fd0b3400700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.553+0000 7fd0b3400700 1 -- 172.21.15.6:0/3393924978 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c00a14f0 con 0x555a60be9920 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.553+0000 7fd0b3400700 1 -- 172.21.15.6:0/3393924978 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8600+0+0 (secure 0 0 0) 0x7fd0c00ad3b0 con 0x555a60be9920 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.553+0000 7fd0de779b80 1 -- 172.21.15.6:0/3393924978 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0c4004d50 con 0x555a60be9920 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.553+0000 7fd0b3400700 1 -- 172.21.15.6:0/3393924978 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd0c0004b90 con 0x555a60be9920 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.553+0000 7fd0de779b80 10 client.26086 fetch_fsmap learned FSMap version 40 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.553+0000 7fd0de779b80 1 -- 172.21.15.6:0/3393924978 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555a60c64fb0 con 0x555a60be9920 2024-04-26T01:53:02.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.553+0000 7fd0b3400700 1 -- 172.21.15.6:0/3393924978 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0c4004d50 con 0x555a60be9920 2024-04-26T01:53:02.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.554+0000 7fd0b3400700 1 -- 172.21.15.6:0/3393924978 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 40) ==== 8342+0+0 (secure 0 0 0) 0x7fd0c0003a70 con 0x555a60be9920 2024-04-26T01:53:02.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.554+0000 7fd0de779b80 10 client.26086 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T01:53:02.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.554+0000 7fd0de779b80 10 client.26086 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:53:02.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.554+0000 7fd0de779b80 20 client.26086 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:02.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.554+0000 7fd0de779b80 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a60c6c640 0x555a60c6ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:02.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.554+0000 7fd0de779b80 4 client.26086 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5f38731d", "release": true, "await": true}] 2024-04-26T01:53:02.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.554+0000 7fd0de779b80 1 -- 172.21.15.6:0/3393924978 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5f38731d", "release": true, "await": true}) -- 0x555a60b44820 con 0x555a60c6c640 2024-04-26T01:53:02.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.554+0000 7fd0d0a00700 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a60c6c640 0x555a60c6ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:02.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.555+0000 7fd0d0a00700 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a60c6c640 0x555a60c6ea30 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b83b9cbcc272cfcd server_cookie=ac1f01a59ed0e815 in_seq=0 out_seq=0 2024-04-26T01:53:02.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.555+0000 7fd0b3400700 10 client.26086 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0b3400700 1 -- 172.21.15.6:0/3393924978 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x555a60b44820 con 0x555a60c6c640 2024-04-26T01:53:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0b3400700 10 client.26086 handle_command_reply: tid=0 2024-04-26T01:53:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0de779b80 1 client.26086 shutdown 2024-04-26T01:53:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0de779b80 1 -- 172.21.15.6:0/3393924978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 msgr2=0x555a60bfc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0de779b80 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60bfc120 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd0c009b3f0 tx=0x7fd0c0004300 comp rx=0 tx=0).stop 2024-04-26T01:53:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0de779b80 1 -- 172.21.15.6:0/3393924978 shutdown_connections 2024-04-26T01:53:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0de779b80 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a60bf50f0 0x555a60c649f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0de779b80 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a60c6c640 0x555a60c6ea30 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0de779b80 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a60be9920 0x555a60bfc120 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0de779b80 1 --2- 172.21.15.6:0/3393924978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a60bea240 0x555a60bfc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0de779b80 1 -- 172.21.15.6:0/3393924978 >> 172.21.15.6:0/3393924978 conn(0x555a60b186a0 msgr2=0x555a60bff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0b3400700 0 client.26086 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:02.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0de779b80 1 -- 172.21.15.6:0/3393924978 shutdown_connections 2024-04-26T01:53:02.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0de779b80 1 -- 172.21.15.6:0/3393924978 wait complete. 2024-04-26T01:53:02.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.561+0000 7fd0de779b80 20 client.26086 trim_cache size 0 max 16384 2024-04-26T01:53:02.561 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:53:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 40, 2024-04-26T01:53:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:53:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 20, 2024-04-26T01:53:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:53:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "5f38731d": { 2024-04-26T01:53:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 20, 2024-04-26T01:53:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:53:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 75.9, 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:53:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:53:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:53:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:53:02.563 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:53:02.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.562+0000 7fd0d3e00700 1 -- 172.21.15.6:0/126635114 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0b40a3db0 msgr2=0x7fd0b40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:02.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.562+0000 7fd0d3e00700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0b40a3db0 0x7fd0b40a6270 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc108910 tx=0x7fd0b800b410 comp rx=0 tx=0).stop 2024-04-26T01:53:02.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.562+0000 7fd0d3e00700 1 -- 172.21.15.6:0/126635114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0cc1092d0 msgr2=0x7fd0cc1a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.562+0000 7fd0d3e00700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0cc1092d0 0x7fd0cc1a53f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0009990 tx=0x7fd0c00047b0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.562+0000 7fd0d3e00700 1 -- 172.21.15.6:0/126635114 shutdown_connections 2024-04-26T01:53:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.562+0000 7fd0d3e00700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd0b40a3db0 0x7fd0b40a6270 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.562+0000 7fd0d3e00700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0cc10e3b0 0x7fd0cc19f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.562+0000 7fd0d3e00700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0cc1092d0 0x7fd0cc1a53f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.562+0000 7fd0d3e00700 1 --2- 172.21.15.6:0/126635114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc109c80 0x7fd0cc1a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.562+0000 7fd0d3e00700 1 -- 172.21.15.6:0/126635114 >> 172.21.15.6:0/126635114 conn(0x7fd0cc076210 msgr2=0x7fd0cc077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.562+0000 7fd0d3e00700 1 -- 172.21.15.6:0/126635114 shutdown_connections 2024-04-26T01:53:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:02.563+0000 7fd0d3e00700 1 -- 172.21.15.6:0/126635114 wait complete. 2024-04-26T01:53:02.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:02 smithi006 ceph-mon[28419]: pgmap v222: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 1.1 MiB/s wr, 70 op/s 2024-04-26T01:53:02.592 INFO:tasks.workunit.client.0.smithi006.stdout: 1 235842 3.74 MB/sec execute 206 sec latency 23580.515 ms 2024-04-26T01:53:02.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:02 smithi188 ceph-mon[32751]: pgmap v222: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 1.1 MiB/s wr, 70 op/s 2024-04-26T01:53:02.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:02 smithi029 ceph-mon[32775]: pgmap v222: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 1.1 MiB/s wr, 70 op/s 2024-04-26T01:53:03.454 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5f38731d', seconds elapsed: 31.4 2024-04-26T01:53:03.592 INFO:tasks.workunit.client.0.smithi006.stdout: 1 237335 3.76 MB/sec execute 207 sec latency 39.567 ms 2024-04-26T01:53:04.593 INFO:tasks.workunit.client.0.smithi006.stdout: 1 238159 3.77 MB/sec execute 208 sec latency 32.083 ms 2024-04-26T01:53:04.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:04 smithi188 ceph-mon[32751]: pgmap v223: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 800 KiB/s wr, 46 op/s 2024-04-26T01:53:04.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:04 smithi029 ceph-mon[32775]: pgmap v223: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 800 KiB/s wr, 46 op/s 2024-04-26T01:53:04.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:04 smithi006 ceph-mon[28419]: pgmap v223: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 800 KiB/s wr, 46 op/s 2024-04-26T01:53:05.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:05 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:53:05.593 INFO:tasks.workunit.client.0.smithi006.stdout: 1 238947 3.76 MB/sec execute 209 sec latency 79.036 ms 2024-04-26T01:53:05.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:05 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:53:05.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:05 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:53:06.593 INFO:tasks.workunit.client.0.smithi006.stdout: 1 240473 3.78 MB/sec execute 210 sec latency 23.479 ms 2024-04-26T01:53:06.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:06 smithi188 ceph-mon[32751]: pgmap v224: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 800 KiB/s wr, 46 op/s 2024-04-26T01:53:06.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:06 smithi188 ceph-mon[32751]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T01:53:06.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:06 smithi029 ceph-mon[32775]: pgmap v224: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 800 KiB/s wr, 46 op/s 2024-04-26T01:53:06.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:06 smithi029 ceph-mon[32775]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T01:53:06.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:06 smithi006 ceph-mon[28419]: pgmap v224: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 800 KiB/s wr, 46 op/s 2024-04-26T01:53:06.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:06 smithi006 ceph-mon[28419]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T01:53:07.593 INFO:tasks.workunit.client.0.smithi006.stdout: 1 241620 3.80 MB/sec execute 211 sec latency 13.570 ms 2024-04-26T01:53:08.593 INFO:tasks.workunit.client.0.smithi006.stdout: 1 242490 3.80 MB/sec execute 212 sec latency 25.997 ms 2024-04-26T01:53:08.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:08 smithi188 ceph-mon[32751]: pgmap v226: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-26T01:53:08.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:08 smithi029 ceph-mon[32775]: pgmap v226: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-26T01:53:08.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:08 smithi006 ceph-mon[28419]: pgmap v226: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-26T01:53:09.593 INFO:tasks.workunit.client.0.smithi006.stdout: 1 243156 3.79 MB/sec execute 213 sec latency 28.910 ms 2024-04-26T01:53:10.594 INFO:tasks.workunit.client.0.smithi006.stdout: 1 244579 3.82 MB/sec execute 214 sec latency 17.958 ms 2024-04-26T01:53:10.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:10 smithi188 ceph-mon[32751]: pgmap v227: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-26T01:53:10.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:10 smithi188 ceph-mon[32751]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T01:53:10.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:10 smithi029 ceph-mon[32775]: pgmap v227: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-26T01:53:10.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:10 smithi029 ceph-mon[32775]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T01:53:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:10 smithi006 ceph-mon[28419]: pgmap v227: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-26T01:53:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:10 smithi006 ceph-mon[28419]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T01:53:11.594 INFO:tasks.workunit.client.0.smithi006.stdout: 1 245311 3.82 MB/sec execute 215 sec latency 20.233 ms 2024-04-26T01:53:12.594 INFO:tasks.workunit.client.0.smithi006.stdout: 1 246345 3.83 MB/sec execute 216 sec latency 22.772 ms 2024-04-26T01:53:12.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:12 smithi188 ceph-mon[32751]: pgmap v229: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 210 op/s 2024-04-26T01:53:12.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:12 smithi029 ceph-mon[32775]: pgmap v229: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 210 op/s 2024-04-26T01:53:12.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:12 smithi006 ceph-mon[28419]: pgmap v229: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 210 op/s 2024-04-26T01:53:13.594 INFO:tasks.workunit.client.0.smithi006.stdout: 1 247778 3.85 MB/sec execute 217 sec latency 20.514 ms 2024-04-26T01:53:14.594 INFO:tasks.workunit.client.0.smithi006.stdout: 1 248757 3.86 MB/sec execute 218 sec latency 27.301 ms 2024-04-26T01:53:14.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:14 smithi188 ceph-mon[32751]: pgmap v230: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 210 op/s 2024-04-26T01:53:14.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:14 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:53:14.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:14 smithi029 ceph-mon[32775]: pgmap v230: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 210 op/s 2024-04-26T01:53:14.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:14 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:53:14.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:14 smithi006 ceph-mon[28419]: pgmap v230: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 210 op/s 2024-04-26T01:53:14.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:14 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:53:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.002+0000 7f6e7e400640 1 -- 172.21.15.6:0/2569850199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78102f30 msgr2=0x7f6e78103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.002+0000 7f6e7e400640 1 --2- 172.21.15.6:0/2569850199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78102f30 0x7f6e78103310 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009900 tx=0x7f6e6c02a480 comp rx=0 tx=0).stop 2024-04-26T01:53:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.003+0000 7f6e7e400640 1 -- 172.21.15.6:0/2569850199 shutdown_connections 2024-04-26T01:53:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.003+0000 7f6e7e400640 1 --2- 172.21.15.6:0/2569850199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e78103850 0x7f6e7810bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.003+0000 7f6e7e400640 1 --2- 172.21.15.6:0/2569850199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e7810c350 0x7f6e7810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.003+0000 7f6e7e400640 1 --2- 172.21.15.6:0/2569850199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78102f30 0x7f6e78103310 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.003+0000 7f6e7e400640 1 -- 172.21.15.6:0/2569850199 >> 172.21.15.6:0/2569850199 conn(0x7f6e780fc730 msgr2=0x7f6e780feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.003+0000 7f6e7e400640 1 -- 172.21.15.6:0/2569850199 shutdown_connections 2024-04-26T01:53:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.003+0000 7f6e7e400640 1 -- 172.21.15.6:0/2569850199 wait complete. 2024-04-26T01:53:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.004+0000 7f6e7e400640 1 Processor -- start 2024-04-26T01:53:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.004+0000 7f6e7e400640 1 -- start start 2024-04-26T01:53:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.005+0000 7f6e7e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e78102f30 0x7f6e78198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.005+0000 7f6e7e400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103850 0x7f6e78198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.005+0000 7f6e7e400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e7810c350 0x7f6e7819ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.005+0000 7f6e7e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6e78111fe0 con 0x7f6e78102f30 2024-04-26T01:53:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.005+0000 7f6e7e400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6e78111e60 con 0x7f6e78103850 2024-04-26T01:53:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.005+0000 7f6e7e400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6e78112160 con 0x7f6e7810c350 2024-04-26T01:53:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.005+0000 7f6e77e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103850 0x7f6e78198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.005+0000 7f6e77e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103850 0x7f6e78198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:45692/0 (socket says 172.21.15.6:45692) 2024-04-26T01:53:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.005+0000 7f6e77e00640 1 -- 172.21.15.6:0/2736507114 learned_addr learned my addr 172.21.15.6:0/2736507114 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.005+0000 7f6e7d000640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e78102f30 0x7f6e78198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.005+0000 7f6e7da00640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e7810c350 0x7f6e7819ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.006+0000 7f6e77e00640 1 -- 172.21.15.6:0/2736507114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e7810c350 msgr2=0x7f6e7819ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.006+0000 7f6e77e00640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e7810c350 0x7f6e7819ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.006+0000 7f6e77e00640 1 -- 172.21.15.6:0/2736507114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e78102f30 msgr2=0x7f6e78198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.006+0000 7f6e77e00640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e78102f30 0x7f6e78198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.006+0000 7f6e77e00640 1 -- 172.21.15.6:0/2736507114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e7819d530 con 0x7f6e78103850 2024-04-26T01:53:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.006+0000 7f6e77e00640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103850 0x7f6e78198a80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6e6400d950 tx=0x7f6e6400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.006+0000 7f6e75600640 1 -- 172.21.15.6:0/2736507114 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e64017030 con 0x7f6e78103850 2024-04-26T01:53:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.006+0000 7f6e75600640 1 -- 172.21.15.6:0/2736507114 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e64004020 con 0x7f6e78103850 2024-04-26T01:53:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.006+0000 7f6e7e400640 1 -- 172.21.15.6:0/2736507114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e7819d820 con 0x7f6e78103850 2024-04-26T01:53:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.007+0000 7f6e75600640 1 -- 172.21.15.6:0/2736507114 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e640055d0 con 0x7f6e78103850 2024-04-26T01:53:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.007+0000 7f6e7e400640 1 -- 172.21.15.6:0/2736507114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e781a50b0 con 0x7f6e78103850 2024-04-26T01:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.011+0000 7f6e827fe740 1 Processor -- start 2024-04-26T01:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.012+0000 7f6e827fe740 1 -- start start 2024-04-26T01:53:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.012+0000 7f6e75600640 1 -- 172.21.15.6:0/2736507114 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f6e64009a30 con 0x7f6e78103850 2024-04-26T01:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.012+0000 7f6e827fe740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 0x55d30f193c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.012+0000 7f6e827fe740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f194170 0x55d30f1a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.012+0000 7f6e827fe740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f1a2fa0 0x55d30f1a5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.012+0000 7f6e827fe740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d30f0f9900 con 0x55d30f1a2fa0 2024-04-26T01:53:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.012+0000 7f6e827fe740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d30f09fef0 con 0x55d30f193850 2024-04-26T01:53:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.012+0000 7f6e827fe740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d30f0f58c0 con 0x55d30f194170 2024-04-26T01:53:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.013+0000 7f6e7da00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 0x55d30f193c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.013+0000 7f6e7da00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 0x55d30f193c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:45700/0 (socket says 172.21.15.6:45700) 2024-04-26T01:53:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.013+0000 7f6e7da00640 1 -- 172.21.15.6:0/1070946219 learned_addr learned my addr 172.21.15.6:0/1070946219 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.013+0000 7f6e7d000640 1 --2- 172.21.15.6:0/1070946219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f194170 0x55d30f1a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.012 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.013+0000 7f6e77e00640 1 --2- 172.21.15.6:0/1070946219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f1a2fa0 0x55d30f1a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.013+0000 7f6e7da00640 1 -- 172.21.15.6:0/1070946219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f194170 msgr2=0x55d30f1a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.013+0000 7f6e7da00640 1 --2- 172.21.15.6:0/1070946219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f194170 0x55d30f1a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.013+0000 7f6e7da00640 1 -- 172.21.15.6:0/1070946219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f1a2fa0 msgr2=0x55d30f1a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.013+0000 7f6e7da00640 1 --2- 172.21.15.6:0/1070946219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f1a2fa0 0x55d30f1a5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.013+0000 7f6e7da00640 1 -- 172.21.15.6:0/1070946219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d30ee0c210 con 0x55d30f193850 2024-04-26T01:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.014+0000 7f6e7da00640 1 --2- 172.21.15.6:0/1070946219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 0x55d30f193c30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009f50 tx=0x7f6e700118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5c8c988ab2dd233 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.014+0000 7f6e6a000640 1 -- 172.21.15.6:0/1070946219 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e70004d20 con 0x55d30f193850 2024-04-26T01:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.014+0000 7f6e6a000640 1 -- 172.21.15.6:0/1070946219 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e70004050 con 0x55d30f193850 2024-04-26T01:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.014+0000 7f6e6a000640 1 -- 172.21.15.6:0/1070946219 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e70005280 con 0x55d30f193850 2024-04-26T01:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.014+0000 7f6e75600640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f6e600a3d80 0x7f6e600a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.015+0000 7f6e75600640 1 -- 172.21.15.6:0/2736507114 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f6e640cfff0 con 0x7f6e78103850 2024-04-26T01:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.015+0000 7f6e7d000640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f6e600a3d80 0x7f6e600a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.015+0000 7f6e827fe740 1 -- 172.21.15.6:0/1070946219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 msgr2=0x55d30f193c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:15.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.015+0000 7f6e827fe740 1 --2- 172.21.15.6:0/1070946219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 0x55d30f193c30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009f50 tx=0x7f6e700118d0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.015+0000 7f6e827fe740 1 -- 172.21.15.6:0/1070946219 shutdown_connections 2024-04-26T01:53:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.015+0000 7f6e827fe740 1 --2- 172.21.15.6:0/1070946219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f1a2fa0 0x55d30f1a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.015+0000 7f6e827fe740 1 --2- 172.21.15.6:0/1070946219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f194170 0x55d30f1a2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.015+0000 7f6e827fe740 1 --2- 172.21.15.6:0/1070946219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 0x55d30f193c30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.015+0000 7f6e827fe740 1 -- 172.21.15.6:0/1070946219 >> 172.21.15.6:0/1070946219 conn(0x55d30f18fec0 msgr2=0x55d30f1902a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.015+0000 7f6e827fe740 1 -- 172.21.15.6:0/1070946219 shutdown_connections 2024-04-26T01:53:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.015+0000 7f6e827fe740 1 -- 172.21.15.6:0/1070946219 wait complete. 2024-04-26T01:53:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.016+0000 7f6e827fe740 1 -- start start 2024-04-26T01:53:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.016+0000 7f6e7d000640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f6e600a3d80 0x7f6e600a6240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c031440 tx=0x7f6e6c005b90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:15.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.016+0000 7f6e827fe740 1 Processor -- start 2024-04-26T01:53:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.017+0000 7f6e827fe740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 0x55d30f19fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.017+0000 7f6e827fe740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f194170 0x55d30f1a0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.017+0000 7f6e827fe740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f1a2fa0 0x55d30f1a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.017+0000 7f6e827fe740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d30f1a5bd0 con 0x55d30f194170 2024-04-26T01:53:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.017+0000 7f6e827fe740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d30f0b3310 con 0x55d30f193850 2024-04-26T01:53:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.017+0000 7f6e827fe740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d30f1a5d50 con 0x55d30f1a2fa0 2024-04-26T01:53:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.017+0000 7f6e827fe740 10 client.0 _ll_register_callbacks cb 0x55d30f081cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.017+0000 7f6e7d000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 0x55d30f19fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.017+0000 7f6e7d000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 0x55d30f19fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:45714/0 (socket says 172.21.15.6:45714) 2024-04-26T01:53:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.017+0000 7f6e7d000640 1 -- 172.21.15.6:0/2898752827 learned_addr learned my addr 172.21.15.6:0/2898752827 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.017+0000 7f6e77e00640 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f194170 0x55d30f1a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.018+0000 7f6e7da00640 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f1a2fa0 0x55d30f1a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.018+0000 7f6e7d000640 1 -- 172.21.15.6:0/2898752827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f1a2fa0 msgr2=0x55d30f1a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.018+0000 7f6e7d000640 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f1a2fa0 0x55d30f1a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.018+0000 7f6e7d000640 1 -- 172.21.15.6:0/2898752827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f194170 msgr2=0x55d30f1a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.018+0000 7f6e7d000640 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f194170 0x55d30f1a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.019 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.018+0000 7f6e7d000640 1 -- 172.21.15.6:0/2898752827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d30eed8620 con 0x55d30f193850 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.018+0000 7f6e77e00640 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f194170 0x55d30f1a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.018+0000 7f6e7da00640 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f1a2fa0 0x55d30f1a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.018+0000 7f6e7d000640 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 0x55d30f19fee0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c033000 tx=0x7f6e6c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bb3cf9c05373529 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.018+0000 7f6e5ea00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.019+0000 7f6e5ea00640 1 -- 172.21.15.6:0/2898752827 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c04d070 con 0x55d30f193850 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.019+0000 7f6e827fe740 1 -- 172.21.15.6:0/2898752827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e70004050 con 0x55d30f193850 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.019+0000 7f6e5ea00640 1 -- 172.21.15.6:0/2898752827 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f6e6c005230 con 0x55d30f193850 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.019+0000 7f6e5ea00640 1 -- 172.21.15.6:0/2898752827 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f6e6c057050 con 0x55d30f193850 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.019+0000 7f6e827fe740 10 client.26051 fetch_fsmap learned FSMap version 41 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.019+0000 7f6e827fe740 1 -- 172.21.15.6:0/2898752827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55d30f1b1e70 con 0x55d30f193850 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.020+0000 7f6e5ea00640 1 -- 172.21.15.6:0/2898752827 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e70004050 con 0x55d30f193850 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.020+0000 7f6e5ea00640 1 -- 172.21.15.6:0/2898752827 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f6e6c003900 con 0x55d30f193850 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.020+0000 7f6e827fe740 10 client.26051 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.020+0000 7f6e827fe740 10 client.26051 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:15.020 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.020+0000 7f6e827fe740 20 client.26051 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:15.021 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.021+0000 7f6e827fe740 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d30f1b3330 0x55d30f1b5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.021 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.021+0000 7f6e827fe740 4 client.26051 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:15.021 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.021+0000 7f6e827fe740 1 -- 172.21.15.6:0/2898752827 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d30f10d8b0 con 0x55d30f1b3330 2024-04-26T01:53:15.021 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.021+0000 7f6e77e00640 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d30f1b3330 0x55d30f1b5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.021+0000 7f6e5ea00640 10 client.26051 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.021+0000 7f6e77e00640 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d30f1b3330 0x55d30f1b5780 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1efcee58e314d22a server_cookie=bd9f08a9b16a4ed6 in_seq=0 out_seq=0 2024-04-26T01:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.022+0000 7f6e5ea00640 1 -- 172.21.15.6:0/2898752827 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d30f10d8b0 con 0x55d30f1b3330 2024-04-26T01:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.022+0000 7f6e5ea00640 10 client.26051 handle_command_reply: tid=0 2024-04-26T01:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.023+0000 7f6e827fe740 1 client.26051 shutdown 2024-04-26T01:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.023+0000 7f6e827fe740 1 -- 172.21.15.6:0/2898752827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 msgr2=0x55d30f19fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.023+0000 7f6e827fe740 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 0x55d30f19fee0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c033000 tx=0x7f6e6c009260 comp rx=0 tx=0).stop 2024-04-26T01:53:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.023+0000 7f6e827fe740 1 -- 172.21.15.6:0/2898752827 shutdown_connections 2024-04-26T01:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.023+0000 7f6e827fe740 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d30f1b3330 0x55d30f1b5780 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.023+0000 7f6e827fe740 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f194170 0x55d30f1a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.023+0000 7f6e827fe740 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f1a2fa0 0x55d30f1a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.023+0000 7f6e827fe740 1 --2- 172.21.15.6:0/2898752827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f193850 0x55d30f19fee0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.023+0000 7f6e827fe740 1 -- 172.21.15.6:0/2898752827 >> 172.21.15.6:0/2898752827 conn(0x55d30f18fec0 msgr2=0x55d30f195d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.023+0000 7f6e5ea00640 0 client.26051 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.023+0000 7f6e827fe740 1 -- 172.21.15.6:0/2898752827 shutdown_connections 2024-04-26T01:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.023+0000 7f6e827fe740 1 -- 172.21.15.6:0/2898752827 wait complete. 2024-04-26T01:53:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.024+0000 7f6e827fe740 20 client.26051 trim_cache size 0 max 16384 2024-04-26T01:53:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.043+0000 7f6e827fe740 1 Processor -- start 2024-04-26T01:53:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.043+0000 7f6e827fe740 1 -- start start 2024-04-26T01:53:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.043+0000 7f6e827fe740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f194170 0x55d30f1aea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.043+0000 7f6e827fe740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f1a2fa0 0x55d30f1bb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.043+0000 7f6e827fe740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f1a2720 0x55d30f1bb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.043+0000 7f6e827fe740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781ae1b0 con 0x55d30f194170 2024-04-26T01:53:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.043+0000 7f6e827fe740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781aba20 con 0x55d30f1a2fa0 2024-04-26T01:53:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.043+0000 7f6e827fe740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781a9890 con 0x55d30f1a2720 2024-04-26T01:53:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.044+0000 7f6e77e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f1a2fa0 0x55d30f1bb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.044+0000 7f6e77e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f1a2fa0 0x55d30f1bb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:45722/0 (socket says 172.21.15.6:45722) 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.044+0000 7f6e77e00640 1 -- 172.21.15.6:0/591559770 learned_addr learned my addr 172.21.15.6:0/591559770 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.044+0000 7f6e7da00640 1 --2- 172.21.15.6:0/591559770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f1a2720 0x55d30f1bb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.044+0000 7f6e7d000640 1 --2- 172.21.15.6:0/591559770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f194170 0x55d30f1aea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.044+0000 7f6e77e00640 1 -- 172.21.15.6:0/591559770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f1a2720 msgr2=0x55d30f1bb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.044+0000 7f6e77e00640 1 --2- 172.21.15.6:0/591559770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f1a2720 0x55d30f1bb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.044+0000 7f6e77e00640 1 -- 172.21.15.6:0/591559770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f194170 msgr2=0x55d30f1aea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.044+0000 7f6e77e00640 1 --2- 172.21.15.6:0/591559770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f194170 0x55d30f1aea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.044+0000 7f6e77e00640 1 -- 172.21.15.6:0/591559770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d30f1aff30 con 0x55d30f1a2fa0 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.044+0000 7f6e7da00640 1 --2- 172.21.15.6:0/591559770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f1a2720 0x55d30f1bb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.044+0000 7f6e77e00640 1 --2- 172.21.15.6:0/591559770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f1a2fa0 0x55d30f1bb090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6e6409bc20 tx=0x7f6e640d2480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f7469e069d540c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.045+0000 7f6e69600640 1 -- 172.21.15.6:0/591559770 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6409b880 con 0x55d30f1a2fa0 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.045+0000 7f6e69600640 1 -- 172.21.15.6:0/591559770 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e6409c620 con 0x55d30f1a2fa0 2024-04-26T01:53:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.045+0000 7f6e69600640 1 -- 172.21.15.6:0/591559770 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e640041d0 con 0x55d30f1a2fa0 2024-04-26T01:53:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.045+0000 7f6e827fe740 1 -- 172.21.15.6:0/591559770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f1a2fa0 msgr2=0x55d30f1bb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.045+0000 7f6e827fe740 1 --2- 172.21.15.6:0/591559770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f1a2fa0 0x55d30f1bb090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6e6409bc20 tx=0x7f6e640d2480 comp rx=0 tx=0).stop 2024-04-26T01:53:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.046+0000 7f6e827fe740 1 -- 172.21.15.6:0/591559770 shutdown_connections 2024-04-26T01:53:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.046+0000 7f6e827fe740 1 --2- 172.21.15.6:0/591559770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f1a2720 0x55d30f1bb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.046+0000 7f6e827fe740 1 --2- 172.21.15.6:0/591559770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f1a2fa0 0x55d30f1bb090 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.046+0000 7f6e827fe740 1 --2- 172.21.15.6:0/591559770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f194170 0x55d30f1aea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.046+0000 7f6e827fe740 1 -- 172.21.15.6:0/591559770 >> 172.21.15.6:0/591559770 conn(0x55d30f18fec0 msgr2=0x55d30f19b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.046+0000 7f6e827fe740 1 -- 172.21.15.6:0/591559770 shutdown_connections 2024-04-26T01:53:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.046+0000 7f6e827fe740 1 -- 172.21.15.6:0/591559770 wait complete. 2024-04-26T01:53:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.046+0000 7f6e827fe740 1 -- start start 2024-04-26T01:53:15.046 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.047+0000 7f6e827fe740 1 Processor -- start 2024-04-26T01:53:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e827fe740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f194170 0x55d30f198c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e827fe740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f1a2720 0x55d30f199140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e827fe740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f1a2fa0 0x55d30f199680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e827fe740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781a9ee0 con 0x55d30f1a2720 2024-04-26T01:53:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e827fe740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781ab350 con 0x55d30f1a2fa0 2024-04-26T01:53:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e827fe740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781aa5b0 con 0x55d30f194170 2024-04-26T01:53:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e827fe740 10 client.0 _ll_register_callbacks cb 0x7f6e6c032ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e7d000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f194170 0x55d30f198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e7d000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f194170 0x55d30f198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:48420/0 (socket says 172.21.15.6:48420) 2024-04-26T01:53:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e7d000640 1 -- 172.21.15.6:0/112563644 learned_addr learned my addr 172.21.15.6:0/112563644 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e77e00640 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f1a2720 0x55d30f199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e7da00640 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f1a2fa0 0x55d30f199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.048+0000 7f6e7d000640 1 -- 172.21.15.6:0/112563644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f1a2fa0 msgr2=0x55d30f199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.049+0000 7f6e7d000640 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f1a2fa0 0x55d30f199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.049+0000 7f6e7d000640 1 -- 172.21.15.6:0/112563644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f1a2720 msgr2=0x55d30f199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.049+0000 7f6e7d000640 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f1a2720 0x55d30f199140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.049+0000 7f6e7d000640 1 -- 172.21.15.6:0/112563644 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d30f1aa240 con 0x55d30f194170 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.049+0000 7f6e77e00640 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f1a2720 0x55d30f199140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.049+0000 7f6e7d000640 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f194170 0x55d30f198c00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009900 tx=0x7f6e6c004550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b815f1c4712889c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.049+0000 7f6e5ea00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.049+0000 7f6e5ea00640 1 -- 172.21.15.6:0/112563644 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c04d040 con 0x55d30f194170 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.049+0000 7f6e827fe740 1 -- 172.21.15.6:0/112563644 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e6409c620 con 0x55d30f194170 2024-04-26T01:53:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.049+0000 7f6e5ea00640 1 -- 172.21.15.6:0/112563644 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f6e6c0037f0 con 0x55d30f194170 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.050+0000 7f6e5ea00640 1 -- 172.21.15.6:0/112563644 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f6e6c035040 con 0x55d30f194170 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.050+0000 7f6e827fe740 10 client.26104 fetch_fsmap learned FSMap version 41 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.050+0000 7f6e827fe740 1 -- 172.21.15.6:0/112563644 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55d30f1ab510 con 0x55d30f194170 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.050+0000 7f6e5ea00640 1 -- 172.21.15.6:0/112563644 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e6409c620 con 0x55d30f194170 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.050+0000 7f6e5ea00640 1 -- 172.21.15.6:0/112563644 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f6e6c03d660 con 0x55d30f194170 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.050+0000 7f6e827fe740 10 client.26104 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.050+0000 7f6e827fe740 10 client.26104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.050+0000 7f6e827fe740 20 client.26104 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.050+0000 7f6e827fe740 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d30f21e2a0 0x55d30f220690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.050+0000 7f6e827fe740 4 client.26104 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.050+0000 7f6e827fe740 1 -- 172.21.15.6:0/112563644 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d30f107320 con 0x55d30f21e2a0 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.051+0000 7f6e77e00640 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d30f21e2a0 0x55d30f220690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.051+0000 7f6e77e00640 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d30f21e2a0 0x55d30f220690 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcc5f4c4dd7d655d server_cookie=194ca36e87c9250b in_seq=0 out_seq=0 2024-04-26T01:53:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.051+0000 7f6e5ea00640 10 client.26104 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:15.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.102+0000 7f6e5ea00640 1 -- 172.21.15.6:0/112563644 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d30f107320 con 0x55d30f21e2a0 2024-04-26T01:53:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.102+0000 7f6e5ea00640 10 client.26104 handle_command_reply: tid=0 2024-04-26T01:53:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.102+0000 7f6e827fe740 1 client.26104 shutdown 2024-04-26T01:53:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.103+0000 7f6e827fe740 1 -- 172.21.15.6:0/112563644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f194170 msgr2=0x55d30f198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.103+0000 7f6e827fe740 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f194170 0x55d30f198c00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009900 tx=0x7f6e6c004550 comp rx=0 tx=0).stop 2024-04-26T01:53:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.103+0000 7f6e827fe740 1 -- 172.21.15.6:0/112563644 shutdown_connections 2024-04-26T01:53:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.103+0000 7f6e827fe740 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d30f21e2a0 0x55d30f220690 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.103+0000 7f6e827fe740 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d30f1a2720 0x55d30f199140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.103+0000 7f6e827fe740 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d30f1a2fa0 0x55d30f199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.103+0000 7f6e827fe740 1 --2- 172.21.15.6:0/112563644 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d30f194170 0x55d30f198c00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.103+0000 7f6e5ea00640 0 client.26104 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:15.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.103+0000 7f6e827fe740 1 -- 172.21.15.6:0/112563644 >> 172.21.15.6:0/112563644 conn(0x55d30f18fec0 msgr2=0x55d30f18e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:15.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.103+0000 7f6e827fe740 1 -- 172.21.15.6:0/112563644 shutdown_connections 2024-04-26T01:53:15.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.103+0000 7f6e827fe740 1 -- 172.21.15.6:0/112563644 wait complete. 2024-04-26T01:53:15.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.104+0000 7f6e827fe740 20 client.26104 trim_cache size 0 max 16384 2024-04-26T01:53:15.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:53:15.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:53:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:53:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:53:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.104+0000 7f6e7e400640 1 -- 172.21.15.6:0/2736507114 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f6e600a3d80 msgr2=0x7f6e600a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.104+0000 7f6e7e400640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f6e600a3d80 0x7f6e600a6240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c031440 tx=0x7f6e6c005b90 comp rx=0 tx=0).stop 2024-04-26T01:53:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.104+0000 7f6e7e400640 1 -- 172.21.15.6:0/2736507114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103850 msgr2=0x7f6e78198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.104+0000 7f6e7e400640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103850 0x7f6e78198a80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6e6400d950 tx=0x7f6e6400de10 comp rx=0 tx=0).stop 2024-04-26T01:53:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.105+0000 7f6e7e400640 1 -- 172.21.15.6:0/2736507114 shutdown_connections 2024-04-26T01:53:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.105+0000 7f6e7e400640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f6e600a3d80 0x7f6e600a6240 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.105+0000 7f6e7e400640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e7810c350 0x7f6e7819ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.105+0000 7f6e7e400640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e78103850 0x7f6e78198a80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.105+0000 7f6e7e400640 1 --2- 172.21.15.6:0/2736507114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e78102f30 0x7f6e78198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.105+0000 7f6e7e400640 1 -- 172.21.15.6:0/2736507114 >> 172.21.15.6:0/2736507114 conn(0x7f6e780fc730 msgr2=0x7f6e780fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.105+0000 7f6e7e400640 1 -- 172.21.15.6:0/2736507114 shutdown_connections 2024-04-26T01:53:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:15.105+0000 7f6e7e400640 1 -- 172.21.15.6:0/2736507114 wait complete. 2024-04-26T01:53:15.594 INFO:tasks.workunit.client.0.smithi006.stdout: 1 249517 3.85 MB/sec execute 219 sec latency 105.488 ms 2024-04-26T01:53:15.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:15 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:53:15.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:15 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:53:15.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:15 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:53:15.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:15 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:53:15.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:15 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:53:15.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:15 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:53:15.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:15 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:53:15.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:15 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:53:15.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:15 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:53:16.594 INFO:tasks.workunit.client.0.smithi006.stdout: 1 250925 3.86 MB/sec execute 220 sec latency 26.124 ms 2024-04-26T01:53:16.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:16 smithi188 ceph-mon[32751]: pgmap v231: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-04-26T01:53:16.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:16 smithi029 ceph-mon[32775]: pgmap v231: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-04-26T01:53:16.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:16 smithi006 ceph-mon[28419]: pgmap v231: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-04-26T01:53:17.594 INFO:tasks.workunit.client.0.smithi006.stdout: 1 252274 3.89 MB/sec execute 221 sec latency 12.385 ms 2024-04-26T01:53:18.594 INFO:tasks.workunit.client.0.smithi006.stdout: 1 253289 3.90 MB/sec execute 222 sec latency 24.097 ms 2024-04-26T01:53:18.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:18 smithi188 ceph-mon[32751]: pgmap v232: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 175 op/s 2024-04-26T01:53:18.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:18 smithi029 ceph-mon[32775]: pgmap v232: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 175 op/s 2024-04-26T01:53:18.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:18 smithi006 ceph-mon[28419]: pgmap v232: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 175 op/s 2024-04-26T01:53:19.594 INFO:tasks.workunit.client.0.smithi006.stdout: 1 254129 3.88 MB/sec execute 223 sec latency 22.822 ms 2024-04-26T01:53:20.595 INFO:tasks.workunit.client.0.smithi006.stdout: 1 255653 3.92 MB/sec execute 224 sec latency 12.053 ms 2024-04-26T01:53:20.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:20 smithi029 ceph-mon[32775]: pgmap v233: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 175 op/s 2024-04-26T01:53:20.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:20 smithi006 ceph-mon[28419]: pgmap v233: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 175 op/s 2024-04-26T01:53:20.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:20 smithi188 ceph-mon[32751]: pgmap v233: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 175 op/s 2024-04-26T01:53:21.595 INFO:tasks.workunit.client.0.smithi006.stdout: 1 256480 3.92 MB/sec execute 225 sec latency 17.221 ms 2024-04-26T01:53:22.595 INFO:tasks.workunit.client.0.smithi006.stdout: 1 257919 3.93 MB/sec execute 226 sec latency 21.808 ms 2024-04-26T01:53:22.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:22 smithi029 ceph-mon[32775]: pgmap v234: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 226 op/s 2024-04-26T01:53:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:22 smithi006 ceph-mon[28419]: pgmap v234: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 226 op/s 2024-04-26T01:53:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:22 smithi188 ceph-mon[32751]: pgmap v234: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 226 op/s 2024-04-26T01:53:23.595 INFO:tasks.workunit.client.0.smithi006.stdout: 1 259330 3.96 MB/sec execute 227 sec latency 16.418 ms 2024-04-26T01:53:24.595 INFO:tasks.workunit.client.0.smithi006.stdout: 1 260220 3.95 MB/sec execute 228 sec latency 17.104 ms 2024-04-26T01:53:24.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:24 smithi029 ceph-mon[32775]: pgmap v235: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 143 op/s 2024-04-26T01:53:24.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:24 smithi006 ceph-mon[28419]: pgmap v235: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 143 op/s 2024-04-26T01:53:24.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:24 smithi188 ceph-mon[32751]: pgmap v235: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 143 op/s 2024-04-26T01:53:25.595 INFO:tasks.workunit.client.0.smithi006.stdout: 1 260948 3.95 MB/sec execute 229 sec latency 23.312 ms 2024-04-26T01:53:26.596 INFO:tasks.workunit.client.0.smithi006.stdout: 1 262591 3.98 MB/sec execute 230 sec latency 15.116 ms 2024-04-26T01:53:26.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:26 smithi029 ceph-mon[32775]: pgmap v236: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 143 op/s 2024-04-26T01:53:26.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:26 smithi006 ceph-mon[28419]: pgmap v236: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 143 op/s 2024-04-26T01:53:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:26 smithi188 ceph-mon[32751]: pgmap v236: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 143 op/s 2024-04-26T01:53:27.596 INFO:tasks.workunit.client.0.smithi006.stdout: 1 263315 3.97 MB/sec execute 231 sec latency 16.135 ms 2024-04-26T01:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.257+0000 7ff77a800640 1 -- 172.21.15.6:0/1274449126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff774102970 msgr2=0x7ff7741069b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.257+0000 7ff77a800640 1 --2- 172.21.15.6:0/1274449126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff774102970 0x7ff7741069b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff760009900 tx=0x7ff76002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.258+0000 7ff77a800640 1 -- 172.21.15.6:0/1274449126 shutdown_connections 2024-04-26T01:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.258+0000 7ff77a800640 1 --2- 172.21.15.6:0/1274449126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff774102970 0x7ff7741069b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.258+0000 7ff77a800640 1 --2- 172.21.15.6:0/1274449126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff774107090 0x7ff77410ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.258+0000 7ff77a800640 1 --2- 172.21.15.6:0/1274449126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff774101fc0 0x7ff7741023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.258+0000 7ff77a800640 1 -- 172.21.15.6:0/1274449126 >> 172.21.15.6:0/1274449126 conn(0x7ff7740fd880 msgr2=0x7ff7740ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.258+0000 7ff77a800640 1 -- 172.21.15.6:0/1274449126 shutdown_connections 2024-04-26T01:53:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.258+0000 7ff77a800640 1 -- 172.21.15.6:0/1274449126 wait complete. 2024-04-26T01:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.259+0000 7ff77a800640 1 Processor -- start 2024-04-26T01:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.259+0000 7ff77a800640 1 -- start start 2024-04-26T01:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff77a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff774101fc0 0x7ff7741986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff77a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff774107090 0x7ff774198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff77a800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff77419cf10 0x7ff77419d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff77a800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff774112000 con 0x7ff774107090 2024-04-26T01:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff77a800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff774111e80 con 0x7ff774101fc0 2024-04-26T01:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff77a800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff774112180 con 0x7ff77419cf10 2024-04-26T01:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff779400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff774101fc0 0x7ff7741986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff779400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff774101fc0 0x7ff7741986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:57490/0 (socket says 172.21.15.6:57490) 2024-04-26T01:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff778a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff774107090 0x7ff774198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff779400640 1 -- 172.21.15.6:0/2016007323 learned_addr learned my addr 172.21.15.6:0/2016007323 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff779e00640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff77419cf10 0x7ff77419d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff779400640 1 -- 172.21.15.6:0/2016007323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff77419cf10 msgr2=0x7ff77419d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff779400640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff77419cf10 0x7ff77419d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff779400640 1 -- 172.21.15.6:0/2016007323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff774107090 msgr2=0x7ff774198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff779400640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff774107090 0x7ff774198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.260+0000 7ff779400640 1 -- 172.21.15.6:0/2016007323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77419da00 con 0x7ff774101fc0 2024-04-26T01:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.261+0000 7ff779400640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff774101fc0 0x7ff7741986a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff76800e9f0 tx=0x7ff76800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.261+0000 7ff772000640 1 -- 172.21.15.6:0/2016007323 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76800cd80 con 0x7ff774101fc0 2024-04-26T01:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.261+0000 7ff772000640 1 -- 172.21.15.6:0/2016007323 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7680108e0 con 0x7ff774101fc0 2024-04-26T01:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.261+0000 7ff77a800640 1 -- 172.21.15.6:0/2016007323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff77419dcf0 con 0x7ff774101fc0 2024-04-26T01:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.261+0000 7ff772000640 1 -- 172.21.15.6:0/2016007323 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768018980 con 0x7ff774101fc0 2024-04-26T01:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.261+0000 7ff77a800640 1 -- 172.21.15.6:0/2016007323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7741a5500 con 0x7ff774101fc0 2024-04-26T01:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff77ebd3740 1 Processor -- start 2024-04-26T01:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff77ebd3740 1 -- start start 2024-04-26T01:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff772000640 1 -- 172.21.15.6:0/2016007323 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7ff768010430 con 0x7ff774101fc0 2024-04-26T01:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff77ebd3740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189a890 0x5621c189ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff77ebd3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189b1b0 0x5621c18a9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff77ebd3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c18a9fa0 0x5621c18ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff77ebd3740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5621c1800900 con 0x5621c189a890 2024-04-26T01:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff77ebd3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5621c17a6ef0 con 0x5621c189b1b0 2024-04-26T01:53:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff77ebd3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5621c17fc8c0 con 0x5621c18a9fa0 2024-04-26T01:53:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff779400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189b1b0 0x5621c18a9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff779e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189a890 0x5621c189ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff779400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189b1b0 0x5621c18a9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:57496/0 (socket says 172.21.15.6:57496) 2024-04-26T01:53:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff778a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c18a9fa0 0x5621c18ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.266+0000 7ff779400640 1 -- 172.21.15.6:0/1812744069 learned_addr learned my addr 172.21.15.6:0/1812744069 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.265+0000 7ff779e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189a890 0x5621c189ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48154/0 (socket says 172.21.15.6:48154) 2024-04-26T01:53:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.266+0000 7ff779400640 1 -- 172.21.15.6:0/1812744069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c18a9fa0 msgr2=0x5621c18ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.266+0000 7ff779400640 1 --2- 172.21.15.6:0/1812744069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c18a9fa0 0x5621c18ac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.266+0000 7ff779400640 1 -- 172.21.15.6:0/1812744069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189a890 msgr2=0x5621c189ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.266+0000 7ff779400640 1 --2- 172.21.15.6:0/1812744069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189a890 0x5621c189ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.266+0000 7ff779400640 1 -- 172.21.15.6:0/1812744069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621c1513210 con 0x5621c189b1b0 2024-04-26T01:53:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.266+0000 7ff779e00640 1 --2- 172.21.15.6:0/1812744069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189a890 0x5621c189ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.266+0000 7ff779400640 1 --2- 172.21.15.6:0/1812744069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189b1b0 0x5621c18a9a60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff7680d4b60 tx=0x7ff7680d9640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=154d41e0b10b858b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.267+0000 7ff766600640 1 -- 172.21.15.6:0/1812744069 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7680e0020 con 0x5621c189b1b0 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.267+0000 7ff766600640 1 -- 172.21.15.6:0/1812744069 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7680dc8a0 con 0x5621c189b1b0 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.267+0000 7ff766600640 1 -- 172.21.15.6:0/1812744069 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7680db5b0 con 0x5621c189b1b0 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.267+0000 7ff772000640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff75c0a3d80 0x7ff75c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.267+0000 7ff772000640 1 -- 172.21.15.6:0/2016007323 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7ff7680d0b80 con 0x7ff774101fc0 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.267+0000 7ff779e00640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff75c0a3d80 0x7ff75c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.267+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1812744069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189b1b0 msgr2=0x5621c18a9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.267+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1812744069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189b1b0 0x5621c18a9a60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff7680d4b60 tx=0x7ff7680d9640 comp rx=0 tx=0).stop 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.268+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1812744069 shutdown_connections 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.268+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1812744069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c18a9fa0 0x5621c18ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.268+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1812744069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189b1b0 0x5621c18a9a60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.268+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1812744069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189a890 0x5621c189ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.268+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1812744069 >> 172.21.15.6:0/1812744069 conn(0x5621c1896ec0 msgr2=0x5621c18972a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.268+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1812744069 shutdown_connections 2024-04-26T01:53:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.268+0000 7ff779e00640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff75c0a3d80 0x7ff75c0a6240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff76c009fd0 tx=0x7ff76c009500 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.268+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1812744069 wait complete. 2024-04-26T01:53:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.268+0000 7ff77ebd3740 1 -- start start 2024-04-26T01:53:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.269+0000 7ff77ebd3740 1 Processor -- start 2024-04-26T01:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.269+0000 7ff77ebd3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189a890 0x5621c18a6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.269+0000 7ff77ebd3740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189b1b0 0x5621c18a73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.269+0000 7ff77ebd3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c18a9fa0 0x5621c18a7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.269+0000 7ff77ebd3740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5621c18acbd0 con 0x5621c189b1b0 2024-04-26T01:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.269+0000 7ff77ebd3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5621c17ba310 con 0x5621c18a9fa0 2024-04-26T01:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.269+0000 7ff77ebd3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5621c18acd50 con 0x5621c189a890 2024-04-26T01:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.269+0000 7ff77ebd3740 10 client.0 _ll_register_callbacks cb 0x5621c1788cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.269+0000 7ff779400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189a890 0x5621c18a6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff779400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189a890 0x5621c18a6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:52892/0 (socket says 172.21.15.6:52892) 2024-04-26T01:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff779400640 1 -- 172.21.15.6:0/1675571142 learned_addr learned my addr 172.21.15.6:0/1675571142 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff778a00640 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189b1b0 0x5621c18a73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff779e00640 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c18a9fa0 0x5621c18a7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff779400640 1 -- 172.21.15.6:0/1675571142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c18a9fa0 msgr2=0x5621c18a7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff779400640 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c18a9fa0 0x5621c18a7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff779400640 1 -- 172.21.15.6:0/1675571142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189b1b0 msgr2=0x5621c18a73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff779400640 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189b1b0 0x5621c18a73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff779400640 1 -- 172.21.15.6:0/1675571142 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621c15df620 con 0x5621c189a890 2024-04-26T01:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff779e00640 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c18a9fa0 0x5621c18a7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff778a00640 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189b1b0 0x5621c18a73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff779400640 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189a890 0x5621c18a6eb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff768098f80 tx=0x7ff76801f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d825d15b5513c527 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff75aa00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff75aa00640 1 -- 172.21.15.6:0/1675571142 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7680e0030 con 0x5621c189a890 2024-04-26T01:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1675571142 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7680dc8a0 con 0x5621c189a890 2024-04-26T01:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.270+0000 7ff75aa00640 1 -- 172.21.15.6:0/1675571142 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7ff768099190 con 0x5621c189a890 2024-04-26T01:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.271+0000 7ff75aa00640 1 -- 172.21.15.6:0/1675571142 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7ff768098dc0 con 0x5621c189a890 2024-04-26T01:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.271+0000 7ff77ebd3740 10 client.26116 fetch_fsmap learned FSMap version 41 2024-04-26T01:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.271+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1675571142 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5621c18acbd0 con 0x5621c189a890 2024-04-26T01:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.271+0000 7ff75aa00640 1 -- 172.21.15.6:0/1675571142 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7680dc8a0 con 0x5621c189a890 2024-04-26T01:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.271+0000 7ff75aa00640 1 -- 172.21.15.6:0/1675571142 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7ff7680ce4a0 con 0x5621c189a890 2024-04-26T01:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.271+0000 7ff77ebd3740 10 client.26116 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.271+0000 7ff77ebd3740 10 client.26116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.271+0000 7ff77ebd3740 20 client.26116 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.271+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621c18b9fb0 0x5621c18bc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.271+0000 7ff77ebd3740 4 client.26116 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.271+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1675571142 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621c18148b0 con 0x5621c18b9fb0 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.272+0000 7ff778a00640 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621c18b9fb0 0x5621c18bc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.272+0000 7ff778a00640 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621c18b9fb0 0x5621c18bc3a0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49d4efc44e3fecfc server_cookie=373be5649db37881 in_seq=0 out_seq=0 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.272+0000 7ff75aa00640 10 client.26116 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff75aa00640 1 -- 172.21.15.6:0/1675571142 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621c18148b0 con 0x5621c18b9fb0 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff75aa00640 10 client.26116 handle_command_reply: tid=0 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff77ebd3740 1 client.26116 shutdown 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1675571142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189a890 msgr2=0x5621c18a6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189a890 0x5621c18a6eb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff768098f80 tx=0x7ff76801f000 comp rx=0 tx=0).stop 2024-04-26T01:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1675571142 shutdown_connections 2024-04-26T01:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621c18b9fb0 0x5621c18bc3a0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189b1b0 0x5621c18a73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c18a9fa0 0x5621c18a7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff75aa00640 0 client.26116 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1675571142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189a890 0x5621c18a6eb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1675571142 >> 172.21.15.6:0/1675571142 conn(0x5621c1896ec0 msgr2=0x5621c189cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1675571142 shutdown_connections 2024-04-26T01:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.273+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1675571142 wait complete. 2024-04-26T01:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.274+0000 7ff77ebd3740 20 client.26116 trim_cache size 0 max 16384 2024-04-26T01:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff77ebd3740 1 Processor -- start 2024-04-26T01:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff77ebd3740 1 -- start start 2024-04-26T01:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff77ebd3740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189a890 0x5621c18b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff77ebd3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189b1b0 0x5621c1895ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff77ebd3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c18a9fa0 0x5621c18c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff77ebd3740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741ae350 con 0x5621c189a890 2024-04-26T01:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff77ebd3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741abbc0 con 0x5621c18a9fa0 2024-04-26T01:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff77ebd3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741a9a30 con 0x5621c189b1b0 2024-04-26T01:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff779e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c18a9fa0 0x5621c18c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff778a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189b1b0 0x5621c1895ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff779400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189a890 0x5621c18b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff779e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c18a9fa0 0x5621c18c1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:57504/0 (socket says 172.21.15.6:57504) 2024-04-26T01:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff779e00640 1 -- 172.21.15.6:0/119306783 learned_addr learned my addr 172.21.15.6:0/119306783 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.291+0000 7ff778a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189b1b0 0x5621c1895ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:52904/0 (socket says 172.21.15.6:52904) 2024-04-26T01:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.292+0000 7ff778a00640 1 -- 172.21.15.6:0/119306783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c18a9fa0 msgr2=0x5621c18c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.292+0000 7ff778a00640 1 --2- 172.21.15.6:0/119306783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c18a9fa0 0x5621c18c1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.292+0000 7ff778a00640 1 -- 172.21.15.6:0/119306783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189a890 msgr2=0x5621c18b5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.292+0000 7ff778a00640 1 --2- 172.21.15.6:0/119306783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189a890 0x5621c18b5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.292+0000 7ff778a00640 1 -- 172.21.15.6:0/119306783 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621c1898080 con 0x5621c189b1b0 2024-04-26T01:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.292+0000 7ff778a00640 1 --2- 172.21.15.6:0/119306783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189b1b0 0x5621c1895ea0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff76002a910 tx=0x7ff760005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9958774c18a0c8ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.292+0000 7ff765c00640 1 -- 172.21.15.6:0/119306783 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760035040 con 0x5621c189b1b0 2024-04-26T01:53:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.292+0000 7ff765c00640 1 -- 172.21.15.6:0/119306783 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff760045830 con 0x5621c189b1b0 2024-04-26T01:53:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.292+0000 7ff765c00640 1 -- 172.21.15.6:0/119306783 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760005400 con 0x5621c189b1b0 2024-04-26T01:53:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.293+0000 7ff77ebd3740 1 -- 172.21.15.6:0/119306783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189b1b0 msgr2=0x5621c1895ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.293+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/119306783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189b1b0 0x5621c1895ea0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff76002a910 tx=0x7ff760005a20 comp rx=0 tx=0).stop 2024-04-26T01:53:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.293+0000 7ff77ebd3740 1 -- 172.21.15.6:0/119306783 shutdown_connections 2024-04-26T01:53:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.293+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/119306783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c18a9fa0 0x5621c18c1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.293+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/119306783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c189b1b0 0x5621c1895ea0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.293+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/119306783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189a890 0x5621c18b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.293+0000 7ff77ebd3740 1 -- 172.21.15.6:0/119306783 >> 172.21.15.6:0/119306783 conn(0x5621c1896ec0 msgr2=0x5621c1895790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.293+0000 7ff77ebd3740 1 -- 172.21.15.6:0/119306783 shutdown_connections 2024-04-26T01:53:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.293+0000 7ff77ebd3740 1 -- 172.21.15.6:0/119306783 wait complete. 2024-04-26T01:53:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.293+0000 7ff77ebd3740 1 -- start start 2024-04-26T01:53:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff77ebd3740 1 Processor -- start 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff77ebd3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189a890 0x5621c18aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff77ebd3740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189b1b0 0x5621c18af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff77ebd3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c18a9fa0 0x5621c18afa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff77ebd3740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741aa080 con 0x5621c189b1b0 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff77ebd3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741ab4f0 con 0x5621c189a890 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff77ebd3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741aa750 con 0x5621c18a9fa0 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff77ebd3740 10 client.0 _ll_register_callbacks cb 0x7ff7680d49e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff779400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189a890 0x5621c18aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff779400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189a890 0x5621c18aefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:57520/0 (socket says 172.21.15.6:57520) 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff779400640 1 -- 172.21.15.6:0/1497992841 learned_addr learned my addr 172.21.15.6:0/1497992841 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff779e00640 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c18a9fa0 0x5621c18afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.294+0000 7ff778a00640 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189b1b0 0x5621c18af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.295+0000 7ff779400640 1 -- 172.21.15.6:0/1497992841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c18a9fa0 msgr2=0x5621c18afa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.295+0000 7ff779400640 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c18a9fa0 0x5621c18afa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.295+0000 7ff779400640 1 -- 172.21.15.6:0/1497992841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189b1b0 msgr2=0x5621c18af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.295+0000 7ff779400640 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189b1b0 0x5621c18af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.295+0000 7ff779400640 1 -- 172.21.15.6:0/1497992841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621c191e520 con 0x5621c189a890 2024-04-26T01:53:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.295+0000 7ff779400640 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189a890 0x5621c18aefb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff76809cab0 tx=0x7ff7680d9640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a6f0cfbf7bf119c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.295+0000 7ff75aa00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:53:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.295+0000 7ff75aa00640 1 -- 172.21.15.6:0/1497992841 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7680e0070 con 0x5621c189a890 2024-04-26T01:53:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.295+0000 7ff75aa00640 1 -- 172.21.15.6:0/1497992841 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7ff7680d4400 con 0x5621c189a890 2024-04-26T01:53:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.295+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1497992841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff760045830 con 0x5621c189a890 2024-04-26T01:53:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.296+0000 7ff75aa00640 1 -- 172.21.15.6:0/1497992841 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7ff76809c480 con 0x5621c189a890 2024-04-26T01:53:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.296+0000 7ff77ebd3740 10 client.26075 fetch_fsmap learned FSMap version 41 2024-04-26T01:53:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.296+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1497992841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5621c191f830 con 0x5621c189a890 2024-04-26T01:53:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.296+0000 7ff75aa00640 1 -- 172.21.15.6:0/1497992841 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff760045830 con 0x5621c189a890 2024-04-26T01:53:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.296+0000 7ff75aa00640 1 -- 172.21.15.6:0/1497992841 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7ff76801abd0 con 0x5621c189a890 2024-04-26T01:53:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.296+0000 7ff77ebd3740 10 client.26075 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:53:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.296+0000 7ff77ebd3740 10 client.26075 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.296+0000 7ff77ebd3740 20 client.26075 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.296+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621c191fd60 0x5621c1922150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.296+0000 7ff77ebd3740 4 client.26075 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.296+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1497992841 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5621c180e320 con 0x5621c191fd60 2024-04-26T01:53:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.296+0000 7ff778a00640 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621c191fd60 0x5621c1922150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.297+0000 7ff778a00640 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621c191fd60 0x5621c1922150 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8227797152903eb6 server_cookie=d940d1fff8362356 in_seq=0 out_seq=0 2024-04-26T01:53:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.297+0000 7ff75aa00640 10 client.26075 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff75aa00640 1 -- 172.21.15.6:0/1497992841 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5621c180e320 con 0x5621c191fd60 2024-04-26T01:53:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff75aa00640 10 client.26075 handle_command_reply: tid=0 2024-04-26T01:53:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff77ebd3740 1 client.26075 shutdown 2024-04-26T01:53:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1497992841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189a890 msgr2=0x5621c18aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189a890 0x5621c18aefb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff76809cab0 tx=0x7ff7680d9640 comp rx=0 tx=0).stop 2024-04-26T01:53:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1497992841 shutdown_connections 2024-04-26T01:53:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621c191fd60 0x5621c1922150 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621c189b1b0 0x5621c18af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621c18a9fa0 0x5621c18afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff77ebd3740 1 --2- 172.21.15.6:0/1497992841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621c189a890 0x5621c18aefb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff75aa00640 0 client.26075 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1497992841 >> 172.21.15.6:0/1497992841 conn(0x5621c1896ec0 msgr2=0x5621c1896020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1497992841 shutdown_connections 2024-04-26T01:53:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.341+0000 7ff77ebd3740 1 -- 172.21.15.6:0/1497992841 wait complete. 2024-04-26T01:53:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.342+0000 7ff77ebd3740 20 client.26075 trim_cache size 0 max 16384 2024-04-26T01:53:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:53:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:53:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:53:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:53:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.342+0000 7ff77a800640 1 -- 172.21.15.6:0/2016007323 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff75c0a3d80 msgr2=0x7ff75c0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.342+0000 7ff77a800640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff75c0a3d80 0x7ff75c0a6240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff76c009fd0 tx=0x7ff76c009500 comp rx=0 tx=0).stop 2024-04-26T01:53:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.342+0000 7ff77a800640 1 -- 172.21.15.6:0/2016007323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff774101fc0 msgr2=0x7ff7741986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.342+0000 7ff77a800640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff774101fc0 0x7ff7741986a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff76800e9f0 tx=0x7ff76800eeb0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.343+0000 7ff77a800640 1 -- 172.21.15.6:0/2016007323 shutdown_connections 2024-04-26T01:53:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.343+0000 7ff77a800640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff75c0a3d80 0x7ff75c0a6240 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.343+0000 7ff77a800640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff774107090 0x7ff774198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.343+0000 7ff77a800640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff77419cf10 0x7ff77419d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.343+0000 7ff77a800640 1 --2- 172.21.15.6:0/2016007323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff774101fc0 0x7ff7741986a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:28.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.343+0000 7ff77a800640 1 -- 172.21.15.6:0/2016007323 >> 172.21.15.6:0/2016007323 conn(0x7ff7740fd880 msgr2=0x7ff7740fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:28.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.343+0000 7ff77a800640 1 -- 172.21.15.6:0/2016007323 shutdown_connections 2024-04-26T01:53:28.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:28.343+0000 7ff77a800640 1 -- 172.21.15.6:0/2016007323 wait complete. 2024-04-26T01:53:28.596 INFO:tasks.workunit.client.0.smithi006.stdout: 1 264872 3.98 MB/sec execute 232 sec latency 21.123 ms 2024-04-26T01:53:28.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:28 smithi029 ceph-mon[32775]: pgmap v237: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 201 op/s 2024-04-26T01:53:28.691 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:28 smithi188 ceph-mon[32751]: pgmap v237: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 201 op/s 2024-04-26T01:53:28.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:28 smithi006 ceph-mon[28419]: pgmap v237: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 201 op/s 2024-04-26T01:53:29.596 INFO:tasks.workunit.client.0.smithi006.stdout: 1 266255 4.02 MB/sec execute 233 sec latency 29.944 ms 2024-04-26T01:53:30.596 INFO:tasks.workunit.client.0.smithi006.stdout: 1 266861 4.00 MB/sec execute 234 sec latency 23.647 ms 2024-04-26T01:53:30.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:30 smithi029 ceph-mon[32775]: pgmap v238: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1015 KiB/s wr, 116 op/s 2024-04-26T01:53:30.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:30 smithi006 ceph-mon[28419]: pgmap v238: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1015 KiB/s wr, 116 op/s 2024-04-26T01:53:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:30 smithi188 ceph-mon[32751]: pgmap v238: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1015 KiB/s wr, 116 op/s 2024-04-26T01:53:31.596 INFO:tasks.workunit.client.0.smithi006.stdout: 1 267673 4.01 MB/sec execute 235 sec latency 30.701 ms 2024-04-26T01:53:32.596 INFO:tasks.workunit.client.0.smithi006.stdout: 1 268619 4.00 MB/sec execute 236 sec latency 22.113 ms 2024-04-26T01:53:32.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:32 smithi029 ceph-mon[32775]: pgmap v239: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-04-26T01:53:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:32 smithi006 ceph-mon[28419]: pgmap v239: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-04-26T01:53:32.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:32 smithi188 ceph-mon[32751]: pgmap v239: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-04-26T01:53:32.984 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.9 2024-04-26T01:53:32.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:53:33.471 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:53:33.596 INFO:tasks.workunit.client.0.smithi006.stdout: 1 269882 4.03 MB/sec execute 237 sec latency 23.331 ms 2024-04-26T01:53:34.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:34 smithi006 ceph-mon[28419]: pgmap v240: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 146 op/s 2024-04-26T01:53:34.597 INFO:tasks.workunit.client.0.smithi006.stdout: 1 270581 4.02 MB/sec execute 238 sec latency 23.694 ms 2024-04-26T01:53:34.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:34 smithi029 ceph-mon[32775]: pgmap v240: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 146 op/s 2024-04-26T01:53:34.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:34 smithi188 ceph-mon[32751]: pgmap v240: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 146 op/s 2024-04-26T01:53:35.597 INFO:tasks.workunit.client.0.smithi006.stdout: 1 271316 4.02 MB/sec execute 239 sec latency 29.066 ms 2024-04-26T01:53:36.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.192+0000 7f9e77800700 1 -- 172.21.15.6:0/730957443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e70102a90 msgr2=0x7f9e70102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:36.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.192+0000 7f9e77800700 1 --2- 172.21.15.6:0/730957443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e70102a90 0x7f9e70102f10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c009990 tx=0x7f9e5c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:53:36.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.193+0000 7f9e77800700 1 -- 172.21.15.6:0/730957443 shutdown_connections 2024-04-26T01:53:36.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.193+0000 7f9e77800700 1 --2- 172.21.15.6:0/730957443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e70103450 0x7f9e7010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:36.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.193+0000 7f9e77800700 1 --2- 172.21.15.6:0/730957443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e70108a50 0x7f9e70108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:36.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.193+0000 7f9e77800700 1 --2- 172.21.15.6:0/730957443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e70102a90 0x7f9e70102f10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:36.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.193+0000 7f9e77800700 1 -- 172.21.15.6:0/730957443 >> 172.21.15.6:0/730957443 conn(0x7f9e70079ba0 msgr2=0x7f9e70079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:36.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.193+0000 7f9e77800700 1 -- 172.21.15.6:0/730957443 shutdown_connections 2024-04-26T01:53:36.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.193+0000 7f9e77800700 1 -- 172.21.15.6:0/730957443 wait complete. 2024-04-26T01:53:36.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.194+0000 7f9e77800700 1 Processor -- start 2024-04-26T01:53:36.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.194+0000 7f9e77800700 1 -- start start 2024-04-26T01:53:36.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.194+0000 7f9e77800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e70102a90 0x7f9e70074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:36.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.194+0000 7f9e77800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e70103450 0x7f9e70075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.194+0000 7f9e77800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e70108a50 0x7f9e700757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.194+0000 7f9e77800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9e70112c50 con 0x7f9e70103450 2024-04-26T01:53:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.194+0000 7f9e77800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9e70112ad0 con 0x7f9e70108a50 2024-04-26T01:53:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.194+0000 7f9e77800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9e70112dd0 con 0x7f9e70102a90 2024-04-26T01:53:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.195+0000 7f9e74e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e70102a90 0x7f9e70074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.195+0000 7f9e6fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e70103450 0x7f9e70075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.195+0000 7f9e74e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e70102a90 0x7f9e70074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:52938/0 (socket says 172.21.15.6:52938) 2024-04-26T01:53:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.195+0000 7f9e74e00700 1 -- 172.21.15.6:0/1670888346 learned_addr learned my addr 172.21.15.6:0/1670888346 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.195+0000 7f9e75800700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e70108a50 0x7f9e700757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.195+0000 7f9e74e00700 1 -- 172.21.15.6:0/1670888346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e70108a50 msgr2=0x7f9e700757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.195+0000 7f9e74e00700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e70108a50 0x7f9e700757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.195+0000 7f9e74e00700 1 -- 172.21.15.6:0/1670888346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e70103450 msgr2=0x7f9e70075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.195+0000 7f9e74e00700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e70103450 0x7f9e70075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.195+0000 7f9e74e00700 1 -- 172.21.15.6:0/1670888346 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e701abb70 con 0x7f9e70102a90 2024-04-26T01:53:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.195+0000 7f9e74e00700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e70102a90 0x7f9e70074d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400d950 tx=0x7f9e6400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.196+0000 7f9e6d600700 1 -- 172.21.15.6:0/1670888346 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e64016070 con 0x7f9e70102a90 2024-04-26T01:53:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.196+0000 7f9e6d600700 1 -- 172.21.15.6:0/1670888346 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e64009df0 con 0x7f9e70102a90 2024-04-26T01:53:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.196+0000 7f9e6d600700 1 -- 172.21.15.6:0/1670888346 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6400f3c0 con 0x7f9e70102a90 2024-04-26T01:53:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.196+0000 7f9e77800700 1 -- 172.21.15.6:0/1670888346 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e701abe00 con 0x7f9e70102a90 2024-04-26T01:53:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.196+0000 7f9e77800700 1 -- 172.21.15.6:0/1670888346 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e701ac250 con 0x7f9e70102a90 2024-04-26T01:53:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.198+0000 7f9e77800700 1 -- 172.21.15.6:0/1670888346 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e70104180 con 0x7f9e70102a90 2024-04-26T01:53:36.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.200+0000 7f9e6d600700 1 -- 172.21.15.6:0/1670888346 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f9e64019070 con 0x7f9e70102a90 2024-04-26T01:53:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.201+0000 7f9e6d600700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9e580a3e30 0x7f9e580a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.202+0000 7f9e6d600700 1 -- 172.21.15.6:0/1670888346 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f9e640d0320 con 0x7f9e70102a90 2024-04-26T01:53:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.202+0000 7f9e6fe00700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9e580a3e30 0x7f9e580a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:36.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.203+0000 7f9e6fe00700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9e580a3e30 0x7f9e580a62f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9e70076590 tx=0x7f9e5c01a040 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:36.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.208+0000 7f9e6d600700 1 -- 172.21.15.6:0/1670888346 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e6409c910 con 0x7f9e70102a90 2024-04-26T01:53:36.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.546+0000 7f9e77800700 1 -- 172.21.15.6:0/1670888346 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e7002e0a0 con 0x7f9e70102a90 2024-04-26T01:53:36.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.547+0000 7f9e6d600700 1 -- 172.21.15.6:0/1670888346 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11853 (secure 0 0 0) 0x7f9e6409c210 con 0x7f9e70102a90 2024-04-26T01:53:36.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:53:36.546 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":41,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:53:03.909439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:53:36.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.550+0000 7f9e77800700 1 -- 172.21.15.6:0/1670888346 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9e580a3e30 msgr2=0x7f9e580a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:36.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.551+0000 7f9e77800700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9e580a3e30 0x7f9e580a62f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9e70076590 tx=0x7f9e5c01a040 comp rx=0 tx=0).stop 2024-04-26T01:53:36.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.551+0000 7f9e77800700 1 -- 172.21.15.6:0/1670888346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e70102a90 msgr2=0x7f9e70074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:36.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.551+0000 7f9e77800700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e70102a90 0x7f9e70074d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400d950 tx=0x7f9e6400dc60 comp rx=0 tx=0).stop 2024-04-26T01:53:36.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.551+0000 7f9e77800700 1 -- 172.21.15.6:0/1670888346 shutdown_connections 2024-04-26T01:53:36.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.551+0000 7f9e77800700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9e580a3e30 0x7f9e580a62f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:36.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.551+0000 7f9e77800700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e70108a50 0x7f9e700757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:36.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.551+0000 7f9e77800700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e70102a90 0x7f9e70074d40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:36.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.551+0000 7f9e77800700 1 --2- 172.21.15.6:0/1670888346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e70103450 0x7f9e70075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:36.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.551+0000 7f9e77800700 1 -- 172.21.15.6:0/1670888346 >> 172.21.15.6:0/1670888346 conn(0x7f9e70079ba0 msgr2=0x7f9e701019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:36.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.551+0000 7f9e77800700 1 -- 172.21.15.6:0/1670888346 shutdown_connections 2024-04-26T01:53:36.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:36.551+0000 7f9e77800700 1 -- 172.21.15.6:0/1670888346 wait complete. 2024-04-26T01:53:36.551 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 41 2024-04-26T01:53:36.597 INFO:tasks.workunit.client.0.smithi006.stdout: 1 272531 4.03 MB/sec execute 240 sec latency 54.888 ms 2024-04-26T01:53:36.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:36 smithi029 ceph-mon[32775]: pgmap v241: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 146 op/s 2024-04-26T01:53:36.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:36 smithi006 ceph-mon[28419]: pgmap v241: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 146 op/s 2024-04-26T01:53:36.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:36 smithi188 ceph-mon[32751]: pgmap v241: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 146 op/s 2024-04-26T01:53:37.367 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 82.9 --await-for 42.0' 2024-04-26T01:53:37.367 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 82.9 --await-for 42.0 2024-04-26T01:53:37.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:37 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1670888346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:53:37.597 INFO:tasks.workunit.client.0.smithi006.stdout: 1 273550 4.05 MB/sec execute 241 sec latency 49.576 ms 2024-04-26T01:53:37.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:37 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1670888346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:53:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:53:37.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:37 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1670888346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:53:38.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:38 smithi006 ceph-mon[28419]: pgmap v242: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 195 op/s 2024-04-26T01:53:38.597 INFO:tasks.workunit.client.0.smithi006.stdout: 1 274760 4.05 MB/sec execute 242 sec latency 19.564 ms 2024-04-26T01:53:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:38 smithi188 ceph-mon[32751]: pgmap v242: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 195 op/s 2024-04-26T01:53:38.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:38 smithi029 ceph-mon[32775]: pgmap v242: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 195 op/s 2024-04-26T01:53:39.597 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276423 4.07 MB/sec execute 243 sec latency 19.095 ms 2024-04-26T01:53:40.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.008+0000 7f3695200700 1 -- 172.21.15.6:0/1932149062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f369010e3b0 msgr2=0x7f3690115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:40.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.008+0000 7f3695200700 1 --2- 172.21.15.6:0/1932149062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f369010e3b0 0x7f3690115f80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3688009f90 tx=0x7f368800b330 comp rx=0 tx=0).stop 2024-04-26T01:53:40.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.008+0000 7f3695200700 1 -- 172.21.15.6:0/1932149062 shutdown_connections 2024-04-26T01:53:40.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.008+0000 7f3695200700 1 --2- 172.21.15.6:0/1932149062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f369010e3b0 0x7f3690115f80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.008+0000 7f3695200700 1 --2- 172.21.15.6:0/1932149062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36901092d0 0x7f36901096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.008+0000 7f3695200700 1 --2- 172.21.15.6:0/1932149062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3690109c80 0x7f369010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.008+0000 7f3695200700 1 -- 172.21.15.6:0/1932149062 >> 172.21.15.6:0/1932149062 conn(0x7f3690076210 msgr2=0x7f3690107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:40.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.009+0000 7f3695200700 1 -- 172.21.15.6:0/1932149062 shutdown_connections 2024-04-26T01:53:40.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.009+0000 7f3695200700 1 -- 172.21.15.6:0/1932149062 wait complete. 2024-04-26T01:53:40.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.010+0000 7f3695200700 1 Processor -- start 2024-04-26T01:53:40.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.010+0000 7f3695200700 1 -- start start 2024-04-26T01:53:40.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.011+0000 7f3695200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36901092d0 0x7f369019f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.011+0000 7f3695200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3690109c80 0x7f369019fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.011+0000 7f3695200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f369010e3b0 0x7f36901a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.011+0000 7f3695200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f36901192a0 con 0x7f369010e3b0 2024-04-26T01:53:40.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.011+0000 7f3695200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3690119120 con 0x7f3690109c80 2024-04-26T01:53:40.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.011+0000 7f3695200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3690119420 con 0x7f36901092d0 2024-04-26T01:53:40.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.011+0000 7f368de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36901092d0 0x7f369019f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.011+0000 7f368de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36901092d0 0x7f369019f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37238/0 (socket says 172.21.15.6:37238) 2024-04-26T01:53:40.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.011+0000 7f368e800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f369010e3b0 0x7f36901a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.011+0000 7f368de00700 1 -- 172.21.15.6:0/2549624444 learned_addr learned my addr 172.21.15.6:0/2549624444 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:40.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.011+0000 7f368d400700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3690109c80 0x7f369019fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.012+0000 7f368d400700 1 -- 172.21.15.6:0/2549624444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36901092d0 msgr2=0x7f369019f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.012+0000 7f368d400700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36901092d0 0x7f369019f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.012+0000 7f368d400700 1 -- 172.21.15.6:0/2549624444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f369010e3b0 msgr2=0x7f36901a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.012+0000 7f368d400700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f369010e3b0 0x7f36901a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.012+0000 7f368d400700 1 -- 172.21.15.6:0/2549624444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36901a45f0 con 0x7f3690109c80 2024-04-26T01:53:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.012+0000 7f368de00700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36901092d0 0x7f369019f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.012+0000 7f368e800700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f369010e3b0 0x7f36901a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.012+0000 7f368d400700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3690109c80 0x7f369019fb70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f367c005fd0 tx=0x7f367c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.012+0000 7f3686a00700 1 -- 172.21.15.6:0/2549624444 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c00cbd0 con 0x7f3690109c80 2024-04-26T01:53:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.012+0000 7f3686a00700 1 -- 172.21.15.6:0/2549624444 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f367c0044d0 con 0x7f3690109c80 2024-04-26T01:53:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.012+0000 7f3686a00700 1 -- 172.21.15.6:0/2549624444 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c010430 con 0x7f3690109c80 2024-04-26T01:53:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.012+0000 7f3695200700 1 -- 172.21.15.6:0/2549624444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36901a48e0 con 0x7f3690109c80 2024-04-26T01:53:40.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.013+0000 7f3695200700 1 -- 172.21.15.6:0/2549624444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36901ac160 con 0x7f3690109c80 2024-04-26T01:53:40.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.016+0000 7f3686a00700 1 -- 172.21.15.6:0/2549624444 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f367c003eb0 con 0x7f3690109c80 2024-04-26T01:53:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.019+0000 7f3686a00700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36740a3ed0 0x7f36740a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.019+0000 7f369fab1b80 1 Processor -- start 2024-04-26T01:53:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.019+0000 7f3686a00700 1 -- 172.21.15.6:0/2549624444 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f367c0d09a0 con 0x7f3690109c80 2024-04-26T01:53:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.019+0000 7f368de00700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36740a3ed0 0x7f36740a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.019+0000 7f369fab1b80 1 -- start start 2024-04-26T01:53:40.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.020+0000 7f369fab1b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f81dd0 0x555693f821b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.020+0000 7f369fab1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f826f0 0x555693f8d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.020+0000 7f369fab1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 0x555693f8f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.020+0000 7f369fab1b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555693ed5020 con 0x555693f81dd0 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.020+0000 7f369fab1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555693e7b8c0 con 0x555693f8d560 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.020+0000 7f369fab1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555693ed0fe0 con 0x555693f826f0 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.020+0000 7f368de00700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36740a3ed0 0x7f36740a6390 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3680011550 tx=0x7f3680009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.020+0000 7f368d400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f826f0 0x555693f8d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.020+0000 7f368de00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f81dd0 0x555693f821b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.020+0000 7f368d400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f826f0 0x555693f8d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37248/0 (socket says 172.21.15.6:37248) 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.020+0000 7f368d400700 1 -- 172.21.15.6:0/3233156636 learned_addr learned my addr 172.21.15.6:0/3233156636 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.020+0000 7f368e800700 1 --2- 172.21.15.6:0/3233156636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 0x555693f8f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.021+0000 7f368d400700 1 -- 172.21.15.6:0/3233156636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 msgr2=0x555693f8f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.021+0000 7f368d400700 1 --2- 172.21.15.6:0/3233156636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 0x555693f8f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.021+0000 7f368d400700 1 -- 172.21.15.6:0/3233156636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f81dd0 msgr2=0x555693f821b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:40.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.021+0000 7f368d400700 1 --2- 172.21.15.6:0/3233156636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f81dd0 0x555693f821b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.021+0000 7f368d400700 1 -- 172.21.15.6:0/3233156636 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555693e71810 con 0x555693f826f0 2024-04-26T01:53:40.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.021+0000 7f368de00700 1 --2- 172.21.15.6:0/3233156636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f81dd0 0x555693f821b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:40.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.021+0000 7f368d400700 1 --2- 172.21.15.6:0/3233156636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f826f0 0x555693f8d020 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f367c00c5b0 tx=0x7f367c098530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20e52c19c7d070e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.021+0000 7f367b400700 1 -- 172.21.15.6:0/3233156636 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c09c1f0 con 0x555693f826f0 2024-04-26T01:53:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.021+0000 7f367b400700 1 -- 172.21.15.6:0/3233156636 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f367c09cf90 con 0x555693f826f0 2024-04-26T01:53:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.021+0000 7f367b400700 1 -- 172.21.15.6:0/3233156636 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c0041a0 con 0x555693f826f0 2024-04-26T01:53:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.022+0000 7f369fab1b80 1 -- 172.21.15.6:0/3233156636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f826f0 msgr2=0x555693f8d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.022+0000 7f369fab1b80 1 --2- 172.21.15.6:0/3233156636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f826f0 0x555693f8d020 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f367c00c5b0 tx=0x7f367c098530 comp rx=0 tx=0).stop 2024-04-26T01:53:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.022+0000 7f369fab1b80 1 -- 172.21.15.6:0/3233156636 shutdown_connections 2024-04-26T01:53:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.022+0000 7f369fab1b80 1 --2- 172.21.15.6:0/3233156636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 0x555693f8f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.022+0000 7f369fab1b80 1 --2- 172.21.15.6:0/3233156636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f81dd0 0x555693f821b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.022+0000 7f369fab1b80 1 --2- 172.21.15.6:0/3233156636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f826f0 0x555693f8d020 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.022+0000 7f369fab1b80 1 -- 172.21.15.6:0/3233156636 >> 172.21.15.6:0/3233156636 conn(0x555693eb10e0 msgr2=0x555693f7e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.023+0000 7f369fab1b80 1 -- 172.21.15.6:0/3233156636 shutdown_connections 2024-04-26T01:53:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.023+0000 7f369fab1b80 1 -- 172.21.15.6:0/3233156636 wait complete. 2024-04-26T01:53:40.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.023+0000 7f369fab1b80 1 -- start start 2024-04-26T01:53:40.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.024+0000 7f369fab1b80 1 Processor -- start 2024-04-26T01:53:40.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.024+0000 7f369fab1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f8ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.024+0000 7f369fab1b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f826f0 0x555693f89ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.025+0000 7f369fab1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 0x555693f8a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.025+0000 7f369fab1b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555693f90050 con 0x555693f826f0 2024-04-26T01:53:40.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.025+0000 7f369fab1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555693e8e9f0 con 0x555693f8d560 2024-04-26T01:53:40.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.025+0000 7f369fab1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555693f901d0 con 0x555693f81dd0 2024-04-26T01:53:40.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.025+0000 7f369fab1b80 10 client.0 _ll_register_callbacks cb 0x555693e6d8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:40.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.025+0000 7f368de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f8ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.025+0000 7f368de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f8ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37262/0 (socket says 172.21.15.6:37262) 2024-04-26T01:53:40.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.025+0000 7f368de00700 1 -- 172.21.15.6:0/2449375024 learned_addr learned my addr 172.21.15.6:0/2449375024 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:40.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.025+0000 7f368d400700 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f826f0 0x555693f89ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.025+0000 7f368e800700 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 0x555693f8a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.026+0000 7f368de00700 1 -- 172.21.15.6:0/2449375024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 msgr2=0x555693f8a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:40.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.026+0000 7f368de00700 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 0x555693f8a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.026+0000 7f368de00700 1 -- 172.21.15.6:0/2449375024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f826f0 msgr2=0x555693f89ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.026+0000 7f368de00700 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f826f0 0x555693f89ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.026+0000 7f368de00700 1 -- 172.21.15.6:0/2449375024 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555693e22a60 con 0x555693f81dd0 2024-04-26T01:53:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.026+0000 7f368d400700 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f826f0 0x555693f89ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.026+0000 7f368de00700 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f8ccf0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f368001ed10 tx=0x7f36800047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6d21e3cfe395110 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.026+0000 7f366fe00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:53:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.027+0000 7f366fe00700 1 -- 172.21.15.6:0/2449375024 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368000eb20 con 0x555693f81dd0 2024-04-26T01:53:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.027+0000 7f366fe00700 1 -- 172.21.15.6:0/2449375024 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f368000ecc0 con 0x555693f81dd0 2024-04-26T01:53:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.027+0000 7f369fab1b80 1 -- 172.21.15.6:0/2449375024 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f367c09cf90 con 0x555693f81dd0 2024-04-26T01:53:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.027+0000 7f366fe00700 1 -- 172.21.15.6:0/2449375024 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f3680004830 con 0x555693f81dd0 2024-04-26T01:53:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.027+0000 7f369fab1b80 10 client.26140 fetch_fsmap learned FSMap version 41 2024-04-26T01:53:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.027+0000 7f369fab1b80 1 -- 172.21.15.6:0/2449375024 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x555693e6bc80 con 0x555693f81dd0 2024-04-26T01:53:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.027+0000 7f366fe00700 1 -- 172.21.15.6:0/2449375024 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f367c09cf90 con 0x555693f81dd0 2024-04-26T01:53:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.027+0000 7f366fe00700 1 -- 172.21.15.6:0/2449375024 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f3680025070 con 0x555693f81dd0 2024-04-26T01:53:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.027+0000 7f369fab1b80 10 client.26140 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:53:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.027+0000 7f369fab1b80 10 client.26140 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:53:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.028+0000 7f369fab1b80 20 client.26140 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.028+0000 7f369fab1b80 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555693f9cdf0 0x555693f9f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.028+0000 7f369fab1b80 4 client.26140 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.028+0000 7f369fab1b80 1 -- 172.21.15.6:0/2449375024 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555693ee2810 con 0x555693f9cdf0 2024-04-26T01:53:40.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.028+0000 7f368d400700 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555693f9cdf0 0x555693f9f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.028+0000 7f368d400700 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555693f9cdf0 0x555693f9f220 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=506bea4ef81980ed server_cookie=dac8bfc6f82db56e in_seq=0 out_seq=0 2024-04-26T01:53:40.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.028+0000 7f366fe00700 10 client.26140 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.029+0000 7f366fe00700 1 -- 172.21.15.6:0/2449375024 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555693ee2810 con 0x555693f9cdf0 2024-04-26T01:53:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.029+0000 7f366fe00700 10 client.26140 handle_command_reply: tid=0 2024-04-26T01:53:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f369fab1b80 1 client.26140 shutdown 2024-04-26T01:53:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f369fab1b80 1 -- 172.21.15.6:0/2449375024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 msgr2=0x555693f8ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f369fab1b80 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f8ccf0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f368001ed10 tx=0x7f36800047c0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f369fab1b80 1 -- 172.21.15.6:0/2449375024 shutdown_connections 2024-04-26T01:53:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f369fab1b80 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 0x555693f8a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f369fab1b80 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555693f9cdf0 0x555693f9f220 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f369fab1b80 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f8ccf0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f369fab1b80 1 --2- 172.21.15.6:0/2449375024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f826f0 0x555693f89ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f366fe00700 0 client.26140 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:40.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f369fab1b80 1 -- 172.21.15.6:0/2449375024 >> 172.21.15.6:0/2449375024 conn(0x555693eb10e0 msgr2=0x555693f7ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:40.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f369fab1b80 1 -- 172.21.15.6:0/2449375024 shutdown_connections 2024-04-26T01:53:40.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f369fab1b80 1 -- 172.21.15.6:0/2449375024 wait complete. 2024-04-26T01:53:40.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.030+0000 7f369fab1b80 20 client.26140 trim_cache size 0 max 16384 2024-04-26T01:53:40.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.054+0000 7f369fab1b80 1 Processor -- start 2024-04-26T01:53:40.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.054+0000 7f369fab1b80 1 -- start start 2024-04-26T01:53:40.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.054+0000 7f369fab1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f98f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.054+0000 7f369fab1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f826f0 0x555693f7dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.054+0000 7f369fab1b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f8d560 0x555693f7e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.054+0000 7f369fab1b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f36901b6340 con 0x555693f8d560 2024-04-26T01:53:40.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.054+0000 7f369fab1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36901b3b70 con 0x555693f826f0 2024-04-26T01:53:40.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.054+0000 7f369fab1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f36901b19e0 con 0x555693f81dd0 2024-04-26T01:53:40.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368d400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f826f0 0x555693f7dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f98f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368d400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f826f0 0x555693f7dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47606/0 (socket says 172.21.15.6:47606) 2024-04-26T01:53:40.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f98f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37276/0 (socket says 172.21.15.6:37276) 2024-04-26T01:53:40.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368d400700 1 -- 172.21.15.6:0/3976940891 learned_addr learned my addr 172.21.15.6:0/3976940891 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:40.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368e800700 1 --2- 172.21.15.6:0/3976940891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f8d560 0x555693f7e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368de00700 1 -- 172.21.15.6:0/3976940891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f826f0 msgr2=0x555693f7dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368de00700 1 --2- 172.21.15.6:0/3976940891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f826f0 0x555693f7dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368de00700 1 -- 172.21.15.6:0/3976940891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f8d560 msgr2=0x555693f7e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368de00700 1 --2- 172.21.15.6:0/3976940891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f8d560 0x555693f7e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368de00700 1 -- 172.21.15.6:0/3976940891 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555693f83c40 con 0x555693f81dd0 2024-04-26T01:53:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368d400700 1 --2- 172.21.15.6:0/3976940891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f826f0 0x555693f7dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.055+0000 7f368de00700 1 --2- 172.21.15.6:0/3976940891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f98f50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f368001d550 tx=0x7f3680009e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db0073d6096b6efd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.056+0000 7f367aa00700 1 -- 172.21.15.6:0/3976940891 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3680004bd0 con 0x555693f81dd0 2024-04-26T01:53:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.056+0000 7f367aa00700 1 -- 172.21.15.6:0/3976940891 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f368000ee20 con 0x555693f81dd0 2024-04-26T01:53:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.056+0000 7f367aa00700 1 -- 172.21.15.6:0/3976940891 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368001d830 con 0x555693f81dd0 2024-04-26T01:53:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.057+0000 7f369fab1b80 1 -- 172.21.15.6:0/3976940891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 msgr2=0x555693f98f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.057+0000 7f369fab1b80 1 --2- 172.21.15.6:0/3976940891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f98f50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f368001d550 tx=0x7f3680009e40 comp rx=0 tx=0).stop 2024-04-26T01:53:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.057+0000 7f369fab1b80 1 -- 172.21.15.6:0/3976940891 shutdown_connections 2024-04-26T01:53:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.057+0000 7f369fab1b80 1 --2- 172.21.15.6:0/3976940891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f8d560 0x555693f7e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.057+0000 7f369fab1b80 1 --2- 172.21.15.6:0/3976940891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f98f50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.057+0000 7f369fab1b80 1 --2- 172.21.15.6:0/3976940891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f826f0 0x555693f7dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.057+0000 7f369fab1b80 1 -- 172.21.15.6:0/3976940891 >> 172.21.15.6:0/3976940891 conn(0x555693eb10e0 msgr2=0x555693f7d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.057+0000 7f369fab1b80 1 -- 172.21.15.6:0/3976940891 shutdown_connections 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.057+0000 7f369fab1b80 1 -- 172.21.15.6:0/3976940891 wait complete. 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.057+0000 7f369fab1b80 1 -- start start 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f369fab1b80 1 Processor -- start 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f369fab1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f92430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f369fab1b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f826f0 0x555693f92970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f369fab1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 0x555693f92eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f369fab1b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f36901b2030 con 0x555693f826f0 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f369fab1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36901b34a0 con 0x555693f8d560 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f369fab1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f36901b2700 con 0x555693f81dd0 2024-04-26T01:53:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f369fab1b80 10 client.0 _ll_register_callbacks cb 0x555693f8ff00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f368de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f92430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f368de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f92430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37284/0 (socket says 172.21.15.6:37284) 2024-04-26T01:53:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f368de00700 1 -- 172.21.15.6:0/4258457873 learned_addr learned my addr 172.21.15.6:0/4258457873 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f368d400700 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f826f0 0x555693f92970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.058+0000 7f368e800700 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 0x555693f92eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.059+0000 7f368de00700 1 -- 172.21.15.6:0/4258457873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 msgr2=0x555693f92eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.059+0000 7f368de00700 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 0x555693f92eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.059+0000 7f368de00700 1 -- 172.21.15.6:0/4258457873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f826f0 msgr2=0x555693f92970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.059+0000 7f368de00700 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f826f0 0x555693f92970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.059+0000 7f368de00700 1 -- 172.21.15.6:0/4258457873 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555693f82d70 con 0x555693f81dd0 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.059+0000 7f368de00700 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f92430 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3680009f50 tx=0x7f36800056f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5618d4c618103225 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.059+0000 7f366fe00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.059+0000 7f366fe00700 1 -- 172.21.15.6:0/4258457873 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36800049d0 con 0x555693f81dd0 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.060+0000 7f366fe00700 1 -- 172.21.15.6:0/4258457873 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f3680004b70 con 0x555693f81dd0 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.060+0000 7f369fab1b80 1 -- 172.21.15.6:0/4258457873 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f368000ee20 con 0x555693f81dd0 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.060+0000 7f366fe00700 1 -- 172.21.15.6:0/4258457873 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f3680004d70 con 0x555693f81dd0 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.060+0000 7f369fab1b80 10 client.26146 fetch_fsmap learned FSMap version 41 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.060+0000 7f369fab1b80 1 -- 172.21.15.6:0/4258457873 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x555694004420 con 0x555693f81dd0 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.060+0000 7f366fe00700 1 -- 172.21.15.6:0/4258457873 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f368000ee20 con 0x555693f81dd0 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.060+0000 7f366fe00700 1 -- 172.21.15.6:0/4258457873 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f3680025040 con 0x555693f81dd0 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.061+0000 7f369fab1b80 10 client.26146 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.061+0000 7f369fab1b80 10 client.26146 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.061+0000 7f369fab1b80 20 client.26146 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.061+0000 7f369fab1b80 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555694004a60 0x555694006e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.061+0000 7f369fab1b80 4 client.26146 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.9, "await_for": 42.0}] 2024-04-26T01:53:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.061+0000 7f369fab1b80 1 -- 172.21.15.6:0/4258457873 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.9, "await_for": 42.0}) -- 0x555693edc8b0 con 0x555694004a60 2024-04-26T01:53:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.061+0000 7f368d400700 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555694004a60 0x555694006e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.061+0000 7f368d400700 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555694004a60 0x555694006e80 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3c77e2d0d3e459b server_cookie=952c263436524d3c in_seq=0 out_seq=0 2024-04-26T01:53:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.061+0000 7f366fe00700 10 client.26146 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:40.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.138+0000 7f366fe00700 1 -- 172.21.15.6:0/4258457873 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x555693edc8b0 con 0x555694004a60 2024-04-26T01:53:40.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.138+0000 7f366fe00700 10 client.26146 handle_command_reply: tid=0 2024-04-26T01:53:40.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.138+0000 7f369fab1b80 1 client.26146 shutdown 2024-04-26T01:53:40.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.138+0000 7f369fab1b80 1 -- 172.21.15.6:0/4258457873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 msgr2=0x555693f92430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:40.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.138+0000 7f369fab1b80 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f92430 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3680009f50 tx=0x7f36800056f0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.138+0000 7f369fab1b80 1 -- 172.21.15.6:0/4258457873 shutdown_connections 2024-04-26T01:53:40.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.138+0000 7f369fab1b80 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555693f8d560 0x555693f92eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.138+0000 7f369fab1b80 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555694004a60 0x555694006e80 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.138+0000 7f369fab1b80 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555693f81dd0 0x555693f92430 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.138+0000 7f369fab1b80 1 --2- 172.21.15.6:0/4258457873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555693f826f0 0x555693f92970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.138+0000 7f369fab1b80 1 -- 172.21.15.6:0/4258457873 >> 172.21.15.6:0/4258457873 conn(0x555693eb10e0 msgr2=0x555693f7dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:40.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.138+0000 7f366fe00700 0 client.26146 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:40.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.139+0000 7f369fab1b80 1 -- 172.21.15.6:0/4258457873 shutdown_connections 2024-04-26T01:53:40.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.139+0000 7f369fab1b80 1 -- 172.21.15.6:0/4258457873 wait complete. 2024-04-26T01:53:40.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.139+0000 7f369fab1b80 20 client.26146 trim_cache size 0 max 16384 2024-04-26T01:53:40.139 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:53:40.139 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 41, 2024-04-26T01:53:40.139 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 22, 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "1f0250cf": { 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 22, 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 82.9, 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:53:40.140 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:53:40.141 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:53:40.141 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:53:40.141 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:53:40.141 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:53:40.141 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:53:40.141 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:53:40.141 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:53:40.141 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:53:40.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.141+0000 7f3695200700 1 -- 172.21.15.6:0/2549624444 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36740a3ed0 msgr2=0x7f36740a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:40.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.141+0000 7f3695200700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36740a3ed0 0x7f36740a6390 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3680011550 tx=0x7f3680009260 comp rx=0 tx=0).stop 2024-04-26T01:53:40.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.141+0000 7f3695200700 1 -- 172.21.15.6:0/2549624444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3690109c80 msgr2=0x7f369019fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:40.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.141+0000 7f3695200700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3690109c80 0x7f369019fb70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f367c005fd0 tx=0x7f367c00ee70 comp rx=0 tx=0).stop 2024-04-26T01:53:40.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.141+0000 7f3695200700 1 -- 172.21.15.6:0/2549624444 shutdown_connections 2024-04-26T01:53:40.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.141+0000 7f3695200700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36740a3ed0 0x7f36740a6390 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.141+0000 7f3695200700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f369010e3b0 0x7f36901a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.141+0000 7f3695200700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36901092d0 0x7f369019f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.141+0000 7f3695200700 1 --2- 172.21.15.6:0/2549624444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3690109c80 0x7f369019fb70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.141+0000 7f3695200700 1 -- 172.21.15.6:0/2549624444 >> 172.21.15.6:0/2549624444 conn(0x7f3690076210 msgr2=0x7f3690076690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:40.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.141+0000 7f3695200700 1 -- 172.21.15.6:0/2549624444 shutdown_connections 2024-04-26T01:53:40.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:53:40.141+0000 7f3695200700 1 -- 172.21.15.6:0/2549624444 wait complete. 2024-04-26T01:53:40.597 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 4.07 MB/sec execute 244 sec latency 520.074 ms 2024-04-26T01:53:40.743 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:40 smithi029 ceph-mon[32775]: pgmap v243: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 952 KiB/s wr, 137 op/s 2024-04-26T01:53:40.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:40 smithi006 ceph-mon[28419]: pgmap v243: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 952 KiB/s wr, 137 op/s 2024-04-26T01:53:40.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:40 smithi188 ceph-mon[32751]: pgmap v243: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 952 KiB/s wr, 137 op/s 2024-04-26T01:53:41.020 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1f0250cf', quiesce took 8.0 seconds. Will release after: 22.9 2024-04-26T01:53:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.494+0000 7f3d76000640 1 -- 172.21.15.6:0/2612159912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d7010e470 msgr2=0x7f3d7010e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.494+0000 7f3d76000640 1 --2- 172.21.15.6:0/2612159912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d7010e470 0x7f3d7010e850 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3d64009900 tx=0x7f3d6402a490 comp rx=0 tx=0).stop 2024-04-26T01:53:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.494+0000 7f3d76000640 1 -- 172.21.15.6:0/2612159912 shutdown_connections 2024-04-26T01:53:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.494+0000 7f3d76000640 1 --2- 172.21.15.6:0/2612159912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d70108d50 0x7f3d70115210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.494+0000 7f3d76000640 1 --2- 172.21.15.6:0/2612159912 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d701083b0 0x7f3d70108810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.494+0000 7f3d76000640 1 --2- 172.21.15.6:0/2612159912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d7010e470 0x7f3d7010e850 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.494+0000 7f3d76000640 1 -- 172.21.15.6:0/2612159912 >> 172.21.15.6:0/2612159912 conn(0x7f3d70077b00 msgr2=0x7f3d70079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.494+0000 7f3d76000640 1 -- 172.21.15.6:0/2612159912 shutdown_connections 2024-04-26T01:53:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.494+0000 7f3d76000640 1 -- 172.21.15.6:0/2612159912 wait complete. 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.495+0000 7f3d76000640 1 Processor -- start 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.495+0000 7f3d76000640 1 -- start start 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.495+0000 7f3d76000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d701083b0 0x7f3d701a47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d76000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d70108d50 0x7f3d701a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d76000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7010e470 0x7f3d7019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d76000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3d70118650 con 0x7f3d70108d50 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d76000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701184d0 con 0x7f3d7010e470 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d76000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701187d0 con 0x7f3d701083b0 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d74c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d701083b0 0x7f3d701a47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d74c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d701083b0 0x7f3d701a47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37308/0 (socket says 172.21.15.6:37308) 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d74c00640 1 -- 172.21.15.6:0/2954199680 learned_addr learned my addr 172.21.15.6:0/2954199680 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d6fe00640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d70108d50 0x7f3d701a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d75600640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7010e470 0x7f3d7019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d74c00640 1 -- 172.21.15.6:0/2954199680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7010e470 msgr2=0x7f3d7019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d74c00640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7010e470 0x7f3d7019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d74c00640 1 -- 172.21.15.6:0/2954199680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d70108d50 msgr2=0x7f3d701a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d74c00640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d70108d50 0x7f3d701a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.496+0000 7f3d74c00640 1 -- 172.21.15.6:0/2954199680 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7019f0f0 con 0x7f3d701083b0 2024-04-26T01:53:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.497+0000 7f3d74c00640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d701083b0 0x7f3d701a47c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3d64005ab0 tx=0x7f3d64003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.497+0000 7f3d6d600640 1 -- 172.21.15.6:0/2954199680 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6403f070 con 0x7f3d701083b0 2024-04-26T01:53:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.497+0000 7f3d6d600640 1 -- 172.21.15.6:0/2954199680 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d64031b60 con 0x7f3d701083b0 2024-04-26T01:53:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.497+0000 7f3d6d600640 1 -- 172.21.15.6:0/2954199680 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d64035030 con 0x7f3d701083b0 2024-04-26T01:53:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.497+0000 7f3d76000640 1 -- 172.21.15.6:0/2954199680 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d7019f380 con 0x7f3d701083b0 2024-04-26T01:53:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.497+0000 7f3d76000640 1 -- 172.21.15.6:0/2954199680 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d701ab680 con 0x7f3d701083b0 2024-04-26T01:53:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.500+0000 7f3d6d600640 1 -- 172.21.15.6:0/2954199680 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f3d64042070 con 0x7f3d701083b0 2024-04-26T01:53:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.500+0000 7f3d7a585740 1 Processor -- start 2024-04-26T01:53:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.501+0000 7f3d7a585740 1 -- start start 2024-04-26T01:53:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.501+0000 7f3d7a585740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 0x55565f31ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.501+0000 7f3d7a585740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31b2d0 0x55565f325650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.501+0000 7f3d7a585740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f327f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.501+0000 7f3d7a585740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55565f280900 con 0x55565f325b90 2024-04-26T01:53:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.501+0000 7f3d7a585740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55565f226ef0 con 0x55565f31b2d0 2024-04-26T01:53:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.501+0000 7f3d7a585740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55565f27c8c0 con 0x55565f31a9b0 2024-04-26T01:53:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.501+0000 7f3d75600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 0x55565f31ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.501+0000 7f3d75600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 0x55565f31ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37318/0 (socket says 172.21.15.6:37318) 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.501+0000 7f3d6fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f327f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.501+0000 7f3d75600640 1 -- 172.21.15.6:0/3981467367 learned_addr learned my addr 172.21.15.6:0/3981467367 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.501+0000 7f3d74c00640 1 --2- 172.21.15.6:0/3981467367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31b2d0 0x55565f325650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.502+0000 7f3d75600640 1 -- 172.21.15.6:0/3981467367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31b2d0 msgr2=0x55565f325650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.502+0000 7f3d75600640 1 --2- 172.21.15.6:0/3981467367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31b2d0 0x55565f325650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.502+0000 7f3d75600640 1 -- 172.21.15.6:0/3981467367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 msgr2=0x55565f327f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.502+0000 7f3d75600640 1 --2- 172.21.15.6:0/3981467367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f327f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.502+0000 7f3d75600640 1 -- 172.21.15.6:0/3981467367 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55565ef93210 con 0x55565f31a9b0 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.502+0000 7f3d75600640 1 --2- 172.21.15.6:0/3981467367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 0x55565f31ad90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3d68011440 tx=0x7f3d68011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15832b38570c56b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.502+0000 7f3d6d600640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3d580a3d80 0x7f3d580a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.502+0000 7f3d61c00640 1 -- 172.21.15.6:0/3981467367 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d68003c30 con 0x55565f31a9b0 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.502+0000 7f3d61c00640 1 -- 172.21.15.6:0/3981467367 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d68003dd0 con 0x55565f31a9b0 2024-04-26T01:53:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.502+0000 7f3d61c00640 1 -- 172.21.15.6:0/3981467367 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6800fa80 con 0x55565f31a9b0 2024-04-26T01:53:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.503+0000 7f3d6d600640 1 -- 172.21.15.6:0/2954199680 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f3d640f0e50 con 0x7f3d701083b0 2024-04-26T01:53:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.503+0000 7f3d74c00640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3d580a3d80 0x7f3d580a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.503+0000 7f3d7a585740 1 -- 172.21.15.6:0/3981467367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 msgr2=0x55565f31ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.503+0000 7f3d7a585740 1 --2- 172.21.15.6:0/3981467367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 0x55565f31ad90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3d68011440 tx=0x7f3d68011900 comp rx=0 tx=0).stop 2024-04-26T01:53:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.503+0000 7f3d74c00640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3d580a3d80 0x7f3d580a6240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x55565f31bf50 tx=0x7f3d640f4900 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.503+0000 7f3d7a585740 1 -- 172.21.15.6:0/3981467367 shutdown_connections 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.503+0000 7f3d7a585740 1 --2- 172.21.15.6:0/3981467367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f327f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.503+0000 7f3d7a585740 1 --2- 172.21.15.6:0/3981467367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31b2d0 0x55565f325650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.503+0000 7f3d7a585740 1 --2- 172.21.15.6:0/3981467367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 0x55565f31ad90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.503+0000 7f3d7a585740 1 -- 172.21.15.6:0/3981467367 >> 172.21.15.6:0/3981467367 conn(0x55565f316ec0 msgr2=0x55565f3172a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.503+0000 7f3d7a585740 1 -- 172.21.15.6:0/3981467367 shutdown_connections 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.503+0000 7f3d7a585740 1 -- 172.21.15.6:0/3981467367 wait complete. 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.504+0000 7f3d7a585740 1 -- start start 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.504+0000 7f3d7a585740 1 Processor -- start 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d7a585740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31a9b0 0x55565f322b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d7a585740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 0x55565f323040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d7a585740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f323580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d7a585740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55565f3287c0 con 0x55565f325b90 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d7a585740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55565f23a310 con 0x55565f31a9b0 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d7a585740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55565f328940 con 0x55565f31b2d0 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d7a585740 10 client.0 _ll_register_callbacks cb 0x55565f208cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d75600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f323580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d6fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 0x55565f323040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d75600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f323580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33248/0 (socket says 172.21.15.6:33248) 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d75600640 1 -- 172.21.15.6:0/1301811096 learned_addr learned my addr 172.21.15.6:0/1301811096 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d75600640 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31a9b0 0x55565f322b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d6fe00640 1 -- 172.21.15.6:0/1301811096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31a9b0 msgr2=0x55565f322b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d6fe00640 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31a9b0 0x55565f322b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.505+0000 7f3d6fe00640 1 -- 172.21.15.6:0/1301811096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 msgr2=0x55565f323580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.506+0000 7f3d6fe00640 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f323580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.506+0000 7f3d6fe00640 1 -- 172.21.15.6:0/1301811096 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55565f05f620 con 0x55565f31b2d0 2024-04-26T01:53:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.506+0000 7f3d75600640 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31a9b0 0x55565f322b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.506+0000 7f3d6fe00640 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 0x55565f323040 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3d7010db50 tx=0x7f3d5c012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab4fc42dd674fc36 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.506+0000 7f3d56000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.506+0000 7f3d56000640 1 -- 172.21.15.6:0/1301811096 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c004020 con 0x55565f31b2d0 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.506+0000 7f3d56000640 1 -- 172.21.15.6:0/1301811096 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f3d5c0041c0 con 0x55565f31b2d0 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.506+0000 7f3d7a585740 1 -- 172.21.15.6:0/1301811096 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d68003dd0 con 0x55565f31b2d0 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.506+0000 7f3d56000640 1 -- 172.21.15.6:0/1301811096 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f3d5c0043c0 con 0x55565f31b2d0 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.506+0000 7f3d7a585740 10 client.26158 fetch_fsmap learned FSMap version 41 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.506+0000 7f3d7a585740 1 -- 172.21.15.6:0/1301811096 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55565f334ad0 con 0x55565f31b2d0 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.507+0000 7f3d56000640 1 -- 172.21.15.6:0/1301811096 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d68003dd0 con 0x55565f31b2d0 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.507+0000 7f3d56000640 1 -- 172.21.15.6:0/1301811096 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f3d5c003ab0 con 0x55565f31b2d0 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.507+0000 7f3d7a585740 10 client.26158 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.507+0000 7f3d7a585740 10 client.26158 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.507+0000 7f3d7a585740 20 client.26158 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.507+0000 7f3d7a585740 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55565f336030 0x55565f338440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.507+0000 7f3d7a585740 4 client.26158 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.507+0000 7f3d7a585740 1 -- 172.21.15.6:0/1301811096 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55565f2948b0 con 0x55565f336030 2024-04-26T01:53:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.507+0000 7f3d75600640 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55565f336030 0x55565f338440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.508+0000 7f3d75600640 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55565f336030 0x55565f338440 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=898422565dc99cf server_cookie=feed580d24b0c49 in_seq=0 out_seq=0 2024-04-26T01:53:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.508+0000 7f3d56000640 10 client.26158 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d56000640 1 -- 172.21.15.6:0/1301811096 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55565f2948b0 con 0x55565f336030 2024-04-26T01:53:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d56000640 10 client.26158 handle_command_reply: tid=0 2024-04-26T01:53:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d7a585740 1 client.26158 shutdown 2024-04-26T01:53:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d7a585740 1 -- 172.21.15.6:0/1301811096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 msgr2=0x55565f323040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d7a585740 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 0x55565f323040 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3d7010db50 tx=0x7f3d5c012b60 comp rx=0 tx=0).stop 2024-04-26T01:53:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d7a585740 1 -- 172.21.15.6:0/1301811096 shutdown_connections 2024-04-26T01:53:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d7a585740 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55565f336030 0x55565f338440 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d7a585740 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f323580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d7a585740 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 0x55565f323040 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d7a585740 1 --2- 172.21.15.6:0/1301811096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31a9b0 0x55565f322b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d56000640 0 client.26158 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:41.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d7a585740 1 -- 172.21.15.6:0/1301811096 >> 172.21.15.6:0/1301811096 conn(0x55565f316ec0 msgr2=0x55565f31cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:41.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d7a585740 1 -- 172.21.15.6:0/1301811096 shutdown_connections 2024-04-26T01:53:41.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d7a585740 1 -- 172.21.15.6:0/1301811096 wait complete. 2024-04-26T01:53:41.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.509+0000 7f3d7a585740 20 client.26158 trim_cache size 0 max 16384 2024-04-26T01:53:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.526+0000 7f3d7a585740 1 Processor -- start 2024-04-26T01:53:41.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.526+0000 7f3d7a585740 1 -- start start 2024-04-26T01:53:41.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.526+0000 7f3d7a585740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 0x55565f3316b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.526+0000 7f3d7a585740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f31b2d0 0x55565f3233a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.526+0000 7f3d7a585740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f325b90 0x55565f33dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.526+0000 7f3d7a585740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701b6a30 con 0x55565f31b2d0 2024-04-26T01:53:41.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.526+0000 7f3d7a585740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701b42a0 con 0x55565f325b90 2024-04-26T01:53:41.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.526+0000 7f3d7a585740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701b2110 con 0x55565f31a9b0 2024-04-26T01:53:41.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.527+0000 7f3d75600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 0x55565f3316b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.527+0000 7f3d6fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f31b2d0 0x55565f3233a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.527+0000 7f3d75600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 0x55565f3316b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37338/0 (socket says 172.21.15.6:37338) 2024-04-26T01:53:41.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.527+0000 7f3d75600640 1 -- 172.21.15.6:0/683953767 learned_addr learned my addr 172.21.15.6:0/683953767 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:41.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.527+0000 7f3d75600640 1 --2- 172.21.15.6:0/683953767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f325b90 0x55565f33dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.527+0000 7f3d6fe00640 1 -- 172.21.15.6:0/683953767 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 msgr2=0x55565f3316b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.527+0000 7f3d6fe00640 1 --2- 172.21.15.6:0/683953767 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 0x55565f3316b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.527+0000 7f3d6fe00640 1 -- 172.21.15.6:0/683953767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f325b90 msgr2=0x55565f33dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.527+0000 7f3d6fe00640 1 --2- 172.21.15.6:0/683953767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f325b90 0x55565f33dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.527+0000 7f3d6fe00640 1 -- 172.21.15.6:0/683953767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55565f31bc60 con 0x55565f31b2d0 2024-04-26T01:53:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.527+0000 7f3d6fe00640 1 --2- 172.21.15.6:0/683953767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f31b2d0 0x55565f3233a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c003f50 tx=0x7f3d5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48108d80cd32f929 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.528+0000 7f3d61200640 1 -- 172.21.15.6:0/683953767 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c004020 con 0x55565f31b2d0 2024-04-26T01:53:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.528+0000 7f3d61200640 1 -- 172.21.15.6:0/683953767 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d5c0041c0 con 0x55565f31b2d0 2024-04-26T01:53:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.528+0000 7f3d61200640 1 -- 172.21.15.6:0/683953767 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c003c80 con 0x55565f31b2d0 2024-04-26T01:53:41.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.528+0000 7f3d7a585740 1 -- 172.21.15.6:0/683953767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f31b2d0 msgr2=0x55565f3233a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:41.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.528+0000 7f3d7a585740 1 --2- 172.21.15.6:0/683953767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f31b2d0 0x55565f3233a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c003f50 tx=0x7f3d5c005830 comp rx=0 tx=0).stop 2024-04-26T01:53:41.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.528+0000 7f3d7a585740 1 -- 172.21.15.6:0/683953767 shutdown_connections 2024-04-26T01:53:41.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.528+0000 7f3d7a585740 1 --2- 172.21.15.6:0/683953767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f31b2d0 0x55565f3233a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.528+0000 7f3d7a585740 1 --2- 172.21.15.6:0/683953767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f325b90 0x55565f33dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.528+0000 7f3d7a585740 1 --2- 172.21.15.6:0/683953767 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31a9b0 0x55565f3316b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.528+0000 7f3d7a585740 1 -- 172.21.15.6:0/683953767 >> 172.21.15.6:0/683953767 conn(0x55565f316ec0 msgr2=0x55565f316020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:41.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.529+0000 7f3d7a585740 1 -- 172.21.15.6:0/683953767 shutdown_connections 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.529+0000 7f3d7a585740 1 -- 172.21.15.6:0/683953767 wait complete. 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.529+0000 7f3d7a585740 1 -- start start 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.529+0000 7f3d7a585740 1 Processor -- start 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.529+0000 7f3d7a585740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31a9b0 0x55565f32dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d7a585740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 0x55565f32b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d7a585740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f32b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d7a585740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701b2760 con 0x55565f325b90 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d7a585740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701b3bd0 con 0x55565f31a9b0 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d7a585740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701b2e30 con 0x55565f31b2d0 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d7a585740 10 client.0 _ll_register_callbacks cb 0x7f3d5c009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d6fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 0x55565f32b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d6fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 0x55565f32b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37348/0 (socket says 172.21.15.6:37348) 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d6fe00640 1 -- 172.21.15.6:0/4132807518 learned_addr learned my addr 172.21.15.6:0/4132807518 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d75600640 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31a9b0 0x55565f32dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d75600640 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f32b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d6fe00640 1 -- 172.21.15.6:0/4132807518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31a9b0 msgr2=0x55565f32dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d6fe00640 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31a9b0 0x55565f32dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d6fe00640 1 -- 172.21.15.6:0/4132807518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 msgr2=0x55565f32b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d6fe00640 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f32b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d6fe00640 1 -- 172.21.15.6:0/4132807518 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55565f32bcb0 con 0x55565f31b2d0 2024-04-26T01:53:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.530+0000 7f3d75600640 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f32b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.531+0000 7f3d6fe00640 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 0x55565f32b130 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c01c490 tx=0x7f3d5c003880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c4629d973b3ba35 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.531+0000 7f3d56000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:53:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.531+0000 7f3d56000640 1 -- 172.21.15.6:0/4132807518 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c005250 con 0x55565f31b2d0 2024-04-26T01:53:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.531+0000 7f3d7a585740 1 -- 172.21.15.6:0/4132807518 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d5c0041c0 con 0x55565f31b2d0 2024-04-26T01:53:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.531+0000 7f3d56000640 1 -- 172.21.15.6:0/4132807518 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f3d5c0053f0 con 0x55565f31b2d0 2024-04-26T01:53:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.531+0000 7f3d56000640 1 -- 172.21.15.6:0/4132807518 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f3d5c03d050 con 0x55565f31b2d0 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.531+0000 7f3d7a585740 10 client.26164 fetch_fsmap learned FSMap version 41 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.531+0000 7f3d7a585740 1 -- 172.21.15.6:0/4132807518 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55565f39c3b0 con 0x55565f31b2d0 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.532+0000 7f3d56000640 1 -- 172.21.15.6:0/4132807518 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d5c0041c0 con 0x55565f31b2d0 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.532+0000 7f3d56000640 1 -- 172.21.15.6:0/4132807518 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f3d5c02b020 con 0x55565f31b2d0 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.532+0000 7f3d7a585740 10 client.26164 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.532+0000 7f3d7a585740 10 client.26164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.532+0000 7f3d7a585740 20 client.26164 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.532+0000 7f3d7a585740 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55565f39c9c0 0x55565f39edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.532+0000 7f3d7a585740 4 client.26164 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.532+0000 7f3d7a585740 1 -- 172.21.15.6:0/4132807518 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55565f28e320 con 0x55565f39c9c0 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.532+0000 7f3d75600640 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55565f39c9c0 0x55565f39edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.532+0000 7f3d75600640 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55565f39c9c0 0x55565f39edb0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9738a532b2eb4067 server_cookie=f48279d87214cf06 in_seq=0 out_seq=0 2024-04-26T01:53:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.532+0000 7f3d56000640 10 client.26164 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.584+0000 7f3d56000640 1 -- 172.21.15.6:0/4132807518 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55565f28e320 con 0x55565f39c9c0 2024-04-26T01:53:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.584+0000 7f3d56000640 10 client.26164 handle_command_reply: tid=0 2024-04-26T01:53:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.585+0000 7f3d7a585740 1 client.26164 shutdown 2024-04-26T01:53:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.585+0000 7f3d7a585740 1 -- 172.21.15.6:0/4132807518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 msgr2=0x55565f32b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.585+0000 7f3d7a585740 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 0x55565f32b130 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c01c490 tx=0x7f3d5c003880 comp rx=0 tx=0).stop 2024-04-26T01:53:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.585+0000 7f3d7a585740 1 -- 172.21.15.6:0/4132807518 shutdown_connections 2024-04-26T01:53:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.585+0000 7f3d7a585740 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55565f39c9c0 0x55565f39edb0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.585+0000 7f3d7a585740 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55565f325b90 0x55565f32b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.585+0000 7f3d7a585740 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55565f31b2d0 0x55565f32b130 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.585+0000 7f3d7a585740 1 --2- 172.21.15.6:0/4132807518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55565f31a9b0 0x55565f32dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.585+0000 7f3d56000640 0 client.26164 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.585+0000 7f3d7a585740 1 -- 172.21.15.6:0/4132807518 >> 172.21.15.6:0/4132807518 conn(0x55565f316ec0 msgr2=0x55565f31c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.585+0000 7f3d7a585740 1 -- 172.21.15.6:0/4132807518 shutdown_connections 2024-04-26T01:53:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.585+0000 7f3d7a585740 1 -- 172.21.15.6:0/4132807518 wait complete. 2024-04-26T01:53:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.586+0000 7f3d7a585740 20 client.26164 trim_cache size 0 max 16384 2024-04-26T01:53:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:53:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:53:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:53:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:53:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.586+0000 7f3d76000640 1 -- 172.21.15.6:0/2954199680 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3d580a3d80 msgr2=0x7f3d580a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.586+0000 7f3d76000640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3d580a3d80 0x7f3d580a6240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x55565f31bf50 tx=0x7f3d640f4900 comp rx=0 tx=0).stop 2024-04-26T01:53:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.586+0000 7f3d76000640 1 -- 172.21.15.6:0/2954199680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d701083b0 msgr2=0x7f3d701a47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.586+0000 7f3d76000640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d701083b0 0x7f3d701a47c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3d64005ab0 tx=0x7f3d64003c10 comp rx=0 tx=0).stop 2024-04-26T01:53:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.587+0000 7f3d76000640 1 -- 172.21.15.6:0/2954199680 shutdown_connections 2024-04-26T01:53:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.587+0000 7f3d76000640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3d580a3d80 0x7f3d580a6240 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.587+0000 7f3d76000640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d70108d50 0x7f3d701a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.587+0000 7f3d76000640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7010e470 0x7f3d7019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.587+0000 7f3d76000640 1 --2- 172.21.15.6:0/2954199680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d701083b0 0x7f3d701a47c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.587+0000 7f3d76000640 1 -- 172.21.15.6:0/2954199680 >> 172.21.15.6:0/2954199680 conn(0x7f3d70077b00 msgr2=0x7f3d70079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.587+0000 7f3d76000640 1 -- 172.21.15.6:0/2954199680 shutdown_connections 2024-04-26T01:53:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:41.587+0000 7f3d76000640 1 -- 172.21.15.6:0/2954199680 wait complete. 2024-04-26T01:53:41.597 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 4.06 MB/sec execute 245 sec latency 1520.199 ms 2024-04-26T01:53:42.597 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 4.04 MB/sec execute 246 sec latency 2520.345 ms 2024-04-26T01:53:42.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:42 smithi006 ceph-mon[28419]: pgmap v244: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 208 op/s 2024-04-26T01:53:42.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:42 smithi188 ceph-mon[32751]: pgmap v244: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 208 op/s 2024-04-26T01:53:42.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:42 smithi029 ceph-mon[32775]: pgmap v244: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 208 op/s 2024-04-26T01:53:43.598 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 4.02 MB/sec execute 247 sec latency 3520.509 ms 2024-04-26T01:53:44.598 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 4.01 MB/sec execute 248 sec latency 4520.678 ms 2024-04-26T01:53:44.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:44 smithi006 ceph-mon[28419]: pgmap v245: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-04-26T01:53:44.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:44 smithi188 ceph-mon[32751]: pgmap v245: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-04-26T01:53:44.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:44 smithi029 ceph-mon[32775]: pgmap v245: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-04-26T01:53:45.598 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.99 MB/sec execute 249 sec latency 5520.815 ms 2024-04-26T01:53:46.598 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.97 MB/sec execute 250 sec latency 6520.970 ms 2024-04-26T01:53:46.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:46 smithi006 ceph-mon[28419]: pgmap v246: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-04-26T01:53:46.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:46 smithi188 ceph-mon[32751]: pgmap v246: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-04-26T01:53:46.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:46 smithi029 ceph-mon[32775]: pgmap v246: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-04-26T01:53:47.598 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.96 MB/sec execute 251 sec latency 7521.156 ms 2024-04-26T01:53:48.598 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.94 MB/sec execute 252 sec latency 8521.252 ms 2024-04-26T01:53:48.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:48 smithi006 ceph-mon[28419]: pgmap v247: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-04-26T01:53:48.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:48 smithi188 ceph-mon[32751]: pgmap v247: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-04-26T01:53:48.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:48 smithi029 ceph-mon[32775]: pgmap v247: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-04-26T01:53:49.599 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.93 MB/sec execute 253 sec latency 9521.348 ms 2024-04-26T01:53:50.599 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.91 MB/sec execute 254 sec latency 10521.493 ms 2024-04-26T01:53:50.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:50 smithi006 ceph-mon[28419]: pgmap v248: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 1.2 MiB/s wr, 112 op/s 2024-04-26T01:53:50.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:50 smithi188 ceph-mon[32751]: pgmap v248: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 1.2 MiB/s wr, 112 op/s 2024-04-26T01:53:50.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:50 smithi029 ceph-mon[32775]: pgmap v248: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 1.2 MiB/s wr, 112 op/s 2024-04-26T01:53:51.599 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.90 MB/sec execute 255 sec latency 11521.656 ms 2024-04-26T01:53:52.599 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.88 MB/sec execute 256 sec latency 12521.763 ms 2024-04-26T01:53:52.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:52 smithi006 ceph-mon[28419]: pgmap v249: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-04-26T01:53:52.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:52 smithi188 ceph-mon[32751]: pgmap v249: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-04-26T01:53:52.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:52 smithi029 ceph-mon[32775]: pgmap v249: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-04-26T01:53:53.599 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.87 MB/sec execute 257 sec latency 13521.943 ms 2024-04-26T01:53:54.599 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.85 MB/sec execute 258 sec latency 14522.137 ms 2024-04-26T01:53:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.732+0000 7fcc38e00640 1 -- 172.21.15.6:0/2863460919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc34074c40 msgr2=0x7fcc3410e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.732+0000 7fcc38e00640 1 --2- 172.21.15.6:0/2863460919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc34074c40 0x7fcc3410e0b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009900 tx=0x7fcc2002a480 comp rx=0 tx=0).stop 2024-04-26T01:53:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.732+0000 7fcc38e00640 1 -- 172.21.15.6:0/2863460919 shutdown_connections 2024-04-26T01:53:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.732+0000 7fcc38e00640 1 --2- 172.21.15.6:0/2863460919 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc3410e5f0 0x7fcc34110a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.732+0000 7fcc38e00640 1 --2- 172.21.15.6:0/2863460919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc34074c40 0x7fcc3410e0b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.732+0000 7fcc38e00640 1 --2- 172.21.15.6:0/2863460919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc34074320 0x7fcc34074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.732+0000 7fcc38e00640 1 -- 172.21.15.6:0/2863460919 >> 172.21.15.6:0/2863460919 conn(0x7fcc340fd880 msgr2=0x7fcc340ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.733+0000 7fcc38e00640 1 -- 172.21.15.6:0/2863460919 shutdown_connections 2024-04-26T01:53:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.733+0000 7fcc38e00640 1 -- 172.21.15.6:0/2863460919 wait complete. 2024-04-26T01:53:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.734+0000 7fcc38e00640 1 Processor -- start 2024-04-26T01:53:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.734+0000 7fcc38e00640 1 -- start start 2024-04-26T01:53:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.734+0000 7fcc38e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc34074320 0x7fcc341a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.735+0000 7fcc38e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc34074c40 0x7fcc341a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.735+0000 7fcc38e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc3410e5f0 0x7fcc3419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.735+0000 7fcc38e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34114270 con 0x7fcc34074320 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.735+0000 7fcc38e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341140f0 con 0x7fcc34074c40 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.735+0000 7fcc38e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341143f0 con 0x7fcc3410e5f0 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.735+0000 7fcc33400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc34074320 0x7fcc341a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.735+0000 7fcc33400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc34074320 0x7fcc341a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46706/0 (socket says 172.21.15.6:46706) 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.735+0000 7fcc33400640 1 -- 172.21.15.6:0/2157455902 learned_addr learned my addr 172.21.15.6:0/2157455902 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.735+0000 7fcc32a00640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc34074c40 0x7fcc341a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.735+0000 7fcc33e00640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc3410e5f0 0x7fcc3419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.736+0000 7fcc33400640 1 -- 172.21.15.6:0/2157455902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc3410e5f0 msgr2=0x7fcc3419a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.736+0000 7fcc33400640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc3410e5f0 0x7fcc3419a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.736+0000 7fcc33400640 1 -- 172.21.15.6:0/2157455902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc34074c40 msgr2=0x7fcc341a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.736+0000 7fcc33400640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc34074c40 0x7fcc341a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.736+0000 7fcc33400640 1 -- 172.21.15.6:0/2157455902 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3419ad50 con 0x7fcc34074320 2024-04-26T01:53:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.736+0000 7fcc32a00640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc34074c40 0x7fcc341a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.736+0000 7fcc33400640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc34074320 0x7fcc341a0420 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400d950 tx=0x7fcc2400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.736+0000 7fcc2be00640 1 -- 172.21.15.6:0/2157455902 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24017070 con 0x7fcc34074320 2024-04-26T01:53:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.736+0000 7fcc38e00640 1 -- 172.21.15.6:0/2157455902 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3419b040 con 0x7fcc34074320 2024-04-26T01:53:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.736+0000 7fcc2be00640 1 -- 172.21.15.6:0/2157455902 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc24009c30 con 0x7fcc34074320 2024-04-26T01:53:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.736+0000 7fcc2be00640 1 -- 172.21.15.6:0/2157455902 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2400f620 con 0x7fcc34074320 2024-04-26T01:53:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.736+0000 7fcc38e00640 1 -- 172.21.15.6:0/2157455902 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc341a72e0 con 0x7fcc34074320 2024-04-26T01:53:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc3cf22740 1 Processor -- start 2024-04-26T01:53:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc3cf22740 1 -- start start 2024-04-26T01:53:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc3cf22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75241d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc3cf22740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc3cf22740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d7524ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc2be00640 1 -- 172.21.15.6:0/2157455902 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fcc24010460 con 0x7fcc34074320 2024-04-26T01:53:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc3cf22740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x562d751a7900 con 0x562d7524cb90 2024-04-26T01:53:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc3cf22740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562d7514def0 con 0x562d752422d0 2024-04-26T01:53:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc3cf22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562d751a38c0 con 0x562d752419b0 2024-04-26T01:53:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc33e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75241d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc33e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75241d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:55000/0 (socket says 172.21.15.6:55000) 2024-04-26T01:53:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc32a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d7524ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.740+0000 7fcc33e00640 1 -- 172.21.15.6:0/511972931 learned_addr learned my addr 172.21.15.6:0/511972931 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.741+0000 7fcc33400640 1 --2- 172.21.15.6:0/511972931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.741+0000 7fcc33e00640 1 -- 172.21.15.6:0/511972931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 msgr2=0x562d7524c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.741+0000 7fcc33e00640 1 --2- 172.21.15.6:0/511972931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.741+0000 7fcc33e00640 1 -- 172.21.15.6:0/511972931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 msgr2=0x562d7524ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.741+0000 7fcc33e00640 1 --2- 172.21.15.6:0/511972931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d7524ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.741+0000 7fcc33e00640 1 -- 172.21.15.6:0/511972931 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d74eba210 con 0x562d752419b0 2024-04-26T01:53:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.741+0000 7fcc32a00640 1 --2- 172.21.15.6:0/511972931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d7524ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:54.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.741+0000 7fcc33e00640 1 --2- 172.21.15.6:0/511972931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75241d90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c009f30 tx=0x7fcc2c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26c656e9b3cf4711 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:54.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.742+0000 7fcc28c00640 1 -- 172.21.15.6:0/511972931 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c004d20 con 0x562d752419b0 2024-04-26T01:53:54.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.742+0000 7fcc28c00640 1 -- 172.21.15.6:0/511972931 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc2c004050 con 0x562d752419b0 2024-04-26T01:53:54.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.742+0000 7fcc28c00640 1 -- 172.21.15.6:0/511972931 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c005280 con 0x562d752419b0 2024-04-26T01:53:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.742+0000 7fcc2be00640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcc180a3d00 0x7fcc180a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.742+0000 7fcc2be00640 1 -- 172.21.15.6:0/2157455902 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7fcc240d0440 con 0x7fcc34074320 2024-04-26T01:53:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.742+0000 7fcc3cf22740 1 -- 172.21.15.6:0/511972931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 msgr2=0x562d75241d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.742+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/511972931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75241d90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c009f30 tx=0x7fcc2c0118d0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.742+0000 7fcc33400640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcc180a3d00 0x7fcc180a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.743+0000 7fcc3cf22740 1 -- 172.21.15.6:0/511972931 shutdown_connections 2024-04-26T01:53:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.743+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/511972931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d7524ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.743+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/511972931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.743+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/511972931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75241d90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.743+0000 7fcc3cf22740 1 -- 172.21.15.6:0/511972931 >> 172.21.15.6:0/511972931 conn(0x562d7523dec0 msgr2=0x562d7523e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:54.747 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.743+0000 7fcc3cf22740 1 -- 172.21.15.6:0/511972931 shutdown_connections 2024-04-26T01:53:54.747 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.743+0000 7fcc3cf22740 1 -- 172.21.15.6:0/511972931 wait complete. 2024-04-26T01:53:54.747 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.743+0000 7fcc3cf22740 1 -- start start 2024-04-26T01:53:54.747 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.743+0000 7fcc33400640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcc180a3d00 0x7fcc180a61c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x562d75242f50 tx=0x7fcc24098040 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:54.747 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.743+0000 7fcc3cf22740 1 Processor -- start 2024-04-26T01:53:54.747 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.744+0000 7fcc3cf22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75249b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.747 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.744+0000 7fcc3cf22740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.747 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.744+0000 7fcc3cf22740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d7524a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.747 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.744+0000 7fcc3cf22740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x562d7524f7c0 con 0x562d7524cb90 2024-04-26T01:53:54.747 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.744+0000 7fcc3cf22740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562d75161310 con 0x562d752422d0 2024-04-26T01:53:54.747 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.744+0000 7fcc3cf22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562d7524f940 con 0x562d752419b0 2024-04-26T01:53:54.747 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.744+0000 7fcc3cf22740 10 client.0 _ll_register_callbacks cb 0x562d7512fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.744+0000 7fcc33e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75249b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.744+0000 7fcc33e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75249b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:55006/0 (socket says 172.21.15.6:55006) 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.744+0000 7fcc33e00640 1 -- 172.21.15.6:0/4090775979 learned_addr learned my addr 172.21.15.6:0/4090775979 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.744+0000 7fcc32a00640 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.744+0000 7fcc33e00640 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d7524a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.745+0000 7fcc32a00640 1 -- 172.21.15.6:0/4090775979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 msgr2=0x562d75249b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.745+0000 7fcc32a00640 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75249b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.745+0000 7fcc32a00640 1 -- 172.21.15.6:0/4090775979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 msgr2=0x562d7524a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.745+0000 7fcc32a00640 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d7524a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.745+0000 7fcc32a00640 1 -- 172.21.15.6:0/4090775979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d74f86620 con 0x562d752422d0 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.745+0000 7fcc32a00640 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524a090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcc2002a9a0 tx=0x7fcc20005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d7cc007fc5552dd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.745+0000 7fcc1d600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.745+0000 7fcc1d600640 1 -- 172.21.15.6:0/4090775979 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc20035040 con 0x562d752422d0 2024-04-26T01:53:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc1d600640 1 -- 172.21.15.6:0/4090775979 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7fcc20004970 con 0x562d752422d0 2024-04-26T01:53:54.749 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc3cf22740 1 -- 172.21.15.6:0/4090775979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc2c004050 con 0x562d752422d0 2024-04-26T01:53:54.749 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc1d600640 1 -- 172.21.15.6:0/4090775979 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fcc20004b70 con 0x562d752422d0 2024-04-26T01:53:54.749 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc3cf22740 10 client.26129 fetch_fsmap learned FSMap version 41 2024-04-26T01:53:54.749 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc3cf22740 1 -- 172.21.15.6:0/4090775979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x562d7525ba50 con 0x562d752422d0 2024-04-26T01:53:54.749 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc1d600640 1 -- 172.21.15.6:0/4090775979 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc2c004050 con 0x562d752422d0 2024-04-26T01:53:54.749 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc1d600640 1 -- 172.21.15.6:0/4090775979 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7fcc200314a0 con 0x562d752422d0 2024-04-26T01:53:54.749 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc3cf22740 10 client.26129 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:53:54.749 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc3cf22740 10 client.26129 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:54.749 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc3cf22740 20 client.26129 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:54.749 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d7525cf60 0x562d7525f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc3cf22740 4 client.26129 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.746+0000 7fcc3cf22740 1 -- 172.21.15.6:0/4090775979 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d751bb8b0 con 0x562d7525cf60 2024-04-26T01:53:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.747+0000 7fcc33e00640 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d7525cf60 0x562d7525f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.747+0000 7fcc33e00640 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d7525cf60 0x562d7525f350 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72e49ccf3254feee server_cookie=82ed2d49b6be0596 in_seq=0 out_seq=0 2024-04-26T01:53:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.747+0000 7fcc1d600640 10 client.26129 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.748+0000 7fcc1d600640 1 -- 172.21.15.6:0/4090775979 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562d751bb8b0 con 0x562d7525cf60 2024-04-26T01:53:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.748+0000 7fcc1d600640 10 client.26129 handle_command_reply: tid=0 2024-04-26T01:53:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc3cf22740 1 client.26129 shutdown 2024-04-26T01:53:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc3cf22740 1 -- 172.21.15.6:0/4090775979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 msgr2=0x562d7524a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524a090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcc2002a9a0 tx=0x7fcc20005270 comp rx=0 tx=0).stop 2024-04-26T01:53:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc3cf22740 1 -- 172.21.15.6:0/4090775979 shutdown_connections 2024-04-26T01:53:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d7525cf60 0x562d7525f350 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d7524a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524a090 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/4090775979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75249b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc1d600640 0 client.26129 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc3cf22740 1 -- 172.21.15.6:0/4090775979 >> 172.21.15.6:0/4090775979 conn(0x562d7523dec0 msgr2=0x562d75243bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc3cf22740 1 -- 172.21.15.6:0/4090775979 shutdown_connections 2024-04-26T01:53:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc3cf22740 1 -- 172.21.15.6:0/4090775979 wait complete. 2024-04-26T01:53:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.749+0000 7fcc3cf22740 20 client.26129 trim_cache size 0 max 16384 2024-04-26T01:53:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.765+0000 7fcc3cf22740 1 Processor -- start 2024-04-26T01:53:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.766+0000 7fcc3cf22740 1 -- start start 2024-04-26T01:53:54.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.766+0000 7fcc3cf22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75258680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.766+0000 7fcc3cf22740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.766+0000 7fcc3cf22740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d75264c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.766+0000 7fcc3cf22740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341b2610 con 0x562d7524cb90 2024-04-26T01:53:54.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.766+0000 7fcc3cf22740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341afe80 con 0x562d752422d0 2024-04-26T01:53:54.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.766+0000 7fcc3cf22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341adcf0 con 0x562d752419b0 2024-04-26T01:53:54.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.766+0000 7fcc32a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.766+0000 7fcc33e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75258680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.766+0000 7fcc33e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75258680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:55022/0 (socket says 172.21.15.6:55022) 2024-04-26T01:53:54.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.767+0000 7fcc33e00640 1 -- 172.21.15.6:0/2852586154 learned_addr learned my addr 172.21.15.6:0/2852586154 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.766+0000 7fcc32a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39960/0 (socket says 172.21.15.6:39960) 2024-04-26T01:53:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.767+0000 7fcc33e00640 1 --2- 172.21.15.6:0/2852586154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d75264c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.767+0000 7fcc33e00640 1 -- 172.21.15.6:0/2852586154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 msgr2=0x562d7524a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.767+0000 7fcc33e00640 1 --2- 172.21.15.6:0/2852586154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.767+0000 7fcc33e00640 1 -- 172.21.15.6:0/2852586154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 msgr2=0x562d75264c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.767+0000 7fcc33e00640 1 --2- 172.21.15.6:0/2852586154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d75264c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.767+0000 7fcc33e00640 1 -- 172.21.15.6:0/2852586154 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d7525f9b0 con 0x562d752419b0 2024-04-26T01:53:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.767+0000 7fcc32a00640 1 --2- 172.21.15.6:0/2852586154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.767+0000 7fcc33e00640 1 --2- 172.21.15.6:0/2852586154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75258680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c003b10 tx=0x7fcc2c004bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e036140b7be7244c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.768+0000 7fcc1fe00640 1 -- 172.21.15.6:0/2852586154 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c033020 con 0x562d752419b0 2024-04-26T01:53:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.768+0000 7fcc1fe00640 1 -- 172.21.15.6:0/2852586154 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc2c003e40 con 0x562d752419b0 2024-04-26T01:53:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.768+0000 7fcc1fe00640 1 -- 172.21.15.6:0/2852586154 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c004550 con 0x562d752419b0 2024-04-26T01:53:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.768+0000 7fcc3cf22740 1 -- 172.21.15.6:0/2852586154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 msgr2=0x562d75258680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.768+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/2852586154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75258680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c003b10 tx=0x7fcc2c004bc0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.768+0000 7fcc3cf22740 1 -- 172.21.15.6:0/2852586154 shutdown_connections 2024-04-26T01:53:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.768+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/2852586154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d7524cb90 0x562d75264c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.768+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/2852586154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d752422d0 0x562d7524a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.768+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/2852586154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75258680 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.768+0000 7fcc3cf22740 1 -- 172.21.15.6:0/2852586154 >> 172.21.15.6:0/2852586154 conn(0x562d7523dec0 msgr2=0x562d75249b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.768+0000 7fcc3cf22740 1 -- 172.21.15.6:0/2852586154 shutdown_connections 2024-04-26T01:53:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.768+0000 7fcc3cf22740 1 -- 172.21.15.6:0/2852586154 wait complete. 2024-04-26T01:53:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.769+0000 7fcc3cf22740 1 -- start start 2024-04-26T01:53:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.769+0000 7fcc3cf22740 1 Processor -- start 2024-04-26T01:53:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.769+0000 7fcc3cf22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75252d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc3cf22740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d752422d0 0x562d75253240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc3cf22740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d7524cb90 0x562d752c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc3cf22740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341ae340 con 0x562d752422d0 2024-04-26T01:53:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc3cf22740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341af7b0 con 0x562d7524cb90 2024-04-26T01:53:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc3cf22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341aea10 con 0x562d752419b0 2024-04-26T01:53:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc3cf22740 10 client.0 _ll_register_callbacks cb 0x7fcc20004510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc32a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d752422d0 0x562d75253240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc32a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d752422d0 0x562d75253240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46736/0 (socket says 172.21.15.6:46736) 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc32a00640 1 -- 172.21.15.6:0/3014607831 learned_addr learned my addr 172.21.15.6:0/3014607831 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc33e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75252d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc33e00640 1 --2- 172.21.15.6:0/3014607831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d7524cb90 0x562d752c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc32a00640 1 -- 172.21.15.6:0/3014607831 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 msgr2=0x562d75252d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc32a00640 1 --2- 172.21.15.6:0/3014607831 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75252d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc32a00640 1 -- 172.21.15.6:0/3014607831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d7524cb90 msgr2=0x562d752c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc32a00640 1 --2- 172.21.15.6:0/3014607831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d7524cb90 0x562d752c24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.770+0000 7fcc32a00640 1 -- 172.21.15.6:0/3014607831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d7524f1b0 con 0x562d752422d0 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.771+0000 7fcc32a00640 1 --2- 172.21.15.6:0/3014607831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d752422d0 0x562d75253240 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcc20031a80 tx=0x7fcc20005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3f8483b72e5455 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.771+0000 7fcc1d600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.771+0000 7fcc1d600640 1 -- 172.21.15.6:0/3014607831 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc20035070 con 0x562d752422d0 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.771+0000 7fcc3cf22740 1 -- 172.21.15.6:0/3014607831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562d751b5320 con 0x562d752422d0 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.771+0000 7fcc1d600640 1 -- 172.21.15.6:0/3014607831 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7fcc20031d00 con 0x562d752422d0 2024-04-26T01:53:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.771+0000 7fcc1d600640 1 -- 172.21.15.6:0/3014607831 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fcc20052040 con 0x562d752422d0 2024-04-26T01:53:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.771+0000 7fcc3cf22740 10 client.15999 fetch_fsmap learned FSMap version 41 2024-04-26T01:53:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.771+0000 7fcc3cf22740 1 -- 172.21.15.6:0/3014607831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x562d752c3b70 con 0x562d752422d0 2024-04-26T01:53:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.772+0000 7fcc1d600640 1 -- 172.21.15.6:0/3014607831 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562d751b5320 con 0x562d752422d0 2024-04-26T01:53:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.772+0000 7fcc1d600640 1 -- 172.21.15.6:0/3014607831 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7fcc20003aa0 con 0x562d752422d0 2024-04-26T01:53:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.772+0000 7fcc3cf22740 10 client.15999 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:53:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.772+0000 7fcc3cf22740 10 client.15999 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:53:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.772+0000 7fcc3cf22740 20 client.15999 populate_metadata read hostname 'smithi006' 2024-04-26T01:53:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.772+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/3014607831 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d752c42c0 0x562d752c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.772+0000 7fcc3cf22740 4 client.15999 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.772+0000 7fcc3cf22740 1 -- 172.21.15.6:0/3014607831 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562d751af190 con 0x562d752c42c0 2024-04-26T01:53:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.772+0000 7fcc33e00640 1 --2- 172.21.15.6:0/3014607831 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d752c42c0 0x562d752c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:54.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.773+0000 7fcc33e00640 1 --2- 172.21.15.6:0/3014607831 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d752c42c0 0x562d752c66e0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44a726c56129d45 server_cookie=6d50e6a77f6c1b4b in_seq=0 out_seq=0 2024-04-26T01:53:54.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.773+0000 7fcc1d600640 10 client.15999 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.796+0000 7fcc1d600640 1 -- 172.21.15.6:0/3014607831 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562d751af190 con 0x562d752c42c0 2024-04-26T01:53:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.796+0000 7fcc1d600640 10 client.15999 handle_command_reply: tid=0 2024-04-26T01:53:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.796+0000 7fcc3cf22740 1 client.15999 shutdown 2024-04-26T01:53:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.796+0000 7fcc3cf22740 1 -- 172.21.15.6:0/3014607831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d752422d0 msgr2=0x562d75253240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.796+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/3014607831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d752422d0 0x562d75253240 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcc20031a80 tx=0x7fcc20005fd0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.796+0000 7fcc3cf22740 1 -- 172.21.15.6:0/3014607831 shutdown_connections 2024-04-26T01:53:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.796+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/3014607831 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d752c42c0 0x562d752c66e0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.796+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/3014607831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d752422d0 0x562d75253240 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.797+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/3014607831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d7524cb90 0x562d752c24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.797+0000 7fcc3cf22740 1 --2- 172.21.15.6:0/3014607831 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d752419b0 0x562d75252d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.797+0000 7fcc1d600640 0 client.15999 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:53:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.797+0000 7fcc3cf22740 1 -- 172.21.15.6:0/3014607831 >> 172.21.15.6:0/3014607831 conn(0x562d7523dec0 msgr2=0x562d75243bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.797+0000 7fcc3cf22740 1 -- 172.21.15.6:0/3014607831 shutdown_connections 2024-04-26T01:53:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.797+0000 7fcc3cf22740 1 -- 172.21.15.6:0/3014607831 wait complete. 2024-04-26T01:53:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.797+0000 7fcc3cf22740 20 client.15999 trim_cache size 0 max 16384 2024-04-26T01:53:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:53:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:53:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:53:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:53:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.798+0000 7fcc38e00640 1 -- 172.21.15.6:0/2157455902 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcc180a3d00 msgr2=0x7fcc180a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.798+0000 7fcc38e00640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcc180a3d00 0x7fcc180a61c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x562d75242f50 tx=0x7fcc24098040 comp rx=0 tx=0).stop 2024-04-26T01:53:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.798+0000 7fcc38e00640 1 -- 172.21.15.6:0/2157455902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc34074320 msgr2=0x7fcc341a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.798+0000 7fcc38e00640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc34074320 0x7fcc341a0420 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400d950 tx=0x7fcc2400de10 comp rx=0 tx=0).stop 2024-04-26T01:53:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.798+0000 7fcc38e00640 1 -- 172.21.15.6:0/2157455902 shutdown_connections 2024-04-26T01:53:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.798+0000 7fcc38e00640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fcc180a3d00 0x7fcc180a61c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.798+0000 7fcc38e00640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc3410e5f0 0x7fcc3419a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.798+0000 7fcc38e00640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc34074c40 0x7fcc341a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.798+0000 7fcc38e00640 1 --2- 172.21.15.6:0/2157455902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc34074320 0x7fcc341a0420 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.798+0000 7fcc38e00640 1 -- 172.21.15.6:0/2157455902 >> 172.21.15.6:0/2157455902 conn(0x7fcc340fd880 msgr2=0x7fcc340fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.798+0000 7fcc38e00640 1 -- 172.21.15.6:0/2157455902 shutdown_connections 2024-04-26T01:53:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:53:54.798+0000 7fcc38e00640 1 -- 172.21.15.6:0/2157455902 wait complete. 2024-04-26T01:53:54.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:54 smithi006 ceph-mon[28419]: pgmap v250: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 792 KiB/s wr, 60 op/s 2024-04-26T01:53:54.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:54 smithi188 ceph-mon[32751]: pgmap v250: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 792 KiB/s wr, 60 op/s 2024-04-26T01:53:54.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:54 smithi029 ceph-mon[32775]: pgmap v250: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 792 KiB/s wr, 60 op/s 2024-04-26T01:53:55.599 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.84 MB/sec execute 259 sec latency 15522.295 ms 2024-04-26T01:53:56.599 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.82 MB/sec execute 260 sec latency 16522.382 ms 2024-04-26T01:53:56.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:56 smithi006 ceph-mon[28419]: pgmap v251: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 792 KiB/s wr, 60 op/s 2024-04-26T01:53:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:56 smithi188 ceph-mon[32751]: pgmap v251: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 792 KiB/s wr, 60 op/s 2024-04-26T01:53:56.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:56 smithi029 ceph-mon[32775]: pgmap v251: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 792 KiB/s wr, 60 op/s 2024-04-26T01:53:57.600 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.81 MB/sec execute 261 sec latency 17522.495 ms 2024-04-26T01:53:58.600 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.79 MB/sec execute 262 sec latency 18522.606 ms 2024-04-26T01:53:58.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:53:58 smithi006 ceph-mon[28419]: pgmap v252: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-04-26T01:53:58.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:53:58 smithi188 ceph-mon[32751]: pgmap v252: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-04-26T01:53:58.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:53:58 smithi029 ceph-mon[32775]: pgmap v252: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-04-26T01:53:59.600 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.78 MB/sec execute 263 sec latency 19522.731 ms 2024-04-26T01:54:00.601 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.76 MB/sec execute 264 sec latency 20522.872 ms 2024-04-26T01:54:00.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:00 smithi006 ceph-mon[28419]: pgmap v253: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 806 KiB/s wr, 42 op/s 2024-04-26T01:54:00.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:00 smithi188 ceph-mon[32751]: pgmap v253: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 806 KiB/s wr, 42 op/s 2024-04-26T01:54:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:00 smithi029 ceph-mon[32775]: pgmap v253: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 806 KiB/s wr, 42 op/s 2024-04-26T01:54:01.600 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.75 MB/sec execute 265 sec latency 21523.011 ms 2024-04-26T01:54:02.600 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.74 MB/sec execute 266 sec latency 22523.153 ms 2024-04-26T01:54:02.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:02 smithi006 ceph-mon[28419]: pgmap v254: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.3 MiB/s wr, 80 op/s 2024-04-26T01:54:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:02 smithi188 ceph-mon[32751]: pgmap v254: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.3 MiB/s wr, 80 op/s 2024-04-26T01:54:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:02 smithi029 ceph-mon[32775]: pgmap v254: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.3 MiB/s wr, 80 op/s 2024-04-26T01:54:03.600 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.72 MB/sec execute 267 sec latency 23523.254 ms 2024-04-26T01:54:03.921 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:54:04.412 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:54:04.601 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.71 MB/sec execute 268 sec latency 24523.352 ms 2024-04-26T01:54:04.766 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:04 smithi006 ceph-mon[28419]: pgmap v255: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 1018 KiB/s wr, 61 op/s 2024-04-26T01:54:04.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:04 smithi188 ceph-mon[32751]: pgmap v255: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 1018 KiB/s wr, 61 op/s 2024-04-26T01:54:04.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:04 smithi029 ceph-mon[32775]: pgmap v255: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 1018 KiB/s wr, 61 op/s 2024-04-26T01:54:05.600 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.69 MB/sec execute 269 sec latency 25523.470 ms 2024-04-26T01:54:06.601 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.68 MB/sec execute 270 sec latency 26523.561 ms 2024-04-26T01:54:06.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.609+0000 7f855d200700 1 -- 172.21.15.6:0/314190534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8558108a70 msgr2=0x7f8558108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:06.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.609+0000 7f855d200700 1 --2- 172.21.15.6:0/314190534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8558108a70 0x7f8558108e50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f85480098f0 tx=0x7f8548009c00 comp rx=0 tx=0).stop 2024-04-26T01:54:06.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.609+0000 7f855d200700 1 -- 172.21.15.6:0/314190534 shutdown_connections 2024-04-26T01:54:06.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.609+0000 7f855d200700 1 --2- 172.21.15.6:0/314190534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8558103460 0x7f855810f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:06.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.609+0000 7f855d200700 1 --2- 172.21.15.6:0/314190534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8558108a70 0x7f8558108e50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:06.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.609+0000 7f855d200700 1 --2- 172.21.15.6:0/314190534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8558102ac0 0x7f8558102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:06.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.609+0000 7f855d200700 1 -- 172.21.15.6:0/314190534 >> 172.21.15.6:0/314190534 conn(0x7f8558079b10 msgr2=0x7f8558079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:06.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.609+0000 7f855d200700 1 -- 172.21.15.6:0/314190534 shutdown_connections 2024-04-26T01:54:06.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.609+0000 7f855d200700 1 -- 172.21.15.6:0/314190534 wait complete. 2024-04-26T01:54:06.609 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:06 smithi006 ceph-mon[28419]: pgmap v256: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 1018 KiB/s wr, 61 op/s 2024-04-26T01:54:06.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.610+0000 7f855d200700 1 Processor -- start 2024-04-26T01:54:06.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.610+0000 7f855d200700 1 -- start start 2024-04-26T01:54:06.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f855d200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8558102ac0 0x7f8558074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:06.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f855d200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8558103460 0x7f8558075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:06.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f855d200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8558108a70 0x7f85580757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:06.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f855d200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8558112c70 con 0x7f8558108a70 2024-04-26T01:54:06.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f855d200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8558112af0 con 0x7f8558103460 2024-04-26T01:54:06.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f855d200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8558112df0 con 0x7f8558102ac0 2024-04-26T01:54:06.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f8555e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8558102ac0 0x7f8558074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:06.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f8555e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8558102ac0 0x7f8558074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:52740/0 (socket says 172.21.15.6:52740) 2024-04-26T01:54:06.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f8555e00700 1 -- 172.21.15.6:0/4022777280 learned_addr learned my addr 172.21.15.6:0/4022777280 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:06.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f8556800700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8558108a70 0x7f85580757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:06.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f854d400700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8558103460 0x7f8558075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:06.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f8555e00700 1 -- 172.21.15.6:0/4022777280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8558103460 msgr2=0x7f8558075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:06.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.611+0000 7f8555e00700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8558103460 0x7f8558075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.612+0000 7f8555e00700 1 -- 172.21.15.6:0/4022777280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8558108a70 msgr2=0x7f85580757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.612+0000 7f8555e00700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8558108a70 0x7f85580757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.612+0000 7f8555e00700 1 -- 172.21.15.6:0/4022777280 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85581a7750 con 0x7f8558102ac0 2024-04-26T01:54:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.612+0000 7f8555e00700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8558102ac0 0x7f8558074d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8548009230 tx=0x7f854800fb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.612+0000 7f854f400700 1 -- 172.21.15.6:0/4022777280 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548005220 con 0x7f8558102ac0 2024-04-26T01:54:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.612+0000 7f854f400700 1 -- 172.21.15.6:0/4022777280 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f854800b560 con 0x7f8558102ac0 2024-04-26T01:54:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.612+0000 7f854f400700 1 -- 172.21.15.6:0/4022777280 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548003a80 con 0x7f8558102ac0 2024-04-26T01:54:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.612+0000 7f855d200700 1 -- 172.21.15.6:0/4022777280 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85581a79e0 con 0x7f8558102ac0 2024-04-26T01:54:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.613+0000 7f855d200700 1 -- 172.21.15.6:0/4022777280 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85581a7de0 con 0x7f8558102ac0 2024-04-26T01:54:06.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.614+0000 7f855d200700 1 -- 172.21.15.6:0/4022777280 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8558069730 con 0x7f8558102ac0 2024-04-26T01:54:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.616+0000 7f854f400700 1 -- 172.21.15.6:0/4022777280 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f854800ba10 con 0x7f8558102ac0 2024-04-26T01:54:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.618+0000 7f854f400700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f853c0a3e30 0x7f853c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.618+0000 7f854f400700 1 -- 172.21.15.6:0/4022777280 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f85480d1b10 con 0x7f8558102ac0 2024-04-26T01:54:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.618+0000 7f854d400700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f853c0a3e30 0x7f853c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.619+0000 7f854d400700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f853c0a3e30 0x7f853c0a62f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f855807a1a0 tx=0x7f8540009310 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:06.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.620+0000 7f854f400700 1 -- 172.21.15.6:0/4022777280 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8548013030 con 0x7f8558102ac0 2024-04-26T01:54:06.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:06 smithi188 ceph-mon[32751]: pgmap v256: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 1018 KiB/s wr, 61 op/s 2024-04-26T01:54:06.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:06 smithi029 ceph-mon[32775]: pgmap v256: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 1018 KiB/s wr, 61 op/s 2024-04-26T01:54:06.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.960+0000 7f855d200700 1 -- 172.21.15.6:0/4022777280 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f855802e140 con 0x7f8558102ac0 2024-04-26T01:54:06.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.961+0000 7f854f400700 1 -- 172.21.15.6:0/4022777280 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11853 (secure 0 0 0) 0x7f854809e190 con 0x7f8558102ac0 2024-04-26T01:54:06.960 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:54:06.960 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":41,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:53:03.909439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:54:06.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.964+0000 7f855d200700 1 -- 172.21.15.6:0/4022777280 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f853c0a3e30 msgr2=0x7f853c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:06.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.964+0000 7f855d200700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f853c0a3e30 0x7f853c0a62f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f855807a1a0 tx=0x7f8540009310 comp rx=0 tx=0).stop 2024-04-26T01:54:06.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.964+0000 7f855d200700 1 -- 172.21.15.6:0/4022777280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8558102ac0 msgr2=0x7f8558074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:06.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.964+0000 7f855d200700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8558102ac0 0x7f8558074d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8548009230 tx=0x7f854800fb20 comp rx=0 tx=0).stop 2024-04-26T01:54:06.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.964+0000 7f855d200700 1 -- 172.21.15.6:0/4022777280 shutdown_connections 2024-04-26T01:54:06.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.964+0000 7f855d200700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f853c0a3e30 0x7f853c0a62f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:06.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.964+0000 7f855d200700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8558108a70 0x7f85580757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:06.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.964+0000 7f855d200700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8558102ac0 0x7f8558074d40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:06.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.964+0000 7f855d200700 1 --2- 172.21.15.6:0/4022777280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8558103460 0x7f8558075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:06.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.965+0000 7f855d200700 1 -- 172.21.15.6:0/4022777280 >> 172.21.15.6:0/4022777280 conn(0x7f8558079b10 msgr2=0x7f8558100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:06.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.965+0000 7f855d200700 1 -- 172.21.15.6:0/4022777280 shutdown_connections 2024-04-26T01:54:06.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:06.965+0000 7f855d200700 1 -- 172.21.15.6:0/4022777280 wait complete. 2024-04-26T01:54:06.965 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 41 2024-04-26T01:54:07.601 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.67 MB/sec execute 271 sec latency 27523.685 ms 2024-04-26T01:54:07.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:07 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/4022777280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:54:07.817 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id 1f0250cf --release --await' 2024-04-26T01:54:07.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id 1f0250cf --release --await 2024-04-26T01:54:07.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:07 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/4022777280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:54:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.941+0000 7f6871200640 1 -- 172.21.15.6:0/3128031816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f686c10c600 msgr2=0x7f686c10ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.941+0000 7f6871200640 1 --2- 172.21.15.6:0/3128031816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f686c10c600 0x7f686c10ea50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6864009ca0 tx=0x7f686402a950 comp rx=0 tx=0).stop 2024-04-26T01:54:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.941+0000 7f6871200640 1 -- 172.21.15.6:0/3128031816 shutdown_connections 2024-04-26T01:54:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.941+0000 7f6871200640 1 --2- 172.21.15.6:0/3128031816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f686c10c600 0x7f686c10ea50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.941+0000 7f6871200640 1 --2- 172.21.15.6:0/3128031816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f686c103b50 0x7f686c10c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.941+0000 7f6871200640 1 --2- 172.21.15.6:0/3128031816 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f686c103230 0x7f686c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.941+0000 7f6871200640 1 -- 172.21.15.6:0/3128031816 >> 172.21.15.6:0/3128031816 conn(0x7f686c0fc970 msgr2=0x7f686c0fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.941+0000 7f6871200640 1 -- 172.21.15.6:0/3128031816 shutdown_connections 2024-04-26T01:54:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.941+0000 7f6871200640 1 -- 172.21.15.6:0/3128031816 wait complete. 2024-04-26T01:54:07.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:07 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/4022777280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:54:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.942+0000 7f6871200640 1 Processor -- start 2024-04-26T01:54:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.942+0000 7f6871200640 1 -- start start 2024-04-26T01:54:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.942+0000 7f6871200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f686c103230 0x7f686c1a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.942+0000 7f6871200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f686c103b50 0x7f686c1a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.942+0000 7f6871200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f686c10c600 0x7f686c19a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.942+0000 7f6871200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f686c112280 con 0x7f686c103b50 2024-04-26T01:54:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.942+0000 7f6871200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f686c112100 con 0x7f686c10c600 2024-04-26T01:54:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.942+0000 7f6871200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f686c112400 con 0x7f686c103230 2024-04-26T01:54:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.943+0000 7f6869400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f686c103b50 0x7f686c1a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.943+0000 7f6869400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f686c103b50 0x7f686c1a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58624/0 (socket says 172.21.15.6:58624) 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.943+0000 7f6869400640 1 -- 172.21.15.6:0/1259542932 learned_addr learned my addr 172.21.15.6:0/1259542932 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.943+0000 7f686a800640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f686c10c600 0x7f686c19a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.943+0000 7f6869e00640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f686c103230 0x7f686c1a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.943+0000 7f6869400640 1 -- 172.21.15.6:0/1259542932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f686c103230 msgr2=0x7f686c1a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.943+0000 7f6869400640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f686c103230 0x7f686c1a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.943+0000 7f6869400640 1 -- 172.21.15.6:0/1259542932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f686c10c600 msgr2=0x7f686c19a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.943+0000 7f6869400640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f686c10c600 0x7f686c19a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.943+0000 7f6869400640 1 -- 172.21.15.6:0/1259542932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f686c19b060 con 0x7f686c103b50 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.943+0000 7f6869e00640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f686c103230 0x7f686c1a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.944+0000 7f6869400640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f686c103b50 0x7f686c1a0b80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f685800ea60 tx=0x7f685800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.944+0000 7f6862a00640 1 -- 172.21.15.6:0/1259542932 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685800cde0 con 0x7f686c103b50 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.944+0000 7f6862a00640 1 -- 172.21.15.6:0/1259542932 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68580035f0 con 0x7f686c103b50 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.944+0000 7f6871200640 1 -- 172.21.15.6:0/1259542932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f686c19b350 con 0x7f686c103b50 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.944+0000 7f6862a00640 1 -- 172.21.15.6:0/1259542932 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6858010690 con 0x7f686c103b50 2024-04-26T01:54:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.944+0000 7f6871200640 1 -- 172.21.15.6:0/1259542932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f686c1a7500 con 0x7f686c103b50 2024-04-26T01:54:07.946 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.947+0000 7f687405c740 1 Processor -- start 2024-04-26T01:54:07.946 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.947+0000 7f687405c740 1 -- start start 2024-04-26T01:54:07.947 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f687405c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c371c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f687405c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c372140 0x55556c37c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f687405c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 0x55556c37edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f687405c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55556c2d7900 con 0x55556c372140 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f687405c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55556c27def0 con 0x55556c371820 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f687405c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55556c2d38c0 con 0x55556c37ca00 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f6869e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c372140 0x55556c37c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f6869e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c372140 0x55556c37c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58626/0 (socket says 172.21.15.6:58626) 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f6869e00640 1 -- 172.21.15.6:0/2243600999 learned_addr learned my addr 172.21.15.6:0/2243600999 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f686a800640 1 --2- 172.21.15.6:0/2243600999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c371c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f6869e00640 1 -- 172.21.15.6:0/2243600999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 msgr2=0x55556c37edf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f6869e00640 1 --2- 172.21.15.6:0/2243600999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 0x55556c37edf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f6869e00640 1 -- 172.21.15.6:0/2243600999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 msgr2=0x55556c371c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f6869e00640 1 --2- 172.21.15.6:0/2243600999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c371c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.948+0000 7f6869e00640 1 -- 172.21.15.6:0/2243600999 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55556bfea210 con 0x55556c372140 2024-04-26T01:54:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.949+0000 7f6869e00640 1 --2- 172.21.15.6:0/2243600999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c372140 0x55556c37c4c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f685c0114f0 tx=0x7f685c0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e61a4e97f9deb48 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.949+0000 7f6862a00640 1 -- 172.21.15.6:0/1259542932 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f68580037a0 con 0x7f686c103b50 2024-04-26T01:54:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.949+0000 7f6857400640 1 -- 172.21.15.6:0/2243600999 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c004020 con 0x55556c372140 2024-04-26T01:54:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.949+0000 7f6857400640 1 -- 172.21.15.6:0/2243600999 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f685c0041c0 con 0x55556c372140 2024-04-26T01:54:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.949+0000 7f6857400640 1 -- 172.21.15.6:0/2243600999 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c00f8f0 con 0x55556c372140 2024-04-26T01:54:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.950+0000 7f687405c740 1 -- 172.21.15.6:0/2243600999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c372140 msgr2=0x55556c37c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.950+0000 7f687405c740 1 --2- 172.21.15.6:0/2243600999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c372140 0x55556c37c4c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f685c0114f0 tx=0x7f685c0119b0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.950+0000 7f6862a00640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68500a3d90 0x7f68500a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.950+0000 7f687405c740 1 -- 172.21.15.6:0/2243600999 shutdown_connections 2024-04-26T01:54:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.950+0000 7f687405c740 1 --2- 172.21.15.6:0/2243600999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c372140 0x55556c37c4c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.950+0000 7f687405c740 1 --2- 172.21.15.6:0/2243600999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 0x55556c37edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.950+0000 7f687405c740 1 --2- 172.21.15.6:0/2243600999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c371c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.950+0000 7f687405c740 1 -- 172.21.15.6:0/2243600999 >> 172.21.15.6:0/2243600999 conn(0x55556c36dd30 msgr2=0x55556c36e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.951+0000 7f6862a00640 1 -- 172.21.15.6:0/1259542932 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f68580d09e0 con 0x7f686c103b50 2024-04-26T01:54:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.951+0000 7f687405c740 1 -- 172.21.15.6:0/2243600999 shutdown_connections 2024-04-26T01:54:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.951+0000 7f686a800640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68500a3d90 0x7f68500a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.951+0000 7f687405c740 1 -- 172.21.15.6:0/2243600999 wait complete. 2024-04-26T01:54:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.951+0000 7f687405c740 1 -- start start 2024-04-26T01:54:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.951+0000 7f686a800640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68500a3d90 0x7f68500a6250 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6864006010 tx=0x7f6864005d80 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.952+0000 7f687405c740 1 Processor -- start 2024-04-26T01:54:07.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.952+0000 7f687405c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c379960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.952+0000 7f687405c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c372140 0x55556c379ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.952+0000 7f687405c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 0x55556c37a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.952+0000 7f687405c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55556c37f630 con 0x55556c372140 2024-04-26T01:54:07.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.952+0000 7f687405c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55556c291310 con 0x55556c371820 2024-04-26T01:54:07.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.952+0000 7f687405c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55556c37f7b0 con 0x55556c37ca00 2024-04-26T01:54:07.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.952+0000 7f687405c740 10 client.0 _ll_register_callbacks cb 0x55556c25fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:07.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.953+0000 7f6869400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c372140 0x55556c379ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.953+0000 7f6869e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c379960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.953+0000 7f686a800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 0x55556c37a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.953+0000 7f6869e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c379960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34394/0 (socket says 172.21.15.6:34394) 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.953+0000 7f6869e00640 1 -- 172.21.15.6:0/890851118 learned_addr learned my addr 172.21.15.6:0/890851118 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.953+0000 7f6869e00640 1 -- 172.21.15.6:0/890851118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 msgr2=0x55556c37a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.953+0000 7f6869e00640 1 --2- 172.21.15.6:0/890851118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 0x55556c37a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.953+0000 7f6869e00640 1 -- 172.21.15.6:0/890851118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c372140 msgr2=0x55556c379ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.953+0000 7f6869e00640 1 --2- 172.21.15.6:0/890851118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c372140 0x55556c379ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.953+0000 7f6869e00640 1 -- 172.21.15.6:0/890851118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55556c0b6620 con 0x55556c371820 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.954+0000 7f6869e00640 1 --2- 172.21.15.6:0/890851118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c379960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f685c01b040 tx=0x7f685c005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acb99884e6b4609 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.954+0000 7f684be00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.954+0000 7f684be00640 1 -- 172.21.15.6:0/890851118 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c028020 con 0x55556c371820 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.954+0000 7f687405c740 1 -- 172.21.15.6:0/890851118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f685c0041c0 con 0x55556c371820 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.954+0000 7f684be00640 1 -- 172.21.15.6:0/890851118 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f685c016040 con 0x55556c371820 2024-04-26T01:54:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.954+0000 7f684be00640 1 -- 172.21.15.6:0/890851118 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f685c033050 con 0x55556c371820 2024-04-26T01:54:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.954+0000 7f687405c740 10 client.26153 fetch_fsmap learned FSMap version 41 2024-04-26T01:54:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.954+0000 7f687405c740 1 -- 172.21.15.6:0/890851118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55556c38b850 con 0x55556c371820 2024-04-26T01:54:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.955+0000 7f684be00640 1 -- 172.21.15.6:0/890851118 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f685c0041c0 con 0x55556c371820 2024-04-26T01:54:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.955+0000 7f684be00640 1 -- 172.21.15.6:0/890851118 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f685c004950 con 0x55556c371820 2024-04-26T01:54:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.955+0000 7f687405c740 10 client.26153 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:54:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.955+0000 7f687405c740 10 client.26153 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.955+0000 7f687405c740 20 client.26153 populate_metadata read hostname 'smithi006' 2024-04-26T01:54:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.955+0000 7f687405c740 1 --2- 172.21.15.6:0/890851118 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55556c38ccc0 0x55556c38f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.955+0000 7f687405c740 4 client.26153 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.955+0000 7f687405c740 1 -- 172.21.15.6:0/890851118 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55556c2eb8b0 con 0x55556c38ccc0 2024-04-26T01:54:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.956+0000 7f6869400640 1 --2- 172.21.15.6:0/890851118 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55556c38ccc0 0x55556c38f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.957+0000 7f6869400640 1 --2- 172.21.15.6:0/890851118 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55556c38ccc0 0x55556c38f0b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2d5e08c03be4748 server_cookie=20c0188ec418a995 in_seq=0 out_seq=0 2024-04-26T01:54:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.957+0000 7f684be00640 10 client.26153 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.958+0000 7f684be00640 1 -- 172.21.15.6:0/890851118 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55556c2eb8b0 con 0x55556c38ccc0 2024-04-26T01:54:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.958+0000 7f684be00640 10 client.26153 handle_command_reply: tid=0 2024-04-26T01:54:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.958+0000 7f687405c740 1 client.26153 shutdown 2024-04-26T01:54:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.959+0000 7f687405c740 1 -- 172.21.15.6:0/890851118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 msgr2=0x55556c379960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.959+0000 7f687405c740 1 --2- 172.21.15.6:0/890851118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c379960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f685c01b040 tx=0x7f685c005770 comp rx=0 tx=0).stop 2024-04-26T01:54:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.959+0000 7f687405c740 1 -- 172.21.15.6:0/890851118 shutdown_connections 2024-04-26T01:54:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.959+0000 7f687405c740 1 --2- 172.21.15.6:0/890851118 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55556c38ccc0 0x55556c38f0b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.959+0000 7f687405c740 1 --2- 172.21.15.6:0/890851118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c372140 0x55556c379ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.959+0000 7f687405c740 1 --2- 172.21.15.6:0/890851118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 0x55556c37a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.959+0000 7f684be00640 0 client.26153 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.959+0000 7f687405c740 1 --2- 172.21.15.6:0/890851118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c379960 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.959+0000 7f687405c740 1 -- 172.21.15.6:0/890851118 >> 172.21.15.6:0/890851118 conn(0x55556c36dd30 msgr2=0x55556c373ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.959+0000 7f687405c740 1 -- 172.21.15.6:0/890851118 shutdown_connections 2024-04-26T01:54:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.959+0000 7f687405c740 1 -- 172.21.15.6:0/890851118 wait complete. 2024-04-26T01:54:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.959+0000 7f687405c740 20 client.26153 trim_cache size 0 max 16384 2024-04-26T01:54:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.993+0000 7f687405c740 1 Processor -- start 2024-04-26T01:54:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.993+0000 7f687405c740 1 -- start start 2024-04-26T01:54:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.993+0000 7f687405c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c371820 0x55556c3884f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.993+0000 7f687405c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c372140 0x55556c3741a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.993+0000 7f687405c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 0x55556c3947a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.993+0000 7f687405c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f686c1b2870 con 0x55556c371820 2024-04-26T01:54:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.993+0000 7f687405c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f686c1b00e0 con 0x55556c372140 2024-04-26T01:54:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.993+0000 7f687405c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f686c1adf50 con 0x55556c37ca00 2024-04-26T01:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.994+0000 7f6869e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c371820 0x55556c3884f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.994+0000 7f6869e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c371820 0x55556c3884f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58638/0 (socket says 172.21.15.6:58638) 2024-04-26T01:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.994+0000 7f6869e00640 1 -- 172.21.15.6:0/3816893629 learned_addr learned my addr 172.21.15.6:0/3816893629 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.994+0000 7f686a800640 1 --2- 172.21.15.6:0/3816893629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 0x55556c3947a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.994+0000 7f6869400640 1 --2- 172.21.15.6:0/3816893629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c372140 0x55556c3741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.994+0000 7f6869400640 1 -- 172.21.15.6:0/3816893629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 msgr2=0x55556c3947a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.994+0000 7f6869400640 1 --2- 172.21.15.6:0/3816893629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 0x55556c3947a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.994+0000 7f6869400640 1 -- 172.21.15.6:0/3816893629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c371820 msgr2=0x55556c3884f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.994+0000 7f6869400640 1 --2- 172.21.15.6:0/3816893629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c371820 0x55556c3884f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.994+0000 7f6869400640 1 -- 172.21.15.6:0/3816893629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55556c377ed0 con 0x55556c372140 2024-04-26T01:54:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.994+0000 7f6869e00640 1 --2- 172.21.15.6:0/3816893629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c371820 0x55556c3884f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.995+0000 7f6869400640 1 --2- 172.21.15.6:0/3816893629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c372140 0x55556c3741a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f68580ce0e0 tx=0x7f68580bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7674781dd76017d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.995+0000 7f6856a00640 1 -- 172.21.15.6:0/3816893629 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68580a0ce0 con 0x55556c372140 2024-04-26T01:54:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.995+0000 7f6856a00640 1 -- 172.21.15.6:0/3816893629 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68580a55a0 con 0x55556c372140 2024-04-26T01:54:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.995+0000 7f6856a00640 1 -- 172.21.15.6:0/3816893629 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68580a5110 con 0x55556c372140 2024-04-26T01:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.996+0000 7f687405c740 1 -- 172.21.15.6:0/3816893629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c372140 msgr2=0x55556c3741a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.996+0000 7f687405c740 1 --2- 172.21.15.6:0/3816893629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c372140 0x55556c3741a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f68580ce0e0 tx=0x7f68580bd040 comp rx=0 tx=0).stop 2024-04-26T01:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.996+0000 7f687405c740 1 -- 172.21.15.6:0/3816893629 shutdown_connections 2024-04-26T01:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.996+0000 7f687405c740 1 --2- 172.21.15.6:0/3816893629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c37ca00 0x55556c3947a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.996+0000 7f687405c740 1 --2- 172.21.15.6:0/3816893629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c372140 0x55556c3741a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.996+0000 7f687405c740 1 --2- 172.21.15.6:0/3816893629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c371820 0x55556c3884f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.996+0000 7f687405c740 1 -- 172.21.15.6:0/3816893629 >> 172.21.15.6:0/3816893629 conn(0x55556c36dd30 msgr2=0x55556c36ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.996+0000 7f687405c740 1 -- 172.21.15.6:0/3816893629 shutdown_connections 2024-04-26T01:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.996+0000 7f687405c740 1 -- 172.21.15.6:0/3816893629 wait complete. 2024-04-26T01:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.997+0000 7f687405c740 1 -- start start 2024-04-26T01:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.997+0000 7f687405c740 1 Processor -- start 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.997+0000 7f687405c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c382370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.997+0000 7f687405c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c372140 0x55556c3828b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.997+0000 7f687405c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c37ca00 0x55556c382df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.997+0000 7f687405c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f686c1ae5a0 con 0x55556c37ca00 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.997+0000 7f687405c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f686c1afa10 con 0x55556c371820 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.997+0000 7f687405c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f686c1aec70 con 0x55556c372140 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.997+0000 7f687405c740 10 client.0 _ll_register_callbacks cb 0x7f685c005400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f686a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c37ca00 0x55556c382df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f6869400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c372140 0x55556c3828b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f6869e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c382370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f6869400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c372140 0x55556c3828b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:60932/0 (socket says 172.21.15.6:60932) 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f6869400640 1 -- 172.21.15.6:0/601020649 learned_addr learned my addr 172.21.15.6:0/601020649 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f6869400640 1 -- 172.21.15.6:0/601020649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 msgr2=0x55556c382370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f6869400640 1 --2- 172.21.15.6:0/601020649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c382370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f6869400640 1 -- 172.21.15.6:0/601020649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c37ca00 msgr2=0x55556c382df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f6869400640 1 --2- 172.21.15.6:0/601020649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c37ca00 0x55556c382df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f6869400640 1 -- 172.21.15.6:0/601020649 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55556c3f2020 con 0x55556c372140 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f6869400640 1 --2- 172.21.15.6:0/601020649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c372140 0x55556c3828b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6858004610 tx=0x7f6858003e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0f21c308a6cc0ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f684be00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.998+0000 7f684be00640 1 -- 172.21.15.6:0/601020649 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68580ce360 con 0x55556c372140 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f687405c740 1 -- 172.21.15.6:0/601020649 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68580a55a0 con 0x55556c372140 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f684be00640 1 -- 172.21.15.6:0/601020649 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8600+0+0 (secure 0 0 0) 0x7f6858098fa0 con 0x55556c372140 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f684be00640 1 -- 172.21.15.6:0/601020649 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f68580a5110 con 0x55556c372140 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f687405c740 10 client.26203 fetch_fsmap learned FSMap version 41 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f687405c740 1 -- 172.21.15.6:0/601020649 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55556c3f3250 con 0x55556c372140 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f684be00640 1 -- 172.21.15.6:0/601020649 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68580a55a0 con 0x55556c372140 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f684be00640 1 -- 172.21.15.6:0/601020649 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f68580bbaf0 con 0x55556c372140 2024-04-26T01:54:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f687405c740 10 client.26203 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:54:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f687405c740 10 client.26203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f687405c740 20 client.26203 populate_metadata read hostname 'smithi006' 2024-04-26T01:54:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f687405c740 1 --2- 172.21.15.6:0/601020649 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55556c3f39d0 0x55556c3f5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f687405c740 4 client.26203 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:07.999+0000 7f687405c740 1 -- 172.21.15.6:0/601020649 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55556c2e5320 con 0x55556c3f39d0 2024-04-26T01:54:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.000+0000 7f6869e00640 1 --2- 172.21.15.6:0/601020649 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55556c3f39d0 0x55556c3f5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.000+0000 7f6869e00640 1 --2- 172.21.15.6:0/601020649 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55556c3f39d0 0x55556c3f5df0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53600be03fd8ac84 server_cookie=8aa8cce775a80b31 in_seq=0 out_seq=0 2024-04-26T01:54:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.000+0000 7f684be00640 10 client.26203 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f684be00640 1 -- 172.21.15.6:0/601020649 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55556c2e5320 con 0x55556c3f39d0 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f684be00640 10 client.26203 handle_command_reply: tid=0 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f687405c740 1 client.26203 shutdown 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f687405c740 1 -- 172.21.15.6:0/601020649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c372140 msgr2=0x55556c3828b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f687405c740 1 --2- 172.21.15.6:0/601020649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c372140 0x55556c3828b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6858004610 tx=0x7f6858003e90 comp rx=0 tx=0).stop 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f687405c740 1 -- 172.21.15.6:0/601020649 shutdown_connections 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f687405c740 1 --2- 172.21.15.6:0/601020649 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55556c3f39d0 0x55556c3f5df0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f687405c740 1 --2- 172.21.15.6:0/601020649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55556c37ca00 0x55556c382df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f687405c740 1 --2- 172.21.15.6:0/601020649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55556c372140 0x55556c3828b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f687405c740 1 --2- 172.21.15.6:0/601020649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55556c371820 0x55556c382370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f684be00640 0 client.26203 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f687405c740 1 -- 172.21.15.6:0/601020649 >> 172.21.15.6:0/601020649 conn(0x55556c36dd30 msgr2=0x55556c37b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f687405c740 1 -- 172.21.15.6:0/601020649 shutdown_connections 2024-04-26T01:54:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f687405c740 1 -- 172.21.15.6:0/601020649 wait complete. 2024-04-26T01:54:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.041+0000 7f687405c740 20 client.26203 trim_cache size 0 max 16384 2024-04-26T01:54:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:54:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:54:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:54:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:54:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.042+0000 7f6871200640 1 -- 172.21.15.6:0/1259542932 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68500a3d90 msgr2=0x7f68500a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.042+0000 7f6871200640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68500a3d90 0x7f68500a6250 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6864006010 tx=0x7f6864005d80 comp rx=0 tx=0).stop 2024-04-26T01:54:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.042+0000 7f6871200640 1 -- 172.21.15.6:0/1259542932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f686c103b50 msgr2=0x7f686c1a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.042+0000 7f6871200640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f686c103b50 0x7f686c1a0b80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f685800ea60 tx=0x7f685800ef20 comp rx=0 tx=0).stop 2024-04-26T01:54:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.042+0000 7f6871200640 1 -- 172.21.15.6:0/1259542932 shutdown_connections 2024-04-26T01:54:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.042+0000 7f6871200640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68500a3d90 0x7f68500a6250 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.042+0000 7f6871200640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f686c103b50 0x7f686c1a0b80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.042+0000 7f6871200640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f686c10c600 0x7f686c19a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.042+0000 7f6871200640 1 --2- 172.21.15.6:0/1259542932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f686c103230 0x7f686c1a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.042+0000 7f6871200640 1 -- 172.21.15.6:0/1259542932 >> 172.21.15.6:0/1259542932 conn(0x7f686c0fc970 msgr2=0x7f686c100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.042+0000 7f6871200640 1 -- 172.21.15.6:0/1259542932 shutdown_connections 2024-04-26T01:54:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:08.042+0000 7f6871200640 1 -- 172.21.15.6:0/1259542932 wait complete. 2024-04-26T01:54:08.276 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:54:08.601 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.65 MB/sec execute 272 sec latency 28523.778 ms 2024-04-26T01:54:08.766 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:08 smithi006 ceph-mon[28419]: pgmap v257: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 1.3 MiB/s wr, 79 op/s 2024-04-26T01:54:08.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:08 smithi188 ceph-mon[32751]: pgmap v257: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 1.3 MiB/s wr, 79 op/s 2024-04-26T01:54:08.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:08 smithi029 ceph-mon[32775]: pgmap v257: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 1.3 MiB/s wr, 79 op/s 2024-04-26T01:54:09.601 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.64 MB/sec execute 273 sec latency 29524.306 ms 2024-04-26T01:54:10.602 INFO:tasks.workunit.client.0.smithi006.stdout: 1 276953 3.63 MB/sec execute 274 sec latency 30524.431 ms 2024-04-26T01:54:10.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.802+0000 7f7c24400700 1 -- 172.21.15.6:0/144702067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c1c10e3b0 msgr2=0x7f7c1c115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.802+0000 7f7c24400700 1 --2- 172.21.15.6:0/144702067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c1c10e3b0 0x7f7c1c115f80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400b0a0 tx=0x7f7c1400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.802+0000 7f7c24400700 1 -- 172.21.15.6:0/144702067 shutdown_connections 2024-04-26T01:54:10.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.802+0000 7f7c24400700 1 --2- 172.21.15.6:0/144702067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c1c10e3b0 0x7f7c1c115f80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.802+0000 7f7c24400700 1 --2- 172.21.15.6:0/144702067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c1c1092d0 0x7f7c1c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.802+0000 7f7c24400700 1 --2- 172.21.15.6:0/144702067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c1c109c80 0x7f7c1c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.802+0000 7f7c24400700 1 -- 172.21.15.6:0/144702067 >> 172.21.15.6:0/144702067 conn(0x7f7c1c076210 msgr2=0x7f7c1c107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:10.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.802+0000 7f7c24400700 1 -- 172.21.15.6:0/144702067 shutdown_connections 2024-04-26T01:54:10.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.802+0000 7f7c24400700 1 -- 172.21.15.6:0/144702067 wait complete. 2024-04-26T01:54:10.802 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:10 smithi006 ceph-mon[28419]: pgmap v258: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 821 KiB/s wr, 55 op/s 2024-04-26T01:54:10.802 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:10 smithi006 ceph-mon[28419]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.804+0000 7f7c24400700 1 Processor -- start 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.804+0000 7f7c24400700 1 -- start start 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.805+0000 7f7c24400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c1c1092d0 0x7f7c1c19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.805+0000 7f7c24400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c1c109c80 0x7f7c1c19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.805+0000 7f7c24400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c1c10e3b0 0x7f7c1c1a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.805+0000 7f7c24400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c119270 con 0x7f7c1c109c80 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.805+0000 7f7c24400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1190f0 con 0x7f7c1c10e3b0 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.805+0000 7f7c24400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1193f0 con 0x7f7c1c1092d0 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.805+0000 7f7c21000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c1c109c80 0x7f7c1c19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.805+0000 7f7c21a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c1c1092d0 0x7f7c1c19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.805+0000 7f7c21000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c1c109c80 0x7f7c1c19fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58664/0 (socket says 172.21.15.6:58664) 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.805+0000 7f7c21a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c1c1092d0 0x7f7c1c19f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:60956/0 (socket says 172.21.15.6:60956) 2024-04-26T01:54:10.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.805+0000 7f7c21000700 1 -- 172.21.15.6:0/862792506 learned_addr learned my addr 172.21.15.6:0/862792506 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:10.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.805+0000 7f7c22400700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c1c10e3b0 0x7f7c1c1a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c21a00700 1 -- 172.21.15.6:0/862792506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c1c10e3b0 msgr2=0x7f7c1c1a3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c21a00700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c1c10e3b0 0x7f7c1c1a3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c21a00700 1 -- 172.21.15.6:0/862792506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c1c109c80 msgr2=0x7f7c1c19fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c21a00700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c1c109c80 0x7f7c1c19fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c21a00700 1 -- 172.21.15.6:0/862792506 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1c1a4700 con 0x7f7c1c1092d0 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c21000700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c1c109c80 0x7f7c1c19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c22400700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c1c10e3b0 0x7f7c1c1a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c21a00700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c1c1092d0 0x7f7c1c19f740 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000e9f0 tx=0x7f7c1000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c1a000700 1 -- 172.21.15.6:0/862792506 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c10004220 con 0x7f7c1c1092d0 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c1a000700 1 -- 172.21.15.6:0/862792506 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c100043c0 con 0x7f7c1c1092d0 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c24400700 1 -- 172.21.15.6:0/862792506 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1c1a49f0 con 0x7f7c1c1092d0 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c1a000700 1 -- 172.21.15.6:0/862792506 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c10005020 con 0x7f7c1c1092d0 2024-04-26T01:54:10.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.806+0000 7f7c24400700 1 -- 172.21.15.6:0/862792506 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1c1ac270 con 0x7f7c1c1092d0 2024-04-26T01:54:10.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.810+0000 7f7c1a000700 1 -- 172.21.15.6:0/862792506 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f7c10004750 con 0x7f7c1c1092d0 2024-04-26T01:54:10.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.811+0000 7f7c2eddbb80 1 Processor -- start 2024-04-26T01:54:10.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.811+0000 7f7c1a000700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7c040a3e30 0x7f7c040a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.811+0000 7f7c2eddbb80 1 -- start start 2024-04-26T01:54:10.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.811+0000 7f7c1a000700 1 -- 172.21.15.6:0/862792506 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8644+0+0 (secure 0 0 0) 0x7f7c100d07a0 con 0x7f7c1c1092d0 2024-04-26T01:54:10.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.811+0000 7f7c22400700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7c040a3e30 0x7f7c040a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.812+0000 7f7c2eddbb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfcfe6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.812+0000 7f7c2eddbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 0x564cfcff1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.812+0000 7f7c2eddbb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcff44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.812+0000 7f7c2eddbb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564cfcf39f90 con 0x564cfcfe6920 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.812+0000 7f7c2eddbb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564cfcee0830 con 0x564cfcff20f0 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.812+0000 7f7c2eddbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564cfcf35f50 con 0x564cfcfe7240 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.812+0000 7f7c21000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 0x564cfcff1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.812+0000 7f7c21a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfcfe6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.812+0000 7f7c21000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 0x564cfcff1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:60964/0 (socket says 172.21.15.6:60964) 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.812+0000 7f7c21000700 1 -- 172.21.15.6:0/4202953280 learned_addr learned my addr 172.21.15.6:0/4202953280 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.812+0000 7f7c22400700 1 --2- 172.21.15.6:0/4202953280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcff44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.813+0000 7f7c21000700 1 -- 172.21.15.6:0/4202953280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 msgr2=0x564cfcff44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.813+0000 7f7c21000700 1 --2- 172.21.15.6:0/4202953280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcff44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.813+0000 7f7c21000700 1 -- 172.21.15.6:0/4202953280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 msgr2=0x564cfcfe6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.813+0000 7f7c21000700 1 --2- 172.21.15.6:0/4202953280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfcfe6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.813+0000 7f7c21000700 1 -- 172.21.15.6:0/4202953280 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cfced6780 con 0x564cfcfe7240 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.813+0000 7f7c22400700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7c040a3e30 0x7f7c040a62f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7c14009fd0 tx=0x7f7c14009db0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.813+0000 7f7c21a00700 1 --2- 172.21.15.6:0/4202953280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfcfe6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:10.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.813+0000 7f7c21000700 1 --2- 172.21.15.6:0/4202953280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 0x564cfcff1bb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7c080115f0 tx=0x7f7c08009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f66f45d5acea80c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.813+0000 7f7c0ea00700 1 -- 172.21.15.6:0/4202953280 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c08004220 con 0x564cfcfe7240 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.814+0000 7f7c0ea00700 1 -- 172.21.15.6:0/4202953280 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c080043c0 con 0x564cfcfe7240 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.814+0000 7f7c0ea00700 1 -- 172.21.15.6:0/4202953280 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c08019990 con 0x564cfcfe7240 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.814+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/4202953280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 msgr2=0x564cfcff1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.814+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/4202953280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 0x564cfcff1bb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7c080115f0 tx=0x7f7c08009f90 comp rx=0 tx=0).stop 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.814+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/4202953280 shutdown_connections 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.814+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/4202953280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcff44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.814+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/4202953280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfcfe6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.814+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/4202953280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 0x564cfcff1bb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.814+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/4202953280 >> 172.21.15.6:0/4202953280 conn(0x564cfcf156a0 msgr2=0x564cfcfe2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.815+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/4202953280 shutdown_connections 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.815+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/4202953280 wait complete. 2024-04-26T01:54:10.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.815+0000 7f7c2eddbb80 1 -- start start 2024-04-26T01:54:10.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.816+0000 7f7c2eddbb80 1 Processor -- start 2024-04-26T01:54:10.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.816+0000 7f7c2eddbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe6920 0x564cfcff1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.816+0000 7f7c2eddbb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe7240 0x564cfcfeeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.816+0000 7f7c2eddbb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcfef070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.816+0000 7f7c2eddbb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564cfcff4c10 con 0x564cfcfe7240 2024-04-26T01:54:10.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.816+0000 7f7c2eddbb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564cfcef3960 con 0x564cfcff20f0 2024-04-26T01:54:10.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.816+0000 7f7c2eddbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564cfcff4d90 con 0x564cfcfe6920 2024-04-26T01:54:10.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.816+0000 7f7c2eddbb80 10 client.0 _ll_register_callbacks cb 0x564cfced2820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:10.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.817+0000 7f7c21000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe7240 0x564cfcfeeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.817+0000 7f7c21a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe6920 0x564cfcff1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.817+0000 7f7c21a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe6920 0x564cfcff1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:60968/0 (socket says 172.21.15.6:60968) 2024-04-26T01:54:10.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.817+0000 7f7c21000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe7240 0x564cfcfeeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58688/0 (socket says 172.21.15.6:58688) 2024-04-26T01:54:10.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.817+0000 7f7c21a00700 1 -- 172.21.15.6:0/82153568 learned_addr learned my addr 172.21.15.6:0/82153568 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:10.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.817+0000 7f7c22400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcfef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.817+0000 7f7c21000700 1 -- 172.21.15.6:0/82153568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe6920 msgr2=0x564cfcff1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.817+0000 7f7c21000700 1 --2- 172.21.15.6:0/82153568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe6920 0x564cfcff1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.817+0000 7f7c21000700 1 -- 172.21.15.6:0/82153568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 msgr2=0x564cfcfef070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.817+0000 7f7c21000700 1 --2- 172.21.15.6:0/82153568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcfef070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.817+0000 7f7c21000700 1 -- 172.21.15.6:0/82153568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cfce879d0 con 0x564cfcfe7240 2024-04-26T01:54:10.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.817+0000 7f7c22400700 1 --2- 172.21.15.6:0/82153568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcfef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:10.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.818+0000 7f7c21000700 1 --2- 172.21.15.6:0/82153568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe7240 0x564cfcfeeb30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7c08011be0 tx=0x7f7c08005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a386082d5af018ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:10.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.818+0000 7f7c03400700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:54:10.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.818+0000 7f7c03400700 1 -- 172.21.15.6:0/82153568 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0801d040 con 0x564cfcfe7240 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.818+0000 7f7c03400700 1 -- 172.21.15.6:0/82153568 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8644+0+0 (secure 0 0 0) 0x7f7c08004750 con 0x564cfcfe7240 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.818+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/82153568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c080043c0 con 0x564cfcfe7240 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.819+0000 7f7c03400700 1 -- 172.21.15.6:0/82153568 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f7c08031050 con 0x564cfcfe7240 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.819+0000 7f7c2eddbb80 10 client.16035 fetch_fsmap learned FSMap version 41 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.819+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/82153568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x564cfced0bf0 con 0x564cfcfe7240 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.819+0000 7f7c03400700 1 -- 172.21.15.6:0/82153568 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c080043c0 con 0x564cfcfe7240 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.819+0000 7f7c03400700 1 -- 172.21.15.6:0/82153568 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f7c08003a90 con 0x564cfcfe7240 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.819+0000 7f7c2eddbb80 10 client.16035 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.819+0000 7f7c2eddbb80 10 client.16035 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.819+0000 7f7c2eddbb80 20 client.16035 populate_metadata read hostname 'smithi006' 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.819+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/82153568 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564cfd0018f0 0x564cfd003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.819+0000 7f7c2eddbb80 4 client.16035 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.819+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/82153568 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cfcf47780 con 0x564cfd0018f0 2024-04-26T01:54:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.820+0000 7f7c21a00700 1 --2- 172.21.15.6:0/82153568 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564cfd0018f0 0x564cfd003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.820+0000 7f7c21a00700 1 --2- 172.21.15.6:0/82153568 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564cfd0018f0 0x564cfd003ce0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9416072b140b76df server_cookie=7786297333ffd8bc in_seq=0 out_seq=0 2024-04-26T01:54:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.820+0000 7f7c03400700 10 client.16035 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.821+0000 7f7c03400700 1 -- 172.21.15.6:0/82153568 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564cfcf47780 con 0x564cfd0018f0 2024-04-26T01:54:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.821+0000 7f7c03400700 10 client.16035 handle_command_reply: tid=0 2024-04-26T01:54:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.821+0000 7f7c2eddbb80 1 client.16035 shutdown 2024-04-26T01:54:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.821+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/82153568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe7240 msgr2=0x564cfcfeeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.821+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/82153568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe7240 0x564cfcfeeb30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7c08011be0 tx=0x7f7c08005270 comp rx=0 tx=0).stop 2024-04-26T01:54:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.822+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/82153568 shutdown_connections 2024-04-26T01:54:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.822+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/82153568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcfef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.822+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/82153568 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564cfd0018f0 0x564cfd003ce0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.822+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/82153568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe6920 0x564cfcff1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.822+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/82153568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe7240 0x564cfcfeeb30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.822+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/82153568 >> 172.21.15.6:0/82153568 conn(0x564cfcf156a0 msgr2=0x564cfcfe29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:10.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.822+0000 7f7c03400700 0 client.16035 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:10.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.822+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/82153568 shutdown_connections 2024-04-26T01:54:10.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.822+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/82153568 wait complete. 2024-04-26T01:54:10.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.822+0000 7f7c2eddbb80 20 client.16035 trim_cache size 0 max 16384 2024-04-26T01:54:10.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.843+0000 7f7c2eddbb80 1 Processor -- start 2024-04-26T01:54:10.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.843+0000 7f7c2eddbb80 1 -- start start 2024-04-26T01:54:10.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.844+0000 7f7c2eddbb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfd009400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.844+0000 7f7c2eddbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 0x564cfd009940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.844+0000 7f7c2eddbb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcfebe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.844+0000 7f7c2eddbb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1b63f0 con 0x564cfcfe6920 2024-04-26T01:54:10.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.844+0000 7f7c2eddbb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1b3c20 con 0x564cfcff20f0 2024-04-26T01:54:10.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.844+0000 7f7c2eddbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1b1a90 con 0x564cfcfe7240 2024-04-26T01:54:10.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.844+0000 7f7c21a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfd009400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.844+0000 7f7c22400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcfebe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.844+0000 7f7c21000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 0x564cfd009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.844+0000 7f7c22400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcfebe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34448/0 (socket says 172.21.15.6:34448) 2024-04-26T01:54:10.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.844+0000 7f7c22400700 1 -- 172.21.15.6:0/3665541843 learned_addr learned my addr 172.21.15.6:0/3665541843 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:10.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.845+0000 7f7c22400700 1 -- 172.21.15.6:0/3665541843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 msgr2=0x564cfd009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.845+0000 7f7c22400700 1 --2- 172.21.15.6:0/3665541843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 0x564cfd009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.845+0000 7f7c22400700 1 -- 172.21.15.6:0/3665541843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 msgr2=0x564cfd009400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.845+0000 7f7c22400700 1 --2- 172.21.15.6:0/3665541843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfd009400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.845+0000 7f7c22400700 1 -- 172.21.15.6:0/3665541843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cfcff11a0 con 0x564cfcff20f0 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.845+0000 7f7c21000700 1 --2- 172.21.15.6:0/3665541843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 0x564cfd009940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.845+0000 7f7c21a00700 1 --2- 172.21.15.6:0/3665541843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfd009400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.845+0000 7f7c22400700 1 --2- 172.21.15.6:0/3665541843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcfebe50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7c14015000 tx=0x7f7c14025000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a6ce0d65a892b73 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.845+0000 7f7c0e000700 1 -- 172.21.15.6:0/3665541843 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c14023070 con 0x564cfcff20f0 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.845+0000 7f7c0e000700 1 -- 172.21.15.6:0/3665541843 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c14005110 con 0x564cfcff20f0 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.845+0000 7f7c0e000700 1 -- 172.21.15.6:0/3665541843 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c14027660 con 0x564cfcff20f0 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.846+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/3665541843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 msgr2=0x564cfcfebe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.846+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/3665541843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcfebe50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7c14015000 tx=0x7f7c14025000 comp rx=0 tx=0).stop 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.846+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/3665541843 shutdown_connections 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.846+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/3665541843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcff20f0 0x564cfcfebe50 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.846+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/3665541843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfd009400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.846+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/3665541843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcfe7240 0x564cfd009940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.846+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/3665541843 >> 172.21.15.6:0/3665541843 conn(0x564cfcf156a0 msgr2=0x564cfcfedc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.846+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/3665541843 shutdown_connections 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.846+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/3665541843 wait complete. 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.847+0000 7f7c2eddbb80 1 -- start start 2024-04-26T01:54:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.847+0000 7f7c2eddbb80 1 Processor -- start 2024-04-26T01:54:10.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.847+0000 7f7c2eddbb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfcffa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.847+0000 7f7c2eddbb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcfe7240 0x564cfcff8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.847+0000 7f7c2eddbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcff20f0 0x564cfcff8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.847+0000 7f7c2eddbb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1b20e0 con 0x564cfcfe6920 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.847+0000 7f7c2eddbb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1b3550 con 0x564cfcfe7240 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.847+0000 7f7c2eddbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1b27b0 con 0x564cfcff20f0 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.847+0000 7f7c2eddbb80 10 client.0 _ll_register_callbacks cb 0x564cfced2820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.848+0000 7f7c21a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfcffa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.848+0000 7f7c21a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfcffa3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58710/0 (socket says 172.21.15.6:58710) 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.848+0000 7f7c21a00700 1 -- 172.21.15.6:0/2839645019 learned_addr learned my addr 172.21.15.6:0/2839645019 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.848+0000 7f7c21000700 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcfe7240 0x564cfcff8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.848+0000 7f7c22400700 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcff20f0 0x564cfcff8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.848+0000 7f7c21a00700 1 -- 172.21.15.6:0/2839645019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcff20f0 msgr2=0x564cfcff8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.848+0000 7f7c21a00700 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcff20f0 0x564cfcff8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.848+0000 7f7c21a00700 1 -- 172.21.15.6:0/2839645019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcfe7240 msgr2=0x564cfcff8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.848+0000 7f7c21a00700 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcfe7240 0x564cfcff8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.848+0000 7f7c21a00700 1 -- 172.21.15.6:0/2839645019 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cfcff94f0 con 0x564cfcfe6920 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.848+0000 7f7c21a00700 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfcffa3c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7c100097e0 tx=0x7f7c100cde00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea172a9b85af3df server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.849+0000 7f7c03400700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.849+0000 7f7c03400700 1 -- 172.21.15.6:0/2839645019 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1007f070 con 0x564cfcfe6920 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.849+0000 7f7c03400700 1 -- 172.21.15.6:0/2839645019 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8644+0+0 (secure 0 0 0) 0x7f7c1009a700 con 0x564cfcfe6920 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.849+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/2839645019 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c14005110 con 0x564cfcfe6920 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.849+0000 7f7c03400700 1 -- 172.21.15.6:0/2839645019 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f7c100b3040 con 0x564cfcfe6920 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.849+0000 7f7c2eddbb80 10 client.16041 fetch_fsmap learned FSMap version 41 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.849+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/2839645019 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x564cfd068d40 con 0x564cfcfe6920 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.849+0000 7f7c03400700 1 -- 172.21.15.6:0/2839645019 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c14005110 con 0x564cfcfe6920 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.850+0000 7f7c03400700 1 -- 172.21.15.6:0/2839645019 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f7c10082030 con 0x564cfcfe6920 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.850+0000 7f7c2eddbb80 10 client.16041 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.850+0000 7f7c2eddbb80 10 client.16041 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.850+0000 7f7c2eddbb80 20 client.16041 populate_metadata read hostname 'smithi006' 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.850+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564cfd069440 0x564cfd06b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.850+0000 7f7c2eddbb80 4 client.16041 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1f0250cf", "release": true, "await": true}] 2024-04-26T01:54:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.850+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/2839645019 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1f0250cf", "release": true, "await": true}) -- 0x564cfcf41820 con 0x564cfd069440 2024-04-26T01:54:10.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.850+0000 7f7c21000700 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564cfd069440 0x564cfd06b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:10.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.851+0000 7f7c21000700 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564cfd069440 0x564cfd06b860 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d32cde2280a74693 server_cookie=323e8f54096e211e in_seq=0 out_seq=0 2024-04-26T01:54:10.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.851+0000 7f7c03400700 10 client.16041 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c03400700 1 -- 172.21.15.6:0/2839645019 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x564cfcf41820 con 0x564cfd069440 2024-04-26T01:54:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c03400700 10 client.16041 handle_command_reply: tid=0 2024-04-26T01:54:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c2eddbb80 1 client.16041 shutdown 2024-04-26T01:54:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/2839645019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 msgr2=0x564cfcffa3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfcffa3c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7c100097e0 tx=0x7f7c100cde00 comp rx=0 tx=0).stop 2024-04-26T01:54:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/2839645019 shutdown_connections 2024-04-26T01:54:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564cfd069440 0x564cfd06b860 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564cfcff20f0 0x564cfcff8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cfcfe6920 0x564cfcffa3c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c2eddbb80 1 --2- 172.21.15.6:0/2839645019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564cfcfe7240 0x564cfcff8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/2839645019 >> 172.21.15.6:0/2839645019 conn(0x564cfcf156a0 msgr2=0x564cfcfe1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:10.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c03400700 0 client.16041 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:10.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.860+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/2839645019 shutdown_connections 2024-04-26T01:54:10.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.861+0000 7f7c2eddbb80 1 -- 172.21.15.6:0/2839645019 wait complete. 2024-04-26T01:54:10.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.861+0000 7f7c2eddbb80 20 client.16041 trim_cache size 0 max 16384 2024-04-26T01:54:10.859 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:54:10.859 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 41, 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 24, 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "1f0250cf": { 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 24, 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 82.9, 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:54:10.860 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:54:10.861 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:54:10.861 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:54:10.861 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:54:10.861 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:54:10.861 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:54:10.861 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:54:10.861 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:54:10.861 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:54:10.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.862+0000 7f7c24400700 1 -- 172.21.15.6:0/862792506 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7c040a3e30 msgr2=0x7f7c040a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.862+0000 7f7c24400700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7c040a3e30 0x7f7c040a62f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7c14009fd0 tx=0x7f7c14009db0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.862+0000 7f7c24400700 1 -- 172.21.15.6:0/862792506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c1c1092d0 msgr2=0x7f7c1c19f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.862+0000 7f7c24400700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c1c1092d0 0x7f7c1c19f740 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000e9f0 tx=0x7f7c1000ed00 comp rx=0 tx=0).stop 2024-04-26T01:54:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.862+0000 7f7c24400700 1 -- 172.21.15.6:0/862792506 shutdown_connections 2024-04-26T01:54:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.862+0000 7f7c24400700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7c040a3e30 0x7f7c040a62f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.862+0000 7f7c24400700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c1c10e3b0 0x7f7c1c1a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.862+0000 7f7c24400700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c1c1092d0 0x7f7c1c19f740 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.862+0000 7f7c24400700 1 --2- 172.21.15.6:0/862792506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c1c109c80 0x7f7c1c19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.862+0000 7f7c24400700 1 -- 172.21.15.6:0/862792506 >> 172.21.15.6:0/862792506 conn(0x7f7c1c076210 msgr2=0x7f7c1c0767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.862+0000 7f7c24400700 1 -- 172.21.15.6:0/862792506 shutdown_connections 2024-04-26T01:54:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:10.862+0000 7f7c24400700 1 -- 172.21.15.6:0/862792506 wait complete. 2024-04-26T01:54:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:10 smithi188 ceph-mon[32751]: pgmap v258: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 821 KiB/s wr, 55 op/s 2024-04-26T01:54:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:10 smithi188 ceph-mon[32751]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T01:54:10.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:10 smithi029 ceph-mon[32775]: pgmap v258: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 821 KiB/s wr, 55 op/s 2024-04-26T01:54:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:10 smithi029 ceph-mon[32775]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T01:54:11.602 INFO:tasks.workunit.client.0.smithi006.stdout: 1 277263 3.61 MB/sec execute 275 sec latency 30786.606 ms 2024-04-26T01:54:11.774 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1f0250cf', seconds elapsed: 38.8 2024-04-26T01:54:12.602 INFO:tasks.workunit.client.0.smithi006.stdout: 1 278252 3.62 MB/sec execute 276 sec latency 23.828 ms 2024-04-26T01:54:12.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:12 smithi006 ceph-mon[28419]: pgmap v260: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.0 MiB/s wr, 52 op/s 2024-04-26T01:54:12.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:12 smithi188 ceph-mon[32751]: pgmap v260: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.0 MiB/s wr, 52 op/s 2024-04-26T01:54:12.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:12 smithi029 ceph-mon[32775]: pgmap v260: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.0 MiB/s wr, 52 op/s 2024-04-26T01:54:13.602 INFO:tasks.workunit.client.0.smithi006.stdout: 1 279622 3.63 MB/sec execute 277 sec latency 24.599 ms 2024-04-26T01:54:14.602 INFO:tasks.workunit.client.0.smithi006.stdout: 1 280534 3.64 MB/sec execute 278 sec latency 23.003 ms 2024-04-26T01:54:14.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:14 smithi006 ceph-mon[28419]: pgmap v261: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.0 MiB/s wr, 52 op/s 2024-04-26T01:54:14.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:14 smithi188 ceph-mon[32751]: pgmap v261: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.0 MiB/s wr, 52 op/s 2024-04-26T01:54:14.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:14 smithi029 ceph-mon[32775]: pgmap v261: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.0 MiB/s wr, 52 op/s 2024-04-26T01:54:15.602 INFO:tasks.workunit.client.0.smithi006.stdout: 1 281223 3.63 MB/sec execute 279 sec latency 23.992 ms 2024-04-26T01:54:16.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:16 smithi006 ceph-mon[28419]: pgmap v262: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.0 MiB/s wr, 52 op/s 2024-04-26T01:54:16.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:16 smithi006 ceph-mon[28419]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T01:54:16.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:16 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:54:16.602 INFO:tasks.workunit.client.0.smithi006.stdout: 1 282672 3.64 MB/sec execute 280 sec latency 23.523 ms 2024-04-26T01:54:16.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:16 smithi188 ceph-mon[32751]: pgmap v262: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.0 MiB/s wr, 52 op/s 2024-04-26T01:54:16.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:16 smithi188 ceph-mon[32751]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T01:54:16.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:16 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:54:16.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:16 smithi029 ceph-mon[32775]: pgmap v262: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.0 MiB/s wr, 52 op/s 2024-04-26T01:54:16.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:16 smithi029 ceph-mon[32775]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T01:54:16.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:16 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:54:17.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:17 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:54:17.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:17 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:54:17.602 INFO:tasks.workunit.client.0.smithi006.stdout: 1 284012 3.67 MB/sec execute 281 sec latency 24.530 ms 2024-04-26T01:54:17.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:17 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:54:17.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:17 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:54:17.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:17 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:54:17.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:17 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:54:18.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:18 smithi006 ceph-mon[28419]: pgmap v264: 97 pgs: 97 active+clean; 95 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 140 op/s 2024-04-26T01:54:18.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:18 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:54:18.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:18 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:54:18.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:18 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:54:18.602 INFO:tasks.workunit.client.0.smithi006.stdout: 1 284558 3.66 MB/sec execute 282 sec latency 30.210 ms 2024-04-26T01:54:18.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:18 smithi188 ceph-mon[32751]: pgmap v264: 97 pgs: 97 active+clean; 95 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 140 op/s 2024-04-26T01:54:18.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:18 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:54:18.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:18 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:54:18.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:18 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:54:18.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:18 smithi029 ceph-mon[32775]: pgmap v264: 97 pgs: 97 active+clean; 95 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 140 op/s 2024-04-26T01:54:18.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:18 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:54:18.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:18 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:54:18.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:18 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:54:19.605 INFO:tasks.workunit.client.0.smithi006.stdout: 1 285376 3.67 MB/sec execute 283 sec latency 22.496 ms 2024-04-26T01:54:20.603 INFO:tasks.workunit.client.0.smithi006.stdout: 1 286347 3.66 MB/sec execute 284 sec latency 22.811 ms 2024-04-26T01:54:20.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:20 smithi188 ceph-mon[32751]: pgmap v265: 97 pgs: 97 active+clean; 95 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 116 op/s 2024-04-26T01:54:20.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:20 smithi029 ceph-mon[32775]: pgmap v265: 97 pgs: 97 active+clean; 95 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 116 op/s 2024-04-26T01:54:20.806 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:20 smithi006 ceph-mon[28419]: pgmap v265: 97 pgs: 97 active+clean; 95 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 116 op/s 2024-04-26T01:54:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.192+0000 7f4384400640 1 -- 172.21.15.6:0/2413749339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f437c1021a0 msgr2=0x7f437c102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.192+0000 7f4384400640 1 --2- 172.21.15.6:0/2413749339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f437c1021a0 0x7f437c102580 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f436c009ca0 tx=0x7f436c02a8c0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.192+0000 7f4384400640 1 -- 172.21.15.6:0/2413749339 shutdown_connections 2024-04-26T01:54:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.192+0000 7f4384400640 1 --2- 172.21.15.6:0/2413749339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f437c102b50 0x7f437c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.192+0000 7f4384400640 1 --2- 172.21.15.6:0/2413749339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c107270 0x7f437c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.192+0000 7f4384400640 1 --2- 172.21.15.6:0/2413749339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f437c1021a0 0x7f437c102580 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.192+0000 7f4384400640 1 -- 172.21.15.6:0/2413749339 >> 172.21.15.6:0/2413749339 conn(0x7f437c0fd9e0 msgr2=0x7f437c0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.192+0000 7f4384400640 1 -- 172.21.15.6:0/2413749339 shutdown_connections 2024-04-26T01:54:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.192+0000 7f4384400640 1 -- 172.21.15.6:0/2413749339 wait complete. 2024-04-26T01:54:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.193+0000 7f4384400640 1 Processor -- start 2024-04-26T01:54:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.193+0000 7f4384400640 1 -- start start 2024-04-26T01:54:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4384400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c1021a0 0x7f437c1a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4384400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f437c107270 0x7f437c1a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4384400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f437c19a850 0x7f437c19ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4384400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f437c112200 con 0x7f437c107270 2024-04-26T01:54:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4384400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f437c112080 con 0x7f437c1021a0 2024-04-26T01:54:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4384400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f437c112380 con 0x7f437c19a850 2024-04-26T01:54:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4381a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c1021a0 0x7f437c1a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4381a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c1021a0 0x7f437c1a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43280/0 (socket says 172.21.15.6:43280) 2024-04-26T01:54:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4381a00640 1 -- 172.21.15.6:0/1559776474 learned_addr learned my addr 172.21.15.6:0/1559776474 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4381000640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f437c107270 0x7f437c1a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4382400640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f437c19a850 0x7f437c19ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4381a00640 1 -- 172.21.15.6:0/1559776474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f437c19a850 msgr2=0x7f437c19ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4381a00640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f437c19a850 0x7f437c19ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4381a00640 1 -- 172.21.15.6:0/1559776474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f437c107270 msgr2=0x7f437c1a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4381a00640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f437c107270 0x7f437c1a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.194+0000 7f4381a00640 1 -- 172.21.15.6:0/1559776474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437c19b3e0 con 0x7f437c1021a0 2024-04-26T01:54:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.195+0000 7f4381a00640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c1021a0 0x7f437c1a0640 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f436c005e30 tx=0x7f436c031840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.195+0000 7f437a000640 1 -- 172.21.15.6:0/1559776474 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c003b70 con 0x7f437c1021a0 2024-04-26T01:54:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.195+0000 7f437a000640 1 -- 172.21.15.6:0/1559776474 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f436c003d10 con 0x7f437c1021a0 2024-04-26T01:54:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.195+0000 7f4384400640 1 -- 172.21.15.6:0/1559776474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f437c19b670 con 0x7f437c1021a0 2024-04-26T01:54:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.195+0000 7f437a000640 1 -- 172.21.15.6:0/1559776474 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c02fa50 con 0x7f437c1021a0 2024-04-26T01:54:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.195+0000 7f4384400640 1 -- 172.21.15.6:0/1559776474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f437c1a7800 con 0x7f437c1021a0 2024-04-26T01:54:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.199+0000 7f437a000640 1 -- 172.21.15.6:0/1559776474 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f436c02fbf0 con 0x7f437c1021a0 2024-04-26T01:54:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.199+0000 7f4387392740 1 Processor -- start 2024-04-26T01:54:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.200+0000 7f4387392740 1 -- start start 2024-04-26T01:54:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.200+0000 7f4387392740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8137820 0x560dd8137c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.200+0000 7f4387392740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8138140 0x560dd81424c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.200+0000 7f4387392740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8142a00 0x560dd8144df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.200+0000 7f4387392740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560dd809d900 con 0x560dd8138140 2024-04-26T01:54:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.200+0000 7f4387392740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560dd8043ef0 con 0x560dd8137820 2024-04-26T01:54:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.200+0000 7f4387392740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560dd80998c0 con 0x560dd8142a00 2024-04-26T01:54:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.200+0000 7f4381000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8142a00 0x560dd8144df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.200+0000 7f4381a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8138140 0x560dd81424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.200+0000 7f4381000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8142a00 0x560dd8144df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:40108/0 (socket says 172.21.15.6:40108) 2024-04-26T01:54:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.200+0000 7f4381000640 1 -- 172.21.15.6:0/4080752317 learned_addr learned my addr 172.21.15.6:0/4080752317 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.200+0000 7f4382400640 1 --2- 172.21.15.6:0/4080752317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8137820 0x560dd8137c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.201+0000 7f4381000640 1 -- 172.21.15.6:0/4080752317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8137820 msgr2=0x560dd8137c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.201+0000 7f4381000640 1 --2- 172.21.15.6:0/4080752317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8137820 0x560dd8137c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.201+0000 7f4381000640 1 -- 172.21.15.6:0/4080752317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8138140 msgr2=0x560dd81424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.201+0000 7f4381000640 1 --2- 172.21.15.6:0/4080752317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8138140 0x560dd81424c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.201+0000 7f4381000640 1 -- 172.21.15.6:0/4080752317 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dd7db0210 con 0x560dd8142a00 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.201+0000 7f4381000640 1 --2- 172.21.15.6:0/4080752317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8142a00 0x560dd8144df0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4368009c20 tx=0x7f4368011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c95186cb698e7851 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.201+0000 7f437a000640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43600a3d90 0x7f43600a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.201+0000 7f4372a00640 1 -- 172.21.15.6:0/4080752317 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4368004230 con 0x560dd8142a00 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.202+0000 7f4372a00640 1 -- 172.21.15.6:0/4080752317 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43680043d0 con 0x560dd8142a00 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.202+0000 7f4372a00640 1 -- 172.21.15.6:0/4080752317 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4368004d50 con 0x560dd8142a00 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.202+0000 7f437a000640 1 -- 172.21.15.6:0/1559776474 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f436c0f1a40 con 0x7f437c1021a0 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.202+0000 7f4382400640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43600a3d90 0x7f43600a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.202+0000 7f4387392740 1 -- 172.21.15.6:0/4080752317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8142a00 msgr2=0x560dd8144df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.202+0000 7f4387392740 1 --2- 172.21.15.6:0/4080752317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8142a00 0x560dd8144df0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4368009c20 tx=0x7f4368011810 comp rx=0 tx=0).stop 2024-04-26T01:54:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.203+0000 7f4382400640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43600a3d90 0x7f43600a6250 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4374005fd0 tx=0x7f4374005cc0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.203+0000 7f4387392740 1 -- 172.21.15.6:0/4080752317 shutdown_connections 2024-04-26T01:54:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.203+0000 7f4387392740 1 --2- 172.21.15.6:0/4080752317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8138140 0x560dd81424c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.203+0000 7f4387392740 1 --2- 172.21.15.6:0/4080752317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8142a00 0x560dd8144df0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.203+0000 7f4387392740 1 --2- 172.21.15.6:0/4080752317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8137820 0x560dd8137c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.203+0000 7f4387392740 1 -- 172.21.15.6:0/4080752317 >> 172.21.15.6:0/4080752317 conn(0x560dd8133d30 msgr2=0x560dd8134110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.203+0000 7f4387392740 1 -- 172.21.15.6:0/4080752317 shutdown_connections 2024-04-26T01:54:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.203+0000 7f4387392740 1 -- 172.21.15.6:0/4080752317 wait complete. 2024-04-26T01:54:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.203+0000 7f4387392740 1 -- start start 2024-04-26T01:54:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.203+0000 7f4387392740 1 Processor -- start 2024-04-26T01:54:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.204+0000 7f4387392740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 0x560dd813f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.204+0000 7f4387392740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8138140 0x560dd813fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.204+0000 7f4387392740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8142a00 0x560dd8140360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.204+0000 7f4387392740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560dd8145630 con 0x560dd8142a00 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.204+0000 7f4387392740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560dd8057310 con 0x560dd8138140 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.204+0000 7f4387392740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560dd81457b0 con 0x560dd8137820 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.204+0000 7f4387392740 10 client.0 _ll_register_callbacks cb 0x560dd8025cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.204+0000 7f4381a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 0x560dd813f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.204+0000 7f4381000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8138140 0x560dd813fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.204+0000 7f4381000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8138140 0x560dd813fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43298/0 (socket says 172.21.15.6:43298) 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.204+0000 7f4381000640 1 -- 172.21.15.6:0/2041899473 learned_addr learned my addr 172.21.15.6:0/2041899473 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.204+0000 7f4382400640 1 --2- 172.21.15.6:0/2041899473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8142a00 0x560dd8140360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.205+0000 7f4381a00640 1 -- 172.21.15.6:0/2041899473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8138140 msgr2=0x560dd813fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.205+0000 7f4381a00640 1 --2- 172.21.15.6:0/2041899473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8138140 0x560dd813fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.205+0000 7f4381a00640 1 -- 172.21.15.6:0/2041899473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8142a00 msgr2=0x560dd8140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.205+0000 7f4381a00640 1 --2- 172.21.15.6:0/2041899473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8142a00 0x560dd8140360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.205+0000 7f4381a00640 1 -- 172.21.15.6:0/2041899473 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dd7e7c620 con 0x560dd8137820 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.205+0000 7f4381a00640 1 --2- 172.21.15.6:0/2041899473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 0x560dd813f8e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f436c0ef250 tx=0x7f436c0099c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eda0ea450868ccd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.205+0000 7f4367400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.205+0000 7f4367400640 1 -- 172.21.15.6:0/2041899473 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c0bd5e0 con 0x560dd8137820 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.205+0000 7f4367400640 1 -- 172.21.15.6:0/2041899473 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f436c0ba110 con 0x560dd8137820 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.205+0000 7f4387392740 1 -- 172.21.15.6:0/2041899473 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43680043d0 con 0x560dd8137820 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.205+0000 7f4367400640 1 -- 172.21.15.6:0/2041899473 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f436c0bd400 con 0x560dd8137820 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.206+0000 7f4387392740 10 client.26233 fetch_fsmap learned FSMap version 41 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.206+0000 7f4387392740 1 -- 172.21.15.6:0/2041899473 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x560dd8145630 con 0x560dd8137820 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.206+0000 7f4367400640 1 -- 172.21.15.6:0/2041899473 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43680043d0 con 0x560dd8137820 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.206+0000 7f4367400640 1 -- 172.21.15.6:0/2041899473 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f436c0bdd00 con 0x560dd8137820 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.206+0000 7f4387392740 10 client.26233 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.206+0000 7f4387392740 10 client.26233 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.206+0000 7f4387392740 20 client.26233 populate_metadata read hostname 'smithi006' 2024-04-26T01:54:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.206+0000 7f4387392740 1 --2- 172.21.15.6:0/2041899473 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560dd8152a10 0x560dd8154e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.206+0000 7f4387392740 4 client.26233 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.206+0000 7f4387392740 1 -- 172.21.15.6:0/2041899473 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560dd80b18b0 con 0x560dd8152a10 2024-04-26T01:54:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.206+0000 7f4381000640 1 --2- 172.21.15.6:0/2041899473 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560dd8152a10 0x560dd8154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.207+0000 7f4381000640 1 --2- 172.21.15.6:0/2041899473 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560dd8152a10 0x560dd8154e00 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97767d48ea5ef902 server_cookie=9b0e9fd24e0554f1 in_seq=0 out_seq=0 2024-04-26T01:54:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.207+0000 7f4367400640 10 client.26233 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.208+0000 7f4367400640 1 -- 172.21.15.6:0/2041899473 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560dd80b18b0 con 0x560dd8152a10 2024-04-26T01:54:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.208+0000 7f4367400640 10 client.26233 handle_command_reply: tid=0 2024-04-26T01:54:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.208+0000 7f4387392740 1 client.26233 shutdown 2024-04-26T01:54:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.208+0000 7f4387392740 1 -- 172.21.15.6:0/2041899473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 msgr2=0x560dd813f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.208+0000 7f4387392740 1 --2- 172.21.15.6:0/2041899473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 0x560dd813f8e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f436c0ef250 tx=0x7f436c0099c0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.208+0000 7f4387392740 1 -- 172.21.15.6:0/2041899473 shutdown_connections 2024-04-26T01:54:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.208+0000 7f4387392740 1 --2- 172.21.15.6:0/2041899473 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560dd8152a10 0x560dd8154e00 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.208+0000 7f4387392740 1 --2- 172.21.15.6:0/2041899473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8142a00 0x560dd8140360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.208+0000 7f4387392740 1 --2- 172.21.15.6:0/2041899473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8138140 0x560dd813fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.208+0000 7f4367400640 0 client.26233 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.208+0000 7f4387392740 1 --2- 172.21.15.6:0/2041899473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 0x560dd813f8e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.208+0000 7f4387392740 1 -- 172.21.15.6:0/2041899473 >> 172.21.15.6:0/2041899473 conn(0x560dd8133d30 msgr2=0x560dd8139ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.209+0000 7f4387392740 1 -- 172.21.15.6:0/2041899473 shutdown_connections 2024-04-26T01:54:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.209+0000 7f4387392740 1 -- 172.21.15.6:0/2041899473 wait complete. 2024-04-26T01:54:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.209+0000 7f4387392740 20 client.26233 trim_cache size 0 max 16384 2024-04-26T01:54:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.228+0000 7f4387392740 1 Processor -- start 2024-04-26T01:54:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.228+0000 7f4387392740 1 -- start start 2024-04-26T01:54:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.228+0000 7f4387392740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 0x560dd814eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.228+0000 7f4387392740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8138140 0x560dd814e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.228+0000 7f4387392740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8142a00 0x560dd815a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.228+0000 7f4387392740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f437c1b27b0 con 0x560dd8138140 2024-04-26T01:54:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.228+0000 7f4387392740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f437c1b0020 con 0x560dd8142a00 2024-04-26T01:54:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.228+0000 7f4387392740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f437c1ade90 con 0x560dd8137820 2024-04-26T01:54:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.228+0000 7f4381a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 0x560dd814eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.229+0000 7f4381a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 0x560dd814eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:40136/0 (socket says 172.21.15.6:40136) 2024-04-26T01:54:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.229+0000 7f4381a00640 1 -- 172.21.15.6:0/1401021808 learned_addr learned my addr 172.21.15.6:0/1401021808 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.229+0000 7f4381000640 1 --2- 172.21.15.6:0/1401021808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8138140 0x560dd814e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.229+0000 7f4382400640 1 --2- 172.21.15.6:0/1401021808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8142a00 0x560dd815a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.229+0000 7f4381a00640 1 -- 172.21.15.6:0/1401021808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8142a00 msgr2=0x560dd815a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.229+0000 7f4381a00640 1 --2- 172.21.15.6:0/1401021808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8142a00 0x560dd815a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.229+0000 7f4381a00640 1 -- 172.21.15.6:0/1401021808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8138140 msgr2=0x560dd814e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.229+0000 7f4381a00640 1 --2- 172.21.15.6:0/1401021808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8138140 0x560dd814e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.229+0000 7f4381a00640 1 -- 172.21.15.6:0/1401021808 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dd8155130 con 0x560dd8137820 2024-04-26T01:54:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.229+0000 7f4381a00640 1 --2- 172.21.15.6:0/1401021808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 0x560dd814eef0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f436c0bda30 tx=0x7f436c0be0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45016826aab51c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.229+0000 7f4372000640 1 -- 172.21.15.6:0/1401021808 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c0bd5e0 con 0x560dd8137820 2024-04-26T01:54:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.230+0000 7f4372000640 1 -- 172.21.15.6:0/1401021808 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f436c036070 con 0x560dd8137820 2024-04-26T01:54:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.230+0000 7f4372000640 1 -- 172.21.15.6:0/1401021808 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c0bdaa0 con 0x560dd8137820 2024-04-26T01:54:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.230+0000 7f4387392740 1 -- 172.21.15.6:0/1401021808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 msgr2=0x560dd814eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.230+0000 7f4387392740 1 --2- 172.21.15.6:0/1401021808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 0x560dd814eef0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f436c0bda30 tx=0x7f436c0be0c0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.230+0000 7f4387392740 1 -- 172.21.15.6:0/1401021808 shutdown_connections 2024-04-26T01:54:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.230+0000 7f4387392740 1 --2- 172.21.15.6:0/1401021808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8138140 0x560dd814e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.230+0000 7f4387392740 1 --2- 172.21.15.6:0/1401021808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8142a00 0x560dd815a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.230+0000 7f4387392740 1 --2- 172.21.15.6:0/1401021808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8137820 0x560dd814eef0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.231+0000 7f4387392740 1 -- 172.21.15.6:0/1401021808 >> 172.21.15.6:0/1401021808 conn(0x560dd8133d30 msgr2=0x560dd8134110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.231+0000 7f4387392740 1 -- 172.21.15.6:0/1401021808 shutdown_connections 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.231+0000 7f4387392740 1 -- 172.21.15.6:0/1401021808 wait complete. 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.231+0000 7f4387392740 1 -- start start 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.231+0000 7f4387392740 1 Processor -- start 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.232+0000 7f4387392740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8137820 0x560dd8148540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.232+0000 7f4387392740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8138140 0x560dd8148a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.232+0000 7f4387392740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8142a00 0x560dd8148fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.232+0000 7f4387392740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f437c1ae4e0 con 0x560dd8142a00 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.232+0000 7f4387392740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f437c1af950 con 0x560dd8137820 2024-04-26T01:54:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.232+0000 7f4387392740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f437c1aebb0 con 0x560dd8138140 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.232+0000 7f4387392740 10 client.0 _ll_register_callbacks cb 0x7f436c003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.232+0000 7f4382400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8142a00 0x560dd8148fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.232+0000 7f4381a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8137820 0x560dd8148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.232+0000 7f4382400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8142a00 0x560dd8148fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48552/0 (socket says 172.21.15.6:48552) 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.232+0000 7f4382400640 1 -- 172.21.15.6:0/3990199364 learned_addr learned my addr 172.21.15.6:0/3990199364 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.232+0000 7f4381000640 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8138140 0x560dd8148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.233+0000 7f4381a00640 1 -- 172.21.15.6:0/3990199364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8138140 msgr2=0x560dd8148a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.233+0000 7f4381a00640 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8138140 0x560dd8148a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.233+0000 7f4381a00640 1 -- 172.21.15.6:0/3990199364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8142a00 msgr2=0x560dd8148fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.233+0000 7f4381a00640 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8142a00 0x560dd8148fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.233+0000 7f4381a00640 1 -- 172.21.15.6:0/3990199364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dd81b8040 con 0x560dd8137820 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.233+0000 7f4381000640 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8138140 0x560dd8148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.233+0000 7f4381a00640 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8137820 0x560dd8148540 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f436c0bd7a0 tx=0x7f436c0f1dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89dadd6bd8945c85 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.233+0000 7f4367400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.233+0000 7f4367400640 1 -- 172.21.15.6:0/3990199364 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c0bd5e0 con 0x560dd8137820 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.233+0000 7f4387392740 1 -- 172.21.15.6:0/3990199364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f436c036070 con 0x560dd8137820 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.234+0000 7f4367400640 1 -- 172.21.15.6:0/3990199364 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f436c0c6480 con 0x560dd8137820 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.234+0000 7f4367400640 1 -- 172.21.15.6:0/3990199364 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f436c0ef510 con 0x560dd8137820 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.234+0000 7f4387392740 10 client.26195 fetch_fsmap learned FSMap version 41 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.234+0000 7f4387392740 1 -- 172.21.15.6:0/3990199364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x560dd81b8040 con 0x560dd8137820 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.234+0000 7f4367400640 1 -- 172.21.15.6:0/3990199364 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f436c036070 con 0x560dd8137820 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.234+0000 7f4367400640 1 -- 172.21.15.6:0/3990199364 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f436c0b97f0 con 0x560dd8137820 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.234+0000 7f4387392740 10 client.26195 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.234+0000 7f4387392740 10 client.26195 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.234+0000 7f4387392740 20 client.26195 populate_metadata read hostname 'smithi006' 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.234+0000 7f4387392740 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560dd81b9630 0x560dd81bba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.234+0000 7f4387392740 4 client.26195 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.234+0000 7f4387392740 1 -- 172.21.15.6:0/3990199364 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560dd80ab320 con 0x560dd81b9630 2024-04-26T01:54:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.235+0000 7f4381000640 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560dd81b9630 0x560dd81bba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.235+0000 7f4381000640 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560dd81b9630 0x560dd81bba20 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2be8ea50613b9d1c server_cookie=b9ecbc8ce2ed73a9 in_seq=0 out_seq=0 2024-04-26T01:54:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.235+0000 7f4367400640 10 client.26195 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.282+0000 7f4367400640 1 -- 172.21.15.6:0/3990199364 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560dd80ab320 con 0x560dd81b9630 2024-04-26T01:54:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.282+0000 7f4367400640 10 client.26195 handle_command_reply: tid=0 2024-04-26T01:54:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.282+0000 7f4387392740 1 client.26195 shutdown 2024-04-26T01:54:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.283+0000 7f4387392740 1 -- 172.21.15.6:0/3990199364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8137820 msgr2=0x560dd8148540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.283+0000 7f4387392740 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8137820 0x560dd8148540 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f436c0bd7a0 tx=0x7f436c0f1dd0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.283+0000 7f4387392740 1 -- 172.21.15.6:0/3990199364 shutdown_connections 2024-04-26T01:54:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.283+0000 7f4387392740 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560dd81b9630 0x560dd81bba20 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.283+0000 7f4387392740 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560dd8142a00 0x560dd8148fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.283+0000 7f4387392740 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560dd8138140 0x560dd8148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.283+0000 7f4387392740 1 --2- 172.21.15.6:0/3990199364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560dd8137820 0x560dd8148540 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.283+0000 7f4367400640 0 client.26195 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.283+0000 7f4387392740 1 -- 172.21.15.6:0/3990199364 >> 172.21.15.6:0/3990199364 conn(0x560dd8133d30 msgr2=0x560dd813f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.283+0000 7f4387392740 1 -- 172.21.15.6:0/3990199364 shutdown_connections 2024-04-26T01:54:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.283+0000 7f4387392740 1 -- 172.21.15.6:0/3990199364 wait complete. 2024-04-26T01:54:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.283+0000 7f4387392740 20 client.26195 trim_cache size 0 max 16384 2024-04-26T01:54:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:54:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:54:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:54:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:54:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.284+0000 7f4384400640 1 -- 172.21.15.6:0/1559776474 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43600a3d90 msgr2=0x7f43600a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.284+0000 7f4384400640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43600a3d90 0x7f43600a6250 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4374005fd0 tx=0x7f4374005cc0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.284+0000 7f4384400640 1 -- 172.21.15.6:0/1559776474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c1021a0 msgr2=0x7f437c1a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.284+0000 7f4384400640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c1021a0 0x7f437c1a0640 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f436c005e30 tx=0x7f436c031840 comp rx=0 tx=0).stop 2024-04-26T01:54:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.284+0000 7f4384400640 1 -- 172.21.15.6:0/1559776474 shutdown_connections 2024-04-26T01:54:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.284+0000 7f4384400640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43600a3d90 0x7f43600a6250 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.284+0000 7f4384400640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f437c107270 0x7f437c1a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.284+0000 7f4384400640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f437c19a850 0x7f437c19ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.284+0000 7f4384400640 1 --2- 172.21.15.6:0/1559776474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f437c1021a0 0x7f437c1a0640 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.284+0000 7f4384400640 1 -- 172.21.15.6:0/1559776474 >> 172.21.15.6:0/1559776474 conn(0x7f437c0fd9e0 msgr2=0x7f437c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.285+0000 7f4384400640 1 -- 172.21.15.6:0/1559776474 shutdown_connections 2024-04-26T01:54:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:21.285+0000 7f4384400640 1 -- 172.21.15.6:0/1559776474 wait complete. 2024-04-26T01:54:21.603 INFO:tasks.workunit.client.0.smithi006.stdout: 1 287565 3.69 MB/sec execute 285 sec latency 48.095 ms 2024-04-26T01:54:22.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:22 smithi006 ceph-mon[28419]: pgmap v266: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-26T01:54:22.603 INFO:tasks.workunit.client.0.smithi006.stdout: 1 288122 3.68 MB/sec execute 286 sec latency 52.044 ms 2024-04-26T01:54:22.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:22 smithi188 ceph-mon[32751]: pgmap v266: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-26T01:54:22.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:22 smithi029 ceph-mon[32775]: pgmap v266: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-26T01:54:23.603 INFO:tasks.workunit.client.0.smithi006.stdout: 1 289158 3.68 MB/sec execute 287 sec latency 22.450 ms 2024-04-26T01:54:24.604 INFO:tasks.workunit.client.0.smithi006.stdout: 1 290520 3.70 MB/sec execute 288 sec latency 25.958 ms 2024-04-26T01:54:24.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:24 smithi188 ceph-mon[32751]: pgmap v267: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-26T01:54:24.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:24 smithi029 ceph-mon[32775]: pgmap v267: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-26T01:54:24.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:24 smithi006 ceph-mon[28419]: pgmap v267: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-26T01:54:25.604 INFO:tasks.workunit.client.0.smithi006.stdout: 1 291282 3.70 MB/sec execute 289 sec latency 19.203 ms 2024-04-26T01:54:26.604 INFO:tasks.workunit.client.0.smithi006.stdout: 1 292285 3.71 MB/sec execute 290 sec latency 27.318 ms 2024-04-26T01:54:26.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:26 smithi188 ceph-mon[32751]: pgmap v268: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-26T01:54:26.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:26 smithi029 ceph-mon[32775]: pgmap v268: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-26T01:54:26.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:26 smithi006 ceph-mon[28419]: pgmap v268: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-26T01:54:27.604 INFO:tasks.workunit.client.0.smithi006.stdout: 1 293752 3.72 MB/sec execute 291 sec latency 23.368 ms 2024-04-26T01:54:28.604 INFO:tasks.workunit.client.0.smithi006.stdout: 1 294807 3.73 MB/sec execute 292 sec latency 17.798 ms 2024-04-26T01:54:28.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:28 smithi188 ceph-mon[32751]: pgmap v269: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 203 op/s 2024-04-26T01:54:28.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:28 smithi029 ceph-mon[32775]: pgmap v269: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 203 op/s 2024-04-26T01:54:28.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:28 smithi006 ceph-mon[28419]: pgmap v269: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 203 op/s 2024-04-26T01:54:29.604 INFO:tasks.workunit.client.0.smithi006.stdout: 1 295826 3.74 MB/sec execute 293 sec latency 23.304 ms 2024-04-26T01:54:30.604 INFO:tasks.workunit.client.0.smithi006.stdout: 1 296881 3.73 MB/sec execute 294 sec latency 48.007 ms 2024-04-26T01:54:30.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:30 smithi188 ceph-mon[32751]: pgmap v270: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 984 KiB/s wr, 133 op/s 2024-04-26T01:54:30.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:30 smithi029 ceph-mon[32775]: pgmap v270: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 984 KiB/s wr, 133 op/s 2024-04-26T01:54:30.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:30 smithi006 ceph-mon[28419]: pgmap v270: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 984 KiB/s wr, 133 op/s 2024-04-26T01:54:31.605 INFO:tasks.workunit.client.0.smithi006.stdout: 1 298302 3.76 MB/sec execute 295 sec latency 52.530 ms 2024-04-26T01:54:32.605 INFO:tasks.workunit.client.0.smithi006.stdout: 1 299152 3.75 MB/sec execute 296 sec latency 25.229 ms 2024-04-26T01:54:32.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:32 smithi188 ceph-mon[32751]: pgmap v271: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 198 op/s 2024-04-26T01:54:32.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:32 smithi029 ceph-mon[32775]: pgmap v271: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 198 op/s 2024-04-26T01:54:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:32 smithi006 ceph-mon[28419]: pgmap v271: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 198 op/s 2024-04-26T01:54:33.605 INFO:tasks.workunit.client.0.smithi006.stdout: 1 300435 3.76 MB/sec execute 297 sec latency 29.387 ms 2024-04-26T01:54:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.440+0000 7f1806000640 1 -- 172.21.15.6:0/739038115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1800104d70 msgr2=0x7f1800108db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.440+0000 7f1806000640 1 --2- 172.21.15.6:0/739038115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1800104d70 0x7f1800108db0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009900 tx=0x7f17e802a410 comp rx=0 tx=0).stop 2024-04-26T01:54:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.440+0000 7f1806000640 1 -- 172.21.15.6:0/739038115 shutdown_connections 2024-04-26T01:54:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.440+0000 7f1806000640 1 --2- 172.21.15.6:0/739038115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1800109490 0x7f1800111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.440+0000 7f1806000640 1 --2- 172.21.15.6:0/739038115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1800104d70 0x7f1800108db0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.440+0000 7f1806000640 1 --2- 172.21.15.6:0/739038115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18001043c0 0x7f18001047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.440+0000 7f1806000640 1 -- 172.21.15.6:0/739038115 >> 172.21.15.6:0/739038115 conn(0x7f18000ffc40 msgr2=0x7f1800102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.441+0000 7f1806000640 1 -- 172.21.15.6:0/739038115 shutdown_connections 2024-04-26T01:54:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.441+0000 7f1806000640 1 -- 172.21.15.6:0/739038115 wait complete. 2024-04-26T01:54:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.442+0000 7f1806000640 1 Processor -- start 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.442+0000 7f1806000640 1 -- start start 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.442+0000 7f1806000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18001043c0 0x7f18001a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.442+0000 7f1806000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1800109490 0x7f18001a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.442+0000 7f1806000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180019a800 0x7f180019acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.442+0000 7f1806000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18001143b0 con 0x7f1800109490 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.442+0000 7f1806000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1800114230 con 0x7f180019a800 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.442+0000 7f1806000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1800114530 con 0x7f18001043c0 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.442+0000 7f17ff400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18001043c0 0x7f18001a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.442+0000 7f17ff400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18001043c0 0x7f18001a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:49952/0 (socket says 172.21.15.6:49952) 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.442+0000 7f17ff400640 1 -- 172.21.15.6:0/951188795 learned_addr learned my addr 172.21.15.6:0/951188795 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.443+0000 7f17f6a00640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1800109490 0x7f18001a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.443+0000 7f17ffe00640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180019a800 0x7f180019acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.443+0000 7f17ff400640 1 -- 172.21.15.6:0/951188795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180019a800 msgr2=0x7f180019acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.443+0000 7f17ff400640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180019a800 0x7f180019acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.443+0000 7f17ff400640 1 -- 172.21.15.6:0/951188795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1800109490 msgr2=0x7f18001a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.443+0000 7f17ff400640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1800109490 0x7f18001a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.443+0000 7f17ff400640 1 -- 172.21.15.6:0/951188795 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180019b390 con 0x7f18001043c0 2024-04-26T01:54:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.443+0000 7f17ff400640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18001043c0 0x7f18001a0640 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f17f000e9f0 tx=0x7f17f000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.443+0000 7f17fcc00640 1 -- 172.21.15.6:0/951188795 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f000ccc0 con 0x7f18001043c0 2024-04-26T01:54:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.444+0000 7f17fcc00640 1 -- 172.21.15.6:0/951188795 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17f000ce60 con 0x7f18001043c0 2024-04-26T01:54:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.444+0000 7f1806000640 1 -- 172.21.15.6:0/951188795 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180019b680 con 0x7f18001043c0 2024-04-26T01:54:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.444+0000 7f17fcc00640 1 -- 172.21.15.6:0/951188795 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0010430 con 0x7f18001043c0 2024-04-26T01:54:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.444+0000 7f1806000640 1 -- 172.21.15.6:0/951188795 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18001a7790 con 0x7f18001043c0 2024-04-26T01:54:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.447+0000 7f17fcc00640 1 -- 172.21.15.6:0/951188795 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f17f00046c0 con 0x7f18001043c0 2024-04-26T01:54:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.447+0000 7f1808e2d740 1 Processor -- start 2024-04-26T01:54:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.447+0000 7f1808e2d740 1 -- start start 2024-04-26T01:54:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.448+0000 7f1808e2d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b7820 0x5619f44b7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.448+0000 7f1808e2d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b8140 0x5619f44c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.448+0000 7f1808e2d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 0x5619f44c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.448+0000 7f1808e2d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5619f441d900 con 0x5619f44c2a00 2024-04-26T01:54:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.448+0000 7f1808e2d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5619f43c3ef0 con 0x5619f44b8140 2024-04-26T01:54:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.448+0000 7f1808e2d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5619f44198c0 con 0x5619f44b7820 2024-04-26T01:54:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.448+0000 7f17ff400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b8140 0x5619f44c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.448+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b7820 0x5619f44b7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.448+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b7820 0x5619f44b7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:49960/0 (socket says 172.21.15.6:49960) 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.448+0000 7f17ffe00640 1 -- 172.21.15.6:0/3719212120 learned_addr learned my addr 172.21.15.6:0/3719212120 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.448+0000 7f17ff400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b8140 0x5619f44c24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47622/0 (socket says 172.21.15.6:47622) 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.448+0000 7f17f6a00640 1 --2- 172.21.15.6:0/3719212120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 0x5619f44c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.449+0000 7f17ffe00640 1 -- 172.21.15.6:0/3719212120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b8140 msgr2=0x5619f44c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.449+0000 7f17ffe00640 1 --2- 172.21.15.6:0/3719212120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b8140 0x5619f44c24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.449+0000 7f17ffe00640 1 -- 172.21.15.6:0/3719212120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 msgr2=0x5619f44c4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.449+0000 7f17ffe00640 1 --2- 172.21.15.6:0/3719212120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 0x5619f44c4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.449+0000 7f17ffe00640 1 -- 172.21.15.6:0/3719212120 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619f4130210 con 0x5619f44b7820 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.449+0000 7f17fcc00640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f17e40a3d10 0x7f17e40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.449+0000 7f17ffe00640 1 --2- 172.21.15.6:0/3719212120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b7820 0x5619f44b7c00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009f50 tx=0x7f17f80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=516721634de6eb5c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.449+0000 7f17fcc00640 1 -- 172.21.15.6:0/951188795 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f17f00d0ca0 con 0x7f18001043c0 2024-04-26T01:54:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.449+0000 7f17ff400640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f17e40a3d10 0x7f17e40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.449+0000 7f17efe00640 1 -- 172.21.15.6:0/3719212120 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8004d20 con 0x5619f44b7820 2024-04-26T01:54:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.449+0000 7f17efe00640 1 -- 172.21.15.6:0/3719212120 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17f8004050 con 0x5619f44b7820 2024-04-26T01:54:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.449+0000 7f17efe00640 1 -- 172.21.15.6:0/3719212120 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8005280 con 0x5619f44b7820 2024-04-26T01:54:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.450+0000 7f1808e2d740 1 -- 172.21.15.6:0/3719212120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b7820 msgr2=0x5619f44b7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.450+0000 7f1808e2d740 1 --2- 172.21.15.6:0/3719212120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b7820 0x5619f44b7c00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009f50 tx=0x7f17f80118d0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.450+0000 7f17ff400640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f17e40a3d10 0x7f17e40a61d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x5619f44b8dc0 tx=0x7f17f00d4750 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.450+0000 7f1808e2d740 1 -- 172.21.15.6:0/3719212120 shutdown_connections 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.450+0000 7f1808e2d740 1 --2- 172.21.15.6:0/3719212120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 0x5619f44c4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.450+0000 7f1808e2d740 1 --2- 172.21.15.6:0/3719212120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b8140 0x5619f44c24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.450+0000 7f1808e2d740 1 --2- 172.21.15.6:0/3719212120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b7820 0x5619f44b7c00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.450+0000 7f1808e2d740 1 -- 172.21.15.6:0/3719212120 >> 172.21.15.6:0/3719212120 conn(0x5619f44b3d30 msgr2=0x5619f44b4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.450+0000 7f1808e2d740 1 -- 172.21.15.6:0/3719212120 shutdown_connections 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.450+0000 7f1808e2d740 1 -- 172.21.15.6:0/3719212120 wait complete. 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.451+0000 7f1808e2d740 1 -- start start 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.451+0000 7f1808e2d740 1 Processor -- start 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.451+0000 7f1808e2d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44b7820 0x5619f44bf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.451+0000 7f1808e2d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44bfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f1808e2d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44c2a00 0x5619f44c03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f1808e2d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5619f44c5630 con 0x5619f44b7820 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f1808e2d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5619f43d7310 con 0x5619f44c2a00 2024-04-26T01:54:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f1808e2d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5619f44c57b0 con 0x5619f44b8140 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f1808e2d740 10 client.0 _ll_register_callbacks cb 0x5619f43a5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44bfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44bfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:49968/0 (socket says 172.21.15.6:49968) 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f17f6a00640 1 -- 172.21.15.6:0/4198406409 learned_addr learned my addr 172.21.15.6:0/4198406409 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f17ffe00640 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44b7820 0x5619f44bf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f17ffe00640 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44c2a00 0x5619f44c03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f17f6a00640 1 -- 172.21.15.6:0/4198406409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44c2a00 msgr2=0x5619f44c03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f17f6a00640 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44c2a00 0x5619f44c03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f17f6a00640 1 -- 172.21.15.6:0/4198406409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44b7820 msgr2=0x5619f44bf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f17f6a00640 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44b7820 0x5619f44bf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f17f6a00640 1 -- 172.21.15.6:0/4198406409 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619f41fc620 con 0x5619f44b8140 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.452+0000 7f17ffe00640 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44c2a00 0x5619f44c03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.453+0000 7f17f6a00640 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44bfea0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009900 tx=0x7f17e8032dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b2e06a728b76984 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.453+0000 7f17ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.453+0000 7f17ecc00640 1 -- 172.21.15.6:0/4198406409 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8035040 con 0x5619f44b8140 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.453+0000 7f1808e2d740 1 -- 172.21.15.6:0/4198406409 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17f8004050 con 0x5619f44b8140 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.453+0000 7f17ecc00640 1 -- 172.21.15.6:0/4198406409 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f17e80316a0 con 0x5619f44b8140 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.453+0000 7f17ecc00640 1 -- 172.21.15.6:0/4198406409 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f17e80318a0 con 0x5619f44b8140 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.453+0000 7f1808e2d740 10 client.26251 fetch_fsmap learned FSMap version 41 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.453+0000 7f1808e2d740 1 -- 172.21.15.6:0/4198406409 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5619f44d1850 con 0x5619f44b8140 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.453+0000 7f17ecc00640 1 -- 172.21.15.6:0/4198406409 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17f8004050 con 0x5619f44b8140 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.454+0000 7f17ecc00640 1 -- 172.21.15.6:0/4198406409 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f17e8004750 con 0x5619f44b8140 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.454+0000 7f1808e2d740 10 client.26251 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.454+0000 7f1808e2d740 10 client.26251 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.454+0000 7f1808e2d740 20 client.26251 populate_metadata read hostname 'smithi006' 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.454+0000 7f1808e2d740 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619f44d2d10 0x5619f44d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.454+0000 7f1808e2d740 4 client.26251 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.454+0000 7f1808e2d740 1 -- 172.21.15.6:0/4198406409 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619f44318b0 con 0x5619f44d2d10 2024-04-26T01:54:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.454+0000 7f17ffe00640 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619f44d2d10 0x5619f44d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.455+0000 7f17ffe00640 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619f44d2d10 0x5619f44d5100 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f36718fe1daebe3 server_cookie=a3d263f803fb82eb in_seq=0 out_seq=0 2024-04-26T01:54:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.455+0000 7f17ecc00640 10 client.26251 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f17ecc00640 1 -- 172.21.15.6:0/4198406409 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5619f44318b0 con 0x5619f44d2d10 2024-04-26T01:54:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f17ecc00640 10 client.26251 handle_command_reply: tid=0 2024-04-26T01:54:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f1808e2d740 1 client.26251 shutdown 2024-04-26T01:54:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f1808e2d740 1 -- 172.21.15.6:0/4198406409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 msgr2=0x5619f44bfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f1808e2d740 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44bfea0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009900 tx=0x7f17e8032dc0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f1808e2d740 1 -- 172.21.15.6:0/4198406409 shutdown_connections 2024-04-26T01:54:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f1808e2d740 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619f44d2d10 0x5619f44d5100 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f1808e2d740 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44c2a00 0x5619f44c03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f17ecc00640 0 client.26251 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f1808e2d740 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44bfea0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f1808e2d740 1 --2- 172.21.15.6:0/4198406409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44b7820 0x5619f44bf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f1808e2d740 1 -- 172.21.15.6:0/4198406409 >> 172.21.15.6:0/4198406409 conn(0x5619f44b3d30 msgr2=0x5619f44b9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f1808e2d740 1 -- 172.21.15.6:0/4198406409 shutdown_connections 2024-04-26T01:54:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f1808e2d740 1 -- 172.21.15.6:0/4198406409 wait complete. 2024-04-26T01:54:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.456+0000 7f1808e2d740 20 client.26251 trim_cache size 0 max 16384 2024-04-26T01:54:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.474+0000 7f1808e2d740 1 Processor -- start 2024-04-26T01:54:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.474+0000 7f1808e2d740 1 -- start start 2024-04-26T01:54:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f1808e2d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 0x5619f44ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f1808e2d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44c0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f1808e2d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 0x5619f44da9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f1808e2d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18001b2920 con 0x5619f44c2a00 2024-04-26T01:54:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f1808e2d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f18001b0190 con 0x5619f44b7820 2024-04-26T01:54:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f1808e2d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f18001ae000 con 0x5619f44b8140 2024-04-26T01:54:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 0x5619f44ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 0x5619f44ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47634/0 (socket says 172.21.15.6:47634) 2024-04-26T01:54:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f17ffe00640 1 -- 172.21.15.6:0/1112771550 learned_addr learned my addr 172.21.15.6:0/1112771550 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44c0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:49970/0 (socket says 172.21.15.6:49970) 2024-04-26T01:54:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f17ffe00640 1 --2- 172.21.15.6:0/1112771550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 0x5619f44da9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f17ffe00640 1 -- 172.21.15.6:0/1112771550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 msgr2=0x5619f44c0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f17ffe00640 1 --2- 172.21.15.6:0/1112771550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44c0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f17ffe00640 1 -- 172.21.15.6:0/1112771550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 msgr2=0x5619f44da9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f17ffe00640 1 --2- 172.21.15.6:0/1112771550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 0x5619f44da9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.475+0000 7f17ffe00640 1 -- 172.21.15.6:0/1112771550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619f44b8ea0 con 0x5619f44b7820 2024-04-26T01:54:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.476+0000 7f17ffe00640 1 --2- 172.21.15.6:0/1112771550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 0x5619f44ce4f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f17f8011dd0 tx=0x7f17f80096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c63fc3f9efbaf01c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.476+0000 7f17ef400640 1 -- 172.21.15.6:0/1112771550 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8004250 con 0x5619f44b7820 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.476+0000 7f17ef400640 1 -- 172.21.15.6:0/1112771550 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17f80043f0 con 0x5619f44b7820 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.476+0000 7f17ef400640 1 -- 172.21.15.6:0/1112771550 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f802c580 con 0x5619f44b7820 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.477+0000 7f1808e2d740 1 -- 172.21.15.6:0/1112771550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 msgr2=0x5619f44ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.477+0000 7f1808e2d740 1 --2- 172.21.15.6:0/1112771550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 0x5619f44ce4f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f17f8011dd0 tx=0x7f17f80096c0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.477+0000 7f1808e2d740 1 -- 172.21.15.6:0/1112771550 shutdown_connections 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.477+0000 7f1808e2d740 1 --2- 172.21.15.6:0/1112771550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 0x5619f44da9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.477+0000 7f1808e2d740 1 --2- 172.21.15.6:0/1112771550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44c0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.477+0000 7f1808e2d740 1 --2- 172.21.15.6:0/1112771550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 0x5619f44ce4f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.477+0000 7f1808e2d740 1 -- 172.21.15.6:0/1112771550 >> 172.21.15.6:0/1112771550 conn(0x5619f44b3d30 msgr2=0x5619f44bf920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.477+0000 7f1808e2d740 1 -- 172.21.15.6:0/1112771550 shutdown_connections 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.477+0000 7f1808e2d740 1 -- 172.21.15.6:0/1112771550 wait complete. 2024-04-26T01:54:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.477+0000 7f1808e2d740 1 -- start start 2024-04-26T01:54:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.478+0000 7f1808e2d740 1 Processor -- start 2024-04-26T01:54:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.478+0000 7f1808e2d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 0x5619f44caa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.478+0000 7f1808e2d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44caf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.478+0000 7f1808e2d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 0x5619f44c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.478+0000 7f1808e2d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18001ae650 con 0x5619f44c2a00 2024-04-26T01:54:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.478+0000 7f1808e2d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f18001afac0 con 0x5619f44b7820 2024-04-26T01:54:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.478+0000 7f1808e2d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f18001aed20 con 0x5619f44b8140 2024-04-26T01:54:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.478+0000 7f1808e2d740 10 client.0 _ll_register_callbacks cb 0x7f17e80095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.478+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.478+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 0x5619f44caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.478+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 0x5619f44caa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47646/0 (socket says 172.21.15.6:47646) 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.478+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44caf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:49972/0 (socket says 172.21.15.6:49972) 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.479+0000 7f17ffe00640 1 -- 172.21.15.6:0/3745957399 learned_addr learned my addr 172.21.15.6:0/3745957399 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.479+0000 7f17ffe00640 1 --2- 172.21.15.6:0/3745957399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 0x5619f44c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.479+0000 7f17f6a00640 1 -- 172.21.15.6:0/3745957399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 msgr2=0x5619f44caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.479+0000 7f17f6a00640 1 --2- 172.21.15.6:0/3745957399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 0x5619f44caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.479+0000 7f17f6a00640 1 -- 172.21.15.6:0/3745957399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 msgr2=0x5619f44c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.479+0000 7f17f6a00640 1 --2- 172.21.15.6:0/3745957399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 0x5619f44c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.479+0000 7f17f6a00640 1 -- 172.21.15.6:0/3745957399 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619f44c87a0 con 0x5619f44b8140 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.479+0000 7f17f6a00640 1 --2- 172.21.15.6:0/3745957399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44caf70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f17e8000c00 tx=0x7f17e80316d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=322423f6fcda813 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.479+0000 7f17ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:54:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f17ecc00640 1 -- 172.21.15.6:0/3745957399 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8035050 con 0x5619f44b8140 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f17ecc00640 1 -- 172.21.15.6:0/3745957399 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f17e8004d90 con 0x5619f44b8140 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f1808e2d740 1 -- 172.21.15.6:0/3745957399 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17f80043f0 con 0x5619f44b8140 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f17ecc00640 1 -- 172.21.15.6:0/3745957399 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f17e8005850 con 0x5619f44b8140 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f1808e2d740 10 client.26257 fetch_fsmap learned FSMap version 41 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f1808e2d740 1 -- 172.21.15.6:0/3745957399 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5619f4538ec0 con 0x5619f44b8140 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f17ecc00640 1 -- 172.21.15.6:0/3745957399 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17f80043f0 con 0x5619f44b8140 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f17ecc00640 1 -- 172.21.15.6:0/3745957399 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f17e8054020 con 0x5619f44b8140 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f1808e2d740 10 client.26257 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f1808e2d740 10 client.26257 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f1808e2d740 20 client.26257 populate_metadata read hostname 'smithi006' 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f1808e2d740 1 --2- 172.21.15.6:0/3745957399 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619f4539670 0x5619f453ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f1808e2d740 4 client.26257 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.480+0000 7f1808e2d740 1 -- 172.21.15.6:0/3745957399 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5619f442b320 con 0x5619f4539670 2024-04-26T01:54:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.481+0000 7f17ffe00640 1 --2- 172.21.15.6:0/3745957399 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619f4539670 0x5619f453ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.481+0000 7f17ffe00640 1 --2- 172.21.15.6:0/3745957399 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619f4539670 0x5619f453ba60 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=550712cfb2ad3df1 server_cookie=6d70ef99ed42c905 in_seq=0 out_seq=0 2024-04-26T01:54:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.481+0000 7f17ecc00640 10 client.26257 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.530+0000 7f17ecc00640 1 -- 172.21.15.6:0/3745957399 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5619f442b320 con 0x5619f4539670 2024-04-26T01:54:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.530+0000 7f17ecc00640 10 client.26257 handle_command_reply: tid=0 2024-04-26T01:54:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.530+0000 7f1808e2d740 1 client.26257 shutdown 2024-04-26T01:54:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.531+0000 7f1808e2d740 1 -- 172.21.15.6:0/3745957399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 msgr2=0x5619f44caf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.531+0000 7f1808e2d740 1 --2- 172.21.15.6:0/3745957399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44caf70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f17e8000c00 tx=0x7f17e80316d0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.531+0000 7f1808e2d740 1 -- 172.21.15.6:0/3745957399 shutdown_connections 2024-04-26T01:54:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.531+0000 7f1808e2d740 1 --2- 172.21.15.6:0/3745957399 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619f4539670 0x5619f453ba60 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.531+0000 7f1808e2d740 1 --2- 172.21.15.6:0/3745957399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619f44c2a00 0x5619f44c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.531+0000 7f1808e2d740 1 --2- 172.21.15.6:0/3745957399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619f44b8140 0x5619f44caf70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.531+0000 7f17ecc00640 0 client.26257 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.531+0000 7f1808e2d740 1 --2- 172.21.15.6:0/3745957399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619f44b7820 0x5619f44caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.531+0000 7f1808e2d740 1 -- 172.21.15.6:0/3745957399 >> 172.21.15.6:0/3745957399 conn(0x5619f44b3d30 msgr2=0x5619f44bf920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.531+0000 7f1808e2d740 1 -- 172.21.15.6:0/3745957399 shutdown_connections 2024-04-26T01:54:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.531+0000 7f1808e2d740 1 -- 172.21.15.6:0/3745957399 wait complete. 2024-04-26T01:54:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.531+0000 7f1808e2d740 20 client.26257 trim_cache size 0 max 16384 2024-04-26T01:54:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:54:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:54:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:54:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:54:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.532+0000 7f1806000640 1 -- 172.21.15.6:0/951188795 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f17e40a3d10 msgr2=0x7f17e40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.532+0000 7f1806000640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f17e40a3d10 0x7f17e40a61d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x5619f44b8dc0 tx=0x7f17f00d4750 comp rx=0 tx=0).stop 2024-04-26T01:54:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.532+0000 7f1806000640 1 -- 172.21.15.6:0/951188795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18001043c0 msgr2=0x7f18001a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.532+0000 7f1806000640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18001043c0 0x7f18001a0640 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f17f000e9f0 tx=0x7f17f000eeb0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.532+0000 7f1806000640 1 -- 172.21.15.6:0/951188795 shutdown_connections 2024-04-26T01:54:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.532+0000 7f1806000640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f17e40a3d10 0x7f17e40a61d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.532+0000 7f1806000640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1800109490 0x7f18001a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.532+0000 7f1806000640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180019a800 0x7f180019acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.532+0000 7f1806000640 1 --2- 172.21.15.6:0/951188795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18001043c0 0x7f18001a0640 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.532+0000 7f1806000640 1 -- 172.21.15.6:0/951188795 >> 172.21.15.6:0/951188795 conn(0x7f18000ffc40 msgr2=0x7f1800100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.532+0000 7f1806000640 1 -- 172.21.15.6:0/951188795 shutdown_connections 2024-04-26T01:54:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:34.532+0000 7f1806000640 1 -- 172.21.15.6:0/951188795 wait complete. 2024-04-26T01:54:34.605 INFO:tasks.workunit.client.0.smithi006.stdout: 1 301783 3.78 MB/sec execute 298 sec latency 22.303 ms 2024-04-26T01:54:34.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:34 smithi188 ceph-mon[32751]: pgmap v272: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 144 op/s 2024-04-26T01:54:34.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:34 smithi029 ceph-mon[32775]: pgmap v272: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 144 op/s 2024-04-26T01:54:34.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:34 smithi006 ceph-mon[28419]: pgmap v272: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 144 op/s 2024-04-26T01:54:35.605 INFO:tasks.workunit.client.0.smithi006.stdout: 1 302512 3.78 MB/sec execute 299 sec latency 21.147 ms 2024-04-26T01:54:36.605 INFO:tasks.workunit.client.0.smithi006.stdout: 1 303711 3.78 MB/sec execute 300 sec latency 26.347 ms 2024-04-26T01:54:36.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:36 smithi188 ceph-mon[32751]: pgmap v273: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 144 op/s 2024-04-26T01:54:36.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:36 smithi029 ceph-mon[32775]: pgmap v273: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 144 op/s 2024-04-26T01:54:36.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:36 smithi006 ceph-mon[28419]: pgmap v273: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 144 op/s 2024-04-26T01:54:37.605 INFO:tasks.workunit.client.0.smithi006.stdout: 1 305155 3.81 MB/sec execute 301 sec latency 17.962 ms 2024-04-26T01:54:38.605 INFO:tasks.workunit.client.0.smithi006.stdout: 1 305817 3.80 MB/sec execute 302 sec latency 19.222 ms 2024-04-26T01:54:38.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:38 smithi188 ceph-mon[32751]: pgmap v274: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 231 op/s 2024-04-26T01:54:38.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:38 smithi029 ceph-mon[32775]: pgmap v274: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 231 op/s 2024-04-26T01:54:38.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:38 smithi006 ceph-mon[28419]: pgmap v274: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 231 op/s 2024-04-26T01:54:39.606 INFO:tasks.workunit.client.0.smithi006.stdout: 1 306841 3.80 MB/sec execute 303 sec latency 25.749 ms 2024-04-26T01:54:40.606 INFO:tasks.workunit.client.0.smithi006.stdout: 1 308349 3.82 MB/sec execute 304 sec latency 82.172 ms 2024-04-26T01:54:40.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:40 smithi029 ceph-mon[32775]: pgmap v275: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-04-26T01:54:40.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:40 smithi006 ceph-mon[28419]: pgmap v275: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-04-26T01:54:40.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:40 smithi188 ceph-mon[32751]: pgmap v275: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-04-26T01:54:41.606 INFO:tasks.workunit.client.0.smithi006.stdout: 1 309137 3.82 MB/sec execute 305 sec latency 13.521 ms 2024-04-26T01:54:42.607 INFO:tasks.workunit.client.0.smithi006.stdout: 1 310117 3.83 MB/sec execute 306 sec latency 26.542 ms 2024-04-26T01:54:42.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:42 smithi029 ceph-mon[32775]: pgmap v276: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 213 op/s 2024-04-26T01:54:42.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:42 smithi006 ceph-mon[28419]: pgmap v276: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 213 op/s 2024-04-26T01:54:42.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:42 smithi188 ceph-mon[32751]: pgmap v276: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 213 op/s 2024-04-26T01:54:43.607 INFO:tasks.workunit.client.0.smithi006.stdout: 1 311491 3.83 MB/sec execute 307 sec latency 25.345 ms 2024-04-26T01:54:44.607 INFO:tasks.workunit.client.0.smithi006.stdout: 1 312547 3.85 MB/sec execute 308 sec latency 13.152 ms 2024-04-26T01:54:44.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:44 smithi029 ceph-mon[32775]: pgmap v277: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1014 KiB/s wr, 148 op/s 2024-04-26T01:54:44.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:44 smithi006 ceph-mon[28419]: pgmap v277: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1014 KiB/s wr, 148 op/s 2024-04-26T01:54:44.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:44 smithi188 ceph-mon[32751]: pgmap v277: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1014 KiB/s wr, 148 op/s 2024-04-26T01:54:45.607 INFO:tasks.workunit.client.0.smithi006.stdout: 1 313393 3.84 MB/sec execute 309 sec latency 30.650 ms 2024-04-26T01:54:46.607 INFO:tasks.workunit.client.0.smithi006.stdout: 1 314610 3.85 MB/sec execute 310 sec latency 22.960 ms 2024-04-26T01:54:46.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:46 smithi029 ceph-mon[32775]: pgmap v278: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1014 KiB/s wr, 148 op/s 2024-04-26T01:54:46.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:46 smithi006 ceph-mon[28419]: pgmap v278: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1014 KiB/s wr, 148 op/s 2024-04-26T01:54:46.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:46 smithi188 ceph-mon[32751]: pgmap v278: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1014 KiB/s wr, 148 op/s 2024-04-26T01:54:47.608 INFO:tasks.workunit.client.0.smithi006.stdout: 1 315969 3.87 MB/sec execute 311 sec latency 18.492 ms 2024-04-26T01:54:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.681+0000 7f60a2e00640 1 -- 172.21.15.6:0/490900327 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f609c10c590 msgr2=0x7f609c10ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.681+0000 7f60a2e00640 1 --2- 172.21.15.6:0/490900327 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f609c10c590 0x7f609c10ea00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6094009ca0 tx=0x7f609402a950 comp rx=0 tx=0).stop 2024-04-26T01:54:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.682+0000 7f60a2e00640 1 -- 172.21.15.6:0/490900327 shutdown_connections 2024-04-26T01:54:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.682+0000 7f60a2e00640 1 --2- 172.21.15.6:0/490900327 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f609c103ae0 0x7f609c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.682+0000 7f60a2e00640 1 --2- 172.21.15.6:0/490900327 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f609c10c590 0x7f609c10ea00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.682+0000 7f60a2e00640 1 --2- 172.21.15.6:0/490900327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c1031c0 0x7f609c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.682+0000 7f60a2e00640 1 -- 172.21.15.6:0/490900327 >> 172.21.15.6:0/490900327 conn(0x7f609c077fb0 msgr2=0x7f609c0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.682+0000 7f60a2e00640 1 -- 172.21.15.6:0/490900327 shutdown_connections 2024-04-26T01:54:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.682+0000 7f60a2e00640 1 -- 172.21.15.6:0/490900327 wait complete. 2024-04-26T01:54:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.683+0000 7f60a2e00640 1 Processor -- start 2024-04-26T01:54:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.683+0000 7f60a2e00640 1 -- start start 2024-04-26T01:54:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.683+0000 7f60a2e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c1031c0 0x7f609c19e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.683+0000 7f60a2e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f609c103ae0 0x7f609c19e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.683+0000 7f60a2e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f609c10c590 0x7f609c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.683+0000 7f60a2e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f609c112240 con 0x7f609c103ae0 2024-04-26T01:54:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.683+0000 7f60a2e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f609c1120c0 con 0x7f609c1031c0 2024-04-26T01:54:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.683+0000 7f60a2e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f609c1123c0 con 0x7f609c10c590 2024-04-26T01:54:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.683+0000 7f60a0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f609c10c590 0x7f609c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.684+0000 7f60a0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f609c10c590 0x7f609c1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:48492/0 (socket says 172.21.15.6:48492) 2024-04-26T01:54:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.684+0000 7f60a0e00640 1 -- 172.21.15.6:0/3439794599 learned_addr learned my addr 172.21.15.6:0/3439794599 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.684+0000 7f609be00640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c1031c0 0x7f609c19e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.684+0000 7f609b400640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f609c103ae0 0x7f609c19e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.684+0000 7f60a0e00640 1 -- 172.21.15.6:0/3439794599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c1031c0 msgr2=0x7f609c19e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.684+0000 7f60a0e00640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c1031c0 0x7f609c19e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.684+0000 7f60a0e00640 1 -- 172.21.15.6:0/3439794599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f609c103ae0 msgr2=0x7f609c19e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.684+0000 7f60a0e00640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f609c103ae0 0x7f609c19e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.684+0000 7f60a0e00640 1 -- 172.21.15.6:0/3439794599 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609c198d40 con 0x7f609c10c590 2024-04-26T01:54:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.684+0000 7f609be00640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c1031c0 0x7f609c19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.684+0000 7f60a0e00640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f609c10c590 0x7f609c1985c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6094009dc0 tx=0x7f6094003ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.685+0000 7f6098c00640 1 -- 172.21.15.6:0/3439794599 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6094034040 con 0x7f609c10c590 2024-04-26T01:54:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.685+0000 7f6098c00640 1 -- 172.21.15.6:0/3439794599 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6094003e50 con 0x7f609c10c590 2024-04-26T01:54:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.685+0000 7f60a2e00640 1 -- 172.21.15.6:0/3439794599 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f609c198fd0 con 0x7f609c10c590 2024-04-26T01:54:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.685+0000 7f6098c00640 1 -- 172.21.15.6:0/3439794599 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6094004600 con 0x7f609c10c590 2024-04-26T01:54:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.685+0000 7f60a2e00640 1 -- 172.21.15.6:0/3439794599 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f609c1a52d0 con 0x7f609c10c590 2024-04-26T01:54:47.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.688+0000 7f60a5db6740 1 Processor -- start 2024-04-26T01:54:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.689+0000 7f6098c00640 1 -- 172.21.15.6:0/3439794599 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f6094037020 con 0x7f609c10c590 2024-04-26T01:54:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.689+0000 7f60a5db6740 1 -- start start 2024-04-26T01:54:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.689+0000 7f60a5db6740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dbc30 0x55af7f2dc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.689+0000 7f60a5db6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2dc550 0x55af7f2e68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.689+0000 7f60a5db6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2e6e10 0x55af7f2e9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.689+0000 7f60a5db6740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55af7f23d900 con 0x55af7f2dbc30 2024-04-26T01:54:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.689+0000 7f60a5db6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55af7f1e3ef0 con 0x55af7f2dc550 2024-04-26T01:54:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.689+0000 7f60a5db6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55af7f2398c0 con 0x55af7f2e6e10 2024-04-26T01:54:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.690+0000 7f609be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2dc550 0x55af7f2e68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.690+0000 7f609b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dbc30 0x55af7f2dc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.690+0000 7f609be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2dc550 0x55af7f2e68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:40680/0 (socket says 172.21.15.6:40680) 2024-04-26T01:54:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.690+0000 7f609be00640 1 -- 172.21.15.6:0/4037660792 learned_addr learned my addr 172.21.15.6:0/4037660792 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.690+0000 7f609b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dbc30 0x55af7f2dc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50924/0 (socket says 172.21.15.6:50924) 2024-04-26T01:54:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.690+0000 7f609b400640 1 --2- 172.21.15.6:0/4037660792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2e6e10 0x55af7f2e9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.690+0000 7f609be00640 1 -- 172.21.15.6:0/4037660792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2e6e10 msgr2=0x55af7f2e9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.690+0000 7f609be00640 1 --2- 172.21.15.6:0/4037660792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2e6e10 0x55af7f2e9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.690+0000 7f609be00640 1 -- 172.21.15.6:0/4037660792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dbc30 msgr2=0x55af7f2dc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.690+0000 7f609be00640 1 --2- 172.21.15.6:0/4037660792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dbc30 0x55af7f2dc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.690+0000 7f609be00640 1 -- 172.21.15.6:0/4037660792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af7ef50210 con 0x55af7f2dc550 2024-04-26T01:54:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.691+0000 7f609be00640 1 --2- 172.21.15.6:0/4037660792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2dc550 0x55af7f2e68d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f608c0115c0 tx=0x7f608c011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aecaa06236a742cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.691+0000 7f6091600640 1 -- 172.21.15.6:0/4037660792 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c00f640 con 0x55af7f2dc550 2024-04-26T01:54:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.691+0000 7f6098c00640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f60800a3d10 0x7f60800a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.691+0000 7f6091600640 1 -- 172.21.15.6:0/4037660792 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f608c00fc90 con 0x55af7f2dc550 2024-04-26T01:54:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.691+0000 7f6091600640 1 -- 172.21.15.6:0/4037660792 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c019bf0 con 0x55af7f2dc550 2024-04-26T01:54:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.691+0000 7f6098c00640 1 -- 172.21.15.6:0/3439794599 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f60940f1450 con 0x7f609c10c590 2024-04-26T01:54:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.691+0000 7f609b400640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f60800a3d10 0x7f60800a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.692+0000 7f60a5db6740 1 -- 172.21.15.6:0/4037660792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2dc550 msgr2=0x55af7f2e68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.692+0000 7f60a5db6740 1 --2- 172.21.15.6:0/4037660792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2dc550 0x55af7f2e68d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f608c0115c0 tx=0x7f608c011a80 comp rx=0 tx=0).stop 2024-04-26T01:54:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.692+0000 7f609b400640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f60800a3d10 0x7f60800a61d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55af7f2d9560 tx=0x7f6088005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.692+0000 7f60a5db6740 1 -- 172.21.15.6:0/4037660792 shutdown_connections 2024-04-26T01:54:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.692+0000 7f60a5db6740 1 --2- 172.21.15.6:0/4037660792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2e6e10 0x55af7f2e9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.692+0000 7f60a5db6740 1 --2- 172.21.15.6:0/4037660792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2dc550 0x55af7f2e68d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.692+0000 7f60a5db6740 1 --2- 172.21.15.6:0/4037660792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dbc30 0x55af7f2dc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.692+0000 7f60a5db6740 1 -- 172.21.15.6:0/4037660792 >> 172.21.15.6:0/4037660792 conn(0x55af7f2d8140 msgr2=0x55af7f2d8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.692+0000 7f60a5db6740 1 -- 172.21.15.6:0/4037660792 shutdown_connections 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.692+0000 7f60a5db6740 1 -- 172.21.15.6:0/4037660792 wait complete. 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.692+0000 7f60a5db6740 1 -- start start 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.693+0000 7f60a5db6740 1 Processor -- start 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.693+0000 7f60a5db6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 0x55af7f2e3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.693+0000 7f60a5db6740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 0x55af7f2e4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.693+0000 7f60a5db6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2e4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.693+0000 7f60a5db6740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55af7f2e9a40 con 0x55af7f2dc550 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.693+0000 7f60a5db6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55af7f1f7310 con 0x55af7f2e6e10 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.693+0000 7f60a5db6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55af7f2e9bc0 con 0x55af7f2dbc30 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.693+0000 7f60a5db6740 10 client.0 _ll_register_callbacks cb 0x55af7f1c5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.693+0000 7f609b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 0x55af7f2e4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.693+0000 7f609be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 0x55af7f2e3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f60a0e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2e4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f609be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 0x55af7f2e3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:48512/0 (socket says 172.21.15.6:48512) 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f609b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 0x55af7f2e4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50938/0 (socket says 172.21.15.6:50938) 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f609be00640 1 -- 172.21.15.6:0/685762838 learned_addr learned my addr 172.21.15.6:0/685762838 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f60a0e00640 1 -- 172.21.15.6:0/685762838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 msgr2=0x55af7f2e3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f60a0e00640 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 0x55af7f2e3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f60a0e00640 1 -- 172.21.15.6:0/685762838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 msgr2=0x55af7f2e4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f60a0e00640 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 0x55af7f2e4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f60a0e00640 1 -- 172.21.15.6:0/685762838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af7f01c620 con 0x55af7f2e6e10 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f609be00640 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 0x55af7f2e3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f609b400640 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 0x55af7f2e4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f60a0e00640 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2e4790 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f60940eec00 tx=0x7f60940bcdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e250df8acdc837d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.694+0000 7f6086000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.695+0000 7f6086000640 1 -- 172.21.15.6:0/685762838 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60940d6060 con 0x55af7f2e6e10 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.695+0000 7f6086000640 1 -- 172.21.15.6:0/685762838 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f60940c5c30 con 0x55af7f2e6e10 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.695+0000 7f60a5db6740 1 -- 172.21.15.6:0/685762838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f608c00fc90 con 0x55af7f2e6e10 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.695+0000 7f6086000640 1 -- 172.21.15.6:0/685762838 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f609402c040 con 0x55af7f2e6e10 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.695+0000 7f60a5db6740 10 client.26225 fetch_fsmap learned FSMap version 41 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.695+0000 7f60a5db6740 1 -- 172.21.15.6:0/685762838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55af7f2f5c70 con 0x55af7f2e6e10 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.695+0000 7f6086000640 1 -- 172.21.15.6:0/685762838 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f608c00fc90 con 0x55af7f2e6e10 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.695+0000 7f6086000640 1 -- 172.21.15.6:0/685762838 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f609409e070 con 0x55af7f2e6e10 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.696+0000 7f60a5db6740 10 client.26225 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.696+0000 7f60a5db6740 10 client.26225 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.696+0000 7f60a5db6740 20 client.26225 populate_metadata read hostname 'smithi006' 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.696+0000 7f60a5db6740 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af7f2f7120 0x55af7f2f9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.696+0000 7f60a5db6740 4 client.26225 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.696+0000 7f60a5db6740 1 -- 172.21.15.6:0/685762838 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af7f2518b0 con 0x55af7f2f7120 2024-04-26T01:54:47.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.696+0000 7f609be00640 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af7f2f7120 0x55af7f2f9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.696+0000 7f609be00640 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af7f2f7120 0x55af7f2f9510 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b8ddb072e1694ff server_cookie=3149809cdc103a3d in_seq=0 out_seq=0 2024-04-26T01:54:47.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.696+0000 7f6086000640 10 client.26225 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.697+0000 7f6086000640 1 -- 172.21.15.6:0/685762838 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55af7f2518b0 con 0x55af7f2f7120 2024-04-26T01:54:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.697+0000 7f6086000640 10 client.26225 handle_command_reply: tid=0 2024-04-26T01:54:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.697+0000 7f60a5db6740 1 client.26225 shutdown 2024-04-26T01:54:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.697+0000 7f60a5db6740 1 -- 172.21.15.6:0/685762838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 msgr2=0x55af7f2e4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.697+0000 7f60a5db6740 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2e4790 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f60940eec00 tx=0x7f60940bcdb0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.698+0000 7f60a5db6740 1 -- 172.21.15.6:0/685762838 shutdown_connections 2024-04-26T01:54:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.698+0000 7f60a5db6740 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af7f2f7120 0x55af7f2f9510 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.698+0000 7f60a5db6740 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 0x55af7f2e4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.698+0000 7f60a5db6740 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2e4790 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.698+0000 7f6086000640 0 client.26225 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.698+0000 7f60a5db6740 1 --2- 172.21.15.6:0/685762838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 0x55af7f2e3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.698+0000 7f60a5db6740 1 -- 172.21.15.6:0/685762838 >> 172.21.15.6:0/685762838 conn(0x55af7f2d8140 msgr2=0x55af7f2ddec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.698+0000 7f60a5db6740 1 -- 172.21.15.6:0/685762838 shutdown_connections 2024-04-26T01:54:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.698+0000 7f60a5db6740 1 -- 172.21.15.6:0/685762838 wait complete. 2024-04-26T01:54:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.698+0000 7f60a5db6740 20 client.26225 trim_cache size 0 max 16384 2024-04-26T01:54:47.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.728+0000 7f60a5db6740 1 Processor -- start 2024-04-26T01:54:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.729+0000 7f60a5db6740 1 -- start start 2024-04-26T01:54:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.729+0000 7f60a5db6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 0x55af7f2f2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.729+0000 7f60a5db6740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 0x55af7f2ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.729+0000 7f60a5db6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2fee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.729+0000 7f60a5db6740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f609c1ae410 con 0x55af7f2dc550 2024-04-26T01:54:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.729+0000 7f60a5db6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f609c1abc80 con 0x55af7f2e6e10 2024-04-26T01:54:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.729+0000 7f60a5db6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f609c1a9af0 con 0x55af7f2dbc30 2024-04-26T01:54:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.730+0000 7f609b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 0x55af7f2ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.730+0000 7f609be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 0x55af7f2f2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.730+0000 7f60a0e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2fee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.730+0000 7f60a0e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2fee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:40690/0 (socket says 172.21.15.6:40690) 2024-04-26T01:54:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.730+0000 7f60a0e00640 1 -- 172.21.15.6:0/3008120060 learned_addr learned my addr 172.21.15.6:0/3008120060 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.731+0000 7f609be00640 1 -- 172.21.15.6:0/3008120060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 msgr2=0x55af7f2fee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.731+0000 7f609be00640 1 --2- 172.21.15.6:0/3008120060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2fee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.731+0000 7f609be00640 1 -- 172.21.15.6:0/3008120060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 msgr2=0x55af7f2ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.732+0000 7f609be00640 1 --2- 172.21.15.6:0/3008120060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 0x55af7f2ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.732+0000 7f609be00640 1 -- 172.21.15.6:0/3008120060 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af7f2f9880 con 0x55af7f2dbc30 2024-04-26T01:54:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.732+0000 7f609b400640 1 --2- 172.21.15.6:0/3008120060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 0x55af7f2ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:54:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.733+0000 7f609be00640 1 --2- 172.21.15.6:0/3008120060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 0x55af7f2f2930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x55af7f2f9d70 tx=0x7f608c0096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df114b3ad482655b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.734+0000 7f6090c00640 1 -- 172.21.15.6:0/3008120060 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c0051c0 con 0x55af7f2dbc30 2024-04-26T01:54:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.734+0000 7f6090c00640 1 -- 172.21.15.6:0/3008120060 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f608c019d70 con 0x55af7f2dbc30 2024-04-26T01:54:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.734+0000 7f6090c00640 1 -- 172.21.15.6:0/3008120060 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c0051c0 con 0x55af7f2dbc30 2024-04-26T01:54:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.735+0000 7f60a5db6740 1 -- 172.21.15.6:0/3008120060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 msgr2=0x55af7f2f2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.735+0000 7f60a5db6740 1 --2- 172.21.15.6:0/3008120060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 0x55af7f2f2930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x55af7f2f9d70 tx=0x7f608c0096a0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.736+0000 7f60a5db6740 1 -- 172.21.15.6:0/3008120060 shutdown_connections 2024-04-26T01:54:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.736+0000 7f60a5db6740 1 --2- 172.21.15.6:0/3008120060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2dc550 0x55af7f2ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.736+0000 7f60a5db6740 1 --2- 172.21.15.6:0/3008120060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.736+0000 7f60a5db6740 1 --2- 172.21.15.6:0/3008120060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dbc30 0x55af7f2f2930 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.736+0000 7f60a5db6740 1 -- 172.21.15.6:0/3008120060 >> 172.21.15.6:0/3008120060 conn(0x55af7f2d8140 msgr2=0x55af7f2e25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.736+0000 7f60a5db6740 1 -- 172.21.15.6:0/3008120060 shutdown_connections 2024-04-26T01:54:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.737+0000 7f60a5db6740 1 -- 172.21.15.6:0/3008120060 wait complete. 2024-04-26T01:54:47.736 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.737+0000 7f60a5db6740 1 -- start start 2024-04-26T01:54:47.736 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.738+0000 7f60a5db6740 1 Processor -- start 2024-04-26T01:54:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.739+0000 7f60a5db6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dc550 0x55af7f2d5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.739+0000 7f60a5db6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2d6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.739+0000 7f60a5db6740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2d6740 0x55af7f2ede40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.739+0000 7f60a5db6740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f609c1aa140 con 0x55af7f2d6740 2024-04-26T01:54:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.739+0000 7f60a5db6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f609c1ab5b0 con 0x55af7f2e6e10 2024-04-26T01:54:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.739+0000 7f60a5db6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f609c1aa810 con 0x55af7f2dc550 2024-04-26T01:54:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.740+0000 7f60a5db6740 10 client.0 _ll_register_callbacks cb 0x7f60940bd2f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.740+0000 7f609b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2d6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.740+0000 7f609be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dc550 0x55af7f2d5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.740+0000 7f609b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2d6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:40704/0 (socket says 172.21.15.6:40704) 2024-04-26T01:54:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.740+0000 7f609b400640 1 -- 172.21.15.6:0/384013378 learned_addr learned my addr 172.21.15.6:0/384013378 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.741+0000 7f60a0e00640 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2d6740 0x55af7f2ede40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.741+0000 7f609b400640 1 -- 172.21.15.6:0/384013378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dc550 msgr2=0x55af7f2d5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.741+0000 7f609b400640 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dc550 0x55af7f2d5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.741+0000 7f609b400640 1 -- 172.21.15.6:0/384013378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2d6740 msgr2=0x55af7f2ede40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.741+0000 7f609b400640 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2d6740 0x55af7f2ede40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.741+0000 7f609b400640 1 -- 172.21.15.6:0/384013378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af7f2ee380 con 0x55af7f2e6e10 2024-04-26T01:54:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.742+0000 7f60a0e00640 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2d6740 0x55af7f2ede40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:54:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.742+0000 7f609b400640 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2d6200 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f609c10e290 tx=0x7f6088004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=666aba1fb8199ce7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.742+0000 7f6086000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:54:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.742+0000 7f6086000640 1 -- 172.21.15.6:0/384013378 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088013d80 con 0x55af7f2e6e10 2024-04-26T01:54:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.743+0000 7f60a5db6740 1 -- 172.21.15.6:0/384013378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f608c019d70 con 0x55af7f2e6e10 2024-04-26T01:54:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.743+0000 7f6086000640 1 -- 172.21.15.6:0/384013378 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f608801c030 con 0x55af7f2e6e10 2024-04-26T01:54:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.743+0000 7f6086000640 1 -- 172.21.15.6:0/384013378 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f6088039050 con 0x55af7f2e6e10 2024-04-26T01:54:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.743+0000 7f60a5db6740 10 client.26231 fetch_fsmap learned FSMap version 41 2024-04-26T01:54:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.743+0000 7f60a5db6740 1 -- 172.21.15.6:0/384013378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55af7f2ef6d0 con 0x55af7f2e6e10 2024-04-26T01:54:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.744+0000 7f6086000640 1 -- 172.21.15.6:0/384013378 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f608c019d70 con 0x55af7f2e6e10 2024-04-26T01:54:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.744+0000 7f6086000640 1 -- 172.21.15.6:0/384013378 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f60880053f0 con 0x55af7f2e6e10 2024-04-26T01:54:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.744+0000 7f60a5db6740 10 client.26231 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:54:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.744+0000 7f60a5db6740 10 client.26231 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:54:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.745+0000 7f60a5db6740 20 client.26231 populate_metadata read hostname 'smithi006' 2024-04-26T01:54:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.745+0000 7f60a5db6740 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af7f3622b0 0x55af7f3646d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.745+0000 7f60a5db6740 4 client.26231 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.745+0000 7f60a5db6740 1 -- 172.21.15.6:0/384013378 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55af7f24b320 con 0x55af7f3622b0 2024-04-26T01:54:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.745+0000 7f609be00640 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af7f3622b0 0x55af7f3646d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:47.745 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.746+0000 7f609be00640 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af7f3622b0 0x55af7f3646d0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33ef048a526ae81b server_cookie=82c4735853d1b11d in_seq=0 out_seq=0 2024-04-26T01:54:47.745 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.746+0000 7f6086000640 10 client.26231 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.807+0000 7f6086000640 1 -- 172.21.15.6:0/384013378 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55af7f24b320 con 0x55af7f3622b0 2024-04-26T01:54:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.807+0000 7f6086000640 10 client.26231 handle_command_reply: tid=0 2024-04-26T01:54:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.807+0000 7f60a5db6740 1 client.26231 shutdown 2024-04-26T01:54:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.808+0000 7f60a5db6740 1 -- 172.21.15.6:0/384013378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 msgr2=0x55af7f2d6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.808+0000 7f60a5db6740 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2d6200 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f609c10e290 tx=0x7f6088004730 comp rx=0 tx=0).stop 2024-04-26T01:54:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.808+0000 7f60a5db6740 1 -- 172.21.15.6:0/384013378 shutdown_connections 2024-04-26T01:54:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.808+0000 7f60a5db6740 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af7f3622b0 0x55af7f3646d0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.808+0000 7f60a5db6740 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af7f2d6740 0x55af7f2ede40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.808+0000 7f60a5db6740 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af7f2e6e10 0x55af7f2d6200 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.808+0000 7f60a5db6740 1 --2- 172.21.15.6:0/384013378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af7f2dc550 0x55af7f2d5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.808+0000 7f6086000640 0 client.26231 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:54:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.808+0000 7f60a5db6740 1 -- 172.21.15.6:0/384013378 >> 172.21.15.6:0/384013378 conn(0x55af7f2d8140 msgr2=0x55af7f2d2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.808+0000 7f60a5db6740 1 -- 172.21.15.6:0/384013378 shutdown_connections 2024-04-26T01:54:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.808+0000 7f60a5db6740 1 -- 172.21.15.6:0/384013378 wait complete. 2024-04-26T01:54:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.808+0000 7f60a5db6740 20 client.26231 trim_cache size 0 max 16384 2024-04-26T01:54:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:54:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:54:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:54:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:54:47.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.809+0000 7f60a2e00640 1 -- 172.21.15.6:0/3439794599 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f60800a3d10 msgr2=0x7f60800a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:47.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.809+0000 7f60a2e00640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f60800a3d10 0x7f60800a61d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55af7f2d9560 tx=0x7f6088005f90 comp rx=0 tx=0).stop 2024-04-26T01:54:47.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.809+0000 7f60a2e00640 1 -- 172.21.15.6:0/3439794599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f609c10c590 msgr2=0x7f609c1985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:47.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.809+0000 7f60a2e00640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f609c10c590 0x7f609c1985c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6094009dc0 tx=0x7f6094003ce0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.809+0000 7f60a2e00640 1 -- 172.21.15.6:0/3439794599 shutdown_connections 2024-04-26T01:54:47.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.809+0000 7f60a2e00640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f60800a3d10 0x7f60800a61d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.809+0000 7f60a2e00640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f609c103ae0 0x7f609c19e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.809+0000 7f60a2e00640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f609c10c590 0x7f609c1985c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.809+0000 7f60a2e00640 1 --2- 172.21.15.6:0/3439794599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f609c1031c0 0x7f609c19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:47.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.810+0000 7f60a2e00640 1 -- 172.21.15.6:0/3439794599 >> 172.21.15.6:0/3439794599 conn(0x7f609c077fb0 msgr2=0x7f609c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:47.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.810+0000 7f60a2e00640 1 -- 172.21.15.6:0/3439794599 shutdown_connections 2024-04-26T01:54:47.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:54:47.810+0000 7f60a2e00640 1 -- 172.21.15.6:0/3439794599 wait complete. 2024-04-26T01:54:48.607 INFO:tasks.workunit.client.0.smithi006.stdout: 1 316661 3.86 MB/sec execute 312 sec latency 38.701 ms 2024-04-26T01:54:48.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:48 smithi029 ceph-mon[32775]: pgmap v279: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 213 op/s 2024-04-26T01:54:48.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:48 smithi006 ceph-mon[28419]: pgmap v279: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 213 op/s 2024-04-26T01:54:48.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:48 smithi188 ceph-mon[32751]: pgmap v279: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 213 op/s 2024-04-26T01:54:49.608 INFO:tasks.workunit.client.0.smithi006.stdout: 1 317530 3.86 MB/sec execute 313 sec latency 25.403 ms 2024-04-26T01:54:50.608 INFO:tasks.workunit.client.0.smithi006.stdout: 1 319053 3.88 MB/sec execute 314 sec latency 18.672 ms 2024-04-26T01:54:50.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:50 smithi029 ceph-mon[32775]: pgmap v280: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-26T01:54:50.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:50 smithi006 ceph-mon[28419]: pgmap v280: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-26T01:54:50.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:50 smithi188 ceph-mon[32751]: pgmap v280: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-26T01:54:51.608 INFO:tasks.workunit.client.0.smithi006.stdout: 1 319773 3.88 MB/sec execute 315 sec latency 14.521 ms 2024-04-26T01:54:52.608 INFO:tasks.workunit.client.0.smithi006.stdout: 1 320805 3.89 MB/sec execute 316 sec latency 25.475 ms 2024-04-26T01:54:52.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:52 smithi029 ceph-mon[32775]: pgmap v281: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 230 op/s 2024-04-26T01:54:52.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:52 smithi006 ceph-mon[28419]: pgmap v281: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 230 op/s 2024-04-26T01:54:52.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:52 smithi188 ceph-mon[32751]: pgmap v281: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 230 op/s 2024-04-26T01:54:53.608 INFO:tasks.workunit.client.0.smithi006.stdout: 1 322175 3.90 MB/sec execute 317 sec latency 46.083 ms 2024-04-26T01:54:54.304 INFO:tasks.quiescer.fs.[cephfs]:Total duration (44.2) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T01:54:54.304 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.1 2024-04-26T01:54:54.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:54:54.608 INFO:tasks.workunit.client.0.smithi006.stdout: 1 323179 3.91 MB/sec execute 318 sec latency 20.006 ms 2024-04-26T01:54:54.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:54 smithi029 ceph-mon[32775]: pgmap v282: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-04-26T01:54:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:54:54.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:54 smithi006 ceph-mon[28419]: pgmap v282: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-04-26T01:54:54.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:54 smithi188 ceph-mon[32751]: pgmap v282: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-04-26T01:54:55.608 INFO:tasks.workunit.client.0.smithi006.stdout: 1 324180 3.91 MB/sec execute 319 sec latency 19.472 ms 2024-04-26T01:54:56.609 INFO:tasks.workunit.client.0.smithi006.stdout: 1 325376 3.91 MB/sec execute 320 sec latency 26.082 ms 2024-04-26T01:54:56.659 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:56 smithi006 ceph-mon[28419]: pgmap v283: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-04-26T01:54:56.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:56 smithi029 ceph-mon[32775]: pgmap v283: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-04-26T01:54:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:56 smithi188 ceph-mon[32751]: pgmap v283: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-04-26T01:54:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.296+0000 7f03b8000700 1 -- 172.21.15.6:0/3676662413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0113920 msgr2=0x7f03b0115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.296+0000 7f03b8000700 1 --2- 172.21.15.6:0/3676662413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0113920 0x7f03b0115d10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f03a800b0a0 tx=0x7f03a800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:54:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.297+0000 7f03b8000700 1 -- 172.21.15.6:0/3676662413 shutdown_connections 2024-04-26T01:54:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.297+0000 7f03b8000700 1 --2- 172.21.15.6:0/3676662413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0113920 0x7f03b0115d10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:57.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.297+0000 7f03b8000700 1 --2- 172.21.15.6:0/3676662413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03b0076f70 0x7f03b0077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:57.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.297+0000 7f03b8000700 1 --2- 172.21.15.6:0/3676662413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b0077890 0x7f03b01133e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:57.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.297+0000 7f03b8000700 1 -- 172.21.15.6:0/3676662413 >> 172.21.15.6:0/3676662413 conn(0x7f03b0100770 msgr2=0x7f03b0102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:57.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.297+0000 7f03b8000700 1 -- 172.21.15.6:0/3676662413 shutdown_connections 2024-04-26T01:54:57.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.297+0000 7f03b8000700 1 -- 172.21.15.6:0/3676662413 wait complete. 2024-04-26T01:54:57.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.299+0000 7f03b8000700 1 Processor -- start 2024-04-26T01:54:57.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.299+0000 7f03b8000700 1 -- start start 2024-04-26T01:54:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.299+0000 7f03b8000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0076f70 0x7f03b019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.299+0000 7f03b8000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b0077890 0x7f03b019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b8000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03b0113920 0x7f03b01a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b8000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f03b01192d0 con 0x7f03b0113920 2024-04-26T01:54:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b8000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03b0119150 con 0x7f03b0076f70 2024-04-26T01:54:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b8000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f03b0119450 con 0x7f03b0077890 2024-04-26T01:54:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b5600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0076f70 0x7f03b019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b5600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0076f70 0x7f03b019f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55012/0 (socket says 172.21.15.6:55012) 2024-04-26T01:54:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b5600700 1 -- 172.21.15.6:0/4006291338 learned_addr learned my addr 172.21.15.6:0/4006291338 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:54:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b4c00700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b0077890 0x7f03b019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b6000700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03b0113920 0x7f03b01a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b5600700 1 -- 172.21.15.6:0/4006291338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b0077890 msgr2=0x7f03b019fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b5600700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b0077890 0x7f03b019fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b5600700 1 -- 172.21.15.6:0/4006291338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03b0113920 msgr2=0x7f03b01a4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b5600700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03b0113920 0x7f03b01a4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b5600700 1 -- 172.21.15.6:0/4006291338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b01a4750 con 0x7f03b0076f70 2024-04-26T01:54:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.300+0000 7f03b6000700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03b0113920 0x7f03b01a4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:54:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.301+0000 7f03b5600700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0076f70 0x7f03b019f740 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f03a400eb30 tx=0x7f03a400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.301+0000 7f03ae000700 1 -- 172.21.15.6:0/4006291338 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a400cc90 con 0x7f03b0076f70 2024-04-26T01:54:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.301+0000 7f03ae000700 1 -- 172.21.15.6:0/4006291338 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03a40044d0 con 0x7f03b0076f70 2024-04-26T01:54:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.301+0000 7f03b8000700 1 -- 172.21.15.6:0/4006291338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03b01a4a40 con 0x7f03b0076f70 2024-04-26T01:54:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.301+0000 7f03ae000700 1 -- 172.21.15.6:0/4006291338 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4010430 con 0x7f03b0076f70 2024-04-26T01:54:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.301+0000 7f03b8000700 1 -- 172.21.15.6:0/4006291338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03b01ac2c0 con 0x7f03b0076f70 2024-04-26T01:54:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.302+0000 7f03b8000700 1 -- 172.21.15.6:0/4006291338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03b0069730 con 0x7f03b0076f70 2024-04-26T01:54:57.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.304+0000 7f03ae000700 1 -- 172.21.15.6:0/4006291338 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f03a4003ea0 con 0x7f03b0076f70 2024-04-26T01:54:57.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.305+0000 7f03ae000700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f03980a3e30 0x7f03980a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:57.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.306+0000 7f03ae000700 1 -- 172.21.15.6:0/4006291338 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f03a40d0ac0 con 0x7f03b0076f70 2024-04-26T01:54:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.306+0000 7f03b4c00700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f03980a3e30 0x7f03980a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.307+0000 7f03b4c00700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f03980a3e30 0x7f03980a62f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f03b0079fa0 tx=0x7f039c009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.310+0000 7f03ae000700 1 -- 172.21.15.6:0/4006291338 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03a409d0b0 con 0x7f03b0076f70 2024-04-26T01:54:57.608 INFO:tasks.workunit.client.0.smithi006.stdout: 1 326602 3.93 MB/sec execute 321 sec latency 21.445 ms 2024-04-26T01:54:57.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.634+0000 7f03b8000700 1 -- 172.21.15.6:0/4006291338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03b002df80 con 0x7f03b0076f70 2024-04-26T01:54:57.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.635+0000 7f03ae000700 1 -- 172.21.15.6:0/4006291338 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11853 (secure 0 0 0) 0x7f03a409c9b0 con 0x7f03b0076f70 2024-04-26T01:54:57.634 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:54:57.634 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":41,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:53:03.909439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:54:57.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.638+0000 7f03b8000700 1 -- 172.21.15.6:0/4006291338 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f03980a3e30 msgr2=0x7f03980a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:57.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.638+0000 7f03b8000700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f03980a3e30 0x7f03980a62f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f03b0079fa0 tx=0x7f039c009260 comp rx=0 tx=0).stop 2024-04-26T01:54:57.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.638+0000 7f03b8000700 1 -- 172.21.15.6:0/4006291338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0076f70 msgr2=0x7f03b019f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:57.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.638+0000 7f03b8000700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0076f70 0x7f03b019f740 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f03a400eb30 tx=0x7f03a400ee40 comp rx=0 tx=0).stop 2024-04-26T01:54:57.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.638+0000 7f03b8000700 1 -- 172.21.15.6:0/4006291338 shutdown_connections 2024-04-26T01:54:57.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.638+0000 7f03b8000700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f03980a3e30 0x7f03980a62f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:57.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.638+0000 7f03b8000700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03b0113920 0x7f03b01a4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:57.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.638+0000 7f03b8000700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0076f70 0x7f03b019f740 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:57.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.638+0000 7f03b8000700 1 --2- 172.21.15.6:0/4006291338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b0077890 0x7f03b019fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:57.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.638+0000 7f03b8000700 1 -- 172.21.15.6:0/4006291338 >> 172.21.15.6:0/4006291338 conn(0x7f03b0100770 msgr2=0x7f03b0102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:57.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.638+0000 7f03b8000700 1 -- 172.21.15.6:0/4006291338 shutdown_connections 2024-04-26T01:54:57.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:54:57.638+0000 7f03b8000700 1 -- 172.21.15.6:0/4006291338 wait complete. 2024-04-26T01:54:57.638 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 41 2024-04-26T01:54:58.438 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:58 smithi006 ceph-mon[28419]: pgmap v284: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 242 op/s 2024-04-26T01:54:58.539 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 82.1 --await-for 42.0' 2024-04-26T01:54:58.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 82.1 --await-for 42.0 2024-04-26T01:54:58.608 INFO:tasks.workunit.client.0.smithi006.stdout: 1 327264 3.92 MB/sec execute 322 sec latency 49.214 ms 2024-04-26T01:54:58.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:58 smithi029 ceph-mon[32775]: pgmap v284: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 242 op/s 2024-04-26T01:54:58.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:54:58 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/4006291338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:54:58.691 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:58 smithi188 ceph-mon[32751]: pgmap v284: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 242 op/s 2024-04-26T01:54:58.691 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:54:58 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/4006291338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:54:58.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:54:58 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/4006291338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:54:59.024 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:54:59.609 INFO:tasks.workunit.client.0.smithi006.stdout: 1 328728 3.93 MB/sec execute 323 sec latency 21.278 ms 2024-04-26T01:55:00.609 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330041 3.95 MB/sec execute 324 sec latency 40.513 ms 2024-04-26T01:55:00.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:00 smithi029 ceph-mon[32775]: pgmap v285: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-04-26T01:55:00.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:00 smithi006 ceph-mon[28419]: pgmap v285: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-04-26T01:55:00.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:00 smithi188 ceph-mon[32751]: pgmap v285: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-04-26T01:55:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.960+0000 7fbaca600640 1 -- 172.21.15.6:0/4153830126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac4100ed0 msgr2=0x7fbac41012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.960+0000 7fbaca600640 1 --2- 172.21.15.6:0/4153830126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac4100ed0 0x7fbac41012b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009ca0 tx=0x7fbab402a8c0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.960+0000 7fbaca600640 1 -- 172.21.15.6:0/4153830126 shutdown_connections 2024-04-26T01:55:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.960+0000 7fbaca600640 1 --2- 172.21.15.6:0/4153830126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbac4105fa0 0x7fbac410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.960+0000 7fbaca600640 1 --2- 172.21.15.6:0/4153830126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac4101880 0x7fbac41058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.960+0000 7fbaca600640 1 --2- 172.21.15.6:0/4153830126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac4100ed0 0x7fbac41012b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.960+0000 7fbaca600640 1 -- 172.21.15.6:0/4153830126 >> 172.21.15.6:0/4153830126 conn(0x7fbac40fc730 msgr2=0x7fbac40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.960+0000 7fbaca600640 1 -- 172.21.15.6:0/4153830126 shutdown_connections 2024-04-26T01:55:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.960+0000 7fbaca600640 1 -- 172.21.15.6:0/4153830126 wait complete. 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.961+0000 7fbaca600640 1 Processor -- start 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.962+0000 7fbaca600640 1 -- start start 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.962+0000 7fbaca600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac4100ed0 0x7fbac41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.962+0000 7fbaca600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac4105fa0 0x7fbac4198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.962+0000 7fbaca600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbac419ce40 0x7fbac419d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.962+0000 7fbaca600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbac4110f00 con 0x7fbac4100ed0 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.962+0000 7fbaca600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbac4110d80 con 0x7fbac4105fa0 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.962+0000 7fbaca600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbac4111080 con 0x7fbac419ce40 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.962+0000 7fbac9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac4100ed0 0x7fbac41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.962+0000 7fbac9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac4100ed0 0x7fbac41985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60788/0 (socket says 172.21.15.6:60788) 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.962+0000 7fbac9200640 1 -- 172.21.15.6:0/2932572819 learned_addr learned my addr 172.21.15.6:0/2932572819 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.962+0000 7fbac3e00640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac4105fa0 0x7fbac4198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.962+0000 7fbac9c00640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbac419ce40 0x7fbac419d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.963+0000 7fbac3e00640 1 -- 172.21.15.6:0/2932572819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbac419ce40 msgr2=0x7fbac419d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.963+0000 7fbac3e00640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbac419ce40 0x7fbac419d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.963+0000 7fbac3e00640 1 -- 172.21.15.6:0/2932572819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac4100ed0 msgr2=0x7fbac41985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.963+0000 7fbac3e00640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac4100ed0 0x7fbac41985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.963+0000 7fbac3e00640 1 -- 172.21.15.6:0/2932572819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac419d930 con 0x7fbac4105fa0 2024-04-26T01:55:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.963+0000 7fbac3e00640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac4105fa0 0x7fbac4198b10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00e9c0 tx=0x7fbaac00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.963+0000 7fbac1600640 1 -- 172.21.15.6:0/2932572819 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac00ccc0 con 0x7fbac4105fa0 2024-04-26T01:55:00.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.963+0000 7fbac1600640 1 -- 172.21.15.6:0/2932572819 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaac0108e0 con 0x7fbac4105fa0 2024-04-26T01:55:00.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.963+0000 7fbac1600640 1 -- 172.21.15.6:0/2932572819 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0187c0 con 0x7fbac4105fa0 2024-04-26T01:55:00.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.963+0000 7fbaca600640 1 -- 172.21.15.6:0/2932572819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac419dc20 con 0x7fbac4105fa0 2024-04-26T01:55:00.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.964+0000 7fbaca600640 1 -- 172.21.15.6:0/2932572819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac41a5460 con 0x7fbac4105fa0 2024-04-26T01:55:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.967+0000 7fbaceba2740 1 Processor -- start 2024-04-26T01:55:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.967+0000 7fbac1600640 1 -- 172.21.15.6:0/2932572819 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fbaac010430 con 0x7fbac4105fa0 2024-04-26T01:55:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.967+0000 7fbaceba2740 1 -- start start 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.967+0000 7fbaceba2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x5639257541a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.967+0000 7fbaceba2740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x56392575ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.967+0000 7fbaceba2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x563925761390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.967+0000 7fbaceba2740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5639256b5900 con 0x5639257546e0 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.967+0000 7fbaceba2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56392565bef0 con 0x56392575efa0 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.967+0000 7fbaceba2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5639256b18c0 con 0x563925753dc0 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.968+0000 7fbac9c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x5639257541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.968+0000 7fbac9c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x5639257541a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:46088/0 (socket says 172.21.15.6:46088) 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.968+0000 7fbac9c00640 1 -- 172.21.15.6:0/2511279456 learned_addr learned my addr 172.21.15.6:0/2511279456 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.968+0000 7fbac9200640 1 --2- 172.21.15.6:0/2511279456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x56392575ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.968+0000 7fbac3e00640 1 --2- 172.21.15.6:0/2511279456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x563925761390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.968+0000 7fbac9c00640 1 -- 172.21.15.6:0/2511279456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 msgr2=0x563925761390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.968+0000 7fbac9c00640 1 --2- 172.21.15.6:0/2511279456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x563925761390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.968+0000 7fbac9c00640 1 -- 172.21.15.6:0/2511279456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 msgr2=0x56392575ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.968+0000 7fbac9c00640 1 --2- 172.21.15.6:0/2511279456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x56392575ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.968+0000 7fbac9c00640 1 -- 172.21.15.6:0/2511279456 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639253c8210 con 0x563925753dc0 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.968+0000 7fbac9200640 1 --2- 172.21.15.6:0/2511279456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x56392575ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.968+0000 7fbac9c00640 1 --2- 172.21.15.6:0/2511279456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x5639257541a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009c20 tx=0x7fbabc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8ffecf3703eea6f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbaba000640 1 -- 172.21.15.6:0/2511279456 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc01e070 con 0x563925753dc0 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbac1600640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbaa80a3d80 0x7fbaa80a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbaba000640 1 -- 172.21.15.6:0/2511279456 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbabc019ae0 con 0x563925753dc0 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbaba000640 1 -- 172.21.15.6:0/2511279456 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc00e960 con 0x563925753dc0 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbac1600640 1 -- 172.21.15.6:0/2932572819 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7fbaac0d0b50 con 0x7fbac4105fa0 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbac9200640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbaa80a3d80 0x7fbaa80a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbaceba2740 1 -- 172.21.15.6:0/2511279456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 msgr2=0x5639257541a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2511279456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x5639257541a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009c20 tx=0x7fbabc011810 comp rx=0 tx=0).stop 2024-04-26T01:55:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbaceba2740 1 -- 172.21.15.6:0/2511279456 shutdown_connections 2024-04-26T01:55:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2511279456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x56392575ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2511279456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x563925761390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2511279456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x5639257541a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.969+0000 7fbaceba2740 1 -- 172.21.15.6:0/2511279456 >> 172.21.15.6:0/2511279456 conn(0x5639257502d0 msgr2=0x5639257506b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.970+0000 7fbaceba2740 1 -- 172.21.15.6:0/2511279456 shutdown_connections 2024-04-26T01:55:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.970+0000 7fbaceba2740 1 -- 172.21.15.6:0/2511279456 wait complete. 2024-04-26T01:55:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.970+0000 7fbac9200640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbaa80a3d80 0x7fbaa80a6240 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbab402f500 tx=0x7fbab402f310 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.970+0000 7fbaceba2740 1 -- start start 2024-04-26T01:55:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.970+0000 7fbaceba2740 1 Processor -- start 2024-04-26T01:55:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbaceba2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x56392575bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbaceba2740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x56392575c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbaceba2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x56392575ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbaceba2740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x563925761bd0 con 0x5639257546e0 2024-04-26T01:55:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbaceba2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56392566f310 con 0x56392575efa0 2024-04-26T01:55:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbaceba2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563925761d50 con 0x563925753dc0 2024-04-26T01:55:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbaceba2740 10 client.0 _ll_register_callbacks cb 0x56392563dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbac9200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x56392575bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbac9c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x56392575ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbac9c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x56392575ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:35322/0 (socket says 172.21.15.6:35322) 2024-04-26T01:55:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbac9c00640 1 -- 172.21.15.6:0/2754395937 learned_addr learned my addr 172.21.15.6:0/2754395937 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbac3e00640 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x56392575c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbac9c00640 1 -- 172.21.15.6:0/2754395937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 msgr2=0x56392575bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbac9c00640 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x56392575bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbac9c00640 1 -- 172.21.15.6:0/2754395937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 msgr2=0x56392575c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbac9c00640 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x56392575c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbac9c00640 1 -- 172.21.15.6:0/2754395937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563925494620 con 0x56392575efa0 2024-04-26T01:55:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.971+0000 7fbac3e00640 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x56392575c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.972+0000 7fbac9c00640 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x56392575ca10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009870 tx=0x7fbabc019660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8187def3f89c65a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.972+0000 7fbab2a00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:55:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.972+0000 7fbab2a00640 1 -- 172.21.15.6:0/2754395937 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc01e050 con 0x56392575efa0 2024-04-26T01:55:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.972+0000 7fbaceba2740 1 -- 172.21.15.6:0/2754395937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbabc019ae0 con 0x56392575efa0 2024-04-26T01:55:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.972+0000 7fbab2a00640 1 -- 172.21.15.6:0/2754395937 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7fbabc004020 con 0x56392575efa0 2024-04-26T01:55:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.973+0000 7fbab2a00640 1 -- 172.21.15.6:0/2754395937 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fbabc032050 con 0x56392575efa0 2024-04-26T01:55:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.973+0000 7fbaceba2740 10 client.26249 fetch_fsmap learned FSMap version 41 2024-04-26T01:55:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.973+0000 7fbaceba2740 1 -- 172.21.15.6:0/2754395937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x56392576de90 con 0x56392575efa0 2024-04-26T01:55:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.973+0000 7fbab2a00640 1 -- 172.21.15.6:0/2754395937 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbabc019ae0 con 0x56392575efa0 2024-04-26T01:55:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.973+0000 7fbab2a00640 1 -- 172.21.15.6:0/2754395937 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7fbabc0046c0 con 0x56392575efa0 2024-04-26T01:55:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.973+0000 7fbaceba2740 10 client.26249 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:55:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.973+0000 7fbaceba2740 10 client.26249 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.974+0000 7fbaceba2740 20 client.26249 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.974+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56392576f2a0 0x563925771690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.974+0000 7fbaceba2740 4 client.26249 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.974+0000 7fbaceba2740 1 -- 172.21.15.6:0/2754395937 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639256c98b0 con 0x56392576f2a0 2024-04-26T01:55:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.974+0000 7fbac9200640 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56392576f2a0 0x563925771690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.975+0000 7fbac9200640 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56392576f2a0 0x563925771690 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7b561ba6b73790a server_cookie=7f3ebe83e3a268ec in_seq=0 out_seq=0 2024-04-26T01:55:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.975+0000 7fbab2a00640 10 client.26249 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.976+0000 7fbab2a00640 1 -- 172.21.15.6:0/2754395937 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5639256c98b0 con 0x56392576f2a0 2024-04-26T01:55:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.976+0000 7fbab2a00640 10 client.26249 handle_command_reply: tid=0 2024-04-26T01:55:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.976+0000 7fbaceba2740 1 client.26249 shutdown 2024-04-26T01:55:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.976+0000 7fbaceba2740 1 -- 172.21.15.6:0/2754395937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 msgr2=0x56392575ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.976+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x56392575ca10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009870 tx=0x7fbabc019660 comp rx=0 tx=0).stop 2024-04-26T01:55:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.976+0000 7fbaceba2740 1 -- 172.21.15.6:0/2754395937 shutdown_connections 2024-04-26T01:55:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.976+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56392576f2a0 0x563925771690 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.976+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x56392575c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.976+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x56392575ca10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.976+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2754395937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x56392575bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.976+0000 7fbab2a00640 0 client.26249 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.976+0000 7fbaceba2740 1 -- 172.21.15.6:0/2754395937 >> 172.21.15.6:0/2754395937 conn(0x5639257502d0 msgr2=0x563925756050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.977+0000 7fbaceba2740 1 -- 172.21.15.6:0/2754395937 shutdown_connections 2024-04-26T01:55:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.977+0000 7fbaceba2740 1 -- 172.21.15.6:0/2754395937 wait complete. 2024-04-26T01:55:00.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.977+0000 7fbaceba2740 20 client.26249 trim_cache size 0 max 16384 2024-04-26T01:55:00.996 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.998+0000 7fbaceba2740 1 Processor -- start 2024-04-26T01:55:00.996 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.998+0000 7fbaceba2740 1 -- start start 2024-04-26T01:55:00.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.998+0000 7fbaceba2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563925753dc0 0x56392576aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.998+0000 7fbaceba2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5639257546e0 0x563925756780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.998+0000 7fbaceba2740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56392575efa0 0x56392575ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.998+0000 7fbaceba2740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbac41ae2f0 con 0x56392575efa0 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.998+0000 7fbaceba2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbac41abb60 con 0x563925753dc0 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.998+0000 7fbaceba2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbac41a99d0 con 0x5639257546e0 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.998+0000 7fbac3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5639257546e0 0x563925756780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.998+0000 7fbac9c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56392575efa0 0x56392575ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.999+0000 7fbac3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5639257546e0 0x563925756780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:46112/0 (socket says 172.21.15.6:46112) 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.999+0000 7fbac3e00640 1 -- 172.21.15.6:0/3021840537 learned_addr learned my addr 172.21.15.6:0/3021840537 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.999+0000 7fbac9200640 1 --2- 172.21.15.6:0/3021840537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563925753dc0 0x56392576aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.999+0000 7fbac3e00640 1 -- 172.21.15.6:0/3021840537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563925753dc0 msgr2=0x56392576aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.999+0000 7fbac3e00640 1 --2- 172.21.15.6:0/3021840537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563925753dc0 0x56392576aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.999+0000 7fbac3e00640 1 -- 172.21.15.6:0/3021840537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56392575efa0 msgr2=0x56392575ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.999+0000 7fbac3e00640 1 --2- 172.21.15.6:0/3021840537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56392575efa0 0x56392575ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.999+0000 7fbac3e00640 1 -- 172.21.15.6:0/3021840537 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56392575a4e0 con 0x5639257546e0 2024-04-26T01:55:01.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.999+0000 7fbac9c00640 1 --2- 172.21.15.6:0/3021840537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56392575efa0 0x56392575ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:01.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.999+0000 7fbac9200640 1 --2- 172.21.15.6:0/3021840537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563925753dc0 0x56392576aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:01.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:00.999+0000 7fbac3e00640 1 --2- 172.21.15.6:0/3021840537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5639257546e0 0x563925756780 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbaac098f50 tx=0x7fbaac00ce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc93c4522030cc91 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:01.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.000+0000 7fbab9600640 1 -- 172.21.15.6:0/3021840537 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac09caf0 con 0x5639257546e0 2024-04-26T01:55:01.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.000+0000 7fbab9600640 1 -- 172.21.15.6:0/3021840537 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaac0a0df0 con 0x5639257546e0 2024-04-26T01:55:01.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.000+0000 7fbab9600640 1 -- 172.21.15.6:0/3021840537 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0ce470 con 0x5639257546e0 2024-04-26T01:55:01.003 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.000+0000 7fbaceba2740 1 -- 172.21.15.6:0/3021840537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5639257546e0 msgr2=0x563925756780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.003 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.000+0000 7fbaceba2740 1 --2- 172.21.15.6:0/3021840537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5639257546e0 0x563925756780 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbaac098f50 tx=0x7fbaac00ce60 comp rx=0 tx=0).stop 2024-04-26T01:55:01.003 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.000+0000 7fbaceba2740 1 -- 172.21.15.6:0/3021840537 shutdown_connections 2024-04-26T01:55:01.003 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.000+0000 7fbaceba2740 1 --2- 172.21.15.6:0/3021840537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56392575efa0 0x56392575ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.000+0000 7fbaceba2740 1 --2- 172.21.15.6:0/3021840537 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5639257546e0 0x563925756780 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.000+0000 7fbaceba2740 1 --2- 172.21.15.6:0/3021840537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563925753dc0 0x56392576aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.000+0000 7fbaceba2740 1 -- 172.21.15.6:0/3021840537 >> 172.21.15.6:0/3021840537 conn(0x5639257502d0 msgr2=0x56392575d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.001+0000 7fbaceba2740 1 -- 172.21.15.6:0/3021840537 shutdown_connections 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.001+0000 7fbaceba2740 1 -- 172.21.15.6:0/3021840537 wait complete. 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.001+0000 7fbaceba2740 1 -- start start 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.001+0000 7fbaceba2740 1 Processor -- start 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbaceba2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x563925764ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbaceba2740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x563925764ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbaceba2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x563925765530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbaceba2740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbac41aa020 con 0x5639257546e0 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbaceba2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbac41ab490 con 0x56392575efa0 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbaceba2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbac41aa6f0 con 0x563925753dc0 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbaceba2740 10 client.0 _ll_register_callbacks cb 0x7fbabc00e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:01.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbac3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x563925764ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbac9c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x563925765530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbac3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x563925764ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60824/0 (socket says 172.21.15.6:60824) 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbac3e00640 1 -- 172.21.15.6:0/2824115514 learned_addr learned my addr 172.21.15.6:0/2824115514 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbac9200640 1 --2- 172.21.15.6:0/2824115514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x563925764ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbac9c00640 1 -- 172.21.15.6:0/2824115514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 msgr2=0x563925764ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbac9c00640 1 --2- 172.21.15.6:0/2824115514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x563925764ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbac9c00640 1 -- 172.21.15.6:0/2824115514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 msgr2=0x563925764ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbac9c00640 1 --2- 172.21.15.6:0/2824115514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x563925764ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.002+0000 7fbac9c00640 1 -- 172.21.15.6:0/2824115514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639257d4720 con 0x56392575efa0 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.003+0000 7fbac9c00640 1 --2- 172.21.15.6:0/2824115514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x563925765530 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009710 tx=0x7fbabc019760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f39fc0d3abddc3ab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.003+0000 7fbab2a00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.003+0000 7fbab2a00640 1 -- 172.21.15.6:0/2824115514 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc01e040 con 0x56392575efa0 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.003+0000 7fbab2a00640 1 -- 172.21.15.6:0/2824115514 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7fbabc004600 con 0x56392575efa0 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.003+0000 7fbaceba2740 1 -- 172.21.15.6:0/2824115514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbaac0a0df0 con 0x56392575efa0 2024-04-26T01:55:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.003+0000 7fbab2a00640 1 -- 172.21.15.6:0/2824115514 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fbabc044050 con 0x56392575efa0 2024-04-26T01:55:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.003+0000 7fbaceba2740 10 client.26255 fetch_fsmap learned FSMap version 41 2024-04-26T01:55:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.003+0000 7fbaceba2740 1 -- 172.21.15.6:0/2824115514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5639257d5a30 con 0x56392575efa0 2024-04-26T01:55:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.004+0000 7fbab2a00640 1 -- 172.21.15.6:0/2824115514 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaac0a0df0 con 0x56392575efa0 2024-04-26T01:55:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.004+0000 7fbab2a00640 1 -- 172.21.15.6:0/2824115514 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7fbabc005300 con 0x56392575efa0 2024-04-26T01:55:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.004+0000 7fbaceba2740 10 client.26255 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:55:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.004+0000 7fbaceba2740 10 client.26255 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.004+0000 7fbaceba2740 20 client.26255 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.004+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2824115514 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5639257d6080 0x5639257d84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.004+0000 7fbaceba2740 4 client.26255 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.004+0000 7fbaceba2740 1 -- 172.21.15.6:0/2824115514 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5639256c3320 con 0x5639257d6080 2024-04-26T01:55:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.005+0000 7fbac9200640 1 --2- 172.21.15.6:0/2824115514 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5639257d6080 0x5639257d84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.005+0000 7fbac9200640 1 --2- 172.21.15.6:0/2824115514 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5639257d6080 0x5639257d84a0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73ed2284db37c89a server_cookie=903c53d9905ae8dd in_seq=0 out_seq=0 2024-04-26T01:55:01.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.005+0000 7fbab2a00640 10 client.26255 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.058+0000 7fbab2a00640 1 -- 172.21.15.6:0/2824115514 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5639256c3320 con 0x5639257d6080 2024-04-26T01:55:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.058+0000 7fbab2a00640 10 client.26255 handle_command_reply: tid=0 2024-04-26T01:55:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.059+0000 7fbaceba2740 1 client.26255 shutdown 2024-04-26T01:55:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.059+0000 7fbaceba2740 1 -- 172.21.15.6:0/2824115514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 msgr2=0x563925765530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.059+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2824115514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x563925765530 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009710 tx=0x7fbabc019760 comp rx=0 tx=0).stop 2024-04-26T01:55:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.059+0000 7fbaceba2740 1 -- 172.21.15.6:0/2824115514 shutdown_connections 2024-04-26T01:55:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.059+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2824115514 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5639257d6080 0x5639257d84a0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.059+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2824115514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639257546e0 0x563925764ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.059+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2824115514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56392575efa0 0x563925765530 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.059+0000 7fbab2a00640 0 client.26255 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.059+0000 7fbaceba2740 1 --2- 172.21.15.6:0/2824115514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563925753dc0 0x563925764ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.059+0000 7fbaceba2740 1 -- 172.21.15.6:0/2824115514 >> 172.21.15.6:0/2824115514 conn(0x5639257502d0 msgr2=0x56392574b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.060+0000 7fbaceba2740 1 -- 172.21.15.6:0/2824115514 shutdown_connections 2024-04-26T01:55:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.060+0000 7fbaceba2740 1 -- 172.21.15.6:0/2824115514 wait complete. 2024-04-26T01:55:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.060+0000 7fbaceba2740 20 client.26255 trim_cache size 0 max 16384 2024-04-26T01:55:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:55:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:55:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:55:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:55:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.061+0000 7fbaca600640 1 -- 172.21.15.6:0/2932572819 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbaa80a3d80 msgr2=0x7fbaa80a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.061+0000 7fbaca600640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbaa80a3d80 0x7fbaa80a6240 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbab402f500 tx=0x7fbab402f310 comp rx=0 tx=0).stop 2024-04-26T01:55:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.061+0000 7fbaca600640 1 -- 172.21.15.6:0/2932572819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac4105fa0 msgr2=0x7fbac4198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.061+0000 7fbaca600640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac4105fa0 0x7fbac4198b10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00e9c0 tx=0x7fbaac00ee80 comp rx=0 tx=0).stop 2024-04-26T01:55:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.061+0000 7fbaca600640 1 -- 172.21.15.6:0/2932572819 shutdown_connections 2024-04-26T01:55:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.061+0000 7fbaca600640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbaa80a3d80 0x7fbaa80a6240 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.061+0000 7fbaca600640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbac419ce40 0x7fbac419d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.061+0000 7fbaca600640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac4105fa0 0x7fbac4198b10 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.061+0000 7fbaca600640 1 --2- 172.21.15.6:0/2932572819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac4100ed0 0x7fbac41985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.061+0000 7fbaca600640 1 -- 172.21.15.6:0/2932572819 >> 172.21.15.6:0/2932572819 conn(0x7fbac40fc730 msgr2=0x7fbac4069ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.061+0000 7fbaca600640 1 -- 172.21.15.6:0/2932572819 shutdown_connections 2024-04-26T01:55:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:01.062+0000 7fbaca600640 1 -- 172.21.15.6:0/2932572819 wait complete. 2024-04-26T01:55:01.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.420+0000 7f51cae00700 1 -- 172.21.15.6:0/2066695749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51c4105870 msgr2=0x7f51c41098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.420+0000 7f51cae00700 1 --2- 172.21.15.6:0/2066695749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51c4105870 0x7f51c41098c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51b00099c0 tx=0x7f51b0009cd0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.421+0000 7f51cae00700 1 -- 172.21.15.6:0/2066695749 shutdown_connections 2024-04-26T01:55:01.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.421+0000 7f51cae00700 1 --2- 172.21.15.6:0/2066695749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51c4109fa0 0x7f51c4111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.421+0000 7f51cae00700 1 --2- 172.21.15.6:0/2066695749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4104ec0 0x7f51c41052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.421+0000 7f51cae00700 1 --2- 172.21.15.6:0/2066695749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51c4105870 0x7f51c41098c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.421+0000 7f51cae00700 1 -- 172.21.15.6:0/2066695749 >> 172.21.15.6:0/2066695749 conn(0x7f51c4100750 msgr2=0x7f51c4102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:01.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.421+0000 7f51cae00700 1 -- 172.21.15.6:0/2066695749 shutdown_connections 2024-04-26T01:55:01.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.421+0000 7f51cae00700 1 -- 172.21.15.6:0/2066695749 wait complete. 2024-04-26T01:55:01.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.422+0000 7f51cae00700 1 Processor -- start 2024-04-26T01:55:01.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.422+0000 7f51cae00700 1 -- start start 2024-04-26T01:55:01.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.423+0000 7f51cae00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51c4104ec0 0x7f51c419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.423+0000 7f51cae00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51c4105870 0x7f51c419b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.423+0000 7f51cae00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4109fa0 0x7f51c419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.423+0000 7f51cae00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f51c4114e80 con 0x7f51c4104ec0 2024-04-26T01:55:01.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.423+0000 7f51cae00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f51c4114d00 con 0x7f51c4109fa0 2024-04-26T01:55:01.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.423+0000 7f51cae00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f51c4115000 con 0x7f51c4105870 2024-04-26T01:55:01.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.423+0000 7f51c3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51c4105870 0x7f51c419b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.423+0000 7f51c3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51c4104ec0 0x7f51c419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.423+0000 7f51c3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51c4105870 0x7f51c419b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:46136/0 (socket says 172.21.15.6:46136) 2024-04-26T01:55:01.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.423+0000 7f51c3400700 1 -- 172.21.15.6:0/76095137 learned_addr learned my addr 172.21.15.6:0/76095137 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:01.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.423+0000 7f51c8e00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4109fa0 0x7f51c419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.424+0000 7f51c8e00700 1 -- 172.21.15.6:0/76095137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51c4105870 msgr2=0x7f51c419b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.424+0000 7f51c8e00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51c4105870 0x7f51c419b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.424+0000 7f51c8e00700 1 -- 172.21.15.6:0/76095137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51c4104ec0 msgr2=0x7f51c419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.424+0000 7f51c8e00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51c4104ec0 0x7f51c419b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.424+0000 7f51c8e00700 1 -- 172.21.15.6:0/76095137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c41a02c0 con 0x7f51c4109fa0 2024-04-26T01:55:01.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.424+0000 7f51c3400700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51c4105870 0x7f51c419b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:01.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.424+0000 7f51c8e00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4109fa0 0x7f51c419fb80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00efc0 tx=0x7f51bc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:01.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.424+0000 7f51c0c00700 1 -- 172.21.15.6:0/76095137 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc0092e0 con 0x7f51c4109fa0 2024-04-26T01:55:01.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.424+0000 7f51c0c00700 1 -- 172.21.15.6:0/76095137 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51bc004bb0 con 0x7f51c4109fa0 2024-04-26T01:55:01.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.424+0000 7f51cae00700 1 -- 172.21.15.6:0/76095137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51c41a0550 con 0x7f51c4109fa0 2024-04-26T01:55:01.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.425+0000 7f51cae00700 1 -- 172.21.15.6:0/76095137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51c41a7e30 con 0x7f51c4109fa0 2024-04-26T01:55:01.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.425+0000 7f51c0c00700 1 -- 172.21.15.6:0/76095137 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc004020 con 0x7f51c4109fa0 2024-04-26T01:55:01.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.428+0000 7f51c0c00700 1 -- 172.21.15.6:0/76095137 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f51bc01b030 con 0x7f51c4109fa0 2024-04-26T01:55:01.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.430+0000 7f51c0c00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f51a80a3ed0 0x7f51a80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.430+0000 7f51c0c00700 1 -- 172.21.15.6:0/76095137 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f51bc0d00d0 con 0x7f51c4109fa0 2024-04-26T01:55:01.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.431+0000 7f51c3e00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f51a80a3ed0 0x7f51a80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.431+0000 7f51d5860b80 1 Processor -- start 2024-04-26T01:55:01.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.431+0000 7f51d5860b80 1 -- start start 2024-04-26T01:55:01.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.432+0000 7f51d5860b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364e9dd0 0x55bd364ea1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.432+0000 7f51d5860b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.432+0000 7f51d5860b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd364f7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.432+0000 7f51c3e00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f51a80a3ed0 0x7f51a80a6390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009c60 tx=0x7f51b4009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:01.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.432+0000 7f51d5860b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55bd3643d020 con 0x55bd364f5560 2024-04-26T01:55:01.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.432+0000 7f51d5860b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bd363e38c0 con 0x55bd364e9dd0 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.432+0000 7f51d5860b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bd36438fe0 con 0x55bd364ea6f0 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.432+0000 7f51c3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.432+0000 7f51c3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364e9dd0 0x55bd364ea1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.432+0000 7f51c3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:46144/0 (socket says 172.21.15.6:46144) 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.432+0000 7f51c3400700 1 -- 172.21.15.6:0/1611417656 learned_addr learned my addr 172.21.15.6:0/1611417656 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.432+0000 7f51c8e00700 1 --2- 172.21.15.6:0/1611417656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd364f7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.433+0000 7f51c3400700 1 -- 172.21.15.6:0/1611417656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364e9dd0 msgr2=0x55bd364ea1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.433+0000 7f51c3400700 1 --2- 172.21.15.6:0/1611417656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364e9dd0 0x55bd364ea1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.433+0000 7f51c3400700 1 -- 172.21.15.6:0/1611417656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 msgr2=0x55bd364f7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.433+0000 7f51c3400700 1 --2- 172.21.15.6:0/1611417656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd364f7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.433+0000 7f51c3400700 1 -- 172.21.15.6:0/1611417656 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd363d9810 con 0x55bd364ea6f0 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.433+0000 7f51c3e00700 1 --2- 172.21.15.6:0/1611417656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364e9dd0 0x55bd364ea1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:01.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.433+0000 7f51c8e00700 1 --2- 172.21.15.6:0/1611417656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd364f7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:01.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.433+0000 7f51c3400700 1 --2- 172.21.15.6:0/1611417656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f5020 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f51b0003710 tx=0x7f51b0005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbfb6a2ffe07167b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:01.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.433+0000 7f51b9600700 1 -- 172.21.15.6:0/1611417656 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0028070 con 0x55bd364ea6f0 2024-04-26T01:55:01.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.434+0000 7f51b9600700 1 -- 172.21.15.6:0/1611417656 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51b0005570 con 0x55bd364ea6f0 2024-04-26T01:55:01.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.434+0000 7f51b9600700 1 -- 172.21.15.6:0/1611417656 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0003d20 con 0x55bd364ea6f0 2024-04-26T01:55:01.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.434+0000 7f51d5860b80 1 -- 172.21.15.6:0/1611417656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 msgr2=0x55bd364f5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.434+0000 7f51d5860b80 1 --2- 172.21.15.6:0/1611417656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f5020 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f51b0003710 tx=0x7f51b0005bc0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.435+0000 7f51d5860b80 1 -- 172.21.15.6:0/1611417656 shutdown_connections 2024-04-26T01:55:01.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.435+0000 7f51d5860b80 1 --2- 172.21.15.6:0/1611417656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd364f7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.435+0000 7f51d5860b80 1 --2- 172.21.15.6:0/1611417656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364e9dd0 0x55bd364ea1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.435+0000 7f51d5860b80 1 --2- 172.21.15.6:0/1611417656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f5020 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.435+0000 7f51d5860b80 1 -- 172.21.15.6:0/1611417656 >> 172.21.15.6:0/1611417656 conn(0x55bd364190e0 msgr2=0x55bd364e62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:01.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.435+0000 7f51d5860b80 1 -- 172.21.15.6:0/1611417656 shutdown_connections 2024-04-26T01:55:01.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.435+0000 7f51d5860b80 1 -- 172.21.15.6:0/1611417656 wait complete. 2024-04-26T01:55:01.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.436+0000 7f51d5860b80 1 -- start start 2024-04-26T01:55:01.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.436+0000 7f51d5860b80 1 Processor -- start 2024-04-26T01:55:01.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.436+0000 7f51d5860b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364e9dd0 0x55bd364f4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.436+0000 7f51d5860b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.436+0000 7f51d5860b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd364f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51d5860b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55bd364f7ff0 con 0x55bd364f5560 2024-04-26T01:55:01.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51d5860b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bd363f69f0 con 0x55bd364e9dd0 2024-04-26T01:55:01.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51d5860b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bd364f8170 con 0x55bd364ea6f0 2024-04-26T01:55:01.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51d5860b80 10 client.0 _ll_register_callbacks cb 0x55bd363d58b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:01.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51c3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364e9dd0 0x55bd364f4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51c3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51c3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364e9dd0 0x55bd364f4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:35380/0 (socket says 172.21.15.6:35380) 2024-04-26T01:55:01.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51c3e00700 1 -- 172.21.15.6:0/1136324226 learned_addr learned my addr 172.21.15.6:0/1136324226 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:01.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51c8e00700 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd364f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51c3400700 1 -- 172.21.15.6:0/1136324226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364e9dd0 msgr2=0x55bd364f4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:01.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51c3400700 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364e9dd0 0x55bd364f4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51c3400700 1 -- 172.21.15.6:0/1136324226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 msgr2=0x55bd364f24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:01.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51c3400700 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd364f24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.437+0000 7f51c3400700 1 -- 172.21.15.6:0/1136324226 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd3638aa60 con 0x55bd364ea6f0 2024-04-26T01:55:01.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.438+0000 7f51c8e00700 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd364f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:55:01.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.438+0000 7f51c3400700 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f1f80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f51b0003760 tx=0x7f51b000b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c01a9163a01746cd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:01.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.438+0000 7f51ae000700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:55:01.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.438+0000 7f51ae000700 1 -- 172.21.15.6:0/1136324226 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0028070 con 0x55bd364ea6f0 2024-04-26T01:55:01.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.438+0000 7f51ae000700 1 -- 172.21.15.6:0/1136324226 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f51b0003ba0 con 0x55bd364ea6f0 2024-04-26T01:55:01.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.438+0000 7f51d5860b80 1 -- 172.21.15.6:0/1136324226 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51b0005570 con 0x55bd364ea6f0 2024-04-26T01:55:01.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.438+0000 7f51ae000700 1 -- 172.21.15.6:0/1136324226 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f51b0003da0 con 0x55bd364ea6f0 2024-04-26T01:55:01.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.438+0000 7f51d5860b80 10 client.26311 fetch_fsmap learned FSMap version 41 2024-04-26T01:55:01.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.438+0000 7f51d5860b80 1 -- 172.21.15.6:0/1136324226 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55bd363d3c80 con 0x55bd364ea6f0 2024-04-26T01:55:01.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.439+0000 7f51ae000700 1 -- 172.21.15.6:0/1136324226 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51b0005570 con 0x55bd364ea6f0 2024-04-26T01:55:01.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.439+0000 7f51ae000700 1 -- 172.21.15.6:0/1136324226 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f51b0023460 con 0x55bd364ea6f0 2024-04-26T01:55:01.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.439+0000 7f51d5860b80 10 client.26311 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:55:01.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.439+0000 7f51d5860b80 10 client.26311 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:55:01.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.439+0000 7f51d5860b80 20 client.26311 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:01.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.439+0000 7f51d5860b80 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd36504bb0 0x55bd36506fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.439+0000 7f51d5860b80 4 client.26311 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:01.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.439+0000 7f51d5860b80 1 -- 172.21.15.6:0/1136324226 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd3644a810 con 0x55bd36504bb0 2024-04-26T01:55:01.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.440+0000 7f51c3e00700 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd36504bb0 0x55bd36506fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.440+0000 7f51c3e00700 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd36504bb0 0x55bd36506fa0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c07a6bb4488e593 server_cookie=cb72ca3a8cadc596 in_seq=0 out_seq=0 2024-04-26T01:55:01.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.440+0000 7f51ae000700 10 client.26311 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:01.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.441+0000 7f51ae000700 1 -- 172.21.15.6:0/1136324226 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bd3644a810 con 0x55bd36504bb0 2024-04-26T01:55:01.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.441+0000 7f51ae000700 10 client.26311 handle_command_reply: tid=0 2024-04-26T01:55:01.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.441+0000 7f51d5860b80 1 client.26311 shutdown 2024-04-26T01:55:01.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.442+0000 7f51d5860b80 1 -- 172.21.15.6:0/1136324226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 msgr2=0x55bd364f1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.442+0000 7f51d5860b80 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f1f80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f51b0003760 tx=0x7f51b000b6b0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.442+0000 7f51d5860b80 1 -- 172.21.15.6:0/1136324226 shutdown_connections 2024-04-26T01:55:01.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.442+0000 7f51d5860b80 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd364f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.442+0000 7f51d5860b80 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364e9dd0 0x55bd364f4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.442+0000 7f51d5860b80 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd36504bb0 0x55bd36506fa0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.442+0000 7f51d5860b80 1 --2- 172.21.15.6:0/1136324226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f1f80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.442+0000 7f51d5860b80 1 -- 172.21.15.6:0/1136324226 >> 172.21.15.6:0/1136324226 conn(0x55bd364190e0 msgr2=0x55bd364e5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:01.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.442+0000 7f51ae000700 0 client.26311 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:01.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.442+0000 7f51d5860b80 1 -- 172.21.15.6:0/1136324226 shutdown_connections 2024-04-26T01:55:01.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.442+0000 7f51d5860b80 1 -- 172.21.15.6:0/1136324226 wait complete. 2024-04-26T01:55:01.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.442+0000 7f51d5860b80 20 client.26311 trim_cache size 0 max 16384 2024-04-26T01:55:01.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.467+0000 7f51d5860b80 1 Processor -- start 2024-04-26T01:55:01.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.467+0000 7f51d5860b80 1 -- start start 2024-04-26T01:55:01.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.468+0000 7f51d5860b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364e9dd0 0x55bd364f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.468+0000 7f51d5860b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.468+0000 7f51d5860b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364f5560 0x55bd364e49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.468+0000 7f51d5860b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f51a4010040 con 0x55bd364e9dd0 2024-04-26T01:55:01.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.468+0000 7f51d5860b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f51a400d870 con 0x55bd364f5560 2024-04-26T01:55:01.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.468+0000 7f51d5860b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f51a400b6e0 con 0x55bd364ea6f0 2024-04-26T01:55:01.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.468+0000 7f51c3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.468+0000 7f51c3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364e9dd0 0x55bd364f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.468+0000 7f51c8e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364f5560 0x55bd364e49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.468+0000 7f51c3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364e9dd0 0x55bd364f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60870/0 (socket says 172.21.15.6:60870) 2024-04-26T01:55:01.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.468+0000 7f51c3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:46158/0 (socket says 172.21.15.6:46158) 2024-04-26T01:55:01.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.468+0000 7f51c3e00700 1 -- 172.21.15.6:0/3208207445 learned_addr learned my addr 172.21.15.6:0/3208207445 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:01.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.469+0000 7f51c8e00700 1 -- 172.21.15.6:0/3208207445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 msgr2=0x55bd364f0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:01.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.469+0000 7f51c8e00700 1 --2- 172.21.15.6:0/3208207445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.469+0000 7f51c8e00700 1 -- 172.21.15.6:0/3208207445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364e9dd0 msgr2=0x55bd364f22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:01.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.469+0000 7f51c8e00700 1 --2- 172.21.15.6:0/3208207445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364e9dd0 0x55bd364f22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.469+0000 7f51c8e00700 1 -- 172.21.15.6:0/3208207445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd364eb7a0 con 0x55bd364f5560 2024-04-26T01:55:01.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.469+0000 7f51c3400700 1 --2- 172.21.15.6:0/3208207445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:01.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.469+0000 7f51c8e00700 1 --2- 172.21.15.6:0/3208207445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364f5560 0x55bd364e49f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0cd770 tx=0x7f51bc09bc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9f372b254134b07 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:01.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.469+0000 7f51b8c00700 1 -- 172.21.15.6:0/3208207445 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc09c750 con 0x55bd364f5560 2024-04-26T01:55:01.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.469+0000 7f51b8c00700 1 -- 172.21.15.6:0/3208207445 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51bc0bc040 con 0x55bd364f5560 2024-04-26T01:55:01.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.469+0000 7f51b8c00700 1 -- 172.21.15.6:0/3208207445 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc09c510 con 0x55bd364f5560 2024-04-26T01:55:01.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.470+0000 7f51d5860b80 1 -- 172.21.15.6:0/3208207445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364f5560 msgr2=0x55bd364e49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.470+0000 7f51d5860b80 1 --2- 172.21.15.6:0/3208207445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364f5560 0x55bd364e49f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0cd770 tx=0x7f51bc09bc60 comp rx=0 tx=0).stop 2024-04-26T01:55:01.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.470+0000 7f51d5860b80 1 -- 172.21.15.6:0/3208207445 shutdown_connections 2024-04-26T01:55:01.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.470+0000 7f51d5860b80 1 --2- 172.21.15.6:0/3208207445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364f5560 0x55bd364e49f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.470+0000 7f51d5860b80 1 --2- 172.21.15.6:0/3208207445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364e9dd0 0x55bd364f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.470+0000 7f51d5860b80 1 --2- 172.21.15.6:0/3208207445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364ea6f0 0x55bd364f0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.470+0000 7f51d5860b80 1 -- 172.21.15.6:0/3208207445 >> 172.21.15.6:0/3208207445 conn(0x55bd364190e0 msgr2=0x55bd36500cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:01.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.470+0000 7f51d5860b80 1 -- 172.21.15.6:0/3208207445 shutdown_connections 2024-04-26T01:55:01.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.470+0000 7f51d5860b80 1 -- 172.21.15.6:0/3208207445 wait complete. 2024-04-26T01:55:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.471+0000 7f51d5860b80 1 -- start start 2024-04-26T01:55:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.471+0000 7f51d5860b80 1 Processor -- start 2024-04-26T01:55:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.472+0000 7f51d5860b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364e9dd0 0x55bd365089a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.472+0000 7f51d5860b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364ea6f0 0x55bd36508ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.472+0000 7f51d5860b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd36509420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.472+0000 7f51d5860b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f51a400bd30 con 0x55bd364f5560 2024-04-26T01:55:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.472+0000 7f51d5860b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f51a400d1a0 con 0x55bd364ea6f0 2024-04-26T01:55:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.472+0000 7f51d5860b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f51a400c400 con 0x55bd364e9dd0 2024-04-26T01:55:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.472+0000 7f51d5860b80 10 client.0 _ll_register_callbacks cb 0x55bd363d58b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.472+0000 7f51c3e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364e9dd0 0x55bd365089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.472+0000 7f51c3e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364e9dd0 0x55bd365089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:46166/0 (socket says 172.21.15.6:46166) 2024-04-26T01:55:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.472+0000 7f51c3e00700 1 -- 172.21.15.6:0/4039230721 learned_addr learned my addr 172.21.15.6:0/4039230721 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.472+0000 7f51c3400700 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364ea6f0 0x55bd36508ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.473+0000 7f51c8e00700 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd36509420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.473+0000 7f51c3e00700 1 -- 172.21.15.6:0/4039230721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364ea6f0 msgr2=0x55bd36508ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.473+0000 7f51c3e00700 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364ea6f0 0x55bd36508ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.473+0000 7f51c3e00700 1 -- 172.21.15.6:0/4039230721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 msgr2=0x55bd36509420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.473+0000 7f51c3e00700 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd36509420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.473+0000 7f51c3e00700 1 -- 172.21.15.6:0/4039230721 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd3656b000 con 0x55bd364e9dd0 2024-04-26T01:55:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.473+0000 7f51c3e00700 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364e9dd0 0x55bd365089a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009f60 tx=0x7f51b40047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d86abb032cabb86 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.473+0000 7f51ae000700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:55:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.473+0000 7f51ae000700 1 -- 172.21.15.6:0/4039230721 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b402a430 con 0x55bd364e9dd0 2024-04-26T01:55:01.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.474+0000 7f51d5860b80 1 -- 172.21.15.6:0/4039230721 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51bc0bc040 con 0x55bd364e9dd0 2024-04-26T01:55:01.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.474+0000 7f51ae000700 1 -- 172.21.15.6:0/4039230721 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7f51b402a5d0 con 0x55bd364e9dd0 2024-04-26T01:55:01.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.474+0000 7f51ae000700 1 -- 172.21.15.6:0/4039230721 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f51b402a7d0 con 0x55bd364e9dd0 2024-04-26T01:55:01.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.474+0000 7f51d5860b80 10 client.26317 fetch_fsmap learned FSMap version 41 2024-04-26T01:55:01.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.474+0000 7f51d5860b80 1 -- 172.21.15.6:0/4039230721 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55bd3656c2e0 con 0x55bd364e9dd0 2024-04-26T01:55:01.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.474+0000 7f51ae000700 1 -- 172.21.15.6:0/4039230721 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51bc0bc040 con 0x55bd364e9dd0 2024-04-26T01:55:01.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.474+0000 7f51ae000700 1 -- 172.21.15.6:0/4039230721 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f51b4005400 con 0x55bd364e9dd0 2024-04-26T01:55:01.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.474+0000 7f51d5860b80 10 client.26317 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:55:01.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.474+0000 7f51d5860b80 10 client.26317 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:55:01.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.474+0000 7f51d5860b80 20 client.26317 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:01.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.475+0000 7f51d5860b80 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd3656cac0 0x55bd3656eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.475+0000 7f51d5860b80 4 client.26317 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.1, "await_for": 42.0}] 2024-04-26T01:55:01.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.475+0000 7f51d5860b80 1 -- 172.21.15.6:0/4039230721 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.1, "await_for": 42.0}) -- 0x55bd364448b0 con 0x55bd3656cac0 2024-04-26T01:55:01.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.475+0000 7f51c3400700 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd3656cac0 0x55bd3656eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.475+0000 7f51c3400700 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd3656cac0 0x55bd3656eee0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f4377460f8bfcca server_cookie=d14ea255bdd0560c in_seq=0 out_seq=0 2024-04-26T01:55:01.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.475+0000 7f51ae000700 10 client.26317 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.564+0000 7f51ae000700 1 -- 172.21.15.6:0/4039230721 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55bd364448b0 con 0x55bd3656cac0 2024-04-26T01:55:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.564+0000 7f51ae000700 10 client.26317 handle_command_reply: tid=0 2024-04-26T01:55:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.564+0000 7f51d5860b80 1 client.26317 shutdown 2024-04-26T01:55:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.564+0000 7f51d5860b80 1 -- 172.21.15.6:0/4039230721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364e9dd0 msgr2=0x55bd365089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.564+0000 7f51d5860b80 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364e9dd0 0x55bd365089a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009f60 tx=0x7f51b40047c0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.564+0000 7f51d5860b80 1 -- 172.21.15.6:0/4039230721 shutdown_connections 2024-04-26T01:55:01.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.564+0000 7f51d5860b80 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55bd364f5560 0x55bd36509420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.564+0000 7f51d5860b80 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55bd3656cac0 0x55bd3656eee0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.564+0000 7f51d5860b80 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd364e9dd0 0x55bd365089a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.564+0000 7f51d5860b80 1 --2- 172.21.15.6:0/4039230721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bd364ea6f0 0x55bd36508ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.564+0000 7f51ae000700 0 client.26317 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:01.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.565+0000 7f51d5860b80 1 -- 172.21.15.6:0/4039230721 >> 172.21.15.6:0/4039230721 conn(0x55bd364190e0 msgr2=0x55bd36500460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:01.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.565+0000 7f51d5860b80 1 -- 172.21.15.6:0/4039230721 shutdown_connections 2024-04-26T01:55:01.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.565+0000 7f51d5860b80 1 -- 172.21.15.6:0/4039230721 wait complete. 2024-04-26T01:55:01.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.565+0000 7f51d5860b80 20 client.26317 trim_cache size 0 max 16384 2024-04-26T01:55:01.565 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:55:01.565 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 41, 2024-04-26T01:55:01.565 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:55:01.565 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 26, 2024-04-26T01:55:01.565 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:55:01.565 INFO:teuthology.orchestra.run.smithi006.stdout: "8a864be1": { 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 26, 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 82.1, 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:55:01.566 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:55:01.567 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:01.567 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:01.567 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:01.567 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:01.567 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:01.567 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:55:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.566+0000 7f51cae00700 1 -- 172.21.15.6:0/76095137 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f51a80a3ed0 msgr2=0x7f51a80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.567+0000 7f51cae00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f51a80a3ed0 0x7f51a80a6390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009c60 tx=0x7f51b4009260 comp rx=0 tx=0).stop 2024-04-26T01:55:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.567+0000 7f51cae00700 1 -- 172.21.15.6:0/76095137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4109fa0 msgr2=0x7f51c419fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.567+0000 7f51cae00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4109fa0 0x7f51c419fb80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00efc0 tx=0x7f51bc00c3a0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.567+0000 7f51cae00700 1 -- 172.21.15.6:0/76095137 shutdown_connections 2024-04-26T01:55:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.567+0000 7f51cae00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f51a80a3ed0 0x7f51a80a6390 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.567+0000 7f51cae00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4109fa0 0x7f51c419fb80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.567+0000 7f51cae00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51c4104ec0 0x7f51c419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.567+0000 7f51cae00700 1 --2- 172.21.15.6:0/76095137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51c4105870 0x7f51c419b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.567+0000 7f51cae00700 1 -- 172.21.15.6:0/76095137 >> 172.21.15.6:0/76095137 conn(0x7f51c4100750 msgr2=0x7f51c4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.567+0000 7f51cae00700 1 -- 172.21.15.6:0/76095137 shutdown_connections 2024-04-26T01:55:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:01.567+0000 7f51cae00700 1 -- 172.21.15.6:0/76095137 wait complete. 2024-04-26T01:55:01.609 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.94 MB/sec execute 325 sec latency 131.793 ms 2024-04-26T01:55:02.354 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '8a864be1', quiesce took 8.1 seconds. Will release after: 22.1 2024-04-26T01:55:02.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:02 smithi006 ceph-mon[28419]: pgmap v286: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 294 op/s 2024-04-26T01:55:02.609 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.93 MB/sec execute 326 sec latency 1131.941 ms 2024-04-26T01:55:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:02 smithi188 ceph-mon[32751]: pgmap v286: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 294 op/s 2024-04-26T01:55:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:02 smithi029 ceph-mon[32775]: pgmap v286: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 294 op/s 2024-04-26T01:55:03.609 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.92 MB/sec execute 327 sec latency 2132.042 ms 2024-04-26T01:55:04.609 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.90 MB/sec execute 328 sec latency 3132.099 ms 2024-04-26T01:55:04.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:04 smithi006 ceph-mon[28419]: pgmap v287: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-04-26T01:55:04.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:04 smithi188 ceph-mon[32751]: pgmap v287: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-04-26T01:55:04.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:04 smithi029 ceph-mon[32775]: pgmap v287: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-04-26T01:55:05.609 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.89 MB/sec execute 329 sec latency 4132.243 ms 2024-04-26T01:55:05.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:05 smithi006 ceph-mon[28419]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T01:55:05.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:05 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:55:05.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:05 smithi188 ceph-mon[32751]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T01:55:05.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:05 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:55:05.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:05 smithi029 ceph-mon[32775]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T01:55:05.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:05 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:55:06.609 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.88 MB/sec execute 330 sec latency 5132.388 ms 2024-04-26T01:55:06.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:06 smithi006 ceph-mon[28419]: pgmap v289: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 228 op/s 2024-04-26T01:55:06.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:06 smithi188 ceph-mon[32751]: pgmap v289: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 228 op/s 2024-04-26T01:55:06.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:06 smithi029 ceph-mon[32775]: pgmap v289: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 228 op/s 2024-04-26T01:55:07.610 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.87 MB/sec execute 331 sec latency 6132.534 ms 2024-04-26T01:55:08.610 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.86 MB/sec execute 332 sec latency 7132.694 ms 2024-04-26T01:55:08.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:08 smithi006 ceph-mon[28419]: pgmap v290: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-04-26T01:55:08.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:08 smithi188 ceph-mon[32751]: pgmap v290: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-04-26T01:55:08.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:08 smithi029 ceph-mon[32775]: pgmap v290: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-04-26T01:55:09.610 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.85 MB/sec execute 333 sec latency 8132.832 ms 2024-04-26T01:55:10.610 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.83 MB/sec execute 334 sec latency 9132.960 ms 2024-04-26T01:55:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:10 smithi006 ceph-mon[28419]: pgmap v291: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-04-26T01:55:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:10 smithi006 ceph-mon[28419]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T01:55:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:10 smithi188 ceph-mon[32751]: pgmap v291: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-04-26T01:55:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:10 smithi188 ceph-mon[32751]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T01:55:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:10 smithi029 ceph-mon[32775]: pgmap v291: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-04-26T01:55:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:10 smithi029 ceph-mon[32775]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T01:55:11.610 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.82 MB/sec execute 335 sec latency 10133.116 ms 2024-04-26T01:55:12.610 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.81 MB/sec execute 336 sec latency 11133.306 ms 2024-04-26T01:55:12.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:12 smithi006 ceph-mon[28419]: pgmap v293: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 1.3 MiB/s wr, 97 op/s 2024-04-26T01:55:12.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:12 smithi188 ceph-mon[32751]: pgmap v293: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 1.3 MiB/s wr, 97 op/s 2024-04-26T01:55:12.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:12 smithi029 ceph-mon[32775]: pgmap v293: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 1.3 MiB/s wr, 97 op/s 2024-04-26T01:55:13.610 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.80 MB/sec execute 337 sec latency 12133.398 ms 2024-04-26T01:55:14.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.211+0000 7f6707600640 1 -- 172.21.15.6:0/2253853995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6700104d90 msgr2=0x7f6700108dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:14.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.211+0000 7f6707600640 1 --2- 172.21.15.6:0/2253853995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6700104d90 0x7f6700108dd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f66f0009910 tx=0x7f66f002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.211+0000 7f6707600640 1 -- 172.21.15.6:0/2253853995 shutdown_connections 2024-04-26T01:55:14.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.211+0000 7f6707600640 1 --2- 172.21.15.6:0/2253853995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6700104d90 0x7f6700108dd0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.211+0000 7f6707600640 1 --2- 172.21.15.6:0/2253853995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67001094b0 0x7f6700111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.211+0000 7f6707600640 1 --2- 172.21.15.6:0/2253853995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67001043e0 0x7f67001047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.211+0000 7f6707600640 1 -- 172.21.15.6:0/2253853995 >> 172.21.15.6:0/2253853995 conn(0x7f67000ffc00 msgr2=0x7f6700102020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:14.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.212+0000 7f6707600640 1 -- 172.21.15.6:0/2253853995 shutdown_connections 2024-04-26T01:55:14.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.212+0000 7f6707600640 1 -- 172.21.15.6:0/2253853995 wait complete. 2024-04-26T01:55:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.212+0000 7f6707600640 1 Processor -- start 2024-04-26T01:55:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.213+0000 7f6707600640 1 -- start start 2024-04-26T01:55:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.213+0000 7f6707600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67001043e0 0x7f67001a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.213+0000 7f6707600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67001094b0 0x7f67001a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.213+0000 7f6707600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f670019a7d0 0x7f670019ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.213+0000 7f6707600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f67001143d0 con 0x7f67001043e0 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.213+0000 7f6707600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6700114250 con 0x7f67001094b0 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.213+0000 7f6707600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6700114550 con 0x7f670019a7d0 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.213+0000 7f6704c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67001043e0 0x7f67001a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.213+0000 7f66ffe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67001094b0 0x7f67001a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.213+0000 7f6704c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67001043e0 0x7f67001a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51994/0 (socket says 172.21.15.6:51994) 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.213+0000 7f6704c00640 1 -- 172.21.15.6:0/1157291561 learned_addr learned my addr 172.21.15.6:0/1157291561 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.213+0000 7f6705600640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f670019a7d0 0x7f670019ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.214+0000 7f6705600640 1 -- 172.21.15.6:0/1157291561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67001094b0 msgr2=0x7f67001a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.214+0000 7f6705600640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67001094b0 0x7f67001a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.214+0000 7f6705600640 1 -- 172.21.15.6:0/1157291561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67001043e0 msgr2=0x7f67001a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.214+0000 7f6705600640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67001043e0 0x7f67001a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.214+0000 7f6705600640 1 -- 172.21.15.6:0/1157291561 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670019b360 con 0x7f670019a7d0 2024-04-26T01:55:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.214+0000 7f6704c00640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67001043e0 0x7f67001a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.214+0000 7f6705600640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f670019a7d0 0x7f670019ac80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009fd0 tx=0x7f66f800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.214+0000 7f66fd600640 1 -- 172.21.15.6:0/1157291561 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f800ce20 con 0x7f670019a7d0 2024-04-26T01:55:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.215+0000 7f6707600640 1 -- 172.21.15.6:0/1157291561 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f670019b650 con 0x7f670019a7d0 2024-04-26T01:55:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.215+0000 7f66fd600640 1 -- 172.21.15.6:0/1157291561 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66f8005020 con 0x7f670019a7d0 2024-04-26T01:55:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.215+0000 7f66fd600640 1 -- 172.21.15.6:0/1157291561 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f8010690 con 0x7f670019a7d0 2024-04-26T01:55:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.215+0000 7f6707600640 1 -- 172.21.15.6:0/1157291561 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67001a77b0 con 0x7f670019a7d0 2024-04-26T01:55:14.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.219+0000 7f670a5b4740 1 Processor -- start 2024-04-26T01:55:14.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.219+0000 7f66fd600640 1 -- 172.21.15.6:0/1157291561 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f66f80054d0 con 0x7f670019a7d0 2024-04-26T01:55:14.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.220+0000 7f670a5b4740 1 -- start start 2024-04-26T01:55:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.220+0000 7f670a5b4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14dd820 0x5619a14ddc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.220+0000 7f670a5b4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14de140 0x5619a14e84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.220+0000 7f670a5b4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14e8a00 0x5619a14eadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.220+0000 7f670a5b4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5619a1443900 con 0x5619a14e8a00 2024-04-26T01:55:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.220+0000 7f670a5b4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5619a13e9ef0 con 0x5619a14dd820 2024-04-26T01:55:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.220+0000 7f670a5b4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5619a143f8c0 con 0x5619a14de140 2024-04-26T01:55:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.220+0000 7f66ffe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14dd820 0x5619a14ddc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.220+0000 7f6704c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14de140 0x5619a14e84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f66ffe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14dd820 0x5619a14ddc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:53616/0 (socket says 172.21.15.6:53616) 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f66ffe00640 1 -- 172.21.15.6:0/3349571786 learned_addr learned my addr 172.21.15.6:0/3349571786 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f66ffe00640 1 --2- 172.21.15.6:0/3349571786 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14e8a00 0x5619a14eadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f6704c00640 1 -- 172.21.15.6:0/3349571786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14dd820 msgr2=0x5619a14ddc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f6704c00640 1 --2- 172.21.15.6:0/3349571786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14dd820 0x5619a14ddc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f6704c00640 1 -- 172.21.15.6:0/3349571786 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14e8a00 msgr2=0x5619a14eadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f6704c00640 1 --2- 172.21.15.6:0/3349571786 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14e8a00 0x5619a14eadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f6704c00640 1 -- 172.21.15.6:0/3349571786 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619a1156210 con 0x5619a14de140 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f66fd600640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f66e80a3d10 0x7f66e80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f6704c00640 1 --2- 172.21.15.6:0/3349571786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14de140 0x5619a14e84c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f66f40058c0 tx=0x7f66f40109e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b84ea0d7830b8f09 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f66fd600640 1 -- 172.21.15.6:0/1157291561 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f66f80d0f30 con 0x7f670019a7d0 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f66edc00640 1 -- 172.21.15.6:0/3349571786 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f40117c0 con 0x5619a14de140 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.221+0000 7f66edc00640 1 -- 172.21.15.6:0/3349571786 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66f4011e10 con 0x5619a14de140 2024-04-26T01:55:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.222+0000 7f66edc00640 1 -- 172.21.15.6:0/3349571786 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f400ee00 con 0x5619a14de140 2024-04-26T01:55:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.222+0000 7f66ffe00640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f66e80a3d10 0x7f66e80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.222+0000 7f670a5b4740 1 -- 172.21.15.6:0/3349571786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14de140 msgr2=0x5619a14e84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.222+0000 7f670a5b4740 1 --2- 172.21.15.6:0/3349571786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14de140 0x5619a14e84c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f66f40058c0 tx=0x7f66f40109e0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.222+0000 7f66ffe00640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f66e80a3d10 0x7f66e80a61d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x5619a14db150 tx=0x7f66f002a3f0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.222+0000 7f670a5b4740 1 -- 172.21.15.6:0/3349571786 shutdown_connections 2024-04-26T01:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.222+0000 7f670a5b4740 1 --2- 172.21.15.6:0/3349571786 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14e8a00 0x5619a14eadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.222+0000 7f670a5b4740 1 --2- 172.21.15.6:0/3349571786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14de140 0x5619a14e84c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.222+0000 7f670a5b4740 1 --2- 172.21.15.6:0/3349571786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14dd820 0x5619a14ddc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.222+0000 7f670a5b4740 1 -- 172.21.15.6:0/3349571786 >> 172.21.15.6:0/3349571786 conn(0x5619a14d9d30 msgr2=0x5619a14da110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.222+0000 7f670a5b4740 1 -- 172.21.15.6:0/3349571786 shutdown_connections 2024-04-26T01:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.223+0000 7f670a5b4740 1 -- 172.21.15.6:0/3349571786 wait complete. 2024-04-26T01:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.223+0000 7f670a5b4740 1 -- start start 2024-04-26T01:55:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.223+0000 7f670a5b4740 1 Processor -- start 2024-04-26T01:55:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f670a5b4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14dd820 0x5619a14e59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f670a5b4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14de140 0x5619a14e5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f670a5b4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14e8a00 0x5619a14e6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f670a5b4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5619a14eb630 con 0x5619a14de140 2024-04-26T01:55:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f670a5b4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5619a13fd310 con 0x5619a14e8a00 2024-04-26T01:55:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f670a5b4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5619a14eb7b0 con 0x5619a14dd820 2024-04-26T01:55:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f670a5b4740 10 client.0 _ll_register_callbacks cb 0x5619a13cbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f6704c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14dd820 0x5619a14e59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f6704c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14dd820 0x5619a14e59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:33370/0 (socket says 172.21.15.6:33370) 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f6704c00640 1 -- 172.21.15.6:0/30294569 learned_addr learned my addr 172.21.15.6:0/30294569 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f66ffe00640 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14de140 0x5619a14e5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f6705600640 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14e8a00 0x5619a14e6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f6704c00640 1 -- 172.21.15.6:0/30294569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14e8a00 msgr2=0x5619a14e6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f6704c00640 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14e8a00 0x5619a14e6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f6704c00640 1 -- 172.21.15.6:0/30294569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14de140 msgr2=0x5619a14e5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f6704c00640 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14de140 0x5619a14e5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.224+0000 7f6704c00640 1 -- 172.21.15.6:0/30294569 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619a1222620 con 0x5619a14dd820 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.225+0000 7f6704c00640 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14dd820 0x5619a14e59d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f66f4006010 tx=0x7f66f400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21e8a860db6ab379 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.225+0000 7f66e2000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.225+0000 7f66e2000640 1 -- 172.21.15.6:0/30294569 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f4004260 con 0x5619a14dd820 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.225+0000 7f670a5b4740 1 -- 172.21.15.6:0/30294569 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66f4011e10 con 0x5619a14dd820 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.225+0000 7f66e2000640 1 -- 172.21.15.6:0/30294569 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f66f4004400 con 0x5619a14dd820 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.225+0000 7f66e2000640 1 -- 172.21.15.6:0/30294569 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f66f400e430 con 0x5619a14dd820 2024-04-26T01:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.225+0000 7f670a5b4740 10 client.26329 fetch_fsmap learned FSMap version 42 2024-04-26T01:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.225+0000 7f670a5b4740 1 -- 172.21.15.6:0/30294569 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5619a14f78e0 con 0x5619a14dd820 2024-04-26T01:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.226+0000 7f66e2000640 1 -- 172.21.15.6:0/30294569 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66f4011e10 con 0x5619a14dd820 2024-04-26T01:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.226+0000 7f66e2000640 1 -- 172.21.15.6:0/30294569 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f66f4011650 con 0x5619a14dd820 2024-04-26T01:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.226+0000 7f670a5b4740 10 client.26329 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.226+0000 7f670a5b4740 10 client.26329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.226+0000 7f670a5b4740 20 client.26329 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.226+0000 7f670a5b4740 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619a14f8cf0 0x5619a14fb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.226+0000 7f670a5b4740 4 client.26329 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.226+0000 7f670a5b4740 1 -- 172.21.15.6:0/30294569 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619a14578b0 con 0x5619a14f8cf0 2024-04-26T01:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.227+0000 7f66ffe00640 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619a14f8cf0 0x5619a14fb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.227+0000 7f66ffe00640 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619a14f8cf0 0x5619a14fb0e0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7c48ba6b2499cf7 server_cookie=8df115f465eed351 in_seq=0 out_seq=0 2024-04-26T01:55:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.227+0000 7f66e2000640 10 client.26329 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.228+0000 7f66e2000640 1 -- 172.21.15.6:0/30294569 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5619a14578b0 con 0x5619a14f8cf0 2024-04-26T01:55:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.228+0000 7f66e2000640 10 client.26329 handle_command_reply: tid=0 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.228+0000 7f670a5b4740 1 client.26329 shutdown 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.228+0000 7f670a5b4740 1 -- 172.21.15.6:0/30294569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14dd820 msgr2=0x5619a14e59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.228+0000 7f670a5b4740 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14dd820 0x5619a14e59d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f66f4006010 tx=0x7f66f400ee40 comp rx=0 tx=0).stop 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.228+0000 7f670a5b4740 1 -- 172.21.15.6:0/30294569 shutdown_connections 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.228+0000 7f670a5b4740 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619a14f8cf0 0x5619a14fb0e0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.228+0000 7f670a5b4740 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14de140 0x5619a14e5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.228+0000 7f670a5b4740 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14e8a00 0x5619a14e6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.228+0000 7f670a5b4740 1 --2- 172.21.15.6:0/30294569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14dd820 0x5619a14e59d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.228+0000 7f670a5b4740 1 -- 172.21.15.6:0/30294569 >> 172.21.15.6:0/30294569 conn(0x5619a14d9d30 msgr2=0x5619a14dfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.228+0000 7f66e2000640 0 client.26329 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.229+0000 7f670a5b4740 1 -- 172.21.15.6:0/30294569 shutdown_connections 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.229+0000 7f670a5b4740 1 -- 172.21.15.6:0/30294569 wait complete. 2024-04-26T01:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.229+0000 7f670a5b4740 20 client.26329 trim_cache size 0 max 16384 2024-04-26T01:55:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.245+0000 7f670a5b4740 1 Processor -- start 2024-04-26T01:55:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.245+0000 7f670a5b4740 1 -- start start 2024-04-26T01:55:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.245+0000 7f670a5b4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14dd820 0x5619a14f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.245+0000 7f670a5b4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14de140 0x5619a1500870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.245+0000 7f670a5b4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14e8a00 0x5619a1500db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.245+0000 7f670a5b4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f67001b28e0 con 0x5619a14de140 2024-04-26T01:55:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.245+0000 7f670a5b4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f67001b0180 con 0x5619a14dd820 2024-04-26T01:55:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.245+0000 7f670a5b4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f67001adff0 con 0x5619a14e8a00 2024-04-26T01:55:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.246+0000 7f66ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14de140 0x5619a1500870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.246+0000 7f66ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14de140 0x5619a1500870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52010/0 (socket says 172.21.15.6:52010) 2024-04-26T01:55:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.246+0000 7f66ffe00640 1 -- 172.21.15.6:0/1088881524 learned_addr learned my addr 172.21.15.6:0/1088881524 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.246+0000 7f6704c00640 1 --2- 172.21.15.6:0/1088881524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14dd820 0x5619a14f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.246+0000 7f6705600640 1 --2- 172.21.15.6:0/1088881524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14e8a00 0x5619a1500db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.246+0000 7f66ffe00640 1 -- 172.21.15.6:0/1088881524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14e8a00 msgr2=0x5619a1500db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.246+0000 7f66ffe00640 1 --2- 172.21.15.6:0/1088881524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14e8a00 0x5619a1500db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.246+0000 7f66ffe00640 1 -- 172.21.15.6:0/1088881524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14dd820 msgr2=0x5619a14f44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.246+0000 7f66ffe00640 1 --2- 172.21.15.6:0/1088881524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14dd820 0x5619a14f44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.246+0000 7f66ffe00640 1 -- 172.21.15.6:0/1088881524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619a14d5470 con 0x5619a14de140 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.246+0000 7f66ffe00640 1 --2- 172.21.15.6:0/1088881524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14de140 0x5619a1500870 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f66f00098e0 tx=0x7f66f0042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bb917165a7815d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.247+0000 7f66ed200640 1 -- 172.21.15.6:0/1088881524 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f005c070 con 0x5619a14de140 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.247+0000 7f66ed200640 1 -- 172.21.15.6:0/1088881524 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66f0030d50 con 0x5619a14de140 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.247+0000 7f66ed200640 1 -- 172.21.15.6:0/1088881524 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f00046c0 con 0x5619a14de140 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.247+0000 7f670a5b4740 1 -- 172.21.15.6:0/1088881524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14de140 msgr2=0x5619a1500870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.247+0000 7f670a5b4740 1 --2- 172.21.15.6:0/1088881524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14de140 0x5619a1500870 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f66f00098e0 tx=0x7f66f0042000 comp rx=0 tx=0).stop 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.247+0000 7f670a5b4740 1 -- 172.21.15.6:0/1088881524 shutdown_connections 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.247+0000 7f670a5b4740 1 --2- 172.21.15.6:0/1088881524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14de140 0x5619a1500870 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.247+0000 7f670a5b4740 1 --2- 172.21.15.6:0/1088881524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14e8a00 0x5619a1500db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.247+0000 7f670a5b4740 1 --2- 172.21.15.6:0/1088881524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14dd820 0x5619a14f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.247+0000 7f670a5b4740 1 -- 172.21.15.6:0/1088881524 >> 172.21.15.6:0/1088881524 conn(0x5619a14d9d30 msgr2=0x5619a14d8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.247+0000 7f670a5b4740 1 -- 172.21.15.6:0/1088881524 shutdown_connections 2024-04-26T01:55:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.248+0000 7f670a5b4740 1 -- 172.21.15.6:0/1088881524 wait complete. 2024-04-26T01:55:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.248+0000 7f670a5b4740 1 -- start start 2024-04-26T01:55:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.248+0000 7f670a5b4740 1 Processor -- start 2024-04-26T01:55:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f670a5b4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14dd820 0x5619a14f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f670a5b4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14de140 0x5619a14ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f670a5b4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14e8a00 0x5619a14ee730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f670a5b4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f67001ae640 con 0x5619a14dd820 2024-04-26T01:55:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f670a5b4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f67001afab0 con 0x5619a14e8a00 2024-04-26T01:55:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f670a5b4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f67001aed10 con 0x5619a14de140 2024-04-26T01:55:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f670a5b4740 10 client.0 _ll_register_callbacks cb 0x7f66f400bdf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f6704c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14dd820 0x5619a14f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f6704c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14dd820 0x5619a14f0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52020/0 (socket says 172.21.15.6:52020) 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f6704c00640 1 -- 172.21.15.6:0/468549979 learned_addr learned my addr 172.21.15.6:0/468549979 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f66ffe00640 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14de140 0x5619a14ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f6705600640 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14e8a00 0x5619a14ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f6704c00640 1 -- 172.21.15.6:0/468549979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14de140 msgr2=0x5619a14ee1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f6704c00640 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14de140 0x5619a14ee1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f6704c00640 1 -- 172.21.15.6:0/468549979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14e8a00 msgr2=0x5619a14ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f6704c00640 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14e8a00 0x5619a14ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f6704c00640 1 -- 172.21.15.6:0/468549979 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619a14eef60 con 0x5619a14dd820 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.249+0000 7f6705600640 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14e8a00 0x5619a14ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.250+0000 7f6704c00640 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14dd820 0x5619a14f0e10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f66f400bea0 tx=0x7f66f4010fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dabc3b458cd45f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.250+0000 7f66e2000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.250+0000 7f66e2000640 1 -- 172.21.15.6:0/468549979 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f4004470 con 0x5619a14dd820 2024-04-26T01:55:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.250+0000 7f66e2000640 1 -- 172.21.15.6:0/468549979 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f66f40115c0 con 0x5619a14dd820 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.250+0000 7f670a5b4740 1 -- 172.21.15.6:0/468549979 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66f0030d50 con 0x5619a14dd820 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.250+0000 7f66e2000640 1 -- 172.21.15.6:0/468549979 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f66f40117c0 con 0x5619a14dd820 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.250+0000 7f670a5b4740 10 client.16149 fetch_fsmap learned FSMap version 42 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.250+0000 7f670a5b4740 1 -- 172.21.15.6:0/468549979 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5619a155f160 con 0x5619a14dd820 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.251+0000 7f66e2000640 1 -- 172.21.15.6:0/468549979 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66f0030d50 con 0x5619a14dd820 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.251+0000 7f66e2000640 1 -- 172.21.15.6:0/468549979 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 42) ==== 8342+0+0 (secure 0 0 0) 0x7f66f4003a40 con 0x5619a14dd820 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.251+0000 7f670a5b4740 10 client.16149 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.251+0000 7f670a5b4740 10 client.16149 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.251+0000 7f670a5b4740 20 client.16149 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.251+0000 7f670a5b4740 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619a155f7b0 0x5619a1561ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.251+0000 7f670a5b4740 4 client.16149 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.251+0000 7f670a5b4740 1 -- 172.21.15.6:0/468549979 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5619a1451320 con 0x5619a155f7b0 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.251+0000 7f66ffe00640 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619a155f7b0 0x5619a1561ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.252+0000 7f66ffe00640 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619a155f7b0 0x5619a1561ba0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=714628ce252af1fb server_cookie=eba6be05717381f2 in_seq=0 out_seq=0 2024-04-26T01:55:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.252+0000 7f66e2000640 10 client.16149 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.283+0000 7f66e2000640 1 -- 172.21.15.6:0/468549979 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5619a1451320 con 0x5619a155f7b0 2024-04-26T01:55:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.283+0000 7f66e2000640 10 client.16149 handle_command_reply: tid=0 2024-04-26T01:55:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.283+0000 7f670a5b4740 1 client.16149 shutdown 2024-04-26T01:55:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.284+0000 7f670a5b4740 1 -- 172.21.15.6:0/468549979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14dd820 msgr2=0x5619a14f0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.284+0000 7f670a5b4740 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14dd820 0x5619a14f0e10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f66f400bea0 tx=0x7f66f4010fb0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.284+0000 7f670a5b4740 1 -- 172.21.15.6:0/468549979 shutdown_connections 2024-04-26T01:55:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.284+0000 7f670a5b4740 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5619a155f7b0 0x5619a1561ba0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.284+0000 7f670a5b4740 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5619a14e8a00 0x5619a14ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.284+0000 7f670a5b4740 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5619a14de140 0x5619a14ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.284+0000 7f670a5b4740 1 --2- 172.21.15.6:0/468549979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619a14dd820 0x5619a14f0e10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.284+0000 7f66e2000640 0 client.16149 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.284+0000 7f670a5b4740 1 -- 172.21.15.6:0/468549979 >> 172.21.15.6:0/468549979 conn(0x5619a14d9d30 msgr2=0x5619a14df1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.284+0000 7f670a5b4740 1 -- 172.21.15.6:0/468549979 shutdown_connections 2024-04-26T01:55:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.284+0000 7f670a5b4740 1 -- 172.21.15.6:0/468549979 wait complete. 2024-04-26T01:55:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.284+0000 7f670a5b4740 20 client.16149 trim_cache size 0 max 16384 2024-04-26T01:55:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:55:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:55:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:55:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:55:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.285+0000 7f6707600640 1 -- 172.21.15.6:0/1157291561 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f66e80a3d10 msgr2=0x7f66e80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.285+0000 7f6707600640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f66e80a3d10 0x7f66e80a61d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x5619a14db150 tx=0x7f66f002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.285+0000 7f6707600640 1 -- 172.21.15.6:0/1157291561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f670019a7d0 msgr2=0x7f670019ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.285+0000 7f6707600640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f670019a7d0 0x7f670019ac80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009fd0 tx=0x7f66f800ef10 comp rx=0 tx=0).stop 2024-04-26T01:55:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.285+0000 7f6707600640 1 -- 172.21.15.6:0/1157291561 shutdown_connections 2024-04-26T01:55:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.285+0000 7f6707600640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f66e80a3d10 0x7f66e80a61d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.285+0000 7f6707600640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f670019a7d0 0x7f670019ac80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.285+0000 7f6707600640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67001094b0 0x7f67001a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.285+0000 7f6707600640 1 --2- 172.21.15.6:0/1157291561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67001043e0 0x7f67001a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.285+0000 7f6707600640 1 -- 172.21.15.6:0/1157291561 >> 172.21.15.6:0/1157291561 conn(0x7f67000ffc00 msgr2=0x7f6700100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.286+0000 7f6707600640 1 -- 172.21.15.6:0/1157291561 shutdown_connections 2024-04-26T01:55:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:14.286+0000 7f6707600640 1 -- 172.21.15.6:0/1157291561 wait complete. 2024-04-26T01:55:14.611 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.79 MB/sec execute 338 sec latency 13133.495 ms 2024-04-26T01:55:14.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:14 smithi006 ceph-mon[28419]: pgmap v294: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-04-26T01:55:14.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:14 smithi188 ceph-mon[32751]: pgmap v294: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-04-26T01:55:14.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:14 smithi029 ceph-mon[32775]: pgmap v294: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-04-26T01:55:15.611 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.78 MB/sec execute 339 sec latency 14133.641 ms 2024-04-26T01:55:16.611 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.77 MB/sec execute 340 sec latency 15133.791 ms 2024-04-26T01:55:16.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:16 smithi006 ceph-mon[28419]: pgmap v295: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.0 MiB/s wr, 78 op/s 2024-04-26T01:55:16.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:16 smithi188 ceph-mon[32751]: pgmap v295: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.0 MiB/s wr, 78 op/s 2024-04-26T01:55:16.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:16 smithi029 ceph-mon[32775]: pgmap v295: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.0 MiB/s wr, 78 op/s 2024-04-26T01:55:17.611 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.76 MB/sec execute 341 sec latency 16133.928 ms 2024-04-26T01:55:18.611 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.74 MB/sec execute 342 sec latency 17134.056 ms 2024-04-26T01:55:18.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:18 smithi006 ceph-mon[28419]: pgmap v296: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.3 MiB/s wr, 58 op/s 2024-04-26T01:55:18.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:18 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:55:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:18 smithi188 ceph-mon[32751]: pgmap v296: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.3 MiB/s wr, 58 op/s 2024-04-26T01:55:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:18 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:55:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:18 smithi029 ceph-mon[32775]: pgmap v296: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.3 MiB/s wr, 58 op/s 2024-04-26T01:55:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:18 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:55:19.611 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.73 MB/sec execute 343 sec latency 18134.140 ms 2024-04-26T01:55:19.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:19 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:55:19.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:19 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:55:19.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:19 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:55:19.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:19 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:55:19.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:19 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:55:19.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:19 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:55:19.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:19 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:55:19.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:19 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:55:19.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:19 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:55:20.611 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.72 MB/sec execute 344 sec latency 19134.315 ms 2024-04-26T01:55:20.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:20 smithi006 ceph-mon[28419]: pgmap v297: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.3 MiB/s wr, 58 op/s 2024-04-26T01:55:20.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:20 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:55:20.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:20 smithi188 ceph-mon[32751]: pgmap v297: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.3 MiB/s wr, 58 op/s 2024-04-26T01:55:20.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:20 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:55:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:20 smithi029 ceph-mon[32775]: pgmap v297: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.3 MiB/s wr, 58 op/s 2024-04-26T01:55:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:20 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:55:21.612 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.71 MB/sec execute 345 sec latency 20134.450 ms 2024-04-26T01:55:22.611 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.70 MB/sec execute 346 sec latency 21134.539 ms 2024-04-26T01:55:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:22 smithi006 ceph-mon[28419]: pgmap v298: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.7 MiB/s wr, 72 op/s 2024-04-26T01:55:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:22 smithi188 ceph-mon[32751]: pgmap v298: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.7 MiB/s wr, 72 op/s 2024-04-26T01:55:22.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:22 smithi029 ceph-mon[32775]: pgmap v298: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.7 MiB/s wr, 72 op/s 2024-04-26T01:55:23.612 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.69 MB/sec execute 347 sec latency 22134.654 ms 2024-04-26T01:55:24.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:55:24.612 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.68 MB/sec execute 348 sec latency 23134.757 ms 2024-04-26T01:55:24.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:24 smithi006 ceph-mon[28419]: pgmap v299: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 1.1 MiB/s wr, 46 op/s 2024-04-26T01:55:24.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:24 smithi188 ceph-mon[32751]: pgmap v299: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 1.1 MiB/s wr, 46 op/s 2024-04-26T01:55:24.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:24 smithi029 ceph-mon[32775]: pgmap v299: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 1.1 MiB/s wr, 46 op/s 2024-04-26T01:55:24.956 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:55:25.612 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.67 MB/sec execute 349 sec latency 24134.891 ms 2024-04-26T01:55:26.612 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.66 MB/sec execute 350 sec latency 25134.984 ms 2024-04-26T01:55:26.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:26 smithi006 ceph-mon[28419]: pgmap v300: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 1.1 MiB/s wr, 46 op/s 2024-04-26T01:55:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:26 smithi188 ceph-mon[32751]: pgmap v300: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 1.1 MiB/s wr, 46 op/s 2024-04-26T01:55:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:26 smithi029 ceph-mon[32775]: pgmap v300: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 1.1 MiB/s wr, 46 op/s 2024-04-26T01:55:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.167+0000 7f8264600700 1 -- 172.21.15.6:0/2062723366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f825c107940 msgr2=0x7f825c107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.167+0000 7f8264600700 1 --2- 172.21.15.6:0/2062723366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f825c107940 0x7f825c107d20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f82500099c0 tx=0x7f8250009cd0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.167+0000 7f8264600700 1 -- 172.21.15.6:0/2062723366 shutdown_connections 2024-04-26T01:55:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.167+0000 7f8264600700 1 --2- 172.21.15.6:0/2062723366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f825c1034b0 0x7f825c10f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.167+0000 7f8264600700 1 --2- 172.21.15.6:0/2062723366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f825c107940 0x7f825c107d20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.167+0000 7f8264600700 1 --2- 172.21.15.6:0/2062723366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f825c102af0 0x7f825c102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.167+0000 7f8264600700 1 -- 172.21.15.6:0/2062723366 >> 172.21.15.6:0/2062723366 conn(0x7f825c079ba0 msgr2=0x7f825c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.167+0000 7f8264600700 1 -- 172.21.15.6:0/2062723366 shutdown_connections 2024-04-26T01:55:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.167+0000 7f8264600700 1 -- 172.21.15.6:0/2062723366 wait complete. 2024-04-26T01:55:27.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.168+0000 7f8264600700 1 Processor -- start 2024-04-26T01:55:27.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.169+0000 7f8264600700 1 -- start start 2024-04-26T01:55:27.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.169+0000 7f8264600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f825c102af0 0x7f825c199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.169+0000 7f8264600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f825c1034b0 0x7f825c199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.169+0000 7f8264600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f825c107940 0x7f825c19d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.169+0000 7f8264600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f825c112ca0 con 0x7f825c102af0 2024-04-26T01:55:27.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.169+0000 7f8264600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f825c112b20 con 0x7f825c1034b0 2024-04-26T01:55:27.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.169+0000 7f8264600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f825c112e20 con 0x7f825c107940 2024-04-26T01:55:27.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.169+0000 7f8261c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f825c102af0 0x7f825c199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.169+0000 7f8261c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f825c102af0 0x7f825c199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45568/0 (socket says 172.21.15.6:45568) 2024-04-26T01:55:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.169+0000 7f8261c00700 1 -- 172.21.15.6:0/2741082901 learned_addr learned my addr 172.21.15.6:0/2741082901 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.169+0000 7f8261200700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f825c1034b0 0x7f825c199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.170+0000 7f8262600700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f825c107940 0x7f825c19d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.170+0000 7f8261200700 1 -- 172.21.15.6:0/2741082901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f825c107940 msgr2=0x7f825c19d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.170+0000 7f8261200700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f825c107940 0x7f825c19d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.170+0000 7f8261200700 1 -- 172.21.15.6:0/2741082901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f825c102af0 msgr2=0x7f825c199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.170+0000 7f8261200700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f825c102af0 0x7f825c199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.170+0000 7f8261200700 1 -- 172.21.15.6:0/2741082901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c19e120 con 0x7f825c1034b0 2024-04-26T01:55:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.170+0000 7f8261c00700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f825c102af0 0x7f825c199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.170+0000 7f8261200700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f825c1034b0 0x7f825c199650 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f824800e9f0 tx=0x7f824800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:27.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.171+0000 7f825a000700 1 -- 172.21.15.6:0/2741082901 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248004220 con 0x7f825c1034b0 2024-04-26T01:55:27.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.171+0000 7f825a000700 1 -- 172.21.15.6:0/2741082901 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82480043c0 con 0x7f825c1034b0 2024-04-26T01:55:27.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.171+0000 7f8264600700 1 -- 172.21.15.6:0/2741082901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825c19e410 con 0x7f825c1034b0 2024-04-26T01:55:27.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.171+0000 7f825a000700 1 -- 172.21.15.6:0/2741082901 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248005020 con 0x7f825c1034b0 2024-04-26T01:55:27.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.171+0000 7f8264600700 1 -- 172.21.15.6:0/2741082901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f825c1a5c90 con 0x7f825c1034b0 2024-04-26T01:55:27.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.172+0000 7f8264600700 1 -- 172.21.15.6:0/2741082901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f825c069730 con 0x7f825c1034b0 2024-04-26T01:55:27.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.174+0000 7f825a000700 1 -- 172.21.15.6:0/2741082901 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f8248004750 con 0x7f825c1034b0 2024-04-26T01:55:27.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.175+0000 7f825a000700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f82440a3db0 0x7f82440a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.176+0000 7f825a000700 1 -- 172.21.15.6:0/2741082901 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f82480d0900 con 0x7f825c1034b0 2024-04-26T01:55:27.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.176+0000 7f8261c00700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f82440a3db0 0x7f82440a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.177+0000 7f8261c00700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f82440a3db0 0x7f82440a6270 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8250000c00 tx=0x7f8250005fd0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:27.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.178+0000 7f825a000700 1 -- 172.21.15.6:0/2741082901 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f824809cf80 con 0x7f825c1034b0 2024-04-26T01:55:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.428+0000 7f0988600640 1 -- 172.21.15.6:0/301371386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09801043c0 msgr2=0x7f09801047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.428+0000 7f0988600640 1 --2- 172.21.15.6:0/301371386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09801043c0 0x7f09801047a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0974009900 tx=0x7f097402a3f0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.428+0000 7f0988600640 1 -- 172.21.15.6:0/301371386 shutdown_connections 2024-04-26T01:55:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.428+0000 7f0988600640 1 --2- 172.21.15.6:0/301371386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0980104d70 0x7f0980108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.428+0000 7f0988600640 1 --2- 172.21.15.6:0/301371386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0980109490 0x7f0980111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.428+0000 7f0988600640 1 --2- 172.21.15.6:0/301371386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09801043c0 0x7f09801047a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.428+0000 7f0988600640 1 -- 172.21.15.6:0/301371386 >> 172.21.15.6:0/301371386 conn(0x7f09800ffc40 msgr2=0x7f0980102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.428+0000 7f0988600640 1 -- 172.21.15.6:0/301371386 shutdown_connections 2024-04-26T01:55:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.428+0000 7f0988600640 1 -- 172.21.15.6:0/301371386 wait complete. 2024-04-26T01:55:27.429 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.429+0000 7f0988600640 1 Processor -- start 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.429+0000 7f0988600640 1 -- start start 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.429+0000 7f0988600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09801043c0 0x7f09801a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.429+0000 7f0988600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0980109490 0x7f09801a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.429+0000 7f0988600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f098019a7d0 0x7f098019ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.429+0000 7f0988600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0980114450 con 0x7f0980109490 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.429+0000 7f0988600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f09801142d0 con 0x7f09801043c0 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.429+0000 7f0988600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f09801145d0 con 0x7f098019a7d0 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.430+0000 7f0986600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f098019a7d0 0x7f098019ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.430+0000 7f0986600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f098019a7d0 0x7f098019ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:33068/0 (socket says 172.21.15.6:33068) 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.430+0000 7f0985200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0980109490 0x7f09801a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.430+0000 7f0986600640 1 -- 172.21.15.6:0/1471588508 learned_addr learned my addr 172.21.15.6:0/1471588508 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.430+0000 7f0985c00640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09801043c0 0x7f09801a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.430+0000 7f0986600640 1 -- 172.21.15.6:0/1471588508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09801043c0 msgr2=0x7f09801a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.430+0000 7f0986600640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09801043c0 0x7f09801a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.430+0000 7f0986600640 1 -- 172.21.15.6:0/1471588508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0980109490 msgr2=0x7f09801a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.430+0000 7f0986600640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0980109490 0x7f09801a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.430+0000 7f0986600640 1 -- 172.21.15.6:0/1471588508 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f098019b360 con 0x7f098019a7d0 2024-04-26T01:55:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.430+0000 7f0985200640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0980109490 0x7f09801a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.431+0000 7f0986600640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f098019a7d0 0x7f098019ac80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f097800eb10 tx=0x7f097800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.431+0000 7f097e000640 1 -- 172.21.15.6:0/1471588508 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0978004b20 con 0x7f098019a7d0 2024-04-26T01:55:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.431+0000 7f097e000640 1 -- 172.21.15.6:0/1471588508 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0978004cc0 con 0x7f098019a7d0 2024-04-26T01:55:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.431+0000 7f097e000640 1 -- 172.21.15.6:0/1471588508 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0978010670 con 0x7f098019a7d0 2024-04-26T01:55:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.431+0000 7f0988600640 1 -- 172.21.15.6:0/1471588508 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f098019b650 con 0x7f098019a7d0 2024-04-26T01:55:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.431+0000 7f0988600640 1 -- 172.21.15.6:0/1471588508 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09801a7830 con 0x7f098019a7d0 2024-04-26T01:55:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.434+0000 7f098b444740 1 Processor -- start 2024-04-26T01:55:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.434+0000 7f097e000640 1 -- 172.21.15.6:0/1471588508 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f0978005020 con 0x7f098019a7d0 2024-04-26T01:55:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.434+0000 7f098b444740 1 -- start start 2024-04-26T01:55:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.435+0000 7f098b444740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfb7820 0x5627bdfb7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.435+0000 7f098b444740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 0x5627bdfc24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.435+0000 7f098b444740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfc2a00 0x5627bdfc4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.435+0000 7f098b444740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5627bdf1d900 con 0x5627bdfb8140 2024-04-26T01:55:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.435+0000 7f098b444740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5627bdec3ef0 con 0x5627bdfc2a00 2024-04-26T01:55:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.435+0000 7f098b444740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5627bdf198c0 con 0x5627bdfb7820 2024-04-26T01:55:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.435+0000 7f0985200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfb7820 0x5627bdfb7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.435+0000 7f0985200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfb7820 0x5627bdfb7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:33080/0 (socket says 172.21.15.6:33080) 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.435+0000 7f0985200640 1 -- 172.21.15.6:0/2446095746 learned_addr learned my addr 172.21.15.6:0/2446095746 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.435+0000 7f0985200640 1 --2- 172.21.15.6:0/2446095746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfc2a00 0x5627bdfc4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.435+0000 7f0985c00640 1 --2- 172.21.15.6:0/2446095746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 0x5627bdfc24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f0985200640 1 -- 172.21.15.6:0/2446095746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfc2a00 msgr2=0x5627bdfc4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f0985200640 1 --2- 172.21.15.6:0/2446095746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfc2a00 0x5627bdfc4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f0985200640 1 -- 172.21.15.6:0/2446095746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 msgr2=0x5627bdfc24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f0985200640 1 --2- 172.21.15.6:0/2446095746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 0x5627bdfc24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f0985200640 1 -- 172.21.15.6:0/2446095746 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627bdc30210 con 0x5627bdfb7820 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f0985c00640 1 --2- 172.21.15.6:0/2446095746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 0x5627bdfc24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f0985200640 1 --2- 172.21.15.6:0/2446095746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfb7820 0x5627bdfb7c00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f09700058c0 tx=0x7f0970005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48cbd6d359db6342 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f097e000640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f09680a3d10 0x7f09680a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f096ea00640 1 -- 172.21.15.6:0/2446095746 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0970011050 con 0x5627bdfb7820 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f096ea00640 1 -- 172.21.15.6:0/2446095746 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09700111f0 con 0x5627bdfb7820 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f096ea00640 1 -- 172.21.15.6:0/2446095746 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09700035f0 con 0x5627bdfb7820 2024-04-26T01:55:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f097e000640 1 -- 172.21.15.6:0/1471588508 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f09780d0d90 con 0x7f098019a7d0 2024-04-26T01:55:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.436+0000 7f0985c00640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f09680a3d10 0x7f09680a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.437+0000 7f098b444740 1 -- 172.21.15.6:0/2446095746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfb7820 msgr2=0x5627bdfb7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.437+0000 7f098b444740 1 --2- 172.21.15.6:0/2446095746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfb7820 0x5627bdfb7c00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f09700058c0 tx=0x7f0970005850 comp rx=0 tx=0).stop 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.437+0000 7f098b444740 1 -- 172.21.15.6:0/2446095746 shutdown_connections 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.437+0000 7f098b444740 1 --2- 172.21.15.6:0/2446095746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 0x5627bdfc24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.437+0000 7f098b444740 1 --2- 172.21.15.6:0/2446095746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfc2a00 0x5627bdfc4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.437+0000 7f098b444740 1 --2- 172.21.15.6:0/2446095746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfb7820 0x5627bdfb7c00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.437+0000 7f098b444740 1 -- 172.21.15.6:0/2446095746 >> 172.21.15.6:0/2446095746 conn(0x5627bdfb3d30 msgr2=0x5627bdfb4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.437+0000 7f098b444740 1 -- 172.21.15.6:0/2446095746 shutdown_connections 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.437+0000 7f098b444740 1 -- 172.21.15.6:0/2446095746 wait complete. 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.437+0000 7f0985c00640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f09680a3d10 0x7f09680a61d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0974005ab0 tx=0x7f0974005a20 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.438+0000 7f098b444740 1 -- start start 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.438+0000 7f098b444740 1 Processor -- start 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.438+0000 7f098b444740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb7820 0x5627bdfbf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.438+0000 7f098b444740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 0x5627bdfbfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.438+0000 7f098b444740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfc2a00 0x5627bdfc03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.438+0000 7f098b444740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5627bdfc5630 con 0x5627bdfb8140 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.438+0000 7f098b444740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5627bded7310 con 0x5627bdfb7820 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.438+0000 7f098b444740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5627bdfc57b0 con 0x5627bdfc2a00 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.438+0000 7f098b444740 10 client.0 _ll_register_callbacks cb 0x5627bdea5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.439+0000 7f0985200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 0x5627bdfbfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.439+0000 7f0985200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 0x5627bdfbfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45614/0 (socket says 172.21.15.6:45614) 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.439+0000 7f0985200640 1 -- 172.21.15.6:0/20216827 learned_addr learned my addr 172.21.15.6:0/20216827 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.439+0000 7f0985c00640 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb7820 0x5627bdfbf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.439+0000 7f0986600640 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfc2a00 0x5627bdfc03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.439+0000 7f0985200640 1 -- 172.21.15.6:0/20216827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfc2a00 msgr2=0x5627bdfc03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.439+0000 7f0985200640 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfc2a00 0x5627bdfc03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.439+0000 7f0985200640 1 -- 172.21.15.6:0/20216827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb7820 msgr2=0x5627bdfbf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.439+0000 7f0985200640 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb7820 0x5627bdfbf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.439+0000 7f0985200640 1 -- 172.21.15.6:0/20216827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627bdcfc620 con 0x5627bdfb8140 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.440+0000 7f0985200640 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 0x5627bdfbfea0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0970011610 tx=0x7f09700056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191fb560fcfb6c76 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.440+0000 7f0963400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.440+0000 7f0963400640 1 -- 172.21.15.6:0/20216827 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097001f640 con 0x5627bdfb8140 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.440+0000 7f0963400640 1 -- 172.21.15.6:0/20216827 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f097001f7e0 con 0x5627bdfb8140 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.440+0000 7f098b444740 1 -- 172.21.15.6:0/20216827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09700111f0 con 0x5627bdfb8140 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.440+0000 7f0963400640 1 -- 172.21.15.6:0/20216827 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f097001fa20 con 0x5627bdfb8140 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.440+0000 7f098b444740 10 client.16167 fetch_fsmap learned FSMap version 43 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.440+0000 7f098b444740 1 -- 172.21.15.6:0/20216827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5627bdfd1850 con 0x5627bdfb8140 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.441+0000 7f0963400640 1 -- 172.21.15.6:0/20216827 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09700111f0 con 0x5627bdfb8140 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.441+0000 7f0963400640 1 -- 172.21.15.6:0/20216827 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f0970029cb0 con 0x5627bdfb8140 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.441+0000 7f098b444740 10 client.16167 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.441+0000 7f098b444740 10 client.16167 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.441+0000 7f098b444740 20 client.16167 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.441+0000 7f098b444740 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5627bdfd2ba0 0x5627bdfd4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.441+0000 7f098b444740 4 client.16167 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.441+0000 7f098b444740 1 -- 172.21.15.6:0/20216827 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627bdf318b0 con 0x5627bdfd2ba0 2024-04-26T01:55:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.442+0000 7f0985c00640 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5627bdfd2ba0 0x5627bdfd4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.442+0000 7f0985c00640 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5627bdfd2ba0 0x5627bdfd4f90 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cf08dea762a71cf server_cookie=33f2b07634745e57 in_seq=0 out_seq=0 2024-04-26T01:55:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.442+0000 7f0963400640 10 client.16167 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f0963400640 1 -- 172.21.15.6:0/20216827 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5627bdf318b0 con 0x5627bdfd2ba0 2024-04-26T01:55:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f0963400640 10 client.16167 handle_command_reply: tid=0 2024-04-26T01:55:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f098b444740 1 client.16167 shutdown 2024-04-26T01:55:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f098b444740 1 -- 172.21.15.6:0/20216827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 msgr2=0x5627bdfbfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f098b444740 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 0x5627bdfbfea0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0970011610 tx=0x7f09700056d0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f098b444740 1 -- 172.21.15.6:0/20216827 shutdown_connections 2024-04-26T01:55:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f098b444740 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5627bdfd2ba0 0x5627bdfd4f90 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f098b444740 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb8140 0x5627bdfbfea0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f098b444740 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfc2a00 0x5627bdfc03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f0963400640 0 client.16167 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f098b444740 1 --2- 172.21.15.6:0/20216827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb7820 0x5627bdfbf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f098b444740 1 -- 172.21.15.6:0/20216827 >> 172.21.15.6:0/20216827 conn(0x5627bdfb3d30 msgr2=0x5627bdfb9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.443+0000 7f098b444740 1 -- 172.21.15.6:0/20216827 shutdown_connections 2024-04-26T01:55:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.444+0000 7f098b444740 1 -- 172.21.15.6:0/20216827 wait complete. 2024-04-26T01:55:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.444+0000 7f098b444740 20 client.16167 trim_cache size 0 max 16384 2024-04-26T01:55:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.459+0000 7f8264600700 1 -- 172.21.15.6:0/2741082901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f825c02df80 con 0x7f825c1034b0 2024-04-26T01:55:27.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.460+0000 7f098b444740 1 Processor -- start 2024-04-26T01:55:27.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.460+0000 7f098b444740 1 -- start start 2024-04-26T01:55:27.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.460+0000 7f098b444740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb7820 0x5627bdfce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.460+0000 7f098b444740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb8140 0x5627bdfda760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.460+0000 7f098b444740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfc2a00 0x5627bdfdaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.460+0000 7f098b444740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f09801b28e0 con 0x5627bdfb7820 2024-04-26T01:55:27.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.460+0000 7f098b444740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f09801b01b0 con 0x5627bdfb8140 2024-04-26T01:55:27.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.460+0000 7f098b444740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f09801ae020 con 0x5627bdfc2a00 2024-04-26T01:55:27.460 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:55:27.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.460+0000 7f0985c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb7820 0x5627bdfce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.461+0000 7f0985200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb8140 0x5627bdfda760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.461+0000 7f0985c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb7820 0x5627bdfce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45626/0 (socket says 172.21.15.6:45626) 2024-04-26T01:55:27.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.461+0000 7f0985c00640 1 -- 172.21.15.6:0/1219525052 learned_addr learned my addr 172.21.15.6:0/1219525052 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:27.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.461+0000 7f0986600640 1 --2- 172.21.15.6:0/1219525052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfc2a00 0x5627bdfdaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.460+0000 7f825a000700 1 -- 172.21.15.6:0/2741082901 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11850 (secure 0 0 0) 0x7f824809c880 con 0x7f825c1034b0 2024-04-26T01:55:27.461 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":43,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:55:18.912671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:55:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.461+0000 7f0985200640 1 -- 172.21.15.6:0/1219525052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfc2a00 msgr2=0x5627bdfdaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:27.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.461+0000 7f0985200640 1 --2- 172.21.15.6:0/1219525052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfc2a00 0x5627bdfdaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.461+0000 7f0985200640 1 -- 172.21.15.6:0/1219525052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb7820 msgr2=0x5627bdfce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.461+0000 7f0985200640 1 --2- 172.21.15.6:0/1219525052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb7820 0x5627bdfce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.461+0000 7f0985200640 1 -- 172.21.15.6:0/1219525052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627bdfb7f70 con 0x5627bdfb8140 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.461+0000 7f0986600640 1 --2- 172.21.15.6:0/1219525052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfc2a00 0x5627bdfdaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.461+0000 7f0985200640 1 --2- 172.21.15.6:0/1219525052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb8140 0x5627bdfda760 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0970010010 tx=0x7f097000fc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5104034478d65900 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.461+0000 7f096e000640 1 -- 172.21.15.6:0/1219525052 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0970011460 con 0x5627bdfb8140 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.462+0000 7f096e000640 1 -- 172.21.15.6:0/1219525052 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0970005b90 con 0x5627bdfb8140 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.462+0000 7f096e000640 1 -- 172.21.15.6:0/1219525052 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097000ed90 con 0x5627bdfb8140 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.462+0000 7f098b444740 1 -- 172.21.15.6:0/1219525052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb8140 msgr2=0x5627bdfda760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.462+0000 7f098b444740 1 --2- 172.21.15.6:0/1219525052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb8140 0x5627bdfda760 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0970010010 tx=0x7f097000fc50 comp rx=0 tx=0).stop 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.462+0000 7f098b444740 1 -- 172.21.15.6:0/1219525052 shutdown_connections 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.462+0000 7f098b444740 1 --2- 172.21.15.6:0/1219525052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfc2a00 0x5627bdfdaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.462+0000 7f098b444740 1 --2- 172.21.15.6:0/1219525052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb8140 0x5627bdfda760 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.462+0000 7f098b444740 1 --2- 172.21.15.6:0/1219525052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfb7820 0x5627bdfce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.464 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.462+0000 7f098b444740 1 -- 172.21.15.6:0/1219525052 >> 172.21.15.6:0/1219525052 conn(0x5627bdfb3d30 msgr2=0x5627bdfb4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:27.464 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.462+0000 7f098b444740 1 -- 172.21.15.6:0/1219525052 shutdown_connections 2024-04-26T01:55:27.464 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.463+0000 7f098b444740 1 -- 172.21.15.6:0/1219525052 wait complete. 2024-04-26T01:55:27.464 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.463+0000 7f098b444740 1 -- start start 2024-04-26T01:55:27.464 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.463+0000 7f098b444740 1 Processor -- start 2024-04-26T01:55:27.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.463+0000 7f8264600700 1 -- 172.21.15.6:0/2741082901 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f82440a3db0 msgr2=0x7f82440a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.463+0000 7f8264600700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f82440a3db0 0x7f82440a6270 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8250000c00 tx=0x7f8250005fd0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.463+0000 7f8264600700 1 -- 172.21.15.6:0/2741082901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f825c1034b0 msgr2=0x7f825c199650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.463+0000 7f8264600700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f825c1034b0 0x7f825c199650 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f824800e9f0 tx=0x7f824800ed00 comp rx=0 tx=0).stop 2024-04-26T01:55:27.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.463+0000 7f8264600700 1 -- 172.21.15.6:0/2741082901 shutdown_connections 2024-04-26T01:55:27.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.463+0000 7f8264600700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f82440a3db0 0x7f82440a6270 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f8264600700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f825c107940 0x7f825c19d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f8264600700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f825c102af0 0x7f825c199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f8264600700 1 --2- 172.21.15.6:0/2741082901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f825c1034b0 0x7f825c199650 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f8264600700 1 -- 172.21.15.6:0/2741082901 >> 172.21.15.6:0/2741082901 conn(0x7f825c079ba0 msgr2=0x7f825c100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f098b444740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb7820 0x5627bdfcace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f098b444740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfb8140 0x5627bdfc8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f098b444740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfc2a00 0x5627bdfc85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f098b444740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f09801ae670 con 0x5627bdfc2a00 2024-04-26T01:55:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f098b444740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f09801afae0 con 0x5627bdfb7820 2024-04-26T01:55:27.466 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f098b444740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f09801aed40 con 0x5627bdfb8140 2024-04-26T01:55:27.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f8264600700 1 -- 172.21.15.6:0/2741082901 shutdown_connections 2024-04-26T01:55:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f098b444740 10 client.0 _ll_register_callbacks cb 0x7f0970005f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f0985c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb7820 0x5627bdfcace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f0985c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb7820 0x5627bdfcace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36598/0 (socket says 172.21.15.6:36598) 2024-04-26T01:55:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f0985c00640 1 -- 172.21.15.6:0/1185706029 learned_addr learned my addr 172.21.15.6:0/1185706029 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f0985200640 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfb8140 0x5627bdfc8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f8264600700 1 -- 172.21.15.6:0/2741082901 wait complete. 2024-04-26T01:55:27.468 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.464+0000 7f0986600640 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfc2a00 0x5627bdfc85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.468 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.465+0000 7f0985c00640 1 -- 172.21.15.6:0/1185706029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfb8140 msgr2=0x5627bdfc8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:27.468 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.465+0000 7f0985c00640 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfb8140 0x5627bdfc8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.468 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.465+0000 7f0985c00640 1 -- 172.21.15.6:0/1185706029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfc2a00 msgr2=0x5627bdfc85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:27.468 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.465+0000 7f0985c00640 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfc2a00 0x5627bdfc85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.465+0000 7f0985c00640 1 -- 172.21.15.6:0/1185706029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627bdfc8cf0 con 0x5627bdfb7820 2024-04-26T01:55:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.465+0000 7f0986600640 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfc2a00 0x5627bdfc85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.465+0000 7f0985c00640 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb7820 0x5627bdfcace0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f097402a910 tx=0x7f0974009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1489a145dd4fc6f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.465+0000 7f0963400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:55:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.465+0000 7f0963400640 1 -- 172.21.15.6:0/1185706029 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974035040 con 0x5627bdfb7820 2024-04-26T01:55:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.465+0000 7f0963400640 1 -- 172.21.15.6:0/1185706029 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f0974053500 con 0x5627bdfb7820 2024-04-26T01:55:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.465+0000 7f098b444740 1 -- 172.21.15.6:0/1185706029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0970005b90 con 0x5627bdfb7820 2024-04-26T01:55:27.469 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-26T01:55:27.470 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.466+0000 7f0963400640 1 -- 172.21.15.6:0/1185706029 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0974053720 con 0x5627bdfb7820 2024-04-26T01:55:27.470 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.466+0000 7f098b444740 10 client.26315 fetch_fsmap learned FSMap version 43 2024-04-26T01:55:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.466+0000 7f098b444740 1 -- 172.21.15.6:0/1185706029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5627be038f20 con 0x5627bdfb7820 2024-04-26T01:55:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.466+0000 7f0963400640 1 -- 172.21.15.6:0/1185706029 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0970005b90 con 0x5627bdfb7820 2024-04-26T01:55:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.466+0000 7f0963400640 1 -- 172.21.15.6:0/1185706029 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f097405a070 con 0x5627bdfb7820 2024-04-26T01:55:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.466+0000 7f098b444740 10 client.26315 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:55:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.466+0000 7f098b444740 10 client.26315 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.466+0000 7f098b444740 20 client.26315 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.466+0000 7f098b444740 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5627be0395f0 0x5627be03ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.466+0000 7f098b444740 4 client.26315 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.466+0000 7f098b444740 1 -- 172.21.15.6:0/1185706029 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5627bdf2b320 con 0x5627be0395f0 2024-04-26T01:55:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.467+0000 7f0985200640 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5627be0395f0 0x5627be03ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.467+0000 7f0985200640 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5627be0395f0 0x5627be03ba10 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1150cf745b84e897 server_cookie=b3f4eebe5afb4459 in_seq=0 out_seq=0 2024-04-26T01:55:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.467+0000 7f0963400640 10 client.26315 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.494+0000 7f0963400640 1 -- 172.21.15.6:0/1185706029 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5627bdf2b320 con 0x5627be0395f0 2024-04-26T01:55:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.494+0000 7f0963400640 10 client.26315 handle_command_reply: tid=0 2024-04-26T01:55:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.494+0000 7f098b444740 1 client.26315 shutdown 2024-04-26T01:55:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.494+0000 7f098b444740 1 -- 172.21.15.6:0/1185706029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb7820 msgr2=0x5627bdfcace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.494+0000 7f098b444740 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb7820 0x5627bdfcace0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f097402a910 tx=0x7f0974009260 comp rx=0 tx=0).stop 2024-04-26T01:55:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.494+0000 7f098b444740 1 -- 172.21.15.6:0/1185706029 shutdown_connections 2024-04-26T01:55:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.494+0000 7f098b444740 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5627be0395f0 0x5627be03ba10 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.494+0000 7f098b444740 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5627bdfc2a00 0x5627bdfc85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.494+0000 7f098b444740 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627bdfb8140 0x5627bdfc8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.494+0000 7f098b444740 1 --2- 172.21.15.6:0/1185706029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627bdfb7820 0x5627bdfcace0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.494+0000 7f0963400640 0 client.26315 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.494+0000 7f098b444740 1 -- 172.21.15.6:0/1185706029 >> 172.21.15.6:0/1185706029 conn(0x5627bdfb3d30 msgr2=0x5627bdfbf900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.495+0000 7f098b444740 1 -- 172.21.15.6:0/1185706029 shutdown_connections 2024-04-26T01:55:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.495+0000 7f098b444740 1 -- 172.21.15.6:0/1185706029 wait complete. 2024-04-26T01:55:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.495+0000 7f098b444740 20 client.26315 trim_cache size 0 max 16384 2024-04-26T01:55:27.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:55:27.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:55:27.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:55:27.495 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:55:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.495+0000 7f0988600640 1 -- 172.21.15.6:0/1471588508 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f09680a3d10 msgr2=0x7f09680a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.495+0000 7f0988600640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f09680a3d10 0x7f09680a61d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0974005ab0 tx=0x7f0974005a20 comp rx=0 tx=0).stop 2024-04-26T01:55:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.495+0000 7f0988600640 1 -- 172.21.15.6:0/1471588508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f098019a7d0 msgr2=0x7f098019ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.495+0000 7f0988600640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f098019a7d0 0x7f098019ac80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f097800eb10 tx=0x7f097800c2d0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.495+0000 7f0988600640 1 -- 172.21.15.6:0/1471588508 shutdown_connections 2024-04-26T01:55:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.496+0000 7f0988600640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f09680a3d10 0x7f09680a61d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.496+0000 7f0988600640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0980109490 0x7f09801a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.496+0000 7f0988600640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f098019a7d0 0x7f098019ac80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.496+0000 7f0988600640 1 --2- 172.21.15.6:0/1471588508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09801043c0 0x7f09801a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.496+0000 7f0988600640 1 -- 172.21.15.6:0/1471588508 >> 172.21.15.6:0/1471588508 conn(0x7f09800ffc40 msgr2=0x7f0980101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.496+0000 7f0988600640 1 -- 172.21.15.6:0/1471588508 shutdown_connections 2024-04-26T01:55:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:27.496+0000 7f0988600640 1 -- 172.21.15.6:0/1471588508 wait complete. 2024-04-26T01:55:27.612 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.65 MB/sec execute 351 sec latency 26135.125 ms 2024-04-26T01:55:27.808 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:27 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2741082901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:27.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:27 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2741082901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:27.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:27 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2741082901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:28.260 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id 8a864be1 --release --await' 2024-04-26T01:55:28.261 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id 8a864be1 --release --await 2024-04-26T01:55:28.612 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.64 MB/sec execute 352 sec latency 27135.245 ms 2024-04-26T01:55:28.739 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:55:28.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:28 smithi006 ceph-mon[28419]: pgmap v301: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.4 MiB/s wr, 67 op/s 2024-04-26T01:55:28.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:28 smithi188 ceph-mon[32751]: pgmap v301: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.4 MiB/s wr, 67 op/s 2024-04-26T01:55:28.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:28 smithi029 ceph-mon[32775]: pgmap v301: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.4 MiB/s wr, 67 op/s 2024-04-26T01:55:29.612 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.63 MB/sec execute 353 sec latency 28135.384 ms 2024-04-26T01:55:30.613 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330274 3.62 MB/sec execute 354 sec latency 29135.475 ms 2024-04-26T01:55:30.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:30 smithi006 ceph-mon[28419]: pgmap v302: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 895 KiB/s wr, 38 op/s 2024-04-26T01:55:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:30 smithi188 ceph-mon[32751]: pgmap v302: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 895 KiB/s wr, 38 op/s 2024-04-26T01:55:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:30 smithi029 ceph-mon[32775]: pgmap v302: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 895 KiB/s wr, 38 op/s 2024-04-26T01:55:31.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.087+0000 7f277a200700 1 -- 172.21.15.6:0/2505877640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f277406afe0 msgr2=0x7f277406b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:31.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.087+0000 7f277a200700 1 --2- 172.21.15.6:0/2505877640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f277406afe0 0x7f277406b440 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2760009990 tx=0x7f2760009ca0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.087+0000 7f277a200700 1 -- 172.21.15.6:0/2505877640 shutdown_connections 2024-04-26T01:55:31.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.087+0000 7f277a200700 1 --2- 172.21.15.6:0/2505877640 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f277406b980 0x7f277410f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.087+0000 7f277a200700 1 --2- 172.21.15.6:0/2505877640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2774108a20 0x7f2774108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.087+0000 7f277a200700 1 --2- 172.21.15.6:0/2505877640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f277406afe0 0x7f277406b440 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.087+0000 7f277a200700 1 -- 172.21.15.6:0/2505877640 >> 172.21.15.6:0/2505877640 conn(0x7f27740788b0 msgr2=0x7f2774078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:31.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.087+0000 7f277a200700 1 -- 172.21.15.6:0/2505877640 shutdown_connections 2024-04-26T01:55:31.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.087+0000 7f277a200700 1 -- 172.21.15.6:0/2505877640 wait complete. 2024-04-26T01:55:31.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.089+0000 7f277a200700 1 Processor -- start 2024-04-26T01:55:31.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.089+0000 7f277a200700 1 -- start start 2024-04-26T01:55:31.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.090+0000 7f277a200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f277406afe0 0x7f277419b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.090+0000 7f277a200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f277406b980 0x7f277419b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.090+0000 7f277a200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2774108a20 0x7f277419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.090+0000 7f277a200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2774112c10 con 0x7f277406b980 2024-04-26T01:55:31.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.090+0000 7f277a200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2774112a90 con 0x7f2774108a20 2024-04-26T01:55:31.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.090+0000 7f277a200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2774112d90 con 0x7f277406afe0 2024-04-26T01:55:31.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.090+0000 7f2772a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f277406b980 0x7f277419b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.090+0000 7f2773400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f277406afe0 0x7f277419b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.090+0000 7f2773e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2774108a20 0x7f277419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.090+0000 7f2773400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f277406afe0 0x7f277419b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:49358/0 (socket says 172.21.15.6:49358) 2024-04-26T01:55:31.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.090+0000 7f2773400700 1 -- 172.21.15.6:0/2763196242 learned_addr learned my addr 172.21.15.6:0/2763196242 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:31.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.091+0000 7f2773400700 1 -- 172.21.15.6:0/2763196242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2774108a20 msgr2=0x7f277419fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:31.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.091+0000 7f2773400700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2774108a20 0x7f277419fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.091+0000 7f2773400700 1 -- 172.21.15.6:0/2763196242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f277406b980 msgr2=0x7f277419b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.091+0000 7f2773400700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f277406b980 0x7f277419b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.091+0000 7f2773400700 1 -- 172.21.15.6:0/2763196242 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27741a0380 con 0x7f277406afe0 2024-04-26T01:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.091+0000 7f2772a00700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f277406b980 0x7f277419b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.091+0000 7f2773400700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f277406afe0 0x7f277419b370 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f276400e9a0 tx=0x7f276400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.091+0000 7f276be00700 1 -- 172.21.15.6:0/2763196242 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276400ca10 con 0x7f277406afe0 2024-04-26T01:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.091+0000 7f276be00700 1 -- 172.21.15.6:0/2763196242 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f276400cbb0 con 0x7f277406afe0 2024-04-26T01:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.091+0000 7f276be00700 1 -- 172.21.15.6:0/2763196242 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764018680 con 0x7f277406afe0 2024-04-26T01:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.091+0000 7f277a200700 1 -- 172.21.15.6:0/2763196242 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27741a0670 con 0x7f277406afe0 2024-04-26T01:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.091+0000 7f277a200700 1 -- 172.21.15.6:0/2763196242 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27741a7ef0 con 0x7f277406afe0 2024-04-26T01:55:31.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.095+0000 7f276be00700 1 -- 172.21.15.6:0/2763196242 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f2764010430 con 0x7f277406afe0 2024-04-26T01:55:31.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.096+0000 7f2784afab80 1 Processor -- start 2024-04-26T01:55:31.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.096+0000 7f2784afab80 1 -- start start 2024-04-26T01:55:31.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f276be00700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27580a3db0 0x7f27580a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f2784afab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 0x564a1001dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f2784afab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a1001e240 0x564a10028bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f2784afab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a100290f0 0x564a1002b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f2784afab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564a0ff70f90 con 0x564a1001d920 2024-04-26T01:55:31.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f2784afab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564a0ff17830 con 0x564a100290f0 2024-04-26T01:55:31.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f2784afab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564a0ff6cf50 con 0x564a1001e240 2024-04-26T01:55:31.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f276be00700 1 -- 172.21.15.6:0/2763196242 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f27640d09c0 con 0x7f277406afe0 2024-04-26T01:55:31.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f2773e00700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27580a3db0 0x7f27580a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f2773400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 0x564a1001dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f2773400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 0x564a1001dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59994/0 (socket says 172.21.15.6:59994) 2024-04-26T01:55:31.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f2773400700 1 -- 172.21.15.6:0/1039353215 learned_addr learned my addr 172.21.15.6:0/1039353215 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:31.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f2772a00700 1 --2- 172.21.15.6:0/1039353215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a1001e240 0x564a10028bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.097+0000 7f2773e00700 1 --2- 172.21.15.6:0/1039353215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a100290f0 0x564a1002b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.098+0000 7f2773400700 1 -- 172.21.15.6:0/1039353215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a1001e240 msgr2=0x564a10028bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:31.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.098+0000 7f2773400700 1 --2- 172.21.15.6:0/1039353215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a1001e240 0x564a10028bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.098+0000 7f2773400700 1 -- 172.21.15.6:0/1039353215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a100290f0 msgr2=0x564a1002b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:31.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.098+0000 7f2773400700 1 --2- 172.21.15.6:0/1039353215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a100290f0 0x564a1002b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.098+0000 7f2773400700 1 -- 172.21.15.6:0/1039353215 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a0ff0d780 con 0x564a1001d920 2024-04-26T01:55:31.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.098+0000 7f2772a00700 1 --2- 172.21.15.6:0/1039353215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a1001e240 0x564a10028bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:31.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.098+0000 7f2773e00700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27580a3db0 0x7f27580a6270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f276c009ec0 tx=0x7f276c0092f0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:31.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.098+0000 7f2773400700 1 --2- 172.21.15.6:0/1039353215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 0x564a1001dd00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2764099280 tx=0x7f276409cfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45da68ddf4cd386c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:31.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.098+0000 7f2768c00700 1 -- 172.21.15.6:0/1039353215 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27640a0c60 con 0x564a1001d920 2024-04-26T01:55:31.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.099+0000 7f2768c00700 1 -- 172.21.15.6:0/1039353215 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27640a5520 con 0x564a1001d920 2024-04-26T01:55:31.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.099+0000 7f2768c00700 1 -- 172.21.15.6:0/1039353215 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27640ce2e0 con 0x564a1001d920 2024-04-26T01:55:31.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.099+0000 7f2784afab80 1 -- 172.21.15.6:0/1039353215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 msgr2=0x564a1001dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:31.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.099+0000 7f2784afab80 1 --2- 172.21.15.6:0/1039353215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 0x564a1001dd00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2764099280 tx=0x7f276409cfb0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.099+0000 7f2784afab80 1 -- 172.21.15.6:0/1039353215 shutdown_connections 2024-04-26T01:55:31.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.099+0000 7f2784afab80 1 --2- 172.21.15.6:0/1039353215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a100290f0 0x564a1002b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.099+0000 7f2784afab80 1 --2- 172.21.15.6:0/1039353215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 0x564a1001dd00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.099+0000 7f2784afab80 1 --2- 172.21.15.6:0/1039353215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a1001e240 0x564a10028bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.099+0000 7f2784afab80 1 -- 172.21.15.6:0/1039353215 >> 172.21.15.6:0/1039353215 conn(0x564a0ff4c6a0 msgr2=0x564a10019dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:31.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.100+0000 7f2784afab80 1 -- 172.21.15.6:0/1039353215 shutdown_connections 2024-04-26T01:55:31.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.100+0000 7f2784afab80 1 -- 172.21.15.6:0/1039353215 wait complete. 2024-04-26T01:55:31.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.100+0000 7f2784afab80 1 -- start start 2024-04-26T01:55:31.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.100+0000 7f2784afab80 1 Processor -- start 2024-04-26T01:55:31.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.101+0000 7f2784afab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a1001d920 0x564a10028870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.101+0000 7f2784afab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001e240 0x564a10025b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.101+0000 7f2784afab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a100290f0 0x564a100260b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.101+0000 7f2784afab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564a1002bc10 con 0x564a100290f0 2024-04-26T01:55:31.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.101+0000 7f2784afab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564a0ff2a960 con 0x564a1001e240 2024-04-26T01:55:31.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.101+0000 7f2784afab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564a1002bd90 con 0x564a1001d920 2024-04-26T01:55:31.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.101+0000 7f2784afab80 10 client.0 _ll_register_callbacks cb 0x564a0ff09820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:31.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.101+0000 7f2772a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001e240 0x564a10025b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.101+0000 7f2772a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001e240 0x564a10025b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52194/0 (socket says 172.21.15.6:52194) 2024-04-26T01:55:31.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.101+0000 7f2773400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a1001d920 0x564a10028870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.101+0000 7f2772a00700 1 -- 172.21.15.6:0/2922481962 learned_addr learned my addr 172.21.15.6:0/2922481962 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:31.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.101+0000 7f2773e00700 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a100290f0 0x564a100260b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.102+0000 7f2772a00700 1 -- 172.21.15.6:0/2922481962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a1001d920 msgr2=0x564a10028870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:31.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.102+0000 7f2772a00700 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a1001d920 0x564a10028870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.102+0000 7f2772a00700 1 -- 172.21.15.6:0/2922481962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a100290f0 msgr2=0x564a100260b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:31.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.102+0000 7f2772a00700 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a100290f0 0x564a100260b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.102+0000 7f2772a00700 1 -- 172.21.15.6:0/2922481962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a0febe9d0 con 0x564a1001e240 2024-04-26T01:55:31.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.102+0000 7f2773e00700 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a100290f0 0x564a100260b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:31.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.103+0000 7f2772a00700 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001e240 0x564a10025b70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2760015000 tx=0x7f2760005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2be4fd8e5df1e704 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:31.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.103+0000 7f275d600700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:55:31.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.103+0000 7f275d600700 1 -- 172.21.15.6:0/2922481962 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2760029060 con 0x564a1001e240 2024-04-26T01:55:31.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.103+0000 7f275d600700 1 -- 172.21.15.6:0/2922481962 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f2760003a90 con 0x564a1001e240 2024-04-26T01:55:31.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.103+0000 7f2784afab80 1 -- 172.21.15.6:0/2922481962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27640a5520 con 0x564a1001e240 2024-04-26T01:55:31.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.103+0000 7f275d600700 1 -- 172.21.15.6:0/2922481962 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f2760003c90 con 0x564a1001e240 2024-04-26T01:55:31.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.104+0000 7f2784afab80 10 client.26327 fetch_fsmap learned FSMap version 43 2024-04-26T01:55:31.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.104+0000 7f2784afab80 1 -- 172.21.15.6:0/2922481962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x564a0ff07bf0 con 0x564a1001e240 2024-04-26T01:55:31.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.104+0000 7f275d600700 1 -- 172.21.15.6:0/2922481962 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27640a5520 con 0x564a1001e240 2024-04-26T01:55:31.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.104+0000 7f275d600700 1 -- 172.21.15.6:0/2922481962 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f276002c400 con 0x564a1001e240 2024-04-26T01:55:31.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.104+0000 7f2784afab80 10 client.26327 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:55:31.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.104+0000 7f2784afab80 10 client.26327 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:55:31.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.104+0000 7f2784afab80 20 client.26327 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:31.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.104+0000 7f2784afab80 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a10038980 0x564a1003ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.104+0000 7f2784afab80 4 client.26327 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:31.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.104+0000 7f2784afab80 1 -- 172.21.15.6:0/2922481962 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a0ff7e780 con 0x564a10038980 2024-04-26T01:55:31.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.105+0000 7f2773400700 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a10038980 0x564a1003ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.105+0000 7f2773400700 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a10038980 0x564a1003ad70 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=547c33907a4b3687 server_cookie=f45d22aa4bad1194 in_seq=0 out_seq=0 2024-04-26T01:55:31.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.105+0000 7f275d600700 10 client.26327 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:31.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.106+0000 7f275d600700 1 -- 172.21.15.6:0/2922481962 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a0ff7e780 con 0x564a10038980 2024-04-26T01:55:31.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.106+0000 7f275d600700 10 client.26327 handle_command_reply: tid=0 2024-04-26T01:55:31.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.106+0000 7f2784afab80 1 client.26327 shutdown 2024-04-26T01:55:31.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.106+0000 7f2784afab80 1 -- 172.21.15.6:0/2922481962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001e240 msgr2=0x564a10025b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:31.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.107+0000 7f2784afab80 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001e240 0x564a10025b70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2760015000 tx=0x7f2760005270 comp rx=0 tx=0).stop 2024-04-26T01:55:31.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.107+0000 7f2784afab80 1 -- 172.21.15.6:0/2922481962 shutdown_connections 2024-04-26T01:55:31.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.107+0000 7f2784afab80 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a100290f0 0x564a100260b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.107+0000 7f2784afab80 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a10038980 0x564a1003ad70 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.107+0000 7f2784afab80 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a1001d920 0x564a10028870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.107+0000 7f2784afab80 1 --2- 172.21.15.6:0/2922481962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001e240 0x564a10025b70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.107+0000 7f275d600700 0 client.26327 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:31.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.107+0000 7f2784afab80 1 -- 172.21.15.6:0/2922481962 >> 172.21.15.6:0/2922481962 conn(0x564a0ff4c6a0 msgr2=0x564a100199c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:31.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.107+0000 7f2784afab80 1 -- 172.21.15.6:0/2922481962 shutdown_connections 2024-04-26T01:55:31.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.107+0000 7f2784afab80 1 -- 172.21.15.6:0/2922481962 wait complete. 2024-04-26T01:55:31.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.107+0000 7f2784afab80 20 client.26327 trim_cache size 0 max 16384 2024-04-26T01:55:31.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.129+0000 7f2784afab80 1 Processor -- start 2024-04-26T01:55:31.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.129+0000 7f2784afab80 1 -- start start 2024-04-26T01:55:31.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.129+0000 7f2784afab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001d920 0x564a10040620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.129+0000 7f2784afab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001e240 0x564a10040b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.129+0000 7f2784afab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a100290f0 0x564a10031010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.129+0000 7f2784afab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f27741b1ed0 con 0x564a1001e240 2024-04-26T01:55:31.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.129+0000 7f2784afab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f27741af760 con 0x564a1001d920 2024-04-26T01:55:31.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.129+0000 7f2784afab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f27741ad5d0 con 0x564a100290f0 2024-04-26T01:55:31.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.130+0000 7f2773e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a100290f0 0x564a10031010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.130+0000 7f2772a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001e240 0x564a10040b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.130+0000 7f2773400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001d920 0x564a10040620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.130+0000 7f2773e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a100290f0 0x564a10031010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:49382/0 (socket says 172.21.15.6:49382) 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.130+0000 7f2773e00700 1 -- 172.21.15.6:0/2040387846 learned_addr learned my addr 172.21.15.6:0/2040387846 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.130+0000 7f2773400700 1 -- 172.21.15.6:0/2040387846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a100290f0 msgr2=0x564a10031010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.130+0000 7f2773400700 1 --2- 172.21.15.6:0/2040387846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a100290f0 0x564a10031010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.130+0000 7f2773400700 1 -- 172.21.15.6:0/2040387846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001e240 msgr2=0x564a10040b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.130+0000 7f2773400700 1 --2- 172.21.15.6:0/2040387846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001e240 0x564a10040b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.130+0000 7f2773400700 1 -- 172.21.15.6:0/2040387846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a1003b3d0 con 0x564a1001d920 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.131+0000 7f2773400700 1 --2- 172.21.15.6:0/2040387846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001d920 0x564a10040620 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f276400c2d0 tx=0x7f2764098c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c7b86f951c805ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.131+0000 7f275fe00700 1 -- 172.21.15.6:0/2040387846 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276409abb0 con 0x564a1001d920 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.131+0000 7f275fe00700 1 -- 172.21.15.6:0/2040387846 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27640ce2e0 con 0x564a1001d920 2024-04-26T01:55:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.131+0000 7f275fe00700 1 -- 172.21.15.6:0/2040387846 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27640a5090 con 0x564a1001d920 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.132+0000 7f2784afab80 1 -- 172.21.15.6:0/2040387846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001d920 msgr2=0x564a10040620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.132+0000 7f2784afab80 1 --2- 172.21.15.6:0/2040387846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001d920 0x564a10040620 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f276400c2d0 tx=0x7f2764098c80 comp rx=0 tx=0).stop 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.132+0000 7f2784afab80 1 -- 172.21.15.6:0/2040387846 shutdown_connections 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.132+0000 7f2784afab80 1 --2- 172.21.15.6:0/2040387846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a100290f0 0x564a10031010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.132+0000 7f2784afab80 1 --2- 172.21.15.6:0/2040387846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001d920 0x564a10040620 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.132+0000 7f2784afab80 1 --2- 172.21.15.6:0/2040387846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001e240 0x564a10040b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.132+0000 7f2784afab80 1 -- 172.21.15.6:0/2040387846 >> 172.21.15.6:0/2040387846 conn(0x564a0ff4c6a0 msgr2=0x564a10027c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.132+0000 7f2784afab80 1 -- 172.21.15.6:0/2040387846 shutdown_connections 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.132+0000 7f2784afab80 1 -- 172.21.15.6:0/2040387846 wait complete. 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.132+0000 7f2784afab80 1 -- start start 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.132+0000 7f2784afab80 1 Processor -- start 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.133+0000 7f2784afab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 0x564a10031590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.133+0000 7f2784afab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001e240 0x564a1002d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.133+0000 7f2784afab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a100290f0 0x564a1002e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.133+0000 7f2784afab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f27741adc20 con 0x564a1001d920 2024-04-26T01:55:31.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.133+0000 7f2784afab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f27741af090 con 0x564a1001e240 2024-04-26T01:55:31.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.133+0000 7f2784afab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f27741ae2f0 con 0x564a100290f0 2024-04-26T01:55:31.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.133+0000 7f2784afab80 10 client.0 _ll_register_callbacks cb 0x564a0ff09820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:31.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.133+0000 7f2773400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 0x564a10031590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.133+0000 7f2773400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 0x564a10031590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60018/0 (socket says 172.21.15.6:60018) 2024-04-26T01:55:31.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.133+0000 7f2773400700 1 -- 172.21.15.6:0/2723973903 learned_addr learned my addr 172.21.15.6:0/2723973903 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:31.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.133+0000 7f2773e00700 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a100290f0 0x564a1002e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.133+0000 7f2772a00700 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001e240 0x564a1002d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.134+0000 7f2773400700 1 -- 172.21.15.6:0/2723973903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a100290f0 msgr2=0x564a1002e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:31.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.134+0000 7f2773400700 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a100290f0 0x564a1002e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.134+0000 7f2773400700 1 -- 172.21.15.6:0/2723973903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001e240 msgr2=0x564a1002d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:31.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.134+0000 7f2773400700 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001e240 0x564a1002d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.134+0000 7f2773400700 1 -- 172.21.15.6:0/2723973903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a1009eea0 con 0x564a1001d920 2024-04-26T01:55:31.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.134+0000 7f2773400700 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 0x564a10031590 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f27640c2c70 tx=0x7f276409d0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee17bd7ff8d90d5c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:31.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.134+0000 7f275d600700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.134+0000 7f275d600700 1 -- 172.21.15.6:0/2723973903 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276409abb0 con 0x564a1001d920 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.134+0000 7f275d600700 1 -- 172.21.15.6:0/2723973903 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f276409cbb0 con 0x564a1001d920 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.134+0000 7f2784afab80 1 -- 172.21.15.6:0/2723973903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27640ce2e0 con 0x564a1001d920 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.135+0000 7f275d600700 1 -- 172.21.15.6:0/2723973903 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f276409dbe0 con 0x564a1001d920 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.135+0000 7f2784afab80 10 client.16191 fetch_fsmap learned FSMap version 43 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.135+0000 7f2784afab80 1 -- 172.21.15.6:0/2723973903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x564a100a00d0 con 0x564a1001d920 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.135+0000 7f275d600700 1 -- 172.21.15.6:0/2723973903 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27640ce2e0 con 0x564a1001d920 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.135+0000 7f275d600700 1 -- 172.21.15.6:0/2723973903 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f2764010720 con 0x564a1001d920 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.135+0000 7f2784afab80 10 client.16191 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.135+0000 7f2784afab80 10 client.16191 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.135+0000 7f2784afab80 20 client.16191 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.135+0000 7f2784afab80 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a100a02a0 0x564a100a2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.135+0000 7f2784afab80 4 client.16191 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "8a864be1", "release": true, "await": true}] 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.135+0000 7f2784afab80 1 -- 172.21.15.6:0/2723973903 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "8a864be1", "release": true, "await": true}) -- 0x564a0ff78820 con 0x564a100a02a0 2024-04-26T01:55:31.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.136+0000 7f2772a00700 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a100a02a0 0x564a100a2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:31.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.136+0000 7f2772a00700 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a100a02a0 0x564a100a2690 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=592e29f1e6d05807 server_cookie=3ab52856ac02edab in_seq=0 out_seq=0 2024-04-26T01:55:31.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.136+0000 7f275d600700 10 client.16191 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:31.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f275d600700 1 -- 172.21.15.6:0/2723973903 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x564a0ff78820 con 0x564a100a02a0 2024-04-26T01:55:31.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f275d600700 10 client.16191 handle_command_reply: tid=0 2024-04-26T01:55:31.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f2784afab80 1 client.16191 shutdown 2024-04-26T01:55:31.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f2784afab80 1 -- 172.21.15.6:0/2723973903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 msgr2=0x564a10031590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:31.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f2784afab80 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 0x564a10031590 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f27640c2c70 tx=0x7f276409d0a0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f2784afab80 1 -- 172.21.15.6:0/2723973903 shutdown_connections 2024-04-26T01:55:31.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f2784afab80 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a100a02a0 0x564a100a2690 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f2784afab80 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a100290f0 0x564a1002e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f2784afab80 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a1001d920 0x564a10031590 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f2784afab80 1 --2- 172.21.15.6:0/2723973903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1001e240 0x564a1002d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f2784afab80 1 -- 172.21.15.6:0/2723973903 >> 172.21.15.6:0/2723973903 conn(0x564a0ff4c6a0 msgr2=0x564a10018f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:31.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f275d600700 0 client.16191 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:31.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f2784afab80 1 -- 172.21.15.6:0/2723973903 shutdown_connections 2024-04-26T01:55:31.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f2784afab80 1 -- 172.21.15.6:0/2723973903 wait complete. 2024-04-26T01:55:31.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.142+0000 7f2784afab80 20 client.16191 trim_cache size 0 max 16384 2024-04-26T01:55:31.142 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 43, 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 28, 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "8a864be1": { 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 28, 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 82.1, 2024-04-26T01:55:31.143 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:55:31.144 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:55:31.144 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:55:31.144 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:55:31.144 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:55:31.144 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:55:31.144 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:31.144 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:31.144 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:31.144 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:31.144 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:31.144 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:55:31.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.143+0000 7f277a200700 1 -- 172.21.15.6:0/2763196242 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27580a3db0 msgr2=0x7f27580a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:31.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.143+0000 7f277a200700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27580a3db0 0x7f27580a6270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f276c009ec0 tx=0x7f276c0092f0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.143+0000 7f277a200700 1 -- 172.21.15.6:0/2763196242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f277406afe0 msgr2=0x7f277419b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:31.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.143+0000 7f277a200700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f277406afe0 0x7f277419b370 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f276400e9a0 tx=0x7f276400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.143+0000 7f277a200700 1 -- 172.21.15.6:0/2763196242 shutdown_connections 2024-04-26T01:55:31.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.143+0000 7f277a200700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27580a3db0 0x7f27580a6270 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.143+0000 7f277a200700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2774108a20 0x7f277419fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.143+0000 7f277a200700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f277406afe0 0x7f277419b370 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.143+0000 7f277a200700 1 --2- 172.21.15.6:0/2763196242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f277406b980 0x7f277419b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:31.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.143+0000 7f277a200700 1 -- 172.21.15.6:0/2763196242 >> 172.21.15.6:0/2763196242 conn(0x7f27740788b0 msgr2=0x7f277410df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:31.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.144+0000 7f277a200700 1 -- 172.21.15.6:0/2763196242 shutdown_connections 2024-04-26T01:55:31.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:31.144+0000 7f277a200700 1 -- 172.21.15.6:0/2763196242 wait complete. 2024-04-26T01:55:31.613 INFO:tasks.workunit.client.0.smithi006.stdout: 1 330608 3.61 MB/sec execute 355 sec latency 29668.595 ms 2024-04-26T01:55:32.024 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '8a864be1', seconds elapsed: 37.7 2024-04-26T01:55:32.025 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.1 2024-04-26T01:55:32.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:55:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:55:32.613 INFO:tasks.workunit.client.0.smithi006.stdout: 1 331757 3.61 MB/sec execute 356 sec latency 21.235 ms 2024-04-26T01:55:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:32 smithi006 ceph-mon[28419]: pgmap v303: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1.3 MiB/s wr, 60 op/s 2024-04-26T01:55:32.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:32 smithi188 ceph-mon[32751]: pgmap v303: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1.3 MiB/s wr, 60 op/s 2024-04-26T01:55:32.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:32 smithi029 ceph-mon[32775]: pgmap v303: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1.3 MiB/s wr, 60 op/s 2024-04-26T01:55:33.613 INFO:tasks.workunit.client.0.smithi006.stdout: 1 333448 3.64 MB/sec execute 357 sec latency 21.386 ms 2024-04-26T01:55:34.614 INFO:tasks.workunit.client.0.smithi006.stdout: 1 334112 3.63 MB/sec execute 358 sec latency 21.922 ms 2024-04-26T01:55:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.671+0000 7ff741600700 1 -- 172.21.15.6:0/1599360621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73c10e3b0 msgr2=0x7ff73c115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.671+0000 7ff741600700 1 --2- 172.21.15.6:0/1599360621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73c10e3b0 0x7ff73c115f80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff73400b0d0 tx=0x7ff73400b3e0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.671+0000 7ff741600700 1 -- 172.21.15.6:0/1599360621 shutdown_connections 2024-04-26T01:55:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.671+0000 7ff741600700 1 --2- 172.21.15.6:0/1599360621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73c10e3b0 0x7ff73c115f80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.671+0000 7ff741600700 1 --2- 172.21.15.6:0/1599360621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c1092d0 0x7ff73c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.671+0000 7ff741600700 1 --2- 172.21.15.6:0/1599360621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73c109c80 0x7ff73c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.671+0000 7ff741600700 1 -- 172.21.15.6:0/1599360621 >> 172.21.15.6:0/1599360621 conn(0x7ff73c076210 msgr2=0x7ff73c107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.671+0000 7ff741600700 1 -- 172.21.15.6:0/1599360621 shutdown_connections 2024-04-26T01:55:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.671+0000 7ff741600700 1 -- 172.21.15.6:0/1599360621 wait complete. 2024-04-26T01:55:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.673+0000 7ff741600700 1 Processor -- start 2024-04-26T01:55:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.673+0000 7ff741600700 1 -- start start 2024-04-26T01:55:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.673+0000 7ff741600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73c1092d0 0x7ff73c19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.673+0000 7ff741600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73c109c80 0x7ff73c19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.673+0000 7ff741600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c10e3b0 0x7ff73c1a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.673+0000 7ff741600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c1192a0 con 0x7ff73c1092d0 2024-04-26T01:55:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.673+0000 7ff741600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c119120 con 0x7ff73c10e3b0 2024-04-26T01:55:34.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.673+0000 7ff741600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c119420 con 0x7ff73c109c80 2024-04-26T01:55:34.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73b400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c10e3b0 0x7ff73c1a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73a000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73c109c80 0x7ff73c19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73aa00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73c1092d0 0x7ff73c19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73a000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73c109c80 0x7ff73c19fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:49408/0 (socket says 172.21.15.6:49408) 2024-04-26T01:55:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73b400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c10e3b0 0x7ff73c1a3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52228/0 (socket says 172.21.15.6:52228) 2024-04-26T01:55:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73a000700 1 -- 172.21.15.6:0/3872173921 learned_addr learned my addr 172.21.15.6:0/3872173921 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73a000700 1 -- 172.21.15.6:0/3872173921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c10e3b0 msgr2=0x7ff73c1a3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73a000700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c10e3b0 0x7ff73c1a3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73a000700 1 -- 172.21.15.6:0/3872173921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73c1092d0 msgr2=0x7ff73c19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73a000700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73c1092d0 0x7ff73c19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73a000700 1 -- 172.21.15.6:0/3872173921 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73c1a4700 con 0x7ff73c109c80 2024-04-26T01:55:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73b400700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c10e3b0 0x7ff73c1a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.674+0000 7ff73aa00700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73c1092d0 0x7ff73c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.675+0000 7ff73a000700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73c109c80 0x7ff73c19fc30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff72800ea30 tx=0x7ff72800ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.675+0000 7ff733400700 1 -- 172.21.15.6:0/3872173921 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72800cbd0 con 0x7ff73c109c80 2024-04-26T01:55:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.675+0000 7ff733400700 1 -- 172.21.15.6:0/3872173921 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7280044d0 con 0x7ff73c109c80 2024-04-26T01:55:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.675+0000 7ff741600700 1 -- 172.21.15.6:0/3872173921 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff73c1a49f0 con 0x7ff73c109c80 2024-04-26T01:55:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.675+0000 7ff733400700 1 -- 172.21.15.6:0/3872173921 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff728003dc0 con 0x7ff73c109c80 2024-04-26T01:55:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.675+0000 7ff741600700 1 -- 172.21.15.6:0/3872173921 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff73c1ac270 con 0x7ff73c109c80 2024-04-26T01:55:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.676+0000 7ff741600700 1 -- 172.21.15.6:0/3872173921 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff73c069730 con 0x7ff73c109c80 2024-04-26T01:55:34.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.678+0000 7ff733400700 1 -- 172.21.15.6:0/3872173921 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7ff728003f80 con 0x7ff73c109c80 2024-04-26T01:55:34.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.680+0000 7ff733400700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff7200a3db0 0x7ff7200a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.680+0000 7ff733400700 1 -- 172.21.15.6:0/3872173921 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7ff7280d0900 con 0x7ff73c109c80 2024-04-26T01:55:34.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.680+0000 7ff73aa00700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff7200a3db0 0x7ff7200a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.681+0000 7ff73aa00700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff7200a3db0 0x7ff7200a6270 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff72c011490 tx=0x7ff72c009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:34.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.682+0000 7ff733400700 1 -- 172.21.15.6:0/3872173921 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff72809cef0 con 0x7ff73c109c80 2024-04-26T01:55:34.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:34 smithi006 ceph-mon[28419]: pgmap v304: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 785 KiB/s wr, 42 op/s 2024-04-26T01:55:34.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:34 smithi188 ceph-mon[32751]: pgmap v304: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 785 KiB/s wr, 42 op/s 2024-04-26T01:55:34.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:34 smithi029 ceph-mon[32775]: pgmap v304: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 785 KiB/s wr, 42 op/s 2024-04-26T01:55:34.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.988+0000 7ff741600700 1 -- 172.21.15.6:0/3872173921 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff73c02df80 con 0x7ff73c109c80 2024-04-26T01:55:34.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.989+0000 7ff733400700 1 -- 172.21.15.6:0/3872173921 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11850 (secure 0 0 0) 0x7ff72809c7f0 con 0x7ff73c109c80 2024-04-26T01:55:34.988 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:55:34.988 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":43,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:55:18.912671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:55:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.992+0000 7ff741600700 1 -- 172.21.15.6:0/3872173921 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff7200a3db0 msgr2=0x7ff7200a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.992+0000 7ff741600700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff7200a3db0 0x7ff7200a6270 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff72c011490 tx=0x7ff72c009260 comp rx=0 tx=0).stop 2024-04-26T01:55:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.992+0000 7ff741600700 1 -- 172.21.15.6:0/3872173921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73c109c80 msgr2=0x7ff73c19fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.992+0000 7ff741600700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73c109c80 0x7ff73c19fc30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff72800ea30 tx=0x7ff72800ed40 comp rx=0 tx=0).stop 2024-04-26T01:55:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.993+0000 7ff741600700 1 -- 172.21.15.6:0/3872173921 shutdown_connections 2024-04-26T01:55:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.993+0000 7ff741600700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff7200a3db0 0x7ff7200a6270 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.993+0000 7ff741600700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c10e3b0 0x7ff73c1a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.993+0000 7ff741600700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73c1092d0 0x7ff73c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.993+0000 7ff741600700 1 --2- 172.21.15.6:0/3872173921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff73c109c80 0x7ff73c19fc30 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.993+0000 7ff741600700 1 -- 172.21.15.6:0/3872173921 >> 172.21.15.6:0/3872173921 conn(0x7ff73c076210 msgr2=0x7ff73c076750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.993+0000 7ff741600700 1 -- 172.21.15.6:0/3872173921 shutdown_connections 2024-04-26T01:55:34.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:34.993+0000 7ff741600700 1 -- 172.21.15.6:0/3872173921 wait complete. 2024-04-26T01:55:34.993 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-26T01:55:35.614 INFO:tasks.workunit.client.0.smithi006.stdout: 1 335239 3.64 MB/sec execute 359 sec latency 66.017 ms 2024-04-26T01:55:35.882 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:35 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3872173921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:35.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:35 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3872173921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:35.941 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 82.1 --await-for 42.0' 2024-04-26T01:55:35.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 82.1 --await-for 42.0 2024-04-26T01:55:35.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:35 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3872173921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:36.432 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:55:36.614 INFO:tasks.workunit.client.0.smithi006.stdout: 1 336956 3.66 MB/sec execute 360 sec latency 16.626 ms 2024-04-26T01:55:36.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:36 smithi188 ceph-mon[32751]: pgmap v305: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 785 KiB/s wr, 42 op/s 2024-04-26T01:55:36.922 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:36 smithi006 ceph-mon[28419]: pgmap v305: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 785 KiB/s wr, 42 op/s 2024-04-26T01:55:36.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:36 smithi029 ceph-mon[32775]: pgmap v305: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 785 KiB/s wr, 42 op/s 2024-04-26T01:55:37.614 INFO:tasks.workunit.client.0.smithi006.stdout: 1 337562 3.65 MB/sec execute 361 sec latency 38.384 ms 2024-04-26T01:55:38.617 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338539 3.66 MB/sec execute 362 sec latency 25.177 ms 2024-04-26T01:55:38.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.737+0000 7f0ef9400700 1 -- 172.21.15.6:0/1312789690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ef4075720 msgr2=0x7f0ef4113380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:38.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.737+0000 7f0ef9400700 1 --2- 172.21.15.6:0/1312789690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ef4075720 0x7f0ef4113380 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0edc0099c0 tx=0x7f0edc009cd0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.738+0000 7f0ef9400700 1 -- 172.21.15.6:0/1312789690 shutdown_connections 2024-04-26T01:55:38.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.738+0000 7f0ef9400700 1 --2- 172.21.15.6:0/1312789690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ef41138c0 0x7f0ef4115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.738+0000 7f0ef9400700 1 --2- 172.21.15.6:0/1312789690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4074e00 0x7f0ef40751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.738+0000 7f0ef9400700 1 --2- 172.21.15.6:0/1312789690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ef4075720 0x7f0ef4113380 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.738+0000 7f0ef9400700 1 -- 172.21.15.6:0/1312789690 >> 172.21.15.6:0/1312789690 conn(0x7f0ef40fe3a0 msgr2=0x7f0ef41007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:38.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.738+0000 7f0ef9400700 1 -- 172.21.15.6:0/1312789690 shutdown_connections 2024-04-26T01:55:38.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.738+0000 7f0ef9400700 1 -- 172.21.15.6:0/1312789690 wait complete. 2024-04-26T01:55:38.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.739+0000 7f0ef9400700 1 Processor -- start 2024-04-26T01:55:38.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.739+0000 7f0ef9400700 1 -- start start 2024-04-26T01:55:38.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.740+0000 7f0ef9400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ef4074e00 0x7f0ef419f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.740+0000 7f0ef9400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4075720 0x7f0ef419fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.740+0000 7f0ef9400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ef41138c0 0x7f0ef41a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.740+0000 7f0ef9400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41191d0 con 0x7f0ef4074e00 2024-04-26T01:55:38.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.740+0000 7f0ef9400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4119050 con 0x7f0ef4075720 2024-04-26T01:55:38.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.740+0000 7f0ef9400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4119350 con 0x7f0ef41138c0 2024-04-26T01:55:38.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.740+0000 7f0ef3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ef4074e00 0x7f0ef419f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.740+0000 7f0ef8a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ef41138c0 0x7f0ef41a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.740+0000 7f0ef3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ef4074e00 0x7f0ef419f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34618/0 (socket says 172.21.15.6:34618) 2024-04-26T01:55:38.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.740+0000 7f0ef3e00700 1 -- 172.21.15.6:0/4050841132 learned_addr learned my addr 172.21.15.6:0/4050841132 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:38.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.741+0000 7f0eeb400700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4075720 0x7f0ef419fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.741+0000 7f0ef8a00700 1 -- 172.21.15.6:0/4050841132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4075720 msgr2=0x7f0ef419fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:38.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.741+0000 7f0ef8a00700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4075720 0x7f0ef419fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.741+0000 7f0ef8a00700 1 -- 172.21.15.6:0/4050841132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ef4074e00 msgr2=0x7f0ef419f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:38.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.741+0000 7f0ef8a00700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ef4074e00 0x7f0ef419f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.741+0000 7f0ef8a00700 1 -- 172.21.15.6:0/4050841132 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef41a45f0 con 0x7f0ef41138c0 2024-04-26T01:55:38.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.741+0000 7f0ef8a00700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ef41138c0 0x7f0ef41a3eb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00d750 tx=0x7f0eec00da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:38.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.741+0000 7f0ef1600700 1 -- 172.21.15.6:0/4050841132 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec014030 con 0x7f0ef41138c0 2024-04-26T01:55:38.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.741+0000 7f0ef1600700 1 -- 172.21.15.6:0/4050841132 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eec0044d0 con 0x7f0ef41138c0 2024-04-26T01:55:38.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.741+0000 7f0ef9400700 1 -- 172.21.15.6:0/4050841132 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef41a48e0 con 0x7f0ef41138c0 2024-04-26T01:55:38.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.741+0000 7f0ef1600700 1 -- 172.21.15.6:0/4050841132 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec005420 con 0x7f0ef41138c0 2024-04-26T01:55:38.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.742+0000 7f0ef9400700 1 -- 172.21.15.6:0/4050841132 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef41ac160 con 0x7f0ef41138c0 2024-04-26T01:55:38.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.745+0000 7f0ef1600700 1 -- 172.21.15.6:0/4050841132 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f0eec004b50 con 0x7f0ef41138c0 2024-04-26T01:55:38.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.746+0000 7f0f05422b80 1 Processor -- start 2024-04-26T01:55:38.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.747+0000 7f0f05422b80 1 -- start start 2024-04-26T01:55:38.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.747+0000 7f0ef1600700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0ed80a3e50 0x7f0ed80a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.747+0000 7f0f05422b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55733947fb80 0x55733947ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.747+0000 7f0ef1600700 1 -- 172.21.15.6:0/4050841132 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f0eec0cffe0 con 0x7f0ef41138c0 2024-04-26T01:55:38.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.747+0000 7f0f05422b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573394804a0 0x55733948ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.747+0000 7f0f05422b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733948b350 0x55733948d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.747+0000 7f0f05422b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5573393d3020 con 0x55733947fb80 2024-04-26T01:55:38.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.747+0000 7f0f05422b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5573393798c0 con 0x55733948b350 2024-04-26T01:55:38.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.747+0000 7f0f05422b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5573393cefe0 con 0x5573394804a0 2024-04-26T01:55:38.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.747+0000 7f0eeb400700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0ed80a3e50 0x7f0ed80a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.748+0000 7f0ef3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55733947fb80 0x55733947ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.748+0000 7f0ef8a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733948b350 0x55733948d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.748+0000 7f0ef8a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733948b350 0x55733948d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56572/0 (socket says 172.21.15.6:56572) 2024-04-26T01:55:38.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.748+0000 7f0ef3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55733947fb80 0x55733947ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34624/0 (socket says 172.21.15.6:34624) 2024-04-26T01:55:38.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.748+0000 7f0ef8a00700 1 -- 172.21.15.6:0/22863039 learned_addr learned my addr 172.21.15.6:0/22863039 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:38.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.748+0000 7f0ef8a00700 1 -- 172.21.15.6:0/22863039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573394804a0 msgr2=0x55733948ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:38.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.748+0000 7f0ef8a00700 1 --2- 172.21.15.6:0/22863039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573394804a0 0x55733948ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.748+0000 7f0ef8a00700 1 -- 172.21.15.6:0/22863039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55733947fb80 msgr2=0x55733947ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:38.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.748+0000 7f0ef8a00700 1 --2- 172.21.15.6:0/22863039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55733947fb80 0x55733947ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.748+0000 7f0ef8a00700 1 -- 172.21.15.6:0/22863039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55733936f810 con 0x55733948b350 2024-04-26T01:55:38.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.748+0000 7f0ef8a00700 1 --2- 172.21.15.6:0/22863039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733948b350 0x55733948d740 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0a4670 tx=0x7f0eec09bba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=799c5c362a64503e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:38.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.749+0000 7f0eeb400700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0ed80a3e50 0x7f0ed80a6310 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4106670 tx=0x7f0edc009f80 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:38.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.749+0000 7f0ee9600700 1 -- 172.21.15.6:0/22863039 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec0b5070 con 0x55733948b350 2024-04-26T01:55:38.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.749+0000 7f0ee9600700 1 -- 172.21.15.6:0/22863039 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eec0a4ba0 con 0x55733948b350 2024-04-26T01:55:38.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.749+0000 7f0ee9600700 1 -- 172.21.15.6:0/22863039 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec01d040 con 0x55733948b350 2024-04-26T01:55:38.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.750+0000 7f0f05422b80 1 -- 172.21.15.6:0/22863039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733948b350 msgr2=0x55733948d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:38.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.750+0000 7f0f05422b80 1 --2- 172.21.15.6:0/22863039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733948b350 0x55733948d740 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0a4670 tx=0x7f0eec09bba0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.750+0000 7f0f05422b80 1 -- 172.21.15.6:0/22863039 shutdown_connections 2024-04-26T01:55:38.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.750+0000 7f0f05422b80 1 --2- 172.21.15.6:0/22863039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733948b350 0x55733948d740 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.750+0000 7f0f05422b80 1 --2- 172.21.15.6:0/22863039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55733947fb80 0x55733947ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.750+0000 7f0f05422b80 1 --2- 172.21.15.6:0/22863039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573394804a0 0x55733948ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.750+0000 7f0f05422b80 1 -- 172.21.15.6:0/22863039 >> 172.21.15.6:0/22863039 conn(0x5573393ae730 msgr2=0x55733947bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:38.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.750+0000 7f0f05422b80 1 -- 172.21.15.6:0/22863039 shutdown_connections 2024-04-26T01:55:38.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.750+0000 7f0f05422b80 1 -- 172.21.15.6:0/22863039 wait complete. 2024-04-26T01:55:38.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.751+0000 7f0f05422b80 1 -- start start 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.751+0000 7f0f05422b80 1 Processor -- start 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.751+0000 7f0f05422b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x55733948aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.751+0000 7f0f05422b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573394804a0 0x557339487df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.751+0000 7f0f05422b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55733948b350 0x557339488330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.751+0000 7f0f05422b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55733948ded0 con 0x5573394804a0 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.751+0000 7f0f05422b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55733938c9f0 con 0x55733947fb80 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.751+0000 7f0f05422b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55733948e050 con 0x55733948b350 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.752+0000 7f0f05422b80 10 client.0 _ll_register_callbacks cb 0x55733936b8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.752+0000 7f0ef3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x55733948aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.752+0000 7f0ef8a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55733948b350 0x557339488330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.752+0000 7f0ef3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x55733948aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56584/0 (socket says 172.21.15.6:56584) 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.752+0000 7f0ef3e00700 1 -- 172.21.15.6:0/3089674855 learned_addr learned my addr 172.21.15.6:0/3089674855 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.752+0000 7f0eeb400700 1 --2- 172.21.15.6:0/3089674855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573394804a0 0x557339487df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.752+0000 7f0ef3e00700 1 -- 172.21.15.6:0/3089674855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55733948b350 msgr2=0x557339488330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.752+0000 7f0ef3e00700 1 --2- 172.21.15.6:0/3089674855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55733948b350 0x557339488330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.752+0000 7f0ef3e00700 1 -- 172.21.15.6:0/3089674855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573394804a0 msgr2=0x557339487df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.752+0000 7f0ef3e00700 1 --2- 172.21.15.6:0/3089674855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573394804a0 0x557339487df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.752+0000 7f0ef3e00700 1 -- 172.21.15.6:0/3089674855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557339320a60 con 0x55733947fb80 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.753+0000 7f0ef3e00700 1 --2- 172.21.15.6:0/3089674855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x55733948aa80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4007bc0 tx=0x7f0ee4007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c0d92604dd4724b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.753+0000 7f0ee2000700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.753+0000 7f0ee2000700 1 -- 172.21.15.6:0/3089674855 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4014cc0 con 0x55733947fb80 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.753+0000 7f0f05422b80 1 -- 172.21.15.6:0/3089674855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0eec0a4ba0 con 0x55733947fb80 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.753+0000 7f0ee2000700 1 -- 172.21.15.6:0/3089674855 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f0ee4020410 con 0x55733947fb80 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.753+0000 7f0ee2000700 1 -- 172.21.15.6:0/3089674855 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0ee4014e60 con 0x55733947fb80 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.753+0000 7f0f05422b80 10 client.26351 fetch_fsmap learned FSMap version 43 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.753+0000 7f0f05422b80 1 -- 172.21.15.6:0/3089674855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x557339369c80 con 0x55733947fb80 2024-04-26T01:55:38.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.753+0000 7f0ee2000700 1 -- 172.21.15.6:0/3089674855 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eec0a4ba0 con 0x55733947fb80 2024-04-26T01:55:38.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.754+0000 7f0ee2000700 1 -- 172.21.15.6:0/3089674855 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f0ee40287a0 con 0x55733947fb80 2024-04-26T01:55:38.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.754+0000 7f0f05422b80 10 client.26351 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:55:38.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.754+0000 7f0f05422b80 10 client.26351 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:55:38.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.754+0000 7f0f05422b80 20 client.26351 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:38.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.754+0000 7f0f05422b80 1 --2- 172.21.15.6:0/3089674855 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55733949ac90 0x55733949d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.754+0000 7f0f05422b80 4 client.26351 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:38.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.754+0000 7f0f05422b80 1 -- 172.21.15.6:0/3089674855 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573393e0810 con 0x55733949ac90 2024-04-26T01:55:38.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.754+0000 7f0eeb400700 1 --2- 172.21.15.6:0/3089674855 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55733949ac90 0x55733949d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.755+0000 7f0eeb400700 1 --2- 172.21.15.6:0/3089674855 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55733949ac90 0x55733949d0c0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3097777c8dedf4a1 server_cookie=7dee6f1d69ca0072 in_seq=0 out_seq=0 2024-04-26T01:55:38.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.755+0000 7f0ee2000700 10 client.26351 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:38.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0ee2000700 1 -- 172.21.15.6:0/3089674855 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5573393e0810 con 0x55733949ac90 2024-04-26T01:55:38.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0ee2000700 10 client.26351 handle_command_reply: tid=0 2024-04-26T01:55:38.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0f05422b80 1 client.26351 shutdown 2024-04-26T01:55:38.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0f05422b80 1 -- 172.21.15.6:0/3089674855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 msgr2=0x55733948aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:38.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0f05422b80 1 --2- 172.21.15.6:0/3089674855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x55733948aa80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4007bc0 tx=0x7f0ee4007ed0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0f05422b80 1 -- 172.21.15.6:0/3089674855 shutdown_connections 2024-04-26T01:55:38.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0f05422b80 1 --2- 172.21.15.6:0/3089674855 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55733949ac90 0x55733949d0c0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0f05422b80 1 --2- 172.21.15.6:0/3089674855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55733948b350 0x557339488330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0f05422b80 1 --2- 172.21.15.6:0/3089674855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x55733948aa80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0ee2000700 0 client.26351 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:38.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0f05422b80 1 --2- 172.21.15.6:0/3089674855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573394804a0 0x557339487df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0f05422b80 1 -- 172.21.15.6:0/3089674855 >> 172.21.15.6:0/3089674855 conn(0x5573393ae730 msgr2=0x557339481c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:38.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0f05422b80 1 -- 172.21.15.6:0/3089674855 shutdown_connections 2024-04-26T01:55:38.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.756+0000 7f0f05422b80 1 -- 172.21.15.6:0/3089674855 wait complete. 2024-04-26T01:55:38.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.757+0000 7f0f05422b80 20 client.26351 trim_cache size 0 max 16384 2024-04-26T01:55:38.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.781+0000 7f0f05422b80 1 Processor -- start 2024-04-26T01:55:38.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.781+0000 7f0f05422b80 1 -- start start 2024-04-26T01:55:38.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.781+0000 7f0f05422b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x557339496d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.781+0000 7f0f05422b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573394804a0 0x557339489eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.781+0000 7f0f05422b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55733948b350 0x5573394932a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.781+0000 7f0f05422b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b61a0 con 0x55733948b350 2024-04-26T01:55:38.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.781+0000 7f0f05422b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b3a30 con 0x55733947fb80 2024-04-26T01:55:38.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.781+0000 7f0f05422b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b18a0 con 0x5573394804a0 2024-04-26T01:55:38.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.781+0000 7f0ef3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x557339496d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.781+0000 7f0ef3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x557339496d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56592/0 (socket says 172.21.15.6:56592) 2024-04-26T01:55:38.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.781+0000 7f0ef3e00700 1 -- 172.21.15.6:0/2447605021 learned_addr learned my addr 172.21.15.6:0/2447605021 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:38.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.781+0000 7f0eeb400700 1 --2- 172.21.15.6:0/2447605021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573394804a0 0x557339489eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.782+0000 7f0ef8a00700 1 --2- 172.21.15.6:0/2447605021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55733948b350 0x5573394932a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.782+0000 7f0ef3e00700 1 -- 172.21.15.6:0/2447605021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573394804a0 msgr2=0x557339489eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.782+0000 7f0ef3e00700 1 --2- 172.21.15.6:0/2447605021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573394804a0 0x557339489eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.782+0000 7f0ef3e00700 1 -- 172.21.15.6:0/2447605021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55733948b350 msgr2=0x5573394932a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.782+0000 7f0ef3e00700 1 --2- 172.21.15.6:0/2447605021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55733948b350 0x5573394932a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.782+0000 7f0ef3e00700 1 -- 172.21.15.6:0/2447605021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573394877f0 con 0x55733947fb80 2024-04-26T01:55:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.782+0000 7f0ef3e00700 1 --2- 172.21.15.6:0/2447605021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x557339496d90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009550 tx=0x7f0ee4009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa98bd897926e26d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.783+0000 7f0ee8c00700 1 -- 172.21.15.6:0/2447605021 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4014ac0 con 0x55733947fb80 2024-04-26T01:55:38.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.783+0000 7f0ee8c00700 1 -- 172.21.15.6:0/2447605021 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ee4014c60 con 0x55733947fb80 2024-04-26T01:55:38.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.783+0000 7f0ee8c00700 1 -- 172.21.15.6:0/2447605021 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee40287c0 con 0x55733947fb80 2024-04-26T01:55:38.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.783+0000 7f0f05422b80 1 -- 172.21.15.6:0/2447605021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 msgr2=0x557339496d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:38.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.783+0000 7f0f05422b80 1 --2- 172.21.15.6:0/2447605021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x557339496d90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009550 tx=0x7f0ee4009ba0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.783+0000 7f0f05422b80 1 -- 172.21.15.6:0/2447605021 shutdown_connections 2024-04-26T01:55:38.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.783+0000 7f0f05422b80 1 --2- 172.21.15.6:0/2447605021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55733948b350 0x5573394932a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.783+0000 7f0f05422b80 1 --2- 172.21.15.6:0/2447605021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x557339496d90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.783+0000 7f0f05422b80 1 --2- 172.21.15.6:0/2447605021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573394804a0 0x557339489eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.783+0000 7f0f05422b80 1 -- 172.21.15.6:0/2447605021 >> 172.21.15.6:0/2447605021 conn(0x5573393ae730 msgr2=0x5573394752b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.784+0000 7f0f05422b80 1 -- 172.21.15.6:0/2447605021 shutdown_connections 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.784+0000 7f0f05422b80 1 -- 172.21.15.6:0/2447605021 wait complete. 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.784+0000 7f0f05422b80 1 -- start start 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.784+0000 7f0f05422b80 1 Processor -- start 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.785+0000 7f0f05422b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x5573394fb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.785+0000 7f0f05422b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573394804a0 0x5573394fbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.785+0000 7f0f05422b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55733948b350 0x5573394926a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.785+0000 7f0f05422b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b1ef0 con 0x5573394804a0 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.785+0000 7f0f05422b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b3360 con 0x55733947fb80 2024-04-26T01:55:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.785+0000 7f0f05422b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b25c0 con 0x55733948b350 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.785+0000 7f0f05422b80 10 client.0 _ll_register_callbacks cb 0x55733936b8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.785+0000 7f0ef3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x5573394fb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.785+0000 7f0ef3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x5573394fb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56594/0 (socket says 172.21.15.6:56594) 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.785+0000 7f0ef3e00700 1 -- 172.21.15.6:0/730366668 learned_addr learned my addr 172.21.15.6:0/730366668 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.785+0000 7f0eeb400700 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573394804a0 0x5573394fbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0ef8a00700 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55733948b350 0x5573394926a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0ef3e00700 1 -- 172.21.15.6:0/730366668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55733948b350 msgr2=0x5573394926a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0ef3e00700 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55733948b350 0x5573394926a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0ef3e00700 1 -- 172.21.15.6:0/730366668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573394804a0 msgr2=0x5573394fbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0ef3e00700 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573394804a0 0x5573394fbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0ef3e00700 1 -- 172.21.15.6:0/730366668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557339492da0 con 0x55733947fb80 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0eeb400700 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573394804a0 0x5573394fbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0ef8a00700 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55733948b350 0x5573394926a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:55:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0ef3e00700 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x5573394fb670 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4014e30 tx=0x7f0ee4014be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=296a21c1c7f47436 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:38.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0ee2000700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:55:38.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0ee2000700 1 -- 172.21.15.6:0/730366668 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4028990 con 0x55733947fb80 2024-04-26T01:55:38.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0ee2000700 1 -- 172.21.15.6:0/730366668 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f0ee4016590 con 0x55733947fb80 2024-04-26T01:55:38.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.786+0000 7f0f05422b80 1 -- 172.21.15.6:0/730366668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ee4014c60 con 0x55733947fb80 2024-04-26T01:55:38.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.787+0000 7f0ee2000700 1 -- 172.21.15.6:0/730366668 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0ee4041050 con 0x55733947fb80 2024-04-26T01:55:38.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.787+0000 7f0f05422b80 10 client.26357 fetch_fsmap learned FSMap version 43 2024-04-26T01:55:38.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.787+0000 7f0f05422b80 1 -- 172.21.15.6:0/730366668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x557339502e80 con 0x55733947fb80 2024-04-26T01:55:38.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.787+0000 7f0ee2000700 1 -- 172.21.15.6:0/730366668 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ee4014c60 con 0x55733947fb80 2024-04-26T01:55:38.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.787+0000 7f0ee2000700 1 -- 172.21.15.6:0/730366668 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f0ee402f070 con 0x55733947fb80 2024-04-26T01:55:38.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.787+0000 7f0f05422b80 10 client.26357 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:55:38.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.787+0000 7f0f05422b80 10 client.26357 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:55:38.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.787+0000 7f0f05422b80 20 client.26357 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:38.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:38 smithi006 ceph-mon[28419]: pgmap v306: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 113 op/s 2024-04-26T01:55:38.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.787+0000 7f0f05422b80 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55733949ac90 0x557339505470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:38.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.787+0000 7f0f05422b80 4 client.26357 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.1, "await_for": 42.0}] 2024-04-26T01:55:38.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.787+0000 7f0f05422b80 1 -- 172.21.15.6:0/730366668 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.1, "await_for": 42.0}) -- 0x5573393da8b0 con 0x55733949ac90 2024-04-26T01:55:38.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.788+0000 7f0eeb400700 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55733949ac90 0x557339505470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:38.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.788+0000 7f0eeb400700 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55733949ac90 0x557339505470 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3aaf5568af673eb server_cookie=cd3a694036dfd3 in_seq=0 out_seq=0 2024-04-26T01:55:38.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.788+0000 7f0ee2000700 10 client.26357 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.849+0000 7f0ee2000700 1 -- 172.21.15.6:0/730366668 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5573393da8b0 con 0x55733949ac90 2024-04-26T01:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.849+0000 7f0ee2000700 10 client.26357 handle_command_reply: tid=0 2024-04-26T01:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.849+0000 7f0f05422b80 1 client.26357 shutdown 2024-04-26T01:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.849+0000 7f0f05422b80 1 -- 172.21.15.6:0/730366668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 msgr2=0x5573394fb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.849+0000 7f0f05422b80 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x5573394fb670 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4014e30 tx=0x7f0ee4014be0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.849+0000 7f0f05422b80 1 -- 172.21.15.6:0/730366668 shutdown_connections 2024-04-26T01:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.850+0000 7f0f05422b80 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55733949ac90 0x557339505470 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.850+0000 7f0f05422b80 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55733948b350 0x5573394926a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.850+0000 7f0f05422b80 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55733947fb80 0x5573394fb670 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.850+0000 7f0f05422b80 1 --2- 172.21.15.6:0/730366668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573394804a0 0x5573394fbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.850+0000 7f0f05422b80 1 -- 172.21.15.6:0/730366668 >> 172.21.15.6:0/730366668 conn(0x5573393ae730 msgr2=0x557339489eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.850+0000 7f0ee2000700 0 client.26357 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.850+0000 7f0f05422b80 1 -- 172.21.15.6:0/730366668 shutdown_connections 2024-04-26T01:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.850+0000 7f0f05422b80 1 -- 172.21.15.6:0/730366668 wait complete. 2024-04-26T01:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.850+0000 7f0f05422b80 20 client.26357 trim_cache size 0 max 16384 2024-04-26T01:55:38.850 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:55:38.850 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 43, 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 30, 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "eeff187f": { 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 30, 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 82.1, 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:55:38.851 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:55:38.852 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:55:38.852 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:55:38.852 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:55:38.852 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:38.852 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:38.852 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:38.852 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:38.852 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:55:38.852 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.852+0000 7f0ef9400700 1 -- 172.21.15.6:0/4050841132 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0ed80a3e50 msgr2=0x7f0ed80a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.852+0000 7f0ef9400700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0ed80a3e50 0x7f0ed80a6310 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4106670 tx=0x7f0edc009f80 comp rx=0 tx=0).stop 2024-04-26T01:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.852+0000 7f0ef9400700 1 -- 172.21.15.6:0/4050841132 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ef41138c0 msgr2=0x7f0ef41a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.852+0000 7f0ef9400700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ef41138c0 0x7f0ef41a3eb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00d750 tx=0x7f0eec00da60 comp rx=0 tx=0).stop 2024-04-26T01:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.852+0000 7f0ef9400700 1 -- 172.21.15.6:0/4050841132 shutdown_connections 2024-04-26T01:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.852+0000 7f0ef9400700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0ed80a3e50 0x7f0ed80a6310 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.852+0000 7f0ef9400700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ef41138c0 0x7f0ef41a3eb0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.852+0000 7f0ef9400700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ef4074e00 0x7f0ef419f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.852+0000 7f0ef9400700 1 --2- 172.21.15.6:0/4050841132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4075720 0x7f0ef419fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.852+0000 7f0ef9400700 1 -- 172.21.15.6:0/4050841132 >> 172.21.15.6:0/4050841132 conn(0x7f0ef40fe3a0 msgr2=0x7f0ef40ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.852+0000 7f0ef9400700 1 -- 172.21.15.6:0/4050841132 shutdown_connections 2024-04-26T01:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:55:38.852+0000 7f0ef9400700 1 -- 172.21.15.6:0/4050841132 wait complete. 2024-04-26T01:55:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:38 smithi188 ceph-mon[32751]: pgmap v306: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 113 op/s 2024-04-26T01:55:38.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:38 smithi029 ceph-mon[32775]: pgmap v306: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 113 op/s 2024-04-26T01:55:39.614 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.65 MB/sec execute 363 sec latency 753.349 ms 2024-04-26T01:55:39.794 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'eeff187f', quiesce took 7.8 seconds. Will release after: 22.1 2024-04-26T01:55:40.617 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.64 MB/sec execute 364 sec latency 1753.478 ms 2024-04-26T01:55:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.644+0000 7f0df3000640 1 -- 172.21.15.6:0/2086444949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec10e600 msgr2=0x7f0dec110a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.644+0000 7f0df3000640 1 --2- 172.21.15.6:0/2086444949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec10e600 0x7f0dec110a10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0de4009900 tx=0x7f0de402a490 comp rx=0 tx=0).stop 2024-04-26T01:55:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.645+0000 7f0df3000640 1 -- 172.21.15.6:0/2086444949 shutdown_connections 2024-04-26T01:55:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.645+0000 7f0df3000640 1 --2- 172.21.15.6:0/2086444949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec10e600 0x7f0dec110a10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.645+0000 7f0df3000640 1 --2- 172.21.15.6:0/2086444949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dec074c40 0x7f0dec10e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.645+0000 7f0df3000640 1 --2- 172.21.15.6:0/2086444949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dec074320 0x7f0dec074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.645+0000 7f0df3000640 1 -- 172.21.15.6:0/2086444949 >> 172.21.15.6:0/2086444949 conn(0x7f0dec0fd890 msgr2=0x7f0dec0ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.645+0000 7f0df3000640 1 -- 172.21.15.6:0/2086444949 shutdown_connections 2024-04-26T01:55:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.645+0000 7f0df3000640 1 -- 172.21.15.6:0/2086444949 wait complete. 2024-04-26T01:55:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.646+0000 7f0df3000640 1 Processor -- start 2024-04-26T01:55:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.646+0000 7f0df3000640 1 -- start start 2024-04-26T01:55:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.646+0000 7f0df3000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec074320 0x7f0dec1a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.646+0000 7f0df3000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dec074c40 0x7f0dec1a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.646+0000 7f0df3000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dec10e600 0x7f0dec19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.646+0000 7f0df3000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1141d0 con 0x7f0dec074320 2024-04-26T01:55:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.646+0000 7f0df3000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec114050 con 0x7f0dec074c40 2024-04-26T01:55:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.646+0000 7f0df3000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec114350 con 0x7f0dec10e600 2024-04-26T01:55:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.647+0000 7f0df1c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec074320 0x7f0dec1a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.647+0000 7f0df1200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dec074c40 0x7f0dec1a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.647+0000 7f0df1c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec074320 0x7f0dec1a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34652/0 (socket says 172.21.15.6:34652) 2024-04-26T01:55:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.647+0000 7f0df1c00640 1 -- 172.21.15.6:0/2905266550 learned_addr learned my addr 172.21.15.6:0/2905266550 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.647+0000 7f0df2600640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dec10e600 0x7f0dec19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.647+0000 7f0df1200640 1 -- 172.21.15.6:0/2905266550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dec10e600 msgr2=0x7f0dec19a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.647+0000 7f0df1200640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dec10e600 0x7f0dec19a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.647+0000 7f0df1200640 1 -- 172.21.15.6:0/2905266550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec074320 msgr2=0x7f0dec1a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.647+0000 7f0df1200640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec074320 0x7f0dec1a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.647+0000 7f0df1200640 1 -- 172.21.15.6:0/2905266550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dec19ad60 con 0x7f0dec074c40 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.647+0000 7f0df1c00640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec074320 0x7f0dec1a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.647+0000 7f0df1200640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dec074c40 0x7f0dec1a0920 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800e9f0 tx=0x7f0dd800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.648+0000 7f0dea000640 1 -- 172.21.15.6:0/2905266550 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd800ccc0 con 0x7f0dec074c40 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.648+0000 7f0df3000640 1 -- 172.21.15.6:0/2905266550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dec19b050 con 0x7f0dec074c40 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.648+0000 7f0df3000640 1 -- 172.21.15.6:0/2905266550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dec1a72a0 con 0x7f0dec074c40 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.648+0000 7f0dea000640 1 -- 172.21.15.6:0/2905266550 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dd80108e0 con 0x7f0dec074c40 2024-04-26T01:55:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.648+0000 7f0dea000640 1 -- 172.21.15.6:0/2905266550 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8018750 con 0x7f0dec074c40 2024-04-26T01:55:40.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df743d740 1 Processor -- start 2024-04-26T01:55:40.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df743d740 1 -- start start 2024-04-26T01:55:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df743d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe989b0 0x56224fe98d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0dea000640 1 -- 172.21.15.6:0/2905266550 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f0dd8010430 con 0x7f0dec074c40 2024-04-26T01:55:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df743d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 0x56224fea3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df743d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fea3b90 0x56224fea5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df743d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56224fdfe900 con 0x56224fe989b0 2024-04-26T01:55:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df743d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56224fda4ef0 con 0x56224fe992d0 2024-04-26T01:55:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df743d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56224fdfa8c0 con 0x56224fea3b90 2024-04-26T01:55:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df1c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 0x56224fea3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df2600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe989b0 0x56224fe98d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df1c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 0x56224fea3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56628/0 (socket says 172.21.15.6:56628) 2024-04-26T01:55:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df1c00640 1 -- 172.21.15.6:0/849470735 learned_addr learned my addr 172.21.15.6:0/849470735 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.652+0000 7f0df1200640 1 --2- 172.21.15.6:0/849470735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fea3b90 0x56224fea5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.653+0000 7f0df1c00640 1 -- 172.21.15.6:0/849470735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fea3b90 msgr2=0x56224fea5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.653+0000 7f0df1c00640 1 --2- 172.21.15.6:0/849470735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fea3b90 0x56224fea5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.653+0000 7f0df1c00640 1 -- 172.21.15.6:0/849470735 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe989b0 msgr2=0x56224fe98d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.653+0000 7f0df1c00640 1 --2- 172.21.15.6:0/849470735 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe989b0 0x56224fe98d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.653+0000 7f0df1c00640 1 -- 172.21.15.6:0/849470735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56224fb11210 con 0x56224fe992d0 2024-04-26T01:55:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.653+0000 7f0df1c00640 1 --2- 172.21.15.6:0/849470735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 0x56224fea3650 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0de000f8d0 tx=0x7f0de000fd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36cbbccf98837dfc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.653+0000 7f0ddea00640 1 -- 172.21.15.6:0/849470735 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de0011c20 con 0x56224fe992d0 2024-04-26T01:55:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.653+0000 7f0ddea00640 1 -- 172.21.15.6:0/849470735 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0de0011dc0 con 0x56224fe992d0 2024-04-26T01:55:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.653+0000 7f0ddea00640 1 -- 172.21.15.6:0/849470735 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de0018700 con 0x56224fe992d0 2024-04-26T01:55:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.653+0000 7f0dea000640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0dd40a3d00 0x7f0dd40a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.654+0000 7f0dea000640 1 -- 172.21.15.6:0/2905266550 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f0dd80d0ad0 con 0x7f0dec074c40 2024-04-26T01:55:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.654+0000 7f0df2600640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0dd40a3d00 0x7f0dd40a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.654+0000 7f0df743d740 1 -- 172.21.15.6:0/849470735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 msgr2=0x56224fea3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.654+0000 7f0df743d740 1 --2- 172.21.15.6:0/849470735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 0x56224fea3650 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0de000f8d0 tx=0x7f0de000fd90 comp rx=0 tx=0).stop 2024-04-26T01:55:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.654+0000 7f0df743d740 1 -- 172.21.15.6:0/849470735 shutdown_connections 2024-04-26T01:55:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.654+0000 7f0df743d740 1 --2- 172.21.15.6:0/849470735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fea3b90 0x56224fea5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.654+0000 7f0df743d740 1 --2- 172.21.15.6:0/849470735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 0x56224fea3650 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.654+0000 7f0df743d740 1 --2- 172.21.15.6:0/849470735 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe989b0 0x56224fe98d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.654+0000 7f0df743d740 1 -- 172.21.15.6:0/849470735 >> 172.21.15.6:0/849470735 conn(0x56224fe94ec0 msgr2=0x56224fe952a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.654+0000 7f0df743d740 1 -- 172.21.15.6:0/849470735 shutdown_connections 2024-04-26T01:55:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.654+0000 7f0df743d740 1 -- 172.21.15.6:0/849470735 wait complete. 2024-04-26T01:55:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.655+0000 7f0df2600640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0dd40a3d00 0x7f0dd40a61c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0de4005940 tx=0x7f0de40058d0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.655+0000 7f0df743d740 1 -- start start 2024-04-26T01:55:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.655+0000 7f0df743d740 1 Processor -- start 2024-04-26T01:55:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.655+0000 7f0df743d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe989b0 0x56224fea0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.655+0000 7f0df743d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe992d0 0x56224fea0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df743d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fea3b90 0x56224fea1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df743d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56224fea67c0 con 0x56224fe992d0 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df743d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56224fdb8310 con 0x56224fea3b90 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df743d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56224fea6940 con 0x56224fe989b0 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df743d740 10 client.0 _ll_register_callbacks cb 0x56224fd86cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df1c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe989b0 0x56224fea0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df1200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe992d0 0x56224fea0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df2600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fea3b90 0x56224fea1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df1c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe989b0 0x56224fea0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36484/0 (socket says 172.21.15.6:36484) 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df1c00640 1 -- 172.21.15.6:0/1271433810 learned_addr learned my addr 172.21.15.6:0/1271433810 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df1c00640 1 -- 172.21.15.6:0/1271433810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fea3b90 msgr2=0x56224fea1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df1c00640 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fea3b90 0x56224fea1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df1c00640 1 -- 172.21.15.6:0/1271433810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe992d0 msgr2=0x56224fea0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df1c00640 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe992d0 0x56224fea0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df1c00640 1 -- 172.21.15.6:0/1271433810 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56224fbdd620 con 0x56224fe989b0 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df1200640 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe992d0 0x56224fea0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.656+0000 7f0df2600640 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fea3b90 0x56224fea1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.657+0000 7f0df1c00640 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe989b0 0x56224fea0a10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0de0000f80 tx=0x7f0de0003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94d7906255348831 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.657+0000 7f0dd3400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.657+0000 7f0dd3400640 1 -- 172.21.15.6:0/1271433810 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de0004450 con 0x56224fe989b0 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.657+0000 7f0dd3400640 1 -- 172.21.15.6:0/1271433810 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f0de00045f0 con 0x56224fe989b0 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.657+0000 7f0df743d740 1 -- 172.21.15.6:0/1271433810 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0de0011dc0 con 0x56224fe989b0 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.657+0000 7f0dd3400640 1 -- 172.21.15.6:0/1271433810 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0de0003e40 con 0x56224fe989b0 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.657+0000 7f0df743d740 10 client.26407 fetch_fsmap learned FSMap version 43 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.657+0000 7f0df743d740 1 -- 172.21.15.6:0/1271433810 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56224feb2980 con 0x56224fe989b0 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.658+0000 7f0dd3400640 1 -- 172.21.15.6:0/1271433810 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0de0011dc0 con 0x56224fe989b0 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.658+0000 7f0dd3400640 1 -- 172.21.15.6:0/1271433810 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f0de0022430 con 0x56224fe989b0 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.658+0000 7f0df743d740 10 client.26407 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:55:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.658+0000 7f0df743d740 10 client.26407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:40.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.658+0000 7f0df743d740 20 client.26407 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:40.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.658+0000 7f0df743d740 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56224feb3de0 0x56224feb61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.658+0000 7f0df743d740 4 client.26407 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:40.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.658+0000 7f0df743d740 1 -- 172.21.15.6:0/1271433810 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56224fe128b0 con 0x56224feb3de0 2024-04-26T01:55:40.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.658+0000 7f0df1200640 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56224feb3de0 0x56224feb61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.659+0000 7f0df1200640 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56224feb3de0 0x56224feb61d0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f2bb7e881bfd664 server_cookie=e33af7b45bd0713f in_seq=0 out_seq=0 2024-04-26T01:55:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.659+0000 7f0dd3400640 10 client.26407 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:40.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0dd3400640 1 -- 172.21.15.6:0/1271433810 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56224fe128b0 con 0x56224feb3de0 2024-04-26T01:55:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0dd3400640 10 client.26407 handle_command_reply: tid=0 2024-04-26T01:55:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0df743d740 1 client.26407 shutdown 2024-04-26T01:55:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0df743d740 1 -- 172.21.15.6:0/1271433810 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe989b0 msgr2=0x56224fea0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0df743d740 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe989b0 0x56224fea0a10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0de0000f80 tx=0x7f0de0003c20 comp rx=0 tx=0).stop 2024-04-26T01:55:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0df743d740 1 -- 172.21.15.6:0/1271433810 shutdown_connections 2024-04-26T01:55:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0df743d740 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56224feb3de0 0x56224feb61d0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0df743d740 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe992d0 0x56224fea0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0dd3400640 0 client.26407 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0df743d740 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fea3b90 0x56224fea1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0df743d740 1 --2- 172.21.15.6:0/1271433810 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe989b0 0x56224fea0a10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0df743d740 1 -- 172.21.15.6:0/1271433810 >> 172.21.15.6:0/1271433810 conn(0x56224fe94ec0 msgr2=0x56224fe9ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:40.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0df743d740 1 -- 172.21.15.6:0/1271433810 shutdown_connections 2024-04-26T01:55:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0df743d740 1 -- 172.21.15.6:0/1271433810 wait complete. 2024-04-26T01:55:40.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.660+0000 7f0df743d740 20 client.26407 trim_cache size 0 max 16384 2024-04-26T01:55:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.676+0000 7f0df743d740 1 Processor -- start 2024-04-26T01:55:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.677+0000 7f0df743d740 1 -- start start 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.677+0000 7f0df743d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe989b0 0x56224feaf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.677+0000 7f0df743d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 0x56224fe9fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.677+0000 7f0df743d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fea3b90 0x56224febba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.677+0000 7f0df743d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc0111a0 con 0x56224fe989b0 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.677+0000 7f0df743d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc00ea70 con 0x56224fe992d0 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.677+0000 7f0df743d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc00c8e0 con 0x56224fea3b90 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.677+0000 7f0df1c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe989b0 0x56224feaf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.677+0000 7f0df1c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe989b0 0x56224feaf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34672/0 (socket says 172.21.15.6:34672) 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.677+0000 7f0df1c00640 1 -- 172.21.15.6:0/2359344597 learned_addr learned my addr 172.21.15.6:0/2359344597 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.677+0000 7f0df1200640 1 --2- 172.21.15.6:0/2359344597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 0x56224fe9fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.677+0000 7f0df2600640 1 --2- 172.21.15.6:0/2359344597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fea3b90 0x56224febba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.678+0000 7f0df1200640 1 -- 172.21.15.6:0/2359344597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fea3b90 msgr2=0x56224febba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.678+0000 7f0df1200640 1 --2- 172.21.15.6:0/2359344597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fea3b90 0x56224febba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.678 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.678+0000 7f0df1200640 1 -- 172.21.15.6:0/2359344597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe989b0 msgr2=0x56224feaf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:40.678 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.678+0000 7f0df1200640 1 --2- 172.21.15.6:0/2359344597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe989b0 0x56224feaf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.678 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.678+0000 7f0df1200640 1 -- 172.21.15.6:0/2359344597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56224fe99100 con 0x56224fe992d0 2024-04-26T01:55:40.678 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.678+0000 7f0df1c00640 1 --2- 172.21.15.6:0/2359344597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe989b0 0x56224feaf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:40.678 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.678+0000 7f0df1200640 1 --2- 172.21.15.6:0/2359344597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 0x56224fe9fb70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0dd80ce140 tx=0x7f0dd80d2f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38919c525d1624ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.678+0000 7f0dde000640 1 -- 172.21.15.6:0/2359344597 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd80a0d70 con 0x56224fe992d0 2024-04-26T01:55:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.679+0000 7f0dde000640 1 -- 172.21.15.6:0/2359344597 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dd80a5630 con 0x56224fe992d0 2024-04-26T01:55:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.679+0000 7f0dde000640 1 -- 172.21.15.6:0/2359344597 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8098b40 con 0x56224fe992d0 2024-04-26T01:55:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.679+0000 7f0df743d740 1 -- 172.21.15.6:0/2359344597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 msgr2=0x56224fe9fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.679+0000 7f0df743d740 1 --2- 172.21.15.6:0/2359344597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 0x56224fe9fb70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0dd80ce140 tx=0x7f0dd80d2f90 comp rx=0 tx=0).stop 2024-04-26T01:55:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.679+0000 7f0df743d740 1 -- 172.21.15.6:0/2359344597 shutdown_connections 2024-04-26T01:55:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.679+0000 7f0df743d740 1 --2- 172.21.15.6:0/2359344597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fea3b90 0x56224febba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.681 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.679+0000 7f0df743d740 1 --2- 172.21.15.6:0/2359344597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe992d0 0x56224fe9fb70 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.679+0000 7f0df743d740 1 --2- 172.21.15.6:0/2359344597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fe989b0 0x56224feaf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.679+0000 7f0df743d740 1 -- 172.21.15.6:0/2359344597 >> 172.21.15.6:0/2359344597 conn(0x56224fe94ec0 msgr2=0x56224fe952a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.679+0000 7f0df743d740 1 -- 172.21.15.6:0/2359344597 shutdown_connections 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.679+0000 7f0df743d740 1 -- 172.21.15.6:0/2359344597 wait complete. 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.680+0000 7f0df743d740 1 -- start start 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.680+0000 7f0df743d740 1 Processor -- start 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.680+0000 7f0df743d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe989b0 0x56224fea8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.680+0000 7f0df743d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe992d0 0x56224fea90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.680+0000 7f0df743d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fea3b90 0x56224fea9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.680+0000 7f0df743d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc00cf30 con 0x56224fea3b90 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.680+0000 7f0df743d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc00e3a0 con 0x56224fe989b0 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.680+0000 7f0df743d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc00d600 con 0x56224fe992d0 2024-04-26T01:55:40.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.680+0000 7f0df743d740 10 client.0 _ll_register_callbacks cb 0x7f0de0005a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0df1200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe992d0 0x56224fea90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0df1200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe992d0 0x56224fea90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36508/0 (socket says 172.21.15.6:36508) 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0df2600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fea3b90 0x56224fea9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0df1200640 1 -- 172.21.15.6:0/2830452080 learned_addr learned my addr 172.21.15.6:0/2830452080 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0df1c00640 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe989b0 0x56224fea8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0df1200640 1 -- 172.21.15.6:0/2830452080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe989b0 msgr2=0x56224fea8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0df1200640 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe989b0 0x56224fea8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0df1200640 1 -- 172.21.15.6:0/2830452080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fea3b90 msgr2=0x56224fea9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0df1200640 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fea3b90 0x56224fea9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0df1200640 1 -- 172.21.15.6:0/2830452080 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56224ff18270 con 0x56224fe992d0 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0df1c00640 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe989b0 0x56224fea8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0df1200640 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe992d0 0x56224fea90d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0dd80ce1a0 tx=0x7f0dd80bd170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9391df164d5876c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.681+0000 7f0dd3400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:55:40.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.682+0000 7f0dd3400640 1 -- 172.21.15.6:0/2830452080 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd80ce5f0 con 0x56224fe992d0 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.682+0000 7f0df743d740 1 -- 172.21.15.6:0/2830452080 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dd80a5630 con 0x56224fe992d0 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.682+0000 7f0dd3400640 1 -- 172.21.15.6:0/2830452080 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f0dd80988c0 con 0x56224fe992d0 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.682+0000 7f0dd3400640 1 -- 172.21.15.6:0/2830452080 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0dd8098b00 con 0x56224fe992d0 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.682+0000 7f0df743d740 10 client.26413 fetch_fsmap learned FSMap version 43 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.682+0000 7f0df743d740 1 -- 172.21.15.6:0/2830452080 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56224ff194c0 con 0x56224fe992d0 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.682+0000 7f0dd3400640 1 -- 172.21.15.6:0/2830452080 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dd80a5630 con 0x56224fe992d0 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.682+0000 7f0dd3400640 1 -- 172.21.15.6:0/2830452080 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f0dd8010a90 con 0x56224fe992d0 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.682+0000 7f0df743d740 10 client.26413 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.682+0000 7f0df743d740 10 client.26413 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.682+0000 7f0df743d740 20 client.26413 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.683+0000 7f0df743d740 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56224ff19b30 0x56224ff1bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.683+0000 7f0df743d740 4 client.26413 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.683+0000 7f0df743d740 1 -- 172.21.15.6:0/2830452080 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56224fe0c320 con 0x56224ff19b30 2024-04-26T01:55:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.683+0000 7f0df1c00640 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56224ff19b30 0x56224ff1bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.683+0000 7f0df1c00640 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56224ff19b30 0x56224ff1bf50 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=963e057b2abe2ddb server_cookie=4cb9692e0e6c4698 in_seq=0 out_seq=0 2024-04-26T01:55:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.683+0000 7f0dd3400640 10 client.26413 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:40.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0dd3400640 1 -- 172.21.15.6:0/2830452080 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56224fe0c320 con 0x56224ff19b30 2024-04-26T01:55:40.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0dd3400640 10 client.26413 handle_command_reply: tid=0 2024-04-26T01:55:40.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0df743d740 1 client.26413 shutdown 2024-04-26T01:55:40.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0df743d740 1 -- 172.21.15.6:0/2830452080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe992d0 msgr2=0x56224fea90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:40.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0df743d740 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe992d0 0x56224fea90d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0dd80ce1a0 tx=0x7f0dd80bd170 comp rx=0 tx=0).stop 2024-04-26T01:55:40.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0df743d740 1 -- 172.21.15.6:0/2830452080 shutdown_connections 2024-04-26T01:55:40.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0df743d740 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56224ff19b30 0x56224ff1bf50 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0df743d740 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56224fea3b90 0x56224fea9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0df743d740 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56224fe992d0 0x56224fea90d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0dd3400640 0 client.26413 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:40.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0df743d740 1 --2- 172.21.15.6:0/2830452080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56224fe989b0 0x56224fea8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0df743d740 1 -- 172.21.15.6:0/2830452080 >> 172.21.15.6:0/2830452080 conn(0x56224fe94ec0 msgr2=0x56224fea1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:40.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0df743d740 1 -- 172.21.15.6:0/2830452080 shutdown_connections 2024-04-26T01:55:40.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.722+0000 7f0df743d740 1 -- 172.21.15.6:0/2830452080 wait complete. 2024-04-26T01:55:40.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.723+0000 7f0df743d740 20 client.26413 trim_cache size 0 max 16384 2024-04-26T01:55:40.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:55:40.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:55:40.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:55:40.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:55:40.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.723+0000 7f0df3000640 1 -- 172.21.15.6:0/2905266550 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0dd40a3d00 msgr2=0x7f0dd40a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:40.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.723+0000 7f0df3000640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0dd40a3d00 0x7f0dd40a61c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0de4005940 tx=0x7f0de40058d0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.723+0000 7f0df3000640 1 -- 172.21.15.6:0/2905266550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dec074c40 msgr2=0x7f0dec1a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:40.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.723+0000 7f0df3000640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dec074c40 0x7f0dec1a0920 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800e9f0 tx=0x7f0dd800eeb0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.724+0000 7f0df3000640 1 -- 172.21.15.6:0/2905266550 shutdown_connections 2024-04-26T01:55:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.724+0000 7f0df3000640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0dd40a3d00 0x7f0dd40a61c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.724+0000 7f0df3000640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dec10e600 0x7f0dec19a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.724+0000 7f0df3000640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dec074c40 0x7f0dec1a0920 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.724+0000 7f0df3000640 1 --2- 172.21.15.6:0/2905266550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec074320 0x7f0dec1a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.724+0000 7f0df3000640 1 -- 172.21.15.6:0/2905266550 >> 172.21.15.6:0/2905266550 conn(0x7f0dec0fd890 msgr2=0x7f0dec0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.724+0000 7f0df3000640 1 -- 172.21.15.6:0/2905266550 shutdown_connections 2024-04-26T01:55:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:40.724+0000 7f0df3000640 1 -- 172.21.15.6:0/2905266550 wait complete. 2024-04-26T01:55:40.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:40 smithi188 ceph-mon[32751]: pgmap v307: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 993 KiB/s wr, 93 op/s 2024-04-26T01:55:40.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:40 smithi029 ceph-mon[32775]: pgmap v307: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 993 KiB/s wr, 93 op/s 2024-04-26T01:55:41.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:40 smithi006 ceph-mon[28419]: pgmap v307: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 993 KiB/s wr, 93 op/s 2024-04-26T01:55:41.615 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.63 MB/sec execute 365 sec latency 2753.627 ms 2024-04-26T01:55:42.615 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.62 MB/sec execute 366 sec latency 3753.731 ms 2024-04-26T01:55:42.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:42 smithi188 ceph-mon[32751]: pgmap v308: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-04-26T01:55:42.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:42 smithi029 ceph-mon[32775]: pgmap v308: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-04-26T01:55:43.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:42 smithi006 ceph-mon[28419]: pgmap v308: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-04-26T01:55:43.615 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.61 MB/sec execute 367 sec latency 4753.878 ms 2024-04-26T01:55:44.615 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.60 MB/sec execute 368 sec latency 5754.037 ms 2024-04-26T01:55:44.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:44 smithi188 ceph-mon[32751]: pgmap v309: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 920 KiB/s wr, 130 op/s 2024-04-26T01:55:44.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:44 smithi029 ceph-mon[32775]: pgmap v309: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 920 KiB/s wr, 130 op/s 2024-04-26T01:55:45.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:44 smithi006 ceph-mon[28419]: pgmap v309: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 920 KiB/s wr, 130 op/s 2024-04-26T01:55:45.615 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.59 MB/sec execute 369 sec latency 6754.184 ms 2024-04-26T01:55:46.615 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.58 MB/sec execute 370 sec latency 7754.335 ms 2024-04-26T01:55:46.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:46 smithi188 ceph-mon[32751]: pgmap v310: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 920 KiB/s wr, 130 op/s 2024-04-26T01:55:46.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:46 smithi029 ceph-mon[32775]: pgmap v310: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 920 KiB/s wr, 130 op/s 2024-04-26T01:55:47.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:46 smithi006 ceph-mon[28419]: pgmap v310: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 920 KiB/s wr, 130 op/s 2024-04-26T01:55:47.615 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.57 MB/sec execute 371 sec latency 8754.428 ms 2024-04-26T01:55:48.616 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.56 MB/sec execute 372 sec latency 9754.524 ms 2024-04-26T01:55:48.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:48 smithi188 ceph-mon[32751]: pgmap v311: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 923 KiB/s wr, 134 op/s 2024-04-26T01:55:48.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:48 smithi029 ceph-mon[32775]: pgmap v311: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 923 KiB/s wr, 134 op/s 2024-04-26T01:55:49.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:48 smithi006 ceph-mon[28419]: pgmap v311: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 923 KiB/s wr, 134 op/s 2024-04-26T01:55:49.616 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.55 MB/sec execute 373 sec latency 10754.639 ms 2024-04-26T01:55:50.616 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.54 MB/sec execute 374 sec latency 11754.746 ms 2024-04-26T01:55:50.870 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:50 smithi006 ceph-mon[28419]: pgmap v312: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 320 KiB/s wr, 63 op/s 2024-04-26T01:55:50.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:50 smithi188 ceph-mon[32751]: pgmap v312: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 320 KiB/s wr, 63 op/s 2024-04-26T01:55:50.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:50 smithi029 ceph-mon[32775]: pgmap v312: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 320 KiB/s wr, 63 op/s 2024-04-26T01:55:51.616 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.53 MB/sec execute 375 sec latency 12754.840 ms 2024-04-26T01:55:52.616 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.52 MB/sec execute 376 sec latency 13754.968 ms 2024-04-26T01:55:52.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:52 smithi188 ceph-mon[32751]: pgmap v313: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 908 KiB/s wr, 84 op/s 2024-04-26T01:55:52.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:52 smithi029 ceph-mon[32775]: pgmap v313: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 908 KiB/s wr, 84 op/s 2024-04-26T01:55:53.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:52 smithi006 ceph-mon[28419]: pgmap v313: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 908 KiB/s wr, 84 op/s 2024-04-26T01:55:53.617 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.51 MB/sec execute 377 sec latency 14755.510 ms 2024-04-26T01:55:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.871+0000 7f68b6a00640 1 -- 172.21.15.6:0/1679097732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b0102f30 msgr2=0x7f68b0103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.871+0000 7f68b6a00640 1 --2- 172.21.15.6:0/1679097732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b0102f30 0x7f68b0103310 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009900 tx=0x7f68a002a410 comp rx=0 tx=0).stop 2024-04-26T01:55:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.871+0000 7f68b6a00640 1 -- 172.21.15.6:0/1679097732 shutdown_connections 2024-04-26T01:55:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.871+0000 7f68b6a00640 1 --2- 172.21.15.6:0/1679097732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b010c3a0 0x7f68b010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.871+0000 7f68b6a00640 1 --2- 172.21.15.6:0/1679097732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68b0103850 0x7f68b010bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.871+0000 7f68b6a00640 1 --2- 172.21.15.6:0/1679097732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b0102f30 0x7f68b0103310 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.871+0000 7f68b6a00640 1 -- 172.21.15.6:0/1679097732 >> 172.21.15.6:0/1679097732 conn(0x7f68b00fc790 msgr2=0x7f68b00febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.871+0000 7f68b6a00640 1 -- 172.21.15.6:0/1679097732 shutdown_connections 2024-04-26T01:55:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.871+0000 7f68b6a00640 1 -- 172.21.15.6:0/1679097732 wait complete. 2024-04-26T01:55:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.873+0000 7f68b6a00640 1 Processor -- start 2024-04-26T01:55:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.873+0000 7f68b6a00640 1 -- start start 2024-04-26T01:55:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.873+0000 7f68b6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b0102f30 0x7f68b01a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.873+0000 7f68b6a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68b0103850 0x7f68b01a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.873+0000 7f68b6a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b010c3a0 0x7f68b019a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.873+0000 7f68b6a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f68b0112050 con 0x7f68b0102f30 2024-04-26T01:55:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.873+0000 7f68b6a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f68b0111ed0 con 0x7f68b010c3a0 2024-04-26T01:55:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.873+0000 7f68b6a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f68b01121d0 con 0x7f68b0103850 2024-04-26T01:55:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.873+0000 7f68b5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b0102f30 0x7f68b01a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.874+0000 7f68b6000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b010c3a0 0x7f68b019a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.874+0000 7f68b6000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b010c3a0 0x7f68b019a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:60516/0 (socket says 172.21.15.6:60516) 2024-04-26T01:55:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.874+0000 7f68b6000640 1 -- 172.21.15.6:0/1492515135 learned_addr learned my addr 172.21.15.6:0/1492515135 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.874+0000 7f68b4c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68b0103850 0x7f68b01a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.874+0000 7f68b6000640 1 -- 172.21.15.6:0/1492515135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68b0103850 msgr2=0x7f68b01a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.874+0000 7f68b6000640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68b0103850 0x7f68b01a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.874+0000 7f68b6000640 1 -- 172.21.15.6:0/1492515135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b0102f30 msgr2=0x7f68b01a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.874+0000 7f68b6000640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b0102f30 0x7f68b01a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.874+0000 7f68b6000640 1 -- 172.21.15.6:0/1492515135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b019ad30 con 0x7f68b010c3a0 2024-04-26T01:55:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.874+0000 7f68b5600640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b0102f30 0x7f68b01a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.874+0000 7f68b4c00640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68b0103850 0x7f68b01a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.874+0000 7f68b6000640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b010c3a0 0x7f68b019a5b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009fd0 tx=0x7f68a800b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.875+0000 7f68ae000640 1 -- 172.21.15.6:0/1492515135 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a8017070 con 0x7f68b010c3a0 2024-04-26T01:55:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.875+0000 7f68b6a00640 1 -- 172.21.15.6:0/1492515135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68b019b020 con 0x7f68b010c3a0 2024-04-26T01:55:53.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.875+0000 7f68ae000640 1 -- 172.21.15.6:0/1492515135 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a8003e20 con 0x7f68b010c3a0 2024-04-26T01:55:53.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.875+0000 7f68ae000640 1 -- 172.21.15.6:0/1492515135 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a8010460 con 0x7f68b010c3a0 2024-04-26T01:55:53.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.875+0000 7f68b6a00640 1 -- 172.21.15.6:0/1492515135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68b01a72c0 con 0x7f68b010c3a0 2024-04-26T01:55:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.878+0000 7f68bad69740 1 Processor -- start 2024-04-26T01:55:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.878+0000 7f68ae000640 1 -- 172.21.15.6:0/1492515135 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f68a8003970 con 0x7f68b010c3a0 2024-04-26T01:55:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.878+0000 7f68bad69740 1 -- start start 2024-04-26T01:55:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68bad69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c481d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68bad69740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c48c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68bad69740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c48ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68bad69740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555a2c3e7900 con 0x555a2c4822d0 2024-04-26T01:55:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68bad69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555a2c38def0 con 0x555a2c4819b0 2024-04-26T01:55:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68bad69740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555a2c3e38c0 con 0x555a2c48cb90 2024-04-26T01:55:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68b5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c48c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68b5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c48c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41704/0 (socket says 172.21.15.6:41704) 2024-04-26T01:55:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68b5600640 1 -- 172.21.15.6:0/2766835784 learned_addr learned my addr 172.21.15.6:0/2766835784 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68b4c00640 1 --2- 172.21.15.6:0/2766835784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c481d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68b4c00640 1 --2- 172.21.15.6:0/2766835784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c48ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68b5600640 1 -- 172.21.15.6:0/2766835784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 msgr2=0x555a2c48ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:53.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68b5600640 1 --2- 172.21.15.6:0/2766835784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c48ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68b5600640 1 -- 172.21.15.6:0/2766835784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 msgr2=0x555a2c481d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:53.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68b5600640 1 --2- 172.21.15.6:0/2766835784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c481d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.879+0000 7f68b5600640 1 -- 172.21.15.6:0/2766835784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a2c0fa210 con 0x555a2c4822d0 2024-04-26T01:55:53.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.880+0000 7f68b4c00640 1 --2- 172.21.15.6:0/2766835784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c481d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:53.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.880+0000 7f68ae000640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68940a3d00 0x7f68940a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.880+0000 7f68b5600640 1 --2- 172.21.15.6:0/2766835784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c48c650 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009230 tx=0x7f68a0005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a052f2beca4ec20 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:53.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.880+0000 7f68ae000640 1 -- 172.21.15.6:0/1492515135 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f68a80d0520 con 0x7f68b010c3a0 2024-04-26T01:55:53.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.880+0000 7f68b4c00640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68940a3d00 0x7f68940a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68a6600640 1 -- 172.21.15.6:0/2766835784 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0035040 con 0x555a2c4822d0 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68a6600640 1 -- 172.21.15.6:0/2766835784 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a0004c20 con 0x555a2c4822d0 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68a6600640 1 -- 172.21.15.6:0/2766835784 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0003a80 con 0x555a2c4822d0 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68bad69740 1 -- 172.21.15.6:0/2766835784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 msgr2=0x555a2c48c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68bad69740 1 --2- 172.21.15.6:0/2766835784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c48c650 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009230 tx=0x7f68a0005c00 comp rx=0 tx=0).stop 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68b4c00640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68940a3d00 0x7f68940a61c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x555a2c47f2e0 tx=0x7f689c005850 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68bad69740 1 -- 172.21.15.6:0/2766835784 shutdown_connections 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68bad69740 1 --2- 172.21.15.6:0/2766835784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c48c650 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68bad69740 1 --2- 172.21.15.6:0/2766835784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c48ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68bad69740 1 --2- 172.21.15.6:0/2766835784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c481d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68bad69740 1 -- 172.21.15.6:0/2766835784 >> 172.21.15.6:0/2766835784 conn(0x555a2c47dec0 msgr2=0x555a2c47e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68bad69740 1 -- 172.21.15.6:0/2766835784 shutdown_connections 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.881+0000 7f68bad69740 1 -- 172.21.15.6:0/2766835784 wait complete. 2024-04-26T01:55:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.882+0000 7f68bad69740 1 -- start start 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.882+0000 7f68bad69740 1 Processor -- start 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68bad69740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c4819b0 0x555a2c489ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68bad69740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c489ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68bad69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c48cb90 0x555a2c48a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68bad69740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555a2c48f760 con 0x555a2c4822d0 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68bad69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555a2c3a1310 con 0x555a2c48cb90 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68bad69740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555a2c48f8e0 con 0x555a2c4819b0 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68bad69740 10 client.0 _ll_register_callbacks cb 0x555a2c36fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68b4c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c489ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68b4c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c489ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41708/0 (socket says 172.21.15.6:41708) 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68b4c00640 1 -- 172.21.15.6:0/4263237771 learned_addr learned my addr 172.21.15.6:0/4263237771 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68b5600640 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c4819b0 0x555a2c489ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68b6000640 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c48cb90 0x555a2c48a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68b4c00640 1 -- 172.21.15.6:0/4263237771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c4819b0 msgr2=0x555a2c489ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68b4c00640 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c4819b0 0x555a2c489ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68b4c00640 1 -- 172.21.15.6:0/4263237771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c48cb90 msgr2=0x555a2c48a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68b4c00640 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c48cb90 0x555a2c48a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.883+0000 7f68b4c00640 1 -- 172.21.15.6:0/4263237771 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a2c1c6620 con 0x555a2c4822d0 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.884+0000 7f68b4c00640 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c489ff0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f689c004610 tx=0x7f689c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a43ffcb83b6c9c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.884+0000 7f689aa00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.884+0000 7f689aa00640 1 -- 172.21.15.6:0/4263237771 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c003ac0 con 0x555a2c4822d0 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.884+0000 7f689aa00640 1 -- 172.21.15.6:0/4263237771 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f689c003c60 con 0x555a2c4822d0 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.884+0000 7f68bad69740 1 -- 172.21.15.6:0/4263237771 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68a0004c20 con 0x555a2c4822d0 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.884+0000 7f689aa00640 1 -- 172.21.15.6:0/4263237771 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f689c003e60 con 0x555a2c4822d0 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.884+0000 7f68bad69740 10 client.16245 fetch_fsmap learned FSMap version 43 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.884+0000 7f68bad69740 1 -- 172.21.15.6:0/4263237771 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x555a2c49ba00 con 0x555a2c4822d0 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.885+0000 7f689aa00640 1 -- 172.21.15.6:0/4263237771 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a0004c20 con 0x555a2c4822d0 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.885+0000 7f689aa00640 1 -- 172.21.15.6:0/4263237771 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f689c01e5b0 con 0x555a2c4822d0 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.885+0000 7f68bad69740 10 client.16245 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.885+0000 7f68bad69740 10 client.16245 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.885+0000 7f68bad69740 20 client.16245 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.885+0000 7f68bad69740 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a2c49cf60 0x555a2c49f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.885+0000 7f68bad69740 4 client.16245 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.885+0000 7f68bad69740 1 -- 172.21.15.6:0/4263237771 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a2c3fb8b0 con 0x555a2c49cf60 2024-04-26T01:55:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.886+0000 7f68b5600640 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a2c49cf60 0x555a2c49f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.887+0000 7f68b5600640 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a2c49cf60 0x555a2c49f390 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a63bc51ea3284e05 server_cookie=cb6907348b0f53d0 in_seq=0 out_seq=0 2024-04-26T01:55:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.887+0000 7f689aa00640 10 client.16245 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f689aa00640 1 -- 172.21.15.6:0/4263237771 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a2c3fb8b0 con 0x555a2c49cf60 2024-04-26T01:55:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f689aa00640 10 client.16245 handle_command_reply: tid=0 2024-04-26T01:55:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f68bad69740 1 client.16245 shutdown 2024-04-26T01:55:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f68bad69740 1 -- 172.21.15.6:0/4263237771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 msgr2=0x555a2c489ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f68bad69740 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c489ff0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f689c004610 tx=0x7f689c005f90 comp rx=0 tx=0).stop 2024-04-26T01:55:53.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f68bad69740 1 -- 172.21.15.6:0/4263237771 shutdown_connections 2024-04-26T01:55:53.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f68bad69740 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a2c49cf60 0x555a2c49f390 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f68bad69740 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c489ff0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f68bad69740 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c48cb90 0x555a2c48a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f68bad69740 1 --2- 172.21.15.6:0/4263237771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c4819b0 0x555a2c489ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f689aa00640 0 client.16245 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:53.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f68bad69740 1 -- 172.21.15.6:0/4263237771 >> 172.21.15.6:0/4263237771 conn(0x555a2c47dec0 msgr2=0x555a2c483c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:53.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f68bad69740 1 -- 172.21.15.6:0/4263237771 shutdown_connections 2024-04-26T01:55:53.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f68bad69740 1 -- 172.21.15.6:0/4263237771 wait complete. 2024-04-26T01:55:53.888 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.888+0000 7f68bad69740 20 client.16245 trim_cache size 0 max 16384 2024-04-26T01:55:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.904+0000 7f68bad69740 1 Processor -- start 2024-04-26T01:55:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.905+0000 7f68bad69740 1 -- start start 2024-04-26T01:55:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.905+0000 7f68bad69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c498620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.905+0000 7f68bad69740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c48b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.905+0000 7f68bad69740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c4a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.905+0000 7f68bad69740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f68b01b26f0 con 0x555a2c4822d0 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.905+0000 7f68bad69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f68b01affc0 con 0x555a2c4819b0 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.905+0000 7f68bad69740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f68b01ade30 con 0x555a2c48cb90 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.905+0000 7f68b4c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c48b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.905+0000 7f68b4c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c48b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41722/0 (socket says 172.21.15.6:41722) 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.905+0000 7f68b5600640 1 --2- 172.21.15.6:0/464737913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c498620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.905+0000 7f68b4c00640 1 -- 172.21.15.6:0/464737913 learned_addr learned my addr 172.21.15.6:0/464737913 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.906+0000 7f68b6000640 1 --2- 172.21.15.6:0/464737913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c4a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.906+0000 7f68b5600640 1 -- 172.21.15.6:0/464737913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 msgr2=0x555a2c4a4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.906+0000 7f68b5600640 1 --2- 172.21.15.6:0/464737913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c4a4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.906+0000 7f68b5600640 1 -- 172.21.15.6:0/464737913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 msgr2=0x555a2c48b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.906+0000 7f68b5600640 1 --2- 172.21.15.6:0/464737913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c48b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.906+0000 7f68b5600640 1 -- 172.21.15.6:0/464737913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a2c488020 con 0x555a2c4819b0 2024-04-26T01:55:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.906+0000 7f68b4c00640 1 --2- 172.21.15.6:0/464737913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c48b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:53.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.906+0000 7f68b5600640 1 --2- 172.21.15.6:0/464737913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c498620 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f68a0005ab0 tx=0x7f68a0005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dc65dfdfd728264 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:53.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.906+0000 7f68a5c00640 1 -- 172.21.15.6:0/464737913 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a005a070 con 0x555a2c4819b0 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.906+0000 7f68a5c00640 1 -- 172.21.15.6:0/464737913 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a0005630 con 0x555a2c4819b0 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.906+0000 7f68a5c00640 1 -- 172.21.15.6:0/464737913 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0035040 con 0x555a2c4819b0 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.907+0000 7f68bad69740 1 -- 172.21.15.6:0/464737913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 msgr2=0x555a2c498620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.907+0000 7f68bad69740 1 --2- 172.21.15.6:0/464737913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c498620 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f68a0005ab0 tx=0x7f68a0005e50 comp rx=0 tx=0).stop 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.907+0000 7f68bad69740 1 -- 172.21.15.6:0/464737913 shutdown_connections 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.907+0000 7f68bad69740 1 --2- 172.21.15.6:0/464737913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c48b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.907+0000 7f68bad69740 1 --2- 172.21.15.6:0/464737913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c4a4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.907+0000 7f68bad69740 1 --2- 172.21.15.6:0/464737913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c498620 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.907+0000 7f68bad69740 1 -- 172.21.15.6:0/464737913 >> 172.21.15.6:0/464737913 conn(0x555a2c47dec0 msgr2=0x555a2c48b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.907+0000 7f68bad69740 1 -- 172.21.15.6:0/464737913 shutdown_connections 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.907+0000 7f68bad69740 1 -- 172.21.15.6:0/464737913 wait complete. 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.907+0000 7f68bad69740 1 -- start start 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.908+0000 7f68bad69740 1 Processor -- start 2024-04-26T01:55:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.908+0000 7f68bad69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c4929c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.908+0000 7f68bad69740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c492f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.908+0000 7f68bad69740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c502520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.908+0000 7f68bad69740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f68b01ae480 con 0x555a2c4822d0 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.908+0000 7f68bad69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f68b01af8f0 con 0x555a2c4819b0 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.908+0000 7f68bad69740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f68b01aeb50 con 0x555a2c48cb90 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.908+0000 7f68bad69740 10 client.0 _ll_register_callbacks cb 0x7f689c0044d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.908+0000 7f68b4c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c492f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.908+0000 7f68b4c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c492f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41730/0 (socket says 172.21.15.6:41730) 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.908+0000 7f68b4c00640 1 -- 172.21.15.6:0/2527488995 learned_addr learned my addr 172.21.15.6:0/2527488995 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.909+0000 7f68b5600640 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c4929c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.909+0000 7f68b6000640 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c502520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.909+0000 7f68b4c00640 1 -- 172.21.15.6:0/2527488995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 msgr2=0x555a2c502520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.909+0000 7f68b4c00640 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c502520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.909+0000 7f68b4c00640 1 -- 172.21.15.6:0/2527488995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 msgr2=0x555a2c4929c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.909+0000 7f68b4c00640 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c4929c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.909+0000 7f68b4c00640 1 -- 172.21.15.6:0/2527488995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a2c502a60 con 0x555a2c4822d0 2024-04-26T01:55:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.909+0000 7f68b6000640 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c502520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.909+0000 7f68b4c00640 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c492f00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f689c01ea20 tx=0x7f689c003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfab9c21788aa8a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.909+0000 7f689aa00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:55:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.909+0000 7f689aa00640 1 -- 172.21.15.6:0/2527488995 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c03f060 con 0x555a2c4822d0 2024-04-26T01:55:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f68bad69740 1 -- 172.21.15.6:0/2527488995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555a2c3f5320 con 0x555a2c4822d0 2024-04-26T01:55:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f689aa00640 1 -- 172.21.15.6:0/2527488995 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f689c01e630 con 0x555a2c4822d0 2024-04-26T01:55:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f689aa00640 1 -- 172.21.15.6:0/2527488995 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f689c01e830 con 0x555a2c4822d0 2024-04-26T01:55:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f68bad69740 10 client.16251 fetch_fsmap learned FSMap version 43 2024-04-26T01:55:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f68bad69740 1 -- 172.21.15.6:0/2527488995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x555a2c503c40 con 0x555a2c4822d0 2024-04-26T01:55:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f689aa00640 1 -- 172.21.15.6:0/2527488995 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555a2c3f5320 con 0x555a2c4822d0 2024-04-26T01:55:53.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f689aa00640 1 -- 172.21.15.6:0/2527488995 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f689c004ad0 con 0x555a2c4822d0 2024-04-26T01:55:53.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f68bad69740 10 client.16251 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:55:53.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f68bad69740 10 client.16251 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:55:53.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f68bad69740 20 client.16251 populate_metadata read hostname 'smithi006' 2024-04-26T01:55:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f68bad69740 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a2c5043a0 0x555a2c5067c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f68bad69740 4 client.16251 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.910+0000 7f68bad69740 1 -- 172.21.15.6:0/2527488995 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555a2c3ef190 con 0x555a2c5043a0 2024-04-26T01:55:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.911+0000 7f68b5600640 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a2c5043a0 0x555a2c5067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.911+0000 7f68b5600640 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a2c5043a0 0x555a2c5067c0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cb5ca1952f8dd91 server_cookie=10aee9ff544db51f in_seq=0 out_seq=0 2024-04-26T01:55:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.911+0000 7f689aa00640 10 client.16251 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.938+0000 7f689aa00640 1 -- 172.21.15.6:0/2527488995 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555a2c3ef190 con 0x555a2c5043a0 2024-04-26T01:55:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.938+0000 7f689aa00640 10 client.16251 handle_command_reply: tid=0 2024-04-26T01:55:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.938+0000 7f68bad69740 1 client.16251 shutdown 2024-04-26T01:55:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.938+0000 7f68bad69740 1 -- 172.21.15.6:0/2527488995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 msgr2=0x555a2c492f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.938+0000 7f68bad69740 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c492f00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f689c01ea20 tx=0x7f689c003af0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.938+0000 7f68bad69740 1 -- 172.21.15.6:0/2527488995 shutdown_connections 2024-04-26T01:55:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.938+0000 7f68bad69740 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555a2c5043a0 0x555a2c5067c0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.939+0000 7f68bad69740 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555a2c4822d0 0x555a2c492f00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.939+0000 7f68bad69740 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555a2c48cb90 0x555a2c502520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.939+0000 7f68bad69740 1 --2- 172.21.15.6:0/2527488995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a2c4819b0 0x555a2c4929c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.939+0000 7f68bad69740 1 -- 172.21.15.6:0/2527488995 >> 172.21.15.6:0/2527488995 conn(0x555a2c47dec0 msgr2=0x555a2c48bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.939+0000 7f689aa00640 0 client.16251 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:55:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.939+0000 7f68bad69740 1 -- 172.21.15.6:0/2527488995 shutdown_connections 2024-04-26T01:55:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.939+0000 7f68bad69740 1 -- 172.21.15.6:0/2527488995 wait complete. 2024-04-26T01:55:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.939+0000 7f68bad69740 20 client.16251 trim_cache size 0 max 16384 2024-04-26T01:55:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:55:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:55:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:55:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:55:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.940+0000 7f68b6a00640 1 -- 172.21.15.6:0/1492515135 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68940a3d00 msgr2=0x7f68940a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.940+0000 7f68b6a00640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68940a3d00 0x7f68940a61c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x555a2c47f2e0 tx=0x7f689c005850 comp rx=0 tx=0).stop 2024-04-26T01:55:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.940+0000 7f68b6a00640 1 -- 172.21.15.6:0/1492515135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b010c3a0 msgr2=0x7f68b019a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.940+0000 7f68b6a00640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b010c3a0 0x7f68b019a5b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009fd0 tx=0x7f68a800b260 comp rx=0 tx=0).stop 2024-04-26T01:55:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.940+0000 7f68b6a00640 1 -- 172.21.15.6:0/1492515135 shutdown_connections 2024-04-26T01:55:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.940+0000 7f68b6a00640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f68940a3d00 0x7f68940a61c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.940+0000 7f68b6a00640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b010c3a0 0x7f68b019a5b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.940+0000 7f68b6a00640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68b0103850 0x7f68b01a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.940+0000 7f68b6a00640 1 --2- 172.21.15.6:0/1492515135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68b0102f30 0x7f68b01a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.940+0000 7f68b6a00640 1 -- 172.21.15.6:0/1492515135 >> 172.21.15.6:0/1492515135 conn(0x7f68b00fc790 msgr2=0x7f68b00fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.940+0000 7f68b6a00640 1 -- 172.21.15.6:0/1492515135 shutdown_connections 2024-04-26T01:55:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:55:53.940+0000 7f68b6a00640 1 -- 172.21.15.6:0/1492515135 wait complete. 2024-04-26T01:55:54.617 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.50 MB/sec execute 378 sec latency 15755.610 ms 2024-04-26T01:55:54.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:54 smithi188 ceph-mon[32751]: pgmap v314: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 591 KiB/s wr, 25 op/s 2024-04-26T01:55:54.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:54 smithi029 ceph-mon[32775]: pgmap v314: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 591 KiB/s wr, 25 op/s 2024-04-26T01:55:55.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:54 smithi006 ceph-mon[28419]: pgmap v314: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 591 KiB/s wr, 25 op/s 2024-04-26T01:55:55.617 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.49 MB/sec execute 379 sec latency 16755.756 ms 2024-04-26T01:55:56.617 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.49 MB/sec execute 380 sec latency 17755.873 ms 2024-04-26T01:55:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:56 smithi188 ceph-mon[32751]: pgmap v315: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 591 KiB/s wr, 25 op/s 2024-04-26T01:55:56.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:56 smithi029 ceph-mon[32775]: pgmap v315: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 591 KiB/s wr, 25 op/s 2024-04-26T01:55:57.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:56 smithi006 ceph-mon[28419]: pgmap v315: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 591 KiB/s wr, 25 op/s 2024-04-26T01:55:57.617 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.48 MB/sec execute 381 sec latency 18756.026 ms 2024-04-26T01:55:58.617 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.47 MB/sec execute 382 sec latency 19756.157 ms 2024-04-26T01:55:58.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:55:58 smithi188 ceph-mon[32751]: pgmap v316: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 632 KiB/s wr, 39 op/s 2024-04-26T01:55:58.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:55:58 smithi029 ceph-mon[32775]: pgmap v316: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 632 KiB/s wr, 39 op/s 2024-04-26T01:55:59.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:55:58 smithi006 ceph-mon[28419]: pgmap v316: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 632 KiB/s wr, 39 op/s 2024-04-26T01:55:59.617 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.46 MB/sec execute 383 sec latency 20756.327 ms 2024-04-26T01:56:00.617 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.45 MB/sec execute 384 sec latency 21756.422 ms 2024-04-26T01:56:00.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:00 smithi188 ceph-mon[32751]: pgmap v317: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 629 KiB/s wr, 35 op/s 2024-04-26T01:56:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:00 smithi029 ceph-mon[32775]: pgmap v317: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 629 KiB/s wr, 35 op/s 2024-04-26T01:56:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:00 smithi006 ceph-mon[28419]: pgmap v317: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 629 KiB/s wr, 35 op/s 2024-04-26T01:56:01.618 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.44 MB/sec execute 385 sec latency 22756.519 ms 2024-04-26T01:56:01.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:56:02.370 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:56:02.618 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.43 MB/sec execute 386 sec latency 23756.608 ms 2024-04-26T01:56:02.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:02 smithi006 ceph-mon[28419]: pgmap v318: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 684 KiB/s wr, 56 op/s 2024-04-26T01:56:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:02 smithi029 ceph-mon[32775]: pgmap v318: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 684 KiB/s wr, 56 op/s 2024-04-26T01:56:03.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:02 smithi188 ceph-mon[32751]: pgmap v318: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 684 KiB/s wr, 56 op/s 2024-04-26T01:56:03.618 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.42 MB/sec execute 387 sec latency 24756.729 ms 2024-04-26T01:56:04.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.610+0000 7f916cc00700 1 -- 172.21.15.6:0/3630768295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806b070 msgr2=0x7f916810efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:04.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.610+0000 7f916cc00700 1 --2- 172.21.15.6:0/3630768295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806b070 0x7f916810efb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f91540098f0 tx=0x7f9154009c00 comp rx=0 tx=0).stop 2024-04-26T01:56:04.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.611+0000 7f916cc00700 1 -- 172.21.15.6:0/3630768295 shutdown_connections 2024-04-26T01:56:04.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.611+0000 7f916cc00700 1 --2- 172.21.15.6:0/3630768295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f916810f4f0 0x7f91681118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:04.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.611+0000 7f916cc00700 1 --2- 172.21.15.6:0/3630768295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f916806a750 0x7f916806ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:04.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.611+0000 7f916cc00700 1 --2- 172.21.15.6:0/3630768295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806b070 0x7f916810efb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:04.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.611+0000 7f916cc00700 1 -- 172.21.15.6:0/3630768295 >> 172.21.15.6:0/3630768295 conn(0x7f91680788f0 msgr2=0x7f9168078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:04.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.611+0000 7f916cc00700 1 -- 172.21.15.6:0/3630768295 shutdown_connections 2024-04-26T01:56:04.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.611+0000 7f916cc00700 1 -- 172.21.15.6:0/3630768295 wait complete. 2024-04-26T01:56:04.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.612+0000 7f916cc00700 1 Processor -- start 2024-04-26T01:56:04.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.612+0000 7f916cc00700 1 -- start start 2024-04-26T01:56:04.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.612+0000 7f916cc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f916806a750 0x7f916819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:04.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f916cc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f916806b070 0x7f916819b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:04.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f916cc00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916810f4f0 0x7f916819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:04.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f916cc00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9168114e00 con 0x7f916806a750 2024-04-26T01:56:04.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f916cc00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9168114c80 con 0x7f916810f4f0 2024-04-26T01:56:04.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f916cc00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9168114f80 con 0x7f916806b070 2024-04-26T01:56:04.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f9165e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f916806a750 0x7f916819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:04.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f9165400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f916806b070 0x7f916819b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:04.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f9165400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f916806b070 0x7f916819b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:40302/0 (socket says 172.21.15.6:40302) 2024-04-26T01:56:04.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f9165400700 1 -- 172.21.15.6:0/2249119556 learned_addr learned my addr 172.21.15.6:0/2249119556 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:04.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f9166800700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916810f4f0 0x7f916819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:04.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f9165400700 1 -- 172.21.15.6:0/2249119556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916810f4f0 msgr2=0x7f916819fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:04.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f9165400700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916810f4f0 0x7f916819fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:04.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f9165400700 1 -- 172.21.15.6:0/2249119556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f916806a750 msgr2=0x7f916819b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:04.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f9165400700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f916806a750 0x7f916819b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.613+0000 7f9165400700 1 -- 172.21.15.6:0/2249119556 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91681a03a0 con 0x7f916806b070 2024-04-26T01:56:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.614+0000 7f9165400700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f916806b070 0x7f916819b8d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f915400f990 tx=0x7f915400f9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.614+0000 7f915ea00700 1 -- 172.21.15.6:0/2249119556 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9154027070 con 0x7f916806b070 2024-04-26T01:56:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.614+0000 7f916cc00700 1 -- 172.21.15.6:0/2249119556 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91681a0630 con 0x7f916806b070 2024-04-26T01:56:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.614+0000 7f916cc00700 1 -- 172.21.15.6:0/2249119556 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91681a7f10 con 0x7f916806b070 2024-04-26T01:56:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.614+0000 7f915ea00700 1 -- 172.21.15.6:0/2249119556 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9154005570 con 0x7f916806b070 2024-04-26T01:56:04.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.614+0000 7f915ea00700 1 -- 172.21.15.6:0/2249119556 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9154004930 con 0x7f916806b070 2024-04-26T01:56:04.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.615+0000 7f916cc00700 1 -- 172.21.15.6:0/2249119556 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9168069730 con 0x7f916806b070 2024-04-26T01:56:04.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.617+0000 7f915ea00700 1 -- 172.21.15.6:0/2249119556 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f915402a070 con 0x7f916806b070 2024-04-26T01:56:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.619+0000 7f915ea00700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f914c0a3e30 0x7f914c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.619+0000 7f915ea00700 1 -- 172.21.15.6:0/2249119556 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f91540d16c0 con 0x7f916806b070 2024-04-26T01:56:04.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.619+0000 7f9165e00700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f914c0a3e30 0x7f914c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:04.618 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.41 MB/sec execute 388 sec latency 25756.784 ms 2024-04-26T01:56:04.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.620+0000 7f9165e00700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f914c0a3e30 0x7f914c0a62f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9158009680 tx=0x7f9158009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:04.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.621+0000 7f915ea00700 1 -- 172.21.15.6:0/2249119556 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9154013030 con 0x7f916806b070 2024-04-26T01:56:04.706 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:04 smithi006 ceph-mon[28419]: pgmap v319: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 95 KiB/s wr, 35 op/s 2024-04-26T01:56:04.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.907+0000 7f916cc00700 1 -- 172.21.15.6:0/2249119556 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f916819c6f0 con 0x7f916806b070 2024-04-26T01:56:04.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.908+0000 7f915ea00700 1 -- 172.21.15.6:0/2249119556 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11850 (secure 0 0 0) 0x7f915409de60 con 0x7f916806b070 2024-04-26T01:56:04.907 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:56:04.907 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":43,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:55:18.912671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:56:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.911+0000 7f916cc00700 1 -- 172.21.15.6:0/2249119556 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f914c0a3e30 msgr2=0x7f914c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.911+0000 7f916cc00700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f914c0a3e30 0x7f914c0a62f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9158009680 tx=0x7f9158009260 comp rx=0 tx=0).stop 2024-04-26T01:56:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.911+0000 7f916cc00700 1 -- 172.21.15.6:0/2249119556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f916806b070 msgr2=0x7f916819b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.911+0000 7f916cc00700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f916806b070 0x7f916819b8d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f915400f990 tx=0x7f915400f9c0 comp rx=0 tx=0).stop 2024-04-26T01:56:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.911+0000 7f916cc00700 1 -- 172.21.15.6:0/2249119556 shutdown_connections 2024-04-26T01:56:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.911+0000 7f916cc00700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f914c0a3e30 0x7f914c0a62f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.911+0000 7f916cc00700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916810f4f0 0x7f916819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.911+0000 7f916cc00700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f916806a750 0x7f916819b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.911+0000 7f916cc00700 1 --2- 172.21.15.6:0/2249119556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f916806b070 0x7f916819b8d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.912+0000 7f916cc00700 1 -- 172.21.15.6:0/2249119556 >> 172.21.15.6:0/2249119556 conn(0x7f91680788f0 msgr2=0x7f91681109e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.912+0000 7f916cc00700 1 -- 172.21.15.6:0/2249119556 shutdown_connections 2024-04-26T01:56:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:04.912+0000 7f916cc00700 1 -- 172.21.15.6:0/2249119556 wait complete. 2024-04-26T01:56:04.911 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-26T01:56:04.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:04 smithi029 ceph-mon[32775]: pgmap v319: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 95 KiB/s wr, 35 op/s 2024-04-26T01:56:05.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:04 smithi188 ceph-mon[32751]: pgmap v319: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 95 KiB/s wr, 35 op/s 2024-04-26T01:56:05.618 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.40 MB/sec execute 389 sec latency 26756.924 ms 2024-04-26T01:56:05.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:05 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2249119556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:56:05.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:05 smithi006 ceph-mon[28419]: pgmap v320: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 95 KiB/s wr, 35 op/s 2024-04-26T01:56:05.848 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id eeff187f --release --await' 2024-04-26T01:56:05.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id eeff187f --release --await 2024-04-26T01:56:05.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:05 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2249119556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:56:05.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:05 smithi029 ceph-mon[32775]: pgmap v320: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 95 KiB/s wr, 35 op/s 2024-04-26T01:56:06.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:05 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2249119556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:56:06.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:05 smithi188 ceph-mon[32751]: pgmap v320: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 95 KiB/s wr, 35 op/s 2024-04-26T01:56:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:56:06.618 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.40 MB/sec execute 390 sec latency 27757.076 ms 2024-04-26T01:56:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.086+0000 7f610cc00640 1 -- 172.21.15.6:0/469263034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6108076da0 msgr2=0x7f61081128d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.086+0000 7f610cc00640 1 --2- 172.21.15.6:0/469263034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6108076da0 0x7f61081128d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f60f4009940 tx=0x7f60f402a420 comp rx=0 tx=0).stop 2024-04-26T01:56:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.086+0000 7f610cc00640 1 -- 172.21.15.6:0/469263034 shutdown_connections 2024-04-26T01:56:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.087+0000 7f610cc00640 1 --2- 172.21.15.6:0/469263034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6108112e10 0x7f6108115200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.087+0000 7f610cc00640 1 --2- 172.21.15.6:0/469263034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6108076da0 0x7f61081128d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.087+0000 7f610cc00640 1 --2- 172.21.15.6:0/469263034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6108076480 0x7f6108076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.087+0000 7f610cc00640 1 -- 172.21.15.6:0/469263034 >> 172.21.15.6:0/469263034 conn(0x7f61080ffc00 msgr2=0x7f6108102020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.087+0000 7f610cc00640 1 -- 172.21.15.6:0/469263034 shutdown_connections 2024-04-26T01:56:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.087+0000 7f610cc00640 1 -- 172.21.15.6:0/469263034 wait complete. 2024-04-26T01:56:07.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.088+0000 7f610cc00640 1 Processor -- start 2024-04-26T01:56:07.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.088+0000 7f610cc00640 1 -- start start 2024-04-26T01:56:07.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.088+0000 7f610cc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6108076480 0x7f610819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.088+0000 7f610cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6108076da0 0x7f610819f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.088+0000 7f610cc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6108112e10 0x7f61081a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.088+0000 7f610cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6108118810 con 0x7f6108076da0 2024-04-26T01:56:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.088+0000 7f610cc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6108118690 con 0x7f6108112e10 2024-04-26T01:56:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.088+0000 7f610cc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6108118990 con 0x7f6108076480 2024-04-26T01:56:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.088+0000 7f6105400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6108076da0 0x7f610819f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.089+0000 7f6105400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6108076da0 0x7f610819f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45326/0 (socket says 172.21.15.6:45326) 2024-04-26T01:56:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.089+0000 7f6106800640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6108112e10 0x7f61081a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.089+0000 7f6105400640 1 -- 172.21.15.6:0/1001064331 learned_addr learned my addr 172.21.15.6:0/1001064331 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.089+0000 7f6105e00640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6108076480 0x7f610819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.089+0000 7f6106800640 1 -- 172.21.15.6:0/1001064331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6108076480 msgr2=0x7f610819ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:07.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.089+0000 7f6106800640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6108076480 0x7f610819ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.089+0000 7f6106800640 1 -- 172.21.15.6:0/1001064331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6108076da0 msgr2=0x7f610819f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:07.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.089+0000 7f6106800640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6108076da0 0x7f610819f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.089+0000 7f6106800640 1 -- 172.21.15.6:0/1001064331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61081a3db0 con 0x7f6108112e10 2024-04-26T01:56:07.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.089+0000 7f6105400640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6108076da0 0x7f610819f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:07.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.089+0000 7f6106800640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6108112e10 0x7f61081a3680 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6100005fd0 tx=0x7f610000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.090+0000 7f60fea00640 1 -- 172.21.15.6:0/1001064331 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610000cdf0 con 0x7f6108112e10 2024-04-26T01:56:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.090+0000 7f60fea00640 1 -- 172.21.15.6:0/1001064331 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6100005020 con 0x7f6108112e10 2024-04-26T01:56:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.090+0000 7f610cc00640 1 -- 172.21.15.6:0/1001064331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61081a40a0 con 0x7f6108112e10 2024-04-26T01:56:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.090+0000 7f60fea00640 1 -- 172.21.15.6:0/1001064331 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6100010690 con 0x7f6108112e10 2024-04-26T01:56:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.090+0000 7f610cc00640 1 -- 172.21.15.6:0/1001064331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61081ab930 con 0x7f6108112e10 2024-04-26T01:56:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.093+0000 7f610fa2d740 1 Processor -- start 2024-04-26T01:56:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.093+0000 7f60fea00640 1 -- 172.21.15.6:0/1001064331 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f6100005460 con 0x7f6108112e10 2024-04-26T01:56:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.093+0000 7f610fa2d740 1 -- start start 2024-04-26T01:56:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.094+0000 7f610fa2d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98269820 0x55fc98269c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.094+0000 7f610fa2d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc982744c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.094+0000 7f610fa2d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc98276df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.094+0000 7f610fa2d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55fc981cf900 con 0x55fc98274a00 2024-04-26T01:56:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.094+0000 7f610fa2d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55fc98175ef0 con 0x55fc98269820 2024-04-26T01:56:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.094+0000 7f610fa2d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fc981cb8c0 con 0x55fc9826a140 2024-04-26T01:56:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.094+0000 7f6105400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98269820 0x55fc98269c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.094+0000 7f6105400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98269820 0x55fc98269c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:45378/0 (socket says 172.21.15.6:45378) 2024-04-26T01:56:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.094+0000 7f6105400640 1 -- 172.21.15.6:0/3479890124 learned_addr learned my addr 172.21.15.6:0/3479890124 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.094+0000 7f6105e00640 1 --2- 172.21.15.6:0/3479890124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc982744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.094+0000 7f6105400640 1 --2- 172.21.15.6:0/3479890124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc98276df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.095+0000 7f6105e00640 1 -- 172.21.15.6:0/3479890124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98269820 msgr2=0x55fc98269c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.095+0000 7f6105e00640 1 --2- 172.21.15.6:0/3479890124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98269820 0x55fc98269c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.095+0000 7f6105e00640 1 -- 172.21.15.6:0/3479890124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 msgr2=0x55fc98276df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.095+0000 7f6105e00640 1 --2- 172.21.15.6:0/3479890124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc98276df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.095+0000 7f6105e00640 1 -- 172.21.15.6:0/3479890124 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc97ee2210 con 0x55fc9826a140 2024-04-26T01:56:07.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.095+0000 7f6105e00640 1 --2- 172.21.15.6:0/3479890124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc982744c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009c20 tx=0x7f60f8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b0a65dd83231e19 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:07.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.095+0000 7f60fea00640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f60ec0a3d10 0x7f60ec0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.095+0000 7f60f3000640 1 -- 172.21.15.6:0/3479890124 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8004050 con 0x55fc9826a140 2024-04-26T01:56:07.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.095+0000 7f60f3000640 1 -- 172.21.15.6:0/3479890124 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60f80054d0 con 0x55fc9826a140 2024-04-26T01:56:07.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.095+0000 7f60f3000640 1 -- 172.21.15.6:0/3479890124 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8004c80 con 0x55fc9826a140 2024-04-26T01:56:07.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.095+0000 7f60fea00640 1 -- 172.21.15.6:0/1001064331 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f61000d0f60 con 0x7f6108112e10 2024-04-26T01:56:07.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.096+0000 7f6105400640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f60ec0a3d10 0x7f60ec0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.096+0000 7f610fa2d740 1 -- 172.21.15.6:0/3479890124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 msgr2=0x55fc982744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:07.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.096+0000 7f610fa2d740 1 --2- 172.21.15.6:0/3479890124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc982744c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009c20 tx=0x7f60f8011810 comp rx=0 tx=0).stop 2024-04-26T01:56:07.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.096+0000 7f6105400640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f60ec0a3d10 0x7f60ec0a61d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f60f4009660 tx=0x7f60f40059c0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.096+0000 7f610fa2d740 1 -- 172.21.15.6:0/3479890124 shutdown_connections 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.096+0000 7f610fa2d740 1 --2- 172.21.15.6:0/3479890124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc98276df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.096+0000 7f610fa2d740 1 --2- 172.21.15.6:0/3479890124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc982744c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.096+0000 7f610fa2d740 1 --2- 172.21.15.6:0/3479890124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98269820 0x55fc98269c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.096+0000 7f610fa2d740 1 -- 172.21.15.6:0/3479890124 >> 172.21.15.6:0/3479890124 conn(0x55fc98265d30 msgr2=0x55fc98266110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.096+0000 7f610fa2d740 1 -- 172.21.15.6:0/3479890124 shutdown_connections 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.096+0000 7f610fa2d740 1 -- 172.21.15.6:0/3479890124 wait complete. 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.097+0000 7f610fa2d740 1 -- start start 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.097+0000 7f610fa2d740 1 Processor -- start 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f610fa2d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98269820 0x55fc98271860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f610fa2d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc9826a140 0x55fc98271da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f610fa2d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc982722e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f610fa2d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55fc98277630 con 0x55fc98274a00 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f610fa2d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55fc98189310 con 0x55fc9826a140 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f610fa2d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fc982777b0 con 0x55fc98269820 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f610fa2d740 10 client.0 _ll_register_callbacks cb 0x55fc98157cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f6106800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc982722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f6105e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98269820 0x55fc98271860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f6106800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc982722e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45332/0 (socket says 172.21.15.6:45332) 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f6105400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc9826a140 0x55fc98271da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f6106800640 1 -- 172.21.15.6:0/2799226805 learned_addr learned my addr 172.21.15.6:0/2799226805 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f6105e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98269820 0x55fc98271860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:40340/0 (socket says 172.21.15.6:40340) 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f6106800640 1 -- 172.21.15.6:0/2799226805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98269820 msgr2=0x55fc98271860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f6106800640 1 --2- 172.21.15.6:0/2799226805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98269820 0x55fc98271860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f6106800640 1 -- 172.21.15.6:0/2799226805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc9826a140 msgr2=0x55fc98271da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f6106800640 1 --2- 172.21.15.6:0/2799226805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc9826a140 0x55fc98271da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.098+0000 7f6106800640 1 -- 172.21.15.6:0/2799226805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc97fae620 con 0x55fc98274a00 2024-04-26T01:56:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.099+0000 7f6106800640 1 --2- 172.21.15.6:0/2799226805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc982722e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6100099820 tx=0x7f610009ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b432a0aa0175ed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.099+0000 7f60e7400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:56:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.099+0000 7f60e7400640 1 -- 172.21.15.6:0/2799226805 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61000a5ac0 con 0x55fc98274a00 2024-04-26T01:56:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.099+0000 7f60e7400640 1 -- 172.21.15.6:0/2799226805 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f610009b150 con 0x55fc98274a00 2024-04-26T01:56:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.099+0000 7f610fa2d740 1 -- 172.21.15.6:0/2799226805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60f80054d0 con 0x55fc98274a00 2024-04-26T01:56:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.099+0000 7f60e7400640 1 -- 172.21.15.6:0/2799226805 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f61000b6260 con 0x55fc98274a00 2024-04-26T01:56:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.099+0000 7f610fa2d740 10 client.16269 fetch_fsmap learned FSMap version 43 2024-04-26T01:56:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.099+0000 7f610fa2d740 1 -- 172.21.15.6:0/2799226805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55fc98283830 con 0x55fc98274a00 2024-04-26T01:56:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.100+0000 7f60e7400640 1 -- 172.21.15.6:0/2799226805 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60f80054d0 con 0x55fc98274a00 2024-04-26T01:56:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.100+0000 7f60e7400640 1 -- 172.21.15.6:0/2799226805 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f6100098ba0 con 0x55fc98274a00 2024-04-26T01:56:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.100+0000 7f610fa2d740 10 client.16269 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:56:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.100+0000 7f610fa2d740 10 client.16269 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.100+0000 7f610fa2d740 20 client.16269 populate_metadata read hostname 'smithi006' 2024-04-26T01:56:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.100+0000 7f610fa2d740 1 --2- 172.21.15.6:0/2799226805 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fc98284cf0 0x55fc982870e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.100+0000 7f610fa2d740 4 client.16269 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.100+0000 7f610fa2d740 1 -- 172.21.15.6:0/2799226805 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc981e38b0 con 0x55fc98284cf0 2024-04-26T01:56:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.100+0000 7f6105e00640 1 --2- 172.21.15.6:0/2799226805 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fc98284cf0 0x55fc982870e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.101+0000 7f6105e00640 1 --2- 172.21.15.6:0/2799226805 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fc98284cf0 0x55fc982870e0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=608b34a1a42d8675 server_cookie=d4fc3b2a23330319 in_seq=0 out_seq=0 2024-04-26T01:56:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.101+0000 7f60e7400640 10 client.16269 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.102+0000 7f60e7400640 1 -- 172.21.15.6:0/2799226805 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc981e38b0 con 0x55fc98284cf0 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.102+0000 7f60e7400640 10 client.16269 handle_command_reply: tid=0 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.102+0000 7f610fa2d740 1 client.16269 shutdown 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.102+0000 7f610fa2d740 1 -- 172.21.15.6:0/2799226805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 msgr2=0x55fc982722e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.102+0000 7f610fa2d740 1 --2- 172.21.15.6:0/2799226805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc982722e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6100099820 tx=0x7f610009ce50 comp rx=0 tx=0).stop 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.102+0000 7f610fa2d740 1 -- 172.21.15.6:0/2799226805 shutdown_connections 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.102+0000 7f610fa2d740 1 --2- 172.21.15.6:0/2799226805 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fc98284cf0 0x55fc982870e0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.102+0000 7f610fa2d740 1 --2- 172.21.15.6:0/2799226805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc982722e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.102+0000 7f610fa2d740 1 --2- 172.21.15.6:0/2799226805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc9826a140 0x55fc98271da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.102+0000 7f610fa2d740 1 --2- 172.21.15.6:0/2799226805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc98269820 0x55fc98271860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.102+0000 7f60e7400640 0 client.16269 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.102+0000 7f610fa2d740 1 -- 172.21.15.6:0/2799226805 >> 172.21.15.6:0/2799226805 conn(0x55fc98265d30 msgr2=0x55fc9826ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.103+0000 7f610fa2d740 1 -- 172.21.15.6:0/2799226805 shutdown_connections 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.103+0000 7f610fa2d740 1 -- 172.21.15.6:0/2799226805 wait complete. 2024-04-26T01:56:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.103+0000 7f610fa2d740 20 client.16269 trim_cache size 0 max 16384 2024-04-26T01:56:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.119+0000 7f610fa2d740 1 Processor -- start 2024-04-26T01:56:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.119+0000 7f610fa2d740 1 -- start start 2024-04-26T01:56:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.119+0000 7f610fa2d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98269820 0x55fc98280520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.119+0000 7f610fa2d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc98270840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.119+0000 7f610fa2d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc9828c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.119+0000 7f610fa2d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f61081b6c80 con 0x55fc98274a00 2024-04-26T01:56:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.119+0000 7f610fa2d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f61081b4550 con 0x55fc98269820 2024-04-26T01:56:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.119+0000 7f610fa2d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f61081b23c0 con 0x55fc9826a140 2024-04-26T01:56:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.120+0000 7f6106800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc9828c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.120+0000 7f6106800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc9828c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45338/0 (socket says 172.21.15.6:45338) 2024-04-26T01:56:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.120+0000 7f6106800640 1 -- 172.21.15.6:0/666845086 learned_addr learned my addr 172.21.15.6:0/666845086 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.120+0000 7f6105e00640 1 --2- 172.21.15.6:0/666845086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98269820 0x55fc98280520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.120+0000 7f6105400640 1 --2- 172.21.15.6:0/666845086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc98270840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.121+0000 7f6106800640 1 -- 172.21.15.6:0/666845086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 msgr2=0x55fc98270840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.121+0000 7f6106800640 1 --2- 172.21.15.6:0/666845086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc98270840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.121+0000 7f6106800640 1 -- 172.21.15.6:0/666845086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98269820 msgr2=0x55fc98280520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.121+0000 7f6106800640 1 --2- 172.21.15.6:0/666845086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98269820 0x55fc98280520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.121+0000 7f6106800640 1 -- 172.21.15.6:0/666845086 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc98269f70 con 0x55fc98274a00 2024-04-26T01:56:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.121+0000 7f6105e00640 1 --2- 172.21.15.6:0/666845086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98269820 0x55fc98280520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.121+0000 7f6106800640 1 --2- 172.21.15.6:0/666845086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc9828c9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f610009cde0 tx=0x7f610009c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f2abb414051e41 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.121+0000 7f60f2600640 1 -- 172.21.15.6:0/666845086 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61000ba020 con 0x55fc98274a00 2024-04-26T01:56:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.121+0000 7f60f2600640 1 -- 172.21.15.6:0/666845086 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f610009f030 con 0x55fc98274a00 2024-04-26T01:56:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.121+0000 7f60f2600640 1 -- 172.21.15.6:0/666845086 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610009bd10 con 0x55fc98274a00 2024-04-26T01:56:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.122+0000 7f610fa2d740 1 -- 172.21.15.6:0/666845086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 msgr2=0x55fc9828c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.122+0000 7f610fa2d740 1 --2- 172.21.15.6:0/666845086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc9828c9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f610009cde0 tx=0x7f610009c7b0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.122+0000 7f610fa2d740 1 -- 172.21.15.6:0/666845086 shutdown_connections 2024-04-26T01:56:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.122+0000 7f610fa2d740 1 --2- 172.21.15.6:0/666845086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98274a00 0x55fc9828c9a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.122+0000 7f610fa2d740 1 --2- 172.21.15.6:0/666845086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc98270840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.122+0000 7f610fa2d740 1 --2- 172.21.15.6:0/666845086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98269820 0x55fc98280520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.122+0000 7f610fa2d740 1 -- 172.21.15.6:0/666845086 >> 172.21.15.6:0/666845086 conn(0x55fc98265d30 msgr2=0x55fc98270130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.122+0000 7f610fa2d740 1 -- 172.21.15.6:0/666845086 shutdown_connections 2024-04-26T01:56:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.123+0000 7f610fa2d740 1 -- 172.21.15.6:0/666845086 wait complete. 2024-04-26T01:56:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.123+0000 7f610fa2d740 1 -- start start 2024-04-26T01:56:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.123+0000 7f610fa2d740 1 Processor -- start 2024-04-26T01:56:07.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f610fa2d740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98269820 0x55fc98277c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f610fa2d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc98278140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f610fa2d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98274a00 0x55fc982e9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f610fa2d740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f61081b2a10 con 0x55fc98269820 2024-04-26T01:56:07.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f610fa2d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f61081b3e80 con 0x55fc98274a00 2024-04-26T01:56:07.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f610fa2d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f61081b30e0 con 0x55fc9826a140 2024-04-26T01:56:07.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f610fa2d740 10 client.0 _ll_register_callbacks cb 0x7f610009d550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:07.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f6105e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98269820 0x55fc98277c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f6105e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98269820 0x55fc98277c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45350/0 (socket says 172.21.15.6:45350) 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f6105e00640 1 -- 172.21.15.6:0/3232288067 learned_addr learned my addr 172.21.15.6:0/3232288067 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f6105400640 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc98278140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f6106800640 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98274a00 0x55fc982e9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f6105e00640 1 -- 172.21.15.6:0/3232288067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 msgr2=0x55fc98278140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f6105e00640 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc98278140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f6105e00640 1 -- 172.21.15.6:0/3232288067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98274a00 msgr2=0x55fc982e9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.124+0000 7f6105e00640 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98274a00 0x55fc982e9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.125+0000 7f6105e00640 1 -- 172.21.15.6:0/3232288067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc982e9770 con 0x55fc98269820 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.125+0000 7f6106800640 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98274a00 0x55fc982e9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.125+0000 7f6105e00640 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98269820 0x55fc98277c00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f60f8014040 tx=0x7f60f8009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906f68bac2cf1bc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.125+0000 7f60e7400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.125+0000 7f60e7400640 1 -- 172.21.15.6:0/3232288067 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f802e3a0 con 0x55fc98269820 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.125+0000 7f60e7400640 1 -- 172.21.15.6:0/3232288067 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f60f802e540 con 0x55fc98269820 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.125+0000 7f610fa2d740 1 -- 172.21.15.6:0/3232288067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f610009f030 con 0x55fc98269820 2024-04-26T01:56:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.125+0000 7f60e7400640 1 -- 172.21.15.6:0/3232288067 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f60f803e050 con 0x55fc98269820 2024-04-26T01:56:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.126+0000 7f610fa2d740 10 client.16275 fetch_fsmap learned FSMap version 43 2024-04-26T01:56:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.126+0000 7f610fa2d740 1 -- 172.21.15.6:0/3232288067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55fc982eaa60 con 0x55fc98269820 2024-04-26T01:56:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.126+0000 7f60e7400640 1 -- 172.21.15.6:0/3232288067 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f610009f030 con 0x55fc98269820 2024-04-26T01:56:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.126+0000 7f60e7400640 1 -- 172.21.15.6:0/3232288067 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f60f8004840 con 0x55fc98269820 2024-04-26T01:56:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.126+0000 7f610fa2d740 10 client.16275 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:56:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.126+0000 7f610fa2d740 10 client.16275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.126+0000 7f610fa2d740 20 client.16275 populate_metadata read hostname 'smithi006' 2024-04-26T01:56:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.126+0000 7f610fa2d740 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fc982eaff0 0x55fc982ed410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.126+0000 7f610fa2d740 4 client.16275 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.126+0000 7f610fa2d740 1 -- 172.21.15.6:0/3232288067 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fc981dd320 con 0x55fc982eaff0 2024-04-26T01:56:07.126 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.126+0000 7f6105400640 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fc982eaff0 0x55fc982ed410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.127+0000 7f6105400640 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fc982eaff0 0x55fc982ed410 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbfce882e219a099 server_cookie=d1198b6d83bebd20 in_seq=0 out_seq=0 2024-04-26T01:56:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.127+0000 7f60e7400640 10 client.16275 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.148+0000 7f60e7400640 1 -- 172.21.15.6:0/3232288067 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fc981dd320 con 0x55fc982eaff0 2024-04-26T01:56:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.148+0000 7f60e7400640 10 client.16275 handle_command_reply: tid=0 2024-04-26T01:56:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.148+0000 7f610fa2d740 1 client.16275 shutdown 2024-04-26T01:56:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.149+0000 7f610fa2d740 1 -- 172.21.15.6:0/3232288067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98269820 msgr2=0x55fc98277c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.149+0000 7f610fa2d740 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98269820 0x55fc98277c00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f60f8014040 tx=0x7f60f8009f30 comp rx=0 tx=0).stop 2024-04-26T01:56:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.149+0000 7f610fa2d740 1 -- 172.21.15.6:0/3232288067 shutdown_connections 2024-04-26T01:56:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.149+0000 7f610fa2d740 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fc982eaff0 0x55fc982ed410 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.149+0000 7f610fa2d740 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fc98274a00 0x55fc982e9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.149+0000 7f610fa2d740 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc9826a140 0x55fc98278140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.149+0000 7f610fa2d740 1 --2- 172.21.15.6:0/3232288067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fc98269820 0x55fc98277c00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.149+0000 7f610fa2d740 1 -- 172.21.15.6:0/3232288067 >> 172.21.15.6:0/3232288067 conn(0x55fc98265d30 msgr2=0x55fc9826ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.149+0000 7f60e7400640 0 client.16275 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.149+0000 7f610fa2d740 1 -- 172.21.15.6:0/3232288067 shutdown_connections 2024-04-26T01:56:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.149+0000 7f610fa2d740 1 -- 172.21.15.6:0/3232288067 wait complete. 2024-04-26T01:56:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.149+0000 7f610fa2d740 20 client.16275 trim_cache size 0 max 16384 2024-04-26T01:56:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:56:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:56:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:56:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:56:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.150+0000 7f610cc00640 1 -- 172.21.15.6:0/1001064331 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f60ec0a3d10 msgr2=0x7f60ec0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.150+0000 7f610cc00640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f60ec0a3d10 0x7f60ec0a61d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f60f4009660 tx=0x7f60f40059c0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.150+0000 7f610cc00640 1 -- 172.21.15.6:0/1001064331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6108112e10 msgr2=0x7f61081a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.150+0000 7f610cc00640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6108112e10 0x7f61081a3680 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6100005fd0 tx=0x7f610000ef10 comp rx=0 tx=0).stop 2024-04-26T01:56:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.150+0000 7f610cc00640 1 -- 172.21.15.6:0/1001064331 shutdown_connections 2024-04-26T01:56:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.150+0000 7f610cc00640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f60ec0a3d10 0x7f60ec0a61d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.150+0000 7f610cc00640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6108076da0 0x7f610819f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.150+0000 7f610cc00640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6108112e10 0x7f61081a3680 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.150+0000 7f610cc00640 1 --2- 172.21.15.6:0/1001064331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6108076480 0x7f610819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.150+0000 7f610cc00640 1 -- 172.21.15.6:0/1001064331 >> 172.21.15.6:0/1001064331 conn(0x7f61080ffc00 msgr2=0x7f6108114610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.151+0000 7f610cc00640 1 -- 172.21.15.6:0/1001064331 shutdown_connections 2024-04-26T01:56:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:07.151+0000 7f610cc00640 1 -- 172.21.15.6:0/1001064331 wait complete. 2024-04-26T01:56:07.618 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.39 MB/sec execute 391 sec latency 28757.207 ms 2024-04-26T01:56:08.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:08 smithi006 ceph-mon[28419]: pgmap v321: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 96 KiB/s wr, 38 op/s 2024-04-26T01:56:08.618 INFO:tasks.workunit.client.0.smithi006.stdout: 1 338742 3.38 MB/sec execute 392 sec latency 29757.343 ms 2024-04-26T01:56:08.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:08 smithi188 ceph-mon[32751]: pgmap v321: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 96 KiB/s wr, 38 op/s 2024-04-26T01:56:08.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:08 smithi029 ceph-mon[32775]: pgmap v321: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 96 KiB/s wr, 38 op/s 2024-04-26T01:56:08.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.788+0000 7f1682c00700 1 -- 172.21.15.6:0/1099385931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f167c102af0 msgr2=0x7f167c102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:08.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.788+0000 7f1682c00700 1 --2- 172.21.15.6:0/1099385931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f167c102af0 0x7f167c102f70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f16680099c0 tx=0x7f1668009cd0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.789+0000 7f1682c00700 1 -- 172.21.15.6:0/1099385931 shutdown_connections 2024-04-26T01:56:08.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.789+0000 7f1682c00700 1 --2- 172.21.15.6:0/1099385931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f167c1034b0 0x7f167c10f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.789+0000 7f1682c00700 1 --2- 172.21.15.6:0/1099385931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f167c107940 0x7f167c107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.789+0000 7f1682c00700 1 --2- 172.21.15.6:0/1099385931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f167c102af0 0x7f167c102f70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.789+0000 7f1682c00700 1 -- 172.21.15.6:0/1099385931 >> 172.21.15.6:0/1099385931 conn(0x7f167c079ba0 msgr2=0x7f167c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:08.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.789+0000 7f1682c00700 1 -- 172.21.15.6:0/1099385931 shutdown_connections 2024-04-26T01:56:08.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.789+0000 7f1682c00700 1 -- 172.21.15.6:0/1099385931 wait complete. 2024-04-26T01:56:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.790+0000 7f1682c00700 1 Processor -- start 2024-04-26T01:56:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.790+0000 7f1682c00700 1 -- start start 2024-04-26T01:56:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.790+0000 7f1682c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f167c102af0 0x7f167c1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.790+0000 7f1682c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f167c1034b0 0x7f167c1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.790+0000 7f1682c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f167c107940 0x7f167c19da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.790+0000 7f1682c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f167c112c50 con 0x7f167c107940 2024-04-26T01:56:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.790+0000 7f1682c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f167c112ad0 con 0x7f167c102af0 2024-04-26T01:56:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.790+0000 7f1682c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f167c112dd0 con 0x7f167c1034b0 2024-04-26T01:56:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.791+0000 7f167be00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f167c102af0 0x7f167c1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.791+0000 7f167be00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f167c102af0 0x7f167c1991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:51882/0 (socket says 172.21.15.6:51882) 2024-04-26T01:56:08.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.791+0000 7f167be00700 1 -- 172.21.15.6:0/3885858069 learned_addr learned my addr 172.21.15.6:0/3885858069 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:08.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.791+0000 7f1680c00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f167c107940 0x7f167c19da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.791+0000 7f167b400700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f167c1034b0 0x7f167c1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.791+0000 7f167be00700 1 -- 172.21.15.6:0/3885858069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f167c1034b0 msgr2=0x7f167c1996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:08.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.791+0000 7f167be00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f167c1034b0 0x7f167c1996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.791+0000 7f167be00700 1 -- 172.21.15.6:0/3885858069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f167c107940 msgr2=0x7f167c19da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:08.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.791+0000 7f167be00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f167c107940 0x7f167c19da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.791+0000 7f167be00700 1 -- 172.21.15.6:0/3885858069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c19e1b0 con 0x7f167c102af0 2024-04-26T01:56:08.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.792+0000 7f167be00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f167c102af0 0x7f167c1991a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f166c00ef30 tx=0x7f166c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:08.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.792+0000 7f1678c00700 1 -- 172.21.15.6:0/3885858069 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c0092e0 con 0x7f167c102af0 2024-04-26T01:56:08.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.792+0000 7f1682c00700 1 -- 172.21.15.6:0/3885858069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167c19e440 con 0x7f167c102af0 2024-04-26T01:56:08.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.792+0000 7f1682c00700 1 -- 172.21.15.6:0/3885858069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f167c1a5d20 con 0x7f167c102af0 2024-04-26T01:56:08.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.792+0000 7f1678c00700 1 -- 172.21.15.6:0/3885858069 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f166c00ce70 con 0x7f167c102af0 2024-04-26T01:56:08.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.792+0000 7f1678c00700 1 -- 172.21.15.6:0/3885858069 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c0079d0 con 0x7f167c102af0 2024-04-26T01:56:08.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.796+0000 7f1678c00700 1 -- 172.21.15.6:0/3885858069 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f166c01b030 con 0x7f167c102af0 2024-04-26T01:56:08.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.797+0000 7f168d579b80 1 Processor -- start 2024-04-26T01:56:08.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.797+0000 7f168d579b80 1 -- start start 2024-04-26T01:56:08.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.797+0000 7f168d579b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b093d40 0x56429b094120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.797+0000 7f168d579b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 0x56429b0a33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.797+0000 7f168d579b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b0a38e0 0x56429b0a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.797+0000 7f168d579b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56429afe6f90 con 0x56429b094660 2024-04-26T01:56:08.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.797+0000 7f168d579b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56429af8d830 con 0x56429b0a38e0 2024-04-26T01:56:08.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.797+0000 7f168d579b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56429afe2f50 con 0x56429b093d40 2024-04-26T01:56:08.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.798+0000 7f167be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 0x56429b0a33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.798+0000 7f1678c00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f16600a3db0 0x7f16600a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.798+0000 7f1680c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b093d40 0x56429b094120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.798+0000 7f167b400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b0a38e0 0x56429b0a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.798+0000 7f167b400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b0a38e0 0x56429b0a5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:51888/0 (socket says 172.21.15.6:51888) 2024-04-26T01:56:08.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.798+0000 7f1680c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b093d40 0x56429b094120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51506/0 (socket says 172.21.15.6:51506) 2024-04-26T01:56:08.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.798+0000 7f167b400700 1 -- 172.21.15.6:0/2717617240 learned_addr learned my addr 172.21.15.6:0/2717617240 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:08.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.798+0000 7f1678c00700 1 -- 172.21.15.6:0/3885858069 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f166c0d0390 con 0x7f167c102af0 2024-04-26T01:56:08.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.798+0000 7f167b400700 1 -- 172.21.15.6:0/2717617240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b093d40 msgr2=0x56429b094120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:08.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.798+0000 7f167b400700 1 --2- 172.21.15.6:0/2717617240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b093d40 0x56429b094120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.798+0000 7f167b400700 1 -- 172.21.15.6:0/2717617240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 msgr2=0x56429b0a33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:08.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.799+0000 7f167b400700 1 --2- 172.21.15.6:0/2717617240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 0x56429b0a33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.799+0000 7f167b400700 1 -- 172.21.15.6:0/2717617240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56429af83780 con 0x56429b0a38e0 2024-04-26T01:56:08.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.799+0000 7f1680c00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f16600a3db0 0x7f16600a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.799+0000 7f167be00700 1 --2- 172.21.15.6:0/2717617240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 0x56429b0a33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:08.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.799+0000 7f167b400700 1 --2- 172.21.15.6:0/2717617240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b0a38e0 0x56429b0a5cd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f16680099c0 tx=0x7f1668005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0b0689458860173 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:08.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.799+0000 7f1671600700 1 -- 172.21.15.6:0/2717617240 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1668028070 con 0x56429b0a38e0 2024-04-26T01:56:08.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.799+0000 7f1671600700 1 -- 172.21.15.6:0/2717617240 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16680037c0 con 0x56429b0a38e0 2024-04-26T01:56:08.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.799+0000 7f1671600700 1 -- 172.21.15.6:0/2717617240 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166800b9f0 con 0x56429b0a38e0 2024-04-26T01:56:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.800+0000 7f168d579b80 1 -- 172.21.15.6:0/2717617240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b0a38e0 msgr2=0x56429b0a5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.800+0000 7f168d579b80 1 --2- 172.21.15.6:0/2717617240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b0a38e0 0x56429b0a5cd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f16680099c0 tx=0x7f1668005bc0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.800+0000 7f1680c00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f16600a3db0 0x7f16600a6270 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1674005e90 tx=0x7f1674014040 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.800+0000 7f168d579b80 1 -- 172.21.15.6:0/2717617240 shutdown_connections 2024-04-26T01:56:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.800+0000 7f168d579b80 1 --2- 172.21.15.6:0/2717617240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b0a38e0 0x56429b0a5cd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.800+0000 7f168d579b80 1 --2- 172.21.15.6:0/2717617240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b093d40 0x56429b094120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.800+0000 7f168d579b80 1 --2- 172.21.15.6:0/2717617240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 0x56429b0a33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.800+0000 7f168d579b80 1 -- 172.21.15.6:0/2717617240 >> 172.21.15.6:0/2717617240 conn(0x56429afc3050 msgr2=0x56429b090250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.800+0000 7f168d579b80 1 -- 172.21.15.6:0/2717617240 shutdown_connections 2024-04-26T01:56:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.800+0000 7f168d579b80 1 -- 172.21.15.6:0/2717617240 wait complete. 2024-04-26T01:56:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.801+0000 7f168d579b80 1 -- start start 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.801+0000 7f168d579b80 1 Processor -- start 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.802+0000 7f168d579b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b093d40 0x56429b0a2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.802+0000 7f168d579b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b094660 0x56429b0a0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.802+0000 7f168d579b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b0a38e0 0x56429b0a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.802+0000 7f168d579b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56429b0a62b0 con 0x56429b093d40 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.802+0000 7f168d579b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56429afa0960 con 0x56429b0a38e0 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.802+0000 7f168d579b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56429b0a6430 con 0x56429b094660 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.802+0000 7f168d579b80 10 client.0 _ll_register_callbacks cb 0x56429af7f820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.802+0000 7f167be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b093d40 0x56429b0a2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.802+0000 7f167be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b093d40 0x56429b0a2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55000/0 (socket says 172.21.15.6:55000) 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.802+0000 7f167be00700 1 -- 172.21.15.6:0/4077850501 learned_addr learned my addr 172.21.15.6:0/4077850501 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.802+0000 7f167b400700 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b094660 0x56429b0a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.802+0000 7f1680c00700 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b0a38e0 0x56429b0a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.803+0000 7f167be00700 1 -- 172.21.15.6:0/4077850501 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b094660 msgr2=0x56429b0a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:08.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.803+0000 7f167be00700 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b094660 0x56429b0a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.803+0000 7f167be00700 1 -- 172.21.15.6:0/4077850501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b0a38e0 msgr2=0x56429b0a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:08.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.803+0000 7f167be00700 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b0a38e0 0x56429b0a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.803+0000 7f167be00700 1 -- 172.21.15.6:0/4077850501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56429af349d0 con 0x56429b093d40 2024-04-26T01:56:08.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.803+0000 7f167be00700 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b093d40 0x56429b0a2f80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f166c0cda90 tx=0x7f166c0cdf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c96236ad7da18 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:08.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.803+0000 7f1666000700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:56:08.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.803+0000 7f1666000700 1 -- 172.21.15.6:0/4077850501 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c0a4c60 con 0x56429b093d40 2024-04-26T01:56:08.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.803+0000 7f1666000700 1 -- 172.21.15.6:0/4077850501 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f166c0bcbc0 con 0x56429b093d40 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.803+0000 7f168d579b80 1 -- 172.21.15.6:0/4077850501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16680037c0 con 0x56429b093d40 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.804+0000 7f1666000700 1 -- 172.21.15.6:0/4077850501 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f166c0a06c0 con 0x56429b093d40 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.804+0000 7f168d579b80 10 client.16287 fetch_fsmap learned FSMap version 43 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.804+0000 7f168d579b80 1 -- 172.21.15.6:0/4077850501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56429af7dbf0 con 0x56429b093d40 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.804+0000 7f1666000700 1 -- 172.21.15.6:0/4077850501 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16680037c0 con 0x56429b093d40 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.804+0000 7f1666000700 1 -- 172.21.15.6:0/4077850501 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f166c004ad0 con 0x56429b093d40 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.804+0000 7f168d579b80 10 client.16287 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.804+0000 7f168d579b80 10 client.16287 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.804+0000 7f168d579b80 20 client.16287 populate_metadata read hostname 'smithi006' 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.804+0000 7f168d579b80 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56429b0b2fa0 0x56429b0b5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.804+0000 7f168d579b80 4 client.16287 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.804+0000 7f168d579b80 1 -- 172.21.15.6:0/4077850501 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56429aff4780 con 0x56429b0b2fa0 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.805+0000 7f167b400700 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56429b0b2fa0 0x56429b0b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.805+0000 7f167b400700 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56429b0b2fa0 0x56429b0b5400 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3dc677446a94adc server_cookie=fbf8f3413a74aa08 in_seq=0 out_seq=0 2024-04-26T01:56:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.805+0000 7f1666000700 10 client.16287 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:08.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.806+0000 7f1666000700 1 -- 172.21.15.6:0/4077850501 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56429aff4780 con 0x56429b0b2fa0 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.806+0000 7f1666000700 10 client.16287 handle_command_reply: tid=0 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.806+0000 7f168d579b80 1 client.16287 shutdown 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.806+0000 7f168d579b80 1 -- 172.21.15.6:0/4077850501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b093d40 msgr2=0x56429b0a2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.806+0000 7f168d579b80 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b093d40 0x56429b0a2f80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f166c0cda90 tx=0x7f166c0cdf90 comp rx=0 tx=0).stop 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.807+0000 7f168d579b80 1 -- 172.21.15.6:0/4077850501 shutdown_connections 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.807+0000 7f168d579b80 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b0a38e0 0x56429b0a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.807+0000 7f168d579b80 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b093d40 0x56429b0a2f80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.807+0000 7f168d579b80 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56429b0b2fa0 0x56429b0b5400 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.807+0000 7f168d579b80 1 --2- 172.21.15.6:0/4077850501 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b094660 0x56429b0a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.807+0000 7f1666000700 0 client.16287 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.807+0000 7f168d579b80 1 -- 172.21.15.6:0/4077850501 >> 172.21.15.6:0/4077850501 conn(0x56429afc3050 msgr2=0x56429b08fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.807+0000 7f168d579b80 1 -- 172.21.15.6:0/4077850501 shutdown_connections 2024-04-26T01:56:08.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.807+0000 7f168d579b80 1 -- 172.21.15.6:0/4077850501 wait complete. 2024-04-26T01:56:08.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.807+0000 7f168d579b80 20 client.16287 trim_cache size 0 max 16384 2024-04-26T01:56:08.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.830+0000 7f168d579b80 1 Processor -- start 2024-04-26T01:56:08.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.831+0000 7f168d579b80 1 -- start start 2024-04-26T01:56:08.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.831+0000 7f168d579b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 0x56429b0af2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.831+0000 7f168d579b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 0x56429b0bacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.831+0000 7f168d579b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b0a38e0 0x56429b0bb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.831+0000 7f168d579b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f165c00ff30 con 0x56429b094660 2024-04-26T01:56:08.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.831+0000 7f168d579b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f165c00d7c0 con 0x56429b093d40 2024-04-26T01:56:08.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.831+0000 7f168d579b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f165c00b630 con 0x56429b0a38e0 2024-04-26T01:56:08.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.831+0000 7f167b400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 0x56429b0bacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.831+0000 7f167b400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 0x56429b0bacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55008/0 (socket says 172.21.15.6:55008) 2024-04-26T01:56:08.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.831+0000 7f167b400700 1 -- 172.21.15.6:0/724137575 learned_addr learned my addr 172.21.15.6:0/724137575 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:08.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.832+0000 7f167be00700 1 --2- 172.21.15.6:0/724137575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 0x56429b0af2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.832+0000 7f1680c00700 1 --2- 172.21.15.6:0/724137575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b0a38e0 0x56429b0bb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.832+0000 7f167be00700 1 -- 172.21.15.6:0/724137575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b0a38e0 msgr2=0x56429b0bb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.832+0000 7f167be00700 1 --2- 172.21.15.6:0/724137575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b0a38e0 0x56429b0bb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.832+0000 7f167be00700 1 -- 172.21.15.6:0/724137575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 msgr2=0x56429b0bacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.832+0000 7f167be00700 1 --2- 172.21.15.6:0/724137575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 0x56429b0bacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.832+0000 7f167be00700 1 -- 172.21.15.6:0/724137575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56429b094ce0 con 0x56429b093d40 2024-04-26T01:56:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.832+0000 7f167b400700 1 --2- 172.21.15.6:0/724137575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 0x56429b0bacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.832+0000 7f167be00700 1 --2- 172.21.15.6:0/724137575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 0x56429b0af2a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f166c098ce0 tx=0x7f166c007b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7ab9b93c83b32e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.833+0000 7f1670c00700 1 -- 172.21.15.6:0/724137575 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c0985b0 con 0x56429b093d40 2024-04-26T01:56:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.833+0000 7f1670c00700 1 -- 172.21.15.6:0/724137575 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f166c0cf030 con 0x56429b093d40 2024-04-26T01:56:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.833+0000 7f1670c00700 1 -- 172.21.15.6:0/724137575 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c0bce20 con 0x56429b093d40 2024-04-26T01:56:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.833+0000 7f168d579b80 1 -- 172.21.15.6:0/724137575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 msgr2=0x56429b0af2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.833+0000 7f168d579b80 1 --2- 172.21.15.6:0/724137575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 0x56429b0af2a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f166c098ce0 tx=0x7f166c007b90 comp rx=0 tx=0).stop 2024-04-26T01:56:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.833+0000 7f168d579b80 1 -- 172.21.15.6:0/724137575 shutdown_connections 2024-04-26T01:56:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.833+0000 7f168d579b80 1 --2- 172.21.15.6:0/724137575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b0a38e0 0x56429b0bb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.833+0000 7f168d579b80 1 --2- 172.21.15.6:0/724137575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 0x56429b0af2a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.833+0000 7f168d579b80 1 --2- 172.21.15.6:0/724137575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b094660 0x56429b0bacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.833+0000 7f168d579b80 1 -- 172.21.15.6:0/724137575 >> 172.21.15.6:0/724137575 conn(0x56429afc3050 msgr2=0x56429b0a2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.834+0000 7f168d579b80 1 -- 172.21.15.6:0/724137575 shutdown_connections 2024-04-26T01:56:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.834+0000 7f168d579b80 1 -- 172.21.15.6:0/724137575 wait complete. 2024-04-26T01:56:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.834+0000 7f168d579b80 1 -- start start 2024-04-26T01:56:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.834+0000 7f168d579b80 1 Processor -- start 2024-04-26T01:56:08.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.835+0000 7f168d579b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 0x56429b09df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.835+0000 7f168d579b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b094660 0x56429b0ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.835+0000 7f168d579b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b0a38e0 0x56429b09c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.835+0000 7f168d579b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f165c00bc80 con 0x56429b0a38e0 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.835+0000 7f168d579b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f165c00d0f0 con 0x56429b093d40 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.835+0000 7f168d579b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f165c00c350 con 0x56429b094660 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.835+0000 7f168d579b80 10 client.0 _ll_register_callbacks cb 0x56429af7f820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.835+0000 7f1680c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b0a38e0 0x56429b09c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.835+0000 7f167be00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 0x56429b09df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.835+0000 7f167be00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 0x56429b09df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:51894/0 (socket says 172.21.15.6:51894) 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.835+0000 7f167be00700 1 -- 172.21.15.6:0/820699070 learned_addr learned my addr 172.21.15.6:0/820699070 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.835+0000 7f167b400700 1 --2- 172.21.15.6:0/820699070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b094660 0x56429b0ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.836+0000 7f167be00700 1 -- 172.21.15.6:0/820699070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b094660 msgr2=0x56429b0ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.836+0000 7f167be00700 1 --2- 172.21.15.6:0/820699070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b094660 0x56429b0ab7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.836+0000 7f167be00700 1 -- 172.21.15.6:0/820699070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b0a38e0 msgr2=0x56429b09c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.836+0000 7f167be00700 1 --2- 172.21.15.6:0/820699070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b0a38e0 0x56429b09c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.836+0000 7f167be00700 1 -- 172.21.15.6:0/820699070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56429b09ca70 con 0x56429b093d40 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.836+0000 7f167be00700 1 --2- 172.21.15.6:0/820699070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 0x56429b09df60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f166c0074d0 tx=0x7f166c0a4cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57df9a7152ac1a0b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:08.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.836+0000 7f1666000700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:56:08.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.836+0000 7f1666000700 1 -- 172.21.15.6:0/820699070 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c0985b0 con 0x56429b093d40 2024-04-26T01:56:08.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.836+0000 7f1666000700 1 -- 172.21.15.6:0/820699070 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8600+0+0 (secure 0 0 0) 0x7f166c0112c0 con 0x56429b093d40 2024-04-26T01:56:08.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.836+0000 7f168d579b80 1 -- 172.21.15.6:0/820699070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f166c0cf030 con 0x56429b093d40 2024-04-26T01:56:08.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.837+0000 7f1666000700 1 -- 172.21.15.6:0/820699070 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f166c004c60 con 0x56429b093d40 2024-04-26T01:56:08.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.837+0000 7f168d579b80 10 client.26435 fetch_fsmap learned FSMap version 43 2024-04-26T01:56:08.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.837+0000 7f168d579b80 1 -- 172.21.15.6:0/820699070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56429b11ac20 con 0x56429b093d40 2024-04-26T01:56:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.837+0000 7f1666000700 1 -- 172.21.15.6:0/820699070 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f166c0cf030 con 0x56429b093d40 2024-04-26T01:56:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.837+0000 7f1666000700 1 -- 172.21.15.6:0/820699070 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f166c063070 con 0x56429b093d40 2024-04-26T01:56:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.837+0000 7f168d579b80 10 client.26435 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:56:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.837+0000 7f168d579b80 10 client.26435 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:56:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.837+0000 7f168d579b80 20 client.26435 populate_metadata read hostname 'smithi006' 2024-04-26T01:56:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.837+0000 7f168d579b80 1 --2- 172.21.15.6:0/820699070 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56429b11b080 0x56429b11d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.837+0000 7f168d579b80 4 client.26435 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "eeff187f", "release": true, "await": true}] 2024-04-26T01:56:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.837+0000 7f168d579b80 1 -- 172.21.15.6:0/820699070 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "eeff187f", "release": true, "await": true}) -- 0x56429afee820 con 0x56429b11b080 2024-04-26T01:56:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.838+0000 7f167b400700 1 --2- 172.21.15.6:0/820699070 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56429b11b080 0x56429b11d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.838+0000 7f167b400700 1 --2- 172.21.15.6:0/820699070 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56429b11b080 0x56429b11d470 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed0adcc2941f1d01 server_cookie=1f9f33d21b8a1470 in_seq=0 out_seq=0 2024-04-26T01:56:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.838+0000 7f1666000700 10 client.26435 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:08.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.844+0000 7f1666000700 1 -- 172.21.15.6:0/820699070 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56429afee820 con 0x56429b11b080 2024-04-26T01:56:08.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.844+0000 7f1666000700 10 client.26435 handle_command_reply: tid=0 2024-04-26T01:56:08.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.844+0000 7f168d579b80 1 client.26435 shutdown 2024-04-26T01:56:08.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.844+0000 7f168d579b80 1 -- 172.21.15.6:0/820699070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 msgr2=0x56429b09df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:08.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.844+0000 7f168d579b80 1 --2- 172.21.15.6:0/820699070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 0x56429b09df60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f166c0074d0 tx=0x7f166c0a4cf0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.844+0000 7f168d579b80 1 -- 172.21.15.6:0/820699070 shutdown_connections 2024-04-26T01:56:08.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.844+0000 7f168d579b80 1 --2- 172.21.15.6:0/820699070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56429b0a38e0 0x56429b09c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.844+0000 7f168d579b80 1 --2- 172.21.15.6:0/820699070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56429b093d40 0x56429b09df60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.844+0000 7f168d579b80 1 --2- 172.21.15.6:0/820699070 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56429b11b080 0x56429b11d470 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.844+0000 7f168d579b80 1 --2- 172.21.15.6:0/820699070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56429b094660 0x56429b0ab7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.844+0000 7f1666000700 0 client.26435 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:08.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.844+0000 7f168d579b80 1 -- 172.21.15.6:0/820699070 >> 172.21.15.6:0/820699070 conn(0x56429afc3050 msgr2=0x56429b095bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:08.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.845+0000 7f168d579b80 1 -- 172.21.15.6:0/820699070 shutdown_connections 2024-04-26T01:56:08.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.845+0000 7f168d579b80 1 -- 172.21.15.6:0/820699070 wait complete. 2024-04-26T01:56:08.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.845+0000 7f168d579b80 20 client.26435 trim_cache size 0 max 16384 2024-04-26T01:56:08.844 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:56:08.844 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 43, 2024-04-26T01:56:08.844 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:56:08.844 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 32, 2024-04-26T01:56:08.844 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:56:08.844 INFO:teuthology.orchestra.run.smithi006.stdout: "eeff187f": { 2024-04-26T01:56:08.844 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 32, 2024-04-26T01:56:08.844 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:56:08.844 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:56:08.844 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 82.1, 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:56:08.845 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:56:08.846 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:56:08.846 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:56:08.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.845+0000 7f1682c00700 1 -- 172.21.15.6:0/3885858069 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f16600a3db0 msgr2=0x7f16600a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:08.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.845+0000 7f1682c00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f16600a3db0 0x7f16600a6270 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1674005e90 tx=0x7f1674014040 comp rx=0 tx=0).stop 2024-04-26T01:56:08.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.845+0000 7f1682c00700 1 -- 172.21.15.6:0/3885858069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f167c102af0 msgr2=0x7f167c1991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:08.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.845+0000 7f1682c00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f167c102af0 0x7f167c1991a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f166c00ef30 tx=0x7f166c00c3a0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.846+0000 7f1682c00700 1 -- 172.21.15.6:0/3885858069 shutdown_connections 2024-04-26T01:56:08.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.846+0000 7f1682c00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f16600a3db0 0x7f16600a6270 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.846+0000 7f1682c00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f167c107940 0x7f167c19da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.846+0000 7f1682c00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f167c102af0 0x7f167c1991a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.846+0000 7f1682c00700 1 --2- 172.21.15.6:0/3885858069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f167c1034b0 0x7f167c1996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:08.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.846+0000 7f1682c00700 1 -- 172.21.15.6:0/3885858069 >> 172.21.15.6:0/3885858069 conn(0x7f167c079ba0 msgr2=0x7f167c100c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:08.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.846+0000 7f1682c00700 1 -- 172.21.15.6:0/3885858069 shutdown_connections 2024-04-26T01:56:08.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:56:08.846+0000 7f1682c00700 1 -- 172.21.15.6:0/3885858069 wait complete. 2024-04-26T01:56:09.618 INFO:tasks.workunit.client.0.smithi006.stdout: 1 339736 3.38 MB/sec execute 393 sec latency 29988.696 ms 2024-04-26T01:56:09.647 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'eeff187f', seconds elapsed: 37.6 2024-04-26T01:56:10.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:10 smithi006 ceph-mon[28419]: pgmap v322: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 55 KiB/s wr, 24 op/s 2024-04-26T01:56:10.618 INFO:tasks.workunit.client.0.smithi006.stdout: 1 340782 3.40 MB/sec execute 394 sec latency 29.359 ms 2024-04-26T01:56:10.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:10 smithi188 ceph-mon[32751]: pgmap v322: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 55 KiB/s wr, 24 op/s 2024-04-26T01:56:10.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:10 smithi029 ceph-mon[32775]: pgmap v322: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 55 KiB/s wr, 24 op/s 2024-04-26T01:56:11.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:11 smithi006 ceph-mon[28419]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T01:56:11.619 INFO:tasks.workunit.client.0.smithi006.stdout: 1 341390 3.39 MB/sec execute 395 sec latency 26.382 ms 2024-04-26T01:56:11.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:11 smithi188 ceph-mon[32751]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T01:56:11.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:11 smithi029 ceph-mon[32775]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T01:56:12.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:12 smithi006 ceph-mon[28419]: pgmap v324: 97 pgs: 97 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 235 KiB/s wr, 39 op/s 2024-04-26T01:56:12.619 INFO:tasks.workunit.client.0.smithi006.stdout: 1 342344 3.39 MB/sec execute 396 sec latency 28.546 ms 2024-04-26T01:56:12.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:12 smithi188 ceph-mon[32751]: pgmap v324: 97 pgs: 97 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 235 KiB/s wr, 39 op/s 2024-04-26T01:56:12.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:12 smithi029 ceph-mon[32775]: pgmap v324: 97 pgs: 97 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 235 KiB/s wr, 39 op/s 2024-04-26T01:56:13.619 INFO:tasks.workunit.client.0.smithi006.stdout: 1 343714 3.40 MB/sec execute 397 sec latency 20.928 ms 2024-04-26T01:56:14.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:14 smithi006 ceph-mon[28419]: pgmap v325: 97 pgs: 97 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 235 KiB/s wr, 39 op/s 2024-04-26T01:56:14.619 INFO:tasks.workunit.client.0.smithi006.stdout: 1 344491 3.41 MB/sec execute 398 sec latency 22.236 ms 2024-04-26T01:56:14.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:14 smithi188 ceph-mon[32751]: pgmap v325: 97 pgs: 97 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 235 KiB/s wr, 39 op/s 2024-04-26T01:56:14.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:14 smithi029 ceph-mon[32775]: pgmap v325: 97 pgs: 97 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 235 KiB/s wr, 39 op/s 2024-04-26T01:56:15.619 INFO:tasks.workunit.client.0.smithi006.stdout: 1 345353 3.41 MB/sec execute 399 sec latency 22.276 ms 2024-04-26T01:56:16.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:16 smithi006 ceph-mon[28419]: pgmap v326: 97 pgs: 97 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 235 KiB/s wr, 39 op/s 2024-04-26T01:56:16.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:16 smithi006 ceph-mon[28419]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T01:56:16.619 INFO:tasks.workunit.client.0.smithi006.stdout: 1 346436 3.41 MB/sec execute 400 sec latency 25.116 ms 2024-04-26T01:56:16.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:16 smithi188 ceph-mon[32751]: pgmap v326: 97 pgs: 97 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 235 KiB/s wr, 39 op/s 2024-04-26T01:56:16.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:16 smithi188 ceph-mon[32751]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T01:56:16.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:16 smithi029 ceph-mon[32775]: pgmap v326: 97 pgs: 97 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 235 KiB/s wr, 39 op/s 2024-04-26T01:56:16.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:16 smithi029 ceph-mon[32775]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T01:56:17.619 INFO:tasks.workunit.client.0.smithi006.stdout: 1 347699 3.43 MB/sec execute 401 sec latency 18.795 ms 2024-04-26T01:56:18.620 INFO:tasks.workunit.client.0.smithi006.stdout: 1 348204 3.42 MB/sec execute 402 sec latency 21.116 ms 2024-04-26T01:56:18.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:18 smithi188 ceph-mon[32751]: pgmap v328: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 995 KiB/s wr, 125 op/s 2024-04-26T01:56:18.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:18 smithi029 ceph-mon[32775]: pgmap v328: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 995 KiB/s wr, 125 op/s 2024-04-26T01:56:18.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:18 smithi006 ceph-mon[28419]: pgmap v328: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 995 KiB/s wr, 125 op/s 2024-04-26T01:56:19.620 INFO:tasks.workunit.client.0.smithi006.stdout: 1 349191 3.43 MB/sec execute 403 sec latency 29.017 ms 2024-04-26T01:56:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.298+0000 7fb95fe00640 1 -- 172.21.15.6:0/3135317792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb960101fe0 msgr2=0x7fb9601023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.298+0000 7fb95fe00640 1 --2- 172.21.15.6:0/3135317792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb960101fe0 0x7fb9601023c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb950009900 tx=0x7fb95002a410 comp rx=0 tx=0).stop 2024-04-26T01:56:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.298+0000 7fb95fe00640 1 -- 172.21.15.6:0/3135317792 shutdown_connections 2024-04-26T01:56:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.298+0000 7fb95fe00640 1 --2- 172.21.15.6:0/3135317792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb960102990 0x7fb9601069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.298+0000 7fb95fe00640 1 --2- 172.21.15.6:0/3135317792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9601070b0 0x7fb96010ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.298+0000 7fb95fe00640 1 --2- 172.21.15.6:0/3135317792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb960101fe0 0x7fb9601023c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.298+0000 7fb95fe00640 1 -- 172.21.15.6:0/3135317792 >> 172.21.15.6:0/3135317792 conn(0x7fb9600fd860 msgr2=0x7fb9600ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.298+0000 7fb95fe00640 1 -- 172.21.15.6:0/3135317792 shutdown_connections 2024-04-26T01:56:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.298+0000 7fb95fe00640 1 -- 172.21.15.6:0/3135317792 wait complete. 2024-04-26T01:56:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.299+0000 7fb95fe00640 1 Processor -- start 2024-04-26T01:56:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.299+0000 7fb95fe00640 1 -- start start 2024-04-26T01:56:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.300+0000 7fb95fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb960101fe0 0x7fb9601a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.300+0000 7fb95fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9601070b0 0x7fb9601a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.300+0000 7fb95fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb96019a5c0 0x7fb96019aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.300+0000 7fb95fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb960112040 con 0x7fb96019a5c0 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.300+0000 7fb95fe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb960111ec0 con 0x7fb960101fe0 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.300+0000 7fb95fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb9601121c0 con 0x7fb9601070b0 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.300+0000 7fb95ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb960101fe0 0x7fb9601a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.300+0000 7fb95ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb960101fe0 0x7fb9601a0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39112/0 (socket says 172.21.15.6:39112) 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.300+0000 7fb95ea00640 1 -- 172.21.15.6:0/3138662529 learned_addr learned my addr 172.21.15.6:0/3138662529 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.300+0000 7fb95e000640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9601070b0 0x7fb9601a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.300+0000 7fb95f400640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb96019a5c0 0x7fb96019aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.301+0000 7fb95ea00640 1 -- 172.21.15.6:0/3138662529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9601070b0 msgr2=0x7fb9601a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.301+0000 7fb95ea00640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9601070b0 0x7fb9601a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.301+0000 7fb95ea00640 1 -- 172.21.15.6:0/3138662529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb96019a5c0 msgr2=0x7fb96019aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.301+0000 7fb95ea00640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb96019a5c0 0x7fb96019aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.301+0000 7fb95ea00640 1 -- 172.21.15.6:0/3138662529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96019b0e0 con 0x7fb960101fe0 2024-04-26T01:56:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.301+0000 7fb95ea00640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb960101fe0 0x7fb9601a0410 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb950005ab0 tx=0x7fb950003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.301+0000 7fb957400640 1 -- 172.21.15.6:0/3138662529 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95003f070 con 0x7fb960101fe0 2024-04-26T01:56:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.301+0000 7fb957400640 1 -- 172.21.15.6:0/3138662529 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb950031b60 con 0x7fb960101fe0 2024-04-26T01:56:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.301+0000 7fb95fe00640 1 -- 172.21.15.6:0/3138662529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb96019b370 con 0x7fb960101fe0 2024-04-26T01:56:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.301+0000 7fb957400640 1 -- 172.21.15.6:0/3138662529 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb950035030 con 0x7fb960101fe0 2024-04-26T01:56:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.301+0000 7fb95fe00640 1 -- 172.21.15.6:0/3138662529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9601a7560 con 0x7fb960101fe0 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.305+0000 7fb957400640 1 -- 172.21.15.6:0/3138662529 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fb950042070 con 0x7fb960101fe0 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.306+0000 7fb9689e1740 1 Processor -- start 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.306+0000 7fb9689e1740 1 -- start start 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.306+0000 7fb9689e1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6a9b0 0x556dd1b6ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.306+0000 7fb9689e1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b75650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.306+0000 7fb9689e1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b75b90 0x556dd1b77f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.306+0000 7fb9689e1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x556dd1ad0900 con 0x556dd1b75b90 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.306+0000 7fb9689e1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556dd1a76ef0 con 0x556dd1b6b2d0 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.306+0000 7fb9689e1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556dd1acc8c0 con 0x556dd1b6a9b0 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.306+0000 7fb95f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6a9b0 0x556dd1b6ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.306+0000 7fb95f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6a9b0 0x556dd1b6ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51312/0 (socket says 172.21.15.6:51312) 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.306+0000 7fb95f400640 1 -- 172.21.15.6:0/2531787199 learned_addr learned my addr 172.21.15.6:0/2531787199 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.306+0000 7fb95ea00640 1 --2- 172.21.15.6:0/2531787199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b75650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.307+0000 7fb95e000640 1 --2- 172.21.15.6:0/2531787199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b75b90 0x556dd1b77f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.307+0000 7fb957400640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb9440a3d00 0x7fb9440a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.307+0000 7fb95f400640 1 -- 172.21.15.6:0/2531787199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 msgr2=0x556dd1b75650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.307+0000 7fb957400640 1 -- 172.21.15.6:0/3138662529 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8600+0+0 (secure 0 0 0) 0x7fb9500f0f40 con 0x7fb960101fe0 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.307+0000 7fb95f400640 1 --2- 172.21.15.6:0/2531787199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b75650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.307+0000 7fb95f400640 1 -- 172.21.15.6:0/2531787199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b75b90 msgr2=0x556dd1b77f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.307+0000 7fb95f400640 1 --2- 172.21.15.6:0/2531787199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b75b90 0x556dd1b77f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.307+0000 7fb95f400640 1 -- 172.21.15.6:0/2531787199 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dd17e3210 con 0x556dd1b6a9b0 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.307+0000 7fb95ea00640 1 --2- 172.21.15.6:0/2531787199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b75650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.307+0000 7fb95f400640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb9440a3d00 0x7fb9440a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.307+0000 7fb95f400640 1 --2- 172.21.15.6:0/2531787199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6a9b0 0x556dd1b6ad90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb958011590 tx=0x7fb958011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6128944bd9c521 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.308+0000 7fb94be00640 1 -- 172.21.15.6:0/2531787199 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb958017050 con 0x556dd1b6a9b0 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.308+0000 7fb94be00640 1 -- 172.21.15.6:0/2531787199 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb95801bb90 con 0x556dd1b6a9b0 2024-04-26T01:56:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.308+0000 7fb94be00640 1 -- 172.21.15.6:0/2531787199 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95800ed60 con 0x556dd1b6a9b0 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.308+0000 7fb95f400640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb9440a3d00 0x7fb9440a61c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb958022000 tx=0x7fb958023000 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.309+0000 7fb9689e1740 1 -- 172.21.15.6:0/2531787199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6a9b0 msgr2=0x556dd1b6ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.309+0000 7fb9689e1740 1 --2- 172.21.15.6:0/2531787199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6a9b0 0x556dd1b6ad90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb958011590 tx=0x7fb958011a50 comp rx=0 tx=0).stop 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.309+0000 7fb9689e1740 1 -- 172.21.15.6:0/2531787199 shutdown_connections 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.309+0000 7fb9689e1740 1 --2- 172.21.15.6:0/2531787199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b75b90 0x556dd1b77f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.309+0000 7fb9689e1740 1 --2- 172.21.15.6:0/2531787199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b75650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.309+0000 7fb9689e1740 1 --2- 172.21.15.6:0/2531787199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6a9b0 0x556dd1b6ad90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.309+0000 7fb9689e1740 1 -- 172.21.15.6:0/2531787199 >> 172.21.15.6:0/2531787199 conn(0x556dd1b66ec0 msgr2=0x556dd1b672a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.309+0000 7fb9689e1740 1 -- 172.21.15.6:0/2531787199 shutdown_connections 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.309+0000 7fb9689e1740 1 -- 172.21.15.6:0/2531787199 wait complete. 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.310+0000 7fb9689e1740 1 -- start start 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.310+0000 7fb9689e1740 1 Processor -- start 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.310+0000 7fb9689e1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b6a9b0 0x556dd1b72b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.310+0000 7fb9689e1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b730c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.310+0000 7fb9689e1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b75b90 0x556dd1b73600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.310+0000 7fb9689e1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x556dd1b787c0 con 0x556dd1b6a9b0 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.310+0000 7fb9689e1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556dd1a8a310 con 0x556dd1b6b2d0 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.310+0000 7fb9689e1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556dd1b78940 con 0x556dd1b75b90 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.310+0000 7fb9689e1740 10 client.0 _ll_register_callbacks cb 0x556dd1a58cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb95e000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb95e000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b730c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39128/0 (socket says 172.21.15.6:39128) 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb95e000640 1 -- 172.21.15.6:0/4215166254 learned_addr learned my addr 172.21.15.6:0/4215166254 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb95f400640 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b75b90 0x556dd1b73600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb95ea00640 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b6a9b0 0x556dd1b72b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb95e000640 1 -- 172.21.15.6:0/4215166254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b75b90 msgr2=0x556dd1b73600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb95e000640 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b75b90 0x556dd1b73600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb95e000640 1 -- 172.21.15.6:0/4215166254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b6a9b0 msgr2=0x556dd1b72b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb95e000640 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b6a9b0 0x556dd1b72b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb95e000640 1 -- 172.21.15.6:0/4215166254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dd18af620 con 0x556dd1b6b2d0 2024-04-26T01:56:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb95ea00640 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b6a9b0 0x556dd1b72b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:56:20.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb95e000640 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b730c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0126d0 tx=0x7fb94c012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=601c0d91230ec9be server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:20.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.311+0000 7fb948c00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:56:20.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.312+0000 7fb948c00640 1 -- 172.21.15.6:0/4215166254 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c004050 con 0x556dd1b6b2d0 2024-04-26T01:56:20.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.312+0000 7fb948c00640 1 -- 172.21.15.6:0/4215166254 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8600+0+0 (secure 0 0 0) 0x7fb94c0041f0 con 0x556dd1b6b2d0 2024-04-26T01:56:20.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.312+0000 7fb9689e1740 1 -- 172.21.15.6:0/4215166254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb95801bb90 con 0x556dd1b6b2d0 2024-04-26T01:56:20.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.312+0000 7fb948c00640 1 -- 172.21.15.6:0/4215166254 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fb94c0043f0 con 0x556dd1b6b2d0 2024-04-26T01:56:20.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.312+0000 7fb9689e1740 10 client.26447 fetch_fsmap learned FSMap version 43 2024-04-26T01:56:20.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.312+0000 7fb9689e1740 1 -- 172.21.15.6:0/4215166254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x556dd1b84a80 con 0x556dd1b6b2d0 2024-04-26T01:56:20.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.312+0000 7fb948c00640 1 -- 172.21.15.6:0/4215166254 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb95801bb90 con 0x556dd1b6b2d0 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.312+0000 7fb948c00640 1 -- 172.21.15.6:0/4215166254 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7fb94c003ac0 con 0x556dd1b6b2d0 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.313+0000 7fb9689e1740 10 client.26447 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.313+0000 7fb9689e1740 10 client.26447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.313+0000 7fb9689e1740 20 client.26447 populate_metadata read hostname 'smithi006' 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.313+0000 7fb9689e1740 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556dd1b85f40 0x556dd1b883c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.313+0000 7fb9689e1740 4 client.26447 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.313+0000 7fb9689e1740 1 -- 172.21.15.6:0/4215166254 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556dd1ae48b0 con 0x556dd1b85f40 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.313+0000 7fb95ea00640 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556dd1b85f40 0x556dd1b883c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.314+0000 7fb95ea00640 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556dd1b85f40 0x556dd1b883c0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=413c2bab25be69f1 server_cookie=887e7904ac7ea05a in_seq=0 out_seq=0 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.314+0000 7fb948c00640 10 client.26447 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb948c00640 1 -- 172.21.15.6:0/4215166254 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556dd1ae48b0 con 0x556dd1b85f40 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb948c00640 10 client.26447 handle_command_reply: tid=0 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb9689e1740 1 client.26447 shutdown 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb9689e1740 1 -- 172.21.15.6:0/4215166254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 msgr2=0x556dd1b730c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb9689e1740 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b730c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0126d0 tx=0x7fb94c012b90 comp rx=0 tx=0).stop 2024-04-26T01:56:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb9689e1740 1 -- 172.21.15.6:0/4215166254 shutdown_connections 2024-04-26T01:56:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb9689e1740 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556dd1b85f40 0x556dd1b883c0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb9689e1740 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b75b90 0x556dd1b73600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb9689e1740 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b730c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb9689e1740 1 --2- 172.21.15.6:0/4215166254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b6a9b0 0x556dd1b72b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb948c00640 0 client.26447 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb9689e1740 1 -- 172.21.15.6:0/4215166254 >> 172.21.15.6:0/4215166254 conn(0x556dd1b66ec0 msgr2=0x556dd1b6cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb9689e1740 1 -- 172.21.15.6:0/4215166254 shutdown_connections 2024-04-26T01:56:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.315+0000 7fb9689e1740 1 -- 172.21.15.6:0/4215166254 wait complete. 2024-04-26T01:56:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.316+0000 7fb9689e1740 20 client.26447 trim_cache size 0 max 16384 2024-04-26T01:56:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.333+0000 7fb9689e1740 1 Processor -- start 2024-04-26T01:56:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.334+0000 7fb9689e1740 1 -- start start 2024-04-26T01:56:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.334+0000 7fb9689e1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6a9b0 0x556dd1b81680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.334+0000 7fb9689e1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6b2d0 0x556dd1b8dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.334+0000 7fb9689e1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b75b90 0x556dd1b8e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.334+0000 7fb9689e1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb9601b25f0 con 0x556dd1b75b90 2024-04-26T01:56:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.334+0000 7fb9689e1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb9601afec0 con 0x556dd1b6a9b0 2024-04-26T01:56:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.334+0000 7fb9689e1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb9601add30 con 0x556dd1b6b2d0 2024-04-26T01:56:20.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.334+0000 7fb95e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6b2d0 0x556dd1b8dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.334+0000 7fb95ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6a9b0 0x556dd1b81680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.334+0000 7fb95e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6b2d0 0x556dd1b8dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51342/0 (socket says 172.21.15.6:51342) 2024-04-26T01:56:20.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.334+0000 7fb95e000640 1 -- 172.21.15.6:0/2989377563 learned_addr learned my addr 172.21.15.6:0/2989377563 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:20.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.334+0000 7fb95f400640 1 --2- 172.21.15.6:0/2989377563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b75b90 0x556dd1b8e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.335+0000 7fb95e000640 1 -- 172.21.15.6:0/2989377563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6a9b0 msgr2=0x556dd1b81680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:20.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.335+0000 7fb95e000640 1 --2- 172.21.15.6:0/2989377563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6a9b0 0x556dd1b81680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.335+0000 7fb95e000640 1 -- 172.21.15.6:0/2989377563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b75b90 msgr2=0x556dd1b8e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:20.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.335+0000 7fb95e000640 1 --2- 172.21.15.6:0/2989377563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b75b90 0x556dd1b8e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.335+0000 7fb95e000640 1 -- 172.21.15.6:0/2989377563 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dd1b781b0 con 0x556dd1b6b2d0 2024-04-26T01:56:20.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.335+0000 7fb95f400640 1 --2- 172.21.15.6:0/2989377563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b75b90 0x556dd1b8e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:20.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.335+0000 7fb95ea00640 1 --2- 172.21.15.6:0/2989377563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6a9b0 0x556dd1b81680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.335+0000 7fb95e000640 1 --2- 172.21.15.6:0/2989377563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6b2d0 0x556dd1b8dcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb94c012420 tx=0x7fb94c009640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b7ffaa550433d54 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.335+0000 7fb94b400640 1 -- 172.21.15.6:0/2989377563 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c031070 con 0x556dd1b6b2d0 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.335+0000 7fb94b400640 1 -- 172.21.15.6:0/2989377563 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb94c00fd50 con 0x556dd1b6b2d0 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.335+0000 7fb94b400640 1 -- 172.21.15.6:0/2989377563 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c003ab0 con 0x556dd1b6b2d0 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.336+0000 7fb9689e1740 1 -- 172.21.15.6:0/2989377563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6b2d0 msgr2=0x556dd1b8dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.336+0000 7fb9689e1740 1 --2- 172.21.15.6:0/2989377563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6b2d0 0x556dd1b8dcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb94c012420 tx=0x7fb94c009640 comp rx=0 tx=0).stop 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.336+0000 7fb9689e1740 1 -- 172.21.15.6:0/2989377563 shutdown_connections 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.336+0000 7fb9689e1740 1 --2- 172.21.15.6:0/2989377563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b75b90 0x556dd1b8e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.336+0000 7fb9689e1740 1 --2- 172.21.15.6:0/2989377563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b6b2d0 0x556dd1b8dcd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.336+0000 7fb9689e1740 1 --2- 172.21.15.6:0/2989377563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6a9b0 0x556dd1b81680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.336+0000 7fb9689e1740 1 -- 172.21.15.6:0/2989377563 >> 172.21.15.6:0/2989377563 conn(0x556dd1b66ec0 msgr2=0x556dd1b733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.337+0000 7fb9689e1740 1 -- 172.21.15.6:0/2989377563 shutdown_connections 2024-04-26T01:56:20.336 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.337+0000 7fb9689e1740 1 -- 172.21.15.6:0/2989377563 wait complete. 2024-04-26T01:56:20.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.337+0000 7fb9689e1740 1 -- start start 2024-04-26T01:56:20.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.337+0000 7fb9689e1740 1 Processor -- start 2024-04-26T01:56:20.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.338+0000 7fb9689e1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b6a9b0 0x556dd1b7de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.338+0000 7fb9689e1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b7c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.338+0000 7fb9689e1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b75b90 0x556dd1b7c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.338+0000 7fb9689e1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb9601ae380 con 0x556dd1b6a9b0 2024-04-26T01:56:20.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.338+0000 7fb9689e1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb9601af7f0 con 0x556dd1b6b2d0 2024-04-26T01:56:20.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.338+0000 7fb9689e1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb9601aea50 con 0x556dd1b75b90 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.338+0000 7fb9689e1740 10 client.0 _ll_register_callbacks cb 0x7fb94c009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.338+0000 7fb95e000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b7c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.338+0000 7fb95e000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b7c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39150/0 (socket says 172.21.15.6:39150) 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.338+0000 7fb95e000640 1 -- 172.21.15.6:0/1258327447 learned_addr learned my addr 172.21.15.6:0/1258327447 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.338+0000 7fb95ea00640 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b6a9b0 0x556dd1b7de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.338+0000 7fb95f400640 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b75b90 0x556dd1b7c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.339+0000 7fb95f400640 1 -- 172.21.15.6:0/1258327447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 msgr2=0x556dd1b7c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.339+0000 7fb95f400640 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b7c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.339+0000 7fb95f400640 1 -- 172.21.15.6:0/1258327447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b6a9b0 msgr2=0x556dd1b7de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.339+0000 7fb95f400640 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b6a9b0 0x556dd1b7de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.339+0000 7fb95f400640 1 -- 172.21.15.6:0/1258327447 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dd1b7ce70 con 0x556dd1b75b90 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.339+0000 7fb95f400640 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b75b90 0x556dd1b7c730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb95800ea40 tx=0x7fb95800ee90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77caa526531833c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.339+0000 7fb948c00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:56:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.344+0000 7fb948c00640 1 -- 172.21.15.6:0/1258327447 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb958033020 con 0x556dd1b75b90 2024-04-26T01:56:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb948c00640 1 -- 172.21.15.6:0/1258327447 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8600+0+0 (secure 0 0 0) 0x7fb95800e5b0 con 0x556dd1b75b90 2024-04-26T01:56:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb9689e1740 1 -- 172.21.15.6:0/1258327447 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb94c00fd50 con 0x556dd1b75b90 2024-04-26T01:56:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb948c00640 1 -- 172.21.15.6:0/1258327447 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fb958017020 con 0x556dd1b75b90 2024-04-26T01:56:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb9689e1740 10 client.26491 fetch_fsmap learned FSMap version 43 2024-04-26T01:56:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb9689e1740 1 -- 172.21.15.6:0/1258327447 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x556dd1bed4e0 con 0x556dd1b75b90 2024-04-26T01:56:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb948c00640 1 -- 172.21.15.6:0/1258327447 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb94c00fd50 con 0x556dd1b75b90 2024-04-26T01:56:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb948c00640 1 -- 172.21.15.6:0/1258327447 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7fb958025430 con 0x556dd1b75b90 2024-04-26T01:56:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb9689e1740 10 client.26491 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:56:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb9689e1740 10 client.26491 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb9689e1740 20 client.26491 populate_metadata read hostname 'smithi006' 2024-04-26T01:56:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb9689e1740 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556dd1bed980 0x556dd1befda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb9689e1740 4 client.26491 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.345+0000 7fb9689e1740 1 -- 172.21.15.6:0/1258327447 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556dd1ade320 con 0x556dd1bed980 2024-04-26T01:56:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.346+0000 7fb95ea00640 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556dd1bed980 0x556dd1befda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.346+0000 7fb95ea00640 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556dd1bed980 0x556dd1befda0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec8b33859b9df9dc server_cookie=9860afac85da7535 in_seq=0 out_seq=0 2024-04-26T01:56:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.346+0000 7fb948c00640 10 client.26491 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.395+0000 7fb948c00640 1 -- 172.21.15.6:0/1258327447 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556dd1ade320 con 0x556dd1bed980 2024-04-26T01:56:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb948c00640 10 client.26491 handle_command_reply: tid=0 2024-04-26T01:56:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb9689e1740 1 client.26491 shutdown 2024-04-26T01:56:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb9689e1740 1 -- 172.21.15.6:0/1258327447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b75b90 msgr2=0x556dd1b7c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb9689e1740 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b75b90 0x556dd1b7c730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb95800ea40 tx=0x7fb95800ee90 comp rx=0 tx=0).stop 2024-04-26T01:56:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb9689e1740 1 -- 172.21.15.6:0/1258327447 shutdown_connections 2024-04-26T01:56:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb9689e1740 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556dd1bed980 0x556dd1befda0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb9689e1740 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556dd1b75b90 0x556dd1b7c730 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb9689e1740 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dd1b6b2d0 0x556dd1b7c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb9689e1740 1 --2- 172.21.15.6:0/1258327447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556dd1b6a9b0 0x556dd1b7de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb9689e1740 1 -- 172.21.15.6:0/1258327447 >> 172.21.15.6:0/1258327447 conn(0x556dd1b66ec0 msgr2=0x556dd1b72b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb948c00640 0 client.26491 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb9689e1740 1 -- 172.21.15.6:0/1258327447 shutdown_connections 2024-04-26T01:56:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.396+0000 7fb9689e1740 1 -- 172.21.15.6:0/1258327447 wait complete. 2024-04-26T01:56:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.397+0000 7fb9689e1740 20 client.26491 trim_cache size 0 max 16384 2024-04-26T01:56:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:56:20.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:56:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:56:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:56:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.397+0000 7fb95fe00640 1 -- 172.21.15.6:0/3138662529 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb9440a3d00 msgr2=0x7fb9440a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.397+0000 7fb95fe00640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb9440a3d00 0x7fb9440a61c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb958022000 tx=0x7fb958023000 comp rx=0 tx=0).stop 2024-04-26T01:56:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.397+0000 7fb95fe00640 1 -- 172.21.15.6:0/3138662529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb960101fe0 msgr2=0x7fb9601a0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.397+0000 7fb95fe00640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb960101fe0 0x7fb9601a0410 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb950005ab0 tx=0x7fb950003c10 comp rx=0 tx=0).stop 2024-04-26T01:56:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.397+0000 7fb95fe00640 1 -- 172.21.15.6:0/3138662529 shutdown_connections 2024-04-26T01:56:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.397+0000 7fb95fe00640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fb9440a3d00 0x7fb9440a61c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.397+0000 7fb95fe00640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb96019a5c0 0x7fb96019aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.397+0000 7fb95fe00640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9601070b0 0x7fb9601a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.397+0000 7fb95fe00640 1 --2- 172.21.15.6:0/3138662529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb960101fe0 0x7fb9601a0410 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.397+0000 7fb95fe00640 1 -- 172.21.15.6:0/3138662529 >> 172.21.15.6:0/3138662529 conn(0x7fb9600fd860 msgr2=0x7fb9600ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.398+0000 7fb95fe00640 1 -- 172.21.15.6:0/3138662529 shutdown_connections 2024-04-26T01:56:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:20.398+0000 7fb95fe00640 1 -- 172.21.15.6:0/3138662529 wait complete. 2024-04-26T01:56:20.619 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:20 smithi006 ceph-mon[28419]: pgmap v329: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 891 KiB/s wr, 112 op/s 2024-04-26T01:56:20.619 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:20 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:56:20.620 INFO:tasks.workunit.client.0.smithi006.stdout: 1 350218 3.43 MB/sec execute 404 sec latency 266.311 ms 2024-04-26T01:56:20.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:20 smithi188 ceph-mon[32751]: pgmap v329: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 891 KiB/s wr, 112 op/s 2024-04-26T01:56:20.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:20 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:56:20.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:20 smithi029 ceph-mon[32775]: pgmap v329: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 891 KiB/s wr, 112 op/s 2024-04-26T01:56:20.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:20 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:56:21.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:21 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:21.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:21 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:21.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:21 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:21.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:21 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:21.535 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:21 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:21.535 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:21 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:21.535 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:21 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:21.535 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:21 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:21.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:21 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:21.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:21 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:21.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:21 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:21.568 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:21 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:21.620 INFO:tasks.workunit.client.0.smithi006.stdout: 1 351506 3.45 MB/sec execute 405 sec latency 16.476 ms 2024-04-26T01:56:22.620 INFO:tasks.workunit.client.0.smithi006.stdout: 1 352559 3.45 MB/sec execute 406 sec latency 23.754 ms 2024-04-26T01:56:22.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:22 smithi029 ceph-mon[32775]: pgmap v330: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-04-26T01:56:22.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:22 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:22.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:22 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:22.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:22 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:22.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:22 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:22 smithi006 ceph-mon[28419]: pgmap v330: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-04-26T01:56:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:22 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:22 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:22 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:22 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:22 smithi188 ceph-mon[32751]: pgmap v330: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-04-26T01:56:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:22 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:22 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:22 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:22.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:22 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:23.620 INFO:tasks.workunit.client.0.smithi006.stdout: 1 353748 3.45 MB/sec execute 407 sec latency 23.850 ms 2024-04-26T01:56:24.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:24.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:24.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: Unable to set osd_memory_target on smithi029 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-26T01:56:24.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: Unable to set osd_memory_target on smithi188 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-26T01:56:24.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:23 smithi006 ceph-mon[28419]: pgmap v331: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-04-26T01:56:24.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:24.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:24.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: Unable to set osd_memory_target on smithi029 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-26T01:56:24.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: Unable to set osd_memory_target on smithi188 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-26T01:56:24.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:23 smithi188 ceph-mon[32751]: pgmap v331: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-04-26T01:56:24.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:24.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:24.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: Unable to set osd_memory_target on smithi029 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:24.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-26T01:56:24.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: Unable to set osd_memory_target on smithi188 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-26T01:56:24.193 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:23 smithi029 ceph-mon[32775]: pgmap v331: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-04-26T01:56:24.620 INFO:tasks.workunit.client.0.smithi006.stdout: 1 354952 3.47 MB/sec execute 408 sec latency 22.038 ms 2024-04-26T01:56:25.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi006 to 177.5M 2024-04-26T01:56:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: Unable to set osd_memory_target on smithi006 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:56:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:56:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:56:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:24 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:25.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi006 to 177.5M 2024-04-26T01:56:25.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: Unable to set osd_memory_target on smithi006 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:56:25.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:56:25.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:56:25.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:24 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:25.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:56:25.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi006 to 177.5M 2024-04-26T01:56:25.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: Unable to set osd_memory_target on smithi006 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:56:25.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:56:25.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:56:25.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:24 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:56:25.620 INFO:tasks.workunit.client.0.smithi006.stdout: 1 355520 3.46 MB/sec execute 409 sec latency 19.792 ms 2024-04-26T01:56:26.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:25 smithi006 ceph-mon[28419]: pgmap v332: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-04-26T01:56:26.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:25 smithi188 ceph-mon[32751]: pgmap v332: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-04-26T01:56:26.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:25 smithi029 ceph-mon[32775]: pgmap v332: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-04-26T01:56:26.620 INFO:tasks.workunit.client.0.smithi006.stdout: 1 356508 3.46 MB/sec execute 410 sec latency 27.044 ms 2024-04-26T01:56:27.620 INFO:tasks.workunit.client.0.smithi006.stdout: 1 357874 3.47 MB/sec execute 411 sec latency 22.110 ms 2024-04-26T01:56:28.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:28 smithi006 ceph-mon[28419]: pgmap v333: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.5 MiB/s wr, 207 op/s 2024-04-26T01:56:28.621 INFO:tasks.workunit.client.0.smithi006.stdout: 1 358633 3.48 MB/sec execute 412 sec latency 27.193 ms 2024-04-26T01:56:28.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:28 smithi188 ceph-mon[32751]: pgmap v333: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.5 MiB/s wr, 207 op/s 2024-04-26T01:56:28.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:28 smithi029 ceph-mon[32775]: pgmap v333: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.5 MiB/s wr, 207 op/s 2024-04-26T01:56:29.621 INFO:tasks.workunit.client.0.smithi006.stdout: 1 359391 3.48 MB/sec execute 413 sec latency 19.025 ms 2024-04-26T01:56:30.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:30 smithi006 ceph-mon[28419]: pgmap v334: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.0 MiB/s wr, 151 op/s 2024-04-26T01:56:30.621 INFO:tasks.workunit.client.0.smithi006.stdout: 1 360627 3.48 MB/sec execute 414 sec latency 27.904 ms 2024-04-26T01:56:30.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:30 smithi188 ceph-mon[32751]: pgmap v334: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.0 MiB/s wr, 151 op/s 2024-04-26T01:56:30.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:30 smithi029 ceph-mon[32775]: pgmap v334: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.0 MiB/s wr, 151 op/s 2024-04-26T01:56:31.621 INFO:tasks.workunit.client.0.smithi006.stdout: 1 361881 3.50 MB/sec execute 415 sec latency 27.950 ms 2024-04-26T01:56:32.621 INFO:tasks.workunit.client.0.smithi006.stdout: 1 362347 3.49 MB/sec execute 416 sec latency 28.303 ms 2024-04-26T01:56:32.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:32 smithi188 ceph-mon[32751]: pgmap v335: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.5 MiB/s wr, 208 op/s 2024-04-26T01:56:32.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:32 smithi029 ceph-mon[32775]: pgmap v335: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.5 MiB/s wr, 208 op/s 2024-04-26T01:56:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:32 smithi006 ceph-mon[28419]: pgmap v335: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.5 MiB/s wr, 208 op/s 2024-04-26T01:56:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.554+0000 7f38a9000640 1 -- 172.21.15.6:0/1965435315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a410e5b0 msgr2=0x7f38a4110a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.554+0000 7f38a9000640 1 --2- 172.21.15.6:0/1965435315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a410e5b0 0x7f38a4110a00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f389c009ca0 tx=0x7f389c02a950 comp rx=0 tx=0).stop 2024-04-26T01:56:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.554+0000 7f38a9000640 1 -- 172.21.15.6:0/1965435315 shutdown_connections 2024-04-26T01:56:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.554+0000 7f38a9000640 1 --2- 172.21.15.6:0/1965435315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a410e5b0 0x7f38a4110a00 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.554+0000 7f38a9000640 1 --2- 172.21.15.6:0/1965435315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38a4074ca0 0x7f38a410e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.554+0000 7f38a9000640 1 --2- 172.21.15.6:0/1965435315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38a4074380 0x7f38a4074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.554+0000 7f38a9000640 1 -- 172.21.15.6:0/1965435315 >> 172.21.15.6:0/1965435315 conn(0x7f38a40fd860 msgr2=0x7f38a40ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.554+0000 7f38a9000640 1 -- 172.21.15.6:0/1965435315 shutdown_connections 2024-04-26T01:56:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.554+0000 7f38a9000640 1 -- 172.21.15.6:0/1965435315 wait complete. 2024-04-26T01:56:33.555 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.555+0000 7f38a9000640 1 Processor -- start 2024-04-26T01:56:33.555 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.555+0000 7f38a9000640 1 -- start start 2024-04-26T01:56:33.555 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.555+0000 7f38a9000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38a4074380 0x7f38a419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.555 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38a4074ca0 0x7f38a419ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.555 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a9000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a410e5b0 0x7f38a419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.555 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a9000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f38a4114230 con 0x7f38a4074ca0 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a9000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f38a41140b0 con 0x7f38a410e5b0 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a9000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f38a41143b0 con 0x7f38a4074380 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f389aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38a4074ca0 0x7f38a419ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38a4074380 0x7f38a419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a410e5b0 0x7f38a419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f389aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38a4074ca0 0x7f38a419ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51030/0 (socket says 172.21.15.6:51030) 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a410e5b0 0x7f38a419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43292/0 (socket says 172.21.15.6:43292) 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38a4074380 0x7f38a419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:60902/0 (socket says 172.21.15.6:60902) 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f389aa00640 1 -- 172.21.15.6:0/4054488652 learned_addr learned my addr 172.21.15.6:0/4054488652 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a3e00640 1 -- 172.21.15.6:0/4054488652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38a4074380 msgr2=0x7f38a419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a3e00640 1 --2- 172.21.15.6:0/4054488652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38a4074380 0x7f38a419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a3e00640 1 -- 172.21.15.6:0/4054488652 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38a4074ca0 msgr2=0x7f38a419ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a3e00640 1 --2- 172.21.15.6:0/4054488652 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38a4074ca0 0x7f38a419ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.556+0000 7f38a3e00640 1 -- 172.21.15.6:0/4054488652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a419f6f0 con 0x7f38a410e5b0 2024-04-26T01:56:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.557+0000 7f38a3e00640 1 --2- 172.21.15.6:0/4054488652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a410e5b0 0x7f38a419efc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f389c005e30 tx=0x7f389c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.557+0000 7f38a0c00640 1 -- 172.21.15.6:0/4054488652 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c034040 con 0x7f38a410e5b0 2024-04-26T01:56:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.557+0000 7f38a0c00640 1 -- 172.21.15.6:0/4054488652 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f389c02fd20 con 0x7f38a410e5b0 2024-04-26T01:56:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.557+0000 7f38a0c00640 1 -- 172.21.15.6:0/4054488652 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c004540 con 0x7f38a410e5b0 2024-04-26T01:56:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.557+0000 7f38a9000640 1 -- 172.21.15.6:0/4054488652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a419f980 con 0x7f38a410e5b0 2024-04-26T01:56:33.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.557+0000 7f38a9000640 1 -- 172.21.15.6:0/4054488652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a41a7270 con 0x7f38a410e5b0 2024-04-26T01:56:33.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.560+0000 7f38a0c00640 1 -- 172.21.15.6:0/4054488652 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f389c037020 con 0x7f38a410e5b0 2024-04-26T01:56:33.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.561+0000 7f38ad5bc740 1 Processor -- start 2024-04-26T01:56:33.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.561+0000 7f38ad5bc740 1 -- start start 2024-04-26T01:56:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.561+0000 7f38ad5bc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f87cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.561+0000 7f38ad5bc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f887650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38ad5bc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f889f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38ad5bc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55848f7e2900 con 0x55848f87c9b0 2024-04-26T01:56:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38ad5bc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55848f788ef0 con 0x55848f87d2d0 2024-04-26T01:56:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38ad5bc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55848f7de8c0 con 0x55848f887b90 2024-04-26T01:56:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38a3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f887650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38a3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f887650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43300/0 (socket says 172.21.15.6:43300) 2024-04-26T01:56:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38a3400640 1 -- 172.21.15.6:0/3026103703 learned_addr learned my addr 172.21.15.6:0/3026103703 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f389aa00640 1 --2- 172.21.15.6:0/3026103703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f87cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f389aa00640 1 --2- 172.21.15.6:0/3026103703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f889f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38a0c00640 1 --2- 172.21.15.6:0/4054488652 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38880a3d80 0x7f38880a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38a3400640 1 -- 172.21.15.6:0/3026103703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 msgr2=0x55848f889f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38a3400640 1 --2- 172.21.15.6:0/3026103703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f889f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38a3400640 1 -- 172.21.15.6:0/3026103703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 msgr2=0x55848f87cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38a3400640 1 --2- 172.21.15.6:0/3026103703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f87cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.562+0000 7f38a3400640 1 -- 172.21.15.6:0/3026103703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55848f4f5210 con 0x55848f87d2d0 2024-04-26T01:56:33.563 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.563+0000 7f389aa00640 1 --2- 172.21.15.6:0/3026103703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f87cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.563+0000 7f38a0c00640 1 -- 172.21.15.6:0/4054488652 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8600+0+0 (secure 0 0 0) 0x7f389c0f1450 con 0x7f38a410e5b0 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.563+0000 7f38a3e00640 1 --2- 172.21.15.6:0/4054488652 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38880a3d80 0x7f38880a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.563+0000 7f38a3400640 1 --2- 172.21.15.6:0/3026103703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f887650 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3894009cf0 tx=0x7f3894011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=655743cb72ec6b85 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.563+0000 7f3898c00640 1 -- 172.21.15.6:0/3026103703 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3894004230 con 0x55848f87d2d0 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.563+0000 7f3898c00640 1 -- 172.21.15.6:0/3026103703 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38940043d0 con 0x55848f87d2d0 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.563+0000 7f3898c00640 1 -- 172.21.15.6:0/3026103703 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3894019880 con 0x55848f87d2d0 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.564+0000 7f38ad5bc740 1 -- 172.21.15.6:0/3026103703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 msgr2=0x55848f887650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.564+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/3026103703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f887650 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3894009cf0 tx=0x7f3894011810 comp rx=0 tx=0).stop 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.564+0000 7f38a3e00640 1 --2- 172.21.15.6:0/4054488652 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38880a3d80 0x7f38880a6240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f389c0c5b90 tx=0x7f389c0095c0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.564+0000 7f38ad5bc740 1 -- 172.21.15.6:0/3026103703 shutdown_connections 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.564+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/3026103703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f889f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.564+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/3026103703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f887650 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.564+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/3026103703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f87cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.565 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.564+0000 7f38ad5bc740 1 -- 172.21.15.6:0/3026103703 >> 172.21.15.6:0/3026103703 conn(0x55848f878ec0 msgr2=0x55848f8792a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:33.565 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.565+0000 7f38ad5bc740 1 -- 172.21.15.6:0/3026103703 shutdown_connections 2024-04-26T01:56:33.565 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.565+0000 7f38ad5bc740 1 -- 172.21.15.6:0/3026103703 wait complete. 2024-04-26T01:56:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.566+0000 7f38ad5bc740 1 -- start start 2024-04-26T01:56:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.566+0000 7f38ad5bc740 1 Processor -- start 2024-04-26T01:56:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.567+0000 7f38ad5bc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87c9b0 0x55848f884aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.567+0000 7f38ad5bc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87d2d0 0x55848f884fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.567+0000 7f38ad5bc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f885520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.567+0000 7f38ad5bc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55848f88a7c0 con 0x55848f87d2d0 2024-04-26T01:56:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.567+0000 7f38ad5bc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55848f79c310 con 0x55848f87c9b0 2024-04-26T01:56:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.567+0000 7f38ad5bc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55848f88a940 con 0x55848f887b90 2024-04-26T01:56:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.567+0000 7f389aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87c9b0 0x55848f884aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.567+0000 7f389aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87c9b0 0x55848f884aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43308/0 (socket says 172.21.15.6:43308) 2024-04-26T01:56:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.567+0000 7f389aa00640 1 -- 172.21.15.6:0/358239593 learned_addr learned my addr 172.21.15.6:0/358239593 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.567+0000 7f38ad5bc740 10 client.0 _ll_register_callbacks cb 0x55848f76acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.567+0000 7f389aa00640 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f885520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.568+0000 7f389aa00640 1 -- 172.21.15.6:0/358239593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 msgr2=0x55848f885520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.568+0000 7f389aa00640 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f885520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.568+0000 7f389aa00640 1 -- 172.21.15.6:0/358239593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87d2d0 msgr2=0x55848f884fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.568+0000 7f38a3400640 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87d2d0 0x55848f884fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.568+0000 7f389aa00640 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87d2d0 0x55848f884fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.568+0000 7f389aa00640 1 -- 172.21.15.6:0/358239593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55848f5c1620 con 0x55848f87c9b0 2024-04-26T01:56:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.568+0000 7f38a3400640 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87d2d0 0x55848f884fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:56:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.568+0000 7f389aa00640 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87c9b0 0x55848f884aa0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f388c009fa0 tx=0x7f388c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b91548716458caa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.568+0000 7f3891600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:56:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.569+0000 7f3891600640 1 -- 172.21.15.6:0/358239593 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388c0120f0 con 0x55848f87c9b0 2024-04-26T01:56:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.569+0000 7f3891600640 1 -- 172.21.15.6:0/358239593 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8600+0+0 (secure 0 0 0) 0x7f388c005020 con 0x55848f87c9b0 2024-04-26T01:56:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.569+0000 7f38ad5bc740 1 -- 172.21.15.6:0/358239593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38940043d0 con 0x55848f87c9b0 2024-04-26T01:56:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.569+0000 7f3891600640 1 -- 172.21.15.6:0/358239593 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f388c012290 con 0x55848f87c9b0 2024-04-26T01:56:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.569+0000 7f38ad5bc740 10 client.26465 fetch_fsmap learned FSMap version 43 2024-04-26T01:56:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.569+0000 7f38ad5bc740 1 -- 172.21.15.6:0/358239593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55848f896a40 con 0x55848f87c9b0 2024-04-26T01:56:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.569+0000 7f3891600640 1 -- 172.21.15.6:0/358239593 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38940043d0 con 0x55848f87c9b0 2024-04-26T01:56:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.570+0000 7f3891600640 1 -- 172.21.15.6:0/358239593 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f388c020460 con 0x55848f87c9b0 2024-04-26T01:56:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.570+0000 7f38ad5bc740 10 client.26465 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:56:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.570+0000 7f38ad5bc740 10 client.26465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.570+0000 7f38ad5bc740 20 client.26465 populate_metadata read hostname 'smithi006' 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.570+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55848f897f00 0x55848f89a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.570+0000 7f38ad5bc740 4 client.26465 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.570+0000 7f38ad5bc740 1 -- 172.21.15.6:0/358239593 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55848f7f68b0 con 0x55848f897f00 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.570+0000 7f38a3400640 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55848f897f00 0x55848f89a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.571+0000 7f38a3400640 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55848f897f00 0x55848f89a310 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5bc0144fd052c8 server_cookie=29f257f3230f7391 in_seq=0 out_seq=0 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.571+0000 7f3891600640 10 client.26465 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.571+0000 7f3891600640 1 -- 172.21.15.6:0/358239593 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55848f7f68b0 con 0x55848f897f00 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.571+0000 7f3891600640 10 client.26465 handle_command_reply: tid=0 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.571+0000 7f38ad5bc740 1 client.26465 shutdown 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.572+0000 7f38ad5bc740 1 -- 172.21.15.6:0/358239593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87c9b0 msgr2=0x55848f884aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.572+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87c9b0 0x55848f884aa0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f388c009fa0 tx=0x7f388c004730 comp rx=0 tx=0).stop 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.572+0000 7f38ad5bc740 1 -- 172.21.15.6:0/358239593 shutdown_connections 2024-04-26T01:56:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.572+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55848f897f00 0x55848f89a310 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.572+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87d2d0 0x55848f884fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.572+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f885520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.572+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/358239593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87c9b0 0x55848f884aa0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.572+0000 7f38ad5bc740 1 -- 172.21.15.6:0/358239593 >> 172.21.15.6:0/358239593 conn(0x55848f878ec0 msgr2=0x55848f87ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.572+0000 7f3891600640 0 client.26465 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.572+0000 7f38ad5bc740 1 -- 172.21.15.6:0/358239593 shutdown_connections 2024-04-26T01:56:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.572+0000 7f38ad5bc740 1 -- 172.21.15.6:0/358239593 wait complete. 2024-04-26T01:56:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.572+0000 7f38ad5bc740 20 client.26465 trim_cache size 0 max 16384 2024-04-26T01:56:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.589+0000 7f38ad5bc740 1 Processor -- start 2024-04-26T01:56:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.590+0000 7f38ad5bc740 1 -- start start 2024-04-26T01:56:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.590+0000 7f38ad5bc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f89a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.590+0000 7f38ad5bc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f893680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.590+0000 7f38ad5bc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f89fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.590+0000 7f38ad5bc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f38a41b25e0 con 0x55848f87c9b0 2024-04-26T01:56:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.590+0000 7f38ad5bc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f38a41afeb0 con 0x55848f87d2d0 2024-04-26T01:56:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.590+0000 7f38ad5bc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f38a41add20 con 0x55848f887b90 2024-04-26T01:56:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.590+0000 7f38a3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f893680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.590+0000 7f389aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f89a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.590+0000 7f38a3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f893680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43316/0 (socket says 172.21.15.6:43316) 2024-04-26T01:56:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.590+0000 7f38a3400640 1 -- 172.21.15.6:0/577487095 learned_addr learned my addr 172.21.15.6:0/577487095 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.590+0000 7f389aa00640 1 --2- 172.21.15.6:0/577487095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f89fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.591+0000 7f38a3400640 1 -- 172.21.15.6:0/577487095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 msgr2=0x55848f89fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.591+0000 7f38a3400640 1 --2- 172.21.15.6:0/577487095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f89fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.591+0000 7f38a3400640 1 -- 172.21.15.6:0/577487095 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 msgr2=0x55848f89a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.591+0000 7f38a3400640 1 --2- 172.21.15.6:0/577487095 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f89a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.591+0000 7f38a3400640 1 -- 172.21.15.6:0/577487095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55848f882ff0 con 0x55848f87d2d0 2024-04-26T01:56:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.591+0000 7f38a3400640 1 --2- 172.21.15.6:0/577487095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f893680 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3894009250 tx=0x7f3894019420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4a3de0d7202d511 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.591+0000 7f3893e00640 1 -- 172.21.15.6:0/577487095 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3894031070 con 0x55848f87d2d0 2024-04-26T01:56:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.591+0000 7f3893e00640 1 -- 172.21.15.6:0/577487095 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3894005b90 con 0x55848f87d2d0 2024-04-26T01:56:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.591+0000 7f3893e00640 1 -- 172.21.15.6:0/577487095 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3894019490 con 0x55848f87d2d0 2024-04-26T01:56:33.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.592+0000 7f38ad5bc740 1 -- 172.21.15.6:0/577487095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 msgr2=0x55848f893680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:33.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.592+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/577487095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f893680 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3894009250 tx=0x7f3894019420 comp rx=0 tx=0).stop 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.592+0000 7f38ad5bc740 1 -- 172.21.15.6:0/577487095 shutdown_connections 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.592+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/577487095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f89fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.592+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/577487095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f893680 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.592+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/577487095 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f89a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.592+0000 7f38ad5bc740 1 -- 172.21.15.6:0/577487095 >> 172.21.15.6:0/577487095 conn(0x55848f878ec0 msgr2=0x55848f87fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.592+0000 7f38ad5bc740 1 -- 172.21.15.6:0/577487095 shutdown_connections 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.592+0000 7f38ad5bc740 1 -- 172.21.15.6:0/577487095 wait complete. 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.593+0000 7f38ad5bc740 1 -- start start 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.593+0000 7f38ad5bc740 1 Processor -- start 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38ad5bc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f88ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38ad5bc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f88d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38ad5bc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f88d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38ad5bc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f38a41ae370 con 0x55848f87c9b0 2024-04-26T01:56:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38ad5bc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f38a41af7e0 con 0x55848f87d2d0 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38ad5bc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f38a41aea40 con 0x55848f887b90 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38ad5bc740 10 client.0 _ll_register_callbacks cb 0x7f388c012990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38a3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f88d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f389aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f88ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38a3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f88d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:43322/0 (socket says 172.21.15.6:43322) 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38a3400640 1 -- 172.21.15.6:0/102489737 learned_addr learned my addr 172.21.15.6:0/102489737 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f389aa00640 1 --2- 172.21.15.6:0/102489737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f88d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38a3400640 1 -- 172.21.15.6:0/102489737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 msgr2=0x55848f88d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38a3400640 1 --2- 172.21.15.6:0/102489737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f88d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38a3400640 1 -- 172.21.15.6:0/102489737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 msgr2=0x55848f88ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38a3400640 1 --2- 172.21.15.6:0/102489737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f88ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.594+0000 7f38a3400640 1 -- 172.21.15.6:0/102489737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55848f88e040 con 0x55848f87d2d0 2024-04-26T01:56:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.595+0000 7f38a3400640 1 --2- 172.21.15.6:0/102489737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f88d300 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3894000c00 tx=0x7f3894009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25a3c3e62e95fc64 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.595+0000 7f3891600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:56:33.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.595+0000 7f3891600640 1 -- 172.21.15.6:0/102489737 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3894031030 con 0x55848f87d2d0 2024-04-26T01:56:33.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.595+0000 7f3891600640 1 -- 172.21.15.6:0/102489737 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8600+0+0 (secure 0 0 0) 0x7f3894005360 con 0x55848f87d2d0 2024-04-26T01:56:33.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.595+0000 7f38ad5bc740 1 -- 172.21.15.6:0/102489737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3894005b90 con 0x55848f87d2d0 2024-04-26T01:56:33.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.595+0000 7f3891600640 1 -- 172.21.15.6:0/102489737 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3894005560 con 0x55848f87d2d0 2024-04-26T01:56:33.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.595+0000 7f38ad5bc740 10 client.26471 fetch_fsmap learned FSMap version 43 2024-04-26T01:56:33.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.595+0000 7f38ad5bc740 1 -- 172.21.15.6:0/102489737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55848f8fe4e0 con 0x55848f87d2d0 2024-04-26T01:56:33.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.596+0000 7f3891600640 1 -- 172.21.15.6:0/102489737 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3894005b90 con 0x55848f87d2d0 2024-04-26T01:56:33.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.596+0000 7f3891600640 1 -- 172.21.15.6:0/102489737 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f38940198d0 con 0x55848f87d2d0 2024-04-26T01:56:33.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.596+0000 7f38ad5bc740 10 client.26471 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:56:33.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.596+0000 7f38ad5bc740 10 client.26471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:33.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.596+0000 7f38ad5bc740 20 client.26471 populate_metadata read hostname 'smithi006' 2024-04-26T01:56:33.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.596+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/102489737 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55848f8fea30 0x55848f900e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:33.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.596+0000 7f38ad5bc740 4 client.26471 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:33.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.596+0000 7f38ad5bc740 1 -- 172.21.15.6:0/102489737 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55848f7f0320 con 0x55848f8fea30 2024-04-26T01:56:33.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.596+0000 7f389aa00640 1 --2- 172.21.15.6:0/102489737 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55848f8fea30 0x55848f900e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:33.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.597+0000 7f389aa00640 1 --2- 172.21.15.6:0/102489737 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55848f8fea30 0x55848f900e50 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5bb9490a5423e95 server_cookie=e0a270e28ebc4187 in_seq=0 out_seq=0 2024-04-26T01:56:33.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.597+0000 7f3891600640 10 client.26471 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:33.622 INFO:tasks.workunit.client.0.smithi006.stdout: 1 363244 3.50 MB/sec execute 417 sec latency 30.263 ms 2024-04-26T01:56:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.647+0000 7f3891600640 1 -- 172.21.15.6:0/102489737 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55848f7f0320 con 0x55848f8fea30 2024-04-26T01:56:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.647+0000 7f3891600640 10 client.26471 handle_command_reply: tid=0 2024-04-26T01:56:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.647+0000 7f38ad5bc740 1 client.26471 shutdown 2024-04-26T01:56:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.647+0000 7f38ad5bc740 1 -- 172.21.15.6:0/102489737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 msgr2=0x55848f88d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.647+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/102489737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f88d300 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3894000c00 tx=0x7f3894009c40 comp rx=0 tx=0).stop 2024-04-26T01:56:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.647+0000 7f38ad5bc740 1 -- 172.21.15.6:0/102489737 shutdown_connections 2024-04-26T01:56:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.647+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/102489737 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55848f8fea30 0x55848f900e50 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.647+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/102489737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55848f887b90 0x55848f88d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.647+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/102489737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55848f87d2d0 0x55848f88d300 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.647+0000 7f38ad5bc740 1 --2- 172.21.15.6:0/102489737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55848f87c9b0 0x55848f88ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.647+0000 7f3891600640 0 client.26471 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.647+0000 7f38ad5bc740 1 -- 172.21.15.6:0/102489737 >> 172.21.15.6:0/102489737 conn(0x55848f878ec0 msgr2=0x55848f877790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.648+0000 7f38ad5bc740 1 -- 172.21.15.6:0/102489737 shutdown_connections 2024-04-26T01:56:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.648+0000 7f38ad5bc740 1 -- 172.21.15.6:0/102489737 wait complete. 2024-04-26T01:56:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.648+0000 7f38ad5bc740 20 client.26471 trim_cache size 0 max 16384 2024-04-26T01:56:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:56:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:56:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:56:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:56:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.649+0000 7f38a9000640 1 -- 172.21.15.6:0/4054488652 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38880a3d80 msgr2=0x7f38880a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.649+0000 7f38a9000640 1 --2- 172.21.15.6:0/4054488652 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38880a3d80 0x7f38880a6240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f389c0c5b90 tx=0x7f389c0095c0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.649+0000 7f38a9000640 1 -- 172.21.15.6:0/4054488652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a410e5b0 msgr2=0x7f38a419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.649+0000 7f38a9000640 1 --2- 172.21.15.6:0/4054488652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a410e5b0 0x7f38a419efc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f389c005e30 tx=0x7f389c003c10 comp rx=0 tx=0).stop 2024-04-26T01:56:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.649+0000 7f38a9000640 1 -- 172.21.15.6:0/4054488652 shutdown_connections 2024-04-26T01:56:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.649+0000 7f38a9000640 1 --2- 172.21.15.6:0/4054488652 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38880a3d80 0x7f38880a6240 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.649+0000 7f38a9000640 1 --2- 172.21.15.6:0/4054488652 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38a4074ca0 0x7f38a419ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.649+0000 7f38a9000640 1 --2- 172.21.15.6:0/4054488652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38a410e5b0 0x7f38a419efc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.649+0000 7f38a9000640 1 --2- 172.21.15.6:0/4054488652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38a4074380 0x7f38a419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.649+0000 7f38a9000640 1 -- 172.21.15.6:0/4054488652 >> 172.21.15.6:0/4054488652 conn(0x7f38a40fd860 msgr2=0x7f38a40fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.650+0000 7f38a9000640 1 -- 172.21.15.6:0/4054488652 shutdown_connections 2024-04-26T01:56:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:33.650+0000 7f38a9000640 1 -- 172.21.15.6:0/4054488652 wait complete. 2024-04-26T01:56:34.622 INFO:tasks.workunit.client.0.smithi006.stdout: 1 363991 3.49 MB/sec execute 418 sec latency 51.597 ms 2024-04-26T01:56:34.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:34 smithi188 ceph-mon[32751]: pgmap v336: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 947 KiB/s wr, 118 op/s 2024-04-26T01:56:34.690 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:34 smithi029 ceph-mon[32775]: pgmap v336: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 947 KiB/s wr, 118 op/s 2024-04-26T01:56:34.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:34 smithi006 ceph-mon[28419]: pgmap v336: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 947 KiB/s wr, 118 op/s 2024-04-26T01:56:35.622 INFO:tasks.workunit.client.0.smithi006.stdout: 1 365279 3.51 MB/sec execute 419 sec latency 17.386 ms 2024-04-26T01:56:36.623 INFO:tasks.workunit.client.0.smithi006.stdout: 1 365786 3.50 MB/sec execute 420 sec latency 17.812 ms 2024-04-26T01:56:36.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:36 smithi188 ceph-mon[32751]: pgmap v337: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 947 KiB/s wr, 118 op/s 2024-04-26T01:56:36.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:36 smithi029 ceph-mon[32775]: pgmap v337: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 947 KiB/s wr, 118 op/s 2024-04-26T01:56:36.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:36 smithi006 ceph-mon[28419]: pgmap v337: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 947 KiB/s wr, 118 op/s 2024-04-26T01:56:37.623 INFO:tasks.workunit.client.0.smithi006.stdout: 1 366765 3.51 MB/sec execute 421 sec latency 27.481 ms 2024-04-26T01:56:38.623 INFO:tasks.workunit.client.0.smithi006.stdout: 1 367116 3.50 MB/sec execute 422 sec latency 249.379 ms 2024-04-26T01:56:38.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:38 smithi188 ceph-mon[32751]: pgmap v338: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 194 op/s 2024-04-26T01:56:38.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:38 smithi029 ceph-mon[32775]: pgmap v338: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 194 op/s 2024-04-26T01:56:38.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:38 smithi006 ceph-mon[28419]: pgmap v338: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 194 op/s 2024-04-26T01:56:39.623 INFO:tasks.workunit.client.0.smithi006.stdout: 1 368269 3.51 MB/sec execute 423 sec latency 20.896 ms 2024-04-26T01:56:40.624 INFO:tasks.workunit.client.0.smithi006.stdout: 1 369265 3.52 MB/sec execute 424 sec latency 18.958 ms 2024-04-26T01:56:40.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:40 smithi188 ceph-mon[32751]: pgmap v339: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 855 KiB/s wr, 133 op/s 2024-04-26T01:56:40.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:40 smithi029 ceph-mon[32775]: pgmap v339: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 855 KiB/s wr, 133 op/s 2024-04-26T01:56:40.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:40 smithi006 ceph-mon[28419]: pgmap v339: 97 pgs: 97 active+clean; 117 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 855 KiB/s wr, 133 op/s 2024-04-26T01:56:41.623 INFO:tasks.workunit.client.0.smithi006.stdout: 1 370090 3.51 MB/sec execute 425 sec latency 32.253 ms 2024-04-26T01:56:42.623 INFO:tasks.workunit.client.0.smithi006.stdout: 1 371181 3.52 MB/sec execute 426 sec latency 30.215 ms 2024-04-26T01:56:42.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:42 smithi188 ceph-mon[32751]: pgmap v340: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2024-04-26T01:56:42.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:42 smithi029 ceph-mon[32775]: pgmap v340: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2024-04-26T01:56:42.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:42 smithi006 ceph-mon[28419]: pgmap v340: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2024-04-26T01:56:43.623 INFO:tasks.workunit.client.0.smithi006.stdout: 1 372551 3.53 MB/sec execute 427 sec latency 19.261 ms 2024-04-26T01:56:44.624 INFO:tasks.workunit.client.0.smithi006.stdout: 1 373098 3.53 MB/sec execute 428 sec latency 32.382 ms 2024-04-26T01:56:44.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:44 smithi188 ceph-mon[32751]: pgmap v341: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.0 MiB/s wr, 140 op/s 2024-04-26T01:56:44.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:44 smithi029 ceph-mon[32775]: pgmap v341: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.0 MiB/s wr, 140 op/s 2024-04-26T01:56:44.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:44 smithi006 ceph-mon[28419]: pgmap v341: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.0 MiB/s wr, 140 op/s 2024-04-26T01:56:45.624 INFO:tasks.workunit.client.0.smithi006.stdout: 1 374066 3.53 MB/sec execute 429 sec latency 66.740 ms 2024-04-26T01:56:46.624 INFO:tasks.workunit.client.0.smithi006.stdout: 1 375419 3.54 MB/sec execute 430 sec latency 24.096 ms 2024-04-26T01:56:46.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:46 smithi188 ceph-mon[32751]: pgmap v342: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.0 MiB/s wr, 140 op/s 2024-04-26T01:56:46.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:46 smithi029 ceph-mon[32775]: pgmap v342: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.0 MiB/s wr, 140 op/s 2024-04-26T01:56:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.808+0000 7fa4d6400640 1 -- 172.21.15.6:0/1514070004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4d0076da0 msgr2=0x7fa4d01126f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.808+0000 7fa4d6400640 1 --2- 172.21.15.6:0/1514070004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4d0076da0 0x7fa4d01126f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009930 tx=0x7fa4bc02a420 comp rx=0 tx=0).stop 2024-04-26T01:56:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.808+0000 7fa4d6400640 1 -- 172.21.15.6:0/1514070004 shutdown_connections 2024-04-26T01:56:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.808+0000 7fa4d6400640 1 --2- 172.21.15.6:0/1514070004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4d0112c30 0x7fa4d0115020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.808+0000 7fa4d6400640 1 --2- 172.21.15.6:0/1514070004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4d0076da0 0x7fa4d01126f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.808+0000 7fa4d6400640 1 --2- 172.21.15.6:0/1514070004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d0076480 0x7fa4d0076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.808+0000 7fa4d6400640 1 -- 172.21.15.6:0/1514070004 >> 172.21.15.6:0/1514070004 conn(0x7fa4d00ffa60 msgr2=0x7fa4d0101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.808+0000 7fa4d6400640 1 -- 172.21.15.6:0/1514070004 shutdown_connections 2024-04-26T01:56:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.808+0000 7fa4d6400640 1 -- 172.21.15.6:0/1514070004 wait complete. 2024-04-26T01:56:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.809+0000 7fa4d6400640 1 Processor -- start 2024-04-26T01:56:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.810+0000 7fa4d6400640 1 -- start start 2024-04-26T01:56:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.810+0000 7fa4d6400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4d0076480 0x7fa4d019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.810+0000 7fa4d6400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d0076da0 0x7fa4d019f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.810+0000 7fa4d6400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4d0112c30 0x7fa4d01a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.810+0000 7fa4d6400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0118680 con 0x7fa4d0076480 2024-04-26T01:56:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.810+0000 7fa4d6400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0118500 con 0x7fa4d0076da0 2024-04-26T01:56:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.810+0000 7fa4d6400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0118800 con 0x7fa4d0112c30 2024-04-26T01:56:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.810+0000 7fa4d5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4d0076480 0x7fa4d019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.810+0000 7fa4cfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d0076da0 0x7fa4d019f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.811+0000 7fa4d5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4d0076480 0x7fa4d019eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33104/0 (socket says 172.21.15.6:33104) 2024-04-26T01:56:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.811+0000 7fa4d5000640 1 -- 172.21.15.6:0/3049917783 learned_addr learned my addr 172.21.15.6:0/3049917783 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.811+0000 7fa4d5a00640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4d0112c30 0x7fa4d01a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.811+0000 7fa4cfe00640 1 -- 172.21.15.6:0/3049917783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4d0112c30 msgr2=0x7fa4d01a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.811+0000 7fa4cfe00640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4d0112c30 0x7fa4d01a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.811+0000 7fa4cfe00640 1 -- 172.21.15.6:0/3049917783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4d0076480 msgr2=0x7fa4d019eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.811+0000 7fa4cfe00640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4d0076480 0x7fa4d019eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.811+0000 7fa4cfe00640 1 -- 172.21.15.6:0/3049917783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d01a3ed0 con 0x7fa4d0076da0 2024-04-26T01:56:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.811+0000 7fa4cfe00640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d0076da0 0x7fa4d019f420 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc02a940 tx=0x7fa4bc003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.812+0000 7fa4cd600640 1 -- 172.21.15.6:0/3049917783 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc03f070 con 0x7fa4d0076da0 2024-04-26T01:56:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.812+0000 7fa4cd600640 1 -- 172.21.15.6:0/3049917783 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4bc031b60 con 0x7fa4d0076da0 2024-04-26T01:56:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.812+0000 7fa4d6400640 1 -- 172.21.15.6:0/3049917783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d01a4160 con 0x7fa4d0076da0 2024-04-26T01:56:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.812+0000 7fa4cd600640 1 -- 172.21.15.6:0/3049917783 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc035030 con 0x7fa4d0076da0 2024-04-26T01:56:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.812+0000 7fa4d6400640 1 -- 172.21.15.6:0/3049917783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d01aba50 con 0x7fa4d0076da0 2024-04-26T01:56:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.815+0000 7fa4cd600640 1 -- 172.21.15.6:0/3049917783 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fa4bc042070 con 0x7fa4d0076da0 2024-04-26T01:56:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.816+0000 7fa4da872740 1 Processor -- start 2024-04-26T01:56:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.817+0000 7fa4da872740 1 -- start start 2024-04-26T01:56:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.817+0000 7fa4da872740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17be9b0 0x55e5d17bed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.817+0000 7fa4da872740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.817+0000 7fa4da872740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 0x55e5d17cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.817+0000 7fa4da872740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55e5d1724900 con 0x55e5d17c9b90 2024-04-26T01:56:46.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.817+0000 7fa4da872740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e5d16caef0 con 0x55e5d17bf2d0 2024-04-26T01:56:46.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.817+0000 7fa4da872740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e5d17208c0 con 0x55e5d17be9b0 2024-04-26T01:56:46.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.817+0000 7fa4d5000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:46 smithi006 ceph-mon[28419]: pgmap v342: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.0 MiB/s wr, 140 op/s 2024-04-26T01:56:46.825 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.817+0000 7fa4d5a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17be9b0 0x55e5d17bed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.826 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.817+0000 7fa4d5000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:42716/0 (socket says 172.21.15.6:42716) 2024-04-26T01:56:46.826 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.818+0000 7fa4d5000640 1 -- 172.21.15.6:0/2656054004 learned_addr learned my addr 172.21.15.6:0/2656054004 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:46.826 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.817+0000 7fa4cfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 0x55e5d17cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.826 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.818+0000 7fa4cd600640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa4b80a3d80 0x7fa4b80a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.826 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.818+0000 7fa4cd600640 1 -- 172.21.15.6:0/3049917783 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8600+0+0 (secure 0 0 0) 0x7fa4bc0f0cc0 con 0x7fa4d0076da0 2024-04-26T01:56:46.826 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.818+0000 7fa4d5000640 1 -- 172.21.15.6:0/2656054004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17be9b0 msgr2=0x55e5d17bed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:46.826 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.818+0000 7fa4d5000640 1 --2- 172.21.15.6:0/2656054004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17be9b0 0x55e5d17bed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.826 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.818+0000 7fa4d5000640 1 -- 172.21.15.6:0/2656054004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 msgr2=0x55e5d17cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:46.826 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.818+0000 7fa4d5000640 1 --2- 172.21.15.6:0/2656054004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 0x55e5d17cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.818+0000 7fa4d5000640 1 -- 172.21.15.6:0/2656054004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5d1437210 con 0x55e5d17bf2d0 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.818+0000 7fa4d5000640 1 --2- 172.21.15.6:0/2656054004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c9650 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4009dc0 tx=0x7fa4c4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb970cac052981ec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.819+0000 7fa4d5a00640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa4b80a3d80 0x7fa4b80a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.819+0000 7fa4c2000640 1 -- 172.21.15.6:0/2656054004 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4005220 con 0x55e5d17bf2d0 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.819+0000 7fa4c2000640 1 -- 172.21.15.6:0/2656054004 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4c4004c50 con 0x55e5d17bf2d0 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.819+0000 7fa4c2000640 1 -- 172.21.15.6:0/2656054004 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4019890 con 0x55e5d17bf2d0 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.820+0000 7fa4da872740 1 -- 172.21.15.6:0/2656054004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 msgr2=0x55e5d17c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.820+0000 7fa4da872740 1 --2- 172.21.15.6:0/2656054004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c9650 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4009dc0 tx=0x7fa4c4011810 comp rx=0 tx=0).stop 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.820+0000 7fa4d5a00640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa4b80a3d80 0x7fa4b80a6240 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8005c40 tx=0x7fa4c8004ab0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.820+0000 7fa4da872740 1 -- 172.21.15.6:0/2656054004 shutdown_connections 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.820+0000 7fa4da872740 1 --2- 172.21.15.6:0/2656054004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 0x55e5d17cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.820+0000 7fa4da872740 1 --2- 172.21.15.6:0/2656054004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c9650 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.827 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.820+0000 7fa4da872740 1 --2- 172.21.15.6:0/2656054004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17be9b0 0x55e5d17bed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.820+0000 7fa4da872740 1 -- 172.21.15.6:0/2656054004 >> 172.21.15.6:0/2656054004 conn(0x55e5d17baec0 msgr2=0x55e5d17bb2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.820+0000 7fa4da872740 1 -- 172.21.15.6:0/2656054004 shutdown_connections 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.820+0000 7fa4da872740 1 -- 172.21.15.6:0/2656054004 wait complete. 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.821+0000 7fa4da872740 1 -- start start 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.821+0000 7fa4da872740 1 Processor -- start 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.821+0000 7fa4da872740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17be9b0 0x55e5d17c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.821+0000 7fa4da872740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4da872740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17c9b90 0x55e5d17c7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4da872740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55e5d17cc7c0 con 0x55e5d17be9b0 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4da872740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e5d16de310 con 0x55e5d17bf2d0 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4da872740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e5d17cc940 con 0x55e5d17c9b90 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4da872740 10 client.0 _ll_register_callbacks cb 0x55e5d16accb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4cfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.828 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4d5a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17c9b90 0x55e5d17c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4cfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:42718/0 (socket says 172.21.15.6:42718) 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4cfe00640 1 -- 172.21.15.6:0/135455074 learned_addr learned my addr 172.21.15.6:0/135455074 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4d5a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17c9b90 0x55e5d17c7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59104/0 (socket says 172.21.15.6:59104) 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4cfe00640 1 -- 172.21.15.6:0/135455074 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17c9b90 msgr2=0x55e5d17c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4cfe00640 1 --2- 172.21.15.6:0/135455074 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17c9b90 0x55e5d17c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4cfe00640 1 -- 172.21.15.6:0/135455074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17be9b0 msgr2=0x55e5d17c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4cfe00640 1 --2- 172.21.15.6:0/135455074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17be9b0 0x55e5d17c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.822+0000 7fa4cfe00640 1 -- 172.21.15.6:0/135455074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5d1503620 con 0x55e5d17bf2d0 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.823+0000 7fa4cfe00640 1 --2- 172.21.15.6:0/135455074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c7000 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0bc960 tx=0x7fa4bc0bc7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c7919ce7337e91d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.823+0000 7fa4b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.823+0000 7fa4b6a00640 1 -- 172.21.15.6:0/135455074 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc049070 con 0x55e5d17bf2d0 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.823+0000 7fa4b6a00640 1 -- 172.21.15.6:0/135455074 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8600+0+0 (secure 0 0 0) 0x7fa4bc0dd0b0 con 0x55e5d17bf2d0 2024-04-26T01:56:46.829 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.823+0000 7fa4da872740 1 -- 172.21.15.6:0/135455074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4c4004c50 con 0x55e5d17bf2d0 2024-04-26T01:56:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.823+0000 7fa4b6a00640 1 -- 172.21.15.6:0/135455074 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fa4bc0c0f60 con 0x55e5d17bf2d0 2024-04-26T01:56:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.823+0000 7fa4da872740 10 client.26483 fetch_fsmap learned FSMap version 43 2024-04-26T01:56:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.823+0000 7fa4da872740 1 -- 172.21.15.6:0/135455074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55e5d17d89c0 con 0x55e5d17bf2d0 2024-04-26T01:56:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.824+0000 7fa4b6a00640 1 -- 172.21.15.6:0/135455074 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4c4004c50 con 0x55e5d17bf2d0 2024-04-26T01:56:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.824+0000 7fa4b6a00640 1 -- 172.21.15.6:0/135455074 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7fa4bc0ee670 con 0x55e5d17bf2d0 2024-04-26T01:56:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.824+0000 7fa4da872740 10 client.26483 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:56:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.824+0000 7fa4da872740 10 client.26483 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.824+0000 7fa4da872740 20 client.26483 populate_metadata read hostname 'smithi006' 2024-04-26T01:56:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.824+0000 7fa4da872740 1 --2- 172.21.15.6:0/135455074 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e5d17d9e70 0x55e5d17dc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.824+0000 7fa4da872740 4 client.26483 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.824+0000 7fa4da872740 1 -- 172.21.15.6:0/135455074 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5d17388b0 con 0x55e5d17d9e70 2024-04-26T01:56:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.824+0000 7fa4d5000640 1 --2- 172.21.15.6:0/135455074 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e5d17d9e70 0x55e5d17dc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.825+0000 7fa4d5000640 1 --2- 172.21.15.6:0/135455074 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e5d17d9e70 0x55e5d17dc2e0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e08dd3a6fc511a19 server_cookie=2d21c0ea517a74e1 in_seq=0 out_seq=0 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.825+0000 7fa4b6a00640 10 client.26483 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4b6a00640 1 -- 172.21.15.6:0/135455074 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5d17388b0 con 0x55e5d17d9e70 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4b6a00640 10 client.26483 handle_command_reply: tid=0 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4da872740 1 client.26483 shutdown 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4da872740 1 -- 172.21.15.6:0/135455074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 msgr2=0x55e5d17c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4da872740 1 --2- 172.21.15.6:0/135455074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c7000 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0bc960 tx=0x7fa4bc0bc7a0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4da872740 1 -- 172.21.15.6:0/135455074 shutdown_connections 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4da872740 1 --2- 172.21.15.6:0/135455074 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e5d17d9e70 0x55e5d17dc2e0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4da872740 1 --2- 172.21.15.6:0/135455074 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17c9b90 0x55e5d17c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4da872740 1 --2- 172.21.15.6:0/135455074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c7000 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4da872740 1 --2- 172.21.15.6:0/135455074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17be9b0 0x55e5d17c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4da872740 1 -- 172.21.15.6:0/135455074 >> 172.21.15.6:0/135455074 conn(0x55e5d17baec0 msgr2=0x55e5d17c0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4b6a00640 0 client.26483 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:46.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4da872740 1 -- 172.21.15.6:0/135455074 shutdown_connections 2024-04-26T01:56:46.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.826+0000 7fa4da872740 1 -- 172.21.15.6:0/135455074 wait complete. 2024-04-26T01:56:46.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.827+0000 7fa4da872740 20 client.26483 trim_cache size 0 max 16384 2024-04-26T01:56:46.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.844+0000 7fa4da872740 1 Processor -- start 2024-04-26T01:56:46.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.844+0000 7fa4da872740 1 -- start start 2024-04-26T01:56:46.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.844+0000 7fa4da872740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17be9b0 0x55e5d17d5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.844+0000 7fa4da872740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.844+0000 7fa4da872740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 0x55e5d17c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.844+0000 7fa4da872740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b6a00 con 0x55e5d17c9b90 2024-04-26T01:56:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.844+0000 7fa4da872740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b42d0 con 0x55e5d17be9b0 2024-04-26T01:56:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.844+0000 7fa4da872740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b2140 con 0x55e5d17bf2d0 2024-04-26T01:56:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.844+0000 7fa4cfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.844+0000 7fa4d5a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 0x55e5d17c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.845+0000 7fa4cfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59112/0 (socket says 172.21.15.6:59112) 2024-04-26T01:56:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.845+0000 7fa4cfe00640 1 -- 172.21.15.6:0/4090107586 learned_addr learned my addr 172.21.15.6:0/4090107586 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.845+0000 7fa4d5000640 1 --2- 172.21.15.6:0/4090107586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17be9b0 0x55e5d17d5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.845+0000 7fa4cfe00640 1 -- 172.21.15.6:0/4090107586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17be9b0 msgr2=0x55e5d17d5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.845+0000 7fa4cfe00640 1 --2- 172.21.15.6:0/4090107586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17be9b0 0x55e5d17d5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.845+0000 7fa4cfe00640 1 -- 172.21.15.6:0/4090107586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 msgr2=0x55e5d17c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.845+0000 7fa4cfe00640 1 --2- 172.21.15.6:0/4090107586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 0x55e5d17c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.845+0000 7fa4cfe00640 1 -- 172.21.15.6:0/4090107586 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5d17c5010 con 0x55e5d17bf2d0 2024-04-26T01:56:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.845+0000 7fa4cfe00640 1 --2- 172.21.15.6:0/4090107586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c1370 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0bcbf0 tx=0x7fa4bc0bb170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4850e11a4abbb81 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.845+0000 7fa4c1600640 1 -- 172.21.15.6:0/4090107586 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc049070 con 0x55e5d17bf2d0 2024-04-26T01:56:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.845+0000 7fa4c1600640 1 -- 172.21.15.6:0/4090107586 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4bc0d6070 con 0x55e5d17bf2d0 2024-04-26T01:56:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.845+0000 7fa4c1600640 1 -- 172.21.15.6:0/4090107586 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc0ee5e0 con 0x55e5d17bf2d0 2024-04-26T01:56:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.846+0000 7fa4da872740 1 -- 172.21.15.6:0/4090107586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 msgr2=0x55e5d17c1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.846+0000 7fa4da872740 1 --2- 172.21.15.6:0/4090107586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c1370 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0bcbf0 tx=0x7fa4bc0bb170 comp rx=0 tx=0).stop 2024-04-26T01:56:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.846+0000 7fa4da872740 1 -- 172.21.15.6:0/4090107586 shutdown_connections 2024-04-26T01:56:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.846+0000 7fa4da872740 1 --2- 172.21.15.6:0/4090107586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 0x55e5d17c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.846+0000 7fa4da872740 1 --2- 172.21.15.6:0/4090107586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 0x55e5d17c1370 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.846+0000 7fa4da872740 1 --2- 172.21.15.6:0/4090107586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17be9b0 0x55e5d17d5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.846+0000 7fa4da872740 1 -- 172.21.15.6:0/4090107586 >> 172.21.15.6:0/4090107586 conn(0x55e5d17baec0 msgr2=0x55e5d17c82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.846+0000 7fa4da872740 1 -- 172.21.15.6:0/4090107586 shutdown_connections 2024-04-26T01:56:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.846+0000 7fa4da872740 1 -- 172.21.15.6:0/4090107586 wait complete. 2024-04-26T01:56:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.847+0000 7fa4da872740 1 -- start start 2024-04-26T01:56:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.847+0000 7fa4da872740 1 Processor -- start 2024-04-26T01:56:46.850 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.847+0000 7fa4da872740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17be9b0 0x55e5d17cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.850 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4da872740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 0x55e5d17cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.850 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4da872740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 0x55e5d17d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.850 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4da872740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b2790 con 0x55e5d17c9b90 2024-04-26T01:56:46.850 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4da872740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b3c00 con 0x55e5d17be9b0 2024-04-26T01:56:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4da872740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01b2e60 con 0x55e5d17bf2d0 2024-04-26T01:56:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4da872740 10 client.0 _ll_register_callbacks cb 0x7fa4bc009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4cfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 0x55e5d17cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4d5000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17be9b0 0x55e5d17cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4cfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 0x55e5d17cfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59118/0 (socket says 172.21.15.6:59118) 2024-04-26T01:56:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4cfe00640 1 -- 172.21.15.6:0/139723046 learned_addr learned my addr 172.21.15.6:0/139723046 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:56:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4d5a00640 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 0x55e5d17d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4d5000640 1 -- 172.21.15.6:0/139723046 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 msgr2=0x55e5d17cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4d5000640 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 0x55e5d17cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4d5000640 1 -- 172.21.15.6:0/139723046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 msgr2=0x55e5d17d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4d5000640 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 0x55e5d17d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4d5000640 1 -- 172.21.15.6:0/139723046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5d183f5a0 con 0x55e5d17be9b0 2024-04-26T01:56:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.848+0000 7fa4cfe00640 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 0x55e5d17cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.849+0000 7fa4d5000640 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17be9b0 0x55e5d17cf700 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4000c00 tx=0x7fa4c4009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bc8e46554f253fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.849+0000 7fa4b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:56:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.849+0000 7fa4b6a00640 1 -- 172.21.15.6:0/139723046 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c40052c0 con 0x55e5d17be9b0 2024-04-26T01:56:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.849+0000 7fa4b6a00640 1 -- 172.21.15.6:0/139723046 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8600+0+0 (secure 0 0 0) 0x7fa4c402a1d0 con 0x55e5d17be9b0 2024-04-26T01:56:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.849+0000 7fa4da872740 1 -- 172.21.15.6:0/139723046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4bc0d6070 con 0x55e5d17be9b0 2024-04-26T01:56:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.849+0000 7fa4b6a00640 1 -- 172.21.15.6:0/139723046 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fa4c402a3d0 con 0x55e5d17be9b0 2024-04-26T01:56:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.849+0000 7fa4da872740 10 client.26489 fetch_fsmap learned FSMap version 43 2024-04-26T01:56:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.849+0000 7fa4da872740 1 -- 172.21.15.6:0/139723046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55e5d1840890 con 0x55e5d17be9b0 2024-04-26T01:56:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.849+0000 7fa4b6a00640 1 -- 172.21.15.6:0/139723046 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4bc0d6070 con 0x55e5d17be9b0 2024-04-26T01:56:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.850+0000 7fa4b6a00640 1 -- 172.21.15.6:0/139723046 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7fa4c4003940 con 0x55e5d17be9b0 2024-04-26T01:56:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.850+0000 7fa4da872740 10 client.26489 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:56:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.850+0000 7fa4da872740 10 client.26489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:56:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.850+0000 7fa4da872740 20 client.26489 populate_metadata read hostname 'smithi006' 2024-04-26T01:56:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.850+0000 7fa4da872740 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e5d1840d60 0x55e5d1843150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.850+0000 7fa4da872740 4 client.26489 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.850+0000 7fa4da872740 1 -- 172.21.15.6:0/139723046 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e5d1732320 con 0x55e5d1840d60 2024-04-26T01:56:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.850+0000 7fa4cfe00640 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e5d1840d60 0x55e5d1843150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.851+0000 7fa4cfe00640 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e5d1840d60 0x55e5d1843150 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb6e0327d96e2531 server_cookie=89ad557f7edd4e1f in_seq=0 out_seq=0 2024-04-26T01:56:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.851+0000 7fa4b6a00640 10 client.26489 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4b6a00640 1 -- 172.21.15.6:0/139723046 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e5d1732320 con 0x55e5d1840d60 2024-04-26T01:56:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4b6a00640 10 client.26489 handle_command_reply: tid=0 2024-04-26T01:56:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4da872740 1 client.26489 shutdown 2024-04-26T01:56:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4da872740 1 -- 172.21.15.6:0/139723046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17be9b0 msgr2=0x55e5d17cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4da872740 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17be9b0 0x55e5d17cf700 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4000c00 tx=0x7fa4c4009f70 comp rx=0 tx=0).stop 2024-04-26T01:56:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4da872740 1 -- 172.21.15.6:0/139723046 shutdown_connections 2024-04-26T01:56:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4da872740 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e5d1840d60 0x55e5d1843150 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4da872740 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5d17c9b90 0x55e5d17d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4da872740 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d17bf2d0 0x55e5d17cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4da872740 1 --2- 172.21.15.6:0/139723046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e5d17be9b0 0x55e5d17cf700 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4b6a00640 0 client.26489 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:56:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4da872740 1 -- 172.21.15.6:0/139723046 >> 172.21.15.6:0/139723046 conn(0x55e5d17baec0 msgr2=0x55e5d17c8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4da872740 1 -- 172.21.15.6:0/139723046 shutdown_connections 2024-04-26T01:56:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.907+0000 7fa4da872740 1 -- 172.21.15.6:0/139723046 wait complete. 2024-04-26T01:56:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.908+0000 7fa4da872740 20 client.26489 trim_cache size 0 max 16384 2024-04-26T01:56:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:56:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:56:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:56:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:56:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.908+0000 7fa4d6400640 1 -- 172.21.15.6:0/3049917783 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa4b80a3d80 msgr2=0x7fa4b80a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.908+0000 7fa4d6400640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa4b80a3d80 0x7fa4b80a6240 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8005c40 tx=0x7fa4c8004ab0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.908+0000 7fa4d6400640 1 -- 172.21.15.6:0/3049917783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d0076da0 msgr2=0x7fa4d019f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.908+0000 7fa4d6400640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d0076da0 0x7fa4d019f420 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc02a940 tx=0x7fa4bc003c10 comp rx=0 tx=0).stop 2024-04-26T01:56:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.909+0000 7fa4d6400640 1 -- 172.21.15.6:0/3049917783 shutdown_connections 2024-04-26T01:56:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.909+0000 7fa4d6400640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa4b80a3d80 0x7fa4b80a6240 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.909+0000 7fa4d6400640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4d0112c30 0x7fa4d01a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.909+0000 7fa4d6400640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d0076da0 0x7fa4d019f420 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.909+0000 7fa4d6400640 1 --2- 172.21.15.6:0/3049917783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4d0076480 0x7fa4d019eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.909+0000 7fa4d6400640 1 -- 172.21.15.6:0/3049917783 >> 172.21.15.6:0/3049917783 conn(0x7fa4d00ffa60 msgr2=0x7fa4d0101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.909+0000 7fa4d6400640 1 -- 172.21.15.6:0/3049917783 shutdown_connections 2024-04-26T01:56:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:56:46.909+0000 7fa4d6400640 1 -- 172.21.15.6:0/3049917783 wait complete. 2024-04-26T01:56:47.624 INFO:tasks.workunit.client.0.smithi006.stdout: 1 376366 3.55 MB/sec execute 431 sec latency 43.999 ms 2024-04-26T01:56:48.624 INFO:tasks.workunit.client.0.smithi006.stdout: 1 377154 3.54 MB/sec execute 432 sec latency 19.261 ms 2024-04-26T01:56:48.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:48 smithi188 ceph-mon[32751]: pgmap v343: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.8 MiB/s wr, 208 op/s 2024-04-26T01:56:48.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:48 smithi029 ceph-mon[32775]: pgmap v343: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.8 MiB/s wr, 208 op/s 2024-04-26T01:56:48.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:48 smithi006 ceph-mon[28419]: pgmap v343: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.8 MiB/s wr, 208 op/s 2024-04-26T01:56:49.624 INFO:tasks.workunit.client.0.smithi006.stdout: 1 378105 3.55 MB/sec execute 433 sec latency 25.797 ms 2024-04-26T01:56:50.624 INFO:tasks.workunit.client.0.smithi006.stdout: 1 379523 3.56 MB/sec execute 434 sec latency 26.453 ms 2024-04-26T01:56:50.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:50 smithi188 ceph-mon[32751]: pgmap v344: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-04-26T01:56:50.649 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:50 smithi006 ceph-mon[28419]: pgmap v344: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-04-26T01:56:50.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:50 smithi029 ceph-mon[32775]: pgmap v344: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.4 MiB/s wr, 132 op/s 2024-04-26T01:56:51.625 INFO:tasks.workunit.client.0.smithi006.stdout: 1 380132 3.56 MB/sec execute 435 sec latency 22.302 ms 2024-04-26T01:56:52.625 INFO:tasks.workunit.client.0.smithi006.stdout: 1 381255 3.56 MB/sec execute 436 sec latency 22.710 ms 2024-04-26T01:56:52.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:52 smithi188 ceph-mon[32751]: pgmap v345: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.1 MiB/s wr, 222 op/s 2024-04-26T01:56:52.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:52 smithi029 ceph-mon[32775]: pgmap v345: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.1 MiB/s wr, 222 op/s 2024-04-26T01:56:52.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:52 smithi006 ceph-mon[28419]: pgmap v345: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.1 MiB/s wr, 222 op/s 2024-04-26T01:56:53.625 INFO:tasks.workunit.client.0.smithi006.stdout: 1 382845 3.58 MB/sec execute 437 sec latency 25.529 ms 2024-04-26T01:56:54.625 INFO:tasks.workunit.client.0.smithi006.stdout: 1 383587 3.58 MB/sec execute 438 sec latency 20.985 ms 2024-04-26T01:56:54.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:54 smithi029 ceph-mon[32775]: pgmap v346: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 158 op/s 2024-04-26T01:56:54.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:54 smithi006 ceph-mon[28419]: pgmap v346: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 158 op/s 2024-04-26T01:56:54.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:54 smithi188 ceph-mon[32751]: pgmap v346: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 158 op/s 2024-04-26T01:56:55.625 INFO:tasks.workunit.client.0.smithi006.stdout: 1 384610 3.58 MB/sec execute 439 sec latency 65.551 ms 2024-04-26T01:56:56.625 INFO:tasks.workunit.client.0.smithi006.stdout: 1 385737 3.58 MB/sec execute 440 sec latency 26.412 ms 2024-04-26T01:56:56.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:56 smithi029 ceph-mon[32775]: pgmap v347: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 158 op/s 2024-04-26T01:56:56.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:56 smithi006 ceph-mon[28419]: pgmap v347: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 158 op/s 2024-04-26T01:56:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:56 smithi188 ceph-mon[32751]: pgmap v347: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 158 op/s 2024-04-26T01:56:57.625 INFO:tasks.workunit.client.0.smithi006.stdout: 1 386997 3.60 MB/sec execute 441 sec latency 21.202 ms 2024-04-26T01:56:58.625 INFO:tasks.workunit.client.0.smithi006.stdout: 1 387744 3.59 MB/sec execute 442 sec latency 19.711 ms 2024-04-26T01:56:58.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:56:58 smithi029 ceph-mon[32775]: pgmap v348: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 235 op/s 2024-04-26T01:56:58.692 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:56:58 smithi188 ceph-mon[32751]: pgmap v348: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 235 op/s 2024-04-26T01:56:58.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:56:58 smithi006 ceph-mon[28419]: pgmap v348: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 235 op/s 2024-04-26T01:56:59.625 INFO:tasks.workunit.client.0.smithi006.stdout: 1 388484 3.59 MB/sec execute 443 sec latency 24.272 ms 2024-04-26T01:57:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.073+0000 7f3be3e00640 1 -- 172.21.15.6:0/714041260 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be4100b10 msgr2=0x7f3be4100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.073+0000 7f3be3e00640 1 --2- 172.21.15.6:0/714041260 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be4100b10 0x7f3be4100ef0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009900 tx=0x7f3bd402a410 comp rx=0 tx=0).stop 2024-04-26T01:57:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.073+0000 7f3be3e00640 1 -- 172.21.15.6:0/714041260 shutdown_connections 2024-04-26T01:57:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.073+0000 7f3be3e00640 1 --2- 172.21.15.6:0/714041260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3be41014c0 0x7f3be410e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.073+0000 7f3be3e00640 1 --2- 172.21.15.6:0/714041260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3be410e6c0 0x7f3be4110ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.073+0000 7f3be3e00640 1 --2- 172.21.15.6:0/714041260 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be4100b10 0x7f3be4100ef0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.073+0000 7f3be3e00640 1 -- 172.21.15.6:0/714041260 >> 172.21.15.6:0/714041260 conn(0x7f3be40fc730 msgr2=0x7f3be40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.074+0000 7f3be3e00640 1 -- 172.21.15.6:0/714041260 shutdown_connections 2024-04-26T01:57:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.074+0000 7f3be3e00640 1 -- 172.21.15.6:0/714041260 wait complete. 2024-04-26T01:57:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.075+0000 7f3be3e00640 1 Processor -- start 2024-04-26T01:57:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.075+0000 7f3be3e00640 1 -- start start 2024-04-26T01:57:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.075+0000 7f3be3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3be4100b10 0x7f3be419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be41014c0 0x7f3be419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3be410e6c0 0x7f3be419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3be4114260 con 0x7f3be4100b10 2024-04-26T01:57:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be3e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41140e0 con 0x7f3be410e6c0 2024-04-26T01:57:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be3e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41143e0 con 0x7f3be41014c0 2024-04-26T01:57:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3be4100b10 0x7f3be419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be2000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be41014c0 0x7f3be419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be2000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be41014c0 0x7f3be419aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47000/0 (socket says 172.21.15.6:47000) 2024-04-26T01:57:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3be4100b10 0x7f3be419a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33844/0 (socket says 172.21.15.6:33844) 2024-04-26T01:57:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be2000640 1 -- 172.21.15.6:0/3746012465 learned_addr learned my addr 172.21.15.6:0/3746012465 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be3400640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3be410e6c0 0x7f3be419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be2000640 1 -- 172.21.15.6:0/3746012465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3be410e6c0 msgr2=0x7f3be419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be2000640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3be410e6c0 0x7f3be419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be2000640 1 -- 172.21.15.6:0/3746012465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3be4100b10 msgr2=0x7f3be419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be2000640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3be4100b10 0x7f3be419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.076+0000 7f3be2000640 1 -- 172.21.15.6:0/3746012465 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be419f750 con 0x7f3be41014c0 2024-04-26T01:57:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.077+0000 7f3be2000640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be41014c0 0x7f3be419aca0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc00ebc0 tx=0x7f3bcc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.077+0000 7f3be8a00640 1 -- 172.21.15.6:0/3746012465 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc010630 con 0x7f3be41014c0 2024-04-26T01:57:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.077+0000 7f3be8a00640 1 -- 172.21.15.6:0/3746012465 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bcc010c80 con 0x7f3be41014c0 2024-04-26T01:57:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.077+0000 7f3be8a00640 1 -- 172.21.15.6:0/3746012465 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc018b60 con 0x7f3be41014c0 2024-04-26T01:57:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.077+0000 7f3be3e00640 1 -- 172.21.15.6:0/3746012465 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be419fa40 con 0x7f3be41014c0 2024-04-26T01:57:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.077+0000 7f3be3e00640 1 -- 172.21.15.6:0/3746012465 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be41a72d0 con 0x7f3be41014c0 2024-04-26T01:57:00.079 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.081+0000 7f3be8a00640 1 -- 172.21.15.6:0/3746012465 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f3bcc018d00 con 0x7f3be41014c0 2024-04-26T01:57:00.080 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.081+0000 7f3becd9c740 1 Processor -- start 2024-04-26T01:57:00.080 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.081+0000 7f3becd9c740 1 -- start start 2024-04-26T01:57:00.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.082+0000 7f3becd9c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.082+0000 7f3becd9c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a12d0 0x5612761ab650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.082+0000 7f3becd9c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761adf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.082+0000 7f3becd9c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x561276106900 con 0x5612761a12d0 2024-04-26T01:57:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.082+0000 7f3becd9c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5612760acef0 con 0x5612761abb90 2024-04-26T01:57:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.082+0000 7f3becd9c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5612761028c0 con 0x5612761a09b0 2024-04-26T01:57:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.082+0000 7f3be3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.082+0000 7f3be3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761a0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47014/0 (socket says 172.21.15.6:47014) 2024-04-26T01:57:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.082+0000 7f3be3400640 1 -- 172.21.15.6:0/2486707304 learned_addr learned my addr 172.21.15.6:0/2486707304 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.082+0000 7f3be2000640 1 --2- 172.21.15.6:0/2486707304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.082+0000 7f3be2a00640 1 --2- 172.21.15.6:0/2486707304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a12d0 0x5612761ab650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.083+0000 7f3be3400640 1 -- 172.21.15.6:0/2486707304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 msgr2=0x5612761adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.083+0000 7f3be3400640 1 --2- 172.21.15.6:0/2486707304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761adf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.083+0000 7f3be3400640 1 -- 172.21.15.6:0/2486707304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a12d0 msgr2=0x5612761ab650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.083+0000 7f3be3400640 1 --2- 172.21.15.6:0/2486707304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a12d0 0x5612761ab650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.083+0000 7f3be3400640 1 -- 172.21.15.6:0/2486707304 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561275e19210 con 0x5612761a09b0 2024-04-26T01:57:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.083+0000 7f3be2a00640 1 --2- 172.21.15.6:0/2486707304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a12d0 0x5612761ab650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.083+0000 7f3be3400640 1 --2- 172.21.15.6:0/2486707304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761a0d90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009ba0 tx=0x7f3bdc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c478bb5412c57c4d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.083+0000 7f3be8a00640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3bc40a3d80 0x7f3bc40a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.083+0000 7f3bd3e00640 1 -- 172.21.15.6:0/2486707304 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc004230 con 0x5612761a09b0 2024-04-26T01:57:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.083+0000 7f3bd3e00640 1 -- 172.21.15.6:0/2486707304 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bdc009e10 con 0x5612761a09b0 2024-04-26T01:57:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.083+0000 7f3bd3e00640 1 -- 172.21.15.6:0/2486707304 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc004bc0 con 0x5612761a09b0 2024-04-26T01:57:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.083+0000 7f3be8a00640 1 -- 172.21.15.6:0/3746012465 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8600+0+0 (secure 0 0 0) 0x7f3bcc0d0e00 con 0x7f3be41014c0 2024-04-26T01:57:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.084+0000 7f3be2a00640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3bc40a3d80 0x7f3bc40a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.084+0000 7f3becd9c740 1 -- 172.21.15.6:0/2486707304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 msgr2=0x5612761a0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.084+0000 7f3becd9c740 1 --2- 172.21.15.6:0/2486707304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761a0d90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009ba0 tx=0x7f3bdc011810 comp rx=0 tx=0).stop 2024-04-26T01:57:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.084+0000 7f3becd9c740 1 -- 172.21.15.6:0/2486707304 shutdown_connections 2024-04-26T01:57:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.084+0000 7f3becd9c740 1 --2- 172.21.15.6:0/2486707304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a12d0 0x5612761ab650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.084+0000 7f3becd9c740 1 --2- 172.21.15.6:0/2486707304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.084+0000 7f3becd9c740 1 --2- 172.21.15.6:0/2486707304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761a0d90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.084+0000 7f3becd9c740 1 -- 172.21.15.6:0/2486707304 >> 172.21.15.6:0/2486707304 conn(0x56127619cec0 msgr2=0x56127619d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.085+0000 7f3becd9c740 1 -- 172.21.15.6:0/2486707304 shutdown_connections 2024-04-26T01:57:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.085+0000 7f3be2a00640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3bc40a3d80 0x7f3bc40a6240 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4005ab0 tx=0x7f3bd4005a20 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.085+0000 7f3becd9c740 1 -- 172.21.15.6:0/2486707304 wait complete. 2024-04-26T01:57:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.085+0000 7f3becd9c740 1 -- start start 2024-04-26T01:57:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.085+0000 7f3becd9c740 1 Processor -- start 2024-04-26T01:57:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.086+0000 7f3becd9c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761a8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.086+0000 7f3becd9c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a12d0 0x5612761a9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.086+0000 7f3becd9c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761a95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.086+0000 7f3becd9c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5612761ae7c0 con 0x5612761a12d0 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.086+0000 7f3becd9c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5612760c0310 con 0x5612761abb90 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.086+0000 7f3becd9c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5612761ae940 con 0x5612761a09b0 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.086+0000 7f3becd9c740 10 client.0 _ll_register_callbacks cb 0x56127608ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.086+0000 7f3be2a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761a8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.086+0000 7f3be2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a12d0 0x5612761a9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.086+0000 7f3be2a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761a8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47030/0 (socket says 172.21.15.6:47030) 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.086+0000 7f3be2a00640 1 -- 172.21.15.6:0/13833127 learned_addr learned my addr 172.21.15.6:0/13833127 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.086+0000 7f3be3400640 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761a95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.087+0000 7f3be3400640 1 -- 172.21.15.6:0/13833127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 msgr2=0x5612761a8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.087+0000 7f3be3400640 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761a8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.087+0000 7f3be3400640 1 -- 172.21.15.6:0/13833127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a12d0 msgr2=0x5612761a9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.087+0000 7f3be3400640 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a12d0 0x5612761a9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.087+0000 7f3be3400640 1 -- 172.21.15.6:0/13833127 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561275ee5620 con 0x5612761abb90 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.087+0000 7f3be2000640 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a12d0 0x5612761a9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.087+0000 7f3be3400640 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761a95a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009ba0 tx=0x7f3bdc019bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=305db082ec68c3c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.087+0000 7f3bd0c00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.087+0000 7f3bd0c00640 1 -- 172.21.15.6:0/13833127 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc028020 con 0x5612761abb90 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.088+0000 7f3becd9c740 1 -- 172.21.15.6:0/13833127 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bdc009e10 con 0x5612761abb90 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.088+0000 7f3bd0c00640 1 -- 172.21.15.6:0/13833127 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8600+0+0 (secure 0 0 0) 0x7f3bdc024470 con 0x5612761abb90 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.088+0000 7f3bd0c00640 1 -- 172.21.15.6:0/13833127 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3bdc01e040 con 0x5612761abb90 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.088+0000 7f3becd9c740 10 client.26501 fetch_fsmap learned FSMap version 43 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.088+0000 7f3becd9c740 1 -- 172.21.15.6:0/13833127 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5612761baa20 con 0x5612761abb90 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.088+0000 7f3bd0c00640 1 -- 172.21.15.6:0/13833127 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bdc009e10 con 0x5612761abb90 2024-04-26T01:57:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.088+0000 7f3bd0c00640 1 -- 172.21.15.6:0/13833127 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f3bdc018070 con 0x5612761abb90 2024-04-26T01:57:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.089+0000 7f3becd9c740 10 client.26501 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:57:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.089+0000 7f3becd9c740 10 client.26501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:57:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.089+0000 7f3becd9c740 20 client.26501 populate_metadata read hostname 'smithi006' 2024-04-26T01:57:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.089+0000 7f3becd9c740 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5612761bbe80 0x5612761be270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.089+0000 7f3becd9c740 4 client.26501 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.089+0000 7f3becd9c740 1 -- 172.21.15.6:0/13833127 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56127611a8b0 con 0x5612761bbe80 2024-04-26T01:57:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.089+0000 7f3be2a00640 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5612761bbe80 0x5612761be270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.089+0000 7f3be2a00640 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5612761bbe80 0x5612761be270 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bf1a9789d94fad4 server_cookie=920e0fe3ea34a544 in_seq=0 out_seq=0 2024-04-26T01:57:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.089+0000 7f3bd0c00640 10 client.26501 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.090+0000 7f3bd0c00640 1 -- 172.21.15.6:0/13833127 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56127611a8b0 con 0x5612761bbe80 2024-04-26T01:57:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.090+0000 7f3bd0c00640 10 client.26501 handle_command_reply: tid=0 2024-04-26T01:57:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.090+0000 7f3becd9c740 1 client.26501 shutdown 2024-04-26T01:57:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.091+0000 7f3becd9c740 1 -- 172.21.15.6:0/13833127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 msgr2=0x5612761a95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.091+0000 7f3becd9c740 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761a95a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009ba0 tx=0x7f3bdc019bf0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.091+0000 7f3becd9c740 1 -- 172.21.15.6:0/13833127 shutdown_connections 2024-04-26T01:57:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.091+0000 7f3becd9c740 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5612761bbe80 0x5612761be270 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.091+0000 7f3becd9c740 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a12d0 0x5612761a9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.091+0000 7f3becd9c740 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761a95a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.091+0000 7f3becd9c740 1 --2- 172.21.15.6:0/13833127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761a8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.091+0000 7f3becd9c740 1 -- 172.21.15.6:0/13833127 >> 172.21.15.6:0/13833127 conn(0x56127619cec0 msgr2=0x5612761a2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.091+0000 7f3bd0c00640 0 client.26501 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.091+0000 7f3becd9c740 1 -- 172.21.15.6:0/13833127 shutdown_connections 2024-04-26T01:57:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.091+0000 7f3becd9c740 1 -- 172.21.15.6:0/13833127 wait complete. 2024-04-26T01:57:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.091+0000 7f3becd9c740 20 client.26501 trim_cache size 0 max 16384 2024-04-26T01:57:00.110 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.111+0000 7f3becd9c740 1 Processor -- start 2024-04-26T01:57:00.110 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.111+0000 7f3becd9c740 1 -- start start 2024-04-26T01:57:00.110 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.111+0000 7f3becd9c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761b7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.110 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.111+0000 7f3becd9c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761a12d0 0x5612761c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.110 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.111+0000 7f3becd9c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761abb90 0x5612761c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.110 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.111+0000 7f3becd9c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41b2690 con 0x5612761abb90 2024-04-26T01:57:00.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.111+0000 7f3becd9c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41aff60 con 0x5612761a12d0 2024-04-26T01:57:00.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.111+0000 7f3becd9c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41addd0 con 0x5612761a09b0 2024-04-26T01:57:00.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.111+0000 7f3be2a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.111+0000 7f3be3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761abb90 0x5612761c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.112+0000 7f3be2a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761b7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47042/0 (socket says 172.21.15.6:47042) 2024-04-26T01:57:00.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.112+0000 7f3be2a00640 1 -- 172.21.15.6:0/1777572283 learned_addr learned my addr 172.21.15.6:0/1777572283 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:00.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.111+0000 7f3be2000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761a12d0 0x5612761c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.112+0000 7f3be2000640 1 -- 172.21.15.6:0/1777572283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 msgr2=0x5612761b7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.112+0000 7f3be2000640 1 --2- 172.21.15.6:0/1777572283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761b7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.112+0000 7f3be2000640 1 -- 172.21.15.6:0/1777572283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761abb90 msgr2=0x5612761c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.112+0000 7f3be2000640 1 --2- 172.21.15.6:0/1777572283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761abb90 0x5612761c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.112+0000 7f3be2000640 1 -- 172.21.15.6:0/1777572283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612761a1100 con 0x5612761a12d0 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.112+0000 7f3be2a00640 1 --2- 172.21.15.6:0/1777572283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761b7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.112+0000 7f3be2000640 1 --2- 172.21.15.6:0/1777572283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761a12d0 0x5612761c3a30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc0ce4a0 tx=0x7f3bcc0bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=929f00693f2e7cf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.113+0000 7f3bd3400640 1 -- 172.21.15.6:0/1777572283 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc09c8e0 con 0x5612761a12d0 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.113+0000 7f3bd3400640 1 -- 172.21.15.6:0/1777572283 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bcc018420 con 0x5612761a12d0 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.113+0000 7f3bd3400640 1 -- 172.21.15.6:0/1777572283 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc09c700 con 0x5612761a12d0 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.113+0000 7f3becd9c740 1 -- 172.21.15.6:0/1777572283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761a12d0 msgr2=0x5612761c3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.113+0000 7f3becd9c740 1 --2- 172.21.15.6:0/1777572283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761a12d0 0x5612761c3a30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc0ce4a0 tx=0x7f3bcc0bd040 comp rx=0 tx=0).stop 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.114+0000 7f3becd9c740 1 -- 172.21.15.6:0/1777572283 shutdown_connections 2024-04-26T01:57:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.114+0000 7f3becd9c740 1 --2- 172.21.15.6:0/1777572283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761abb90 0x5612761c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.114+0000 7f3becd9c740 1 --2- 172.21.15.6:0/1777572283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761a12d0 0x5612761c3a30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.114+0000 7f3becd9c740 1 --2- 172.21.15.6:0/1777572283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761a09b0 0x5612761b7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.114+0000 7f3becd9c740 1 -- 172.21.15.6:0/1777572283 >> 172.21.15.6:0/1777572283 conn(0x56127619cec0 msgr2=0x56127619c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.114+0000 7f3becd9c740 1 -- 172.21.15.6:0/1777572283 shutdown_connections 2024-04-26T01:57:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.114+0000 7f3becd9c740 1 -- 172.21.15.6:0/1777572283 wait complete. 2024-04-26T01:57:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.114+0000 7f3becd9c740 1 -- start start 2024-04-26T01:57:00.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.114+0000 7f3becd9c740 1 Processor -- start 2024-04-26T01:57:00.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3becd9c740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a09b0 0x5612761b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3becd9c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761b4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3becd9c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761b0fc0 0x5612761b1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3becd9c740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41ae420 con 0x5612761a09b0 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3becd9c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41af890 con 0x5612761abb90 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3becd9c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41aeaf0 con 0x5612761b0fc0 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3becd9c740 10 client.0 _ll_register_callbacks cb 0x7f3bdc019800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3be2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a09b0 0x5612761b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3be2000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3be2000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:57980/0 (socket says 172.21.15.6:57980) 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3be2000640 1 -- 172.21.15.6:0/2239183941 learned_addr learned my addr 172.21.15.6:0/2239183941 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3be3400640 1 --2- 172.21.15.6:0/2239183941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761b0fc0 0x5612761b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3be2000640 1 -- 172.21.15.6:0/2239183941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761b0fc0 msgr2=0x5612761b1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3be2000640 1 --2- 172.21.15.6:0/2239183941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761b0fc0 0x5612761b1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3be2000640 1 -- 172.21.15.6:0/2239183941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a09b0 msgr2=0x5612761b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.115+0000 7f3be2000640 1 --2- 172.21.15.6:0/2239183941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a09b0 0x5612761b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.116+0000 7f3be2000640 1 -- 172.21.15.6:0/2239183941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612761b1c50 con 0x5612761abb90 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.116+0000 7f3be2000640 1 --2- 172.21.15.6:0/2239183941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761b4080 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc099770 tx=0x7f3bcc01f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dd8fb8b89726f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.116+0000 7f3bd0c00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.116+0000 7f3bd0c00640 1 -- 172.21.15.6:0/2239183941 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc09c700 con 0x5612761abb90 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.116+0000 7f3bd0c00640 1 -- 172.21.15.6:0/2239183941 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8600+0+0 (secure 0 0 0) 0x7f3bcc0bd2b0 con 0x5612761abb90 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.116+0000 7f3becd9c740 1 -- 172.21.15.6:0/2239183941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561276114320 con 0x5612761abb90 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.116+0000 7f3bd0c00640 1 -- 172.21.15.6:0/2239183941 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3bcc0ce930 con 0x5612761abb90 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.116+0000 7f3becd9c740 10 client.26507 fetch_fsmap learned FSMap version 43 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.116+0000 7f3becd9c740 1 -- 172.21.15.6:0/2239183941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x561276222290 con 0x5612761abb90 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.117+0000 7f3bd0c00640 1 -- 172.21.15.6:0/2239183941 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561276114320 con 0x5612761abb90 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.117+0000 7f3bd0c00640 1 -- 172.21.15.6:0/2239183941 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f3bcc09c1f0 con 0x5612761abb90 2024-04-26T01:57:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.117+0000 7f3becd9c740 10 client.26507 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:57:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.117+0000 7f3becd9c740 10 client.26507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:57:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.117+0000 7f3becd9c740 20 client.26507 populate_metadata read hostname 'smithi006' 2024-04-26T01:57:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.117+0000 7f3becd9c740 1 --2- 172.21.15.6:0/2239183941 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5612762228f0 0x561276224d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.117+0000 7f3becd9c740 4 client.26507 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:57:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.117+0000 7f3becd9c740 1 -- 172.21.15.6:0/2239183941 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56127610e190 con 0x5612762228f0 2024-04-26T01:57:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.117+0000 7f3be2a00640 1 --2- 172.21.15.6:0/2239183941 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5612762228f0 0x561276224d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.118+0000 7f3be2a00640 1 --2- 172.21.15.6:0/2239183941 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5612762228f0 0x561276224d10 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f44447a80c6b7b41 server_cookie=1d73a67e31a13bee in_seq=0 out_seq=0 2024-04-26T01:57:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.118+0000 7f3bd0c00640 10 client.26507 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3bd0c00640 1 -- 172.21.15.6:0/2239183941 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56127610e190 con 0x5612762228f0 2024-04-26T01:57:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3bd0c00640 10 client.26507 handle_command_reply: tid=0 2024-04-26T01:57:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3becd9c740 1 client.26507 shutdown 2024-04-26T01:57:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3becd9c740 1 -- 172.21.15.6:0/2239183941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 msgr2=0x5612761b4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3becd9c740 1 --2- 172.21.15.6:0/2239183941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761b4080 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc099770 tx=0x7f3bcc01f000 comp rx=0 tx=0).stop 2024-04-26T01:57:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3becd9c740 1 -- 172.21.15.6:0/2239183941 shutdown_connections 2024-04-26T01:57:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3becd9c740 1 --2- 172.21.15.6:0/2239183941 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5612762228f0 0x561276224d10 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3becd9c740 1 --2- 172.21.15.6:0/2239183941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5612761b0fc0 0x5612761b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3becd9c740 1 --2- 172.21.15.6:0/2239183941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5612761abb90 0x5612761b4080 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3becd9c740 1 --2- 172.21.15.6:0/2239183941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5612761a09b0 0x5612761b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3bd0c00640 0 client.26507 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3becd9c740 1 -- 172.21.15.6:0/2239183941 >> 172.21.15.6:0/2239183941 conn(0x56127619cec0 msgr2=0x5612761a2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3becd9c740 1 -- 172.21.15.6:0/2239183941 shutdown_connections 2024-04-26T01:57:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3becd9c740 1 -- 172.21.15.6:0/2239183941 wait complete. 2024-04-26T01:57:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.166+0000 7f3becd9c740 20 client.26507 trim_cache size 0 max 16384 2024-04-26T01:57:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:57:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:57:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:57:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:57:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.167+0000 7f3be3e00640 1 -- 172.21.15.6:0/3746012465 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3bc40a3d80 msgr2=0x7f3bc40a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.167+0000 7f3be3e00640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3bc40a3d80 0x7f3bc40a6240 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4005ab0 tx=0x7f3bd4005a20 comp rx=0 tx=0).stop 2024-04-26T01:57:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.167+0000 7f3be3e00640 1 -- 172.21.15.6:0/3746012465 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be41014c0 msgr2=0x7f3be419aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.167+0000 7f3be3e00640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be41014c0 0x7f3be419aca0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc00ebc0 tx=0x7f3bcc00c6a0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.167+0000 7f3be3e00640 1 -- 172.21.15.6:0/3746012465 shutdown_connections 2024-04-26T01:57:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.167+0000 7f3be3e00640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3bc40a3d80 0x7f3bc40a6240 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.167+0000 7f3be3e00640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3be410e6c0 0x7f3be419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.167+0000 7f3be3e00640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be41014c0 0x7f3be419aca0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.167+0000 7f3be3e00640 1 --2- 172.21.15.6:0/3746012465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3be4100b10 0x7f3be419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.168+0000 7f3be3e00640 1 -- 172.21.15.6:0/3746012465 >> 172.21.15.6:0/3746012465 conn(0x7f3be40fc730 msgr2=0x7f3be40feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.168+0000 7f3be3e00640 1 -- 172.21.15.6:0/3746012465 shutdown_connections 2024-04-26T01:57:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:00.168+0000 7f3be3e00640 1 -- 172.21.15.6:0/3746012465 wait complete. 2024-04-26T01:57:00.626 INFO:tasks.workunit.client.0.smithi006.stdout: 1 389288 3.59 MB/sec execute 444 sec latency 245.945 ms 2024-04-26T01:57:00.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:00 smithi029 ceph-mon[32775]: pgmap v349: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 167 op/s 2024-04-26T01:57:00.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:00 smithi006 ceph-mon[28419]: pgmap v349: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 167 op/s 2024-04-26T01:57:00.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:00 smithi188 ceph-mon[32751]: pgmap v349: 97 pgs: 97 active+clean; 130 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 167 op/s 2024-04-26T01:57:01.626 INFO:tasks.workunit.client.0.smithi006.stdout: 1 389948 3.60 MB/sec execute 445 sec latency 59.607 ms 2024-04-26T01:57:02.626 INFO:tasks.workunit.client.0.smithi006.stdout: 1 390657 3.60 MB/sec execute 446 sec latency 21.841 ms 2024-04-26T01:57:02.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:02 smithi029 ceph-mon[32775]: pgmap v350: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 2.4 MiB/s wr, 286 op/s 2024-04-26T01:57:02.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:02 smithi006 ceph-mon[28419]: pgmap v350: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 2.4 MiB/s wr, 286 op/s 2024-04-26T01:57:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:02 smithi188 ceph-mon[32751]: pgmap v350: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 2.4 MiB/s wr, 286 op/s 2024-04-26T01:57:03.626 INFO:tasks.workunit.client.0.smithi006.stdout: 1 391604 3.60 MB/sec execute 447 sec latency 21.960 ms 2024-04-26T01:57:04.626 INFO:tasks.workunit.client.0.smithi006.stdout: 1 392675 3.60 MB/sec execute 448 sec latency 27.356 ms 2024-04-26T01:57:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:04 smithi029 ceph-mon[32775]: pgmap v351: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-04-26T01:57:04.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:04 smithi006 ceph-mon[28419]: pgmap v351: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-04-26T01:57:04.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:04 smithi188 ceph-mon[32751]: pgmap v351: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-04-26T01:57:05.626 INFO:tasks.workunit.client.0.smithi006.stdout: 1 393881 3.62 MB/sec execute 449 sec latency 65.345 ms 2024-04-26T01:57:06.626 INFO:tasks.workunit.client.0.smithi006.stdout: 1 394359 3.61 MB/sec execute 450 sec latency 31.843 ms 2024-04-26T01:57:06.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:06 smithi029 ceph-mon[32775]: pgmap v352: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-04-26T01:57:06.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:06 smithi029 ceph-mon[32775]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T01:57:06.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:06 smithi006 ceph-mon[28419]: pgmap v352: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-04-26T01:57:06.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:06 smithi006 ceph-mon[28419]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T01:57:06.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:06 smithi188 ceph-mon[32751]: pgmap v352: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.7 MiB/s wr, 195 op/s 2024-04-26T01:57:06.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:06 smithi188 ceph-mon[32751]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T01:57:07.626 INFO:tasks.workunit.client.0.smithi006.stdout: 1 395273 3.61 MB/sec execute 451 sec latency 28.814 ms 2024-04-26T01:57:08.627 INFO:tasks.workunit.client.0.smithi006.stdout: 1 396371 3.61 MB/sec execute 452 sec latency 25.327 ms 2024-04-26T01:57:08.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:08 smithi029 ceph-mon[32775]: pgmap v354: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.0 MiB/s wr, 236 op/s 2024-04-26T01:57:08.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:08 smithi006 ceph-mon[28419]: pgmap v354: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.0 MiB/s wr, 236 op/s 2024-04-26T01:57:08.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:08 smithi188 ceph-mon[32751]: pgmap v354: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.0 MiB/s wr, 236 op/s 2024-04-26T01:57:09.627 INFO:tasks.workunit.client.0.smithi006.stdout: 1 397526 3.63 MB/sec execute 453 sec latency 43.613 ms 2024-04-26T01:57:10.627 INFO:tasks.workunit.client.0.smithi006.stdout: 1 397905 3.62 MB/sec execute 454 sec latency 30.212 ms 2024-04-26T01:57:10.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:10 smithi029 ceph-mon[32775]: pgmap v355: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.0 MiB/s wr, 236 op/s 2024-04-26T01:57:10.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:10 smithi029 ceph-mon[32775]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T01:57:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:10 smithi006 ceph-mon[28419]: pgmap v355: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.0 MiB/s wr, 236 op/s 2024-04-26T01:57:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:10 smithi006 ceph-mon[28419]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T01:57:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:10 smithi188 ceph-mon[32751]: pgmap v355: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.0 MiB/s wr, 236 op/s 2024-04-26T01:57:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:10 smithi188 ceph-mon[32751]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T01:57:11.627 INFO:tasks.workunit.client.0.smithi006.stdout: 1 398839 3.62 MB/sec execute 455 sec latency 30.322 ms 2024-04-26T01:57:12.627 INFO:tasks.workunit.client.0.smithi006.stdout: 1 400069 3.63 MB/sec execute 456 sec latency 29.427 ms 2024-04-26T01:57:12.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:12 smithi029 ceph-mon[32775]: pgmap v357: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 213 op/s 2024-04-26T01:57:12.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:12 smithi006 ceph-mon[28419]: pgmap v357: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 213 op/s 2024-04-26T01:57:12.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:12 smithi188 ceph-mon[32751]: pgmap v357: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 213 op/s 2024-04-26T01:57:13.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.325+0000 7f3681800640 1 -- 172.21.15.6:0/3048071262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f367c10c340 msgr2=0x7f367c10e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.325+0000 7f3681800640 1 --2- 172.21.15.6:0/3048071262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f367c10c340 0x7f367c10e7e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3674009ca0 tx=0x7f367402a950 comp rx=0 tx=0).stop 2024-04-26T01:57:13.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.325+0000 7f3681800640 1 -- 172.21.15.6:0/3048071262 shutdown_connections 2024-04-26T01:57:13.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.325+0000 7f3681800640 1 --2- 172.21.15.6:0/3048071262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f367c10c340 0x7f367c10e7e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.325+0000 7f3681800640 1 --2- 172.21.15.6:0/3048071262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f367c103840 0x7f367c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.325+0000 7f3681800640 1 --2- 172.21.15.6:0/3048071262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f367c102f20 0x7f367c103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.325+0000 7f3681800640 1 -- 172.21.15.6:0/3048071262 >> 172.21.15.6:0/3048071262 conn(0x7f367c0fc780 msgr2=0x7f367c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:13.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.325+0000 7f3681800640 1 -- 172.21.15.6:0/3048071262 shutdown_connections 2024-04-26T01:57:13.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.325+0000 7f3681800640 1 -- 172.21.15.6:0/3048071262 wait complete. 2024-04-26T01:57:13.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.327+0000 7f3681800640 1 Processor -- start 2024-04-26T01:57:13.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.327+0000 7f3681800640 1 -- start start 2024-04-26T01:57:13.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.327+0000 7f3681800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f367c102f20 0x7f367c100b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.327+0000 7f3681800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f367c103840 0x7f367c1010b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.327+0000 7f3681800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f367c10c340 0x7f367c1015f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.327+0000 7f3681800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f367c112020 con 0x7f367c102f20 2024-04-26T01:57:13.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.327+0000 7f3681800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f367c111ea0 con 0x7f367c10c340 2024-04-26T01:57:13.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.327+0000 7f3681800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1121a0 con 0x7f367c103840 2024-04-26T01:57:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.328+0000 7f3673e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f367c103840 0x7f367c1010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.328+0000 7f3673e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f367c103840 0x7f367c1010b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45002/0 (socket says 172.21.15.6:45002) 2024-04-26T01:57:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.328+0000 7f3673e00640 1 -- 172.21.15.6:0/3619096213 learned_addr learned my addr 172.21.15.6:0/3619096213 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.328+0000 7f3680e00640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f367c10c340 0x7f367c1015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.328+0000 7f367be00640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f367c102f20 0x7f367c100b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.328+0000 7f3673e00640 1 -- 172.21.15.6:0/3619096213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f367c10c340 msgr2=0x7f367c1015f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.328+0000 7f3673e00640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f367c10c340 0x7f367c1015f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.328+0000 7f3673e00640 1 -- 172.21.15.6:0/3619096213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f367c102f20 msgr2=0x7f367c100b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.328+0000 7f3673e00640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f367c102f20 0x7f367c100b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.328+0000 7f3673e00640 1 -- 172.21.15.6:0/3619096213 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f367c1a6a40 con 0x7f367c103840 2024-04-26T01:57:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.328+0000 7f367be00640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f367c102f20 0x7f367c100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.328+0000 7f3673e00640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f367c103840 0x7f367c1010b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f366400e9f0 tx=0x7f366400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.329+0000 7f3679600640 1 -- 172.21.15.6:0/3619096213 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366400ccc0 con 0x7f367c103840 2024-04-26T01:57:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.329+0000 7f3679600640 1 -- 172.21.15.6:0/3619096213 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f366400ce60 con 0x7f367c103840 2024-04-26T01:57:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.329+0000 7f3681800640 1 -- 172.21.15.6:0/3619096213 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f367c1a6d30 con 0x7f367c103840 2024-04-26T01:57:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.329+0000 7f3679600640 1 -- 172.21.15.6:0/3619096213 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3664010430 con 0x7f367c103840 2024-04-26T01:57:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.329+0000 7f3681800640 1 -- 172.21.15.6:0/3619096213 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f367c1a71d0 con 0x7f367c103840 2024-04-26T01:57:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.333+0000 7f3685b80740 1 Processor -- start 2024-04-26T01:57:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.333+0000 7f3685b80740 1 -- start start 2024-04-26T01:57:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.333+0000 7f3679600640 1 -- 172.21.15.6:0/3619096213 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f36640046c0 con 0x7f367c103840 2024-04-26T01:57:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.334+0000 7f3685b80740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53adc9b0 0x555c53adcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.334+0000 7f3685b80740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53add2d0 0x555c53ae7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.334+0000 7f3685b80740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53ae7b90 0x555c53ae9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.334+0000 7f3685b80740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555c53a42900 con 0x555c53ae7b90 2024-04-26T01:57:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.334+0000 7f3685b80740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555c539e8ef0 con 0x555c53adc9b0 2024-04-26T01:57:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.334+0000 7f3685b80740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555c53a3e8c0 con 0x555c53add2d0 2024-04-26T01:57:13.333 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.334+0000 7f367be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53add2d0 0x555c53ae7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.333 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.334+0000 7f367be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53add2d0 0x555c53ae7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45014/0 (socket says 172.21.15.6:45014) 2024-04-26T01:57:13.333 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.334+0000 7f367be00640 1 -- 172.21.15.6:0/606508484 learned_addr learned my addr 172.21.15.6:0/606508484 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:13.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.334+0000 7f3680e00640 1 --2- 172.21.15.6:0/606508484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53adc9b0 0x555c53adcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.334+0000 7f3673e00640 1 --2- 172.21.15.6:0/606508484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53ae7b90 0x555c53ae9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.335+0000 7f3680e00640 1 -- 172.21.15.6:0/606508484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53add2d0 msgr2=0x555c53ae7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.335+0000 7f3680e00640 1 --2- 172.21.15.6:0/606508484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53add2d0 0x555c53ae7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.335+0000 7f3680e00640 1 -- 172.21.15.6:0/606508484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53ae7b90 msgr2=0x555c53ae9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.335+0000 7f3680e00640 1 --2- 172.21.15.6:0/606508484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53ae7b90 0x555c53ae9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.335+0000 7f3680e00640 1 -- 172.21.15.6:0/606508484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c53755210 con 0x555c53adc9b0 2024-04-26T01:57:13.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.335+0000 7f3673e00640 1 --2- 172.21.15.6:0/606508484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53ae7b90 0x555c53ae9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:13.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.335+0000 7f3680e00640 1 --2- 172.21.15.6:0/606508484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53adc9b0 0x555c53adcd90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3674005e30 tx=0x7f3674005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb73928c536a162d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:13.334 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.335+0000 7f3671600640 1 -- 172.21.15.6:0/606508484 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674048070 con 0x555c53adc9b0 2024-04-26T01:57:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.335+0000 7f3671600640 1 -- 172.21.15.6:0/606508484 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3674003e20 con 0x555c53adc9b0 2024-04-26T01:57:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.335+0000 7f3671600640 1 -- 172.21.15.6:0/606508484 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674034040 con 0x555c53adc9b0 2024-04-26T01:57:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.336+0000 7f3679600640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36600a3d00 0x7f36600a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.337 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.336+0000 7f3679600640 1 -- 172.21.15.6:0/3619096213 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7f36640d0db0 con 0x7f367c103840 2024-04-26T01:57:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.336+0000 7f367be00640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36600a3d00 0x7f36600a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.337+0000 7f3685b80740 1 -- 172.21.15.6:0/606508484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53adc9b0 msgr2=0x555c53adcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.337+0000 7f3685b80740 1 --2- 172.21.15.6:0/606508484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53adc9b0 0x555c53adcd90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3674005e30 tx=0x7f3674005f80 comp rx=0 tx=0).stop 2024-04-26T01:57:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.337+0000 7f3685b80740 1 -- 172.21.15.6:0/606508484 shutdown_connections 2024-04-26T01:57:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.337+0000 7f3685b80740 1 --2- 172.21.15.6:0/606508484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53ae7b90 0x555c53ae9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.337+0000 7f3685b80740 1 --2- 172.21.15.6:0/606508484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53add2d0 0x555c53ae7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.337+0000 7f3685b80740 1 --2- 172.21.15.6:0/606508484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53adc9b0 0x555c53adcd90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.337+0000 7f3685b80740 1 -- 172.21.15.6:0/606508484 >> 172.21.15.6:0/606508484 conn(0x555c53ad8ec0 msgr2=0x555c53ad92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.337+0000 7f3685b80740 1 -- 172.21.15.6:0/606508484 shutdown_connections 2024-04-26T01:57:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.337+0000 7f3685b80740 1 -- 172.21.15.6:0/606508484 wait complete. 2024-04-26T01:57:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.337+0000 7f367be00640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36600a3d00 0x7f36600a61c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f366c0095f0 tx=0x7f366c005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.337+0000 7f3685b80740 1 -- start start 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.338+0000 7f3685b80740 1 Processor -- start 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.338+0000 7f3685b80740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53ae4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.338+0000 7f3685b80740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53add2d0 0x555c53ae50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.338+0000 7f3685b80740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53ae7b90 0x555c53ae5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.338+0000 7f3685b80740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555c53aea760 con 0x555c53adc9b0 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.338+0000 7f3685b80740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555c539fc310 con 0x555c53ae7b90 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.338+0000 7f3685b80740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555c53aea8e0 con 0x555c53add2d0 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.338+0000 7f3685b80740 10 client.0 _ll_register_callbacks cb 0x555c539cacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f367be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53ae4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f3680e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53ae7b90 0x555c53ae5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f3680e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53ae7b90 0x555c53ae5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47036/0 (socket says 172.21.15.6:47036) 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f3680e00640 1 -- 172.21.15.6:0/3530402916 learned_addr learned my addr 172.21.15.6:0/3530402916 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f3673e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53add2d0 0x555c53ae50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f3680e00640 1 -- 172.21.15.6:0/3530402916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53add2d0 msgr2=0x555c53ae50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f3680e00640 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53add2d0 0x555c53ae50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f3680e00640 1 -- 172.21.15.6:0/3530402916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 msgr2=0x555c53ae4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f3680e00640 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53ae4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f3680e00640 1 -- 172.21.15.6:0/3530402916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c53821620 con 0x555c53ae7b90 2024-04-26T01:57:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f3673e00640 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53add2d0 0x555c53ae50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f367be00640 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53ae4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f3680e00640 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53ae7b90 0x555c53ae5600 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3674006010 tx=0x7f367402f7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb95954d44a7a51a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.339+0000 7f366a000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:57:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.340+0000 7f366a000640 1 -- 172.21.15.6:0/3530402916 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674051060 con 0x555c53ae7b90 2024-04-26T01:57:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.340+0000 7f366a000640 1 -- 172.21.15.6:0/3530402916 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7f3674003970 con 0x555c53ae7b90 2024-04-26T01:57:13.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.340+0000 7f3685b80740 1 -- 172.21.15.6:0/3530402916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3674003e20 con 0x555c53ae7b90 2024-04-26T01:57:13.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.341+0000 7f366a000640 1 -- 172.21.15.6:0/3530402916 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3674048070 con 0x555c53ae7b90 2024-04-26T01:57:13.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.341+0000 7f3685b80740 10 client.26519 fetch_fsmap learned FSMap version 43 2024-04-26T01:57:13.342 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.341+0000 7f3685b80740 1 -- 172.21.15.6:0/3530402916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x555c53af6a20 con 0x555c53ae7b90 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.341+0000 7f366a000640 1 -- 172.21.15.6:0/3530402916 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3674003e20 con 0x555c53ae7b90 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.341+0000 7f366a000640 1 -- 172.21.15.6:0/3530402916 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f3674030440 con 0x555c53ae7b90 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.341+0000 7f3685b80740 10 client.26519 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.341+0000 7f3685b80740 10 client.26519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.341+0000 7f3685b80740 20 client.26519 populate_metadata read hostname 'smithi006' 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.341+0000 7f3685b80740 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555c53af7d70 0x555c53afa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.341+0000 7f3685b80740 4 client.26519 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.341+0000 7f3685b80740 1 -- 172.21.15.6:0/3530402916 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c53a568b0 con 0x555c53af7d70 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.342+0000 7f367be00640 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555c53af7d70 0x555c53afa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.342+0000 7f367be00640 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555c53af7d70 0x555c53afa160 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c4f90a82606cb5f server_cookie=65ef75a0806ad2b3 in_seq=0 out_seq=0 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.342+0000 7f366a000640 10 client.26519 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.343+0000 7f366a000640 1 -- 172.21.15.6:0/3530402916 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c53a568b0 con 0x555c53af7d70 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.343+0000 7f366a000640 10 client.26519 handle_command_reply: tid=0 2024-04-26T01:57:13.343 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.343+0000 7f3685b80740 1 client.26519 shutdown 2024-04-26T01:57:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.343+0000 7f3685b80740 1 -- 172.21.15.6:0/3530402916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53ae7b90 msgr2=0x555c53ae5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.343+0000 7f3685b80740 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53ae7b90 0x555c53ae5600 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3674006010 tx=0x7f367402f7d0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.343+0000 7f3685b80740 1 -- 172.21.15.6:0/3530402916 shutdown_connections 2024-04-26T01:57:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.344+0000 7f3685b80740 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555c53af7d70 0x555c53afa160 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.344+0000 7f3685b80740 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53ae7b90 0x555c53ae5600 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.344+0000 7f3685b80740 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53add2d0 0x555c53ae50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.344+0000 7f3685b80740 1 --2- 172.21.15.6:0/3530402916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53ae4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.344+0000 7f366a000640 0 client.26519 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.344+0000 7f3685b80740 1 -- 172.21.15.6:0/3530402916 >> 172.21.15.6:0/3530402916 conn(0x555c53ad8ec0 msgr2=0x555c53adebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.344+0000 7f3685b80740 1 -- 172.21.15.6:0/3530402916 shutdown_connections 2024-04-26T01:57:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.344+0000 7f3685b80740 1 -- 172.21.15.6:0/3530402916 wait complete. 2024-04-26T01:57:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.344+0000 7f3685b80740 20 client.26519 trim_cache size 0 max 16384 2024-04-26T01:57:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.362+0000 7f3685b80740 1 Processor -- start 2024-04-26T01:57:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.362+0000 7f3685b80740 1 -- start start 2024-04-26T01:57:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.362+0000 7f3685b80740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53ad7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.362+0000 7f3685b80740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53add2d0 0x555c53ae4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3685b80740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 0x555c53ae5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3685b80740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1b2580 con 0x555c53adc9b0 2024-04-26T01:57:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3685b80740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1afe50 con 0x555c53add2d0 2024-04-26T01:57:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3685b80740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1adcc0 con 0x555c53ae7b90 2024-04-26T01:57:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3680e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 0x555c53ae5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3680e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 0x555c53ae5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45034/0 (socket says 172.21.15.6:45034) 2024-04-26T01:57:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3680e00640 1 -- 172.21.15.6:0/668413280 learned_addr learned my addr 172.21.15.6:0/668413280 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3673e00640 1 --2- 172.21.15.6:0/668413280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53add2d0 0x555c53ae4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f367be00640 1 --2- 172.21.15.6:0/668413280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53ad7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3680e00640 1 -- 172.21.15.6:0/668413280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53add2d0 msgr2=0x555c53ae4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3680e00640 1 --2- 172.21.15.6:0/668413280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53add2d0 0x555c53ae4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3680e00640 1 -- 172.21.15.6:0/668413280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 msgr2=0x555c53ad7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3680e00640 1 --2- 172.21.15.6:0/668413280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53ad7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f3680e00640 1 -- 172.21.15.6:0/668413280 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c53ad4820 con 0x555c53ae7b90 2024-04-26T01:57:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.363+0000 7f367be00640 1 --2- 172.21.15.6:0/668413280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53ad7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.364+0000 7f3680e00640 1 --2- 172.21.15.6:0/668413280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 0x555c53ae5080 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3674000f80 tx=0x7f367402b0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=790d8885eac7cd26 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.364+0000 7f3670c00640 1 -- 172.21.15.6:0/668413280 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674051040 con 0x555c53ae7b90 2024-04-26T01:57:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.364+0000 7f3670c00640 1 -- 172.21.15.6:0/668413280 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3674030440 con 0x555c53ae7b90 2024-04-26T01:57:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.364+0000 7f3670c00640 1 -- 172.21.15.6:0/668413280 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674034050 con 0x555c53ae7b90 2024-04-26T01:57:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.365+0000 7f3685b80740 1 -- 172.21.15.6:0/668413280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 msgr2=0x555c53ae5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.365+0000 7f3685b80740 1 --2- 172.21.15.6:0/668413280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 0x555c53ae5080 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3674000f80 tx=0x7f367402b0d0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.365+0000 7f3685b80740 1 -- 172.21.15.6:0/668413280 shutdown_connections 2024-04-26T01:57:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.365+0000 7f3685b80740 1 --2- 172.21.15.6:0/668413280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 0x555c53ae5080 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.365+0000 7f3685b80740 1 --2- 172.21.15.6:0/668413280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53add2d0 0x555c53ae4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.365+0000 7f3685b80740 1 --2- 172.21.15.6:0/668413280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53ad7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.365+0000 7f3685b80740 1 -- 172.21.15.6:0/668413280 >> 172.21.15.6:0/668413280 conn(0x555c53ad8ec0 msgr2=0x555c53ad7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.365+0000 7f3685b80740 1 -- 172.21.15.6:0/668413280 shutdown_connections 2024-04-26T01:57:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.365+0000 7f3685b80740 1 -- 172.21.15.6:0/668413280 wait complete. 2024-04-26T01:57:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.366+0000 7f3685b80740 1 -- start start 2024-04-26T01:57:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.366+0000 7f3685b80740 1 Processor -- start 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.366+0000 7f3685b80740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53afc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.366+0000 7f3685b80740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53add2d0 0x555c53afc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.366+0000 7f3685b80740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 0x555c53b5bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3685b80740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1ae310 con 0x555c53adc9b0 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3685b80740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1af780 con 0x555c53add2d0 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3685b80740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1ae9e0 con 0x555c53ae7b90 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3685b80740 10 client.0 _ll_register_callbacks cb 0x7f3674009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f367be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53afc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3680e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 0x555c53b5bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3673e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53add2d0 0x555c53afc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3680e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 0x555c53b5bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45040/0 (socket says 172.21.15.6:45040) 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3680e00640 1 -- 172.21.15.6:0/3529814976 learned_addr learned my addr 172.21.15.6:0/3529814976 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3673e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53add2d0 0x555c53afc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47056/0 (socket says 172.21.15.6:47056) 2024-04-26T01:57:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3680e00640 1 -- 172.21.15.6:0/3529814976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53add2d0 msgr2=0x555c53afc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3680e00640 1 --2- 172.21.15.6:0/3529814976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53add2d0 0x555c53afc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3680e00640 1 -- 172.21.15.6:0/3529814976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 msgr2=0x555c53afc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3680e00640 1 --2- 172.21.15.6:0/3529814976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53afc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3680e00640 1 -- 172.21.15.6:0/3529814976 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c53b5c4c0 con 0x555c53ae7b90 2024-04-26T01:57:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.367+0000 7f3673e00640 1 --2- 172.21.15.6:0/3529814976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53add2d0 0x555c53afc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.368+0000 7f3680e00640 1 --2- 172.21.15.6:0/3529814976 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 0x555c53b5bf50 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f367402f810 tx=0x7f3674030f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=148b590bbed93832 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.368+0000 7f366a000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:57:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.368+0000 7f366a000640 1 -- 172.21.15.6:0/3529814976 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674051070 con 0x555c53ae7b90 2024-04-26T01:57:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.368+0000 7f366a000640 1 -- 172.21.15.6:0/3529814976 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7f36740305f0 con 0x555c53ae7b90 2024-04-26T01:57:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.368+0000 7f3685b80740 1 -- 172.21.15.6:0/3529814976 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3674030440 con 0x555c53ae7b90 2024-04-26T01:57:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.368+0000 7f366a000640 1 -- 172.21.15.6:0/3529814976 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f367406b050 con 0x555c53ae7b90 2024-04-26T01:57:13.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.368+0000 7f3685b80740 10 client.26563 fetch_fsmap learned FSMap version 43 2024-04-26T01:57:13.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.368+0000 7f3685b80740 1 -- 172.21.15.6:0/3529814976 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x555c53b5d7b0 con 0x555c53ae7b90 2024-04-26T01:57:13.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.368+0000 7f366a000640 1 -- 172.21.15.6:0/3529814976 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3674030440 con 0x555c53ae7b90 2024-04-26T01:57:13.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.369+0000 7f366a000640 1 -- 172.21.15.6:0/3529814976 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f367405d5a0 con 0x555c53ae7b90 2024-04-26T01:57:13.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.369+0000 7f3685b80740 10 client.26563 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:57:13.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.369+0000 7f3685b80740 10 client.26563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:57:13.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.369+0000 7f3685b80740 20 client.26563 populate_metadata read hostname 'smithi006' 2024-04-26T01:57:13.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.369+0000 7f3685b80740 1 --2- 172.21.15.6:0/3529814976 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555c53b5dc80 0x555c53b60070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.369+0000 7f3685b80740 4 client.26563 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:57:13.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.369+0000 7f3685b80740 1 -- 172.21.15.6:0/3529814976 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555c53a50320 con 0x555c53b5dc80 2024-04-26T01:57:13.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.369+0000 7f367be00640 1 --2- 172.21.15.6:0/3529814976 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555c53b5dc80 0x555c53b60070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.369+0000 7f367be00640 1 --2- 172.21.15.6:0/3529814976 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555c53b5dc80 0x555c53b60070 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a86fddebeefe2eb server_cookie=f8f1966395cdb18 in_seq=0 out_seq=0 2024-04-26T01:57:13.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.369+0000 7f366a000640 10 client.26563 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:13.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.423+0000 7f366a000640 1 -- 172.21.15.6:0/3529814976 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555c53a50320 con 0x555c53b5dc80 2024-04-26T01:57:13.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.423+0000 7f366a000640 10 client.26563 handle_command_reply: tid=0 2024-04-26T01:57:13.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.423+0000 7f3685b80740 1 client.26563 shutdown 2024-04-26T01:57:13.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.423+0000 7f3685b80740 1 -- 172.21.15.6:0/3529814976 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 msgr2=0x555c53b5bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.423+0000 7f3685b80740 1 --2- 172.21.15.6:0/3529814976 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 0x555c53b5bf50 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f367402f810 tx=0x7f3674030f70 comp rx=0 tx=0).stop 2024-04-26T01:57:13.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.423+0000 7f3685b80740 1 -- 172.21.15.6:0/3529814976 shutdown_connections 2024-04-26T01:57:13.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.423+0000 7f3685b80740 1 --2- 172.21.15.6:0/3529814976 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555c53b5dc80 0x555c53b60070 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.424+0000 7f3685b80740 1 --2- 172.21.15.6:0/3529814976 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555c53ae7b90 0x555c53b5bf50 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.424+0000 7f3685b80740 1 --2- 172.21.15.6:0/3529814976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555c53add2d0 0x555c53afc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.424+0000 7f366a000640 0 client.26563 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:13.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.424+0000 7f3685b80740 1 --2- 172.21.15.6:0/3529814976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555c53adc9b0 0x555c53afc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.424+0000 7f3685b80740 1 -- 172.21.15.6:0/3529814976 >> 172.21.15.6:0/3529814976 conn(0x555c53ad8ec0 msgr2=0x555c53ae3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:13.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.424+0000 7f3685b80740 1 -- 172.21.15.6:0/3529814976 shutdown_connections 2024-04-26T01:57:13.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.424+0000 7f3685b80740 1 -- 172.21.15.6:0/3529814976 wait complete. 2024-04-26T01:57:13.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.424+0000 7f3685b80740 20 client.26563 trim_cache size 0 max 16384 2024-04-26T01:57:13.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:57:13.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:57:13.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:57:13.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:57:13.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.425+0000 7f3681800640 1 -- 172.21.15.6:0/3619096213 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36600a3d00 msgr2=0x7f36600a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.425+0000 7f3681800640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36600a3d00 0x7f36600a61c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f366c0095f0 tx=0x7f366c005f90 comp rx=0 tx=0).stop 2024-04-26T01:57:13.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.425+0000 7f3681800640 1 -- 172.21.15.6:0/3619096213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f367c103840 msgr2=0x7f367c1010b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.425+0000 7f3681800640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f367c103840 0x7f367c1010b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f366400e9f0 tx=0x7f366400eeb0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.425+0000 7f3681800640 1 -- 172.21.15.6:0/3619096213 shutdown_connections 2024-04-26T01:57:13.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.425+0000 7f3681800640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f36600a3d00 0x7f36600a61c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.425+0000 7f3681800640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f367c10c340 0x7f367c1015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.425+0000 7f3681800640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f367c103840 0x7f367c1010b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.425+0000 7f3681800640 1 --2- 172.21.15.6:0/3619096213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f367c102f20 0x7f367c100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.425+0000 7f3681800640 1 -- 172.21.15.6:0/3619096213 >> 172.21.15.6:0/3619096213 conn(0x7f367c0fc780 msgr2=0x7f367c0fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:13.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.425+0000 7f3681800640 1 -- 172.21.15.6:0/3619096213 shutdown_connections 2024-04-26T01:57:13.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:13.425+0000 7f3681800640 1 -- 172.21.15.6:0/3619096213 wait complete. 2024-04-26T01:57:13.627 INFO:tasks.workunit.client.0.smithi006.stdout: 1 401099 3.64 MB/sec execute 457 sec latency 65.289 ms 2024-04-26T01:57:14.627 INFO:tasks.workunit.client.0.smithi006.stdout: 1 401639 3.63 MB/sec execute 458 sec latency 26.552 ms 2024-04-26T01:57:14.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:14 smithi006 ceph-mon[28419]: pgmap v358: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 213 op/s 2024-04-26T01:57:14.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:14 smithi188 ceph-mon[32751]: pgmap v358: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 213 op/s 2024-04-26T01:57:14.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:14 smithi029 ceph-mon[32775]: pgmap v358: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 213 op/s 2024-04-26T01:57:15.628 INFO:tasks.workunit.client.0.smithi006.stdout: 1 402551 3.64 MB/sec execute 459 sec latency 28.487 ms 2024-04-26T01:57:16.628 INFO:tasks.workunit.client.0.smithi006.stdout: 1 403830 3.64 MB/sec execute 460 sec latency 47.721 ms 2024-04-26T01:57:16.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:16 smithi006 ceph-mon[28419]: pgmap v359: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 173 op/s 2024-04-26T01:57:16.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:16 smithi188 ceph-mon[32751]: pgmap v359: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 173 op/s 2024-04-26T01:57:16.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:16 smithi029 ceph-mon[32775]: pgmap v359: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 173 op/s 2024-04-26T01:57:17.628 INFO:tasks.workunit.client.0.smithi006.stdout: 1 404728 3.65 MB/sec execute 461 sec latency 22.954 ms 2024-04-26T01:57:18.628 INFO:tasks.workunit.client.0.smithi006.stdout: 1 405438 3.65 MB/sec execute 462 sec latency 25.344 ms 2024-04-26T01:57:18.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:18 smithi006 ceph-mon[28419]: pgmap v360: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-04-26T01:57:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:18 smithi188 ceph-mon[32751]: pgmap v360: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-04-26T01:57:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:18 smithi029 ceph-mon[32775]: pgmap v360: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-04-26T01:57:19.628 INFO:tasks.workunit.client.0.smithi006.stdout: 1 406735 3.65 MB/sec execute 463 sec latency 22.356 ms 2024-04-26T01:57:20.628 INFO:tasks.workunit.client.0.smithi006.stdout: 1 408156 3.67 MB/sec execute 464 sec latency 14.383 ms 2024-04-26T01:57:20.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:20 smithi006 ceph-mon[28419]: pgmap v361: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-04-26T01:57:20.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:20 smithi188 ceph-mon[32751]: pgmap v361: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-04-26T01:57:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:20 smithi029 ceph-mon[32775]: pgmap v361: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-04-26T01:57:21.628 INFO:tasks.workunit.client.0.smithi006.stdout: 1 408670 3.66 MB/sec execute 465 sec latency 51.409 ms 2024-04-26T01:57:22.628 INFO:tasks.workunit.client.0.smithi006.stdout: 1 409725 3.66 MB/sec execute 466 sec latency 23.771 ms 2024-04-26T01:57:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:22 smithi006 ceph-mon[28419]: pgmap v362: 97 pgs: 97 active+clean; 135 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 252 op/s 2024-04-26T01:57:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:22 smithi188 ceph-mon[32751]: pgmap v362: 97 pgs: 97 active+clean; 135 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 252 op/s 2024-04-26T01:57:22.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:22 smithi029 ceph-mon[32775]: pgmap v362: 97 pgs: 97 active+clean; 135 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 252 op/s 2024-04-26T01:57:23.628 INFO:tasks.workunit.client.0.smithi006.stdout: 1 411307 3.67 MB/sec execute 467 sec latency 20.627 ms 2024-04-26T01:57:24.628 INFO:tasks.workunit.client.0.smithi006.stdout: 1 412017 3.68 MB/sec execute 468 sec latency 20.255 ms 2024-04-26T01:57:24.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:24 smithi006 ceph-mon[28419]: pgmap v363: 97 pgs: 97 active+clean; 135 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-04-26T01:57:24.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:24 smithi188 ceph-mon[32751]: pgmap v363: 97 pgs: 97 active+clean; 135 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-04-26T01:57:24.940 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:24 smithi029 ceph-mon[32775]: pgmap v363: 97 pgs: 97 active+clean; 135 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-04-26T01:57:25.629 INFO:tasks.workunit.client.0.smithi006.stdout: 1 413150 3.68 MB/sec execute 469 sec latency 24.370 ms 2024-04-26T01:57:25.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:25 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:57:25.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:25 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:57:25.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:25 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:57:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.586+0000 7f16a1800640 1 -- 172.21.15.6:0/1965332243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169c100b30 msgr2=0x7f169c100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.586+0000 7f16a1800640 1 --2- 172.21.15.6:0/1965332243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169c100b30 0x7f169c100f10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f168c009900 tx=0x7f168c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.586+0000 7f16a1800640 1 -- 172.21.15.6:0/1965332243 shutdown_connections 2024-04-26T01:57:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.586+0000 7f16a1800640 1 --2- 172.21.15.6:0/1965332243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f169c10e6e0 0x7f169c110ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.586+0000 7f16a1800640 1 --2- 172.21.15.6:0/1965332243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f169c1014e0 0x7f169c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.586+0000 7f16a1800640 1 --2- 172.21.15.6:0/1965332243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169c100b30 0x7f169c100f10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.587+0000 7f16a1800640 1 -- 172.21.15.6:0/1965332243 >> 172.21.15.6:0/1965332243 conn(0x7f169c0fc730 msgr2=0x7f169c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.587+0000 7f16a1800640 1 -- 172.21.15.6:0/1965332243 shutdown_connections 2024-04-26T01:57:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.587+0000 7f16a1800640 1 -- 172.21.15.6:0/1965332243 wait complete. 2024-04-26T01:57:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.588+0000 7f16a1800640 1 Processor -- start 2024-04-26T01:57:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f16a1800640 1 -- start start 2024-04-26T01:57:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f16a1800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169c100b30 0x7f169c1a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f16a1800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f169c1014e0 0x7f169c1a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f16a1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f169c10e6e0 0x7f169c19a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f16a1800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f169c114270 con 0x7f169c10e6e0 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f16a1800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f169c1140f0 con 0x7f169c100b30 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f16a1800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f169c1143f0 con 0x7f169c1014e0 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f169be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169c100b30 0x7f169c1a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f169b400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f169c1014e0 0x7f169c1a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f169be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169c100b30 0x7f169c1a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36710/0 (socket says 172.21.15.6:36710) 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f169b400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f169c1014e0 0x7f169c1a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47462/0 (socket says 172.21.15.6:47462) 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f169be00640 1 -- 172.21.15.6:0/518468545 learned_addr learned my addr 172.21.15.6:0/518468545 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.589+0000 7f16a0e00640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f169c10e6e0 0x7f169c19a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.590+0000 7f169be00640 1 -- 172.21.15.6:0/518468545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f169c1014e0 msgr2=0x7f169c1a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.590+0000 7f169be00640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f169c1014e0 0x7f169c1a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.590+0000 7f169be00640 1 -- 172.21.15.6:0/518468545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f169c10e6e0 msgr2=0x7f169c19a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.590+0000 7f169be00640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f169c10e6e0 0x7f169c19a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.590+0000 7f169be00640 1 -- 172.21.15.6:0/518468545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c19ad50 con 0x7f169c100b30 2024-04-26T01:57:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.590+0000 7f169be00640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169c100b30 0x7f169c1a0420 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f168c005ab0 tx=0x7f168c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.590+0000 7f1698c00640 1 -- 172.21.15.6:0/518468545 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c03f070 con 0x7f169c100b30 2024-04-26T01:57:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.590+0000 7f1698c00640 1 -- 172.21.15.6:0/518468545 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f168c031b60 con 0x7f169c100b30 2024-04-26T01:57:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.590+0000 7f1698c00640 1 -- 172.21.15.6:0/518468545 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c035030 con 0x7f169c100b30 2024-04-26T01:57:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.590+0000 7f16a1800640 1 -- 172.21.15.6:0/518468545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f169c19afe0 con 0x7f169c100b30 2024-04-26T01:57:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.591+0000 7f16a1800640 1 -- 172.21.15.6:0/518468545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f169c1a72e0 con 0x7f169c100b30 2024-04-26T01:57:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.594+0000 7f1698c00640 1 -- 172.21.15.6:0/518468545 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f168c042070 con 0x7f169c100b30 2024-04-26T01:57:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.594+0000 7f16a5bf7740 1 Processor -- start 2024-04-26T01:57:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f16a5bf7740 1 -- start start 2024-04-26T01:57:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f16a5bf7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 0x5640f6313d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f16a5bf7740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f631e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f16a5bf7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f631eb90 0x5640f6320f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f16a5bf7740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5640f6279900 con 0x5640f63142d0 2024-04-26T01:57:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f16a5bf7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5640f621fef0 con 0x5640f631eb90 2024-04-26T01:57:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f16a5bf7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5640f62758c0 con 0x5640f63139b0 2024-04-26T01:57:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f169b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f631eb90 0x5640f6320f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f16a0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 0x5640f6313d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f169b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f631eb90 0x5640f6320f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36722/0 (socket says 172.21.15.6:36722) 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f16a0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 0x5640f6313d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47468/0 (socket says 172.21.15.6:47468) 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f169b400640 1 -- 172.21.15.6:0/1503212869 learned_addr learned my addr 172.21.15.6:0/1503212869 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.595+0000 7f169be00640 1 --2- 172.21.15.6:0/1503212869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f631e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.596+0000 7f1698c00640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f16800a3d80 0x7f16800a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.596+0000 7f169b400640 1 -- 172.21.15.6:0/1503212869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 msgr2=0x5640f6313d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.596+0000 7f169b400640 1 --2- 172.21.15.6:0/1503212869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 0x5640f6313d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.596+0000 7f169b400640 1 -- 172.21.15.6:0/1503212869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 msgr2=0x5640f631e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.596+0000 7f169b400640 1 --2- 172.21.15.6:0/1503212869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f631e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.596+0000 7f169b400640 1 -- 172.21.15.6:0/1503212869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640f5f8c210 con 0x5640f631eb90 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.596+0000 7f169be00640 1 --2- 172.21.15.6:0/1503212869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f631e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.596+0000 7f1698c00640 1 -- 172.21.15.6:0/518468545 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7f168c0f0e50 con 0x7f169c100b30 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.596+0000 7f16a0e00640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f16800a3d80 0x7f16800a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.596+0000 7f169b400640 1 --2- 172.21.15.6:0/1503212869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f631eb90 0x5640f6320f80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1688009c60 tx=0x7f1688011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa4308282490e80d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.596+0000 7f1691600640 1 -- 172.21.15.6:0/1503212869 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1688004230 con 0x5640f631eb90 2024-04-26T01:57:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.597+0000 7f1691600640 1 -- 172.21.15.6:0/1503212869 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16880043d0 con 0x5640f631eb90 2024-04-26T01:57:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.597+0000 7f1691600640 1 -- 172.21.15.6:0/1503212869 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1688004ca0 con 0x5640f631eb90 2024-04-26T01:57:26.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.597+0000 7f16a5bf7740 1 -- 172.21.15.6:0/1503212869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f631eb90 msgr2=0x5640f6320f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.597+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/1503212869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f631eb90 0x5640f6320f80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1688009c60 tx=0x7f1688011810 comp rx=0 tx=0).stop 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.597+0000 7f16a0e00640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f16800a3d80 0x7f16800a6240 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1694009b80 tx=0x7f1694004a40 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.598+0000 7f16a5bf7740 1 -- 172.21.15.6:0/1503212869 shutdown_connections 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.598+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/1503212869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f631e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.598+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/1503212869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f631eb90 0x5640f6320f80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.598+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/1503212869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 0x5640f6313d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.598+0000 7f16a5bf7740 1 -- 172.21.15.6:0/1503212869 >> 172.21.15.6:0/1503212869 conn(0x5640f630fec0 msgr2=0x5640f63102a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.598+0000 7f16a5bf7740 1 -- 172.21.15.6:0/1503212869 shutdown_connections 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.598+0000 7f16a5bf7740 1 -- 172.21.15.6:0/1503212869 wait complete. 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.598+0000 7f16a5bf7740 1 -- start start 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.599+0000 7f16a5bf7740 1 Processor -- start 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.599+0000 7f16a5bf7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 0x5640f631b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.599+0000 7f16a5bf7740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f631bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.599+0000 7f16a5bf7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f631eb90 0x5640f631c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.599+0000 7f16a5bf7740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5640f63217c0 con 0x5640f63142d0 2024-04-26T01:57:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.599+0000 7f16a5bf7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5640f6233310 con 0x5640f631eb90 2024-04-26T01:57:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.599+0000 7f16a5bf7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5640f6321940 con 0x5640f63139b0 2024-04-26T01:57:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.599+0000 7f16a5bf7740 10 client.0 _ll_register_callbacks cb 0x5640f6201cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.599+0000 7f169be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 0x5640f631b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.600+0000 7f169be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 0x5640f631b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47484/0 (socket says 172.21.15.6:47484) 2024-04-26T01:57:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.600+0000 7f169be00640 1 -- 172.21.15.6:0/283207262 learned_addr learned my addr 172.21.15.6:0/283207262 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.600+0000 7f169b400640 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f631bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.600+0000 7f16a0e00640 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f631eb90 0x5640f631c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.600+0000 7f169be00640 1 -- 172.21.15.6:0/283207262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f631eb90 msgr2=0x5640f631c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.600+0000 7f169be00640 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f631eb90 0x5640f631c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.600+0000 7f169be00640 1 -- 172.21.15.6:0/283207262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 msgr2=0x5640f631bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.600+0000 7f169be00640 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f631bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.600+0000 7f169be00640 1 -- 172.21.15.6:0/283207262 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640f6058620 con 0x5640f63139b0 2024-04-26T01:57:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.600+0000 7f169be00640 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 0x5640f631b9d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f168c000c00 tx=0x7f168c0bc880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12d19f34351037a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.600+0000 7f1686000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:57:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.601+0000 7f1686000640 1 -- 172.21.15.6:0/283207262 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c049070 con 0x5640f63139b0 2024-04-26T01:57:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.601+0000 7f1686000640 1 -- 172.21.15.6:0/283207262 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7f168c0b93b0 con 0x5640f63139b0 2024-04-26T01:57:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.601+0000 7f16a5bf7740 1 -- 172.21.15.6:0/283207262 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16880043d0 con 0x5640f63139b0 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.601+0000 7f1686000640 1 -- 172.21.15.6:0/283207262 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f168c0d62c0 con 0x5640f63139b0 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.601+0000 7f16a5bf7740 10 client.26575 fetch_fsmap learned FSMap version 43 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.601+0000 7f16a5bf7740 1 -- 172.21.15.6:0/283207262 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5640f63217c0 con 0x5640f63139b0 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.601+0000 7f1686000640 1 -- 172.21.15.6:0/283207262 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16880043d0 con 0x5640f63139b0 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.601+0000 7f1686000640 1 -- 172.21.15.6:0/283207262 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f168c0ee770 con 0x5640f63139b0 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.601+0000 7f16a5bf7740 10 client.26575 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.601+0000 7f16a5bf7740 10 client.26575 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.602+0000 7f16a5bf7740 20 client.26575 populate_metadata read hostname 'smithi006' 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.602+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5640f632edb0 0x5640f6331220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.602+0000 7f16a5bf7740 4 client.26575 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.602+0000 7f16a5bf7740 1 -- 172.21.15.6:0/283207262 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640f628d8b0 con 0x5640f632edb0 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.602+0000 7f169b400640 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5640f632edb0 0x5640f6331220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.602+0000 7f169b400640 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5640f632edb0 0x5640f6331220 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4935a09d9ab257b8 server_cookie=67902b9db5ffe680 in_seq=0 out_seq=0 2024-04-26T01:57:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.602+0000 7f1686000640 10 client.26575 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:26.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.603+0000 7f1686000640 1 -- 172.21.15.6:0/283207262 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640f628d8b0 con 0x5640f632edb0 2024-04-26T01:57:26.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.603+0000 7f1686000640 10 client.26575 handle_command_reply: tid=0 2024-04-26T01:57:26.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.603+0000 7f16a5bf7740 1 client.26575 shutdown 2024-04-26T01:57:26.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.604+0000 7f16a5bf7740 1 -- 172.21.15.6:0/283207262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 msgr2=0x5640f631b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.604+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 0x5640f631b9d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f168c000c00 tx=0x7f168c0bc880 comp rx=0 tx=0).stop 2024-04-26T01:57:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.604+0000 7f16a5bf7740 1 -- 172.21.15.6:0/283207262 shutdown_connections 2024-04-26T01:57:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.604+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5640f632edb0 0x5640f6331220 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.604+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f631bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.604+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f631eb90 0x5640f631c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.604+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/283207262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63139b0 0x5640f631b9d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.604+0000 7f16a5bf7740 1 -- 172.21.15.6:0/283207262 >> 172.21.15.6:0/283207262 conn(0x5640f630fec0 msgr2=0x5640f6315c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.604+0000 7f1686000640 0 client.26575 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.604+0000 7f16a5bf7740 1 -- 172.21.15.6:0/283207262 shutdown_connections 2024-04-26T01:57:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.604+0000 7f16a5bf7740 1 -- 172.21.15.6:0/283207262 wait complete. 2024-04-26T01:57:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.604+0000 7f16a5bf7740 20 client.26575 trim_cache size 0 max 16384 2024-04-26T01:57:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.624+0000 7f16a5bf7740 1 Processor -- start 2024-04-26T01:57:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.624+0000 7f16a5bf7740 1 -- start start 2024-04-26T01:57:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f16a5bf7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f63139b0 0x5640f632a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f16a5bf7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63142d0 0x5640f631b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f16a5bf7740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f631eb90 0x5640f6336b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f16a5bf7740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f169c1b2640 con 0x5640f631eb90 2024-04-26T01:57:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f16a5bf7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f169c1aff10 con 0x5640f63139b0 2024-04-26T01:57:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f16a5bf7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f169c1add80 con 0x5640f63142d0 2024-04-26T01:57:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f169be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f63139b0 0x5640f632a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f169be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f63139b0 0x5640f632a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36738/0 (socket says 172.21.15.6:36738) 2024-04-26T01:57:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f169be00640 1 -- 172.21.15.6:0/1971070567 learned_addr learned my addr 172.21.15.6:0/1971070567 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f169b400640 1 --2- 172.21.15.6:0/1971070567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63142d0 0x5640f631b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f16a0e00640 1 --2- 172.21.15.6:0/1971070567 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f631eb90 0x5640f6336b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f169be00640 1 -- 172.21.15.6:0/1971070567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63142d0 msgr2=0x5640f631b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f169be00640 1 --2- 172.21.15.6:0/1971070567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63142d0 0x5640f631b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f169be00640 1 -- 172.21.15.6:0/1971070567 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f631eb90 msgr2=0x5640f6336b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f169be00640 1 --2- 172.21.15.6:0/1971070567 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f631eb90 0x5640f6336b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f169be00640 1 -- 172.21.15.6:0/1971070567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640f630fcf0 con 0x5640f63139b0 2024-04-26T01:57:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f169b400640 1 --2- 172.21.15.6:0/1971070567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63142d0 0x5640f631b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.625+0000 7f16a0e00640 1 --2- 172.21.15.6:0/1971070567 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f631eb90 0x5640f6336b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:57:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.626+0000 7f169be00640 1 --2- 172.21.15.6:0/1971070567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f63139b0 0x5640f632a6b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f168c031440 tx=0x7f168c0bb250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc4570c074c9f4b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.626+0000 7f1690c00640 1 -- 172.21.15.6:0/1971070567 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c049030 con 0x5640f63139b0 2024-04-26T01:57:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.626+0000 7f1690c00640 1 -- 172.21.15.6:0/1971070567 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f168c0c5780 con 0x5640f63139b0 2024-04-26T01:57:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.626+0000 7f1690c00640 1 -- 172.21.15.6:0/1971070567 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c0dd110 con 0x5640f63139b0 2024-04-26T01:57:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.627+0000 7f16a5bf7740 1 -- 172.21.15.6:0/1971070567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f63139b0 msgr2=0x5640f632a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.627+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/1971070567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f63139b0 0x5640f632a6b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f168c031440 tx=0x7f168c0bb250 comp rx=0 tx=0).stop 2024-04-26T01:57:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.627+0000 7f16a5bf7740 1 -- 172.21.15.6:0/1971070567 shutdown_connections 2024-04-26T01:57:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.627+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/1971070567 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f631eb90 0x5640f6336b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.627+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/1971070567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f63142d0 0x5640f631b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.627+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/1971070567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f63139b0 0x5640f632a6b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.627+0000 7f16a5bf7740 1 -- 172.21.15.6:0/1971070567 >> 172.21.15.6:0/1971070567 conn(0x5640f630fec0 msgr2=0x5640f630f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.627+0000 7f16a5bf7740 1 -- 172.21.15.6:0/1971070567 shutdown_connections 2024-04-26T01:57:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.627+0000 7f16a5bf7740 1 -- 172.21.15.6:0/1971070567 wait complete. 2024-04-26T01:57:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.627+0000 7f16a5bf7740 1 -- start start 2024-04-26T01:57:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.628+0000 7f16a5bf7740 1 Processor -- start 2024-04-26T01:57:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.628+0000 7f16a5bf7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f63139b0 0x5640f6326d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.628+0000 7f16a5bf7740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f63240d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.628+0000 7f16a5bf7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f631eb90 0x5640f6324610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.628+0000 7f16a5bf7740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f169c1ae3d0 con 0x5640f63142d0 2024-04-26T01:57:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.628+0000 7f16a5bf7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f169c1af840 con 0x5640f63139b0 2024-04-26T01:57:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.628+0000 7f16a5bf7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f169c1aeaa0 con 0x5640f631eb90 2024-04-26T01:57:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.628+0000 7f16a5bf7740 10 client.0 _ll_register_callbacks cb 0x7f168c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.629+0000 7f169b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f63240d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.629+0000 7f16a0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f631eb90 0x5640f6324610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.629+0000 7f16a0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f631eb90 0x5640f6324610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47502/0 (socket says 172.21.15.6:47502) 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.629+0000 7f16a0e00640 1 -- 172.21.15.6:0/721492632 learned_addr learned my addr 172.21.15.6:0/721492632 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.629+0000 7f169be00640 1 --2- 172.21.15.6:0/721492632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f63139b0 0x5640f6326d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.629+0000 7f16a0e00640 1 -- 172.21.15.6:0/721492632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f63139b0 msgr2=0x5640f6326d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.629+0000 7f16a0e00640 1 --2- 172.21.15.6:0/721492632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f63139b0 0x5640f6326d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.629+0000 7f16a0e00640 1 -- 172.21.15.6:0/721492632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 msgr2=0x5640f63240d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.629+0000 7f16a0e00640 1 --2- 172.21.15.6:0/721492632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f63240d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.629+0000 7f16a0e00640 1 -- 172.21.15.6:0/721492632 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640f6316820 con 0x5640f631eb90 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.629+0000 7f16a0e00640 1 --2- 172.21.15.6:0/721492632 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f631eb90 0x5640f6324610 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1694005fd0 tx=0x7f1694018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=361a0c3777c8a956 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.630+0000 7f1686000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.630+0000 7f1686000640 1 -- 172.21.15.6:0/721492632 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169401cdf0 con 0x5640f631eb90 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.630+0000 7f1686000640 1 -- 172.21.15.6:0/721492632 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7f1694003970 con 0x5640f631eb90 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.630+0000 7f16a5bf7740 1 -- 172.21.15.6:0/721492632 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f168c0c5780 con 0x5640f631eb90 2024-04-26T01:57:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.630+0000 7f1686000640 1 -- 172.21.15.6:0/721492632 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f1694037050 con 0x5640f631eb90 2024-04-26T01:57:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.630+0000 7f16a5bf7740 10 client.26581 fetch_fsmap learned FSMap version 43 2024-04-26T01:57:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.630+0000 7f16a5bf7740 1 -- 172.21.15.6:0/721492632 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5640f630f8e0 con 0x5640f631eb90 2024-04-26T01:57:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.630+0000 7f1686000640 1 -- 172.21.15.6:0/721492632 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f168c0c5780 con 0x5640f631eb90 2024-04-26T01:57:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.631+0000 7f1686000640 1 -- 172.21.15.6:0/721492632 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f1694029400 con 0x5640f631eb90 2024-04-26T01:57:26.630 INFO:tasks.workunit.client.0.smithi006.stdout: 1 414699 3.69 MB/sec execute 470 sec latency 22.165 ms 2024-04-26T01:57:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.631+0000 7f16a5bf7740 10 client.26581 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:57:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.631+0000 7f16a5bf7740 10 client.26581 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:57:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.631+0000 7f16a5bf7740 20 client.26581 populate_metadata read hostname 'smithi006' 2024-04-26T01:57:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.631+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/721492632 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5640f6395510 0x5640f6397930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.631+0000 7f16a5bf7740 4 client.26581 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:57:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.631+0000 7f16a5bf7740 1 -- 172.21.15.6:0/721492632 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640f6287320 con 0x5640f6395510 2024-04-26T01:57:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.631+0000 7f169be00640 1 --2- 172.21.15.6:0/721492632 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5640f6395510 0x5640f6397930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.631+0000 7f169be00640 1 --2- 172.21.15.6:0/721492632 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5640f6395510 0x5640f6397930 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af4371f0accfc1fc server_cookie=e4253748fe101242 in_seq=0 out_seq=0 2024-04-26T01:57:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.631+0000 7f1686000640 10 client.26581 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:26.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f1686000640 1 -- 172.21.15.6:0/721492632 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640f6287320 con 0x5640f6395510 2024-04-26T01:57:26.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f1686000640 10 client.26581 handle_command_reply: tid=0 2024-04-26T01:57:26.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f16a5bf7740 1 client.26581 shutdown 2024-04-26T01:57:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f16a5bf7740 1 -- 172.21.15.6:0/721492632 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f631eb90 msgr2=0x5640f6324610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/721492632 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f631eb90 0x5640f6324610 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1694005fd0 tx=0x7f1694018000 comp rx=0 tx=0).stop 2024-04-26T01:57:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f16a5bf7740 1 -- 172.21.15.6:0/721492632 shutdown_connections 2024-04-26T01:57:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/721492632 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5640f6395510 0x5640f6397930 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/721492632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5640f63142d0 0x5640f63240d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/721492632 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5640f631eb90 0x5640f6324610 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f16a5bf7740 1 --2- 172.21.15.6:0/721492632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640f63139b0 0x5640f6326d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f1686000640 0 client.26581 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f16a5bf7740 1 -- 172.21.15.6:0/721492632 >> 172.21.15.6:0/721492632 conn(0x5640f630fec0 msgr2=0x5640f630e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f16a5bf7740 1 -- 172.21.15.6:0/721492632 shutdown_connections 2024-04-26T01:57:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.687+0000 7f16a5bf7740 1 -- 172.21.15.6:0/721492632 wait complete. 2024-04-26T01:57:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.688+0000 7f16a5bf7740 20 client.26581 trim_cache size 0 max 16384 2024-04-26T01:57:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:57:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:57:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:57:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:57:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.688+0000 7f16a1800640 1 -- 172.21.15.6:0/518468545 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f16800a3d80 msgr2=0x7f16800a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.688+0000 7f16a1800640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f16800a3d80 0x7f16800a6240 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1694009b80 tx=0x7f1694004a40 comp rx=0 tx=0).stop 2024-04-26T01:57:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.688+0000 7f16a1800640 1 -- 172.21.15.6:0/518468545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169c100b30 msgr2=0x7f169c1a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.688+0000 7f16a1800640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169c100b30 0x7f169c1a0420 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f168c005ab0 tx=0x7f168c003c10 comp rx=0 tx=0).stop 2024-04-26T01:57:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.688+0000 7f16a1800640 1 -- 172.21.15.6:0/518468545 shutdown_connections 2024-04-26T01:57:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.688+0000 7f16a1800640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f16800a3d80 0x7f16800a6240 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.688+0000 7f16a1800640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f169c10e6e0 0x7f169c19a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.688+0000 7f16a1800640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f169c1014e0 0x7f169c1a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.688+0000 7f16a1800640 1 --2- 172.21.15.6:0/518468545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169c100b30 0x7f169c1a0420 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.688+0000 7f16a1800640 1 -- 172.21.15.6:0/518468545 >> 172.21.15.6:0/518468545 conn(0x7f169c0fc730 msgr2=0x7f169c0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.689+0000 7f16a1800640 1 -- 172.21.15.6:0/518468545 shutdown_connections 2024-04-26T01:57:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:26.689+0000 7f16a1800640 1 -- 172.21.15.6:0/518468545 wait complete. 2024-04-26T01:57:26.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:26 smithi006 ceph-mon[28419]: pgmap v364: 97 pgs: 97 active+clean; 135 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-04-26T01:57:26.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:26 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:57:26.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:26 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:57:26.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:26 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:57:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:26 smithi188 ceph-mon[32751]: pgmap v364: 97 pgs: 97 active+clean; 135 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-04-26T01:57:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:26 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:57:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:26 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:57:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:26 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:57:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:26 smithi029 ceph-mon[32775]: pgmap v364: 97 pgs: 97 active+clean; 135 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-04-26T01:57:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:26 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:57:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:26 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:57:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:26 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:57:27.629 INFO:tasks.workunit.client.0.smithi006.stdout: 1 415471 3.69 MB/sec execute 471 sec latency 58.033 ms 2024-04-26T01:57:28.629 INFO:tasks.workunit.client.0.smithi006.stdout: 1 416475 3.70 MB/sec execute 472 sec latency 30.075 ms 2024-04-26T01:57:28.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:28 smithi006 ceph-mon[28419]: pgmap v365: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 242 op/s 2024-04-26T01:57:28.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:28 smithi188 ceph-mon[32751]: pgmap v365: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 242 op/s 2024-04-26T01:57:28.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:28 smithi029 ceph-mon[32775]: pgmap v365: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.6 MiB/s wr, 242 op/s 2024-04-26T01:57:29.629 INFO:tasks.workunit.client.0.smithi006.stdout: 1 417913 3.70 MB/sec execute 473 sec latency 24.772 ms 2024-04-26T01:57:30.629 INFO:tasks.workunit.client.0.smithi006.stdout: 1 418947 3.71 MB/sec execute 474 sec latency 15.245 ms 2024-04-26T01:57:30.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:30 smithi006 ceph-mon[28419]: pgmap v366: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 146 op/s 2024-04-26T01:57:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:30 smithi188 ceph-mon[32751]: pgmap v366: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 146 op/s 2024-04-26T01:57:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:30 smithi029 ceph-mon[32775]: pgmap v366: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 146 op/s 2024-04-26T01:57:31.629 INFO:tasks.workunit.client.0.smithi006.stdout: 1 419709 3.71 MB/sec execute 475 sec latency 36.573 ms 2024-04-26T01:57:31.733 INFO:tasks.quiescer.fs.[cephfs]:Total duration (38.5) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T01:57:31.734 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.25 2024-04-26T01:57:31.734 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:57:32.218 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:32.629 INFO:tasks.workunit.client.0.smithi006.stdout: 1 421061 3.71 MB/sec execute 476 sec latency 22.994 ms 2024-04-26T01:57:32.630 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:32 smithi006 ceph-mon[28419]: pgmap v367: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-04-26T01:57:32.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:32 smithi188 ceph-mon[32751]: pgmap v367: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-04-26T01:57:32.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:32 smithi029 ceph-mon[32775]: pgmap v367: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-04-26T01:57:33.629 INFO:tasks.workunit.client.0.smithi006.stdout: 1 422363 3.73 MB/sec execute 477 sec latency 30.940 ms 2024-04-26T01:57:34.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.523+0000 7facc5400700 1 -- 172.21.15.6:0/151787549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facc0074c70 msgr2=0x7facc0075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:34.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.523+0000 7facc5400700 1 --2- 172.21.15.6:0/151787549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facc0074c70 0x7facc0075050 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7facb00098f0 tx=0x7facb0009c00 comp rx=0 tx=0).stop 2024-04-26T01:57:34.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.523+0000 7facc5400700 1 -- 172.21.15.6:0/151787549 shutdown_connections 2024-04-26T01:57:34.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.523+0000 7facc5400700 1 --2- 172.21.15.6:0/151787549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facc010f440 0x7facc0111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:34.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.523+0000 7facc5400700 1 --2- 172.21.15.6:0/151787549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facc0074c70 0x7facc0075050 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:34.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.523+0000 7facc5400700 1 --2- 172.21.15.6:0/151787549 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facc0075590 0x7facc010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:34.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.523+0000 7facc5400700 1 -- 172.21.15.6:0/151787549 >> 172.21.15.6:0/151787549 conn(0x7facc0079ba0 msgr2=0x7facc0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:34.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.523+0000 7facc5400700 1 -- 172.21.15.6:0/151787549 shutdown_connections 2024-04-26T01:57:34.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.523+0000 7facc5400700 1 -- 172.21.15.6:0/151787549 wait complete. 2024-04-26T01:57:34.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.524+0000 7facc5400700 1 Processor -- start 2024-04-26T01:57:34.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.524+0000 7facc5400700 1 -- start start 2024-04-26T01:57:34.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facc5400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facc0074c70 0x7facc019b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:34.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facc5400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facc0075590 0x7facc019b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:34.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facc5400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facc010f440 0x7facc019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:34.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facc5400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7facc0114e60 con 0x7facc0075590 2024-04-26T01:57:34.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facc5400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7facc0114ce0 con 0x7facc010f440 2024-04-26T01:57:34.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facc5400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7facc0114fe0 con 0x7facc0074c70 2024-04-26T01:57:34.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facbe800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facc0074c70 0x7facc019b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:34.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facbe800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facc0074c70 0x7facc019b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:35060/0 (socket says 172.21.15.6:35060) 2024-04-26T01:57:34.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facbe800700 1 -- 172.21.15.6:0/668481972 learned_addr learned my addr 172.21.15.6:0/668481972 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:34.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facbf200700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facc010f440 0x7facc019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:34.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facbde00700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facc0075590 0x7facc019b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:34.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facbe800700 1 -- 172.21.15.6:0/668481972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facc010f440 msgr2=0x7facc019fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:34.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facbe800700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facc010f440 0x7facc019fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:34.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facbe800700 1 -- 172.21.15.6:0/668481972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facc0075590 msgr2=0x7facc019b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:34.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.525+0000 7facbe800700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facc0075590 0x7facc019b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:34.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.526+0000 7facbe800700 1 -- 172.21.15.6:0/668481972 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc01a0370 con 0x7facc0074c70 2024-04-26T01:57:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.526+0000 7facbde00700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facc0075590 0x7facc019b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:57:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.527+0000 7facbe800700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facc0074c70 0x7facc019b3b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7facb0009230 tx=0x7facb000fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.527+0000 7facb7400700 1 -- 172.21.15.6:0/668481972 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb0005140 con 0x7facc0074c70 2024-04-26T01:57:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.527+0000 7facb7400700 1 -- 172.21.15.6:0/668481972 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facb000b560 con 0x7facc0074c70 2024-04-26T01:57:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.527+0000 7facb7400700 1 -- 172.21.15.6:0/668481972 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb0003ae0 con 0x7facc0074c70 2024-04-26T01:57:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.527+0000 7facc5400700 1 -- 172.21.15.6:0/668481972 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc01a0600 con 0x7facc0074c70 2024-04-26T01:57:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.527+0000 7facc5400700 1 -- 172.21.15.6:0/668481972 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc01a7ee0 con 0x7facc0074c70 2024-04-26T01:57:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.528+0000 7facc5400700 1 -- 172.21.15.6:0/668481972 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facc0076360 con 0x7facc0074c70 2024-04-26T01:57:34.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.530+0000 7facb7400700 1 -- 172.21.15.6:0/668481972 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7facb0003c80 con 0x7facc0074c70 2024-04-26T01:57:34.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.532+0000 7facb7400700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7faca40a3e30 0x7faca40a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:34.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.532+0000 7facb7400700 1 -- 172.21.15.6:0/668481972 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7facb00d17d0 con 0x7facc0074c70 2024-04-26T01:57:34.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.532+0000 7facbde00700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7faca40a3e30 0x7faca40a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:34.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.533+0000 7facbde00700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7faca40a3e30 0x7faca40a62f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7facc01111d0 tx=0x7facac009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:34.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.536+0000 7facb7400700 1 -- 172.21.15.6:0/668481972 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7facb0013030 con 0x7facc0074c70 2024-04-26T01:57:34.607 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:34 smithi006 ceph-mon[28419]: pgmap v368: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-04-26T01:57:34.629 INFO:tasks.workunit.client.0.smithi006.stdout: 1 423121 3.72 MB/sec execute 478 sec latency 21.990 ms 2024-04-26T01:57:34.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.843+0000 7facc5400700 1 -- 172.21.15.6:0/668481972 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facc002df80 con 0x7facc0074c70 2024-04-26T01:57:34.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.844+0000 7facb7400700 1 -- 172.21.15.6:0/668481972 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11850 (secure 0 0 0) 0x7facb009de50 con 0x7facc0074c70 2024-04-26T01:57:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:57:34.842 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":43,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:55:18.912671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:57:34.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.848+0000 7facc5400700 1 -- 172.21.15.6:0/668481972 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7faca40a3e30 msgr2=0x7faca40a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:34.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.848+0000 7facc5400700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7faca40a3e30 0x7faca40a62f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7facc01111d0 tx=0x7facac009260 comp rx=0 tx=0).stop 2024-04-26T01:57:34.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.848+0000 7facc5400700 1 -- 172.21.15.6:0/668481972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facc0074c70 msgr2=0x7facc019b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:34.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.848+0000 7facc5400700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facc0074c70 0x7facc019b3b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7facb0009230 tx=0x7facb000fa70 comp rx=0 tx=0).stop 2024-04-26T01:57:34.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.848+0000 7facc5400700 1 -- 172.21.15.6:0/668481972 shutdown_connections 2024-04-26T01:57:34.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.848+0000 7facc5400700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7faca40a3e30 0x7faca40a62f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:34.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.848+0000 7facc5400700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facc010f440 0x7facc019fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:34.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.848+0000 7facc5400700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7facc0074c70 0x7facc019b3b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:34.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.848+0000 7facc5400700 1 --2- 172.21.15.6:0/668481972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facc0075590 0x7facc019b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:34.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.848+0000 7facc5400700 1 -- 172.21.15.6:0/668481972 >> 172.21.15.6:0/668481972 conn(0x7facc0079ba0 msgr2=0x7facc00ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:34.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.848+0000 7facc5400700 1 -- 172.21.15.6:0/668481972 shutdown_connections 2024-04-26T01:57:34.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:34.848+0000 7facc5400700 1 -- 172.21.15.6:0/668481972 wait complete. 2024-04-26T01:57:34.847 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-26T01:57:34.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:34 smithi188 ceph-mon[32751]: pgmap v368: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-04-26T01:57:34.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:34 smithi029 ceph-mon[32775]: pgmap v368: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-04-26T01:57:35.630 INFO:tasks.workunit.client.0.smithi006.stdout: 1 423897 3.72 MB/sec execute 479 sec latency 49.926 ms 2024-04-26T01:57:35.806 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:35 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/668481972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:35.811 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 79.25 --await-for 42.0' 2024-04-26T01:57:35.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 79.25 --await-for 42.0 2024-04-26T01:57:35.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:35 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/668481972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:35.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:35 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/668481972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:36.266 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:36.630 INFO:tasks.workunit.client.0.smithi006.stdout: 1 425136 3.73 MB/sec execute 480 sec latency 18.562 ms 2024-04-26T01:57:36.696 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:36 smithi006 ceph-mon[28419]: pgmap v369: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-04-26T01:57:36.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:36 smithi188 ceph-mon[32751]: pgmap v369: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-04-26T01:57:36.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:36 smithi029 ceph-mon[32775]: pgmap v369: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-04-26T01:57:37.630 INFO:tasks.workunit.client.0.smithi006.stdout: 1 426065 3.74 MB/sec execute 481 sec latency 20.992 ms 2024-04-26T01:57:38.630 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427166 3.74 MB/sec execute 482 sec latency 19.572 ms 2024-04-26T01:57:38.775 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:38 smithi006 ceph-mon[28419]: pgmap v370: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 239 op/s 2024-04-26T01:57:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.860+0000 7fdc87400700 1 -- 172.21.15.6:0/189716360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc80075590 msgr2=0x7fdc8010ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.860+0000 7fdc87400700 1 --2- 172.21.15.6:0/189716360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc80075590 0x7fdc8010ef60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c009990 tx=0x7fdc6c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.860+0000 7fdc87400700 1 -- 172.21.15.6:0/189716360 shutdown_connections 2024-04-26T01:57:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.860+0000 7fdc87400700 1 --2- 172.21.15.6:0/189716360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc8010f4a0 0x7fdc80111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.860+0000 7fdc87400700 1 --2- 172.21.15.6:0/189716360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc80074c70 0x7fdc80075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.860+0000 7fdc87400700 1 --2- 172.21.15.6:0/189716360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc80075590 0x7fdc8010ef60 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.860+0000 7fdc87400700 1 -- 172.21.15.6:0/189716360 >> 172.21.15.6:0/189716360 conn(0x7fdc80079b10 msgr2=0x7fdc80079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.860+0000 7fdc87400700 1 -- 172.21.15.6:0/189716360 shutdown_connections 2024-04-26T01:57:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.860+0000 7fdc87400700 1 -- 172.21.15.6:0/189716360 wait complete. 2024-04-26T01:57:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.861+0000 7fdc87400700 1 Processor -- start 2024-04-26T01:57:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.861+0000 7fdc87400700 1 -- start start 2024-04-26T01:57:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.861+0000 7fdc87400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc80074c70 0x7fdc8019b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc87400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc80075590 0x7fdc8019b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc87400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc8010f4a0 0x7fdc8019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc87400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdc80114e60 con 0x7fdc80074c70 2024-04-26T01:57:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc87400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdc80114ce0 con 0x7fdc80075590 2024-04-26T01:57:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc87400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdc80114fe0 con 0x7fdc8010f4a0 2024-04-26T01:57:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc84a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc80074c70 0x7fdc8019b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc84a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc80074c70 0x7fdc8019b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37856/0 (socket says 172.21.15.6:37856) 2024-04-26T01:57:38.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc84a00700 1 -- 172.21.15.6:0/1696162006 learned_addr learned my addr 172.21.15.6:0/1696162006 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:38.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc7fe00700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc80075590 0x7fdc8019b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc85400700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc8010f4a0 0x7fdc8019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc7fe00700 1 -- 172.21.15.6:0/1696162006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc8010f4a0 msgr2=0x7fdc8019fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc7fe00700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc8010f4a0 0x7fdc8019fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc7fe00700 1 -- 172.21.15.6:0/1696162006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc80074c70 msgr2=0x7fdc8019b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc7fe00700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc80074c70 0x7fdc8019b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.862+0000 7fdc7fe00700 1 -- 172.21.15.6:0/1696162006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc801a02e0 con 0x7fdc80075590 2024-04-26T01:57:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.863+0000 7fdc7fe00700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc80075590 0x7fdc8019b810 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0098e0 tx=0x7fdc6c0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.863+0000 7fdc7d600700 1 -- 172.21.15.6:0/1696162006 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c027070 con 0x7fdc80075590 2024-04-26T01:57:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.863+0000 7fdc7d600700 1 -- 172.21.15.6:0/1696162006 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc6c022470 con 0x7fdc80075590 2024-04-26T01:57:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.863+0000 7fdc7d600700 1 -- 172.21.15.6:0/1696162006 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c017650 con 0x7fdc80075590 2024-04-26T01:57:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.863+0000 7fdc87400700 1 -- 172.21.15.6:0/1696162006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc801a0570 con 0x7fdc80075590 2024-04-26T01:57:38.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.863+0000 7fdc87400700 1 -- 172.21.15.6:0/1696162006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc801a7e50 con 0x7fdc80075590 2024-04-26T01:57:38.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.867+0000 7fdc7d600700 1 -- 172.21.15.6:0/1696162006 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fdc6c02a070 con 0x7fdc80075590 2024-04-26T01:57:38.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.868+0000 7fdc7d600700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdc680a3ed0 0x7fdc680a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.868+0000 7fdc91e7cb80 1 Processor -- start 2024-04-26T01:57:38.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.868+0000 7fdc7d600700 1 -- 172.21.15.6:0/1696162006 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7fdc6c0d14e0 con 0x7fdc80075590 2024-04-26T01:57:38.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.869+0000 7fdc91e7cb80 1 -- start start 2024-04-26T01:57:38.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.869+0000 7fdc84a00700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdc680a3ed0 0x7fdc680a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.869+0000 7fdc91e7cb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3c11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.869+0000 7fdc91e7cb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 0x560a2f3cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.869+0000 7fdc91e7cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f3ce950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.869+0000 7fdc91e7cb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560a2f314020 con 0x560a2f3c0dd0 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.869+0000 7fdc91e7cb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560a2f2ba8c0 con 0x560a2f3c16f0 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.869+0000 7fdc91e7cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560a2f30ffe0 con 0x560a2f3cc560 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.869+0000 7fdc7fe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 0x560a2f3cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.869+0000 7fdc7fe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 0x560a2f3cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59496/0 (socket says 172.21.15.6:59496) 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.869+0000 7fdc7fe00700 1 -- 172.21.15.6:0/1103945007 learned_addr learned my addr 172.21.15.6:0/1103945007 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.869+0000 7fdc85400700 1 --2- 172.21.15.6:0/1103945007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f3ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.870+0000 7fdc84a00700 1 --2- 172.21.15.6:0/1103945007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3c11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.870+0000 7fdc7fe00700 1 -- 172.21.15.6:0/1103945007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 msgr2=0x560a2f3ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.870+0000 7fdc7fe00700 1 --2- 172.21.15.6:0/1103945007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f3ce950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.870+0000 7fdc7fe00700 1 -- 172.21.15.6:0/1103945007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 msgr2=0x560a2f3c11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.870+0000 7fdc7fe00700 1 --2- 172.21.15.6:0/1103945007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3c11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.870+0000 7fdc7fe00700 1 -- 172.21.15.6:0/1103945007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a2f2b0810 con 0x560a2f3c16f0 2024-04-26T01:57:38.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.870+0000 7fdc7fe00700 1 --2- 172.21.15.6:0/1103945007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 0x560a2f3cc020 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0a5b10 tx=0x7fdc6c022cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c08590b9ffae59bc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:38.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.870+0000 7fdc84a00700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdc680a3ed0 0x7fdc680a6390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009d70 tx=0x7fdc74011720 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:38.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.870+0000 7fdc72000700 1 -- 172.21.15.6:0/1103945007 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c09d460 con 0x560a2f3c16f0 2024-04-26T01:57:38.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.870+0000 7fdc72000700 1 -- 172.21.15.6:0/1103945007 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc6c0a1810 con 0x560a2f3c16f0 2024-04-26T01:57:38.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.870+0000 7fdc72000700 1 -- 172.21.15.6:0/1103945007 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c09d6c0 con 0x560a2f3c16f0 2024-04-26T01:57:38.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.871+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/1103945007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 msgr2=0x560a2f3cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:38.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.871+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/1103945007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 0x560a2f3cc020 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0a5b10 tx=0x7fdc6c022cc0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.871+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/1103945007 shutdown_connections 2024-04-26T01:57:38.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.871+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/1103945007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f3ce950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.871+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/1103945007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3c11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.871+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/1103945007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 0x560a2f3cc020 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.871+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/1103945007 >> 172.21.15.6:0/1103945007 conn(0x560a2f2f00e0 msgr2=0x560a2f3bd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:38.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.871+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/1103945007 shutdown_connections 2024-04-26T01:57:38.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.871+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/1103945007 wait complete. 2024-04-26T01:57:38.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.872+0000 7fdc91e7cb80 1 -- start start 2024-04-26T01:57:38.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.872+0000 7fdc91e7cb80 1 Processor -- start 2024-04-26T01:57:38.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.873+0000 7fdc91e7cb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c0dd0 0x560a2f3cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.873+0000 7fdc91e7cb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c16f0 0x560a2f3c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.873+0000 7fdc91e7cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f3c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.873+0000 7fdc91e7cb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560a2f2ba8c0 con 0x560a2f3c16f0 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.873+0000 7fdc91e7cb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560a2f2cd9f0 con 0x560a2f3c0dd0 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.873+0000 7fdc91e7cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560a2f3cee90 con 0x560a2f3cc560 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.873+0000 7fdc91e7cb80 10 client.0 _ll_register_callbacks cb 0x560a2f2ac8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.873+0000 7fdc85400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f3c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.873+0000 7fdc7fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c16f0 0x560a2f3c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.873+0000 7fdc85400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f3c94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38356/0 (socket says 172.21.15.6:38356) 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.873+0000 7fdc84a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c0dd0 0x560a2f3cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.873+0000 7fdc85400700 1 -- 172.21.15.6:0/2027713953 learned_addr learned my addr 172.21.15.6:0/2027713953 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.874+0000 7fdc85400700 1 -- 172.21.15.6:0/2027713953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c0dd0 msgr2=0x560a2f3cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.874+0000 7fdc85400700 1 --2- 172.21.15.6:0/2027713953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c0dd0 0x560a2f3cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.874+0000 7fdc85400700 1 -- 172.21.15.6:0/2027713953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c16f0 msgr2=0x560a2f3c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.874+0000 7fdc85400700 1 --2- 172.21.15.6:0/2027713953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c16f0 0x560a2f3c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.874+0000 7fdc85400700 1 -- 172.21.15.6:0/2027713953 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a2f261a60 con 0x560a2f3cc560 2024-04-26T01:57:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.874+0000 7fdc7fe00700 1 --2- 172.21.15.6:0/2027713953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c16f0 0x560a2f3c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.874+0000 7fdc85400700 1 --2- 172.21.15.6:0/2027713953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f3c94d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdc78011ef0 tx=0x7fdc78014600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3fb986a65bba628 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.874+0000 7fdc66a00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:57:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.874+0000 7fdc66a00700 1 -- 172.21.15.6:0/2027713953 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc78014ce0 con 0x560a2f3cc560 2024-04-26T01:57:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.875+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2027713953 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc6c0a1810 con 0x560a2f3cc560 2024-04-26T01:57:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.875+0000 7fdc66a00700 1 -- 172.21.15.6:0/2027713953 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7fdc78003a00 con 0x560a2f3cc560 2024-04-26T01:57:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.875+0000 7fdc66a00700 1 -- 172.21.15.6:0/2027713953 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fdc780093a0 con 0x560a2f3cc560 2024-04-26T01:57:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.875+0000 7fdc91e7cb80 10 client.26599 fetch_fsmap learned FSMap version 43 2024-04-26T01:57:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.875+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2027713953 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x560a2f2aac80 con 0x560a2f3cc560 2024-04-26T01:57:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.875+0000 7fdc66a00700 1 -- 172.21.15.6:0/2027713953 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc6c0a1810 con 0x560a2f3cc560 2024-04-26T01:57:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.875+0000 7fdc66a00700 1 -- 172.21.15.6:0/2027713953 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7fdc78004770 con 0x560a2f3cc560 2024-04-26T01:57:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.875+0000 7fdc91e7cb80 10 client.26599 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:57:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.875+0000 7fdc91e7cb80 10 client.26599 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:57:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.876+0000 7fdc91e7cb80 20 client.26599 populate_metadata read hostname 'smithi006' 2024-04-26T01:57:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.876+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/2027713953 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560a2f3dba30 0x560a2f3dde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.876+0000 7fdc91e7cb80 4 client.26599 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.876+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2027713953 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a2f321810 con 0x560a2f3dba30 2024-04-26T01:57:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.876+0000 7fdc84a00700 1 --2- 172.21.15.6:0/2027713953 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560a2f3dba30 0x560a2f3dde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.876+0000 7fdc84a00700 1 --2- 172.21.15.6:0/2027713953 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560a2f3dba30 0x560a2f3dde20 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcf749e82a5025bb server_cookie=1a585be1548f505c in_seq=0 out_seq=0 2024-04-26T01:57:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.876+0000 7fdc66a00700 10 client.26599 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.877+0000 7fdc66a00700 1 -- 172.21.15.6:0/2027713953 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a2f321810 con 0x560a2f3dba30 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.877+0000 7fdc66a00700 10 client.26599 handle_command_reply: tid=0 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.877+0000 7fdc91e7cb80 1 client.26599 shutdown 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.877+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2027713953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 msgr2=0x560a2f3c94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.877+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/2027713953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f3c94d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdc78011ef0 tx=0x7fdc78014600 comp rx=0 tx=0).stop 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.878+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2027713953 shutdown_connections 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.878+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/2027713953 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560a2f3dba30 0x560a2f3dde20 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.878+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/2027713953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f3c94d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.878+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/2027713953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c0dd0 0x560a2f3cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.878+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/2027713953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c16f0 0x560a2f3c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.878+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2027713953 >> 172.21.15.6:0/2027713953 conn(0x560a2f2f00e0 msgr2=0x560a2f3bced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.878+0000 7fdc66a00700 0 client.26599 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.878+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2027713953 shutdown_connections 2024-04-26T01:57:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.878+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2027713953 wait complete. 2024-04-26T01:57:38.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.878+0000 7fdc91e7cb80 20 client.26599 trim_cache size 0 max 16384 2024-04-26T01:57:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:38 smithi188 ceph-mon[32751]: pgmap v370: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 239 op/s 2024-04-26T01:57:38.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.900+0000 7fdc91e7cb80 1 Processor -- start 2024-04-26T01:57:38.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.900+0000 7fdc91e7cb80 1 -- start start 2024-04-26T01:57:38.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.900+0000 7fdc91e7cb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.900+0000 7fdc91e7cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3c16f0 0x560a2f3d7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.900+0000 7fdc91e7cb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3cc560 0x560a2f3e36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.900+0000 7fdc91e7cb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdc801b1ea0 con 0x560a2f3c0dd0 2024-04-26T01:57:38.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.900+0000 7fdc91e7cb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdc801af730 con 0x560a2f3cc560 2024-04-26T01:57:38.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.900+0000 7fdc91e7cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdc801ad5a0 con 0x560a2f3c16f0 2024-04-26T01:57:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.900+0000 7fdc84a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.901+0000 7fdc85400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3cc560 0x560a2f3e36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.901+0000 7fdc84a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37888/0 (socket says 172.21.15.6:37888) 2024-04-26T01:57:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.901+0000 7fdc84a00700 1 -- 172.21.15.6:0/568479333 learned_addr learned my addr 172.21.15.6:0/568479333 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.901+0000 7fdc7fe00700 1 --2- 172.21.15.6:0/568479333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3c16f0 0x560a2f3d7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.901+0000 7fdc85400700 1 -- 172.21.15.6:0/568479333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3c16f0 msgr2=0x560a2f3d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.901+0000 7fdc85400700 1 --2- 172.21.15.6:0/568479333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3c16f0 0x560a2f3d7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.901+0000 7fdc85400700 1 -- 172.21.15.6:0/568479333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 msgr2=0x560a2f3c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.901+0000 7fdc85400700 1 --2- 172.21.15.6:0/568479333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.901+0000 7fdc85400700 1 -- 172.21.15.6:0/568479333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a2f3d9500 con 0x560a2f3cc560 2024-04-26T01:57:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.901+0000 7fdc84a00700 1 --2- 172.21.15.6:0/568479333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.901+0000 7fdc85400700 1 --2- 172.21.15.6:0/568479333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3cc560 0x560a2f3e36d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdc780075d0 tx=0x7fdc780112f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=511e10b1d53d4342 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.902+0000 7fdc71600700 1 -- 172.21.15.6:0/568479333 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc78007840 con 0x560a2f3cc560 2024-04-26T01:57:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.902+0000 7fdc71600700 1 -- 172.21.15.6:0/568479333 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc780079e0 con 0x560a2f3cc560 2024-04-26T01:57:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.902+0000 7fdc71600700 1 -- 172.21.15.6:0/568479333 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc780259c0 con 0x560a2f3cc560 2024-04-26T01:57:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.903+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/568479333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3cc560 msgr2=0x560a2f3e36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.903+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/568479333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3cc560 0x560a2f3e36d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdc780075d0 tx=0x7fdc780112f0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.903+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/568479333 shutdown_connections 2024-04-26T01:57:38.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.903+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/568479333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3cc560 0x560a2f3e36d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.903+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/568479333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.903+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/568479333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3c16f0 0x560a2f3d7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.903+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/568479333 >> 172.21.15.6:0/568479333 conn(0x560a2f2f00e0 msgr2=0x560a2f3c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:38.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.903+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/568479333 shutdown_connections 2024-04-26T01:57:38.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.903+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/568479333 wait complete. 2024-04-26T01:57:38.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.903+0000 7fdc91e7cb80 1 -- start start 2024-04-26T01:57:38.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.904+0000 7fdc91e7cb80 1 Processor -- start 2024-04-26T01:57:38.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.904+0000 7fdc91e7cb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3d1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.904+0000 7fdc91e7cb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 0x560a2f3d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.904+0000 7fdc91e7cb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f441f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.904+0000 7fdc91e7cb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdc801adbf0 con 0x560a2f3c0dd0 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.904+0000 7fdc91e7cb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdc801af060 con 0x560a2f3c16f0 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.904+0000 7fdc91e7cb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdc801ae2c0 con 0x560a2f3cc560 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.904+0000 7fdc91e7cb80 10 client.0 _ll_register_callbacks cb 0x560a2f2ac8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.905+0000 7fdc7fe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 0x560a2f3d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.905+0000 7fdc85400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f441f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.905+0000 7fdc7fe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 0x560a2f3d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59520/0 (socket says 172.21.15.6:59520) 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.905+0000 7fdc7fe00700 1 -- 172.21.15.6:0/2653334957 learned_addr learned my addr 172.21.15.6:0/2653334957 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.905+0000 7fdc84a00700 1 --2- 172.21.15.6:0/2653334957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3d1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.905+0000 7fdc85400700 1 -- 172.21.15.6:0/2653334957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 msgr2=0x560a2f3d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.905+0000 7fdc85400700 1 --2- 172.21.15.6:0/2653334957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 0x560a2f3d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.905+0000 7fdc85400700 1 -- 172.21.15.6:0/2653334957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 msgr2=0x560a2f3d1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:38.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.905+0000 7fdc85400700 1 --2- 172.21.15.6:0/2653334957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3d1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.905+0000 7fdc85400700 1 -- 172.21.15.6:0/2653334957 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a2f442490 con 0x560a2f3cc560 2024-04-26T01:57:38.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.905+0000 7fdc85400700 1 --2- 172.21.15.6:0/2653334957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f441f50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdc78007650 tx=0x7fdc78009e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fb140515d28bb68 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:38.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.905+0000 7fdc66a00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:57:38.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.906+0000 7fdc66a00700 1 -- 172.21.15.6:0/2653334957 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc78033060 con 0x560a2f3cc560 2024-04-26T01:57:38.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.906+0000 7fdc66a00700 1 -- 172.21.15.6:0/2653334957 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7fdc78025630 con 0x560a2f3cc560 2024-04-26T01:57:38.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.906+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2653334957 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc780079e0 con 0x560a2f3cc560 2024-04-26T01:57:38.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.906+0000 7fdc66a00700 1 -- 172.21.15.6:0/2653334957 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fdc78025830 con 0x560a2f3cc560 2024-04-26T01:57:38.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.906+0000 7fdc91e7cb80 10 client.26605 fetch_fsmap learned FSMap version 43 2024-04-26T01:57:38.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.906+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2653334957 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x560a2f443660 con 0x560a2f3cc560 2024-04-26T01:57:38.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.906+0000 7fdc66a00700 1 -- 172.21.15.6:0/2653334957 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc780079e0 con 0x560a2f3cc560 2024-04-26T01:57:38.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.906+0000 7fdc66a00700 1 -- 172.21.15.6:0/2653334957 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7fdc7802eaf0 con 0x560a2f3cc560 2024-04-26T01:57:38.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.906+0000 7fdc91e7cb80 10 client.26605 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:57:38.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.906+0000 7fdc91e7cb80 10 client.26605 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:57:38.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.906+0000 7fdc91e7cb80 20 client.26605 populate_metadata read hostname 'smithi006' 2024-04-26T01:57:38.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.906+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/2653334957 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560a2f443c10 0x560a2f446030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:38.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.907+0000 7fdc91e7cb80 4 client.26605 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.25, "await_for": 42.0}] 2024-04-26T01:57:38.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.907+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2653334957 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.25, "await_for": 42.0}) -- 0x560a2f31b8b0 con 0x560a2f443c10 2024-04-26T01:57:38.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.907+0000 7fdc84a00700 1 --2- 172.21.15.6:0/2653334957 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560a2f443c10 0x560a2f446030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:38.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.907+0000 7fdc84a00700 1 --2- 172.21.15.6:0/2653334957 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560a2f443c10 0x560a2f446030 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a256d4bf5c03ae9 server_cookie=3517cd6106fabacc in_seq=0 out_seq=0 2024-04-26T01:57:38.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.907+0000 7fdc66a00700 10 client.26605 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:38.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:38 smithi029 ceph-mon[32775]: pgmap v370: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 239 op/s 2024-04-26T01:57:38.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.971+0000 7fdc66a00700 1 -- 172.21.15.6:0/2653334957 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x560a2f31b8b0 con 0x560a2f443c10 2024-04-26T01:57:38.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.971+0000 7fdc66a00700 10 client.26605 handle_command_reply: tid=0 2024-04-26T01:57:38.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.971+0000 7fdc91e7cb80 1 client.26605 shutdown 2024-04-26T01:57:38.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.972+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2653334957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 msgr2=0x560a2f441f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.972+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/2653334957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f441f50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdc78007650 tx=0x7fdc78009e50 comp rx=0 tx=0).stop 2024-04-26T01:57:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.973+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2653334957 shutdown_connections 2024-04-26T01:57:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.973+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/2653334957 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560a2f443c10 0x560a2f446030 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.973+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/2653334957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560a2f3cc560 0x560a2f441f50 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.973+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/2653334957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560a2f3c0dd0 0x560a2f3d1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.973+0000 7fdc91e7cb80 1 --2- 172.21.15.6:0/2653334957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560a2f3c16f0 0x560a2f3d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.973+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2653334957 >> 172.21.15.6:0/2653334957 conn(0x560a2f2f00e0 msgr2=0x560a2f3ca7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.973+0000 7fdc66a00700 0 client.26605 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:38.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.974+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2653334957 shutdown_connections 2024-04-26T01:57:38.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.975+0000 7fdc91e7cb80 1 -- 172.21.15.6:0/2653334957 wait complete. 2024-04-26T01:57:38.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.975+0000 7fdc91e7cb80 20 client.26605 trim_cache size 0 max 16384 2024-04-26T01:57:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:57:38.975 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 43, 2024-04-26T01:57:38.975 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:57:38.975 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 34, 2024-04-26T01:57:38.975 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:57:38.975 INFO:teuthology.orchestra.run.smithi006.stdout: "8cb585fa": { 2024-04-26T01:57:38.975 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 34, 2024-04-26T01:57:38.975 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:57:38.975 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:57:38.975 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:57:38.975 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 79.2, 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:57:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:57:38.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.978+0000 7fdc87400700 1 -- 172.21.15.6:0/1696162006 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdc680a3ed0 msgr2=0x7fdc680a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:38.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.978+0000 7fdc87400700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdc680a3ed0 0x7fdc680a6390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009d70 tx=0x7fdc74011720 comp rx=0 tx=0).stop 2024-04-26T01:57:38.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.978+0000 7fdc87400700 1 -- 172.21.15.6:0/1696162006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc80075590 msgr2=0x7fdc8019b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:38.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.978+0000 7fdc87400700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc80075590 0x7fdc8019b810 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0098e0 tx=0x7fdc6c0057c0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.978+0000 7fdc87400700 1 -- 172.21.15.6:0/1696162006 shutdown_connections 2024-04-26T01:57:38.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.978+0000 7fdc87400700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdc680a3ed0 0x7fdc680a6390 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.978+0000 7fdc87400700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc8010f4a0 0x7fdc8019fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.978+0000 7fdc87400700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc80074c70 0x7fdc8019b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.978+0000 7fdc87400700 1 --2- 172.21.15.6:0/1696162006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc80075590 0x7fdc8019b810 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:38.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.978+0000 7fdc87400700 1 -- 172.21.15.6:0/1696162006 >> 172.21.15.6:0/1696162006 conn(0x7fdc80079b10 msgr2=0x7fdc80100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:38.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.979+0000 7fdc87400700 1 -- 172.21.15.6:0/1696162006 shutdown_connections 2024-04-26T01:57:38.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:57:38.979+0000 7fdc87400700 1 -- 172.21.15.6:0/1696162006 wait complete. 2024-04-26T01:57:39.631 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.73 MB/sec execute 483 sec latency 648.219 ms 2024-04-26T01:57:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.839+0000 7f4330a00640 1 -- 172.21.15.6:0/3123205566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c107080 msgr2=0x7f432c10ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.839+0000 7f4330a00640 1 --2- 172.21.15.6:0/3123205566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c107080 0x7f432c10ec50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4324009ca0 tx=0x7f432402a8c0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.839+0000 7f4330a00640 1 -- 172.21.15.6:0/3123205566 shutdown_connections 2024-04-26T01:57:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.840+0000 7f4330a00640 1 --2- 172.21.15.6:0/3123205566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c107080 0x7f432c10ec50 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.840+0000 7f4330a00640 1 --2- 172.21.15.6:0/3123205566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f432c102960 0x7f432c1069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.840+0000 7f4330a00640 1 --2- 172.21.15.6:0/3123205566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f432c101fb0 0x7f432c102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.840+0000 7f4330a00640 1 -- 172.21.15.6:0/3123205566 >> 172.21.15.6:0/3123205566 conn(0x7f432c0fd890 msgr2=0x7f432c0ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.840+0000 7f4330a00640 1 -- 172.21.15.6:0/3123205566 shutdown_connections 2024-04-26T01:57:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.840+0000 7f4330a00640 1 -- 172.21.15.6:0/3123205566 wait complete. 2024-04-26T01:57:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.841+0000 7f4330a00640 1 Processor -- start 2024-04-26T01:57:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.841+0000 7f4330a00640 1 -- start start 2024-04-26T01:57:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.841+0000 7f4330a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c101fb0 0x7f432c1a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.841+0000 7f4330a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f432c107080 0x7f432c1a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.841+0000 7f4330a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f432c19a610 0x7f432c19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.841+0000 7f4330a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f432c111fd0 con 0x7f432c101fb0 2024-04-26T01:57:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.841+0000 7f4330a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f432c111e50 con 0x7f432c107080 2024-04-26T01:57:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.841+0000 7f4330a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f432c112150 con 0x7f432c19a610 2024-04-26T01:57:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.841+0000 7f432a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c101fb0 0x7f432c1a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.842+0000 7f432a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c101fb0 0x7f432c1a0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37922/0 (socket says 172.21.15.6:37922) 2024-04-26T01:57:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.842+0000 7f432a800640 1 -- 172.21.15.6:0/12868695 learned_addr learned my addr 172.21.15.6:0/12868695 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.842+0000 7f4329e00640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f432c107080 0x7f432c1a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.842+0000 7f432b200640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f432c19a610 0x7f432c19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.842+0000 7f432a800640 1 -- 172.21.15.6:0/12868695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f432c19a610 msgr2=0x7f432c19aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.842+0000 7f432a800640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f432c19a610 0x7f432c19aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.842+0000 7f432a800640 1 -- 172.21.15.6:0/12868695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f432c107080 msgr2=0x7f432c1a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.842+0000 7f432a800640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f432c107080 0x7f432c1a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.842+0000 7f432a800640 1 -- 172.21.15.6:0/12868695 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432c19b130 con 0x7f432c101fb0 2024-04-26T01:57:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.842+0000 7f432a800640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c101fb0 0x7f432c1a0410 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f431c00d950 tx=0x7f431c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.843+0000 7f4323400640 1 -- 172.21.15.6:0/12868695 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431c017030 con 0x7f432c101fb0 2024-04-26T01:57:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.843+0000 7f4330a00640 1 -- 172.21.15.6:0/12868695 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f432c19b420 con 0x7f432c101fb0 2024-04-26T01:57:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.843+0000 7f4323400640 1 -- 172.21.15.6:0/12868695 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f431c004020 con 0x7f432c101fb0 2024-04-26T01:57:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.843+0000 7f4323400640 1 -- 172.21.15.6:0/12868695 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431c0047a0 con 0x7f432c101fb0 2024-04-26T01:57:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.843+0000 7f4330a00640 1 -- 172.21.15.6:0/12868695 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f432c1a7670 con 0x7f432c101fb0 2024-04-26T01:57:39.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.846+0000 7f4323400640 1 -- 172.21.15.6:0/12868695 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f431c009a30 con 0x7f432c101fb0 2024-04-26T01:57:39.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.846+0000 7f4334203740 1 Processor -- start 2024-04-26T01:57:39.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.846+0000 7f4334203740 1 -- start start 2024-04-26T01:57:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f4334203740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c7399b0 0x55e92c739d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f4334203740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c73a2d0 0x55e92c744650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f4334203740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c744b90 0x55e92c746f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f4334203740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55e92c69f900 con 0x55e92c73a2d0 2024-04-26T01:57:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f4334203740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e92c645ef0 con 0x55e92c7399b0 2024-04-26T01:57:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f4334203740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e92c69b8c0 con 0x55e92c744b90 2024-04-26T01:57:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f432a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c73a2d0 0x55e92c744650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f432b200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c7399b0 0x55e92c739d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f4329e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c744b90 0x55e92c746f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f432b200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c7399b0 0x55e92c739d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59556/0 (socket says 172.21.15.6:59556) 2024-04-26T01:57:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f432b200640 1 -- 172.21.15.6:0/4136695339 learned_addr learned my addr 172.21.15.6:0/4136695339 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f4329e00640 1 --2- 172.21.15.6:0/4136695339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c744b90 0x55e92c746f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38412/0 (socket says 172.21.15.6:38412) 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f432b200640 1 -- 172.21.15.6:0/4136695339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c744b90 msgr2=0x55e92c746f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f432b200640 1 --2- 172.21.15.6:0/4136695339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c744b90 0x55e92c746f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f432b200640 1 -- 172.21.15.6:0/4136695339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c73a2d0 msgr2=0x55e92c744650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f432b200640 1 --2- 172.21.15.6:0/4136695339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c73a2d0 0x55e92c744650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.847+0000 7f432b200640 1 -- 172.21.15.6:0/4136695339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e92c3b2210 con 0x55e92c7399b0 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.848+0000 7f4323400640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43100a3d00 0x7f43100a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.848+0000 7f432b200640 1 --2- 172.21.15.6:0/4136695339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c7399b0 0x55e92c739d90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4324005fa0 tx=0x7f4324009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=116125553c2fdeb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.848+0000 7f432a800640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43100a3d00 0x7f43100a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.848+0000 7f4323400640 1 -- 172.21.15.6:0/12868695 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7f431c0d0280 con 0x7f432c101fb0 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.848+0000 7f4317400640 1 -- 172.21.15.6:0/4136695339 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4324031cf0 con 0x55e92c7399b0 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.848+0000 7f4317400640 1 -- 172.21.15.6:0/4136695339 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4324036030 con 0x55e92c7399b0 2024-04-26T01:57:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.848+0000 7f4317400640 1 -- 172.21.15.6:0/4136695339 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43240042a0 con 0x55e92c7399b0 2024-04-26T01:57:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.849+0000 7f4334203740 1 -- 172.21.15.6:0/4136695339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c7399b0 msgr2=0x55e92c739d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.849+0000 7f432a800640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43100a3d00 0x7f43100a61c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55e92c73af50 tx=0x7f431c0d3d30 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.849+0000 7f4334203740 1 --2- 172.21.15.6:0/4136695339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c7399b0 0x55e92c739d90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4324005fa0 tx=0x7f4324009800 comp rx=0 tx=0).stop 2024-04-26T01:57:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.849+0000 7f4334203740 1 -- 172.21.15.6:0/4136695339 shutdown_connections 2024-04-26T01:57:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.849+0000 7f4334203740 1 --2- 172.21.15.6:0/4136695339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c73a2d0 0x55e92c744650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.849+0000 7f4334203740 1 --2- 172.21.15.6:0/4136695339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c744b90 0x55e92c746f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.849+0000 7f4334203740 1 --2- 172.21.15.6:0/4136695339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c7399b0 0x55e92c739d90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.849+0000 7f4334203740 1 -- 172.21.15.6:0/4136695339 >> 172.21.15.6:0/4136695339 conn(0x55e92c735ec0 msgr2=0x55e92c7362a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.849+0000 7f4334203740 1 -- 172.21.15.6:0/4136695339 shutdown_connections 2024-04-26T01:57:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.849+0000 7f4334203740 1 -- 172.21.15.6:0/4136695339 wait complete. 2024-04-26T01:57:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.850+0000 7f4334203740 1 -- start start 2024-04-26T01:57:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.850+0000 7f4334203740 1 Processor -- start 2024-04-26T01:57:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.850+0000 7f4334203740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c7399b0 0x55e92c741b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.850+0000 7f4334203740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c742070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.850+0000 7f4334203740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c744b90 0x55e92c7425b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.850+0000 7f4334203740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55e92c7475e0 con 0x55e92c7399b0 2024-04-26T01:57:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.850+0000 7f4334203740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e92c659310 con 0x55e92c73a2d0 2024-04-26T01:57:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.850+0000 7f4334203740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e92c747760 con 0x55e92c744b90 2024-04-26T01:57:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.850+0000 7f4334203740 10 client.0 _ll_register_callbacks cb 0x55e92c627cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f432b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c7399b0 0x55e92c741b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f4329e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c742070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f4329e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c742070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59562/0 (socket says 172.21.15.6:59562) 2024-04-26T01:57:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f4329e00640 1 -- 172.21.15.6:0/865316988 learned_addr learned my addr 172.21.15.6:0/865316988 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f432b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c7399b0 0x55e92c741b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37930/0 (socket says 172.21.15.6:37930) 2024-04-26T01:57:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f432b200640 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c744b90 0x55e92c7425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f4329e00640 1 -- 172.21.15.6:0/865316988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c744b90 msgr2=0x55e92c7425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f4329e00640 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c744b90 0x55e92c7425b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f4329e00640 1 -- 172.21.15.6:0/865316988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c7399b0 msgr2=0x55e92c741b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f4329e00640 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c7399b0 0x55e92c741b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f4329e00640 1 -- 172.21.15.6:0/865316988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e92c47e620 con 0x55e92c73a2d0 2024-04-26T01:57:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f432b200640 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c744b90 0x55e92c7425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:57:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f4329e00640 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c742070 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4318005bc0 tx=0x7f4318012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2786ca25e6688cd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.851+0000 7f430be00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:57:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.852+0000 7f430be00640 1 -- 172.21.15.6:0/865316988 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431800f920 con 0x55e92c73a2d0 2024-04-26T01:57:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.852+0000 7f4334203740 1 -- 172.21.15.6:0/865316988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4324036030 con 0x55e92c73a2d0 2024-04-26T01:57:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.852+0000 7f430be00640 1 -- 172.21.15.6:0/865316988 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7f431800fac0 con 0x55e92c73a2d0 2024-04-26T01:57:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.852+0000 7f430be00640 1 -- 172.21.15.6:0/865316988 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f431800fcc0 con 0x55e92c73a2d0 2024-04-26T01:57:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.852+0000 7f4334203740 10 client.26579 fetch_fsmap learned FSMap version 43 2024-04-26T01:57:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.852+0000 7f4334203740 1 -- 172.21.15.6:0/865316988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55e92c753910 con 0x55e92c73a2d0 2024-04-26T01:57:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.852+0000 7f430be00640 1 -- 172.21.15.6:0/865316988 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4324036030 con 0x55e92c73a2d0 2024-04-26T01:57:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.853+0000 7f430be00640 1 -- 172.21.15.6:0/865316988 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f431801c420 con 0x55e92c73a2d0 2024-04-26T01:57:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.853+0000 7f4334203740 10 client.26579 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:57:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.853+0000 7f4334203740 10 client.26579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:57:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.853+0000 7f4334203740 20 client.26579 populate_metadata read hostname 'smithi006' 2024-04-26T01:57:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.853+0000 7f4334203740 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e92c754dd0 0x55e92c757220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.853+0000 7f4334203740 4 client.26579 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.853+0000 7f4334203740 1 -- 172.21.15.6:0/865316988 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e92c6b38b0 con 0x55e92c754dd0 2024-04-26T01:57:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.853+0000 7f432b200640 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e92c754dd0 0x55e92c757220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.853+0000 7f432b200640 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e92c754dd0 0x55e92c757220 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e6d6f900bb9f49b server_cookie=6df29e176f097251 in_seq=0 out_seq=0 2024-04-26T01:57:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.853+0000 7f430be00640 10 client.26579 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.854+0000 7f430be00640 1 -- 172.21.15.6:0/865316988 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e92c6b38b0 con 0x55e92c754dd0 2024-04-26T01:57:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.854+0000 7f430be00640 10 client.26579 handle_command_reply: tid=0 2024-04-26T01:57:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.854+0000 7f4334203740 1 client.26579 shutdown 2024-04-26T01:57:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.855+0000 7f4334203740 1 -- 172.21.15.6:0/865316988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 msgr2=0x55e92c742070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.855+0000 7f4334203740 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c742070 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4318005bc0 tx=0x7f4318012b60 comp rx=0 tx=0).stop 2024-04-26T01:57:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.855+0000 7f4334203740 1 -- 172.21.15.6:0/865316988 shutdown_connections 2024-04-26T01:57:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.855+0000 7f4334203740 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e92c754dd0 0x55e92c757220 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.855+0000 7f4334203740 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c744b90 0x55e92c7425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.855+0000 7f4334203740 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c742070 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.855+0000 7f4334203740 1 --2- 172.21.15.6:0/865316988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c7399b0 0x55e92c741b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.855+0000 7f430be00640 0 client.26579 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.855+0000 7f4334203740 1 -- 172.21.15.6:0/865316988 >> 172.21.15.6:0/865316988 conn(0x55e92c735ec0 msgr2=0x55e92c73bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.855+0000 7f4334203740 1 -- 172.21.15.6:0/865316988 shutdown_connections 2024-04-26T01:57:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.855+0000 7f4334203740 1 -- 172.21.15.6:0/865316988 wait complete. 2024-04-26T01:57:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.855+0000 7f4334203740 20 client.26579 trim_cache size 0 max 16384 2024-04-26T01:57:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.871+0000 7f4334203740 1 Processor -- start 2024-04-26T01:57:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.871+0000 7f4334203740 1 -- start start 2024-04-26T01:57:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f4334203740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 0x55e92c7504a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f4334203740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c75caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f4334203740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c744b90 0x55e92c75d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f4334203740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f432c1b2630 con 0x55e92c744b90 2024-04-26T01:57:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f4334203740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f432c1aff00 con 0x55e92c73a2d0 2024-04-26T01:57:39.870 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f4334203740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f432c1add70 con 0x55e92c7399b0 2024-04-26T01:57:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f432b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c744b90 0x55e92c75d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f4329e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c75caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f432b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 0x55e92c7504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f432b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 0x55e92c7504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38434/0 (socket says 172.21.15.6:38434) 2024-04-26T01:57:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f432b200640 1 -- 172.21.15.6:0/1638466414 learned_addr learned my addr 172.21.15.6:0/1638466414 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f432b200640 1 -- 172.21.15.6:0/1638466414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 msgr2=0x55e92c75caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f432b200640 1 --2- 172.21.15.6:0/1638466414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c75caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f432b200640 1 -- 172.21.15.6:0/1638466414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c744b90 msgr2=0x55e92c75d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f432b200640 1 --2- 172.21.15.6:0/1638466414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c744b90 0x55e92c75d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.872+0000 7f432b200640 1 -- 172.21.15.6:0/1638466414 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e92c7578a0 con 0x55e92c7399b0 2024-04-26T01:57:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.873+0000 7f432b200640 1 --2- 172.21.15.6:0/1638466414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 0x55e92c7504a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55e92c75d570 tx=0x7f4324003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=506cc382e77de5f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.873+0000 7f4316a00640 1 -- 172.21.15.6:0/1638466414 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4324031af0 con 0x55e92c7399b0 2024-04-26T01:57:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.873+0000 7f4316a00640 1 -- 172.21.15.6:0/1638466414 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4324036040 con 0x55e92c7399b0 2024-04-26T01:57:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.873+0000 7f4316a00640 1 -- 172.21.15.6:0/1638466414 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43240576f0 con 0x55e92c7399b0 2024-04-26T01:57:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.874+0000 7f4334203740 1 -- 172.21.15.6:0/1638466414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 msgr2=0x55e92c7504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.874+0000 7f4334203740 1 --2- 172.21.15.6:0/1638466414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 0x55e92c7504a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55e92c75d570 tx=0x7f4324003cd0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.874+0000 7f4334203740 1 -- 172.21.15.6:0/1638466414 shutdown_connections 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.874+0000 7f4334203740 1 --2- 172.21.15.6:0/1638466414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c744b90 0x55e92c75d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.874+0000 7f4334203740 1 --2- 172.21.15.6:0/1638466414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c75caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.874+0000 7f4334203740 1 --2- 172.21.15.6:0/1638466414 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 0x55e92c7504a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.874+0000 7f4334203740 1 -- 172.21.15.6:0/1638466414 >> 172.21.15.6:0/1638466414 conn(0x55e92c735ec0 msgr2=0x55e92c73ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.874+0000 7f4334203740 1 -- 172.21.15.6:0/1638466414 shutdown_connections 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.874+0000 7f4334203740 1 -- 172.21.15.6:0/1638466414 wait complete. 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.874+0000 7f4334203740 1 -- start start 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f4334203740 1 Processor -- start 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f4334203740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 0x55e92c749980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f4334203740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c749ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f4334203740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c744b90 0x55e92c7c1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f4334203740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f432c1ae3c0 con 0x55e92c744b90 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f4334203740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f432c1af830 con 0x55e92c73a2d0 2024-04-26T01:57:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f4334203740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f432c1aea90 con 0x55e92c7399b0 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f4334203740 10 client.0 _ll_register_callbacks cb 0x7f4318009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f432b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 0x55e92c749980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f432b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c744b90 0x55e92c7c1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f432b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 0x55e92c749980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38450/0 (socket says 172.21.15.6:38450) 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f4329e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c749ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.875+0000 7f432b200640 1 -- 172.21.15.6:0/2983702474 learned_addr learned my addr 172.21.15.6:0/2983702474 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.876+0000 7f4329e00640 1 -- 172.21.15.6:0/2983702474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 msgr2=0x55e92c749980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.876+0000 7f4329e00640 1 --2- 172.21.15.6:0/2983702474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 0x55e92c749980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.876+0000 7f4329e00640 1 -- 172.21.15.6:0/2983702474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c744b90 msgr2=0x55e92c7c1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.876+0000 7f4329e00640 1 --2- 172.21.15.6:0/2983702474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c744b90 0x55e92c7c1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.876+0000 7f4329e00640 1 -- 172.21.15.6:0/2983702474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e92c7c20b0 con 0x55e92c73a2d0 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.876+0000 7f432b200640 1 --2- 172.21.15.6:0/2983702474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 0x55e92c749980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.876+0000 7f4329e00640 1 --2- 172.21.15.6:0/2983702474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c749ec0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f431801ca30 tx=0x7f431801c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2591c2d372d53579 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.876+0000 7f430be00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.876+0000 7f430be00640 1 -- 172.21.15.6:0/2983702474 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431801cd80 con 0x55e92c73a2d0 2024-04-26T01:57:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.876+0000 7f4334203740 1 -- 172.21.15.6:0/2983702474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4324036040 con 0x55e92c73a2d0 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.876+0000 7f430be00640 1 -- 172.21.15.6:0/2983702474 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7f43180050e0 con 0x55e92c73a2d0 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.876+0000 7f430be00640 1 -- 172.21.15.6:0/2983702474 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f4318031050 con 0x55e92c73a2d0 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.877+0000 7f4334203740 10 client.26585 fetch_fsmap learned FSMap version 43 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.877+0000 7f4334203740 1 -- 172.21.15.6:0/2983702474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55e92c7c3200 con 0x55e92c73a2d0 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.877+0000 7f430be00640 1 -- 172.21.15.6:0/2983702474 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4324036040 con 0x55e92c73a2d0 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.877+0000 7f430be00640 1 -- 172.21.15.6:0/2983702474 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f4318003a90 con 0x55e92c73a2d0 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.877+0000 7f4334203740 10 client.26585 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.877+0000 7f4334203740 10 client.26585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.877+0000 7f4334203740 20 client.26585 populate_metadata read hostname 'smithi006' 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.877+0000 7f4334203740 1 --2- 172.21.15.6:0/2983702474 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e92c7c38c0 0x55e92c7c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.877+0000 7f4334203740 4 client.26585 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.877+0000 7f4334203740 1 -- 172.21.15.6:0/2983702474 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e92c6ad320 con 0x55e92c7c38c0 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.877+0000 7f432b200640 1 --2- 172.21.15.6:0/2983702474 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e92c7c38c0 0x55e92c7c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.878+0000 7f432b200640 1 --2- 172.21.15.6:0/2983702474 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e92c7c38c0 0x55e92c7c5cb0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c03d765385544451 server_cookie=4c8b37f1dde05711 in_seq=0 out_seq=0 2024-04-26T01:57:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.878+0000 7f430be00640 10 client.26585 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f430be00640 1 -- 172.21.15.6:0/2983702474 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e92c6ad320 con 0x55e92c7c38c0 2024-04-26T01:57:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f430be00640 10 client.26585 handle_command_reply: tid=0 2024-04-26T01:57:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f4334203740 1 client.26585 shutdown 2024-04-26T01:57:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f4334203740 1 -- 172.21.15.6:0/2983702474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 msgr2=0x55e92c749ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f4334203740 1 --2- 172.21.15.6:0/2983702474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c749ec0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f431801ca30 tx=0x7f431801c650 comp rx=0 tx=0).stop 2024-04-26T01:57:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f4334203740 1 -- 172.21.15.6:0/2983702474 shutdown_connections 2024-04-26T01:57:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f4334203740 1 --2- 172.21.15.6:0/2983702474 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e92c7c38c0 0x55e92c7c5cb0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f4334203740 1 --2- 172.21.15.6:0/2983702474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e92c744b90 0x55e92c7c1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f4334203740 1 --2- 172.21.15.6:0/2983702474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e92c73a2d0 0x55e92c749ec0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f4334203740 1 --2- 172.21.15.6:0/2983702474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e92c7399b0 0x55e92c749980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f430be00640 0 client.26585 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f4334203740 1 -- 172.21.15.6:0/2983702474 >> 172.21.15.6:0/2983702474 conn(0x55e92c735ec0 msgr2=0x55e92c73b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f4334203740 1 -- 172.21.15.6:0/2983702474 shutdown_connections 2024-04-26T01:57:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.922+0000 7f4334203740 1 -- 172.21.15.6:0/2983702474 wait complete. 2024-04-26T01:57:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.923+0000 7f4334203740 20 client.26585 trim_cache size 0 max 16384 2024-04-26T01:57:39.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:57:39.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:57:39.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:57:39.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:57:39.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.923+0000 7f4330a00640 1 -- 172.21.15.6:0/12868695 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43100a3d00 msgr2=0x7f43100a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:39.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.923+0000 7f4330a00640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43100a3d00 0x7f43100a61c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55e92c73af50 tx=0x7f431c0d3d30 comp rx=0 tx=0).stop 2024-04-26T01:57:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.923+0000 7f4330a00640 1 -- 172.21.15.6:0/12868695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c101fb0 msgr2=0x7f432c1a0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.923+0000 7f4330a00640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c101fb0 0x7f432c1a0410 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f431c00d950 tx=0x7f431c00de10 comp rx=0 tx=0).stop 2024-04-26T01:57:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.923+0000 7f4330a00640 1 -- 172.21.15.6:0/12868695 shutdown_connections 2024-04-26T01:57:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.923+0000 7f4330a00640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43100a3d00 0x7f43100a61c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.923+0000 7f4330a00640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f432c19a610 0x7f432c19aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.923+0000 7f4330a00640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f432c107080 0x7f432c1a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.923+0000 7f4330a00640 1 --2- 172.21.15.6:0/12868695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c101fb0 0x7f432c1a0410 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.923+0000 7f4330a00640 1 -- 172.21.15.6:0/12868695 >> 172.21.15.6:0/12868695 conn(0x7f432c0fd890 msgr2=0x7f432c0ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.924+0000 7f4330a00640 1 -- 172.21.15.6:0/12868695 shutdown_connections 2024-04-26T01:57:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:39.924+0000 7f4330a00640 1 -- 172.21.15.6:0/12868695 wait complete. 2024-04-26T01:57:40.024 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '8cb585fa', quiesce took 8.3 seconds. Will release after: 19.25 2024-04-26T01:57:40.631 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.72 MB/sec execute 484 sec latency 1648.381 ms 2024-04-26T01:57:40.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:40 smithi006 ceph-mon[28419]: pgmap v371: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-04-26T01:57:40.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:40 smithi188 ceph-mon[32751]: pgmap v371: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-04-26T01:57:40.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:40 smithi029 ceph-mon[32775]: pgmap v371: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-04-26T01:57:41.631 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.72 MB/sec execute 485 sec latency 2648.538 ms 2024-04-26T01:57:42.631 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.71 MB/sec execute 486 sec latency 3648.702 ms 2024-04-26T01:57:42.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:42 smithi006 ceph-mon[28419]: pgmap v372: 97 pgs: 97 active+clean; 132 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 240 op/s 2024-04-26T01:57:42.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:42 smithi188 ceph-mon[32751]: pgmap v372: 97 pgs: 97 active+clean; 132 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 240 op/s 2024-04-26T01:57:42.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:42 smithi029 ceph-mon[32775]: pgmap v372: 97 pgs: 97 active+clean; 132 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 240 op/s 2024-04-26T01:57:43.631 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.70 MB/sec execute 487 sec latency 4648.844 ms 2024-04-26T01:57:44.631 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.69 MB/sec execute 488 sec latency 5648.993 ms 2024-04-26T01:57:44.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:44 smithi006 ceph-mon[28419]: pgmap v373: 97 pgs: 97 active+clean; 132 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-26T01:57:44.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:44 smithi188 ceph-mon[32751]: pgmap v373: 97 pgs: 97 active+clean; 132 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-26T01:57:44.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:44 smithi029 ceph-mon[32775]: pgmap v373: 97 pgs: 97 active+clean; 132 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-26T01:57:45.631 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.69 MB/sec execute 489 sec latency 6649.176 ms 2024-04-26T01:57:46.632 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.68 MB/sec execute 490 sec latency 7649.270 ms 2024-04-26T01:57:46.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:46 smithi006 ceph-mon[28419]: pgmap v374: 97 pgs: 97 active+clean; 132 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-26T01:57:46.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:46 smithi188 ceph-mon[32751]: pgmap v374: 97 pgs: 97 active+clean; 132 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-26T01:57:46.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:46 smithi029 ceph-mon[32775]: pgmap v374: 97 pgs: 97 active+clean; 132 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-26T01:57:47.632 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.67 MB/sec execute 491 sec latency 8649.374 ms 2024-04-26T01:57:48.632 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.66 MB/sec execute 492 sec latency 9649.500 ms 2024-04-26T01:57:48.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:48 smithi006 ceph-mon[28419]: pgmap v375: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-04-26T01:57:48.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:48 smithi188 ceph-mon[32751]: pgmap v375: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-04-26T01:57:48.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:48 smithi029 ceph-mon[32775]: pgmap v375: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-04-26T01:57:49.632 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.66 MB/sec execute 493 sec latency 10649.614 ms 2024-04-26T01:57:50.632 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.65 MB/sec execute 494 sec latency 11649.742 ms 2024-04-26T01:57:50.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:50 smithi006 ceph-mon[28419]: pgmap v376: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 999 KiB/s wr, 94 op/s 2024-04-26T01:57:50.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:50 smithi188 ceph-mon[32751]: pgmap v376: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 999 KiB/s wr, 94 op/s 2024-04-26T01:57:50.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:50 smithi029 ceph-mon[32775]: pgmap v376: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 999 KiB/s wr, 94 op/s 2024-04-26T01:57:51.632 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.64 MB/sec execute 495 sec latency 12649.850 ms 2024-04-26T01:57:52.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:52 smithi006 ceph-mon[28419]: pgmap v377: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-04-26T01:57:52.633 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.63 MB/sec execute 496 sec latency 13649.986 ms 2024-04-26T01:57:52.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:52 smithi188 ceph-mon[32751]: pgmap v377: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-04-26T01:57:52.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:52 smithi029 ceph-mon[32775]: pgmap v377: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-04-26T01:57:53.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.069+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2373120900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b081043c0 msgr2=0x7f9b081047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:53.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.069+0000 7f9b0ea00640 1 --2- 172.21.15.6:0/2373120900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b081043c0 0x7f9b081047a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009900 tx=0x7f9af802a410 comp rx=0 tx=0).stop 2024-04-26T01:57:53.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.069+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2373120900 shutdown_connections 2024-04-26T01:57:53.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.069+0000 7f9b0ea00640 1 --2- 172.21.15.6:0/2373120900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b08109490 0x7f9b08111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.069+0000 7f9b0ea00640 1 --2- 172.21.15.6:0/2373120900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b08104d70 0x7f9b08108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.069+0000 7f9b0ea00640 1 --2- 172.21.15.6:0/2373120900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b081043c0 0x7f9b081047a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.069+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2373120900 >> 172.21.15.6:0/2373120900 conn(0x7f9b080ffc40 msgr2=0x7f9b08102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:53.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.069+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2373120900 shutdown_connections 2024-04-26T01:57:53.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.069+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2373120900 wait complete. 2024-04-26T01:57:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.070+0000 7f9b0ea00640 1 Processor -- start 2024-04-26T01:57:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.070+0000 7f9b0ea00640 1 -- start start 2024-04-26T01:57:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.070+0000 7f9b0ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b081043c0 0x7f9b081a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.070+0000 7f9b0ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b08109490 0x7f9b081a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.070+0000 7f9b0ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b0819a7d0 0x7f9b0819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b0ea00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9b081143f0 con 0x7f9b081043c0 2024-04-26T01:57:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b0ea00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9b08114270 con 0x7f9b08109490 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b0ea00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9b08114570 con 0x7f9b0819a7d0 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b07e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b081043c0 0x7f9b081a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b07e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b081043c0 0x7f9b081a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57264/0 (socket says 172.21.15.6:57264) 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b07e00640 1 -- 172.21.15.6:0/2019941772 learned_addr learned my addr 172.21.15.6:0/2019941772 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b07400640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b08109490 0x7f9b081a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b0ca00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b0819a7d0 0x7f9b0819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b07e00640 1 -- 172.21.15.6:0/2019941772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b0819a7d0 msgr2=0x7f9b0819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b07e00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b0819a7d0 0x7f9b0819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b07e00640 1 -- 172.21.15.6:0/2019941772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b08109490 msgr2=0x7f9b081a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b07e00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b08109490 0x7f9b081a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.071+0000 7f9b07e00640 1 -- 172.21.15.6:0/2019941772 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b0819b360 con 0x7f9b081043c0 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.072+0000 7f9b07e00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b081043c0 0x7f9b081a0660 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9af8005940 tx=0x7f9af80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.072+0000 7f9b04c00640 1 -- 172.21.15.6:0/2019941772 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af803f030 con 0x7f9b081043c0 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.072+0000 7f9b04c00640 1 -- 172.21.15.6:0/2019941772 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9af80044c0 con 0x7f9b081043c0 2024-04-26T01:57:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.072+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2019941772 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b0819b5f0 con 0x7f9b081043c0 2024-04-26T01:57:53.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.072+0000 7f9b04c00640 1 -- 172.21.15.6:0/2019941772 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af8035070 con 0x7f9b081043c0 2024-04-26T01:57:53.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.072+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2019941772 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b081a77d0 con 0x7f9b081043c0 2024-04-26T01:57:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.075+0000 7f9b11867740 1 Processor -- start 2024-04-26T01:57:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.075+0000 7f9b04c00640 1 -- 172.21.15.6:0/2019941772 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f9af8042030 con 0x7f9b081043c0 2024-04-26T01:57:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.075+0000 7f9b11867740 1 -- start start 2024-04-26T01:57:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.076+0000 7f9b11867740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74b6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.076+0000 7f9b11867740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74c14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.076+0000 7f9b11867740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 0x5616d74c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.076+0000 7f9b11867740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5616d741c900 con 0x5616d74b6820 2024-04-26T01:57:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.076+0000 7f9b11867740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5616d73c2ef0 con 0x5616d74c1a00 2024-04-26T01:57:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.076+0000 7f9b11867740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5616d74188c0 con 0x5616d74b7140 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.076+0000 7f9b0ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74b6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.076+0000 7f9b0ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74b6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57280/0 (socket says 172.21.15.6:57280) 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.076+0000 7f9b0ca00640 1 -- 172.21.15.6:0/3926404529 learned_addr learned my addr 172.21.15.6:0/3926404529 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.076+0000 7f9b07e00640 1 --2- 172.21.15.6:0/3926404529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74c14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.076+0000 7f9b07400640 1 --2- 172.21.15.6:0/3926404529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 0x5616d74c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.077+0000 7f9b0ca00640 1 -- 172.21.15.6:0/3926404529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 msgr2=0x5616d74c14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.077+0000 7f9b0ca00640 1 --2- 172.21.15.6:0/3926404529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74c14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.077+0000 7f9b0ca00640 1 -- 172.21.15.6:0/3926404529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 msgr2=0x5616d74c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.077+0000 7f9b0ca00640 1 --2- 172.21.15.6:0/3926404529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 0x5616d74c3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.077+0000 7f9b0ca00640 1 -- 172.21.15.6:0/3926404529 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616d712f210 con 0x5616d74b6820 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.077+0000 7f9b0ca00640 1 --2- 172.21.15.6:0/3926404529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74b6c00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9b00009f40 tx=0x7f9b00010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd8fb5bdd5a7791 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.077+0000 7f9b04c00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9aec0a3d90 0x7f9aec0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.077+0000 7f9afd200640 1 -- 172.21.15.6:0/3926404529 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00003b70 con 0x5616d74b6820 2024-04-26T01:57:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.077+0000 7f9b04c00640 1 -- 172.21.15.6:0/2019941772 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7f9af80f0b80 con 0x7f9b081043c0 2024-04-26T01:57:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.077+0000 7f9afd200640 1 -- 172.21.15.6:0/3926404529 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b00003d10 con 0x5616d74b6820 2024-04-26T01:57:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.077+0000 7f9afd200640 1 -- 172.21.15.6:0/3926404529 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b000119a0 con 0x5616d74b6820 2024-04-26T01:57:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.077+0000 7f9b07e00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9aec0a3d90 0x7f9aec0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.078+0000 7f9b11867740 1 -- 172.21.15.6:0/3926404529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 msgr2=0x5616d74b6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.078+0000 7f9b11867740 1 --2- 172.21.15.6:0/3926404529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74b6c00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9b00009f40 tx=0x7f9b00010860 comp rx=0 tx=0).stop 2024-04-26T01:57:53.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.078+0000 7f9b07e00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9aec0a3d90 0x7f9aec0a6250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9af80ee330 tx=0x7f9af80f46c0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.078+0000 7f9b11867740 1 -- 172.21.15.6:0/3926404529 shutdown_connections 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.078+0000 7f9b11867740 1 --2- 172.21.15.6:0/3926404529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 0x5616d74c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.078+0000 7f9b11867740 1 --2- 172.21.15.6:0/3926404529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74c14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.078+0000 7f9b11867740 1 --2- 172.21.15.6:0/3926404529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74b6c00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.078+0000 7f9b11867740 1 -- 172.21.15.6:0/3926404529 >> 172.21.15.6:0/3926404529 conn(0x5616d74b2d30 msgr2=0x5616d74b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.078+0000 7f9b11867740 1 -- 172.21.15.6:0/3926404529 shutdown_connections 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.078+0000 7f9b11867740 1 -- 172.21.15.6:0/3926404529 wait complete. 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.079+0000 7f9b11867740 1 -- start start 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.079+0000 7f9b11867740 1 Processor -- start 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.079+0000 7f9b11867740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74be990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.079+0000 7f9b11867740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74beed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.079+0000 7f9b11867740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 0x5616d74bf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.079+0000 7f9b11867740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5616d74c4630 con 0x5616d74b6820 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.079+0000 7f9b11867740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5616d73d6310 con 0x5616d74c1a00 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.079+0000 7f9b11867740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5616d74c47b0 con 0x5616d74b7140 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.080+0000 7f9b11867740 10 client.0 _ll_register_callbacks cb 0x5616d73a4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.080+0000 7f9b0ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74be990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.080+0000 7f9b0ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74be990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57284/0 (socket says 172.21.15.6:57284) 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.080+0000 7f9b0ca00640 1 -- 172.21.15.6:0/1866169427 learned_addr learned my addr 172.21.15.6:0/1866169427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.080+0000 7f9b07400640 1 --2- 172.21.15.6:0/1866169427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74beed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.080+0000 7f9b0ca00640 1 -- 172.21.15.6:0/1866169427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 msgr2=0x5616d74beed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.080+0000 7f9b0ca00640 1 --2- 172.21.15.6:0/1866169427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74beed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.080+0000 7f9b0ca00640 1 -- 172.21.15.6:0/1866169427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 msgr2=0x5616d74bf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.080+0000 7f9b0ca00640 1 --2- 172.21.15.6:0/1866169427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 0x5616d74bf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.080+0000 7f9b0ca00640 1 -- 172.21.15.6:0/1866169427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616d71fb620 con 0x5616d74b6820 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.080+0000 7f9b0ca00640 1 --2- 172.21.15.6:0/1866169427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74be990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9b00006010 tx=0x7f9b00009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba474f23561ed707 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.080+0000 7f9af5600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:57:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.081+0000 7f9af5600640 1 -- 172.21.15.6:0/1866169427 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00011940 con 0x5616d74b6820 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.081+0000 7f9af5600640 1 -- 172.21.15.6:0/1866169427 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7f9b00011ae0 con 0x5616d74b6820 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.081+0000 7f9b11867740 1 -- 172.21.15.6:0/1866169427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b00003d10 con 0x5616d74b6820 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.081+0000 7f9af5600640 1 -- 172.21.15.6:0/1866169427 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f9b00011ce0 con 0x5616d74b6820 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.081+0000 7f9b11867740 10 client.16440 fetch_fsmap learned FSMap version 43 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.081+0000 7f9b11867740 1 -- 172.21.15.6:0/1866169427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5616d74d0890 con 0x5616d74b6820 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.081+0000 7f9af5600640 1 -- 172.21.15.6:0/1866169427 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b00003d10 con 0x5616d74b6820 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.082+0000 7f9af5600640 1 -- 172.21.15.6:0/1866169427 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f9b00005240 con 0x5616d74b6820 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.082+0000 7f9b11867740 10 client.16440 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.082+0000 7f9b11867740 10 client.16440 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.082+0000 7f9b11867740 20 client.16440 populate_metadata read hostname 'smithi006' 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.082+0000 7f9b11867740 1 --2- 172.21.15.6:0/1866169427 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5616d74d1c90 0x5616d74d4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.082+0000 7f9b11867740 4 client.16440 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:53.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.082+0000 7f9b11867740 1 -- 172.21.15.6:0/1866169427 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616d74308b0 con 0x5616d74d1c90 2024-04-26T01:57:53.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.082+0000 7f9b07400640 1 --2- 172.21.15.6:0/1866169427 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5616d74d1c90 0x5616d74d4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.083+0000 7f9b07400640 1 --2- 172.21.15.6:0/1866169427 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5616d74d1c90 0x5616d74d4080 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c085acfc0e2bc79 server_cookie=e4dc9e2b0cb97652 in_seq=0 out_seq=0 2024-04-26T01:57:53.085 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.083+0000 7f9af5600640 10 client.16440 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9af5600640 1 -- 172.21.15.6:0/1866169427 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616d74308b0 con 0x5616d74d1c90 2024-04-26T01:57:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9af5600640 10 client.16440 handle_command_reply: tid=0 2024-04-26T01:57:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9b11867740 1 client.16440 shutdown 2024-04-26T01:57:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9b11867740 1 -- 172.21.15.6:0/1866169427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 msgr2=0x5616d74be990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9b11867740 1 --2- 172.21.15.6:0/1866169427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74be990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9b00006010 tx=0x7f9b00009cc0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9b11867740 1 -- 172.21.15.6:0/1866169427 shutdown_connections 2024-04-26T01:57:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9b11867740 1 --2- 172.21.15.6:0/1866169427 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5616d74d1c90 0x5616d74d4080 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9b11867740 1 --2- 172.21.15.6:0/1866169427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 0x5616d74bf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9af5600640 0 client.16440 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9b11867740 1 --2- 172.21.15.6:0/1866169427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74beed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9b11867740 1 --2- 172.21.15.6:0/1866169427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74be990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.086 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9b11867740 1 -- 172.21.15.6:0/1866169427 >> 172.21.15.6:0/1866169427 conn(0x5616d74b2d30 msgr2=0x5616d74b8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9b11867740 1 -- 172.21.15.6:0/1866169427 shutdown_connections 2024-04-26T01:57:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9b11867740 1 -- 172.21.15.6:0/1866169427 wait complete. 2024-04-26T01:57:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.084+0000 7f9b11867740 20 client.16440 trim_cache size 0 max 16384 2024-04-26T01:57:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.101+0000 7f9b11867740 1 Processor -- start 2024-04-26T01:57:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.101+0000 7f9b11867740 1 -- start start 2024-04-26T01:57:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.101+0000 7f9b11867740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74b2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.101+0000 7f9b11867740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74b91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.101+0000 7f9b11867740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 0x5616d74d9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.101+0000 7f9b11867740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9b081b2890 con 0x5616d74b6820 2024-04-26T01:57:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.101+0000 7f9b11867740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9b081b0160 con 0x5616d74c1a00 2024-04-26T01:57:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.101+0000 7f9b11867740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9b081adfd0 con 0x5616d74b7140 2024-04-26T01:57:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b0ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b07400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74b91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b07400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74b91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:42656/0 (socket says 172.21.15.6:42656) 2024-04-26T01:57:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b0ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74b2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57294/0 (socket says 172.21.15.6:57294) 2024-04-26T01:57:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b07400640 1 -- 172.21.15.6:0/1246358778 learned_addr learned my addr 172.21.15.6:0/1246358778 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b07400640 1 -- 172.21.15.6:0/1246358778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 msgr2=0x5616d74d9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b0ca00640 1 --2- 172.21.15.6:0/1246358778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 0x5616d74d9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b07400640 1 --2- 172.21.15.6:0/1246358778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 0x5616d74d9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b07400640 1 -- 172.21.15.6:0/1246358778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 msgr2=0x5616d74b2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b07400640 1 --2- 172.21.15.6:0/1246358778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74b2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b07400640 1 -- 172.21.15.6:0/1246358778 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616d74b78b0 con 0x5616d74b7140 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b0ca00640 1 --2- 172.21.15.6:0/1246358778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 0x5616d74d9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.102+0000 7f9b07400640 1 --2- 172.21.15.6:0/1246358778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74b91e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9b08103390 tx=0x7f9af0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfb68f3f96169e77 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.103+0000 7f9af7e00640 1 -- 172.21.15.6:0/1246358778 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af0003de0 con 0x5616d74b7140 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.103+0000 7f9af7e00640 1 -- 172.21.15.6:0/1246358778 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9af0025070 con 0x5616d74b7140 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.103+0000 7f9af7e00640 1 -- 172.21.15.6:0/1246358778 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af0010990 con 0x5616d74b7140 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.103+0000 7f9b11867740 1 -- 172.21.15.6:0/1246358778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 msgr2=0x5616d74b91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.103+0000 7f9b11867740 1 --2- 172.21.15.6:0/1246358778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74b91e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9b08103390 tx=0x7f9af0005f90 comp rx=0 tx=0).stop 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.103+0000 7f9b11867740 1 -- 172.21.15.6:0/1246358778 shutdown_connections 2024-04-26T01:57:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.103+0000 7f9b11867740 1 --2- 172.21.15.6:0/1246358778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74c1a00 0x5616d74d9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.103+0000 7f9b11867740 1 --2- 172.21.15.6:0/1246358778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b7140 0x5616d74b91e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.103+0000 7f9b11867740 1 --2- 172.21.15.6:0/1246358778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74b6820 0x5616d74b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.103+0000 7f9b11867740 1 -- 172.21.15.6:0/1246358778 >> 172.21.15.6:0/1246358778 conn(0x5616d74b2d30 msgr2=0x5616d74bf1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:53.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.103+0000 7f9b11867740 1 -- 172.21.15.6:0/1246358778 shutdown_connections 2024-04-26T01:57:53.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.104+0000 7f9b11867740 1 -- 172.21.15.6:0/1246358778 wait complete. 2024-04-26T01:57:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.104+0000 7f9b11867740 1 -- start start 2024-04-26T01:57:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.104+0000 7f9b11867740 1 Processor -- start 2024-04-26T01:57:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.104+0000 7f9b11867740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b6820 0x5616d74c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.104+0000 7f9b11867740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74b7140 0x5616d74c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b11867740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74c1a00 0x5616d74c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b11867740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9b081ae620 con 0x5616d74c1a00 2024-04-26T01:57:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b11867740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9b081afa90 con 0x5616d74b7140 2024-04-26T01:57:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b11867740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9b081aecf0 con 0x5616d74b6820 2024-04-26T01:57:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b11867740 10 client.0 _ll_register_callbacks cb 0x7f9b00009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b0ca00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b6820 0x5616d74c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b0ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74c1a00 0x5616d74c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b07400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74b7140 0x5616d74c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b0ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74c1a00 0x5616d74c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57298/0 (socket says 172.21.15.6:57298) 2024-04-26T01:57:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b0ca00640 1 -- 172.21.15.6:0/2621755405 learned_addr learned my addr 172.21.15.6:0/2621755405 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:57:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b07400640 1 -- 172.21.15.6:0/2621755405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b6820 msgr2=0x5616d74c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b07400640 1 --2- 172.21.15.6:0/2621755405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b6820 0x5616d74c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b07400640 1 -- 172.21.15.6:0/2621755405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74c1a00 msgr2=0x5616d74c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b07400640 1 --2- 172.21.15.6:0/2621755405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74c1a00 0x5616d74c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b07400640 1 -- 172.21.15.6:0/2621755405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616d74c9150 con 0x5616d74b7140 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.105+0000 7f9b0ca00640 1 --2- 172.21.15.6:0/2621755405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74c1a00 0x5616d74c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.106+0000 7f9b07400640 1 --2- 172.21.15.6:0/2621755405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74b7140 0x5616d74c86a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x5616d74bfb90 tx=0x7f9af0010430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=553a9a2c1fd672bd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.106+0000 7f9af5600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.106+0000 7f9af5600640 1 -- 172.21.15.6:0/2621755405 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af002d070 con 0x5616d74b7140 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.106+0000 7f9b11867740 1 -- 172.21.15.6:0/2621755405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9af0025070 con 0x5616d74b7140 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.106+0000 7f9af5600640 1 -- 172.21.15.6:0/2621755405 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7f9af0010710 con 0x5616d74b7140 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.106+0000 7f9af5600640 1 -- 172.21.15.6:0/2621755405 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f9af0010910 con 0x5616d74b7140 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.106+0000 7f9b11867740 10 client.26597 fetch_fsmap learned FSMap version 43 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.106+0000 7f9b11867740 1 -- 172.21.15.6:0/2621755405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5616d7537c20 con 0x5616d74b7140 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.107+0000 7f9af5600640 1 -- 172.21.15.6:0/2621755405 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9af0025070 con 0x5616d74b7140 2024-04-26T01:57:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.107+0000 7f9af5600640 1 -- 172.21.15.6:0/2621755405 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f9af0032460 con 0x5616d74b7140 2024-04-26T01:57:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.107+0000 7f9b11867740 10 client.26597 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:57:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.107+0000 7f9b11867740 10 client.26597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:57:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.107+0000 7f9b11867740 20 client.26597 populate_metadata read hostname 'smithi006' 2024-04-26T01:57:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.107+0000 7f9b11867740 1 --2- 172.21.15.6:0/2621755405 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5616d7538380 0x5616d753a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.107+0000 7f9b11867740 4 client.26597 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:57:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.107+0000 7f9b11867740 1 -- 172.21.15.6:0/2621755405 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5616d742a320 con 0x5616d7538380 2024-04-26T01:57:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.108+0000 7f9b0ca00640 1 --2- 172.21.15.6:0/2621755405 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5616d7538380 0x5616d753a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.108+0000 7f9b0ca00640 1 --2- 172.21.15.6:0/2621755405 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5616d7538380 0x5616d753a7a0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=771e1bccd6efb2b0 server_cookie=6ae40c35982c9dda in_seq=0 out_seq=0 2024-04-26T01:57:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.108+0000 7f9af5600640 10 client.26597 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.140+0000 7f9af5600640 1 -- 172.21.15.6:0/2621755405 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5616d742a320 con 0x5616d7538380 2024-04-26T01:57:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.140+0000 7f9af5600640 10 client.26597 handle_command_reply: tid=0 2024-04-26T01:57:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.140+0000 7f9b11867740 1 client.26597 shutdown 2024-04-26T01:57:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.141+0000 7f9b11867740 1 -- 172.21.15.6:0/2621755405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74b7140 msgr2=0x5616d74c86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.141+0000 7f9b11867740 1 --2- 172.21.15.6:0/2621755405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74b7140 0x5616d74c86a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x5616d74bfb90 tx=0x7f9af0010430 comp rx=0 tx=0).stop 2024-04-26T01:57:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.141+0000 7f9b11867740 1 -- 172.21.15.6:0/2621755405 shutdown_connections 2024-04-26T01:57:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.141+0000 7f9b11867740 1 --2- 172.21.15.6:0/2621755405 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5616d7538380 0x5616d753a7a0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.141+0000 7f9b11867740 1 --2- 172.21.15.6:0/2621755405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5616d74c1a00 0x5616d74c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.141+0000 7f9b11867740 1 --2- 172.21.15.6:0/2621755405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616d74b7140 0x5616d74c86a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.141+0000 7f9b11867740 1 --2- 172.21.15.6:0/2621755405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5616d74b6820 0x5616d74c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.141+0000 7f9b11867740 1 -- 172.21.15.6:0/2621755405 >> 172.21.15.6:0/2621755405 conn(0x5616d74b2d30 msgr2=0x5616d74c01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:53.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.141+0000 7f9af5600640 0 client.26597 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:57:53.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.141+0000 7f9b11867740 1 -- 172.21.15.6:0/2621755405 shutdown_connections 2024-04-26T01:57:53.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.141+0000 7f9b11867740 1 -- 172.21.15.6:0/2621755405 wait complete. 2024-04-26T01:57:53.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.141+0000 7f9b11867740 20 client.26597 trim_cache size 0 max 16384 2024-04-26T01:57:53.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:57:53.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:57:53.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:57:53.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:57:53.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.142+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2019941772 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9aec0a3d90 msgr2=0x7f9aec0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:53.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.142+0000 7f9b0ea00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9aec0a3d90 0x7f9aec0a6250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9af80ee330 tx=0x7f9af80f46c0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.142+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2019941772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b081043c0 msgr2=0x7f9b081a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:53.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.142+0000 7f9b0ea00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b081043c0 0x7f9b081a0660 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9af8005940 tx=0x7f9af80040c0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.142+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2019941772 shutdown_connections 2024-04-26T01:57:53.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.142+0000 7f9b0ea00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9aec0a3d90 0x7f9aec0a6250 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.143 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.142+0000 7f9b0ea00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b0819a7d0 0x7f9b0819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.143 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.142+0000 7f9b0ea00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b08109490 0x7f9b081a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.143 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.142+0000 7f9b0ea00640 1 --2- 172.21.15.6:0/2019941772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b081043c0 0x7f9b081a0660 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:53.143 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.142+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2019941772 >> 172.21.15.6:0/2019941772 conn(0x7f9b080ffc40 msgr2=0x7f9b08101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:53.143 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.142+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2019941772 shutdown_connections 2024-04-26T01:57:53.143 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:57:53.142+0000 7f9b0ea00640 1 -- 172.21.15.6:0/2019941772 wait complete. 2024-04-26T01:57:53.633 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.63 MB/sec execute 497 sec latency 14650.144 ms 2024-04-26T01:57:54.633 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.62 MB/sec execute 498 sec latency 15650.295 ms 2024-04-26T01:57:54.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:54 smithi006 ceph-mon[28419]: pgmap v378: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 692 KiB/s wr, 35 op/s 2024-04-26T01:57:54.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:54 smithi188 ceph-mon[32751]: pgmap v378: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 692 KiB/s wr, 35 op/s 2024-04-26T01:57:54.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:54 smithi029 ceph-mon[32775]: pgmap v378: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 692 KiB/s wr, 35 op/s 2024-04-26T01:57:55.633 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.61 MB/sec execute 499 sec latency 16650.445 ms 2024-04-26T01:57:56.633 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.60 MB/sec execute 500 sec latency 17650.556 ms 2024-04-26T01:57:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:56 smithi188 ceph-mon[32751]: pgmap v379: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 692 KiB/s wr, 35 op/s 2024-04-26T01:57:56.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:56 smithi029 ceph-mon[32775]: pgmap v379: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 692 KiB/s wr, 35 op/s 2024-04-26T01:57:57.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:56 smithi006 ceph-mon[28419]: pgmap v379: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 692 KiB/s wr, 35 op/s 2024-04-26T01:57:57.633 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.60 MB/sec execute 501 sec latency 18650.677 ms 2024-04-26T01:57:58.633 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.59 MB/sec execute 502 sec latency 19650.812 ms 2024-04-26T01:57:58.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:57:58 smithi188 ceph-mon[32751]: pgmap v380: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.1 MiB/s wr, 58 op/s 2024-04-26T01:57:58.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:57:58 smithi029 ceph-mon[32775]: pgmap v380: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.1 MiB/s wr, 58 op/s 2024-04-26T01:57:59.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:57:58 smithi006 ceph-mon[28419]: pgmap v380: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.1 MiB/s wr, 58 op/s 2024-04-26T01:57:59.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:57:59.633 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.58 MB/sec execute 503 sec latency 20650.907 ms 2024-04-26T01:57:59.733 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:00.633 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.58 MB/sec execute 504 sec latency 21651.020 ms 2024-04-26T01:58:00.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:00 smithi188 ceph-mon[32751]: pgmap v381: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 793 KiB/s wr, 41 op/s 2024-04-26T01:58:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:00 smithi029 ceph-mon[32775]: pgmap v381: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 793 KiB/s wr, 41 op/s 2024-04-26T01:58:00.945 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:00 smithi006 ceph-mon[28419]: pgmap v381: 97 pgs: 97 active+clean; 131 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 793 KiB/s wr, 41 op/s 2024-04-26T01:58:01.634 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.57 MB/sec execute 505 sec latency 22651.168 ms 2024-04-26T01:58:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.137+0000 7fd175600700 1 -- 172.21.15.6:0/1013193593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd170109940 msgr2=0x7fd170115de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.137+0000 7fd175600700 1 --2- 172.21.15.6:0/1013193593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd170109940 0x7fd170115de0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd168009f90 tx=0x7fd16800b330 comp rx=0 tx=0).stop 2024-04-26T01:58:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.137+0000 7fd175600700 1 -- 172.21.15.6:0/1013193593 shutdown_connections 2024-04-26T01:58:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.137+0000 7fd175600700 1 --2- 172.21.15.6:0/1013193593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd170109940 0x7fd170115de0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.137+0000 7fd175600700 1 --2- 172.21.15.6:0/1013193593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd17010f030 0x7fd17010f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.137+0000 7fd175600700 1 --2- 172.21.15.6:0/1013193593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd170108f80 0x7fd170109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.137+0000 7fd175600700 1 -- 172.21.15.6:0/1013193593 >> 172.21.15.6:0/1013193593 conn(0x7fd1700787d0 msgr2=0x7fd170107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.137+0000 7fd175600700 1 -- 172.21.15.6:0/1013193593 shutdown_connections 2024-04-26T01:58:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.137+0000 7fd175600700 1 -- 172.21.15.6:0/1013193593 wait complete. 2024-04-26T01:58:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.138+0000 7fd175600700 1 Processor -- start 2024-04-26T01:58:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.138+0000 7fd175600700 1 -- start start 2024-04-26T01:58:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.139+0000 7fd175600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd170108f80 0x7fd17019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.139+0000 7fd175600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd170109940 0x7fd17019fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.139+0000 7fd175600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd17010f030 0x7fd1701a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.139+0000 7fd175600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd170119170 con 0x7fd170108f80 2024-04-26T01:58:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.139+0000 7fd175600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd170118ff0 con 0x7fd17010f030 2024-04-26T01:58:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.139+0000 7fd175600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd1701192f0 con 0x7fd170109940 2024-04-26T01:58:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.139+0000 7fd16fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd170108f80 0x7fd17019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.139+0000 7fd16fe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd170108f80 0x7fd17019f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57504/0 (socket says 172.21.15.6:57504) 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.139+0000 7fd16fe00700 1 -- 172.21.15.6:0/1661073701 learned_addr learned my addr 172.21.15.6:0/1661073701 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.139+0000 7fd174c00700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd17010f030 0x7fd1701a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.139+0000 7fd16f400700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd170109940 0x7fd17019fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.140+0000 7fd174c00700 1 -- 172.21.15.6:0/1661073701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd170109940 msgr2=0x7fd17019fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.140+0000 7fd174c00700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd170109940 0x7fd17019fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.140+0000 7fd174c00700 1 -- 172.21.15.6:0/1661073701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd170108f80 msgr2=0x7fd17019f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.140+0000 7fd174c00700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd170108f80 0x7fd17019f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.140+0000 7fd174c00700 1 -- 172.21.15.6:0/1661073701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1701a4610 con 0x7fd17010f030 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.140+0000 7fd174c00700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd17010f030 0x7fd1701a3ed0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd16800b6c0 tx=0x7fd16800cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.140+0000 7fd16cc00700 1 -- 172.21.15.6:0/1661073701 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168021070 con 0x7fd17010f030 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.141+0000 7fd16cc00700 1 -- 172.21.15.6:0/1661073701 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd168007d60 con 0x7fd17010f030 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.141+0000 7fd175600700 1 -- 172.21.15.6:0/1661073701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1701a48a0 con 0x7fd17010f030 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.141+0000 7fd16cc00700 1 -- 172.21.15.6:0/1661073701 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168005040 con 0x7fd17010f030 2024-04-26T01:58:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.141+0000 7fd175600700 1 -- 172.21.15.6:0/1661073701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1701ac180 con 0x7fd17010f030 2024-04-26T01:58:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.142+0000 7fd175600700 1 -- 172.21.15.6:0/1661073701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd170111380 con 0x7fd17010f030 2024-04-26T01:58:02.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.144+0000 7fd16cc00700 1 -- 172.21.15.6:0/1661073701 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fd168024070 con 0x7fd17010f030 2024-04-26T01:58:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.145+0000 7fd16cc00700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd1540a3db0 0x7fd1540a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.146+0000 7fd16cc00700 1 -- 172.21.15.6:0/1661073701 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8600+0+0 (secure 0 0 0) 0x7fd16800f020 con 0x7fd17010f030 2024-04-26T01:58:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.146+0000 7fd16fe00700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd1540a3db0 0x7fd1540a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:02.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.146+0000 7fd16fe00700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd1540a3db0 0x7fd1540a6270 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd160009750 tx=0x7fd160009310 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.148+0000 7fd16cc00700 1 -- 172.21.15.6:0/1661073701 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd16802b020 con 0x7fd17010f030 2024-04-26T01:58:02.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.432+0000 7fd175600700 1 -- 172.21.15.6:0/1661073701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd170069730 con 0x7fd17010f030 2024-04-26T01:58:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.433+0000 7fd16cc00700 1 -- 172.21.15.6:0/1661073701 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11850 (secure 0 0 0) 0x7fd168007420 con 0x7fd17010f030 2024-04-26T01:58:02.432 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:58:02.432 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":43,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:55:18.912671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:58:02.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.436+0000 7fd175600700 1 -- 172.21.15.6:0/1661073701 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd1540a3db0 msgr2=0x7fd1540a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:02.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.436+0000 7fd175600700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd1540a3db0 0x7fd1540a6270 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd160009750 tx=0x7fd160009310 comp rx=0 tx=0).stop 2024-04-26T01:58:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.436+0000 7fd175600700 1 -- 172.21.15.6:0/1661073701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd17010f030 msgr2=0x7fd1701a3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.436+0000 7fd175600700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd17010f030 0x7fd1701a3ed0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd16800b6c0 tx=0x7fd16800cc30 comp rx=0 tx=0).stop 2024-04-26T01:58:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.436+0000 7fd175600700 1 -- 172.21.15.6:0/1661073701 shutdown_connections 2024-04-26T01:58:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.436+0000 7fd175600700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd1540a3db0 0x7fd1540a6270 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.436+0000 7fd175600700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd17010f030 0x7fd1701a3ed0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.436+0000 7fd175600700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd170108f80 0x7fd17019f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.436+0000 7fd175600700 1 --2- 172.21.15.6:0/1661073701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd170109940 0x7fd17019fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.436+0000 7fd175600700 1 -- 172.21.15.6:0/1661073701 >> 172.21.15.6:0/1661073701 conn(0x7fd1700787d0 msgr2=0x7fd170079ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.436+0000 7fd175600700 1 -- 172.21.15.6:0/1661073701 shutdown_connections 2024-04-26T01:58:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:02.436+0000 7fd175600700 1 -- 172.21.15.6:0/1661073701 wait complete. 2024-04-26T01:58:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-26T01:58:02.634 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.56 MB/sec execute 506 sec latency 23651.253 ms 2024-04-26T01:58:02.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:02 smithi006 ceph-mon[28419]: pgmap v382: 97 pgs: 97 active+clean; 130 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 1.3 MiB/s wr, 74 op/s 2024-04-26T01:58:02.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:02 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1661073701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:02 smithi188 ceph-mon[32751]: pgmap v382: 97 pgs: 97 active+clean; 130 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 1.3 MiB/s wr, 74 op/s 2024-04-26T01:58:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:02 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1661073701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:02 smithi029 ceph-mon[32775]: pgmap v382: 97 pgs: 97 active+clean; 130 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 1.3 MiB/s wr, 74 op/s 2024-04-26T01:58:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:02 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1661073701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:03.326 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id 8cb585fa --release --await' 2024-04-26T01:58:03.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id 8cb585fa --release --await 2024-04-26T01:58:03.634 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.56 MB/sec execute 507 sec latency 24651.341 ms 2024-04-26T01:58:03.820 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:04.634 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.55 MB/sec execute 508 sec latency 25651.405 ms 2024-04-26T01:58:04.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:04 smithi188 ceph-mon[32751]: pgmap v383: 97 pgs: 97 active+clean; 130 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 940 KiB/s wr, 54 op/s 2024-04-26T01:58:04.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:04 smithi029 ceph-mon[32775]: pgmap v383: 97 pgs: 97 active+clean; 130 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 940 KiB/s wr, 54 op/s 2024-04-26T01:58:04.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:04 smithi006 ceph-mon[28419]: pgmap v383: 97 pgs: 97 active+clean; 130 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 940 KiB/s wr, 54 op/s 2024-04-26T01:58:05.634 INFO:tasks.workunit.client.0.smithi006.stdout: 1 427484 3.54 MB/sec execute 509 sec latency 26651.486 ms 2024-04-26T01:58:05.762 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:05 smithi006 ceph-mon[28419]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T01:58:05.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:05 smithi188 ceph-mon[32751]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T01:58:05.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:05 smithi029 ceph-mon[32775]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T01:58:05.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.952+0000 7fd423200700 1 -- 172.21.15.6:0/649566819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c113c70 msgr2=0x7fd41c116060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:05.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.952+0000 7fd423200700 1 --2- 172.21.15.6:0/649566819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c113c70 0x7fd41c116060 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd41400b0a0 tx=0x7fd41400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.953+0000 7fd423200700 1 -- 172.21.15.6:0/649566819 shutdown_connections 2024-04-26T01:58:05.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.953+0000 7fd423200700 1 --2- 172.21.15.6:0/649566819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c113c70 0x7fd41c116060 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.953+0000 7fd423200700 1 --2- 172.21.15.6:0/649566819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c074c70 0x7fd41c075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.953+0000 7fd423200700 1 --2- 172.21.15.6:0/649566819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd41c075590 0x7fd41c113730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.953+0000 7fd423200700 1 -- 172.21.15.6:0/649566819 >> 172.21.15.6:0/649566819 conn(0x7fd41c079b90 msgr2=0x7fd41c079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:05.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.953+0000 7fd423200700 1 -- 172.21.15.6:0/649566819 shutdown_connections 2024-04-26T01:58:05.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.953+0000 7fd423200700 1 -- 172.21.15.6:0/649566819 wait complete. 2024-04-26T01:58:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.954+0000 7fd423200700 1 Processor -- start 2024-04-26T01:58:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd423200700 1 -- start start 2024-04-26T01:58:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd423200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c074c70 0x7fd41c1a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd423200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd41c075590 0x7fd41c1a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd423200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c113c70 0x7fd41c19f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd423200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c1192c0 con 0x7fd41c074c70 2024-04-26T01:58:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd423200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c119140 con 0x7fd41c113c70 2024-04-26T01:58:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd423200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c119440 con 0x7fd41c075590 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd41be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c074c70 0x7fd41c1a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd413400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd41c075590 0x7fd41c1a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd413400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd41c075590 0x7fd41c1a5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:57170/0 (socket says 172.21.15.6:57170) 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd413400700 1 -- 172.21.15.6:0/2102866070 learned_addr learned my addr 172.21.15.6:0/2102866070 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd41be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c074c70 0x7fd41c1a53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57526/0 (socket says 172.21.15.6:57526) 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.955+0000 7fd421200700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c113c70 0x7fd41c19f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.956+0000 7fd413400700 1 -- 172.21.15.6:0/2102866070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c113c70 msgr2=0x7fd41c19f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.956+0000 7fd413400700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c113c70 0x7fd41c19f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.956+0000 7fd413400700 1 -- 172.21.15.6:0/2102866070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c074c70 msgr2=0x7fd41c1a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.956+0000 7fd413400700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c074c70 0x7fd41c1a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.956+0000 7fd413400700 1 -- 172.21.15.6:0/2102866070 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41c19fd00 con 0x7fd41c075590 2024-04-26T01:58:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.956+0000 7fd413400700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd41c075590 0x7fd41c1a5920 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd40400e9c0 tx=0x7fd40400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.957+0000 7fd419600700 1 -- 172.21.15.6:0/2102866070 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40400cbd0 con 0x7fd41c075590 2024-04-26T01:58:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.957+0000 7fd419600700 1 -- 172.21.15.6:0/2102866070 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd40400ee60 con 0x7fd41c075590 2024-04-26T01:58:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.957+0000 7fd423200700 1 -- 172.21.15.6:0/2102866070 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd41c19fff0 con 0x7fd41c075590 2024-04-26T01:58:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.957+0000 7fd419600700 1 -- 172.21.15.6:0/2102866070 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4040055c0 con 0x7fd41c075590 2024-04-26T01:58:05.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.957+0000 7fd423200700 1 -- 172.21.15.6:0/2102866070 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd41c1ac2b0 con 0x7fd41c075590 2024-04-26T01:58:05.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.960+0000 7fd419600700 1 -- 172.21.15.6:0/2102866070 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fd404004750 con 0x7fd41c075590 2024-04-26T01:58:05.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.961+0000 7fd419600700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd4000a3e30 0x7fd4000a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.962+0000 7fd42dbf3b80 1 Processor -- start 2024-04-26T01:58:05.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.962+0000 7fd419600700 1 -- 172.21.15.6:0/2102866070 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8644+0+0 (secure 0 0 0) 0x7fd4040d0a40 con 0x7fd41c075590 2024-04-26T01:58:05.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.962+0000 7fd42dbf3b80 1 -- start start 2024-04-26T01:58:05.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.962+0000 7fd41be00700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd4000a3e30 0x7fd4000a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.962+0000 7fd42dbf3b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 0x561134076120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.962+0000 7fd42dbf3b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134076660 0x561134080f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.962+0000 7fd42dbf3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611340814d0 0x5611340838c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.962+0000 7fd42dbf3b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x561133fc8f90 con 0x561134075d40 2024-04-26T01:58:05.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.962+0000 7fd42dbf3b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561133f6f830 con 0x561134076660 2024-04-26T01:58:05.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.962+0000 7fd42dbf3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561133fc4f50 con 0x5611340814d0 2024-04-26T01:58:05.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.963+0000 7fd41be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 0x561134076120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.963+0000 7fd41be00700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd4000a3e30 0x7fd4000a62f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009c60 tx=0x7fd40c009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:05.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.963+0000 7fd41be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 0x561134076120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57530/0 (socket says 172.21.15.6:57530) 2024-04-26T01:58:05.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.963+0000 7fd413400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134076660 0x561134080f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.963+0000 7fd41be00700 1 -- 172.21.15.6:0/3102300261 learned_addr learned my addr 172.21.15.6:0/3102300261 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:05.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.963+0000 7fd421200700 1 --2- 172.21.15.6:0/3102300261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611340814d0 0x5611340838c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.963+0000 7fd41be00700 1 -- 172.21.15.6:0/3102300261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611340814d0 msgr2=0x5611340838c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:05.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.963+0000 7fd41be00700 1 --2- 172.21.15.6:0/3102300261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611340814d0 0x5611340838c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.963+0000 7fd41be00700 1 -- 172.21.15.6:0/3102300261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134076660 msgr2=0x561134080f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:05.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.963+0000 7fd41be00700 1 --2- 172.21.15.6:0/3102300261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134076660 0x561134080f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.963+0000 7fd41be00700 1 -- 172.21.15.6:0/3102300261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561133f65780 con 0x561134075d40 2024-04-26T01:58:05.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.963+0000 7fd413400700 1 --2- 172.21.15.6:0/3102300261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134076660 0x561134080f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:05.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.964+0000 7fd41be00700 1 --2- 172.21.15.6:0/3102300261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 0x561134076120 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009b20 tx=0x7fd40c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2652ef68847c5851 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:05.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.964+0000 7fd411600700 1 -- 172.21.15.6:0/3102300261 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40c00ee20 con 0x561134075d40 2024-04-26T01:58:05.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.964+0000 7fd411600700 1 -- 172.21.15.6:0/3102300261 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd40c01d8c0 con 0x561134075d40 2024-04-26T01:58:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.964+0000 7fd411600700 1 -- 172.21.15.6:0/3102300261 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40c0259d0 con 0x561134075d40 2024-04-26T01:58:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.965+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3102300261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 msgr2=0x561134076120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.965+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/3102300261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 0x561134076120 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009b20 tx=0x7fd40c014000 comp rx=0 tx=0).stop 2024-04-26T01:58:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.965+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3102300261 shutdown_connections 2024-04-26T01:58:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.965+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/3102300261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611340814d0 0x5611340838c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.965+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/3102300261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 0x561134076120 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.965+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/3102300261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134076660 0x561134080f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.965+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3102300261 >> 172.21.15.6:0/3102300261 conn(0x561133fa5050 msgr2=0x561134072250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.965+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3102300261 shutdown_connections 2024-04-26T01:58:05.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.965+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3102300261 wait complete. 2024-04-26T01:58:05.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.966+0000 7fd42dbf3b80 1 -- start start 2024-04-26T01:58:05.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.966+0000 7fd42dbf3b80 1 Processor -- start 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.966+0000 7fd42dbf3b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 0x561134080c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.966+0000 7fd42dbf3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134076660 0x56113407df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.966+0000 7fd42dbf3b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5611340814d0 0x56113407e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.966+0000 7fd42dbf3b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x561134083fc0 con 0x561134075d40 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.966+0000 7fd42dbf3b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561133f82960 con 0x5611340814d0 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.966+0000 7fd42dbf3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561134084140 con 0x561134076660 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.967+0000 7fd42dbf3b80 10 client.0 _ll_register_callbacks cb 0x561133f61820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.967+0000 7fd41be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 0x561134080c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.967+0000 7fd413400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134076660 0x56113407df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.967+0000 7fd41be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 0x561134080c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57534/0 (socket says 172.21.15.6:57534) 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.967+0000 7fd41be00700 1 -- 172.21.15.6:0/2033628651 learned_addr learned my addr 172.21.15.6:0/2033628651 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.967+0000 7fd421200700 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5611340814d0 0x56113407e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.967+0000 7fd421200700 1 -- 172.21.15.6:0/2033628651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134076660 msgr2=0x56113407df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.967+0000 7fd421200700 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134076660 0x56113407df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.967+0000 7fd421200700 1 -- 172.21.15.6:0/2033628651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 msgr2=0x561134080c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.967+0000 7fd421200700 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 0x561134080c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.967+0000 7fd421200700 1 -- 172.21.15.6:0/2033628651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561133f169d0 con 0x5611340814d0 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.967+0000 7fd413400700 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134076660 0x56113407df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.968+0000 7fd421200700 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5611340814d0 0x56113407e480 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd4140062a0 tx=0x7fd41400be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67a6f3e0e510549b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.968+0000 7fd40a000700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.968+0000 7fd40a000700 1 -- 172.21.15.6:0/2033628651 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4140076e0 con 0x5611340814d0 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.968+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/2033628651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd40c01d8c0 con 0x5611340814d0 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.968+0000 7fd40a000700 1 -- 172.21.15.6:0/2033628651 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8644+0+0 (secure 0 0 0) 0x7fd414007880 con 0x5611340814d0 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.968+0000 7fd40a000700 1 -- 172.21.15.6:0/2033628651 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd414007a80 con 0x5611340814d0 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.968+0000 7fd42dbf3b80 10 client.26615 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.968+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/2033628651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x561133f5fbf0 con 0x5611340814d0 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.969+0000 7fd40a000700 1 -- 172.21.15.6:0/2033628651 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd40c01d8c0 con 0x5611340814d0 2024-04-26T01:58:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.969+0000 7fd40a000700 1 -- 172.21.15.6:0/2033628651 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7fd414011070 con 0x5611340814d0 2024-04-26T01:58:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.969+0000 7fd42dbf3b80 10 client.26615 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.969+0000 7fd42dbf3b80 10 client.26615 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:58:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.969+0000 7fd42dbf3b80 20 client.26615 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.969+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x561134090c80 0x561134093070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.969+0000 7fd42dbf3b80 4 client.26615 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.969+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/2033628651 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561133fd6780 con 0x561134090c80 2024-04-26T01:58:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.970+0000 7fd41be00700 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x561134090c80 0x561134093070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.970+0000 7fd41be00700 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x561134090c80 0x561134093070 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff881327ff26d9cf server_cookie=60506700fbf4b2ae in_seq=0 out_seq=0 2024-04-26T01:58:05.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.970+0000 7fd40a000700 10 client.26615 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.971+0000 7fd40a000700 1 -- 172.21.15.6:0/2033628651 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561133fd6780 con 0x561134090c80 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.971+0000 7fd40a000700 10 client.26615 handle_command_reply: tid=0 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.971+0000 7fd42dbf3b80 1 client.26615 shutdown 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.971+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/2033628651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5611340814d0 msgr2=0x56113407e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.971+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5611340814d0 0x56113407e480 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd4140062a0 tx=0x7fd41400be20 comp rx=0 tx=0).stop 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.971+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/2033628651 shutdown_connections 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.971+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5611340814d0 0x56113407e480 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.971+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561134075d40 0x561134080c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.971+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x561134090c80 0x561134093070 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.971+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/2033628651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134076660 0x56113407df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.971+0000 7fd40a000700 0 client.26615 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.971+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/2033628651 >> 172.21.15.6:0/2033628651 conn(0x561133fa5050 msgr2=0x561134071e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.972+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/2033628651 shutdown_connections 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.972+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/2033628651 wait complete. 2024-04-26T01:58:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.972+0000 7fd42dbf3b80 20 client.26615 trim_cache size 0 max 16384 2024-04-26T01:58:05.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.994+0000 7fd42dbf3b80 1 Processor -- start 2024-04-26T01:58:05.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.994+0000 7fd42dbf3b80 1 -- start start 2024-04-26T01:58:05.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.995+0000 7fd42dbf3b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134075d40 0x56113408cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.995+0000 7fd42dbf3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134076660 0x5611340988f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.995+0000 7fd42dbf3b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 0x561134098e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.995+0000 7fd42dbf3b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c1b6360 con 0x5611340814d0 2024-04-26T01:58:05.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.995+0000 7fd42dbf3b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c1b3bf0 con 0x561134075d40 2024-04-26T01:58:05.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.995+0000 7fd42dbf3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c1b1a60 con 0x561134076660 2024-04-26T01:58:05.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.995+0000 7fd41be00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134075d40 0x56113408cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.995+0000 7fd421200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 0x561134098e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.995+0000 7fd41be00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134075d40 0x56113408cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55496/0 (socket says 172.21.15.6:55496) 2024-04-26T01:58:05.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.995+0000 7fd41be00700 1 -- 172.21.15.6:0/1042455717 learned_addr learned my addr 172.21.15.6:0/1042455717 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:05.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.995+0000 7fd413400700 1 --2- 172.21.15.6:0/1042455717 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134076660 0x5611340988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.996+0000 7fd41be00700 1 -- 172.21.15.6:0/1042455717 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134076660 msgr2=0x5611340988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:05.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.996+0000 7fd41be00700 1 --2- 172.21.15.6:0/1042455717 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134076660 0x5611340988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.996+0000 7fd41be00700 1 -- 172.21.15.6:0/1042455717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 msgr2=0x561134098e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:05.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.996+0000 7fd41be00700 1 --2- 172.21.15.6:0/1042455717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 0x561134098e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.996+0000 7fd41be00700 1 -- 172.21.15.6:0/1042455717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611340734c0 con 0x561134075d40 2024-04-26T01:58:05.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.996+0000 7fd421200700 1 --2- 172.21.15.6:0/1042455717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 0x561134098e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:05.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.996+0000 7fd41be00700 1 --2- 172.21.15.6:0/1042455717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134075d40 0x56113408cec0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd40c011ab0 tx=0x7fd40c0092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67a82ddf06181208 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:05.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.996+0000 7fd410c00700 1 -- 172.21.15.6:0/1042455717 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40c039070 con 0x561134075d40 2024-04-26T01:58:05.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.996+0000 7fd410c00700 1 -- 172.21.15.6:0/1042455717 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd40c005c80 con 0x561134075d40 2024-04-26T01:58:05.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.996+0000 7fd410c00700 1 -- 172.21.15.6:0/1042455717 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40c034570 con 0x561134075d40 2024-04-26T01:58:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.997+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/1042455717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134075d40 msgr2=0x56113408cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.997+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/1042455717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134075d40 0x56113408cec0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd40c011ab0 tx=0x7fd40c0092d0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.997+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/1042455717 shutdown_connections 2024-04-26T01:58:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.997+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/1042455717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 0x561134098e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.997+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/1042455717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134075d40 0x56113408cec0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.997+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/1042455717 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134076660 0x5611340988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.997+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/1042455717 >> 172.21.15.6:0/1042455717 conn(0x561133fa5050 msgr2=0x561134080000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.997+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/1042455717 shutdown_connections 2024-04-26T01:58:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.997+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/1042455717 wait complete. 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.998+0000 7fd42dbf3b80 1 -- start start 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.998+0000 7fd42dbf3b80 1 Processor -- start 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd42dbf3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134075d40 0x561134084390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd42dbf3b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134076660 0x5611340848d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd42dbf3b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 0x5611340f7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd42dbf3b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c1b20b0 con 0x5611340814d0 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd42dbf3b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c1b3520 con 0x561134076660 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd42dbf3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c1b2780 con 0x561134075d40 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd42dbf3b80 10 client.0 _ll_register_callbacks cb 0x561133f61820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd421200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 0x5611340f7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd421200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 0x5611340f7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57550/0 (socket says 172.21.15.6:57550) 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd421200700 1 -- 172.21.15.6:0/3545545781 learned_addr learned my addr 172.21.15.6:0/3545545781 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd413400700 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134076660 0x5611340848d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:05.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd41be00700 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134075d40 0x561134084390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd421200700 1 -- 172.21.15.6:0/3545545781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134075d40 msgr2=0x561134084390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd421200700 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134075d40 0x561134084390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd421200700 1 -- 172.21.15.6:0/3545545781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134076660 msgr2=0x5611340848d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd421200700 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134076660 0x5611340848d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:05.999+0000 7fd421200700 1 -- 172.21.15.6:0/3545545781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611340f76a0 con 0x5611340814d0 2024-04-26T01:58:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.000+0000 7fd421200700 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 0x5611340f7160 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd41402f430 tx=0x7fd41402f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd4321ce0fd1aa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.000+0000 7fd40a000700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:58:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.000+0000 7fd40a000700 1 -- 172.21.15.6:0/3545545781 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd414005c60 con 0x5611340814d0 2024-04-26T01:58:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.000+0000 7fd40a000700 1 -- 172.21.15.6:0/3545545781 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8644+0+0 (secure 0 0 0) 0x7fd414005e00 con 0x5611340814d0 2024-04-26T01:58:06.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.000+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3545545781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561133fd0820 con 0x5611340814d0 2024-04-26T01:58:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.000+0000 7fd40a000700 1 -- 172.21.15.6:0/3545545781 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd41403e050 con 0x5611340814d0 2024-04-26T01:58:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.000+0000 7fd42dbf3b80 10 client.16470 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.000+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3545545781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5611340f8900 con 0x5611340814d0 2024-04-26T01:58:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.001+0000 7fd40a000700 1 -- 172.21.15.6:0/3545545781 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561133fd0820 con 0x5611340814d0 2024-04-26T01:58:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.001+0000 7fd40a000700 1 -- 172.21.15.6:0/3545545781 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7fd414036020 con 0x5611340814d0 2024-04-26T01:58:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.001+0000 7fd42dbf3b80 10 client.16470 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.001+0000 7fd42dbf3b80 10 client.16470 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:58:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.001+0000 7fd42dbf3b80 20 client.16470 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.001+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5611340f8fd0 0x5611340fb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.001+0000 7fd42dbf3b80 4 client.16470 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "8cb585fa", "release": true, "await": true}] 2024-04-26T01:58:06.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.001+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3545545781 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "8cb585fa", "release": true, "await": true}) -- 0x561133e49350 con 0x5611340f8fd0 2024-04-26T01:58:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.003+0000 7fd41be00700 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5611340f8fd0 0x5611340fb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.004+0000 7fd41be00700 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5611340f8fd0 0x5611340fb3f0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6713643cb7e8245 server_cookie=195a941b2df93e1c in_seq=0 out_seq=0 2024-04-26T01:58:06.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.004+0000 7fd40a000700 10 client.16470 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:06.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.012+0000 7fd40a000700 1 -- 172.21.15.6:0/3545545781 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x561133e49350 con 0x5611340f8fd0 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.012+0000 7fd40a000700 10 client.16470 handle_command_reply: tid=0 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.012+0000 7fd42dbf3b80 1 client.16470 shutdown 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.012+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3545545781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 msgr2=0x5611340f7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.012+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 0x5611340f7160 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd41402f430 tx=0x7fd41402f540 comp rx=0 tx=0).stop 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.012+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3545545781 shutdown_connections 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.013+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5611340814d0 0x5611340f7160 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.013+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5611340f8fd0 0x5611340fb3f0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.013+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561134075d40 0x561134084390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.013+0000 7fd42dbf3b80 1 --2- 172.21.15.6:0/3545545781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561134076660 0x5611340848d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.013+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3545545781 >> 172.21.15.6:0/3545545781 conn(0x561133fa5050 msgr2=0x561134077de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.013+0000 7fd40a000700 0 client.16470 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.013+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3545545781 shutdown_connections 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.013+0000 7fd42dbf3b80 1 -- 172.21.15.6:0/3545545781 wait complete. 2024-04-26T01:58:06.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.013+0000 7fd42dbf3b80 20 client.16470 trim_cache size 0 max 16384 2024-04-26T01:58:06.013 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:58:06.013 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 43, 2024-04-26T01:58:06.013 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:58:06.013 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 36, 2024-04-26T01:58:06.013 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:58:06.013 INFO:teuthology.orchestra.run.smithi006.stdout: "8cb585fa": { 2024-04-26T01:58:06.013 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 36, 2024-04-26T01:58:06.013 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:58:06.013 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:58:06.013 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 79.2, 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:06.014 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:06.015 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:58:06.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.014+0000 7fd423200700 1 -- 172.21.15.6:0/2102866070 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd4000a3e30 msgr2=0x7fd4000a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.014+0000 7fd423200700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd4000a3e30 0x7fd4000a62f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009c60 tx=0x7fd40c009260 comp rx=0 tx=0).stop 2024-04-26T01:58:06.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.014+0000 7fd423200700 1 -- 172.21.15.6:0/2102866070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd41c075590 msgr2=0x7fd41c1a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.014+0000 7fd423200700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd41c075590 0x7fd41c1a5920 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd40400e9c0 tx=0x7fd40400ecd0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.014+0000 7fd423200700 1 -- 172.21.15.6:0/2102866070 shutdown_connections 2024-04-26T01:58:06.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.014+0000 7fd423200700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd4000a3e30 0x7fd4000a62f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.014+0000 7fd423200700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c113c70 0x7fd41c19f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.014+0000 7fd423200700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c074c70 0x7fd41c1a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.014+0000 7fd423200700 1 --2- 172.21.15.6:0/2102866070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd41c075590 0x7fd41c1a5920 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.014+0000 7fd423200700 1 -- 172.21.15.6:0/2102866070 >> 172.21.15.6:0/2102866070 conn(0x7fd41c079b90 msgr2=0x7fd41c112a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:06.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.014+0000 7fd423200700 1 -- 172.21.15.6:0/2102866070 shutdown_connections 2024-04-26T01:58:06.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:06.014+0000 7fd423200700 1 -- 172.21.15.6:0/2102866070 wait complete. 2024-04-26T01:58:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.281+0000 7fae92200640 1 -- 172.21.15.6:0/1618169618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae8c112c30 msgr2=0x7fae8c115020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.281+0000 7fae92200640 1 --2- 172.21.15.6:0/1618169618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae8c112c30 0x7fae8c115020 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fae84009890 tx=0x7fae8402a370 comp rx=0 tx=0).stop 2024-04-26T01:58:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.281+0000 7fae92200640 1 -- 172.21.15.6:0/1618169618 shutdown_connections 2024-04-26T01:58:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.281+0000 7fae92200640 1 --2- 172.21.15.6:0/1618169618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae8c112c30 0x7fae8c115020 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.281+0000 7fae92200640 1 --2- 172.21.15.6:0/1618169618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8c076da0 0x7fae8c1126f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.281+0000 7fae92200640 1 --2- 172.21.15.6:0/1618169618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae8c076480 0x7fae8c076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.281+0000 7fae92200640 1 -- 172.21.15.6:0/1618169618 >> 172.21.15.6:0/1618169618 conn(0x7fae8c0ffa60 msgr2=0x7fae8c101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.281+0000 7fae92200640 1 -- 172.21.15.6:0/1618169618 shutdown_connections 2024-04-26T01:58:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.282+0000 7fae92200640 1 -- 172.21.15.6:0/1618169618 wait complete. 2024-04-26T01:58:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.282+0000 7fae92200640 1 Processor -- start 2024-04-26T01:58:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.283+0000 7fae92200640 1 -- start start 2024-04-26T01:58:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.283+0000 7fae92200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8c076480 0x7fae8c19ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.283+0000 7fae92200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae8c076da0 0x7fae8c19f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.283+0000 7fae92200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae8c112c30 0x7fae8c1a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.283+0000 7fae92200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c118600 con 0x7fae8c076da0 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.283+0000 7fae92200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c118480 con 0x7fae8c076480 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.283+0000 7fae92200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c118780 con 0x7fae8c112c30 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.283+0000 7fae90e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8c076480 0x7fae8c19ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.283+0000 7fae90e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8c076480 0x7fae8c19ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55526/0 (socket says 172.21.15.6:55526) 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.283+0000 7fae90e00640 1 -- 172.21.15.6:0/2236543889 learned_addr learned my addr 172.21.15.6:0/2236543889 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.283+0000 7fae83e00640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae8c076da0 0x7fae8c19f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.283+0000 7fae91800640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae8c112c30 0x7fae8c1a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.284+0000 7fae90e00640 1 -- 172.21.15.6:0/2236543889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae8c112c30 msgr2=0x7fae8c1a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.284+0000 7fae90e00640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae8c112c30 0x7fae8c1a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.284+0000 7fae90e00640 1 -- 172.21.15.6:0/2236543889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae8c076da0 msgr2=0x7fae8c19f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.284+0000 7fae90e00640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae8c076da0 0x7fae8c19f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.284+0000 7fae90e00640 1 -- 172.21.15.6:0/2236543889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8c1a3e50 con 0x7fae8c076480 2024-04-26T01:58:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.284+0000 7fae90e00640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8c076480 0x7fae8c19ee60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00ebc0 tx=0x7fae7c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.284+0000 7fae8a000640 1 -- 172.21.15.6:0/2236543889 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c010630 con 0x7fae8c076480 2024-04-26T01:58:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.284+0000 7fae8a000640 1 -- 172.21.15.6:0/2236543889 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae7c010c80 con 0x7fae8c076480 2024-04-26T01:58:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.284+0000 7fae92200640 1 -- 172.21.15.6:0/2236543889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae8c1a4140 con 0x7fae8c076480 2024-04-26T01:58:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.284+0000 7fae8a000640 1 -- 172.21.15.6:0/2236543889 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c018b60 con 0x7fae8c076480 2024-04-26T01:58:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.284+0000 7fae92200640 1 -- 172.21.15.6:0/2236543889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae8c1ab9d0 con 0x7fae8c076480 2024-04-26T01:58:06.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.288+0000 7fae8a000640 1 -- 172.21.15.6:0/2236543889 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fae7c018d00 con 0x7fae8c076480 2024-04-26T01:58:06.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.288+0000 7fae96606740 1 Processor -- start 2024-04-26T01:58:06.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.288+0000 7fae96606740 1 -- start start 2024-04-26T01:58:06.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.288+0000 7fae96606740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f48f9b0 0x556d2f48fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.288+0000 7fae96606740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f4902d0 0x556d2f49a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.288+0000 7fae96606740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f49ab90 0x556d2f49cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.288+0000 7fae96606740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x556d2f3f5900 con 0x556d2f48f9b0 2024-04-26T01:58:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.288+0000 7fae96606740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556d2f39bef0 con 0x556d2f4902d0 2024-04-26T01:58:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.288+0000 7fae96606740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556d2f3f18c0 con 0x556d2f49ab90 2024-04-26T01:58:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.289+0000 7fae90e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f4902d0 0x556d2f49a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.289+0000 7fae91800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f48f9b0 0x556d2f48fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.289+0000 7fae90e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f4902d0 0x556d2f49a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55540/0 (socket says 172.21.15.6:55540) 2024-04-26T01:58:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.289+0000 7fae90e00640 1 -- 172.21.15.6:0/804332051 learned_addr learned my addr 172.21.15.6:0/804332051 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.289+0000 7fae83e00640 1 --2- 172.21.15.6:0/804332051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f49ab90 0x556d2f49cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.289+0000 7fae90e00640 1 -- 172.21.15.6:0/804332051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f49ab90 msgr2=0x556d2f49cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.289+0000 7fae90e00640 1 --2- 172.21.15.6:0/804332051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f49ab90 0x556d2f49cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.289+0000 7fae90e00640 1 -- 172.21.15.6:0/804332051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f48f9b0 msgr2=0x556d2f48fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.289+0000 7fae90e00640 1 --2- 172.21.15.6:0/804332051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f48f9b0 0x556d2f48fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.289+0000 7fae90e00640 1 -- 172.21.15.6:0/804332051 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d2f108210 con 0x556d2f4902d0 2024-04-26T01:58:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.289+0000 7fae83e00640 1 --2- 172.21.15.6:0/804332051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f49ab90 0x556d2f49cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.290+0000 7fae90e00640 1 --2- 172.21.15.6:0/804332051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f4902d0 0x556d2f49a650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fae7c0ce410 tx=0x7fae7c0d9570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ea35604703c9279 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.290+0000 7fae82000640 1 -- 172.21.15.6:0/804332051 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c0e4070 con 0x556d2f4902d0 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.290+0000 7fae82000640 1 -- 172.21.15.6:0/804332051 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae7c0dfab0 con 0x556d2f4902d0 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.290+0000 7fae8a000640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fae700a3d80 0x7fae700a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.290+0000 7fae82000640 1 -- 172.21.15.6:0/804332051 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c0dea30 con 0x556d2f4902d0 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.290+0000 7fae8a000640 1 -- 172.21.15.6:0/2236543889 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8644+0+0 (secure 0 0 0) 0x7fae7c0d0dd0 con 0x7fae8c076480 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.290+0000 7fae91800640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fae700a3d80 0x7fae700a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.291+0000 7fae96606740 1 -- 172.21.15.6:0/804332051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f4902d0 msgr2=0x556d2f49a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.291+0000 7fae96606740 1 --2- 172.21.15.6:0/804332051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f4902d0 0x556d2f49a650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fae7c0ce410 tx=0x7fae7c0d9570 comp rx=0 tx=0).stop 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.291+0000 7fae96606740 1 -- 172.21.15.6:0/804332051 shutdown_connections 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.291+0000 7fae96606740 1 --2- 172.21.15.6:0/804332051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f49ab90 0x556d2f49cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.291+0000 7fae96606740 1 --2- 172.21.15.6:0/804332051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f4902d0 0x556d2f49a650 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.291+0000 7fae96606740 1 --2- 172.21.15.6:0/804332051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f48f9b0 0x556d2f48fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.291+0000 7fae96606740 1 -- 172.21.15.6:0/804332051 >> 172.21.15.6:0/804332051 conn(0x556d2f48bec0 msgr2=0x556d2f48c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.291+0000 7fae91800640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fae700a3d80 0x7fae700a6240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae840303f0 tx=0x7fae84030340 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.291+0000 7fae96606740 1 -- 172.21.15.6:0/804332051 shutdown_connections 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.291+0000 7fae96606740 1 -- 172.21.15.6:0/804332051 wait complete. 2024-04-26T01:58:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.291+0000 7fae96606740 1 -- start start 2024-04-26T01:58:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.292+0000 7fae96606740 1 Processor -- start 2024-04-26T01:58:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.292+0000 7fae96606740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 0x556d2f497ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.292+0000 7fae96606740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f4902d0 0x556d2f498010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.292+0000 7fae96606740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 0x556d2f498550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.292+0000 7fae96606740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x556d2f49d7c0 con 0x556d2f4902d0 2024-04-26T01:58:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.292+0000 7fae96606740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556d2f3af310 con 0x556d2f49ab90 2024-04-26T01:58:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.292+0000 7fae96606740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556d2f49d940 con 0x556d2f48f9b0 2024-04-26T01:58:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.292+0000 7fae90e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 0x556d2f497ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.292+0000 7fae96606740 10 client.0 _ll_register_callbacks cb 0x556d2f37dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae90e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 0x556d2f497ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:57244/0 (socket says 172.21.15.6:57244) 2024-04-26T01:58:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae90e00640 1 -- 172.21.15.6:0/2225729908 learned_addr learned my addr 172.21.15.6:0/2225729908 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae83e00640 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f4902d0 0x556d2f498010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae91800640 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 0x556d2f498550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae90e00640 1 -- 172.21.15.6:0/2225729908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 msgr2=0x556d2f498550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae90e00640 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 0x556d2f498550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae90e00640 1 -- 172.21.15.6:0/2225729908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f4902d0 msgr2=0x556d2f498010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae90e00640 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f4902d0 0x556d2f498010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae90e00640 1 -- 172.21.15.6:0/2225729908 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d2f1d4620 con 0x556d2f48f9b0 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae90e00640 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 0x556d2f497ad0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fae7c099770 tx=0x7fae7c01f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47252a58956ad6a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae7aa00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae7aa00640 1 -- 172.21.15.6:0/2225729908 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c099040 con 0x556d2f48f9b0 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae7aa00640 1 -- 172.21.15.6:0/2225729908 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8644+0+0 (secure 0 0 0) 0x7fae7c0dfc60 con 0x556d2f48f9b0 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.293+0000 7fae96606740 1 -- 172.21.15.6:0/2225729908 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae7c0dfab0 con 0x556d2f48f9b0 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.294+0000 7fae7aa00640 1 -- 172.21.15.6:0/2225729908 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fae7c09c700 con 0x556d2f48f9b0 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.294+0000 7fae96606740 10 client.26674 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.294+0000 7fae96606740 1 -- 172.21.15.6:0/2225729908 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x556d2f49d7c0 con 0x556d2f48f9b0 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.294+0000 7fae7aa00640 1 -- 172.21.15.6:0/2225729908 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae7c0dfab0 con 0x556d2f48f9b0 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.294+0000 7fae7aa00640 1 -- 172.21.15.6:0/2225729908 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7fae7c0de3a0 con 0x556d2f48f9b0 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.294+0000 7fae96606740 10 client.26674 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.294+0000 7fae96606740 10 client.26674 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:58:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.294+0000 7fae96606740 20 client.26674 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.294+0000 7fae96606740 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556d2f4aab90 0x556d2f4acf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.294+0000 7fae96606740 4 client.26674 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.294+0000 7fae96606740 1 -- 172.21.15.6:0/2225729908 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d2f4098b0 con 0x556d2f4aab90 2024-04-26T01:58:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.294+0000 7fae83e00640 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556d2f4aab90 0x556d2f4acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.295+0000 7fae83e00640 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556d2f4aab90 0x556d2f4acf80 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd2752084c5be5c4 server_cookie=36bd31eba33cac05 in_seq=0 out_seq=0 2024-04-26T01:58:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.295+0000 7fae7aa00640 10 client.26674 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.296+0000 7fae7aa00640 1 -- 172.21.15.6:0/2225729908 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556d2f4098b0 con 0x556d2f4aab90 2024-04-26T01:58:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae7aa00640 10 client.26674 handle_command_reply: tid=0 2024-04-26T01:58:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae96606740 1 client.26674 shutdown 2024-04-26T01:58:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae96606740 1 -- 172.21.15.6:0/2225729908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 msgr2=0x556d2f497ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae96606740 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 0x556d2f497ad0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fae7c099770 tx=0x7fae7c01f000 comp rx=0 tx=0).stop 2024-04-26T01:58:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae96606740 1 -- 172.21.15.6:0/2225729908 shutdown_connections 2024-04-26T01:58:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae96606740 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556d2f4aab90 0x556d2f4acf80 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae96606740 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f4902d0 0x556d2f498010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae96606740 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 0x556d2f498550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae7aa00640 0 client.26674 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae96606740 1 --2- 172.21.15.6:0/2225729908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 0x556d2f497ad0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae96606740 1 -- 172.21.15.6:0/2225729908 >> 172.21.15.6:0/2225729908 conn(0x556d2f48bec0 msgr2=0x556d2f491c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae96606740 1 -- 172.21.15.6:0/2225729908 shutdown_connections 2024-04-26T01:58:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae96606740 1 -- 172.21.15.6:0/2225729908 wait complete. 2024-04-26T01:58:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.297+0000 7fae96606740 20 client.26674 trim_cache size 0 max 16384 2024-04-26T01:58:06.313 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.314+0000 7fae96606740 1 Processor -- start 2024-04-26T01:58:06.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.315+0000 7fae96606740 1 -- start start 2024-04-26T01:58:06.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.315+0000 7fae96606740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 0x556d2f4a6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.315+0000 7fae96606740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f4902d0 0x556d2f492370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.315+0000 7fae96606740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 0x556d2f48aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.315+0000 7fae96606740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c1b6bb0 con 0x556d2f4902d0 2024-04-26T01:58:06.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.315+0000 7fae96606740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c1b4480 con 0x556d2f49ab90 2024-04-26T01:58:06.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.315+0000 7fae96606740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c1b22f0 con 0x556d2f48f9b0 2024-04-26T01:58:06.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.315+0000 7fae90e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 0x556d2f4a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.315+0000 7fae83e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f4902d0 0x556d2f492370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.315+0000 7fae90e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 0x556d2f4a6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:57250/0 (socket says 172.21.15.6:57250) 2024-04-26T01:58:06.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.315+0000 7fae90e00640 1 -- 172.21.15.6:0/4266289157 learned_addr learned my addr 172.21.15.6:0/4266289157 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.316+0000 7fae91800640 1 --2- 172.21.15.6:0/4266289157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 0x556d2f48aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.316+0000 7fae90e00640 1 -- 172.21.15.6:0/4266289157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 msgr2=0x556d2f48aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.316+0000 7fae90e00640 1 --2- 172.21.15.6:0/4266289157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 0x556d2f48aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.316+0000 7fae90e00640 1 -- 172.21.15.6:0/4266289157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f4902d0 msgr2=0x556d2f492370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.316+0000 7fae90e00640 1 --2- 172.21.15.6:0/4266289157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f4902d0 0x556d2f492370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.316+0000 7fae90e00640 1 -- 172.21.15.6:0/4266289157 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d2f4ad3e0 con 0x556d2f48f9b0 2024-04-26T01:58:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.316+0000 7fae83e00640 1 --2- 172.21.15.6:0/4266289157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f4902d0 0x556d2f492370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.316+0000 7fae90e00640 1 --2- 172.21.15.6:0/4266289157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 0x556d2f4a6680 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fae7c0d43a0 tx=0x7fae7c09b2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=930e7d686c5f1c18 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.316+0000 7fae81600640 1 -- 172.21.15.6:0/4266289157 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c09c8e0 con 0x556d2f48f9b0 2024-04-26T01:58:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.316+0000 7fae81600640 1 -- 172.21.15.6:0/4266289157 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae7c0a10a0 con 0x556d2f48f9b0 2024-04-26T01:58:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.316+0000 7fae81600640 1 -- 172.21.15.6:0/4266289157 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c09cb70 con 0x556d2f48f9b0 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.317+0000 7fae96606740 1 -- 172.21.15.6:0/4266289157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 msgr2=0x556d2f4a6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.317+0000 7fae96606740 1 --2- 172.21.15.6:0/4266289157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 0x556d2f4a6680 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fae7c0d43a0 tx=0x7fae7c09b2b0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.317+0000 7fae96606740 1 -- 172.21.15.6:0/4266289157 shutdown_connections 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.317+0000 7fae96606740 1 --2- 172.21.15.6:0/4266289157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f4902d0 0x556d2f492370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.317+0000 7fae96606740 1 --2- 172.21.15.6:0/4266289157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 0x556d2f48aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.317+0000 7fae96606740 1 --2- 172.21.15.6:0/4266289157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f48f9b0 0x556d2f4a6680 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.317+0000 7fae96606740 1 -- 172.21.15.6:0/4266289157 >> 172.21.15.6:0/4266289157 conn(0x556d2f48bec0 msgr2=0x556d2f48a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.317+0000 7fae96606740 1 -- 172.21.15.6:0/4266289157 shutdown_connections 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.317+0000 7fae96606740 1 -- 172.21.15.6:0/4266289157 wait complete. 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.318+0000 7fae96606740 1 -- start start 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.318+0000 7fae96606740 1 Processor -- start 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.318+0000 7fae96606740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f48f9b0 0x556d2f4a2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.318+0000 7fae96606740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f4902d0 0x556d2f4a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.318+0000 7fae96606740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 0x556d2f4a0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.318+0000 7fae96606740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c1b2940 con 0x556d2f48f9b0 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.318+0000 7fae96606740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c1b3db0 con 0x556d2f49ab90 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.318+0000 7fae96606740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c1b3010 con 0x556d2f4902d0 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.318+0000 7fae96606740 10 client.0 _ll_register_callbacks cb 0x7fae74009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.319+0000 7fae83e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f4902d0 0x556d2f4a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.319+0000 7fae83e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f4902d0 0x556d2f4a02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:57258/0 (socket says 172.21.15.6:57258) 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.319+0000 7fae83e00640 1 -- 172.21.15.6:0/1681856450 learned_addr learned my addr 172.21.15.6:0/1681856450 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.319+0000 7fae91800640 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 0x556d2f4a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.319+0000 7fae90e00640 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f48f9b0 0x556d2f4a2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.319+0000 7fae83e00640 1 -- 172.21.15.6:0/1681856450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 msgr2=0x556d2f4a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.319+0000 7fae83e00640 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 0x556d2f4a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.319+0000 7fae83e00640 1 -- 172.21.15.6:0/1681856450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f48f9b0 msgr2=0x556d2f4a2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.319+0000 7fae83e00640 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f48f9b0 0x556d2f4a2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.319+0000 7fae83e00640 1 -- 172.21.15.6:0/1681856450 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d2f4a1010 con 0x556d2f4902d0 2024-04-26T01:58:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.319+0000 7fae83e00640 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f4902d0 0x556d2f4a02f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fae8c114bb0 tx=0x7fae740108c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c9c6ce790f3fb21 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.319+0000 7fae7aa00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:58:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae7aa00640 1 -- 172.21.15.6:0/1681856450 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74023070 con 0x556d2f4902d0 2024-04-26T01:58:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae7aa00640 1 -- 172.21.15.6:0/1681856450 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8644+0+0 (secure 0 0 0) 0x7fae7401f8d0 con 0x556d2f4902d0 2024-04-26T01:58:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae96606740 1 -- 172.21.15.6:0/1681856450 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae7c0a10a0 con 0x556d2f4902d0 2024-04-26T01:58:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae7aa00640 1 -- 172.21.15.6:0/1681856450 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fae74005020 con 0x556d2f4902d0 2024-04-26T01:58:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae96606740 10 client.26680 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae96606740 1 -- 172.21.15.6:0/1681856450 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x556d2f510f10 con 0x556d2f4902d0 2024-04-26T01:58:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae7aa00640 1 -- 172.21.15.6:0/1681856450 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae7c0a10a0 con 0x556d2f4902d0 2024-04-26T01:58:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae7aa00640 1 -- 172.21.15.6:0/1681856450 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7fae7402e070 con 0x556d2f4902d0 2024-04-26T01:58:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae96606740 10 client.26680 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae96606740 10 client.26680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:58:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae96606740 20 client.26680 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae96606740 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556d2f511670 0x556d2f513a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae96606740 4 client.26680 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:58:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.320+0000 7fae96606740 1 -- 172.21.15.6:0/1681856450 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556d2f403320 con 0x556d2f511670 2024-04-26T01:58:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.321+0000 7fae90e00640 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556d2f511670 0x556d2f513a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.321+0000 7fae90e00640 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556d2f511670 0x556d2f513a90 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1adaffd7543997fe server_cookie=cd56294bf20d3715 in_seq=0 out_seq=0 2024-04-26T01:58:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.321+0000 7fae7aa00640 10 client.26680 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:06.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.355+0000 7fae7aa00640 1 -- 172.21.15.6:0/1681856450 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556d2f403320 con 0x556d2f511670 2024-04-26T01:58:06.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae7aa00640 10 client.26680 handle_command_reply: tid=0 2024-04-26T01:58:06.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae96606740 1 client.26680 shutdown 2024-04-26T01:58:06.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae96606740 1 -- 172.21.15.6:0/1681856450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f4902d0 msgr2=0x556d2f4a02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae96606740 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f4902d0 0x556d2f4a02f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fae8c114bb0 tx=0x7fae740108c0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae96606740 1 -- 172.21.15.6:0/1681856450 shutdown_connections 2024-04-26T01:58:06.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae96606740 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556d2f511670 0x556d2f513a90 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae96606740 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d2f49ab90 0x556d2f4a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae96606740 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556d2f4902d0 0x556d2f4a02f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae96606740 1 --2- 172.21.15.6:0/1681856450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556d2f48f9b0 0x556d2f4a2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae96606740 1 -- 172.21.15.6:0/1681856450 >> 172.21.15.6:0/1681856450 conn(0x556d2f48bec0 msgr2=0x556d2f48b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae7aa00640 0 client.26680 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae96606740 1 -- 172.21.15.6:0/1681856450 shutdown_connections 2024-04-26T01:58:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae96606740 1 -- 172.21.15.6:0/1681856450 wait complete. 2024-04-26T01:58:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.356+0000 7fae96606740 20 client.26680 trim_cache size 0 max 16384 2024-04-26T01:58:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:58:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:58:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:58:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:58:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.357+0000 7fae92200640 1 -- 172.21.15.6:0/2236543889 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fae700a3d80 msgr2=0x7fae700a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.357+0000 7fae92200640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fae700a3d80 0x7fae700a6240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae840303f0 tx=0x7fae84030340 comp rx=0 tx=0).stop 2024-04-26T01:58:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.357+0000 7fae92200640 1 -- 172.21.15.6:0/2236543889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8c076480 msgr2=0x7fae8c19ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.357+0000 7fae92200640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8c076480 0x7fae8c19ee60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00ebc0 tx=0x7fae7c00c6a0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.357+0000 7fae92200640 1 -- 172.21.15.6:0/2236543889 shutdown_connections 2024-04-26T01:58:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.357+0000 7fae92200640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fae700a3d80 0x7fae700a6240 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.357+0000 7fae92200640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae8c076da0 0x7fae8c19f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.357+0000 7fae92200640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae8c112c30 0x7fae8c1a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.357+0000 7fae92200640 1 --2- 172.21.15.6:0/2236543889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8c076480 0x7fae8c19ee60 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.357+0000 7fae92200640 1 -- 172.21.15.6:0/2236543889 >> 172.21.15.6:0/2236543889 conn(0x7fae8c0ffa60 msgr2=0x7fae8c101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.357+0000 7fae92200640 1 -- 172.21.15.6:0/2236543889 shutdown_connections 2024-04-26T01:58:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:06.357+0000 7fae92200640 1 -- 172.21.15.6:0/2236543889 wait complete. 2024-04-26T01:58:06.634 INFO:tasks.workunit.client.0.smithi006.stdout: 1 428459 3.55 MB/sec execute 510 sec latency 27037.117 ms 2024-04-26T01:58:06.760 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:06 smithi006 ceph-mon[28419]: pgmap v385: 97 pgs: 97 active+clean; 130 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.1 MiB/s wr, 65 op/s 2024-04-26T01:58:06.835 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '8cb585fa', seconds elapsed: 35.1 2024-04-26T01:58:06.835 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.25 2024-04-26T01:58:06.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:58:06.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:06 smithi188 ceph-mon[32751]: pgmap v385: 97 pgs: 97 active+clean; 130 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.1 MiB/s wr, 65 op/s 2024-04-26T01:58:06.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:06 smithi029 ceph-mon[32775]: pgmap v385: 97 pgs: 97 active+clean; 130 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.1 MiB/s wr, 65 op/s 2024-04-26T01:58:07.335 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:07.634 INFO:tasks.workunit.client.0.smithi006.stdout: 1 429560 3.55 MB/sec execute 511 sec latency 51.565 ms 2024-04-26T01:58:08.634 INFO:tasks.workunit.client.0.smithi006.stdout: 1 430378 3.55 MB/sec execute 512 sec latency 32.859 ms 2024-04-26T01:58:08.767 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:08 smithi006 ceph-mon[28419]: pgmap v386: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1002 KiB/s wr, 65 op/s 2024-04-26T01:58:08.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:08 smithi188 ceph-mon[32751]: pgmap v386: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1002 KiB/s wr, 65 op/s 2024-04-26T01:58:08.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:08 smithi029 ceph-mon[32775]: pgmap v386: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1002 KiB/s wr, 65 op/s 2024-04-26T01:58:09.635 INFO:tasks.workunit.client.0.smithi006.stdout: 1 431783 3.56 MB/sec execute 513 sec latency 20.385 ms 2024-04-26T01:58:09.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.698+0000 7f4d96e00700 1 -- 172.21.15.6:0/751948851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d9006a750 msgr2=0x7f4d9006ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:09.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.698+0000 7f4d96e00700 1 --2- 172.21.15.6:0/751948851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d9006a750 0x7f4d9006ab30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000b0a0 tx=0x7f4d8000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:58:09.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.698+0000 7f4d96e00700 1 -- 172.21.15.6:0/751948851 shutdown_connections 2024-04-26T01:58:09.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.698+0000 7f4d96e00700 1 --2- 172.21.15.6:0/751948851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d9010f4f0 0x7f4d901118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:09.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.698+0000 7f4d96e00700 1 --2- 172.21.15.6:0/751948851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d9006a750 0x7f4d9006ab30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:09.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.698+0000 7f4d96e00700 1 --2- 172.21.15.6:0/751948851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d9006b070 0x7f4d9010efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:09.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.698+0000 7f4d96e00700 1 -- 172.21.15.6:0/751948851 >> 172.21.15.6:0/751948851 conn(0x7f4d900788f0 msgr2=0x7f4d90078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:09.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.698+0000 7f4d96e00700 1 -- 172.21.15.6:0/751948851 shutdown_connections 2024-04-26T01:58:09.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.699+0000 7f4d96e00700 1 -- 172.21.15.6:0/751948851 wait complete. 2024-04-26T01:58:09.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.700+0000 7f4d96e00700 1 Processor -- start 2024-04-26T01:58:09.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.701+0000 7f4d96e00700 1 -- start start 2024-04-26T01:58:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.701+0000 7f4d96e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d9006a750 0x7f4d9019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.701+0000 7f4d96e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d9006b070 0x7f4d9019b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.701+0000 7f4d96e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d9010f4f0 0x7f4d9019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.701+0000 7f4d96e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4d90114e20 con 0x7f4d9010f4f0 2024-04-26T01:58:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.701+0000 7f4d96e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d90114ca0 con 0x7f4d9006b070 2024-04-26T01:58:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.701+0000 7f4d96e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4d90114fa0 con 0x7f4d9006a750 2024-04-26T01:58:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.702+0000 7f4d8fe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d9006a750 0x7f4d9019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.702+0000 7f4d8fe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d9006a750 0x7f4d9019b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58390/0 (socket says 172.21.15.6:58390) 2024-04-26T01:58:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.702+0000 7f4d8f400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d9006b070 0x7f4d9019b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.702+0000 7f4d8fe00700 1 -- 172.21.15.6:0/3221140012 learned_addr learned my addr 172.21.15.6:0/3221140012 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.702+0000 7f4d94e00700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d9010f4f0 0x7f4d9019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:09.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.702+0000 7f4d8fe00700 1 -- 172.21.15.6:0/3221140012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d9006b070 msgr2=0x7f4d9019b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:09.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.702+0000 7f4d8fe00700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d9006b070 0x7f4d9019b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:09.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.702+0000 7f4d8fe00700 1 -- 172.21.15.6:0/3221140012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d9010f4f0 msgr2=0x7f4d9019fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:09.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.702+0000 7f4d8fe00700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d9010f4f0 0x7f4d9019fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:09.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.702+0000 7f4d8fe00700 1 -- 172.21.15.6:0/3221140012 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d901a0320 con 0x7f4d9006a750 2024-04-26T01:58:09.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.703+0000 7f4d8fe00700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d9006a750 0x7f4d9019b310 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000b070 tx=0x7f4d80004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:09.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.703+0000 7f4d8cc00700 1 -- 172.21.15.6:0/3221140012 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80021070 con 0x7f4d9006a750 2024-04-26T01:58:09.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.703+0000 7f4d8cc00700 1 -- 172.21.15.6:0/3221140012 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d80007e30 con 0x7f4d9006a750 2024-04-26T01:58:09.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.703+0000 7f4d8cc00700 1 -- 172.21.15.6:0/3221140012 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80004730 con 0x7f4d9006a750 2024-04-26T01:58:09.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.703+0000 7f4d96e00700 1 -- 172.21.15.6:0/3221140012 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d901a05b0 con 0x7f4d9006a750 2024-04-26T01:58:09.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.703+0000 7f4d96e00700 1 -- 172.21.15.6:0/3221140012 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d901a7e90 con 0x7f4d9006a750 2024-04-26T01:58:09.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.704+0000 7f4d96e00700 1 -- 172.21.15.6:0/3221140012 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d90101f60 con 0x7f4d9006a750 2024-04-26T01:58:09.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.706+0000 7f4d8cc00700 1 -- 172.21.15.6:0/3221140012 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f4d80007980 con 0x7f4d9006a750 2024-04-26T01:58:09.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.707+0000 7f4d8cc00700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4d740a3db0 0x7f4d740a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:09.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.708+0000 7f4d8cc00700 1 -- 172.21.15.6:0/3221140012 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8644+0+0 (secure 0 0 0) 0x7f4d8000f020 con 0x7f4d9006a750 2024-04-26T01:58:09.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.708+0000 7f4d8f400700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4d740a3db0 0x7f4d740a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:09.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.709+0000 7f4d8f400700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4d740a3db0 0x7f4d740a6270 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4d90102f00 tx=0x7f4d7c009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:09.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:09.713+0000 7f4d8cc00700 1 -- 172.21.15.6:0/3221140012 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4d8002a020 con 0x7f4d9006a750 2024-04-26T01:58:10.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.020+0000 7f4d96e00700 1 -- 172.21.15.6:0/3221140012 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d9002df80 con 0x7f4d9006a750 2024-04-26T01:58:10.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.021+0000 7f4d8cc00700 1 -- 172.21.15.6:0/3221140012 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11850 (secure 0 0 0) 0x7f4d80004000 con 0x7f4d9006a750 2024-04-26T01:58:10.020 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:58:10.020 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":43,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:55:18.912671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:58:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.023+0000 7f4d96e00700 1 -- 172.21.15.6:0/3221140012 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4d740a3db0 msgr2=0x7f4d740a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.023+0000 7f4d96e00700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4d740a3db0 0x7f4d740a6270 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4d90102f00 tx=0x7f4d7c009260 comp rx=0 tx=0).stop 2024-04-26T01:58:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.024+0000 7f4d96e00700 1 -- 172.21.15.6:0/3221140012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d9006a750 msgr2=0x7f4d9019b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.024+0000 7f4d96e00700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d9006a750 0x7f4d9019b310 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000b070 tx=0x7f4d80004d00 comp rx=0 tx=0).stop 2024-04-26T01:58:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.024+0000 7f4d96e00700 1 -- 172.21.15.6:0/3221140012 shutdown_connections 2024-04-26T01:58:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.024+0000 7f4d96e00700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4d740a3db0 0x7f4d740a6270 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.024+0000 7f4d96e00700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d9010f4f0 0x7f4d9019fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.024+0000 7f4d96e00700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d9006a750 0x7f4d9019b310 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.024+0000 7f4d96e00700 1 --2- 172.21.15.6:0/3221140012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d9006b070 0x7f4d9019b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:10.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.024+0000 7f4d96e00700 1 -- 172.21.15.6:0/3221140012 >> 172.21.15.6:0/3221140012 conn(0x7f4d900788f0 msgr2=0x7f4d901109a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:10.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.024+0000 7f4d96e00700 1 -- 172.21.15.6:0/3221140012 shutdown_connections 2024-04-26T01:58:10.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:10.024+0000 7f4d96e00700 1 -- 172.21.15.6:0/3221140012 wait complete. 2024-04-26T01:58:10.025 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-26T01:58:10.634 INFO:tasks.workunit.client.0.smithi006.stdout: 1 433092 3.57 MB/sec execute 514 sec latency 19.446 ms 2024-04-26T01:58:10.809 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:10 smithi006 ceph-mon[28419]: pgmap v387: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1002 KiB/s wr, 65 op/s 2024-04-26T01:58:10.809 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:10 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3221140012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:10.809 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:10 smithi006 ceph-mon[28419]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T01:58:10.878 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 79.25 --await-for 42.0' 2024-04-26T01:58:10.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 79.25 --await-for 42.0 2024-04-26T01:58:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:10 smithi188 ceph-mon[32751]: pgmap v387: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1002 KiB/s wr, 65 op/s 2024-04-26T01:58:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:10 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3221140012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:10 smithi188 ceph-mon[32751]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T01:58:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:10 smithi029 ceph-mon[32775]: pgmap v387: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1002 KiB/s wr, 65 op/s 2024-04-26T01:58:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:10 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3221140012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:10 smithi029 ceph-mon[32775]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T01:58:11.373 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:11.635 INFO:tasks.workunit.client.0.smithi006.stdout: 1 434016 3.57 MB/sec execute 515 sec latency 26.446 ms 2024-04-26T01:58:12.635 INFO:tasks.workunit.client.0.smithi006.stdout: 1 435115 3.57 MB/sec execute 516 sec latency 71.195 ms 2024-04-26T01:58:12.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:12 smithi006 ceph-mon[28419]: pgmap v389: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 161 op/s 2024-04-26T01:58:12.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:12 smithi188 ceph-mon[32751]: pgmap v389: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 161 op/s 2024-04-26T01:58:12.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:12 smithi029 ceph-mon[32775]: pgmap v389: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 161 op/s 2024-04-26T01:58:13.635 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436525 3.59 MB/sec execute 517 sec latency 23.075 ms 2024-04-26T01:58:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.728+0000 7f7973200700 1 -- 172.21.15.6:0/122328721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f796c108a30 msgr2=0x7f796c108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.728+0000 7f7973200700 1 --2- 172.21.15.6:0/122328721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f796c108a30 0x7f796c108e10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f795c009990 tx=0x7f795c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.728+0000 7f7973200700 1 -- 172.21.15.6:0/122328721 shutdown_connections 2024-04-26T01:58:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.728+0000 7f7973200700 1 --2- 172.21.15.6:0/122328721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796c1034b0 0x7f796c10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.728+0000 7f7973200700 1 --2- 172.21.15.6:0/122328721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f796c108a30 0x7f796c108e10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.728+0000 7f7973200700 1 --2- 172.21.15.6:0/122328721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c102af0 0x7f796c102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.728+0000 7f7973200700 1 -- 172.21.15.6:0/122328721 >> 172.21.15.6:0/122328721 conn(0x7f796c079b10 msgr2=0x7f796c079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.728+0000 7f7973200700 1 -- 172.21.15.6:0/122328721 shutdown_connections 2024-04-26T01:58:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.728+0000 7f7973200700 1 -- 172.21.15.6:0/122328721 wait complete. 2024-04-26T01:58:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.729+0000 7f7973200700 1 Processor -- start 2024-04-26T01:58:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.729+0000 7f7973200700 1 -- start start 2024-04-26T01:58:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f7973200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c102af0 0x7f796c074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f7973200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796c1034b0 0x7f796c075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f7973200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f796c108a30 0x7f796c0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f7973200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f796c112c50 con 0x7f796c1034b0 2024-04-26T01:58:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f7973200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f796c112ad0 con 0x7f796c102af0 2024-04-26T01:58:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f7973200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f796c112dd0 con 0x7f796c108a30 2024-04-26T01:58:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f796be00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c102af0 0x7f796c074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f7963e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796c1034b0 0x7f796c075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f7971200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f796c108a30 0x7f796c0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f796be00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c102af0 0x7f796c074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47326/0 (socket says 172.21.15.6:47326) 2024-04-26T01:58:13.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f796be00700 1 -- 172.21.15.6:0/1765322056 learned_addr learned my addr 172.21.15.6:0/1765322056 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:13.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f796be00700 1 -- 172.21.15.6:0/1765322056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f796c108a30 msgr2=0x7f796c0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f796be00700 1 --2- 172.21.15.6:0/1765322056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f796c108a30 0x7f796c0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f796be00700 1 -- 172.21.15.6:0/1765322056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796c1034b0 msgr2=0x7f796c075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f796be00700 1 --2- 172.21.15.6:0/1765322056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796c1034b0 0x7f796c075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.730+0000 7f796be00700 1 -- 172.21.15.6:0/1765322056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796c1a7570 con 0x7f796c102af0 2024-04-26T01:58:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.731+0000 7f796be00700 1 --2- 172.21.15.6:0/1765322056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c102af0 0x7f796c074d40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f795c005910 tx=0x7f795c00bfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.731+0000 7f7969600700 1 -- 172.21.15.6:0/1765322056 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c027070 con 0x7f796c102af0 2024-04-26T01:58:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.731+0000 7f7973200700 1 -- 172.21.15.6:0/1765322056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f796c1a7800 con 0x7f796c102af0 2024-04-26T01:58:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.731+0000 7f7969600700 1 -- 172.21.15.6:0/1765322056 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f795c00b610 con 0x7f796c102af0 2024-04-26T01:58:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.731+0000 7f7969600700 1 -- 172.21.15.6:0/1765322056 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c0173f0 con 0x7f796c102af0 2024-04-26T01:58:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.731+0000 7f7973200700 1 -- 172.21.15.6:0/1765322056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f796c1a7cb0 con 0x7f796c102af0 2024-04-26T01:58:13.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.735+0000 7f7969600700 1 -- 172.21.15.6:0/1765322056 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f795c02a070 con 0x7f796c102af0 2024-04-26T01:58:13.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.736+0000 7f797daddb80 1 Processor -- start 2024-04-26T01:58:13.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.736+0000 7f7969600700 1 --2- 172.21.15.6:0/1765322056 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f79500a3ed0 0x7f79500a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.737+0000 7f797daddb80 1 -- start start 2024-04-26T01:58:13.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.737+0000 7f7969600700 1 -- 172.21.15.6:0/1765322056 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f795c0d15f0 con 0x7f796c102af0 2024-04-26T01:58:13.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.737+0000 7f7971200700 1 --2- 172.21.15.6:0/1765322056 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f79500a3ed0 0x7f79500a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.737+0000 7f797daddb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c144bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.737+0000 7f797daddb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 0x55d3c1456c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.737+0000 7f797daddb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c1459570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.737+0000 7f797daddb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d3c139f020 con 0x55d3c144b9b0 2024-04-26T01:58:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.737+0000 7f797daddb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d3c13458c0 con 0x55d3c144c2d0 2024-04-26T01:58:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.737+0000 7f797daddb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d3c139afe0 con 0x55d3c1457180 2024-04-26T01:58:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.737+0000 7f7963e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 0x55d3c1456c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.737+0000 7f7971200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c1459570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.738+0000 7f796be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c144bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.738+0000 7f7963e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 0x55d3c1456c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47332/0 (socket says 172.21.15.6:47332) 2024-04-26T01:58:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.738+0000 7f7971200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c1459570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58430/0 (socket says 172.21.15.6:58430) 2024-04-26T01:58:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.738+0000 7f7963e00700 1 -- 172.21.15.6:0/3614216752 learned_addr learned my addr 172.21.15.6:0/3614216752 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.738+0000 7f7963e00700 1 -- 172.21.15.6:0/3614216752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 msgr2=0x55d3c1459570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.738+0000 7f7963e00700 1 --2- 172.21.15.6:0/3614216752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c1459570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.738+0000 7f7963e00700 1 -- 172.21.15.6:0/3614216752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 msgr2=0x55d3c144bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.738+0000 7f7963e00700 1 --2- 172.21.15.6:0/3614216752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c144bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.738+0000 7f7963e00700 1 -- 172.21.15.6:0/3614216752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3c133b810 con 0x55d3c144c2d0 2024-04-26T01:58:13.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.738+0000 7f7971200700 1 --2- 172.21.15.6:0/1765322056 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f79500a3ed0 0x7f79500a6390 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f796400f870 tx=0x7f796400fc30 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:13.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.738+0000 7f7963e00700 1 --2- 172.21.15.6:0/3614216752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 0x55d3c1456c40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7954009cf0 tx=0x7f7954011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=522b6188737465d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:13.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.739+0000 7f7961600700 1 -- 172.21.15.6:0/3614216752 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954011d00 con 0x55d3c144c2d0 2024-04-26T01:58:13.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.739+0000 7f7961600700 1 -- 172.21.15.6:0/3614216752 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7954004500 con 0x55d3c144c2d0 2024-04-26T01:58:13.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.739+0000 7f7961600700 1 -- 172.21.15.6:0/3614216752 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954005280 con 0x55d3c144c2d0 2024-04-26T01:58:13.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.740+0000 7f797daddb80 1 -- 172.21.15.6:0/3614216752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 msgr2=0x55d3c1456c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.740+0000 7f797daddb80 1 --2- 172.21.15.6:0/3614216752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 0x55d3c1456c40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7954009cf0 tx=0x7f7954011440 comp rx=0 tx=0).stop 2024-04-26T01:58:13.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.740+0000 7f797daddb80 1 -- 172.21.15.6:0/3614216752 shutdown_connections 2024-04-26T01:58:13.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.740+0000 7f797daddb80 1 --2- 172.21.15.6:0/3614216752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c1459570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.740+0000 7f797daddb80 1 --2- 172.21.15.6:0/3614216752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c144bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.740+0000 7f797daddb80 1 --2- 172.21.15.6:0/3614216752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 0x55d3c1456c40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.740+0000 7f797daddb80 1 -- 172.21.15.6:0/3614216752 >> 172.21.15.6:0/3614216752 conn(0x55d3c137a730 msgr2=0x55d3c1447e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:13.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.740+0000 7f797daddb80 1 -- 172.21.15.6:0/3614216752 shutdown_connections 2024-04-26T01:58:13.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.740+0000 7f797daddb80 1 -- 172.21.15.6:0/3614216752 wait complete. 2024-04-26T01:58:13.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.740+0000 7f797daddb80 1 -- start start 2024-04-26T01:58:13.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.741+0000 7f797daddb80 1 Processor -- start 2024-04-26T01:58:13.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.741+0000 7f797daddb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144b9b0 0x55d3c1456820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.741+0000 7f797daddb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144c2d0 0x55d3c1453b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.741+0000 7f797daddb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c1454060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.741+0000 7f797daddb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d3c1459ca0 con 0x55d3c144c2d0 2024-04-26T01:58:13.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.741+0000 7f797daddb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d3c13589f0 con 0x55d3c144b9b0 2024-04-26T01:58:13.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.741+0000 7f797daddb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d3c1459e20 con 0x55d3c1457180 2024-04-26T01:58:13.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.741+0000 7f797daddb80 10 client.0 _ll_register_callbacks cb 0x55d3c13378b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:13.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f7971200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c1454060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f7963e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144c2d0 0x55d3c1453b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f7971200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c1454060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58438/0 (socket says 172.21.15.6:58438) 2024-04-26T01:58:13.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f7971200700 1 -- 172.21.15.6:0/618392295 learned_addr learned my addr 172.21.15.6:0/618392295 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:13.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f796be00700 1 --2- 172.21.15.6:0/618392295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144b9b0 0x55d3c1456820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f7963e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144c2d0 0x55d3c1453b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48982/0 (socket says 172.21.15.6:48982) 2024-04-26T01:58:13.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f7971200700 1 -- 172.21.15.6:0/618392295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144b9b0 msgr2=0x55d3c1456820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f7971200700 1 --2- 172.21.15.6:0/618392295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144b9b0 0x55d3c1456820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f7971200700 1 -- 172.21.15.6:0/618392295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144c2d0 msgr2=0x55d3c1453b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f7971200700 1 --2- 172.21.15.6:0/618392295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144c2d0 0x55d3c1453b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f7971200700 1 -- 172.21.15.6:0/618392295 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3c12eca60 con 0x55d3c1457180 2024-04-26T01:58:13.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f7971200700 1 --2- 172.21.15.6:0/618392295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c1454060 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f796400f650 tx=0x7f7964004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7458e6181f728a86 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:13.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.742+0000 7f795a000700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:58:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.743+0000 7f795a000700 1 -- 172.21.15.6:0/618392295 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7964027030 con 0x55d3c1457180 2024-04-26T01:58:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.743+0000 7f795a000700 1 -- 172.21.15.6:0/618392295 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f7964008b80 con 0x55d3c1457180 2024-04-26T01:58:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.743+0000 7f797daddb80 1 -- 172.21.15.6:0/618392295 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7954004500 con 0x55d3c1457180 2024-04-26T01:58:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.743+0000 7f795a000700 1 -- 172.21.15.6:0/618392295 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f7964031050 con 0x55d3c1457180 2024-04-26T01:58:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.743+0000 7f797daddb80 10 client.26698 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.743+0000 7f797daddb80 1 -- 172.21.15.6:0/618392295 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55d3c1335c80 con 0x55d3c1457180 2024-04-26T01:58:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.743+0000 7f795a000700 1 -- 172.21.15.6:0/618392295 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7954004500 con 0x55d3c1457180 2024-04-26T01:58:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.744+0000 7f795a000700 1 -- 172.21.15.6:0/618392295 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f7964023420 con 0x55d3c1457180 2024-04-26T01:58:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.744+0000 7f797daddb80 10 client.26698 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.744+0000 7f797daddb80 10 client.26698 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:58:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.744+0000 7f797daddb80 20 client.26698 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.744+0000 7f797daddb80 1 --2- 172.21.15.6:0/618392295 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d3c1466940 0x55d3c1468d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.744+0000 7f797daddb80 4 client.26698 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.744+0000 7f797daddb80 1 -- 172.21.15.6:0/618392295 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3c13ac810 con 0x55d3c1466940 2024-04-26T01:58:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.744+0000 7f796be00700 1 --2- 172.21.15.6:0/618392295 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d3c1466940 0x55d3c1468d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.745+0000 7f796be00700 1 --2- 172.21.15.6:0/618392295 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d3c1466940 0x55d3c1468d30 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79dc735888c16653 server_cookie=35d9b71c79a33e46 in_seq=0 out_seq=0 2024-04-26T01:58:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.745+0000 7f795a000700 10 client.26698 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.745+0000 7f795a000700 1 -- 172.21.15.6:0/618392295 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d3c13ac810 con 0x55d3c1466940 2024-04-26T01:58:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.745+0000 7f795a000700 10 client.26698 handle_command_reply: tid=0 2024-04-26T01:58:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.745+0000 7f797daddb80 1 client.26698 shutdown 2024-04-26T01:58:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.746+0000 7f797daddb80 1 -- 172.21.15.6:0/618392295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 msgr2=0x55d3c1454060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.746+0000 7f797daddb80 1 --2- 172.21.15.6:0/618392295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c1454060 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f796400f650 tx=0x7f7964004b40 comp rx=0 tx=0).stop 2024-04-26T01:58:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.746+0000 7f797daddb80 1 -- 172.21.15.6:0/618392295 shutdown_connections 2024-04-26T01:58:13.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.746+0000 7f797daddb80 1 --2- 172.21.15.6:0/618392295 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d3c1466940 0x55d3c1468d30 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.746+0000 7f797daddb80 1 --2- 172.21.15.6:0/618392295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c1454060 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.746+0000 7f797daddb80 1 --2- 172.21.15.6:0/618392295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144b9b0 0x55d3c1456820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.746+0000 7f797daddb80 1 --2- 172.21.15.6:0/618392295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144c2d0 0x55d3c1453b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.746+0000 7f797daddb80 1 -- 172.21.15.6:0/618392295 >> 172.21.15.6:0/618392295 conn(0x55d3c137a730 msgr2=0x55d3c1447a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:13.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.746+0000 7f795a000700 0 client.26698 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:13.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.746+0000 7f797daddb80 1 -- 172.21.15.6:0/618392295 shutdown_connections 2024-04-26T01:58:13.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.746+0000 7f797daddb80 1 -- 172.21.15.6:0/618392295 wait complete. 2024-04-26T01:58:13.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.746+0000 7f797daddb80 20 client.26698 trim_cache size 0 max 16384 2024-04-26T01:58:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.769+0000 7f797daddb80 1 Processor -- start 2024-04-26T01:58:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.769+0000 7f797daddb80 1 -- start start 2024-04-26T01:58:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.769+0000 7f797daddb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c1447fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.769+0000 7f797daddb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c144c2d0 0x55d3c1453f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.769+0000 7f797daddb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c1457180 0x55d3c146e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.769+0000 7f797daddb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f796c1b1d50 con 0x55d3c144b9b0 2024-04-26T01:58:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.769+0000 7f797daddb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f796c1af5e0 con 0x55d3c1457180 2024-04-26T01:58:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.769+0000 7f797daddb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f796c1ad450 con 0x55d3c144c2d0 2024-04-26T01:58:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.769+0000 7f7963e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c144c2d0 0x55d3c1453f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.770+0000 7f796be00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c1447fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.770+0000 7f7963e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c144c2d0 0x55d3c1453f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58446/0 (socket says 172.21.15.6:58446) 2024-04-26T01:58:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.770+0000 7f7963e00700 1 -- 172.21.15.6:0/3710693966 learned_addr learned my addr 172.21.15.6:0/3710693966 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.770+0000 7f7971200700 1 --2- 172.21.15.6:0/3710693966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c1457180 0x55d3c146e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.770+0000 7f7963e00700 1 -- 172.21.15.6:0/3710693966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c1457180 msgr2=0x55d3c146e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.770+0000 7f7963e00700 1 --2- 172.21.15.6:0/3710693966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c1457180 0x55d3c146e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.770+0000 7f7963e00700 1 -- 172.21.15.6:0/3710693966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 msgr2=0x55d3c1447fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.770+0000 7f7963e00700 1 --2- 172.21.15.6:0/3710693966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c1447fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.770+0000 7f7963e00700 1 -- 172.21.15.6:0/3710693966 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3c1452090 con 0x55d3c144c2d0 2024-04-26T01:58:13.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.770+0000 7f7971200700 1 --2- 172.21.15.6:0/3710693966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c1457180 0x55d3c146e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:13.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.770+0000 7f7963e00700 1 --2- 172.21.15.6:0/3710693966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c144c2d0 0x55d3c1453f70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7954009c10 tx=0x7f7954004da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d8bb6f869056379 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:13.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.771+0000 7f7960c00700 1 -- 172.21.15.6:0/3710693966 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954028020 con 0x55d3c144c2d0 2024-04-26T01:58:13.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.771+0000 7f7960c00700 1 -- 172.21.15.6:0/3710693966 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7954011b00 con 0x55d3c144c2d0 2024-04-26T01:58:13.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.771+0000 7f7960c00700 1 -- 172.21.15.6:0/3710693966 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954005050 con 0x55d3c144c2d0 2024-04-26T01:58:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.771+0000 7f797daddb80 1 -- 172.21.15.6:0/3710693966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c144c2d0 msgr2=0x55d3c1453f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.771+0000 7f797daddb80 1 --2- 172.21.15.6:0/3710693966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c144c2d0 0x55d3c1453f70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7954009c10 tx=0x7f7954004da0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.772+0000 7f797daddb80 1 -- 172.21.15.6:0/3710693966 shutdown_connections 2024-04-26T01:58:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.772+0000 7f797daddb80 1 --2- 172.21.15.6:0/3710693966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c1457180 0x55d3c146e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.772+0000 7f797daddb80 1 --2- 172.21.15.6:0/3710693966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c1447fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.772+0000 7f797daddb80 1 --2- 172.21.15.6:0/3710693966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c144c2d0 0x55d3c1453f70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.772+0000 7f797daddb80 1 -- 172.21.15.6:0/3710693966 >> 172.21.15.6:0/3710693966 conn(0x55d3c137a730 msgr2=0x55d3c1455be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.772+0000 7f797daddb80 1 -- 172.21.15.6:0/3710693966 shutdown_connections 2024-04-26T01:58:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.772+0000 7f797daddb80 1 -- 172.21.15.6:0/3710693966 wait complete. 2024-04-26T01:58:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.772+0000 7f797daddb80 1 -- start start 2024-04-26T01:58:13.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.773+0000 7f797daddb80 1 Processor -- start 2024-04-26T01:58:13.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.773+0000 7f797daddb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c145d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.773+0000 7f797daddb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 0x55d3c145d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.773+0000 7f797daddb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c14cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.773+0000 7f797daddb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f796c1adaa0 con 0x55d3c144b9b0 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.773+0000 7f797daddb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f796c1aef10 con 0x55d3c144c2d0 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.773+0000 7f797daddb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f796c1ae170 con 0x55d3c1457180 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.773+0000 7f797daddb80 10 client.0 _ll_register_callbacks cb 0x55d3c13378b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.774+0000 7f7963e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 0x55d3c145d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.774+0000 7f7963e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 0x55d3c145d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47362/0 (socket says 172.21.15.6:47362) 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.774+0000 7f7963e00700 1 -- 172.21.15.6:0/3768015481 learned_addr learned my addr 172.21.15.6:0/3768015481 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.774+0000 7f7971200700 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c14cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.774+0000 7f796be00700 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c145d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.774+0000 7f7971200700 1 -- 172.21.15.6:0/3768015481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 msgr2=0x55d3c145d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.774+0000 7f7971200700 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 0x55d3c145d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.774+0000 7f7971200700 1 -- 172.21.15.6:0/3768015481 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 msgr2=0x55d3c145d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.774+0000 7f7971200700 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c145d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.774+0000 7f7971200700 1 -- 172.21.15.6:0/3768015481 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3c14cc420 con 0x55d3c1457180 2024-04-26T01:58:13.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.774+0000 7f796be00700 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c145d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:13.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.775+0000 7f7971200700 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c14cbe60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7964023a30 tx=0x7f7964023ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22cdedccf037099c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:13.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.775+0000 7f795a000700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:58:13.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.775+0000 7f795a000700 1 -- 172.21.15.6:0/3768015481 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7964027040 con 0x55d3c1457180 2024-04-26T01:58:13.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.775+0000 7f797daddb80 1 -- 172.21.15.6:0/3768015481 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7954011b00 con 0x55d3c1457180 2024-04-26T01:58:13.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.775+0000 7f795a000700 1 -- 172.21.15.6:0/3768015481 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f7964004bb0 con 0x55d3c1457180 2024-04-26T01:58:13.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.775+0000 7f795a000700 1 -- 172.21.15.6:0/3768015481 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f7964023e60 con 0x55d3c1457180 2024-04-26T01:58:13.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.775+0000 7f797daddb80 10 client.26704 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:13.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.775+0000 7f797daddb80 1 -- 172.21.15.6:0/3768015481 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55d3c14cd6e0 con 0x55d3c1457180 2024-04-26T01:58:13.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.776+0000 7f795a000700 1 -- 172.21.15.6:0/3768015481 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7954011b00 con 0x55d3c1457180 2024-04-26T01:58:13.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.776+0000 7f795a000700 1 -- 172.21.15.6:0/3768015481 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f7964005cf0 con 0x55d3c1457180 2024-04-26T01:58:13.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.776+0000 7f797daddb80 10 client.26704 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:13.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.776+0000 7f797daddb80 10 client.26704 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:58:13.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.776+0000 7f797daddb80 20 client.26704 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:13.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.776+0000 7f797daddb80 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d3c14cdd40 0x55d3c14d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.776+0000 7f797daddb80 4 client.26704 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.25, "await_for": 42.0}] 2024-04-26T01:58:13.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.776+0000 7f797daddb80 1 -- 172.21.15.6:0/3768015481 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.25, "await_for": 42.0}) -- 0x55d3c13a68b0 con 0x55d3c14cdd40 2024-04-26T01:58:13.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.776+0000 7f796be00700 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d3c14cdd40 0x55d3c14d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.777+0000 7f796be00700 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d3c14cdd40 0x55d3c14d0160 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9cd4d1fa3d4b58 server_cookie=39cb274d29d6c37c in_seq=0 out_seq=0 2024-04-26T01:58:13.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.777+0000 7f795a000700 10 client.26704 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:13.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.848+0000 7f795a000700 1 -- 172.21.15.6:0/3768015481 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55d3c13a68b0 con 0x55d3c14cdd40 2024-04-26T01:58:13.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.848+0000 7f795a000700 10 client.26704 handle_command_reply: tid=0 2024-04-26T01:58:13.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.848+0000 7f797daddb80 1 client.26704 shutdown 2024-04-26T01:58:13.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.849+0000 7f797daddb80 1 -- 172.21.15.6:0/3768015481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 msgr2=0x55d3c14cbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.849+0000 7f797daddb80 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c14cbe60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7964023a30 tx=0x7f7964023ce0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.849+0000 7f797daddb80 1 -- 172.21.15.6:0/3768015481 shutdown_connections 2024-04-26T01:58:13.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.849+0000 7f797daddb80 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d3c14cdd40 0x55d3c14d0160 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.849+0000 7f797daddb80 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d3c1457180 0x55d3c14cbe60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.849+0000 7f795a000700 0 client.26704 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:13.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.849+0000 7f797daddb80 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d3c144b9b0 0x55d3c145d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.849+0000 7f797daddb80 1 --2- 172.21.15.6:0/3768015481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3c144c2d0 0x55d3c145d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.849+0000 7f797daddb80 1 -- 172.21.15.6:0/3768015481 >> 172.21.15.6:0/3768015481 conn(0x55d3c137a730 msgr2=0x55d3c1446f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:13.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.849+0000 7f797daddb80 1 -- 172.21.15.6:0/3768015481 shutdown_connections 2024-04-26T01:58:13.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.850+0000 7f797daddb80 1 -- 172.21.15.6:0/3768015481 wait complete. 2024-04-26T01:58:13.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.850+0000 7f797daddb80 20 client.26704 trim_cache size 0 max 16384 2024-04-26T01:58:13.850 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:58:13.850 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 43, 2024-04-26T01:58:13.850 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:58:13.850 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 38, 2024-04-26T01:58:13.850 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:58:13.850 INFO:teuthology.orchestra.run.smithi006.stdout: "db2f2455": { 2024-04-26T01:58:13.850 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 38, 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 79.2, 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:13.851 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:13.852 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:13.852 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:13.852 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:13.852 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:58:13.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.851+0000 7f7973200700 1 -- 172.21.15.6:0/1765322056 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f79500a3ed0 msgr2=0x7f79500a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.851+0000 7f7973200700 1 --2- 172.21.15.6:0/1765322056 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f79500a3ed0 0x7f79500a6390 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f796400f870 tx=0x7f796400fc30 comp rx=0 tx=0).stop 2024-04-26T01:58:13.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.851+0000 7f7973200700 1 -- 172.21.15.6:0/1765322056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c102af0 msgr2=0x7f796c074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.851+0000 7f7973200700 1 --2- 172.21.15.6:0/1765322056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c102af0 0x7f796c074d40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f795c005910 tx=0x7f795c00bfc0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.851+0000 7f7973200700 1 -- 172.21.15.6:0/1765322056 shutdown_connections 2024-04-26T01:58:13.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.851+0000 7f7973200700 1 --2- 172.21.15.6:0/1765322056 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f79500a3ed0 0x7f79500a6390 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.851+0000 7f7973200700 1 --2- 172.21.15.6:0/1765322056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f796c108a30 0x7f796c0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.851+0000 7f7973200700 1 --2- 172.21.15.6:0/1765322056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c102af0 0x7f796c074d40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.851+0000 7f7973200700 1 --2- 172.21.15.6:0/1765322056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796c1034b0 0x7f796c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.851+0000 7f7973200700 1 -- 172.21.15.6:0/1765322056 >> 172.21.15.6:0/1765322056 conn(0x7f796c079b10 msgr2=0x7f796c101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:13.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.852+0000 7f7973200700 1 -- 172.21.15.6:0/1765322056 shutdown_connections 2024-04-26T01:58:13.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:13.852+0000 7f7973200700 1 -- 172.21.15.6:0/1765322056 wait complete. 2024-04-26T01:58:14.535 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'db2f2455', quiesce took 7.7 seconds. Will release after: 19.25 2024-04-26T01:58:14.635 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.58 MB/sec execute 518 sec latency 806.223 ms 2024-04-26T01:58:14.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:14 smithi006 ceph-mon[28419]: pgmap v390: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 149 op/s 2024-04-26T01:58:14.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:14 smithi188 ceph-mon[32751]: pgmap v390: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 149 op/s 2024-04-26T01:58:14.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:14 smithi029 ceph-mon[32775]: pgmap v390: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 149 op/s 2024-04-26T01:58:15.635 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.57 MB/sec execute 519 sec latency 1806.359 ms 2024-04-26T01:58:16.635 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.57 MB/sec execute 520 sec latency 2806.488 ms 2024-04-26T01:58:16.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:16 smithi188 ceph-mon[32751]: pgmap v391: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 129 op/s 2024-04-26T01:58:16.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:16 smithi029 ceph-mon[32775]: pgmap v391: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 129 op/s 2024-04-26T01:58:17.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:16 smithi006 ceph-mon[28419]: pgmap v391: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 129 op/s 2024-04-26T01:58:17.636 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.56 MB/sec execute 521 sec latency 3806.581 ms 2024-04-26T01:58:18.636 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.55 MB/sec execute 522 sec latency 4806.739 ms 2024-04-26T01:58:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:18 smithi029 ceph-mon[32775]: pgmap v392: 97 pgs: 97 active+clean; 137 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-04-26T01:58:19.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:18 smithi006 ceph-mon[28419]: pgmap v392: 97 pgs: 97 active+clean; 137 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-04-26T01:58:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:18 smithi188 ceph-mon[32751]: pgmap v392: 97 pgs: 97 active+clean; 137 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-04-26T01:58:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.508+0000 7f110b200640 1 -- 172.21.15.6:0/509785845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110c112d80 msgr2=0x7f110c115170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.508+0000 7f110b200640 1 --2- 172.21.15.6:0/509785845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110c112d80 0x7f110c115170 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1104009ca0 tx=0x7f110402a940 comp rx=0 tx=0).stop 2024-04-26T01:58:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.508+0000 7f110b200640 1 -- 172.21.15.6:0/509785845 shutdown_connections 2024-04-26T01:58:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.508+0000 7f110b200640 1 --2- 172.21.15.6:0/509785845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c074c10 0x7f110c112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.508+0000 7f110b200640 1 --2- 172.21.15.6:0/509785845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110c112d80 0x7f110c115170 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.508+0000 7f110b200640 1 --2- 172.21.15.6:0/509785845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f110c0742f0 0x7f110c0746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.508+0000 7f110b200640 1 -- 172.21.15.6:0/509785845 >> 172.21.15.6:0/509785845 conn(0x7f110c0fd860 msgr2=0x7f110c0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.508+0000 7f110b200640 1 -- 172.21.15.6:0/509785845 shutdown_connections 2024-04-26T01:58:19.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.508+0000 7f110b200640 1 -- 172.21.15.6:0/509785845 wait complete. 2024-04-26T01:58:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.509+0000 7f110b200640 1 Processor -- start 2024-04-26T01:58:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.509+0000 7f110b200640 1 -- start start 2024-04-26T01:58:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.509+0000 7f110b200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110c0742f0 0x7f110c1a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.509+0000 7f110b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c074c10 0x7f110c1a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.509+0000 7f110b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f110c112d80 0x7f110c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f110b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f110c118640 con 0x7f110c074c10 2024-04-26T01:58:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f110b200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1184c0 con 0x7f110c0742f0 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f110b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1187c0 con 0x7f110c112d80 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f1103e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c074c10 0x7f110c1a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f1103e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c074c10 0x7f110c1a4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33584/0 (socket says 172.21.15.6:33584) 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f1103e00640 1 -- 172.21.15.6:0/2374341780 learned_addr learned my addr 172.21.15.6:0/2374341780 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f1109e00640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110c0742f0 0x7f110c1a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f110a800640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f110c112d80 0x7f110c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f1103e00640 1 -- 172.21.15.6:0/2374341780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f110c112d80 msgr2=0x7f110c19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f1103e00640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f110c112d80 0x7f110c19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f1103e00640 1 -- 172.21.15.6:0/2374341780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110c0742f0 msgr2=0x7f110c1a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f1103e00640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110c0742f0 0x7f110c1a47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f1103e00640 1 -- 172.21.15.6:0/2374341780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110c19f100 con 0x7f110c074c10 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.510+0000 7f1109e00640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110c0742f0 0x7f110c1a47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.511+0000 7f1103e00640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c074c10 0x7f110c1a4d10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f10f400ea60 tx=0x7f10f400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.511+0000 7f1102a00640 1 -- 172.21.15.6:0/2374341780 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f400cde0 con 0x7f110c074c10 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.511+0000 7f110b200640 1 -- 172.21.15.6:0/2374341780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f110c19f3f0 con 0x7f110c074c10 2024-04-26T01:58:19.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.511+0000 7f1102a00640 1 -- 172.21.15.6:0/2374341780 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f40035f0 con 0x7f110c074c10 2024-04-26T01:58:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.511+0000 7f1102a00640 1 -- 172.21.15.6:0/2374341780 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f4010690 con 0x7f110c074c10 2024-04-26T01:58:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.511+0000 7f110b200640 1 -- 172.21.15.6:0/2374341780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f110c1ab690 con 0x7f110c074c10 2024-04-26T01:58:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.514+0000 7f1113c07740 1 Processor -- start 2024-04-26T01:58:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f1113c07740 1 -- start start 2024-04-26T01:58:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f1102a00640 1 -- 172.21.15.6:0/2374341780 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f10f40037a0 con 0x7f110c074c10 2024-04-26T01:58:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f1113c07740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a5244ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f1113c07740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244b2d0 0x559a52455650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f1113c07740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a52457f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f1113c07740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x559a523b0900 con 0x559a5244a9b0 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f1113c07740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559a52356ef0 con 0x559a52455b90 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f1113c07740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559a523ac8c0 con 0x559a5244b2d0 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f110a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a5244ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f110a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a5244ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33594/0 (socket says 172.21.15.6:33594) 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f110a800640 1 -- 172.21.15.6:0/3700339044 learned_addr learned my addr 172.21.15.6:0/3700339044 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f1109e00640 1 --2- 172.21.15.6:0/3700339044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244b2d0 0x559a52455650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.515+0000 7f1103e00640 1 --2- 172.21.15.6:0/3700339044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a52457f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.516+0000 7f1109e00640 1 -- 172.21.15.6:0/3700339044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 msgr2=0x559a52457f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.516+0000 7f1109e00640 1 --2- 172.21.15.6:0/3700339044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a52457f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.516+0000 7f1109e00640 1 -- 172.21.15.6:0/3700339044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 msgr2=0x559a5244ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.516+0000 7f1109e00640 1 --2- 172.21.15.6:0/3700339044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a5244ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.516+0000 7f1109e00640 1 -- 172.21.15.6:0/3700339044 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a520c3210 con 0x559a5244b2d0 2024-04-26T01:58:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.516+0000 7f1109e00640 1 --2- 172.21.15.6:0/3700339044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244b2d0 0x559a52455650 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009fa0 tx=0x7f10fc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=368313e255116230 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.516+0000 7f10fb400640 1 -- 172.21.15.6:0/3700339044 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc0040f0 con 0x559a5244b2d0 2024-04-26T01:58:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.516+0000 7f1102a00640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f10f00a3d00 0x7f10f00a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.516+0000 7f10fb400640 1 -- 172.21.15.6:0/3700339044 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10fc0054d0 con 0x559a5244b2d0 2024-04-26T01:58:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.516+0000 7f10fb400640 1 -- 172.21.15.6:0/3700339044 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc004d20 con 0x559a5244b2d0 2024-04-26T01:58:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.517+0000 7f1102a00640 1 -- 172.21.15.6:0/2374341780 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f10f40d0b10 con 0x7f110c074c10 2024-04-26T01:58:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.517+0000 7f110a800640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f10f00a3d00 0x7f10f00a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.517+0000 7f1113c07740 1 -- 172.21.15.6:0/3700339044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244b2d0 msgr2=0x559a52455650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.517+0000 7f1113c07740 1 --2- 172.21.15.6:0/3700339044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244b2d0 0x559a52455650 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009fa0 tx=0x7f10fc0118d0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.517+0000 7f1113c07740 1 -- 172.21.15.6:0/3700339044 shutdown_connections 2024-04-26T01:58:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.517+0000 7f1113c07740 1 --2- 172.21.15.6:0/3700339044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a52457f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.517+0000 7f1113c07740 1 --2- 172.21.15.6:0/3700339044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244b2d0 0x559a52455650 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.517+0000 7f1113c07740 1 --2- 172.21.15.6:0/3700339044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a5244ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.517+0000 7f1113c07740 1 -- 172.21.15.6:0/3700339044 >> 172.21.15.6:0/3700339044 conn(0x559a52446ec0 msgr2=0x559a524472a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.518+0000 7f1113c07740 1 -- 172.21.15.6:0/3700339044 shutdown_connections 2024-04-26T01:58:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.518+0000 7f110a800640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f10f00a3d00 0x7f10f00a61c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1104009a60 tx=0x7f1104005d80 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.518+0000 7f1113c07740 1 -- 172.21.15.6:0/3700339044 wait complete. 2024-04-26T01:58:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.518+0000 7f1113c07740 1 -- start start 2024-04-26T01:58:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.518+0000 7f1113c07740 1 Processor -- start 2024-04-26T01:58:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.519+0000 7f1113c07740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a52452a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.519+0000 7f1113c07740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244b2d0 0x559a52452fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.519+0000 7f1113c07740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a524534f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.519+0000 7f1113c07740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x559a524587c0 con 0x559a5244a9b0 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.519+0000 7f1113c07740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559a5236a310 con 0x559a52455b90 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.519+0000 7f1113c07740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559a52458940 con 0x559a5244b2d0 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.519+0000 7f1113c07740 10 client.0 _ll_register_callbacks cb 0x559a52338cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.519+0000 7f1109e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a52452a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.519+0000 7f110a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a524534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.519+0000 7f1109e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a52452a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33602/0 (socket says 172.21.15.6:33602) 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.519+0000 7f1109e00640 1 -- 172.21.15.6:0/2803894815 learned_addr learned my addr 172.21.15.6:0/2803894815 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.519+0000 7f1103e00640 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244b2d0 0x559a52452fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.520+0000 7f110a800640 1 -- 172.21.15.6:0/2803894815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244b2d0 msgr2=0x559a52452fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.520+0000 7f110a800640 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244b2d0 0x559a52452fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.520+0000 7f110a800640 1 -- 172.21.15.6:0/2803894815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 msgr2=0x559a52452a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.520+0000 7f110a800640 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a52452a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.520+0000 7f110a800640 1 -- 172.21.15.6:0/2803894815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a5218f620 con 0x559a52455b90 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.520+0000 7f1109e00640 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a52452a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.520+0000 7f110a800640 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a524534f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1104006010 tx=0x7f11040095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc91c5cf1a6031b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.520+0000 7f10efe00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.520+0000 7f10efe00640 1 -- 172.21.15.6:0/2803894815 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104034040 con 0x559a52455b90 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.520+0000 7f1113c07740 1 -- 172.21.15.6:0/2803894815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10fc0054d0 con 0x559a52455b90 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.521+0000 7f10efe00640 1 -- 172.21.15.6:0/2803894815 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f1104030d90 con 0x559a52455b90 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.521+0000 7f10efe00640 1 -- 172.21.15.6:0/2803894815 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f1104003f70 con 0x559a52455b90 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.521+0000 7f1113c07740 10 client.26675 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.521+0000 7f1113c07740 1 -- 172.21.15.6:0/2803894815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x559a524649f0 con 0x559a52455b90 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.521+0000 7f10efe00640 1 -- 172.21.15.6:0/2803894815 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10fc0054d0 con 0x559a52455b90 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.521+0000 7f10efe00640 1 -- 172.21.15.6:0/2803894815 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f1104050070 con 0x559a52455b90 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.521+0000 7f1113c07740 10 client.26675 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.521+0000 7f1113c07740 10 client.26675 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.521+0000 7f1113c07740 20 client.26675 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.521+0000 7f1113c07740 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559a52465e60 0x559a52468250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.521+0000 7f1113c07740 4 client.26675 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.521+0000 7f1113c07740 1 -- 172.21.15.6:0/2803894815 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a523c48b0 con 0x559a52465e60 2024-04-26T01:58:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.522+0000 7f1109e00640 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559a52465e60 0x559a52468250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.522+0000 7f1109e00640 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559a52465e60 0x559a52468250 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce43d75256ced96a server_cookie=d113f2d8a50c6308 in_seq=0 out_seq=0 2024-04-26T01:58:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.522+0000 7f10efe00640 10 client.26675 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.523+0000 7f10efe00640 1 -- 172.21.15.6:0/2803894815 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a523c48b0 con 0x559a52465e60 2024-04-26T01:58:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.523+0000 7f10efe00640 10 client.26675 handle_command_reply: tid=0 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.523+0000 7f1113c07740 1 client.26675 shutdown 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.523+0000 7f1113c07740 1 -- 172.21.15.6:0/2803894815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 msgr2=0x559a524534f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.523+0000 7f1113c07740 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a524534f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1104006010 tx=0x7f11040095c0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.523+0000 7f1113c07740 1 -- 172.21.15.6:0/2803894815 shutdown_connections 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.523+0000 7f1113c07740 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559a52465e60 0x559a52468250 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.523+0000 7f1113c07740 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a524534f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.523+0000 7f1113c07740 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244b2d0 0x559a52452fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.523+0000 7f1113c07740 1 --2- 172.21.15.6:0/2803894815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a52452a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.523+0000 7f1113c07740 1 -- 172.21.15.6:0/2803894815 >> 172.21.15.6:0/2803894815 conn(0x559a52446ec0 msgr2=0x559a5244cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.523+0000 7f10efe00640 0 client.26675 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.524+0000 7f1113c07740 1 -- 172.21.15.6:0/2803894815 shutdown_connections 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.524+0000 7f1113c07740 1 -- 172.21.15.6:0/2803894815 wait complete. 2024-04-26T01:58:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.524+0000 7f1113c07740 20 client.26675 trim_cache size 0 max 16384 2024-04-26T01:58:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.540+0000 7f1113c07740 1 Processor -- start 2024-04-26T01:58:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.540+0000 7f1113c07740 1 -- start start 2024-04-26T01:58:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.541+0000 7f1113c07740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244a9b0 0x559a52461680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.541+0000 7f1113c07740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244b2d0 0x559a5244d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.541+0000 7f1113c07740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a5246db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.541+0000 7f1113c07740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1b6a00 con 0x559a5244b2d0 2024-04-26T01:58:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.541+0000 7f1113c07740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1b42d0 con 0x559a52455b90 2024-04-26T01:58:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.541+0000 7f1113c07740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1b2140 con 0x559a5244a9b0 2024-04-26T01:58:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.541+0000 7f1103e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244b2d0 0x559a5244d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.541+0000 7f1103e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244b2d0 0x559a5244d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33604/0 (socket says 172.21.15.6:33604) 2024-04-26T01:58:19.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.541+0000 7f1103e00640 1 -- 172.21.15.6:0/1192768579 learned_addr learned my addr 172.21.15.6:0/1192768579 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:19.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.541+0000 7f110a800640 1 --2- 172.21.15.6:0/1192768579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a5246db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.541+0000 7f1109e00640 1 --2- 172.21.15.6:0/1192768579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244a9b0 0x559a52461680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.542+0000 7f1103e00640 1 -- 172.21.15.6:0/1192768579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244a9b0 msgr2=0x559a52461680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:19.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.542+0000 7f1103e00640 1 --2- 172.21.15.6:0/1192768579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244a9b0 0x559a52461680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.542+0000 7f1103e00640 1 -- 172.21.15.6:0/1192768579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 msgr2=0x559a5246db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:19.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.542+0000 7f1103e00640 1 --2- 172.21.15.6:0/1192768579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a5246db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.542+0000 7f1103e00640 1 -- 172.21.15.6:0/1192768579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a52446cf0 con 0x559a5244b2d0 2024-04-26T01:58:19.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.542+0000 7f110a800640 1 --2- 172.21.15.6:0/1192768579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a5246db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:19.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.542+0000 7f1103e00640 1 --2- 172.21.15.6:0/1192768579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244b2d0 0x559a5244d370 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f10f40ce210 tx=0x7f10f400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f216a4e390f5830 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:19.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.542+0000 7f10faa00640 1 -- 172.21.15.6:0/1192768579 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f409c360 con 0x559a5244b2d0 2024-04-26T01:58:19.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.542+0000 7f10faa00640 1 -- 172.21.15.6:0/1192768579 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f40d8040 con 0x559a5244b2d0 2024-04-26T01:58:19.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.542+0000 7f10faa00640 1 -- 172.21.15.6:0/1192768579 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f4099070 con 0x559a5244b2d0 2024-04-26T01:58:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.543+0000 7f1113c07740 1 -- 172.21.15.6:0/1192768579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244b2d0 msgr2=0x559a5244d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.543+0000 7f1113c07740 1 --2- 172.21.15.6:0/1192768579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244b2d0 0x559a5244d370 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f10f40ce210 tx=0x7f10f400ef90 comp rx=0 tx=0).stop 2024-04-26T01:58:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.543+0000 7f1113c07740 1 -- 172.21.15.6:0/1192768579 shutdown_connections 2024-04-26T01:58:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.543+0000 7f1113c07740 1 --2- 172.21.15.6:0/1192768579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244b2d0 0x559a5244d370 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.543+0000 7f1113c07740 1 --2- 172.21.15.6:0/1192768579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a52455b90 0x559a5246db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.543+0000 7f1113c07740 1 --2- 172.21.15.6:0/1192768579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a5244a9b0 0x559a52461680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.543+0000 7f1113c07740 1 -- 172.21.15.6:0/1192768579 >> 172.21.15.6:0/1192768579 conn(0x559a52446ec0 msgr2=0x559a5244dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.543+0000 7f1113c07740 1 -- 172.21.15.6:0/1192768579 shutdown_connections 2024-04-26T01:58:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.543+0000 7f1113c07740 1 -- 172.21.15.6:0/1192768579 wait complete. 2024-04-26T01:58:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.543+0000 7f1113c07740 1 -- start start 2024-04-26T01:58:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f1113c07740 1 Processor -- start 2024-04-26T01:58:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f1113c07740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a5245dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f1113c07740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a5244b2d0 0x559a5245b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f1113c07740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a52455b90 0x559a5245b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f1113c07740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1b2790 con 0x559a5244a9b0 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f1113c07740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1b3c00 con 0x559a5244b2d0 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f1113c07740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1b2e60 con 0x559a52455b90 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f1113c07740 10 client.0 _ll_register_callbacks cb 0x7f1104003de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f1109e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a5245dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f1109e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a5245dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33618/0 (socket says 172.21.15.6:33618) 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f1109e00640 1 -- 172.21.15.6:0/4140770032 learned_addr learned my addr 172.21.15.6:0/4140770032 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f1103e00640 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a5244b2d0 0x559a5245b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.544+0000 7f110a800640 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a52455b90 0x559a5245b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.545+0000 7f1109e00640 1 -- 172.21.15.6:0/4140770032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a52455b90 msgr2=0x559a5245b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.545+0000 7f1109e00640 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a52455b90 0x559a5245b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.545+0000 7f1109e00640 1 -- 172.21.15.6:0/4140770032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a5244b2d0 msgr2=0x559a5245b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.545+0000 7f1109e00640 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a5244b2d0 0x559a5245b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.545+0000 7f1109e00640 1 -- 172.21.15.6:0/4140770032 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a5245bbe0 con 0x559a5244a9b0 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.545+0000 7f1103e00640 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a5244b2d0 0x559a5245b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.545+0000 7f1109e00640 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a5245dc50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009710 tx=0x7f10fc005d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b597a22e15d532c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.545+0000 7f10efe00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.545+0000 7f10efe00640 1 -- 172.21.15.6:0/4140770032 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc032060 con 0x559a5244a9b0 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.545+0000 7f1113c07740 1 -- 172.21.15.6:0/4140770032 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10f40d8040 con 0x559a5244a9b0 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.545+0000 7f10efe00640 1 -- 172.21.15.6:0/4140770032 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f10fc0035f0 con 0x559a5244a9b0 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.546+0000 7f10efe00640 1 -- 172.21.15.6:0/4140770032 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f10fc01d040 con 0x559a5244a9b0 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.546+0000 7f1113c07740 10 client.16530 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.546+0000 7f1113c07740 1 -- 172.21.15.6:0/4140770032 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x559a524cc0d0 con 0x559a5244a9b0 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.546+0000 7f10efe00640 1 -- 172.21.15.6:0/4140770032 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f40d8040 con 0x559a5244a9b0 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.546+0000 7f10efe00640 1 -- 172.21.15.6:0/4140770032 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f10fc00f8b0 con 0x559a5244a9b0 2024-04-26T01:58:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.546+0000 7f1113c07740 10 client.16530 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.546+0000 7f1113c07740 10 client.16530 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:58:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.546+0000 7f1113c07740 20 client.16530 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.546+0000 7f1113c07740 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559a524cc660 0x559a524cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.546+0000 7f1113c07740 4 client.16530 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:58:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.546+0000 7f1113c07740 1 -- 172.21.15.6:0/4140770032 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559a523be320 con 0x559a524cc660 2024-04-26T01:58:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.547+0000 7f1103e00640 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559a524cc660 0x559a524cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.547+0000 7f1103e00640 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559a524cc660 0x559a524cea50 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32ce2917ae9f9eb4 server_cookie=6b672abfd6b451ea in_seq=0 out_seq=0 2024-04-26T01:58:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.547+0000 7f10efe00640 10 client.16530 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f10efe00640 1 -- 172.21.15.6:0/4140770032 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559a523be320 con 0x559a524cc660 2024-04-26T01:58:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f10efe00640 10 client.16530 handle_command_reply: tid=0 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f1113c07740 1 client.16530 shutdown 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f1113c07740 1 -- 172.21.15.6:0/4140770032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 msgr2=0x559a5245dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f1113c07740 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a5245dc50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009710 tx=0x7f10fc005d60 comp rx=0 tx=0).stop 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f1113c07740 1 -- 172.21.15.6:0/4140770032 shutdown_connections 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f1113c07740 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559a524cc660 0x559a524cea50 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f1113c07740 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559a52455b90 0x559a5245b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f1113c07740 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559a5244b2d0 0x559a5245b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f1113c07740 1 --2- 172.21.15.6:0/4140770032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559a5244a9b0 0x559a5245dc50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f10efe00640 0 client.16530 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f1113c07740 1 -- 172.21.15.6:0/4140770032 >> 172.21.15.6:0/4140770032 conn(0x559a52446ec0 msgr2=0x559a52445790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.590+0000 7f1113c07740 1 -- 172.21.15.6:0/4140770032 shutdown_connections 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.591+0000 7f1113c07740 1 -- 172.21.15.6:0/4140770032 wait complete. 2024-04-26T01:58:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.591+0000 7f1113c07740 20 client.16530 trim_cache size 0 max 16384 2024-04-26T01:58:19.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:58:19.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:58:19.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:58:19.591 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:58:19.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.591+0000 7f110b200640 1 -- 172.21.15.6:0/2374341780 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f10f00a3d00 msgr2=0x7f10f00a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:19.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.591+0000 7f110b200640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f10f00a3d00 0x7f10f00a61c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1104009a60 tx=0x7f1104005d80 comp rx=0 tx=0).stop 2024-04-26T01:58:19.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.591+0000 7f110b200640 1 -- 172.21.15.6:0/2374341780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c074c10 msgr2=0x7f110c1a4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:19.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.591+0000 7f110b200640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c074c10 0x7f110c1a4d10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f10f400ea60 tx=0x7f10f400ef20 comp rx=0 tx=0).stop 2024-04-26T01:58:19.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.592+0000 7f110b200640 1 -- 172.21.15.6:0/2374341780 shutdown_connections 2024-04-26T01:58:19.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.592+0000 7f110b200640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f10f00a3d00 0x7f10f00a61c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.592+0000 7f110b200640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c074c10 0x7f110c1a4d10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.592+0000 7f110b200640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f110c112d80 0x7f110c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.592+0000 7f110b200640 1 --2- 172.21.15.6:0/2374341780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110c0742f0 0x7f110c1a47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:19.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.592+0000 7f110b200640 1 -- 172.21.15.6:0/2374341780 >> 172.21.15.6:0/2374341780 conn(0x7f110c0fd860 msgr2=0x7f110c0ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:19.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.592+0000 7f110b200640 1 -- 172.21.15.6:0/2374341780 shutdown_connections 2024-04-26T01:58:19.592 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:19.592+0000 7f110b200640 1 -- 172.21.15.6:0/2374341780 wait complete. 2024-04-26T01:58:19.635 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.55 MB/sec execute 523 sec latency 5806.837 ms 2024-04-26T01:58:20.636 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.54 MB/sec execute 524 sec latency 6806.981 ms 2024-04-26T01:58:20.895 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:20 smithi006 ceph-mon[28419]: pgmap v393: 97 pgs: 97 active+clean; 137 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-04-26T01:58:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:20 smithi029 ceph-mon[32775]: pgmap v393: 97 pgs: 97 active+clean; 137 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-04-26T01:58:21.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:20 smithi188 ceph-mon[32751]: pgmap v393: 97 pgs: 97 active+clean; 137 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-04-26T01:58:21.636 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.53 MB/sec execute 525 sec latency 7807.119 ms 2024-04-26T01:58:22.636 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.52 MB/sec execute 526 sec latency 8807.270 ms 2024-04-26T01:58:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:22 smithi006 ceph-mon[28419]: pgmap v394: 97 pgs: 97 active+clean; 133 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-04-26T01:58:22.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:22 smithi029 ceph-mon[32775]: pgmap v394: 97 pgs: 97 active+clean; 133 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-04-26T01:58:23.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:22 smithi188 ceph-mon[32751]: pgmap v394: 97 pgs: 97 active+clean; 133 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 171 op/s 2024-04-26T01:58:23.636 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.52 MB/sec execute 527 sec latency 9807.426 ms 2024-04-26T01:58:24.636 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.51 MB/sec execute 528 sec latency 10807.528 ms 2024-04-26T01:58:24.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:24 smithi029 ceph-mon[32775]: pgmap v395: 97 pgs: 97 active+clean; 133 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 413 KiB/s wr, 70 op/s 2024-04-26T01:58:25.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:24 smithi006 ceph-mon[28419]: pgmap v395: 97 pgs: 97 active+clean; 133 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 413 KiB/s wr, 70 op/s 2024-04-26T01:58:25.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:24 smithi188 ceph-mon[32751]: pgmap v395: 97 pgs: 97 active+clean; 133 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 413 KiB/s wr, 70 op/s 2024-04-26T01:58:25.636 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.50 MB/sec execute 529 sec latency 11807.723 ms 2024-04-26T01:58:26.637 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.50 MB/sec execute 530 sec latency 12807.852 ms 2024-04-26T01:58:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:26 smithi029 ceph-mon[32775]: pgmap v396: 97 pgs: 97 active+clean; 133 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 413 KiB/s wr, 70 op/s 2024-04-26T01:58:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:26 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:58:27.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:26 smithi006 ceph-mon[28419]: pgmap v396: 97 pgs: 97 active+clean; 133 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 413 KiB/s wr, 70 op/s 2024-04-26T01:58:27.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:26 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:58:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:26 smithi188 ceph-mon[32751]: pgmap v396: 97 pgs: 97 active+clean; 133 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 413 KiB/s wr, 70 op/s 2024-04-26T01:58:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:26 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:58:27.637 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.49 MB/sec execute 531 sec latency 13807.949 ms 2024-04-26T01:58:27.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:27 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:58:27.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:27 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:58:27.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:27 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:58:28.042 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:27 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:58:28.042 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:27 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:58:28.042 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:27 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:58:28.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:27 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:58:28.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:27 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:58:28.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:27 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:58:28.637 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.49 MB/sec execute 532 sec latency 14808.045 ms 2024-04-26T01:58:28.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:28 smithi029 ceph-mon[32775]: pgmap v397: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 490 KiB/s wr, 74 op/s 2024-04-26T01:58:29.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:28 smithi006 ceph-mon[28419]: pgmap v397: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 490 KiB/s wr, 74 op/s 2024-04-26T01:58:29.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:28 smithi188 ceph-mon[32751]: pgmap v397: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 490 KiB/s wr, 74 op/s 2024-04-26T01:58:29.637 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.48 MB/sec execute 533 sec latency 15808.176 ms 2024-04-26T01:58:30.637 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.47 MB/sec execute 534 sec latency 16808.326 ms 2024-04-26T01:58:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:30 smithi029 ceph-mon[32775]: pgmap v398: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 110 KiB/s wr, 25 op/s 2024-04-26T01:58:31.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:30 smithi006 ceph-mon[28419]: pgmap v398: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 110 KiB/s wr, 25 op/s 2024-04-26T01:58:31.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:30 smithi188 ceph-mon[32751]: pgmap v398: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 110 KiB/s wr, 25 op/s 2024-04-26T01:58:31.637 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.47 MB/sec execute 535 sec latency 17808.485 ms 2024-04-26T01:58:31.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:31 smithi029 ceph-mon[32775]: pgmap v399: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 126 KiB/s wr, 29 op/s 2024-04-26T01:58:31.963 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:31 smithi006 ceph-mon[28419]: pgmap v399: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 126 KiB/s wr, 29 op/s 2024-04-26T01:58:32.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:31 smithi188 ceph-mon[32751]: pgmap v399: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 126 KiB/s wr, 29 op/s 2024-04-26T01:58:32.637 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.46 MB/sec execute 536 sec latency 18808.578 ms 2024-04-26T01:58:32.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.723+0000 7f1463800640 1 -- 172.21.15.6:0/3418592587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c1031c0 msgr2=0x7f145c1035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.723+0000 7f1463800640 1 --2- 172.21.15.6:0/3418592587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c1031c0 0x7f145c1035a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f144c009ca0 tx=0x7f144c02a950 comp rx=0 tx=0).stop 2024-04-26T01:58:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.723+0000 7f1463800640 1 -- 172.21.15.6:0/3418592587 shutdown_connections 2024-04-26T01:58:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.723+0000 7f1463800640 1 --2- 172.21.15.6:0/3418592587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f145c10c590 0x7f145c10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.723+0000 7f1463800640 1 --2- 172.21.15.6:0/3418592587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f145c103ae0 0x7f145c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.723+0000 7f1463800640 1 --2- 172.21.15.6:0/3418592587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c1031c0 0x7f145c1035a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.723+0000 7f1463800640 1 -- 172.21.15.6:0/3418592587 >> 172.21.15.6:0/3418592587 conn(0x7f145c077f70 msgr2=0x7f145c078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.723+0000 7f1463800640 1 -- 172.21.15.6:0/3418592587 shutdown_connections 2024-04-26T01:58:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.723+0000 7f1463800640 1 -- 172.21.15.6:0/3418592587 wait complete. 2024-04-26T01:58:32.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.724+0000 7f1463800640 1 Processor -- start 2024-04-26T01:58:32.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.724+0000 7f1463800640 1 -- start start 2024-04-26T01:58:32.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.725+0000 7f1463800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f145c1031c0 0x7f145c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.725+0000 7f1463800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c103ae0 0x7f145c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.725+0000 7f1463800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f145c10c590 0x7f145c19a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.725+0000 7f1463800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f145c112230 con 0x7f145c10c590 2024-04-26T01:58:32.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.725+0000 7f1463800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f145c1120b0 con 0x7f145c103ae0 2024-04-26T01:58:32.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.725+0000 7f1463800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f145c1123b0 con 0x7f145c1031c0 2024-04-26T01:58:32.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.725+0000 7f145be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c103ae0 0x7f145c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.725+0000 7f145be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c103ae0 0x7f145c1a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:40556/0 (socket says 172.21.15.6:40556) 2024-04-26T01:58:32.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.725+0000 7f145be00640 1 -- 172.21.15.6:0/1760093335 learned_addr learned my addr 172.21.15.6:0/1760093335 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:32.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.725+0000 7f1460e00640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f145c1031c0 0x7f145c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.725+0000 7f1461800640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f145c10c590 0x7f145c19a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.726+0000 7f145be00640 1 -- 172.21.15.6:0/1760093335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f145c1031c0 msgr2=0x7f145c1a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:32.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.726+0000 7f145be00640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f145c1031c0 0x7f145c1a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.726+0000 7f145be00640 1 -- 172.21.15.6:0/1760093335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f145c10c590 msgr2=0x7f145c19a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:32.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.726+0000 7f145be00640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f145c10c590 0x7f145c19a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.726+0000 7f145be00640 1 -- 172.21.15.6:0/1760093335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145c19afe0 con 0x7f145c103ae0 2024-04-26T01:58:32.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.726+0000 7f145be00640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c103ae0 0x7f145c1a0bf0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f144800e9f0 tx=0x7f144800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:32.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.726+0000 7f1459600640 1 -- 172.21.15.6:0/1760093335 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144800cd80 con 0x7f145c103ae0 2024-04-26T01:58:32.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.726+0000 7f1459600640 1 -- 172.21.15.6:0/1760093335 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14480054d0 con 0x7f145c103ae0 2024-04-26T01:58:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.726+0000 7f1459600640 1 -- 172.21.15.6:0/1760093335 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1448010690 con 0x7f145c103ae0 2024-04-26T01:58:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.726+0000 7f1463800640 1 -- 172.21.15.6:0/1760093335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f145c19b2d0 con 0x7f145c103ae0 2024-04-26T01:58:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.726+0000 7f1463800640 1 -- 172.21.15.6:0/1760093335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f145c1a7570 con 0x7f145c103ae0 2024-04-26T01:58:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.731+0000 7f146676e740 1 Processor -- start 2024-04-26T01:58:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.731+0000 7f146676e740 1 -- start start 2024-04-26T01:58:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.731+0000 7f146676e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d2415c820 0x562d2415cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.731+0000 7f146676e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415d140 0x562d241674c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.731+0000 7f146676e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d24167a00 0x562d24169df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.731+0000 7f146676e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x562d240c2900 con 0x562d2415c820 2024-04-26T01:58:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.731+0000 7f146676e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562d24068ef0 con 0x562d2415d140 2024-04-26T01:58:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.731+0000 7f146676e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562d240be8c0 con 0x562d24167a00 2024-04-26T01:58:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f1461800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d2415c820 0x562d2415cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f1461800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d2415c820 0x562d2415cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54948/0 (socket says 172.21.15.6:54948) 2024-04-26T01:58:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f1460e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415d140 0x562d241674c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f1461800640 1 -- 172.21.15.6:0/166579203 learned_addr learned my addr 172.21.15.6:0/166579203 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f1459600640 1 -- 172.21.15.6:0/1760093335 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f1448004e90 con 0x7f145c103ae0 2024-04-26T01:58:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f1461800640 1 -- 172.21.15.6:0/166579203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d24167a00 msgr2=0x562d24169df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f145be00640 1 --2- 172.21.15.6:0/166579203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d24167a00 0x562d24169df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f1461800640 1 --2- 172.21.15.6:0/166579203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d24167a00 0x562d24169df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f1461800640 1 -- 172.21.15.6:0/166579203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415d140 msgr2=0x562d241674c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f1461800640 1 --2- 172.21.15.6:0/166579203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415d140 0x562d241674c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f1461800640 1 -- 172.21.15.6:0/166579203 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d23dd5210 con 0x562d2415c820 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f1460e00640 1 --2- 172.21.15.6:0/166579203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415d140 0x562d241674c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.732+0000 7f145be00640 1 --2- 172.21.15.6:0/166579203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d24167a00 0x562d24169df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.733+0000 7f1461800640 1 --2- 172.21.15.6:0/166579203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d2415c820 0x562d2415cc00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f145400b600 tx=0x7f145400bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9f2de1ae810c1d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.733+0000 7f1452000640 1 -- 172.21.15.6:0/166579203 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454003b20 con 0x562d2415c820 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.733+0000 7f1452000640 1 -- 172.21.15.6:0/166579203 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1454015070 con 0x562d2415c820 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.733+0000 7f1452000640 1 -- 172.21.15.6:0/166579203 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454010440 con 0x562d2415c820 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.733+0000 7f1459600640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f14400a3d10 0x7f14400a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.734+0000 7f146676e740 1 -- 172.21.15.6:0/166579203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d2415c820 msgr2=0x562d2415cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.734+0000 7f146676e740 1 --2- 172.21.15.6:0/166579203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d2415c820 0x562d2415cc00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f145400b600 tx=0x7f145400bac0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.734+0000 7f1459600640 1 -- 172.21.15.6:0/1760093335 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f14480d0ca0 con 0x7f145c103ae0 2024-04-26T01:58:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.734+0000 7f146676e740 1 -- 172.21.15.6:0/166579203 shutdown_connections 2024-04-26T01:58:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.734+0000 7f146676e740 1 --2- 172.21.15.6:0/166579203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d24167a00 0x562d24169df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.734+0000 7f146676e740 1 --2- 172.21.15.6:0/166579203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415d140 0x562d241674c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.734+0000 7f146676e740 1 --2- 172.21.15.6:0/166579203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d2415c820 0x562d2415cc00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.734+0000 7f1460e00640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f14400a3d10 0x7f14400a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.734+0000 7f146676e740 1 -- 172.21.15.6:0/166579203 >> 172.21.15.6:0/166579203 conn(0x562d24158d30 msgr2=0x562d24159110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.734+0000 7f146676e740 1 -- 172.21.15.6:0/166579203 shutdown_connections 2024-04-26T01:58:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.734+0000 7f146676e740 1 -- 172.21.15.6:0/166579203 wait complete. 2024-04-26T01:58:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.734+0000 7f146676e740 1 -- start start 2024-04-26T01:58:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.735+0000 7f1460e00640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f14400a3d10 0x7f14400a61d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f145c19b9a0 tx=0x7f144c005d80 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.735+0000 7f146676e740 1 Processor -- start 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.735+0000 7f146676e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415c820 0x562d24164900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.735+0000 7f146676e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415d140 0x562d24164e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.735+0000 7f146676e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 0x562d24165380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.735+0000 7f146676e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x562d2416a630 con 0x562d24167a00 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.735+0000 7f146676e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562d2407c310 con 0x562d2415d140 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.735+0000 7f146676e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562d2416a7b0 con 0x562d2415c820 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.735+0000 7f146676e740 10 client.0 _ll_register_callbacks cb 0x562d2404acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.735+0000 7f1461800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 0x562d24165380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.736+0000 7f1460e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415c820 0x562d24164900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.736+0000 7f1460e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415c820 0x562d24164900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:53730/0 (socket says 172.21.15.6:53730) 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.736+0000 7f1461800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 0x562d24165380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54954/0 (socket says 172.21.15.6:54954) 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.736+0000 7f1460e00640 1 -- 172.21.15.6:0/3494826401 learned_addr learned my addr 172.21.15.6:0/3494826401 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.736+0000 7f145be00640 1 --2- 172.21.15.6:0/3494826401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415d140 0x562d24164e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.736+0000 7f1460e00640 1 -- 172.21.15.6:0/3494826401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415d140 msgr2=0x562d24164e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.736+0000 7f1460e00640 1 --2- 172.21.15.6:0/3494826401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415d140 0x562d24164e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.736+0000 7f1460e00640 1 -- 172.21.15.6:0/3494826401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 msgr2=0x562d24165380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.736+0000 7f1460e00640 1 --2- 172.21.15.6:0/3494826401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 0x562d24165380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.736+0000 7f1460e00640 1 -- 172.21.15.6:0/3494826401 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d23ea1620 con 0x562d2415c820 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.736+0000 7f1460e00640 1 --2- 172.21.15.6:0/3494826401 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415c820 0x562d24164900 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f144c035000 tx=0x7f144c0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd025bcad7aa38bf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.736+0000 7f1446a00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.737+0000 7f1446a00640 1 -- 172.21.15.6:0/3494826401 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c034040 con 0x562d2415c820 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.737+0000 7f1446a00640 1 -- 172.21.15.6:0/3494826401 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f144c0041d0 con 0x562d2415c820 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.737+0000 7f146676e740 1 -- 172.21.15.6:0/3494826401 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1454015070 con 0x562d2415c820 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.737+0000 7f1446a00640 1 -- 172.21.15.6:0/3494826401 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f144c0043d0 con 0x562d2415c820 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.737+0000 7f146676e740 10 client.26731 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.737+0000 7f146676e740 1 -- 172.21.15.6:0/3494826401 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562d24176760 con 0x562d2415c820 2024-04-26T01:58:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.737+0000 7f1446a00640 1 -- 172.21.15.6:0/3494826401 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1454015070 con 0x562d2415c820 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.738+0000 7f1446a00640 1 -- 172.21.15.6:0/3494826401 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f144c050070 con 0x562d2415c820 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.738+0000 7f146676e740 10 client.26731 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.738+0000 7f146676e740 10 client.26731 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.738+0000 7f146676e740 20 client.26731 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.738+0000 7f146676e740 1 --2- 172.21.15.6:0/3494826401 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d24177c70 0x562d2417a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.738+0000 7f146676e740 4 client.26731 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.738+0000 7f146676e740 1 -- 172.21.15.6:0/3494826401 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d240d68b0 con 0x562d24177c70 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.738+0000 7f145be00640 1 --2- 172.21.15.6:0/3494826401 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d24177c70 0x562d2417a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.739+0000 7f145be00640 1 --2- 172.21.15.6:0/3494826401 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d24177c70 0x562d2417a060 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4aa001d17369c12 server_cookie=ed5eeb71afeee8ed in_seq=0 out_seq=0 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.739+0000 7f1446a00640 10 client.26731 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.739+0000 7f1446a00640 1 -- 172.21.15.6:0/3494826401 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562d240d68b0 con 0x562d24177c70 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.739+0000 7f1446a00640 10 client.26731 handle_command_reply: tid=0 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.739+0000 7f146676e740 1 client.26731 shutdown 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.740+0000 7f146676e740 1 -- 172.21.15.6:0/3494826401 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415c820 msgr2=0x562d24164900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.740+0000 7f146676e740 1 --2- 172.21.15.6:0/3494826401 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415c820 0x562d24164900 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f144c035000 tx=0x7f144c0095c0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.740+0000 7f146676e740 1 -- 172.21.15.6:0/3494826401 shutdown_connections 2024-04-26T01:58:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.740+0000 7f146676e740 1 --2- 172.21.15.6:0/3494826401 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d24177c70 0x562d2417a060 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.740+0000 7f146676e740 1 --2- 172.21.15.6:0/3494826401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 0x562d24165380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.740+0000 7f146676e740 1 --2- 172.21.15.6:0/3494826401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415d140 0x562d24164e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.740+0000 7f146676e740 1 --2- 172.21.15.6:0/3494826401 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415c820 0x562d24164900 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.740+0000 7f146676e740 1 -- 172.21.15.6:0/3494826401 >> 172.21.15.6:0/3494826401 conn(0x562d24158d30 msgr2=0x562d2415eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.740+0000 7f1446a00640 0 client.26731 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.740+0000 7f146676e740 1 -- 172.21.15.6:0/3494826401 shutdown_connections 2024-04-26T01:58:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.740+0000 7f146676e740 1 -- 172.21.15.6:0/3494826401 wait complete. 2024-04-26T01:58:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.740+0000 7f146676e740 20 client.26731 trim_cache size 0 max 16384 2024-04-26T01:58:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.756+0000 7f146676e740 1 Processor -- start 2024-04-26T01:58:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.756+0000 7f146676e740 1 -- start start 2024-04-26T01:58:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.757+0000 7f146676e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415c820 0x562d241734f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.757+0000 7f146676e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415d140 0x562d2415f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.757+0000 7f146676e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 0x562d24164fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.757+0000 7f146676e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f145c1b28c0 con 0x562d24167a00 2024-04-26T01:58:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.757+0000 7f146676e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f145c1b0190 con 0x562d2415c820 2024-04-26T01:58:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.757+0000 7f146676e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f145c1ae000 con 0x562d2415d140 2024-04-26T01:58:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.757+0000 7f1461800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 0x562d24164fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.757+0000 7f1461800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 0x562d24164fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54968/0 (socket says 172.21.15.6:54968) 2024-04-26T01:58:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.757+0000 7f1461800640 1 -- 172.21.15.6:0/1548104013 learned_addr learned my addr 172.21.15.6:0/1548104013 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.757+0000 7f1460e00640 1 --2- 172.21.15.6:0/1548104013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415c820 0x562d241734f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.757+0000 7f145be00640 1 --2- 172.21.15.6:0/1548104013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415d140 0x562d2415f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.758+0000 7f1461800640 1 -- 172.21.15.6:0/1548104013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415d140 msgr2=0x562d2415f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.758+0000 7f1461800640 1 --2- 172.21.15.6:0/1548104013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415d140 0x562d2415f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.758+0000 7f1461800640 1 -- 172.21.15.6:0/1548104013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415c820 msgr2=0x562d241734f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.758+0000 7f1461800640 1 --2- 172.21.15.6:0/1548104013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415c820 0x562d241734f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.758+0000 7f1461800640 1 -- 172.21.15.6:0/1548104013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d24159900 con 0x562d24167a00 2024-04-26T01:58:32.759 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.758+0000 7f1461800640 1 --2- 172.21.15.6:0/1548104013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 0x562d24164fd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1454011bf0 tx=0x7f1454011b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f157075f4d511d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:32.759 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.758+0000 7f1451600640 1 -- 172.21.15.6:0/1548104013 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454028070 con 0x562d24167a00 2024-04-26T01:58:32.759 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.758+0000 7f1451600640 1 -- 172.21.15.6:0/1548104013 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1454015040 con 0x562d24167a00 2024-04-26T01:58:32.759 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.758+0000 7f1451600640 1 -- 172.21.15.6:0/1548104013 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454003db0 con 0x562d24167a00 2024-04-26T01:58:32.762 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.759+0000 7f146676e740 1 -- 172.21.15.6:0/1548104013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 msgr2=0x562d24164fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:32.762 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.759+0000 7f146676e740 1 --2- 172.21.15.6:0/1548104013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 0x562d24164fd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1454011bf0 tx=0x7f1454011b80 comp rx=0 tx=0).stop 2024-04-26T01:58:32.762 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.759+0000 7f146676e740 1 -- 172.21.15.6:0/1548104013 shutdown_connections 2024-04-26T01:58:32.762 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.759+0000 7f146676e740 1 --2- 172.21.15.6:0/1548104013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d24167a00 0x562d24164fd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.759+0000 7f146676e740 1 --2- 172.21.15.6:0/1548104013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415d140 0x562d2415f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.759+0000 7f146676e740 1 --2- 172.21.15.6:0/1548104013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2415c820 0x562d241734f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.759+0000 7f146676e740 1 -- 172.21.15.6:0/1548104013 >> 172.21.15.6:0/1548104013 conn(0x562d24158d30 msgr2=0x562d241648c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.759+0000 7f146676e740 1 -- 172.21.15.6:0/1548104013 shutdown_connections 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.759+0000 7f146676e740 1 -- 172.21.15.6:0/1548104013 wait complete. 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.759+0000 7f146676e740 1 -- start start 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.760+0000 7f146676e740 1 Processor -- start 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.760+0000 7f146676e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d2415c820 0x562d2416fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.760+0000 7f146676e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415d140 0x562d2416d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.760+0000 7f146676e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d24167a00 0x562d2416d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.760+0000 7f146676e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f145c1ae650 con 0x562d2415c820 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.760+0000 7f146676e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f145c1afac0 con 0x562d24167a00 2024-04-26T01:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.760+0000 7f146676e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f145c1aed20 con 0x562d2415d140 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.760+0000 7f146676e740 10 client.0 _ll_register_callbacks cb 0x7f144c003a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.760+0000 7f145be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415d140 0x562d2416d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f1461800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d24167a00 0x562d2416d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f145be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415d140 0x562d2416d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:53750/0 (socket says 172.21.15.6:53750) 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f145be00640 1 -- 172.21.15.6:0/526244427 learned_addr learned my addr 172.21.15.6:0/526244427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f1460e00640 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d2415c820 0x562d2416fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f145be00640 1 -- 172.21.15.6:0/526244427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d24167a00 msgr2=0x562d2416d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f145be00640 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d24167a00 0x562d2416d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f145be00640 1 -- 172.21.15.6:0/526244427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d2415c820 msgr2=0x562d2416fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f145be00640 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d2415c820 0x562d2416fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f145be00640 1 -- 172.21.15.6:0/526244427 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d2416deb0 con 0x562d2415d140 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f1461800640 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d24167a00 0x562d2416d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:32.764 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f145be00640 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415d140 0x562d2416d140 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f144809c890 tx=0x7f14480d30f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e19885c3428d2ce8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f1446a00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f1446a00640 1 -- 172.21.15.6:0/526244427 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144809cb90 con 0x562d2415d140 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f1446a00640 1 -- 172.21.15.6:0/526244427 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f14480bd0b0 con 0x562d2415d140 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.761+0000 7f146676e740 1 -- 172.21.15.6:0/526244427 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1454015040 con 0x562d2415d140 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.762+0000 7f1446a00640 1 -- 172.21.15.6:0/526244427 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f14480a0f40 con 0x562d2415d140 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.762+0000 7f146676e740 10 client.26737 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.762+0000 7f146676e740 1 -- 172.21.15.6:0/526244427 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562d241de230 con 0x562d2415d140 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.762+0000 7f1446a00640 1 -- 172.21.15.6:0/526244427 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1454015040 con 0x562d2415d140 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.762+0000 7f1446a00640 1 -- 172.21.15.6:0/526244427 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f14480ce650 con 0x562d2415d140 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.762+0000 7f146676e740 10 client.26737 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.762+0000 7f146676e740 10 client.26737 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.762+0000 7f146676e740 20 client.26737 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.762+0000 7f146676e740 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d241de8e0 0x562d241e0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.762+0000 7f146676e740 4 client.26737 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:58:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.762+0000 7f146676e740 1 -- 172.21.15.6:0/526244427 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562d240d0320 con 0x562d241de8e0 2024-04-26T01:58:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.762+0000 7f1460e00640 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d241de8e0 0x562d241e0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.763+0000 7f1460e00640 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d241de8e0 0x562d241e0d00 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35012654e8f34d3d server_cookie=a1165ab300e2832b in_seq=0 out_seq=0 2024-04-26T01:58:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.763+0000 7f1446a00640 10 client.26737 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.799+0000 7f1446a00640 1 -- 172.21.15.6:0/526244427 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562d240d0320 con 0x562d241de8e0 2024-04-26T01:58:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.799+0000 7f1446a00640 10 client.26737 handle_command_reply: tid=0 2024-04-26T01:58:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.799+0000 7f146676e740 1 client.26737 shutdown 2024-04-26T01:58:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.799+0000 7f146676e740 1 -- 172.21.15.6:0/526244427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415d140 msgr2=0x562d2416d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.799+0000 7f146676e740 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415d140 0x562d2416d140 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f144809c890 tx=0x7f14480d30f0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.799+0000 7f146676e740 1 -- 172.21.15.6:0/526244427 shutdown_connections 2024-04-26T01:58:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.799+0000 7f146676e740 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562d241de8e0 0x562d241e0d00 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.799+0000 7f146676e740 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d24167a00 0x562d2416d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.799+0000 7f146676e740 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562d2415d140 0x562d2416d140 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.799+0000 7f146676e740 1 --2- 172.21.15.6:0/526244427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562d2415c820 0x562d2416fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.799+0000 7f146676e740 1 -- 172.21.15.6:0/526244427 >> 172.21.15.6:0/526244427 conn(0x562d24158d30 msgr2=0x562d24165130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.799+0000 7f1446a00640 0 client.26737 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.800+0000 7f146676e740 1 -- 172.21.15.6:0/526244427 shutdown_connections 2024-04-26T01:58:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.800+0000 7f146676e740 1 -- 172.21.15.6:0/526244427 wait complete. 2024-04-26T01:58:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.800+0000 7f146676e740 20 client.26737 trim_cache size 0 max 16384 2024-04-26T01:58:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:58:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:58:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:58:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:58:32.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.801+0000 7f1463800640 1 -- 172.21.15.6:0/1760093335 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f14400a3d10 msgr2=0x7f14400a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:32.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.801+0000 7f1463800640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f14400a3d10 0x7f14400a61d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f145c19b9a0 tx=0x7f144c005d80 comp rx=0 tx=0).stop 2024-04-26T01:58:32.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.801+0000 7f1463800640 1 -- 172.21.15.6:0/1760093335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c103ae0 msgr2=0x7f145c1a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:32.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.801+0000 7f1463800640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c103ae0 0x7f145c1a0bf0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f144800e9f0 tx=0x7f144800edf0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.801+0000 7f1463800640 1 -- 172.21.15.6:0/1760093335 shutdown_connections 2024-04-26T01:58:32.801 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.801+0000 7f1463800640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f14400a3d10 0x7f14400a61d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.801+0000 7f1463800640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f145c10c590 0x7f145c19a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.801+0000 7f1463800640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c103ae0 0x7f145c1a0bf0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.801+0000 7f1463800640 1 --2- 172.21.15.6:0/1760093335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f145c1031c0 0x7f145c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.801+0000 7f1463800640 1 -- 172.21.15.6:0/1760093335 >> 172.21.15.6:0/1760093335 conn(0x7f145c077f70 msgr2=0x7f145c0ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.801+0000 7f1463800640 1 -- 172.21.15.6:0/1760093335 shutdown_connections 2024-04-26T01:58:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:32.801+0000 7f1463800640 1 -- 172.21.15.6:0/1760093335 wait complete. 2024-04-26T01:58:33.637 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.45 MB/sec execute 537 sec latency 19808.736 ms 2024-04-26T01:58:33.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:58:34.287 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:34.638 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.45 MB/sec execute 538 sec latency 20808.882 ms 2024-04-26T01:58:34.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:34 smithi188 ceph-mon[32751]: pgmap v400: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 92 KiB/s wr, 7 op/s 2024-04-26T01:58:34.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:34 smithi029 ceph-mon[32775]: pgmap v400: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 92 KiB/s wr, 7 op/s 2024-04-26T01:58:34.756 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:34 smithi006 ceph-mon[28419]: pgmap v400: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 92 KiB/s wr, 7 op/s 2024-04-26T01:58:35.638 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.44 MB/sec execute 539 sec latency 21809.444 ms 2024-04-26T01:58:36.516 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:36 smithi006 ceph-mon[28419]: pgmap v401: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 92 KiB/s wr, 7 op/s 2024-04-26T01:58:36.639 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.43 MB/sec execute 540 sec latency 22809.572 ms 2024-04-26T01:58:36.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:36 smithi188 ceph-mon[32751]: pgmap v401: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 92 KiB/s wr, 7 op/s 2024-04-26T01:58:36.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:36 smithi029 ceph-mon[32775]: pgmap v401: 97 pgs: 97 active+clean; 134 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 92 KiB/s wr, 7 op/s 2024-04-26T01:58:36.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.856+0000 7f2c24400700 1 -- 172.21.15.6:0/4188345131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c1c108a50 msgr2=0x7f2c1c108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:36.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.856+0000 7f2c24400700 1 --2- 172.21.15.6:0/4188345131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c1c108a50 0x7f2c1c108e30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009990 tx=0x7f2c10009ca0 comp rx=0 tx=0).stop 2024-04-26T01:58:36.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.856+0000 7f2c24400700 1 -- 172.21.15.6:0/4188345131 shutdown_connections 2024-04-26T01:58:36.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.856+0000 7f2c24400700 1 --2- 172.21.15.6:0/4188345131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c1c103450 0x7f2c1c10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:36.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.856+0000 7f2c24400700 1 --2- 172.21.15.6:0/4188345131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c1c108a50 0x7f2c1c108e30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:36.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.856+0000 7f2c24400700 1 --2- 172.21.15.6:0/4188345131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c1c102a90 0x7f2c1c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:36.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.856+0000 7f2c24400700 1 -- 172.21.15.6:0/4188345131 >> 172.21.15.6:0/4188345131 conn(0x7f2c1c079ba0 msgr2=0x7f2c1c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:36.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.856+0000 7f2c24400700 1 -- 172.21.15.6:0/4188345131 shutdown_connections 2024-04-26T01:58:36.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.856+0000 7f2c24400700 1 -- 172.21.15.6:0/4188345131 wait complete. 2024-04-26T01:58:36.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.858+0000 7f2c24400700 1 Processor -- start 2024-04-26T01:58:36.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.859+0000 7f2c24400700 1 -- start start 2024-04-26T01:58:36.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.859+0000 7f2c24400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c1c102a90 0x7f2c1c074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:36.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.859+0000 7f2c24400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c1c103450 0x7f2c1c075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:36.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.859+0000 7f2c24400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c1c108a50 0x7f2c1c0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:36.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.859+0000 7f2c24400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1c112be0 con 0x7f2c1c103450 2024-04-26T01:58:36.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.859+0000 7f2c24400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1c112a60 con 0x7f2c1c108a50 2024-04-26T01:58:36.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.859+0000 7f2c24400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1c112d60 con 0x7f2c1c102a90 2024-04-26T01:58:36.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.860+0000 7f2c21a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c1c102a90 0x7f2c1c074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:36.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.860+0000 7f2c21a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c1c102a90 0x7f2c1c074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:53762/0 (socket says 172.21.15.6:53762) 2024-04-26T01:58:36.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.860+0000 7f2c21a00700 1 -- 172.21.15.6:0/1898410753 learned_addr learned my addr 172.21.15.6:0/1898410753 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:36.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.860+0000 7f2c22400700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c1c108a50 0x7f2c1c0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:36.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.860+0000 7f2c21000700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c1c103450 0x7f2c1c075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.860+0000 7f2c21a00700 1 -- 172.21.15.6:0/1898410753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c1c108a50 msgr2=0x7f2c1c0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.860+0000 7f2c21a00700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c1c108a50 0x7f2c1c0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.860+0000 7f2c21a00700 1 -- 172.21.15.6:0/1898410753 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c1c103450 msgr2=0x7f2c1c075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.860+0000 7f2c21a00700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c1c103450 0x7f2c1c075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.860+0000 7f2c21a00700 1 -- 172.21.15.6:0/1898410753 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1c1a76c0 con 0x7f2c1c102a90 2024-04-26T01:58:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.860+0000 7f2c21000700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c1c103450 0x7f2c1c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:58:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.861+0000 7f2c21a00700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c1c102a90 0x7f2c1c074d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2c100057f0 tx=0x7f2c1000bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.861+0000 7f2c1a000700 1 -- 172.21.15.6:0/1898410753 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c10026070 con 0x7f2c1c102a90 2024-04-26T01:58:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.861+0000 7f2c24400700 1 -- 172.21.15.6:0/1898410753 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c1c1a7950 con 0x7f2c1c102a90 2024-04-26T01:58:36.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.861+0000 7f2c1a000700 1 -- 172.21.15.6:0/1898410753 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c10021410 con 0x7f2c1c102a90 2024-04-26T01:58:36.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.861+0000 7f2c1a000700 1 -- 172.21.15.6:0/1898410753 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c10017650 con 0x7f2c1c102a90 2024-04-26T01:58:36.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.861+0000 7f2c24400700 1 -- 172.21.15.6:0/1898410753 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c1c1a7e30 con 0x7f2c1c102a90 2024-04-26T01:58:36.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.863+0000 7f2c24400700 1 -- 172.21.15.6:0/1898410753 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c1c069730 con 0x7f2c1c102a90 2024-04-26T01:58:36.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.865+0000 7f2c1a000700 1 -- 172.21.15.6:0/1898410753 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f2c10029070 con 0x7f2c1c102a90 2024-04-26T01:58:36.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.866+0000 7f2c1a000700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2c040a3db0 0x7f2c040a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:36.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.867+0000 7f2c1a000700 1 -- 172.21.15.6:0/1898410753 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f2c100d1980 con 0x7f2c1c102a90 2024-04-26T01:58:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.867+0000 7f2c21000700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2c040a3db0 0x7f2c040a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.868+0000 7f2c21000700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2c040a3db0 0x7f2c040a6270 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c07a2c0 tx=0x7f2c08009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:36.869+0000 7f2c1a000700 1 -- 172.21.15.6:0/1898410753 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c10030080 con 0x7f2c1c102a90 2024-04-26T01:58:37.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.149+0000 7f2c24400700 1 -- 172.21.15.6:0/1898410753 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c1c02df80 con 0x7f2c1c102a90 2024-04-26T01:58:37.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.151+0000 7f2c1a000700 1 -- 172.21.15.6:0/1898410753 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11850 (secure 0 0 0) 0x7f2c10013070 con 0x7f2c1c102a90 2024-04-26T01:58:37.150 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:58:37.150 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":43,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:55:18.912671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T01:58:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.154+0000 7f2c24400700 1 -- 172.21.15.6:0/1898410753 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2c040a3db0 msgr2=0x7f2c040a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.154+0000 7f2c24400700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2c040a3db0 0x7f2c040a6270 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c07a2c0 tx=0x7f2c08009260 comp rx=0 tx=0).stop 2024-04-26T01:58:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.154+0000 7f2c24400700 1 -- 172.21.15.6:0/1898410753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c1c102a90 msgr2=0x7f2c1c074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.154+0000 7f2c24400700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c1c102a90 0x7f2c1c074d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2c100057f0 tx=0x7f2c1000bfd0 comp rx=0 tx=0).stop 2024-04-26T01:58:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.154+0000 7f2c24400700 1 -- 172.21.15.6:0/1898410753 shutdown_connections 2024-04-26T01:58:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.154+0000 7f2c24400700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f2c040a3db0 0x7f2c040a6270 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.154+0000 7f2c24400700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c1c108a50 0x7f2c1c0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.154+0000 7f2c24400700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c1c102a90 0x7f2c1c074d40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.154+0000 7f2c24400700 1 --2- 172.21.15.6:0/1898410753 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c1c103450 0x7f2c1c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.154+0000 7f2c24400700 1 -- 172.21.15.6:0/1898410753 >> 172.21.15.6:0/1898410753 conn(0x7f2c1c079ba0 msgr2=0x7f2c1c101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.155+0000 7f2c24400700 1 -- 172.21.15.6:0/1898410753 shutdown_connections 2024-04-26T01:58:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:37.155+0000 7f2c24400700 1 -- 172.21.15.6:0/1898410753 wait complete. 2024-04-26T01:58:37.155 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-26T01:58:37.513 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:37 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1898410753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:37.638 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.43 MB/sec execute 541 sec latency 23809.645 ms 2024-04-26T01:58:37.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:37 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1898410753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:37.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:37 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1898410753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:37.993 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id db2f2455 --release --await' 2024-04-26T01:58:37.993 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id db2f2455 --release --await 2024-04-26T01:58:38.452 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:38.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:38 smithi006 ceph-mon[28419]: pgmap v402: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 833 KiB/s wr, 38 op/s 2024-04-26T01:58:38.639 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.42 MB/sec execute 542 sec latency 24809.755 ms 2024-04-26T01:58:38.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:38 smithi188 ceph-mon[32751]: pgmap v402: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 833 KiB/s wr, 38 op/s 2024-04-26T01:58:38.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:38 smithi029 ceph-mon[32775]: pgmap v402: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 833 KiB/s wr, 38 op/s 2024-04-26T01:58:39.639 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.41 MB/sec execute 543 sec latency 25809.887 ms 2024-04-26T01:58:40.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:40 smithi006 ceph-mon[28419]: pgmap v403: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 755 KiB/s wr, 35 op/s 2024-04-26T01:58:40.639 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436601 3.41 MB/sec execute 544 sec latency 26810.012 ms 2024-04-26T01:58:40.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:40 smithi188 ceph-mon[32751]: pgmap v403: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 755 KiB/s wr, 35 op/s 2024-04-26T01:58:40.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:40 smithi029 ceph-mon[32775]: pgmap v403: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 755 KiB/s wr, 35 op/s 2024-04-26T01:58:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.837+0000 7f0864400700 1 -- 172.21.15.6:0/2985074852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c108a30 msgr2=0x7f085c108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.837+0000 7f0864400700 1 --2- 172.21.15.6:0/2985074852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c108a30 0x7f085c108e10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0850009990 tx=0x7f0850009ca0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.837+0000 7f0864400700 1 -- 172.21.15.6:0/2985074852 shutdown_connections 2024-04-26T01:58:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.837+0000 7f0864400700 1 --2- 172.21.15.6:0/2985074852 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f085c06b0f0 0x7f085c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.837+0000 7f0864400700 1 --2- 172.21.15.6:0/2985074852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c108a30 0x7f085c108e10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.837+0000 7f0864400700 1 --2- 172.21.15.6:0/2985074852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f085c06a750 0x7f085c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.837+0000 7f0864400700 1 -- 172.21.15.6:0/2985074852 >> 172.21.15.6:0/2985074852 conn(0x7f085c0788f0 msgr2=0x7f085c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.838+0000 7f0864400700 1 -- 172.21.15.6:0/2985074852 shutdown_connections 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.838+0000 7f0864400700 1 -- 172.21.15.6:0/2985074852 wait complete. 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.839+0000 7f0864400700 1 Processor -- start 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.839+0000 7f0864400700 1 -- start start 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.839+0000 7f0864400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f085c06a750 0x7f085c19b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.839+0000 7f0864400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c06b0f0 0x7f085c19b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.839+0000 7f0864400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f085c108a30 0x7f085c19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.839+0000 7f0864400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f085c112bd0 con 0x7f085c06b0f0 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.839+0000 7f0864400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f085c112a50 con 0x7f085c06a750 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.839+0000 7f0864400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f085c112d50 con 0x7f085c108a30 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.840+0000 7f0861a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f085c06a750 0x7f085c19b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.840+0000 7f0861a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f085c06a750 0x7f085c19b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:42212/0 (socket says 172.21.15.6:42212) 2024-04-26T01:58:40.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.840+0000 7f0861000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c06b0f0 0x7f085c19b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.840+0000 7f0861a00700 1 -- 172.21.15.6:0/546507231 learned_addr learned my addr 172.21.15.6:0/546507231 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.840+0000 7f0862400700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f085c108a30 0x7f085c19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.840+0000 7f0861a00700 1 -- 172.21.15.6:0/546507231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f085c108a30 msgr2=0x7f085c19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.840+0000 7f0861a00700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f085c108a30 0x7f085c19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.840+0000 7f0861a00700 1 -- 172.21.15.6:0/546507231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c06b0f0 msgr2=0x7f085c19b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.840+0000 7f0861a00700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c06b0f0 0x7f085c19b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.840+0000 7f0861a00700 1 -- 172.21.15.6:0/546507231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085c1a0330 con 0x7f085c06a750 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.841+0000 7f0861a00700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f085c06a750 0x7f085c19b320 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f08500057f0 tx=0x7f085000bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.841+0000 7f085a000700 1 -- 172.21.15.6:0/546507231 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850026070 con 0x7f085c06a750 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.841+0000 7f085a000700 1 -- 172.21.15.6:0/546507231 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0850021410 con 0x7f085c06a750 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.841+0000 7f0864400700 1 -- 172.21.15.6:0/546507231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f085c1a05c0 con 0x7f085c06a750 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.841+0000 7f085a000700 1 -- 172.21.15.6:0/546507231 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850017650 con 0x7f085c06a750 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.841+0000 7f0864400700 1 -- 172.21.15.6:0/546507231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f085c1a7ea0 con 0x7f085c06a750 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.844+0000 7f085a000700 1 -- 172.21.15.6:0/546507231 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f0850029070 con 0x7f085c06a750 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.846+0000 7f086eca1b80 1 Processor -- start 2024-04-26T01:58:40.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.846+0000 7f086eca1b80 1 -- start start 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.846+0000 7f086eca1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 0x5634eb2f7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.846+0000 7f085a000700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f08440a3e30 0x7f08440a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.846+0000 7f086eca1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb301f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.846+0000 7f086eca1b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb3024d0 0x5634eb3048c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.846+0000 7f086eca1b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5634eb249f90 con 0x5634eb3024d0 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.846+0000 7f086eca1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5634eb1f0830 con 0x5634eb2f7660 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.846+0000 7f086eca1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5634eb245f50 con 0x5634eb2f6d40 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f0862400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 0x5634eb2f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f0862400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 0x5634eb2f7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:54884/0 (socket says 172.21.15.6:54884) 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f0862400700 1 -- 172.21.15.6:0/89960504 learned_addr learned my addr 172.21.15.6:0/89960504 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f085a000700 1 -- 172.21.15.6:0/546507231 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f08500d1400 con 0x7f085c06a750 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f0861a00700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f08440a3e30 0x7f08440a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f0861000700 1 --2- 172.21.15.6:0/89960504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb301f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f0862400700 1 --2- 172.21.15.6:0/89960504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb3024d0 0x5634eb3048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f0862400700 1 -- 172.21.15.6:0/89960504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 msgr2=0x5634eb301f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f0862400700 1 --2- 172.21.15.6:0/89960504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb301f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f0862400700 1 -- 172.21.15.6:0/89960504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb3024d0 msgr2=0x5634eb3048c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f0862400700 1 --2- 172.21.15.6:0/89960504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb3024d0 0x5634eb3048c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f0862400700 1 -- 172.21.15.6:0/89960504 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634eb1e6780 con 0x5634eb2f6d40 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.847+0000 7f0861000700 1 --2- 172.21.15.6:0/89960504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb301f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.848+0000 7f0862400700 1 --2- 172.21.15.6:0/89960504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 0x5634eb2f7120 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0854008920 tx=0x7f0854009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62261139d9d3140a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.848+0000 7f0861a00700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f08440a3e30 0x7f08440a62f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f08500a5ac0 tx=0x7f0850021c60 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.848+0000 7f084ea00700 1 -- 172.21.15.6:0/89960504 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854003a00 con 0x5634eb2f6d40 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.848+0000 7f084ea00700 1 -- 172.21.15.6:0/89960504 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0854003ba0 con 0x5634eb2f6d40 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.848+0000 7f084ea00700 1 -- 172.21.15.6:0/89960504 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854004220 con 0x5634eb2f6d40 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.849+0000 7f086eca1b80 1 -- 172.21.15.6:0/89960504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 msgr2=0x5634eb2f7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.849+0000 7f086eca1b80 1 --2- 172.21.15.6:0/89960504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 0x5634eb2f7120 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0854008920 tx=0x7f0854009920 comp rx=0 tx=0).stop 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.849+0000 7f086eca1b80 1 -- 172.21.15.6:0/89960504 shutdown_connections 2024-04-26T01:58:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.849+0000 7f086eca1b80 1 --2- 172.21.15.6:0/89960504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb3024d0 0x5634eb3048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.849+0000 7f086eca1b80 1 --2- 172.21.15.6:0/89960504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 0x5634eb2f7120 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.849+0000 7f086eca1b80 1 --2- 172.21.15.6:0/89960504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb301f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.849+0000 7f086eca1b80 1 -- 172.21.15.6:0/89960504 >> 172.21.15.6:0/89960504 conn(0x5634eb226050 msgr2=0x5634eb2f3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.849+0000 7f086eca1b80 1 -- 172.21.15.6:0/89960504 shutdown_connections 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.849+0000 7f086eca1b80 1 -- 172.21.15.6:0/89960504 wait complete. 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.850+0000 7f086eca1b80 1 -- start start 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.850+0000 7f086eca1b80 1 Processor -- start 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f086eca1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f6d40 0x5634eb301c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f086eca1b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f7660 0x5634eb2fef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f086eca1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb3024d0 0x5634eb2ff470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f086eca1b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5634eb304f60 con 0x5634eb2f7660 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f086eca1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5634eb203960 con 0x5634eb2f6d40 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f086eca1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5634eb3050e0 con 0x5634eb3024d0 2024-04-26T01:58:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f086eca1b80 10 client.0 _ll_register_callbacks cb 0x5634eb1e2820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f0861000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f7660 0x5634eb2fef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f0862400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f6d40 0x5634eb301c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f0861000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f7660 0x5634eb2fef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45610/0 (socket says 172.21.15.6:45610) 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f0862400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f6d40 0x5634eb301c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:42228/0 (socket says 172.21.15.6:42228) 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f0861000700 1 -- 172.21.15.6:0/1786107172 learned_addr learned my addr 172.21.15.6:0/1786107172 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.851+0000 7f0862400700 1 --2- 172.21.15.6:0/1786107172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb3024d0 0x5634eb2ff470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.852+0000 7f0861000700 1 -- 172.21.15.6:0/1786107172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb3024d0 msgr2=0x5634eb2ff470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.852+0000 7f0861000700 1 --2- 172.21.15.6:0/1786107172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb3024d0 0x5634eb2ff470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.852+0000 7f0861000700 1 -- 172.21.15.6:0/1786107172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f6d40 msgr2=0x5634eb301c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.852+0000 7f0861000700 1 --2- 172.21.15.6:0/1786107172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f6d40 0x5634eb301c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.852+0000 7f0861000700 1 -- 172.21.15.6:0/1786107172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634eb1979d0 con 0x5634eb2f7660 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.852+0000 7f0861000700 1 --2- 172.21.15.6:0/1786107172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f7660 0x5634eb2fef30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0848011940 tx=0x7f0848011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5e1f9d73640fef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.852+0000 7f0843400700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.852+0000 7f0843400700 1 -- 172.21.15.6:0/1786107172 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084800f850 con 0x5634eb2f7660 2024-04-26T01:58:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.852+0000 7f086eca1b80 1 -- 172.21.15.6:0/1786107172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0854003ba0 con 0x5634eb2f7660 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.852+0000 7f0843400700 1 -- 172.21.15.6:0/1786107172 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f084800f9f0 con 0x5634eb2f7660 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.853+0000 7f0843400700 1 -- 172.21.15.6:0/1786107172 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0848027050 con 0x5634eb2f7660 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.853+0000 7f086eca1b80 10 client.16566 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.853+0000 7f086eca1b80 1 -- 172.21.15.6:0/1786107172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5634eb1e0bf0 con 0x5634eb2f7660 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.853+0000 7f0843400700 1 -- 172.21.15.6:0/1786107172 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0854003ba0 con 0x5634eb2f7660 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.854+0000 7f0843400700 1 -- 172.21.15.6:0/1786107172 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f0848022030 con 0x5634eb2f7660 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.854+0000 7f086eca1b80 10 client.16566 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.854+0000 7f086eca1b80 10 client.16566 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.854+0000 7f086eca1b80 20 client.16566 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.854+0000 7f086eca1b80 1 --2- 172.21.15.6:0/1786107172 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634eb311d40 0x5634eb314130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.854+0000 7f086eca1b80 4 client.16566 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.854+0000 7f086eca1b80 1 -- 172.21.15.6:0/1786107172 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634eb257780 con 0x5634eb311d40 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.854+0000 7f0862400700 1 --2- 172.21.15.6:0/1786107172 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634eb311d40 0x5634eb314130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.855+0000 7f0862400700 1 --2- 172.21.15.6:0/1786107172 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634eb311d40 0x5634eb314130 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fda1c1be16ea52de server_cookie=436ec4a3aca5cd87 in_seq=0 out_seq=0 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.855+0000 7f0843400700 10 client.16566 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.855+0000 7f0843400700 1 -- 172.21.15.6:0/1786107172 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5634eb257780 con 0x5634eb311d40 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.855+0000 7f0843400700 10 client.16566 handle_command_reply: tid=0 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.855+0000 7f086eca1b80 1 client.16566 shutdown 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.856+0000 7f086eca1b80 1 -- 172.21.15.6:0/1786107172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f7660 msgr2=0x5634eb2fef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.856+0000 7f086eca1b80 1 --2- 172.21.15.6:0/1786107172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f7660 0x5634eb2fef30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0848011940 tx=0x7f0848011c50 comp rx=0 tx=0).stop 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.856+0000 7f086eca1b80 1 -- 172.21.15.6:0/1786107172 shutdown_connections 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.856+0000 7f086eca1b80 1 --2- 172.21.15.6:0/1786107172 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634eb311d40 0x5634eb314130 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.856+0000 7f086eca1b80 1 --2- 172.21.15.6:0/1786107172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb3024d0 0x5634eb2ff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.856+0000 7f086eca1b80 1 --2- 172.21.15.6:0/1786107172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f6d40 0x5634eb301c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.856+0000 7f086eca1b80 1 --2- 172.21.15.6:0/1786107172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f7660 0x5634eb2fef30 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.856+0000 7f0843400700 0 client.16566 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.856+0000 7f086eca1b80 1 -- 172.21.15.6:0/1786107172 >> 172.21.15.6:0/1786107172 conn(0x5634eb226050 msgr2=0x5634eb2f2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.856+0000 7f086eca1b80 1 -- 172.21.15.6:0/1786107172 shutdown_connections 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.856+0000 7f086eca1b80 1 -- 172.21.15.6:0/1786107172 wait complete. 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.856+0000 7f086eca1b80 20 client.16566 trim_cache size 0 max 16384 2024-04-26T01:58:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.878+0000 7f086eca1b80 1 Processor -- start 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.878+0000 7f086eca1b80 1 -- start start 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.878+0000 7f086eca1b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f6d40 0x5634eb30de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.878+0000 7f086eca1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb3199c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.878+0000 7f086eca1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb3024d0 0x5634eb319f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.878+0000 7f086eca1b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f085c1b1ee0 con 0x5634eb2f6d40 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.878+0000 7f086eca1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f085c1af770 con 0x5634eb2f7660 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.878+0000 7f086eca1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f085c1ad5e0 con 0x5634eb3024d0 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.879+0000 7f0862400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f6d40 0x5634eb30de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.879+0000 7f0862400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f6d40 0x5634eb30de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45612/0 (socket says 172.21.15.6:45612) 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.879+0000 7f0862400700 1 -- 172.21.15.6:0/3614145603 learned_addr learned my addr 172.21.15.6:0/3614145603 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.879+0000 7f0861000700 1 --2- 172.21.15.6:0/3614145603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb3199c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.879+0000 7f0862400700 1 --2- 172.21.15.6:0/3614145603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb3024d0 0x5634eb319f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.879+0000 7f0862400700 1 -- 172.21.15.6:0/3614145603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb3024d0 msgr2=0x5634eb319f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.879+0000 7f0862400700 1 --2- 172.21.15.6:0/3614145603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb3024d0 0x5634eb319f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.879+0000 7f0862400700 1 -- 172.21.15.6:0/3614145603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 msgr2=0x5634eb3199c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.879+0000 7f0862400700 1 --2- 172.21.15.6:0/3614145603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb3199c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.879+0000 7f0862400700 1 -- 172.21.15.6:0/3614145603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634eb2f8bb0 con 0x5634eb2f6d40 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.879+0000 7f0862400700 1 --2- 172.21.15.6:0/3614145603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f6d40 0x5634eb30de90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0854008f10 tx=0x7f0854008750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=978140976f6c993e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.880+0000 7f084e000700 1 -- 172.21.15.6:0/3614145603 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085402d020 con 0x5634eb2f6d40 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.880+0000 7f084e000700 1 -- 172.21.15.6:0/3614145603 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0854018c50 con 0x5634eb2f6d40 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.880+0000 7f084e000700 1 -- 172.21.15.6:0/3614145603 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854005700 con 0x5634eb2f6d40 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.880+0000 7f086eca1b80 1 -- 172.21.15.6:0/3614145603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f6d40 msgr2=0x5634eb30de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.880+0000 7f086eca1b80 1 --2- 172.21.15.6:0/3614145603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f6d40 0x5634eb30de90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0854008f10 tx=0x7f0854008750 comp rx=0 tx=0).stop 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.880+0000 7f086eca1b80 1 -- 172.21.15.6:0/3614145603 shutdown_connections 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.880+0000 7f086eca1b80 1 --2- 172.21.15.6:0/3614145603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb3024d0 0x5634eb319f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.880+0000 7f086eca1b80 1 --2- 172.21.15.6:0/3614145603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb2f6d40 0x5634eb30de90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.880+0000 7f086eca1b80 1 --2- 172.21.15.6:0/3614145603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb3199c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.880+0000 7f086eca1b80 1 -- 172.21.15.6:0/3614145603 >> 172.21.15.6:0/3614145603 conn(0x5634eb226050 msgr2=0x5634eb300ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.880+0000 7f086eca1b80 1 -- 172.21.15.6:0/3614145603 shutdown_connections 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.881+0000 7f086eca1b80 1 -- 172.21.15.6:0/3614145603 wait complete. 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.881+0000 7f086eca1b80 1 -- start start 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.881+0000 7f086eca1b80 1 Processor -- start 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f086eca1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 0x5634eb305540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f086eca1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb305a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f086eca1b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb3024d0 0x5634eb378230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f086eca1b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f085c1adc30 con 0x5634eb3024d0 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f086eca1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f085c1af0a0 con 0x5634eb2f7660 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f086eca1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f085c1ae300 con 0x5634eb2f6d40 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f086eca1b80 10 client.0 _ll_register_callbacks cb 0x5634eb1e2820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f0862400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 0x5634eb305540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f0862400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 0x5634eb305540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:54906/0 (socket says 172.21.15.6:54906) 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f0862400700 1 -- 172.21.15.6:0/2702660632 learned_addr learned my addr 172.21.15.6:0/2702660632 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f0862400700 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb3024d0 0x5634eb378230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f0861000700 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb305a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f0862400700 1 -- 172.21.15.6:0/2702660632 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 msgr2=0x5634eb305540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f0862400700 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 0x5634eb305540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f0862400700 1 -- 172.21.15.6:0/2702660632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 msgr2=0x5634eb305a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f0862400700 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb305a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f0862400700 1 -- 172.21.15.6:0/2702660632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634eb30f760 con 0x5634eb3024d0 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.882+0000 7f0861000700 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb305a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.883+0000 7f0862400700 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb3024d0 0x5634eb378230 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0854005b90 tx=0x7f08540090a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adbc0c45f2a61da2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.883+0000 7f0843400700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.883+0000 7f0843400700 1 -- 172.21.15.6:0/2702660632 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085402d030 con 0x5634eb3024d0 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.883+0000 7f0843400700 1 -- 172.21.15.6:0/2702660632 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f0854018860 con 0x5634eb3024d0 2024-04-26T01:58:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.883+0000 7f086eca1b80 1 -- 172.21.15.6:0/2702660632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0854018c50 con 0x5634eb3024d0 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.883+0000 7f0843400700 1 -- 172.21.15.6:0/2702660632 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0854018a60 con 0x5634eb3024d0 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.883+0000 7f086eca1b80 10 client.16572 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.883+0000 7f086eca1b80 1 -- 172.21.15.6:0/2702660632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5634eb3798b0 con 0x5634eb3024d0 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.884+0000 7f0843400700 1 -- 172.21.15.6:0/2702660632 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0854018c50 con 0x5634eb3024d0 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.884+0000 7f0843400700 1 -- 172.21.15.6:0/2702660632 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f0854012040 con 0x5634eb3024d0 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.884+0000 7f086eca1b80 10 client.16572 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.884+0000 7f086eca1b80 10 client.16572 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.884+0000 7f086eca1b80 20 client.16572 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.884+0000 7f086eca1b80 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634eb379ec0 0x5634eb37c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.884+0000 7f086eca1b80 4 client.16572 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "db2f2455", "release": true, "await": true}] 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.884+0000 7f086eca1b80 1 -- 172.21.15.6:0/2702660632 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "db2f2455", "release": true, "await": true}) -- 0x5634eb251820 con 0x5634eb379ec0 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.884+0000 7f0861000700 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634eb379ec0 0x5634eb37c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.885+0000 7f0861000700 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634eb379ec0 0x5634eb37c2b0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ec5d0d25f0bdea1 server_cookie=6a46392f279149d7 in_seq=0 out_seq=0 2024-04-26T01:58:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.885+0000 7f0843400700 10 client.16572 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.893+0000 7f0843400700 1 -- 172.21.15.6:0/2702660632 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5634eb251820 con 0x5634eb379ec0 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.893+0000 7f0843400700 10 client.16572 handle_command_reply: tid=0 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.893+0000 7f086eca1b80 1 client.16572 shutdown 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.894+0000 7f086eca1b80 1 -- 172.21.15.6:0/2702660632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb3024d0 msgr2=0x5634eb378230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.894+0000 7f086eca1b80 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb3024d0 0x5634eb378230 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0854005b90 tx=0x7f08540090a0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.894+0000 7f086eca1b80 1 -- 172.21.15.6:0/2702660632 shutdown_connections 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.894+0000 7f086eca1b80 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634eb3024d0 0x5634eb378230 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.894+0000 7f086eca1b80 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634eb379ec0 0x5634eb37c2b0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.894+0000 7f086eca1b80 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634eb2f6d40 0x5634eb305540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.894+0000 7f086eca1b80 1 --2- 172.21.15.6:0/2702660632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634eb2f7660 0x5634eb305a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.894+0000 7f086eca1b80 1 -- 172.21.15.6:0/2702660632 >> 172.21.15.6:0/2702660632 conn(0x5634eb226050 msgr2=0x5634eb300760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.894+0000 7f0843400700 0 client.16572 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.894+0000 7f086eca1b80 1 -- 172.21.15.6:0/2702660632 shutdown_connections 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.894+0000 7f086eca1b80 1 -- 172.21.15.6:0/2702660632 wait complete. 2024-04-26T01:58:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.894+0000 7f086eca1b80 20 client.16572 trim_cache size 0 max 16384 2024-04-26T01:58:40.908 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T01:58:40.908 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 43, 2024-04-26T01:58:40.908 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T01:58:40.908 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 40, 2024-04-26T01:58:40.908 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T01:58:40.908 INFO:teuthology.orchestra.run.smithi006.stdout: "db2f2455": { 2024-04-26T01:58:40.908 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 40, 2024-04-26T01:58:40.908 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T01:58:40.908 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:58:40.908 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:58:40.908 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 79.2, 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T01:58:40.909 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T01:58:40.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.895+0000 7f0864400700 1 -- 172.21.15.6:0/546507231 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f08440a3e30 msgr2=0x7f08440a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:40.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.895+0000 7f0864400700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f08440a3e30 0x7f08440a62f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f08500a5ac0 tx=0x7f0850021c60 comp rx=0 tx=0).stop 2024-04-26T01:58:40.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.895+0000 7f0864400700 1 -- 172.21.15.6:0/546507231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f085c06a750 msgr2=0x7f085c19b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:40.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.895+0000 7f0864400700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f085c06a750 0x7f085c19b320 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f08500057f0 tx=0x7f085000bfd0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.895+0000 7f0864400700 1 -- 172.21.15.6:0/546507231 shutdown_connections 2024-04-26T01:58:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.895+0000 7f0864400700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f08440a3e30 0x7f08440a62f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.895+0000 7f0864400700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f085c108a30 0x7f085c19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.895+0000 7f0864400700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f085c06a750 0x7f085c19b320 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.895+0000 7f0864400700 1 --2- 172.21.15.6:0/546507231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c06b0f0 0x7f085c19b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.895+0000 7f0864400700 1 -- 172.21.15.6:0/546507231 >> 172.21.15.6:0/546507231 conn(0x7f085c0788f0 msgr2=0x7f085c10def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.896+0000 7f0864400700 1 -- 172.21.15.6:0/546507231 shutdown_connections 2024-04-26T01:58:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:58:40.896+0000 7f0864400700 1 -- 172.21.15.6:0/546507231 wait complete. 2024-04-26T01:58:41.639 INFO:tasks.workunit.client.0.smithi006.stdout: 1 436961 3.40 MB/sec execute 545 sec latency 27072.183 ms 2024-04-26T01:58:41.808 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'db2f2455', seconds elapsed: 35.0 2024-04-26T01:58:42.639 INFO:tasks.workunit.client.0.smithi006.stdout: 1 437832 3.41 MB/sec execute 546 sec latency 24.063 ms 2024-04-26T01:58:42.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:42 smithi188 ceph-mon[32751]: pgmap v404: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 788 KiB/s wr, 41 op/s 2024-04-26T01:58:42.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:42 smithi029 ceph-mon[32775]: pgmap v404: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 788 KiB/s wr, 41 op/s 2024-04-26T01:58:42.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:42 smithi006 ceph-mon[28419]: pgmap v404: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 788 KiB/s wr, 41 op/s 2024-04-26T01:58:43.639 INFO:tasks.workunit.client.0.smithi006.stdout: 1 439032 3.41 MB/sec execute 547 sec latency 23.337 ms 2024-04-26T01:58:44.639 INFO:tasks.workunit.client.0.smithi006.stdout: 1 440079 3.42 MB/sec execute 548 sec latency 18.354 ms 2024-04-26T01:58:44.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:44 smithi188 ceph-mon[32751]: pgmap v405: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 773 KiB/s wr, 37 op/s 2024-04-26T01:58:44.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:44 smithi029 ceph-mon[32775]: pgmap v405: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 773 KiB/s wr, 37 op/s 2024-04-26T01:58:44.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:44 smithi006 ceph-mon[28419]: pgmap v405: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 773 KiB/s wr, 37 op/s 2024-04-26T01:58:45.640 INFO:tasks.workunit.client.0.smithi006.stdout: 1 440577 3.41 MB/sec execute 549 sec latency 23.436 ms 2024-04-26T01:58:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.964+0000 7f78d6600640 1 -- 172.21.15.6:0/2255770350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d010e600 msgr2=0x7f78d0110a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.964+0000 7f78d6600640 1 --2- 172.21.15.6:0/2255770350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d010e600 0x7f78d0110a10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f78c8009ca0 tx=0x7f78c802a940 comp rx=0 tx=0).stop 2024-04-26T01:58:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.964+0000 7f78d6600640 1 -- 172.21.15.6:0/2255770350 shutdown_connections 2024-04-26T01:58:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.964+0000 7f78d6600640 1 --2- 172.21.15.6:0/2255770350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d010e600 0x7f78d0110a10 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.964+0000 7f78d6600640 1 --2- 172.21.15.6:0/2255770350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78d0074c40 0x7f78d010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.964+0000 7f78d6600640 1 --2- 172.21.15.6:0/2255770350 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d0074320 0x7f78d0074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.964+0000 7f78d6600640 1 -- 172.21.15.6:0/2255770350 >> 172.21.15.6:0/2255770350 conn(0x7f78d00fd890 msgr2=0x7f78d00ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.965+0000 7f78d6600640 1 -- 172.21.15.6:0/2255770350 shutdown_connections 2024-04-26T01:58:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.965+0000 7f78d6600640 1 -- 172.21.15.6:0/2255770350 wait complete. 2024-04-26T01:58:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.966+0000 7f78d6600640 1 Processor -- start 2024-04-26T01:58:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.966+0000 7f78d6600640 1 -- start start 2024-04-26T01:58:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.967+0000 7f78d6600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78d0074320 0x7f78d01a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.967+0000 7f78d6600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0074c40 0x7f78d01a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.967+0000 7f78d6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d010e600 0x7f78d019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.967+0000 7f78d6600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f78d01141f0 con 0x7f78d010e600 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.967+0000 7f78d6600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f78d0114070 con 0x7f78d0074c40 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.967+0000 7f78d6600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f78d0114370 con 0x7f78d0074320 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.967+0000 7f78d5200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78d0074320 0x7f78d01a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.967+0000 7f78d5200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78d0074320 0x7f78d01a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:54924/0 (socket says 172.21.15.6:54924) 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.967+0000 7f78d5200640 1 -- 172.21.15.6:0/315767576 learned_addr learned my addr 172.21.15.6:0/315767576 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.967+0000 7f78c7e00640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0074c40 0x7f78d01a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.967+0000 7f78d5c00640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d010e600 0x7f78d019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.968+0000 7f78c7e00640 1 -- 172.21.15.6:0/315767576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78d0074320 msgr2=0x7f78d01a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.968+0000 7f78c7e00640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78d0074320 0x7f78d01a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.968+0000 7f78c7e00640 1 -- 172.21.15.6:0/315767576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d010e600 msgr2=0x7f78d019a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.968+0000 7f78c7e00640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d010e600 0x7f78d019a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.968+0000 7f78c7e00640 1 -- 172.21.15.6:0/315767576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d019ad60 con 0x7f78d0074c40 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.968+0000 7f78c7e00640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0074c40 0x7f78d01a0920 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f78b800e9f0 tx=0x7f78b800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.968+0000 7f78ce000640 1 -- 172.21.15.6:0/315767576 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b800ccc0 con 0x7f78d0074c40 2024-04-26T01:58:45.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.968+0000 7f78ce000640 1 -- 172.21.15.6:0/315767576 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b8005020 con 0x7f78d0074c40 2024-04-26T01:58:45.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.968+0000 7f78ce000640 1 -- 172.21.15.6:0/315767576 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b8004a00 con 0x7f78d0074c40 2024-04-26T01:58:45.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.968+0000 7f78d6600640 1 -- 172.21.15.6:0/315767576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d019b050 con 0x7f78d0074c40 2024-04-26T01:58:45.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.968+0000 7f78d6600640 1 -- 172.21.15.6:0/315767576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d01a72a0 con 0x7f78d0074c40 2024-04-26T01:58:45.970 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.972+0000 7f78daa7b740 1 Processor -- start 2024-04-26T01:58:45.970 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.972+0000 7f78ce000640 1 -- 172.21.15.6:0/315767576 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f78b8004ba0 con 0x7f78d0074c40 2024-04-26T01:58:45.970 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.972+0000 7f78daa7b740 1 -- start start 2024-04-26T01:58:45.970 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.972+0000 7f78daa7b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d24f89b0 0x55a9d24f8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.972+0000 7f78daa7b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f92d0 0x55a9d2503650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.972+0000 7f78daa7b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d2503b90 0x55a9d2505f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.972+0000 7f78daa7b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a9d245e900 con 0x55a9d24f89b0 2024-04-26T01:58:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.972+0000 7f78daa7b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a9d2404ef0 con 0x55a9d2503b90 2024-04-26T01:58:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.972+0000 7f78daa7b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a9d245a8c0 con 0x55a9d24f92d0 2024-04-26T01:58:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78d5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d24f89b0 0x55a9d24f8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78d5200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f92d0 0x55a9d2503650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78c7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d2503b90 0x55a9d2505f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78d5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d24f89b0 0x55a9d24f8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45646/0 (socket says 172.21.15.6:45646) 2024-04-26T01:58:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78d5200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f92d0 0x55a9d2503650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:54930/0 (socket says 172.21.15.6:54930) 2024-04-26T01:58:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78d5c00640 1 -- 172.21.15.6:0/3322534176 learned_addr learned my addr 172.21.15.6:0/3322534176 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:45.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78d5200640 1 -- 172.21.15.6:0/3322534176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d2503b90 msgr2=0x55a9d2505f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:45.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78d5200640 1 --2- 172.21.15.6:0/3322534176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d2503b90 0x55a9d2505f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78d5200640 1 -- 172.21.15.6:0/3322534176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d24f89b0 msgr2=0x55a9d24f8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78d5200640 1 --2- 172.21.15.6:0/3322534176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d24f89b0 0x55a9d24f8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78d5200640 1 -- 172.21.15.6:0/3322534176 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9d2171210 con 0x55a9d24f92d0 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78d5c00640 1 --2- 172.21.15.6:0/3322534176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d24f89b0 0x55a9d24f8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78d5200640 1 --2- 172.21.15.6:0/3322534176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f92d0 0x55a9d2503650 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f78c0009c80 tx=0x7f78c0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0700c24f69d85ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.973+0000 7f78ce000640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f78b40a3d80 0x7f78b40a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.974+0000 7f78c6000640 1 -- 172.21.15.6:0/3322534176 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0004230 con 0x55a9d24f92d0 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.974+0000 7f78d5c00640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f78b40a3d80 0x7f78b40a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.974+0000 7f78ce000640 1 -- 172.21.15.6:0/315767576 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f78b80d06d0 con 0x7f78d0074c40 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.974+0000 7f78c6000640 1 -- 172.21.15.6:0/3322534176 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78c00043d0 con 0x55a9d24f92d0 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.974+0000 7f78c6000640 1 -- 172.21.15.6:0/3322534176 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0004d80 con 0x55a9d24f92d0 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.975+0000 7f78daa7b740 1 -- 172.21.15.6:0/3322534176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f92d0 msgr2=0x55a9d2503650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.975+0000 7f78daa7b740 1 --2- 172.21.15.6:0/3322534176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f92d0 0x55a9d2503650 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f78c0009c80 tx=0x7f78c0011810 comp rx=0 tx=0).stop 2024-04-26T01:58:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.975+0000 7f78d5c00640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f78b40a3d80 0x7f78b40a6240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f78c8005e30 tx=0x7f78c8005c50 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.975+0000 7f78daa7b740 1 -- 172.21.15.6:0/3322534176 shutdown_connections 2024-04-26T01:58:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.975+0000 7f78daa7b740 1 --2- 172.21.15.6:0/3322534176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d2503b90 0x55a9d2505f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.975+0000 7f78daa7b740 1 --2- 172.21.15.6:0/3322534176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f92d0 0x55a9d2503650 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.975+0000 7f78daa7b740 1 --2- 172.21.15.6:0/3322534176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d24f89b0 0x55a9d24f8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.975+0000 7f78daa7b740 1 -- 172.21.15.6:0/3322534176 >> 172.21.15.6:0/3322534176 conn(0x55a9d24f4ec0 msgr2=0x55a9d24f52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.975+0000 7f78daa7b740 1 -- 172.21.15.6:0/3322534176 shutdown_connections 2024-04-26T01:58:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.975+0000 7f78daa7b740 1 -- 172.21.15.6:0/3322534176 wait complete. 2024-04-26T01:58:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.975+0000 7f78daa7b740 1 -- start start 2024-04-26T01:58:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.976+0000 7f78daa7b740 1 Processor -- start 2024-04-26T01:58:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.976+0000 7f78daa7b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d24f89b0 0x55a9d2500b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.976+0000 7f78daa7b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d2501070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.976+0000 7f78daa7b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d2503b90 0x55a9d25015b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.976+0000 7f78daa7b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a9d25067c0 con 0x55a9d24f89b0 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.976+0000 7f78daa7b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a9d2418310 con 0x55a9d24f92d0 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.976+0000 7f78daa7b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a9d2506940 con 0x55a9d2503b90 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.976+0000 7f78daa7b740 10 client.0 _ll_register_callbacks cb 0x55a9d23e6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.977+0000 7f78c7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d2501070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.977+0000 7f78d5c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d2503b90 0x55a9d25015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.977+0000 7f78c7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d2501070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:42296/0 (socket says 172.21.15.6:42296) 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.977+0000 7f78c7e00640 1 -- 172.21.15.6:0/632645305 learned_addr learned my addr 172.21.15.6:0/632645305 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.977+0000 7f78d5200640 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d24f89b0 0x55a9d2500b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.977+0000 7f78d5c00640 1 -- 172.21.15.6:0/632645305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 msgr2=0x55a9d2501070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.977+0000 7f78d5c00640 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d2501070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.977+0000 7f78d5c00640 1 -- 172.21.15.6:0/632645305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d24f89b0 msgr2=0x55a9d2500b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.977+0000 7f78d5c00640 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d24f89b0 0x55a9d2500b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.977+0000 7f78d5c00640 1 -- 172.21.15.6:0/632645305 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9d223d620 con 0x55a9d2503b90 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.977+0000 7f78c7e00640 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d2501070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.977+0000 7f78d5c00640 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d2503b90 0x55a9d25015b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f78c8035000 tx=0x7f78c80095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c836cb4369dda9df server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.978+0000 7f78bea00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.978+0000 7f78bea00640 1 -- 172.21.15.6:0/632645305 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c8055060 con 0x55a9d2503b90 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.978+0000 7f78daa7b740 1 -- 172.21.15.6:0/632645305 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78c00043d0 con 0x55a9d2503b90 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.978+0000 7f78bea00640 1 -- 172.21.15.6:0/632645305 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f78c8030ac0 con 0x55a9d2503b90 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.978+0000 7f78bea00640 1 -- 172.21.15.6:0/632645305 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f78c8034040 con 0x55a9d2503b90 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.978+0000 7f78daa7b740 10 client.26773 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.978+0000 7f78daa7b740 1 -- 172.21.15.6:0/632645305 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55a9d2512a90 con 0x55a9d2503b90 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.979+0000 7f78bea00640 1 -- 172.21.15.6:0/632645305 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78c00043d0 con 0x55a9d2503b90 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.979+0000 7f78bea00640 1 -- 172.21.15.6:0/632645305 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f78c802f070 con 0x55a9d2503b90 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.979+0000 7f78daa7b740 10 client.26773 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.979+0000 7f78daa7b740 10 client.26773 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:58:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.979+0000 7f78daa7b740 20 client.26773 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.979+0000 7f78daa7b740 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a9d2513f00 0x55a9d25162f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.979+0000 7f78daa7b740 4 client.26773 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.979+0000 7f78daa7b740 1 -- 172.21.15.6:0/632645305 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9d24728b0 con 0x55a9d2513f00 2024-04-26T01:58:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.979+0000 7f78d5200640 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a9d2513f00 0x55a9d25162f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.980+0000 7f78d5200640 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a9d2513f00 0x55a9d25162f0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa38de546e3954ba server_cookie=c6a367bd09550fe0 in_seq=0 out_seq=0 2024-04-26T01:58:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.980+0000 7f78bea00640 10 client.26773 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78bea00640 1 -- 172.21.15.6:0/632645305 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a9d24728b0 con 0x55a9d2513f00 2024-04-26T01:58:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78bea00640 10 client.26773 handle_command_reply: tid=0 2024-04-26T01:58:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78daa7b740 1 client.26773 shutdown 2024-04-26T01:58:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78daa7b740 1 -- 172.21.15.6:0/632645305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d2503b90 msgr2=0x55a9d25015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78daa7b740 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d2503b90 0x55a9d25015b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f78c8035000 tx=0x7f78c80095c0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78daa7b740 1 -- 172.21.15.6:0/632645305 shutdown_connections 2024-04-26T01:58:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78daa7b740 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a9d2513f00 0x55a9d25162f0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78daa7b740 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d2503b90 0x55a9d25015b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78daa7b740 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d2501070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78daa7b740 1 --2- 172.21.15.6:0/632645305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d24f89b0 0x55a9d2500b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78daa7b740 1 -- 172.21.15.6:0/632645305 >> 172.21.15.6:0/632645305 conn(0x55a9d24f4ec0 msgr2=0x55a9d24fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78bea00640 0 client.26773 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78daa7b740 1 -- 172.21.15.6:0/632645305 shutdown_connections 2024-04-26T01:58:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.981+0000 7f78daa7b740 1 -- 172.21.15.6:0/632645305 wait complete. 2024-04-26T01:58:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.982+0000 7f78daa7b740 20 client.26773 trim_cache size 0 max 16384 2024-04-26T01:58:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.999+0000 7f78daa7b740 1 Processor -- start 2024-04-26T01:58:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:45.999+0000 7f78daa7b740 1 -- start start 2024-04-26T01:58:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78daa7b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 0x55a9d250f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78daa7b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d24fb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78daa7b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d2503b90 0x55a9d251bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78daa7b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f78d01b2660 con 0x55a9d2503b90 2024-04-26T01:58:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78daa7b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f78d01aff30 con 0x55a9d24f92d0 2024-04-26T01:58:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78daa7b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f78d01adda0 con 0x55a9d24f89b0 2024-04-26T01:58:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78d5200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 0x55a9d250f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78d5200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 0x55a9d250f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:54946/0 (socket says 172.21.15.6:54946) 2024-04-26T01:58:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78d5200640 1 -- 172.21.15.6:0/2594249426 learned_addr learned my addr 172.21.15.6:0/2594249426 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78d5c00640 1 --2- 172.21.15.6:0/2594249426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d2503b90 0x55a9d251bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78d5200640 1 -- 172.21.15.6:0/2594249426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 msgr2=0x55a9d24fb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78d5200640 1 --2- 172.21.15.6:0/2594249426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d24fb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78d5200640 1 -- 172.21.15.6:0/2594249426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d2503b90 msgr2=0x55a9d251bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78d5200640 1 --2- 172.21.15.6:0/2594249426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d2503b90 0x55a9d251bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.000+0000 7f78d5200640 1 -- 172.21.15.6:0/2594249426 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9d24f9c60 con 0x55a9d24f89b0 2024-04-26T01:58:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.001+0000 7f78d5200640 1 --2- 172.21.15.6:0/2594249426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 0x55a9d250f680 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x55a9d24f9f50 tx=0x7f78c0009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d09db9f183c1bac9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.001+0000 7f78c5600640 1 -- 172.21.15.6:0/2594249426 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0032070 con 0x55a9d24f89b0 2024-04-26T01:58:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.001+0000 7f78c5600640 1 -- 172.21.15.6:0/2594249426 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78c001f040 con 0x55a9d24f89b0 2024-04-26T01:58:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.001+0000 7f78c5600640 1 -- 172.21.15.6:0/2594249426 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c00193f0 con 0x55a9d24f89b0 2024-04-26T01:58:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.002+0000 7f78daa7b740 1 -- 172.21.15.6:0/2594249426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 msgr2=0x55a9d250f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.002+0000 7f78daa7b740 1 --2- 172.21.15.6:0/2594249426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 0x55a9d250f680 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x55a9d24f9f50 tx=0x7f78c0009ef0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.002+0000 7f78daa7b740 1 -- 172.21.15.6:0/2594249426 shutdown_connections 2024-04-26T01:58:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.002+0000 7f78daa7b740 1 --2- 172.21.15.6:0/2594249426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d2503b90 0x55a9d251bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.002+0000 7f78daa7b740 1 --2- 172.21.15.6:0/2594249426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d24fb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.002+0000 7f78daa7b740 1 --2- 172.21.15.6:0/2594249426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 0x55a9d250f680 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.002+0000 7f78daa7b740 1 -- 172.21.15.6:0/2594249426 >> 172.21.15.6:0/2594249426 conn(0x55a9d24f4ec0 msgr2=0x55a9d24ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.002+0000 7f78daa7b740 1 -- 172.21.15.6:0/2594249426 shutdown_connections 2024-04-26T01:58:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.002+0000 7f78daa7b740 1 -- 172.21.15.6:0/2594249426 wait complete. 2024-04-26T01:58:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.003+0000 7f78daa7b740 1 -- start start 2024-04-26T01:58:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.003+0000 7f78daa7b740 1 Processor -- start 2024-04-26T01:58:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.003+0000 7f78daa7b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 0x55a9d2508db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.003+0000 7f78daa7b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d25092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.003+0000 7f78daa7b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d2503b90 0x55a9d2578450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.003+0000 7f78daa7b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f78d01ae3f0 con 0x55a9d2503b90 2024-04-26T01:58:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.003+0000 7f78daa7b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f78d01af860 con 0x55a9d24f92d0 2024-04-26T01:58:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.003+0000 7f78daa7b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f78d01aeac0 con 0x55a9d24f89b0 2024-04-26T01:58:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78daa7b740 10 client.0 _ll_register_callbacks cb 0x7f78c802f860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78d5200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 0x55a9d2508db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78d5200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 0x55a9d2508db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:54950/0 (socket says 172.21.15.6:54950) 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78d5200640 1 -- 172.21.15.6:0/3883800415 learned_addr learned my addr 172.21.15.6:0/3883800415 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78c7e00640 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d25092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78d5c00640 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d2503b90 0x55a9d2578450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78d5200640 1 -- 172.21.15.6:0/3883800415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 msgr2=0x55a9d25092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78d5200640 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d25092f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78d5200640 1 -- 172.21.15.6:0/3883800415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d2503b90 msgr2=0x55a9d2578450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78d5200640 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d2503b90 0x55a9d2578450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78d5200640 1 -- 172.21.15.6:0/3883800415 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9d2578a60 con 0x55a9d24f89b0 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78d5200640 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 0x55a9d2508db0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f78c0009620 tx=0x7f78c0009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb09de333d91b47f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.004+0000 7f78bea00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.005+0000 7f78bea00640 1 -- 172.21.15.6:0/3883800415 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c003a060 con 0x55a9d24f89b0 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.005+0000 7f78daa7b740 1 -- 172.21.15.6:0/3883800415 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78c001f040 con 0x55a9d24f89b0 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.005+0000 7f78bea00640 1 -- 172.21.15.6:0/3883800415 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f78c0026440 con 0x55a9d24f89b0 2024-04-26T01:58:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.005+0000 7f78bea00640 1 -- 172.21.15.6:0/3883800415 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f78c0032070 con 0x55a9d24f89b0 2024-04-26T01:58:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.005+0000 7f78daa7b740 10 client.26779 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.005+0000 7f78daa7b740 1 -- 172.21.15.6:0/3883800415 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55a9d2579d70 con 0x55a9d24f89b0 2024-04-26T01:58:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.005+0000 7f78bea00640 1 -- 172.21.15.6:0/3883800415 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78c001f040 con 0x55a9d24f89b0 2024-04-26T01:58:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.005+0000 7f78bea00640 1 -- 172.21.15.6:0/3883800415 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f78c0024040 con 0x55a9d24f89b0 2024-04-26T01:58:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.006+0000 7f78daa7b740 10 client.26779 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.006+0000 7f78daa7b740 10 client.26779 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:58:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.006+0000 7f78daa7b740 20 client.26779 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.006+0000 7f78daa7b740 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a9d257a1e0 0x55a9d257c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.006+0000 7f78daa7b740 4 client.26779 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:58:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.006+0000 7f78daa7b740 1 -- 172.21.15.6:0/3883800415 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a9d246c320 con 0x55a9d257a1e0 2024-04-26T01:58:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.006+0000 7f78c7e00640 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a9d257a1e0 0x55a9d257c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.006+0000 7f78c7e00640 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a9d257a1e0 0x55a9d257c5d0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb97af3df888c33f server_cookie=fc1c54f15fabf70c in_seq=0 out_seq=0 2024-04-26T01:58:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.006+0000 7f78bea00640 10 client.26779 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:46.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.049+0000 7f78bea00640 1 -- 172.21.15.6:0/3883800415 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a9d246c320 con 0x55a9d257a1e0 2024-04-26T01:58:46.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.049+0000 7f78bea00640 10 client.26779 handle_command_reply: tid=0 2024-04-26T01:58:46.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.049+0000 7f78daa7b740 1 client.26779 shutdown 2024-04-26T01:58:46.048 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.049+0000 7f78daa7b740 1 -- 172.21.15.6:0/3883800415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 msgr2=0x55a9d2508db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.049+0000 7f78daa7b740 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 0x55a9d2508db0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f78c0009620 tx=0x7f78c0009700 comp rx=0 tx=0).stop 2024-04-26T01:58:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.049+0000 7f78daa7b740 1 -- 172.21.15.6:0/3883800415 shutdown_connections 2024-04-26T01:58:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.049+0000 7f78daa7b740 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a9d257a1e0 0x55a9d257c5d0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.049+0000 7f78daa7b740 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a9d2503b90 0x55a9d2578450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.049+0000 7f78daa7b740 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a9d24f92d0 0x55a9d25092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.049+0000 7f78daa7b740 1 --2- 172.21.15.6:0/3883800415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a9d24f89b0 0x55a9d2508db0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.049+0000 7f78bea00640 0 client.26779 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.049+0000 7f78daa7b740 1 -- 172.21.15.6:0/3883800415 >> 172.21.15.6:0/3883800415 conn(0x55a9d24f4ec0 msgr2=0x55a9d2500af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.050+0000 7f78daa7b740 1 -- 172.21.15.6:0/3883800415 shutdown_connections 2024-04-26T01:58:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.050+0000 7f78daa7b740 1 -- 172.21.15.6:0/3883800415 wait complete. 2024-04-26T01:58:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.050+0000 7f78daa7b740 20 client.26779 trim_cache size 0 max 16384 2024-04-26T01:58:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:58:46.050 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:58:46.050 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:58:46.050 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:58:46.050 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.050+0000 7f78d6600640 1 -- 172.21.15.6:0/315767576 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f78b40a3d80 msgr2=0x7f78b40a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.050+0000 7f78d6600640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f78b40a3d80 0x7f78b40a6240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f78c8005e30 tx=0x7f78c8005c50 comp rx=0 tx=0).stop 2024-04-26T01:58:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.050+0000 7f78d6600640 1 -- 172.21.15.6:0/315767576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0074c40 msgr2=0x7f78d01a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.050+0000 7f78d6600640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0074c40 0x7f78d01a0920 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f78b800e9f0 tx=0x7f78b800eeb0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.051+0000 7f78d6600640 1 -- 172.21.15.6:0/315767576 shutdown_connections 2024-04-26T01:58:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.051+0000 7f78d6600640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f78b40a3d80 0x7f78b40a6240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.051+0000 7f78d6600640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d010e600 0x7f78d019a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.051+0000 7f78d6600640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0074c40 0x7f78d01a0920 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.051+0000 7f78d6600640 1 --2- 172.21.15.6:0/315767576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78d0074320 0x7f78d01a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.051+0000 7f78d6600640 1 -- 172.21.15.6:0/315767576 >> 172.21.15.6:0/315767576 conn(0x7f78d00fd890 msgr2=0x7f78d00fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.051+0000 7f78d6600640 1 -- 172.21.15.6:0/315767576 shutdown_connections 2024-04-26T01:58:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:46.051+0000 7f78d6600640 1 -- 172.21.15.6:0/315767576 wait complete. 2024-04-26T01:58:46.640 INFO:tasks.workunit.client.0.smithi006.stdout: 1 441514 3.42 MB/sec execute 550 sec latency 32.374 ms 2024-04-26T01:58:46.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:46 smithi188 ceph-mon[32751]: pgmap v406: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 773 KiB/s wr, 37 op/s 2024-04-26T01:58:46.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:46 smithi029 ceph-mon[32775]: pgmap v406: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 773 KiB/s wr, 37 op/s 2024-04-26T01:58:46.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:46 smithi006 ceph-mon[28419]: pgmap v406: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 773 KiB/s wr, 37 op/s 2024-04-26T01:58:47.640 INFO:tasks.workunit.client.0.smithi006.stdout: 1 442733 3.42 MB/sec execute 551 sec latency 26.178 ms 2024-04-26T01:58:48.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:48 smithi006 ceph-mon[28419]: pgmap v407: 97 pgs: 97 active+clean; 139 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 109 op/s 2024-04-26T01:58:48.640 INFO:tasks.workunit.client.0.smithi006.stdout: 1 443680 3.43 MB/sec execute 552 sec latency 17.931 ms 2024-04-26T01:58:48.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:48 smithi188 ceph-mon[32751]: pgmap v407: 97 pgs: 97 active+clean; 139 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 109 op/s 2024-04-26T01:58:48.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:48 smithi029 ceph-mon[32775]: pgmap v407: 97 pgs: 97 active+clean; 139 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 109 op/s 2024-04-26T01:58:49.640 INFO:tasks.workunit.client.0.smithi006.stdout: 1 444528 3.43 MB/sec execute 553 sec latency 21.646 ms 2024-04-26T01:58:50.640 INFO:tasks.workunit.client.0.smithi006.stdout: 1 445231 3.43 MB/sec execute 554 sec latency 22.887 ms 2024-04-26T01:58:50.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:50 smithi029 ceph-mon[32775]: pgmap v408: 97 pgs: 97 active+clean; 139 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 509 KiB/s wr, 77 op/s 2024-04-26T01:58:50.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:50 smithi006 ceph-mon[28419]: pgmap v408: 97 pgs: 97 active+clean; 139 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 509 KiB/s wr, 77 op/s 2024-04-26T01:58:50.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:50 smithi188 ceph-mon[32751]: pgmap v408: 97 pgs: 97 active+clean; 139 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 509 KiB/s wr, 77 op/s 2024-04-26T01:58:51.641 INFO:tasks.workunit.client.0.smithi006.stdout: 1 446385 3.43 MB/sec execute 555 sec latency 45.685 ms 2024-04-26T01:58:52.641 INFO:tasks.workunit.client.0.smithi006.stdout: 1 447404 3.44 MB/sec execute 556 sec latency 19.649 ms 2024-04-26T01:58:52.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:52 smithi029 ceph-mon[32775]: pgmap v409: 97 pgs: 97 active+clean; 138 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 892 KiB/s wr, 135 op/s 2024-04-26T01:58:52.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:52 smithi006 ceph-mon[28419]: pgmap v409: 97 pgs: 97 active+clean; 138 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 892 KiB/s wr, 135 op/s 2024-04-26T01:58:52.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:52 smithi188 ceph-mon[32751]: pgmap v409: 97 pgs: 97 active+clean; 138 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 892 KiB/s wr, 135 op/s 2024-04-26T01:58:53.641 INFO:tasks.workunit.client.0.smithi006.stdout: 1 448469 3.44 MB/sec execute 557 sec latency 25.755 ms 2024-04-26T01:58:54.641 INFO:tasks.workunit.client.0.smithi006.stdout: 1 449878 3.45 MB/sec execute 558 sec latency 23.574 ms 2024-04-26T01:58:54.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:54 smithi029 ceph-mon[32775]: pgmap v410: 97 pgs: 97 active+clean; 138 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 859 KiB/s wr, 128 op/s 2024-04-26T01:58:54.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:54 smithi006 ceph-mon[28419]: pgmap v410: 97 pgs: 97 active+clean; 138 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 859 KiB/s wr, 128 op/s 2024-04-26T01:58:54.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:54 smithi188 ceph-mon[32751]: pgmap v410: 97 pgs: 97 active+clean; 138 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 859 KiB/s wr, 128 op/s 2024-04-26T01:58:55.641 INFO:tasks.workunit.client.0.smithi006.stdout: 1 450887 3.45 MB/sec execute 559 sec latency 19.174 ms 2024-04-26T01:58:56.641 INFO:tasks.workunit.client.0.smithi006.stdout: 1 451650 3.45 MB/sec execute 560 sec latency 46.187 ms 2024-04-26T01:58:56.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:56 smithi029 ceph-mon[32775]: pgmap v411: 97 pgs: 97 active+clean; 138 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 859 KiB/s wr, 128 op/s 2024-04-26T01:58:56.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:56 smithi006 ceph-mon[28419]: pgmap v411: 97 pgs: 97 active+clean; 138 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 859 KiB/s wr, 128 op/s 2024-04-26T01:58:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:56 smithi188 ceph-mon[32751]: pgmap v411: 97 pgs: 97 active+clean; 138 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 859 KiB/s wr, 128 op/s 2024-04-26T01:58:57.641 INFO:tasks.workunit.client.0.smithi006.stdout: 1 452698 3.45 MB/sec execute 561 sec latency 30.577 ms 2024-04-26T01:58:58.641 INFO:tasks.workunit.client.0.smithi006.stdout: 1 454020 3.47 MB/sec execute 562 sec latency 24.077 ms 2024-04-26T01:58:58.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:58:58 smithi029 ceph-mon[32775]: pgmap v412: 97 pgs: 97 active+clean; 144 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.3 MiB/s wr, 197 op/s 2024-04-26T01:58:58.692 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:58:58 smithi188 ceph-mon[32751]: pgmap v412: 97 pgs: 97 active+clean; 144 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.3 MiB/s wr, 197 op/s 2024-04-26T01:58:58.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:58:58 smithi006 ceph-mon[28419]: pgmap v412: 97 pgs: 97 active+clean; 144 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.3 MiB/s wr, 197 op/s 2024-04-26T01:58:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.204+0000 7f85af200640 1 -- 172.21.15.6:0/2917149241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85a8106190 msgr2=0x7f85a810dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.204+0000 7f85af200640 1 --2- 172.21.15.6:0/2917149241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85a8106190 0x7f85a810dd60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009900 tx=0x7f85a002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.204+0000 7f85af200640 1 -- 172.21.15.6:0/2917149241 shutdown_connections 2024-04-26T01:58:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.204+0000 7f85af200640 1 --2- 172.21.15.6:0/2917149241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85a8101a70 0x7f85a8105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.204+0000 7f85af200640 1 --2- 172.21.15.6:0/2917149241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85a8106190 0x7f85a810dd60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.204+0000 7f85af200640 1 --2- 172.21.15.6:0/2917149241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85a81010c0 0x7f85a81014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.204+0000 7f85af200640 1 -- 172.21.15.6:0/2917149241 >> 172.21.15.6:0/2917149241 conn(0x7f85a8077fd0 msgr2=0x7f85a80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:59.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.204+0000 7f85af200640 1 -- 172.21.15.6:0/2917149241 shutdown_connections 2024-04-26T01:58:59.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.204+0000 7f85af200640 1 -- 172.21.15.6:0/2917149241 wait complete. 2024-04-26T01:58:59.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.205+0000 7f85af200640 1 Processor -- start 2024-04-26T01:58:59.203 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.205+0000 7f85af200640 1 -- start start 2024-04-26T01:58:59.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.206+0000 7f85af200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85a81010c0 0x7f85a81a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.206+0000 7f85af200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85a8106190 0x7f85a81a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.206+0000 7f85af200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85a819a780 0x7f85a819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.206+0000 7f85af200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f85a8111150 con 0x7f85a819a780 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.206+0000 7f85af200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f85a8110fd0 con 0x7f85a8106190 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.206+0000 7f85af200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f85a81112d0 con 0x7f85a81010c0 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.206+0000 7f85a7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85a81010c0 0x7f85a81a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.206+0000 7f85a7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85a81010c0 0x7f85a81a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:52994/0 (socket says 172.21.15.6:52994) 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.206+0000 7f85a7e00640 1 -- 172.21.15.6:0/4261548902 learned_addr learned my addr 172.21.15.6:0/4261548902 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.206+0000 7f85a7400640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85a8106190 0x7f85a81a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.206+0000 7f85ad200640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85a819a780 0x7f85a819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.207+0000 7f85a7e00640 1 -- 172.21.15.6:0/4261548902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85a8106190 msgr2=0x7f85a81a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.207+0000 7f85a7e00640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85a8106190 0x7f85a81a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.207+0000 7f85a7e00640 1 -- 172.21.15.6:0/4261548902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85a819a780 msgr2=0x7f85a819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.207+0000 7f85a7e00640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85a819a780 0x7f85a819ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.207+0000 7f85a7e00640 1 -- 172.21.15.6:0/4261548902 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a819b310 con 0x7f85a81010c0 2024-04-26T01:58:59.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.207+0000 7f85a7e00640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85a81010c0 0x7f85a81a0610 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f859800eb90 tx=0x7f859800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:59.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.207+0000 7f85a4c00640 1 -- 172.21.15.6:0/4261548902 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598004d20 con 0x7f85a81010c0 2024-04-26T01:58:59.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.207+0000 7f85a4c00640 1 -- 172.21.15.6:0/4261548902 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f859800ee90 con 0x7f85a81010c0 2024-04-26T01:58:59.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.207+0000 7f85a4c00640 1 -- 172.21.15.6:0/4261548902 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598010670 con 0x7f85a81010c0 2024-04-26T01:58:59.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.208+0000 7f85af200640 1 -- 172.21.15.6:0/4261548902 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a819b600 con 0x7f85a81010c0 2024-04-26T01:58:59.207 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.208+0000 7f85af200640 1 -- 172.21.15.6:0/4261548902 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a81a7790 con 0x7f85a81010c0 2024-04-26T01:58:59.209 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.211+0000 7f85a4c00640 1 -- 172.21.15.6:0/4261548902 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f8598005020 con 0x7f85a81010c0 2024-04-26T01:58:59.210 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.212+0000 7f85b21a4740 1 Processor -- start 2024-04-26T01:58:59.210 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.212+0000 7f85b21a4740 1 -- start start 2024-04-26T01:58:59.210 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.212+0000 7f85a4c00640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f858c0a3d10 0x7f858c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85a4c00640 1 -- 172.21.15.6:0/4261548902 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f85980d0ee0 con 0x7f85a81010c0 2024-04-26T01:58:59.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85b21a4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b10820 0x55c073b10c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85b21a4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b11140 0x55c073b1b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85b21a4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b1ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85ad200640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f858c0a3d10 0x7f858c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85b21a4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55c073a76900 con 0x55c073b10820 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85b21a4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c073a1cef0 con 0x55c073b11140 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85b21a4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c073a728c0 con 0x55c073b1ba00 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85a7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b10820 0x55c073b10c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85ad200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b1ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85a7400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b11140 0x55c073b1b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85a7400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b11140 0x55c073b1b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:60844/0 (socket says 172.21.15.6:60844) 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85a7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b10820 0x55c073b10c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37178/0 (socket says 172.21.15.6:37178) 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.213+0000 7f85a7400640 1 -- 172.21.15.6:0/2611747290 learned_addr learned my addr 172.21.15.6:0/2611747290 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.214+0000 7f85a7400640 1 -- 172.21.15.6:0/2611747290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 msgr2=0x55c073b1ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.214+0000 7f85a7400640 1 --2- 172.21.15.6:0/2611747290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b1ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.214+0000 7f85a7400640 1 -- 172.21.15.6:0/2611747290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b10820 msgr2=0x55c073b10c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:59.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.214+0000 7f85a7400640 1 --2- 172.21.15.6:0/2611747290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b10820 0x55c073b10c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.214+0000 7f85a7400640 1 -- 172.21.15.6:0/2611747290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c073789210 con 0x55c073b11140 2024-04-26T01:58:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.214+0000 7f85a7400640 1 --2- 172.21.15.6:0/2611747290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b11140 0x55c073b1b4c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8590009ba0 tx=0x7f8590011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8cbce0af55c9f0c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.214+0000 7f85ad200640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f858c0a3d10 0x7f858c0a61d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f85a0006010 tx=0x7f85a0009680 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.214+0000 7f859d600640 1 -- 172.21.15.6:0/2611747290 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8590004230 con 0x55c073b11140 2024-04-26T01:58:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.215+0000 7f859d600640 1 -- 172.21.15.6:0/2611747290 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8590009e10 con 0x55c073b11140 2024-04-26T01:58:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.215+0000 7f859d600640 1 -- 172.21.15.6:0/2611747290 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8590004d10 con 0x55c073b11140 2024-04-26T01:58:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.215+0000 7f85b21a4740 1 -- 172.21.15.6:0/2611747290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b11140 msgr2=0x55c073b1b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.215+0000 7f85b21a4740 1 --2- 172.21.15.6:0/2611747290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b11140 0x55c073b1b4c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8590009ba0 tx=0x7f8590011810 comp rx=0 tx=0).stop 2024-04-26T01:58:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.216+0000 7f85b21a4740 1 -- 172.21.15.6:0/2611747290 shutdown_connections 2024-04-26T01:58:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.216+0000 7f85b21a4740 1 --2- 172.21.15.6:0/2611747290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b1ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.216+0000 7f85b21a4740 1 --2- 172.21.15.6:0/2611747290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b11140 0x55c073b1b4c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.216+0000 7f85b21a4740 1 --2- 172.21.15.6:0/2611747290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b10820 0x55c073b10c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.216+0000 7f85b21a4740 1 -- 172.21.15.6:0/2611747290 >> 172.21.15.6:0/2611747290 conn(0x55c073b0cd30 msgr2=0x55c073b0d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.216+0000 7f85b21a4740 1 -- 172.21.15.6:0/2611747290 shutdown_connections 2024-04-26T01:58:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.216+0000 7f85b21a4740 1 -- 172.21.15.6:0/2611747290 wait complete. 2024-04-26T01:58:59.216 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.216+0000 7f85b21a4740 1 -- start start 2024-04-26T01:58:59.216 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.217+0000 7f85b21a4740 1 Processor -- start 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.217+0000 7f85b21a4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b18960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.217+0000 7f85b21a4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b11140 0x55c073b18ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.217+0000 7f85b21a4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b193e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.217+0000 7f85b21a4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55c073b1e630 con 0x55c073b11140 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.217+0000 7f85b21a4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c073a30310 con 0x55c073b10820 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.217+0000 7f85b21a4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c073b1e7b0 con 0x55c073b1ba00 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.217+0000 7f85b21a4740 10 client.0 _ll_register_callbacks cb 0x55c0739fecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.218+0000 7f85ad200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b193e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.218+0000 7f85ad200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b193e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:53014/0 (socket says 172.21.15.6:53014) 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.218+0000 7f85ad200640 1 -- 172.21.15.6:0/40421977 learned_addr learned my addr 172.21.15.6:0/40421977 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.218+0000 7f85a7400640 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b11140 0x55c073b18ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.218+0000 7f85a7e00640 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b18960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.218+0000 7f85ad200640 1 -- 172.21.15.6:0/40421977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 msgr2=0x55c073b18960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.218+0000 7f85ad200640 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b18960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.218+0000 7f85ad200640 1 -- 172.21.15.6:0/40421977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b11140 msgr2=0x55c073b18ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:59.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.218+0000 7f85ad200640 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b11140 0x55c073b18ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.218+0000 7f85ad200640 1 -- 172.21.15.6:0/40421977 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c073855620 con 0x55c073b1ba00 2024-04-26T01:58:59.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.218+0000 7f85a7400640 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b11140 0x55c073b18ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:59.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.219+0000 7f85ad200640 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b193e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f85a0031440 tx=0x7f85a002a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c39fce6052d020e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:59.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.219+0000 7f8596000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:58:59.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.219+0000 7f8596000640 1 -- 172.21.15.6:0/40421977 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0050070 con 0x55c073b1ba00 2024-04-26T01:58:59.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.219+0000 7f85b21a4740 1 -- 172.21.15.6:0/40421977 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8590009e10 con 0x55c073b1ba00 2024-04-26T01:58:59.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.219+0000 7f8596000640 1 -- 172.21.15.6:0/40421977 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f85a00318b0 con 0x55c073b1ba00 2024-04-26T01:58:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.219+0000 7f8596000640 1 -- 172.21.15.6:0/40421977 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f85a005c050 con 0x55c073b1ba00 2024-04-26T01:58:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.219+0000 7f85b21a4740 10 client.26791 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.219+0000 7f85b21a4740 1 -- 172.21.15.6:0/40421977 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55c073b2a850 con 0x55c073b1ba00 2024-04-26T01:58:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.220+0000 7f8596000640 1 -- 172.21.15.6:0/40421977 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8590009e10 con 0x55c073b1ba00 2024-04-26T01:58:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.220+0000 7f8596000640 1 -- 172.21.15.6:0/40421977 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f85a004cab0 con 0x55c073b1ba00 2024-04-26T01:58:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.220+0000 7f85b21a4740 10 client.26791 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.220+0000 7f85b21a4740 10 client.26791 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.220+0000 7f85b21a4740 20 client.26791 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.220+0000 7f85b21a4740 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c073b2bd10 0x55c073b2e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.220+0000 7f85b21a4740 4 client.26791 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.220+0000 7f85b21a4740 1 -- 172.21.15.6:0/40421977 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c073a8a8b0 con 0x55c073b2bd10 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.220+0000 7f85a7e00640 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c073b2bd10 0x55c073b2e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.221+0000 7f85a7e00640 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c073b2bd10 0x55c073b2e100 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5859ae75ee343155 server_cookie=5829fe09f923edec in_seq=0 out_seq=0 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.221+0000 7f8596000640 10 client.26791 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f8596000640 1 -- 172.21.15.6:0/40421977 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c073a8a8b0 con 0x55c073b2bd10 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f8596000640 10 client.26791 handle_command_reply: tid=0 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f85b21a4740 1 client.26791 shutdown 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f85b21a4740 1 -- 172.21.15.6:0/40421977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 msgr2=0x55c073b193e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f85b21a4740 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b193e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f85a0031440 tx=0x7f85a002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f85b21a4740 1 -- 172.21.15.6:0/40421977 shutdown_connections 2024-04-26T01:58:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f85b21a4740 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c073b2bd10 0x55c073b2e100 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f85b21a4740 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b11140 0x55c073b18ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f85b21a4740 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b193e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f8596000640 0 client.26791 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f85b21a4740 1 --2- 172.21.15.6:0/40421977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b18960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f85b21a4740 1 -- 172.21.15.6:0/40421977 >> 172.21.15.6:0/40421977 conn(0x55c073b0cd30 msgr2=0x55c073b12ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f85b21a4740 1 -- 172.21.15.6:0/40421977 shutdown_connections 2024-04-26T01:58:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.222+0000 7f85b21a4740 1 -- 172.21.15.6:0/40421977 wait complete. 2024-04-26T01:58:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.223+0000 7f85b21a4740 20 client.26791 trim_cache size 0 max 16384 2024-04-26T01:58:59.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.242+0000 7f85b21a4740 1 Processor -- start 2024-04-26T01:58:59.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.242+0000 7f85b21a4740 1 -- start start 2024-04-26T01:58:59.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.242+0000 7f85b21a4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b274f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.242+0000 7f85b21a4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b11140 0x55c073b13230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.242+0000 7f85b21a4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b19030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.242+0000 7f85b21a4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f85a81b2850 con 0x55c073b11140 2024-04-26T01:58:59.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.242+0000 7f85b21a4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f85a81b0120 con 0x55c073b10820 2024-04-26T01:58:59.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.242+0000 7f85b21a4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f85a81adf90 con 0x55c073b1ba00 2024-04-26T01:58:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.243+0000 7f85a7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b274f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.243+0000 7f85a7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b274f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:60868/0 (socket says 172.21.15.6:60868) 2024-04-26T01:58:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.243+0000 7f85a7e00640 1 -- 172.21.15.6:0/427864438 learned_addr learned my addr 172.21.15.6:0/427864438 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.243+0000 7f85ad200640 1 --2- 172.21.15.6:0/427864438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b19030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.243+0000 7f85a7400640 1 --2- 172.21.15.6:0/427864438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b11140 0x55c073b13230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.243+0000 7f85a7e00640 1 -- 172.21.15.6:0/427864438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 msgr2=0x55c073b19030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.243+0000 7f85a7e00640 1 --2- 172.21.15.6:0/427864438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b19030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.243+0000 7f85a7e00640 1 -- 172.21.15.6:0/427864438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b11140 msgr2=0x55c073b13230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.243+0000 7f85a7e00640 1 --2- 172.21.15.6:0/427864438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b11140 0x55c073b13230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.243+0000 7f85a7e00640 1 -- 172.21.15.6:0/427864438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c073b10f70 con 0x55c073b10820 2024-04-26T01:58:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.243+0000 7f85a7400640 1 --2- 172.21.15.6:0/427864438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b11140 0x55c073b13230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:58:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.244+0000 7f85a7e00640 1 --2- 172.21.15.6:0/427864438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b274f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f859809cd60 tx=0x7f85980bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eecad719d2b84298 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.244+0000 7f859cc00640 1 -- 172.21.15.6:0/427864438 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85980a0040 con 0x55c073b10820 2024-04-26T01:58:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.244+0000 7f859cc00640 1 -- 172.21.15.6:0/427864438 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f859809b0d0 con 0x55c073b10820 2024-04-26T01:58:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.244+0000 7f859cc00640 1 -- 172.21.15.6:0/427864438 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85980a55b0 con 0x55c073b10820 2024-04-26T01:58:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.245+0000 7f85b21a4740 1 -- 172.21.15.6:0/427864438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 msgr2=0x55c073b274f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.245+0000 7f85b21a4740 1 --2- 172.21.15.6:0/427864438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b274f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f859809cd60 tx=0x7f85980bd040 comp rx=0 tx=0).stop 2024-04-26T01:58:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.245+0000 7f85b21a4740 1 -- 172.21.15.6:0/427864438 shutdown_connections 2024-04-26T01:58:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.245+0000 7f85b21a4740 1 --2- 172.21.15.6:0/427864438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b11140 0x55c073b13230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.245+0000 7f85b21a4740 1 --2- 172.21.15.6:0/427864438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b1ba00 0x55c073b19030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.245+0000 7f85b21a4740 1 --2- 172.21.15.6:0/427864438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b274f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.245+0000 7f85b21a4740 1 -- 172.21.15.6:0/427864438 >> 172.21.15.6:0/427864438 conn(0x55c073b0cd30 msgr2=0x55c073b18920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.245+0000 7f85b21a4740 1 -- 172.21.15.6:0/427864438 shutdown_connections 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.245+0000 7f85b21a4740 1 -- 172.21.15.6:0/427864438 wait complete. 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.245+0000 7f85b21a4740 1 -- start start 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85b21a4740 1 Processor -- start 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85b21a4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b23b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85b21a4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b11140 0x55c073b20fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85b21a4740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b1ba00 0x55c073b21510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85b21a4740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f85a81ae5e0 con 0x55c073b1ba00 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85b21a4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f85a81afa50 con 0x55c073b10820 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85b21a4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f85a81aecb0 con 0x55c073b11140 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85b21a4740 10 client.0 _ll_register_callbacks cb 0x7f85a0009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85a7400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b11140 0x55c073b20fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85a7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b23b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85a7400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b11140 0x55c073b20fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:53032/0 (socket says 172.21.15.6:53032) 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85a7400640 1 -- 172.21.15.6:0/2775618299 learned_addr learned my addr 172.21.15.6:0/2775618299 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.246+0000 7f85ad200640 1 --2- 172.21.15.6:0/2775618299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b1ba00 0x55c073b21510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f85a7e00640 1 -- 172.21.15.6:0/2775618299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b11140 msgr2=0x55c073b20fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f85a7e00640 1 --2- 172.21.15.6:0/2775618299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b11140 0x55c073b20fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f85a7e00640 1 -- 172.21.15.6:0/2775618299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b1ba00 msgr2=0x55c073b21510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f85a7e00640 1 --2- 172.21.15.6:0/2775618299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b1ba00 0x55c073b21510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f85a7e00640 1 -- 172.21.15.6:0/2775618299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c073b21a50 con 0x55c073b10820 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f85a7e00640 1 --2- 172.21.15.6:0/2775618299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b23b10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f85980997a0 tx=0x7f85980a1300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aaf0c5089e886db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f8596000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f8596000640 1 -- 172.21.15.6:0/2775618299 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85980a0070 con 0x55c073b10820 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f85b21a4740 1 -- 172.21.15.6:0/2775618299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f859809b0d0 con 0x55c073b10820 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f8596000640 1 -- 172.21.15.6:0/2775618299 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8600+0+0 (secure 0 0 0) 0x7f85980a55b0 con 0x55c073b10820 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f8596000640 1 -- 172.21.15.6:0/2775618299 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f85980b6330 con 0x55c073b10820 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f85b21a4740 10 client.26756 fetch_fsmap learned FSMap version 43 2024-04-26T01:58:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.247+0000 7f85b21a4740 1 -- 172.21.15.6:0/2775618299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55c073b91f00 con 0x55c073b10820 2024-04-26T01:58:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.248+0000 7f8596000640 1 -- 172.21.15.6:0/2775618299 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f859809b0d0 con 0x55c073b10820 2024-04-26T01:58:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.248+0000 7f8596000640 1 -- 172.21.15.6:0/2775618299 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 43) ==== 8338+0+0 (secure 0 0 0) 0x7f85980cfce0 con 0x55c073b10820 2024-04-26T01:58:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.248+0000 7f85b21a4740 10 client.26756 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:58:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.248+0000 7f85b21a4740 10 client.26756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:58:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.248+0000 7f85b21a4740 20 client.26756 populate_metadata read hostname 'smithi006' 2024-04-26T01:58:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.248+0000 7f85b21a4740 1 --2- 172.21.15.6:0/2775618299 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c073b925c0 0x55c073b949e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.248+0000 7f85b21a4740 4 client.26756 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:58:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.248+0000 7f85b21a4740 1 -- 172.21.15.6:0/2775618299 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c073a84320 con 0x55c073b925c0 2024-04-26T01:58:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.248+0000 7f85a7400640 1 --2- 172.21.15.6:0/2775618299 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c073b925c0 0x55c073b949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.249+0000 7f85a7400640 1 --2- 172.21.15.6:0/2775618299 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c073b925c0 0x55c073b949e0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34a0af97f04e4338 server_cookie=cf7a69c61d864a8b in_seq=0 out_seq=0 2024-04-26T01:58:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.249+0000 7f8596000640 10 client.26756 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f8596000640 1 -- 172.21.15.6:0/2775618299 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c073a84320 con 0x55c073b925c0 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f8596000640 10 client.26756 handle_command_reply: tid=0 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f85b21a4740 1 client.26756 shutdown 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f85b21a4740 1 -- 172.21.15.6:0/2775618299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 msgr2=0x55c073b23b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f85b21a4740 1 --2- 172.21.15.6:0/2775618299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b23b10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f85980997a0 tx=0x7f85980a1300 comp rx=0 tx=0).stop 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f85b21a4740 1 -- 172.21.15.6:0/2775618299 shutdown_connections 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f85b21a4740 1 --2- 172.21.15.6:0/2775618299 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c073b925c0 0x55c073b949e0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f85b21a4740 1 --2- 172.21.15.6:0/2775618299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c073b1ba00 0x55c073b21510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f85b21a4740 1 --2- 172.21.15.6:0/2775618299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c073b11140 0x55c073b20fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f8596000640 0 client.26756 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f85b21a4740 1 --2- 172.21.15.6:0/2775618299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c073b10820 0x55c073b23b10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f85b21a4740 1 -- 172.21.15.6:0/2775618299 >> 172.21.15.6:0/2775618299 conn(0x55c073b0cd30 msgr2=0x55c073b191b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f85b21a4740 1 -- 172.21.15.6:0/2775618299 shutdown_connections 2024-04-26T01:58:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.296+0000 7f85b21a4740 1 -- 172.21.15.6:0/2775618299 wait complete. 2024-04-26T01:58:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.297+0000 7f85b21a4740 20 client.26756 trim_cache size 0 max 16384 2024-04-26T01:58:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:58:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:58:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:58:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:58:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.297+0000 7f85af200640 1 -- 172.21.15.6:0/4261548902 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f858c0a3d10 msgr2=0x7f858c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.297+0000 7f85af200640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f858c0a3d10 0x7f858c0a61d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f85a0006010 tx=0x7f85a0009680 comp rx=0 tx=0).stop 2024-04-26T01:58:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.297+0000 7f85af200640 1 -- 172.21.15.6:0/4261548902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85a81010c0 msgr2=0x7f85a81a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.297+0000 7f85af200640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85a81010c0 0x7f85a81a0610 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f859800eb90 tx=0x7f859800c6a0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.297+0000 7f85af200640 1 -- 172.21.15.6:0/4261548902 shutdown_connections 2024-04-26T01:58:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.297+0000 7f85af200640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f858c0a3d10 0x7f858c0a61d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.297+0000 7f85af200640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85a819a780 0x7f85a819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.298+0000 7f85af200640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85a8106190 0x7f85a81a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.298+0000 7f85af200640 1 --2- 172.21.15.6:0/4261548902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85a81010c0 0x7f85a81a0610 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.298+0000 7f85af200640 1 -- 172.21.15.6:0/4261548902 >> 172.21.15.6:0/4261548902 conn(0x7f85a8077fd0 msgr2=0x7f85a806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.298+0000 7f85af200640 1 -- 172.21.15.6:0/4261548902 shutdown_connections 2024-04-26T01:58:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:58:59.298+0000 7f85af200640 1 -- 172.21.15.6:0/4261548902 wait complete. 2024-04-26T01:58:59.641 INFO:tasks.workunit.client.0.smithi006.stdout: 1 454563 3.46 MB/sec execute 563 sec latency 40.467 ms 2024-04-26T01:59:00.642 INFO:tasks.workunit.client.0.smithi006.stdout: 1 455070 3.46 MB/sec execute 564 sec latency 64.991 ms 2024-04-26T01:59:00.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:00 smithi029 ceph-mon[32775]: pgmap v413: 97 pgs: 97 active+clean; 144 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 894 KiB/s wr, 125 op/s 2024-04-26T01:59:00.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:00 smithi006 ceph-mon[28419]: pgmap v413: 97 pgs: 97 active+clean; 144 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 894 KiB/s wr, 125 op/s 2024-04-26T01:59:00.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:00 smithi188 ceph-mon[32751]: pgmap v413: 97 pgs: 97 active+clean; 144 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 894 KiB/s wr, 125 op/s 2024-04-26T01:59:01.642 INFO:tasks.workunit.client.0.smithi006.stdout: 1 455678 3.46 MB/sec execute 565 sec latency 29.242 ms 2024-04-26T01:59:02.642 INFO:tasks.workunit.client.0.smithi006.stdout: 1 456423 3.46 MB/sec execute 566 sec latency 66.397 ms 2024-04-26T01:59:02.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:02 smithi029 ceph-mon[32775]: pgmap v414: 97 pgs: 97 active+clean; 140 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.4 MiB/s wr, 239 op/s 2024-04-26T01:59:02.748 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:02 smithi006 ceph-mon[28419]: pgmap v414: 97 pgs: 97 active+clean; 140 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.4 MiB/s wr, 239 op/s 2024-04-26T01:59:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:02 smithi188 ceph-mon[32751]: pgmap v414: 97 pgs: 97 active+clean; 140 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.4 MiB/s wr, 239 op/s 2024-04-26T01:59:03.642 INFO:tasks.workunit.client.0.smithi006.stdout: 1 457661 3.47 MB/sec execute 567 sec latency 19.822 ms 2024-04-26T01:59:04.642 INFO:tasks.workunit.client.0.smithi006.stdout: 1 458164 3.47 MB/sec execute 568 sec latency 22.303 ms 2024-04-26T01:59:04.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:04 smithi029 ceph-mon[32775]: pgmap v415: 97 pgs: 97 active+clean; 140 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 181 op/s 2024-04-26T01:59:04.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:04 smithi006 ceph-mon[28419]: pgmap v415: 97 pgs: 97 active+clean; 140 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 181 op/s 2024-04-26T01:59:04.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:04 smithi188 ceph-mon[32751]: pgmap v415: 97 pgs: 97 active+clean; 140 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 181 op/s 2024-04-26T01:59:05.642 INFO:tasks.workunit.client.0.smithi006.stdout: 1 458641 3.48 MB/sec execute 569 sec latency 97.474 ms 2024-04-26T01:59:06.642 INFO:tasks.workunit.client.0.smithi006.stdout: 1 459520 3.50 MB/sec execute 570 sec latency 32.890 ms 2024-04-26T01:59:06.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:06 smithi029 ceph-mon[32775]: pgmap v416: 97 pgs: 97 active+clean; 140 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-04-26T01:59:06.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:06 smithi029 ceph-mon[32775]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T01:59:06.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:06 smithi006 ceph-mon[28419]: pgmap v416: 97 pgs: 97 active+clean; 140 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-04-26T01:59:06.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:06 smithi006 ceph-mon[28419]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T01:59:06.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:06 smithi188 ceph-mon[32751]: pgmap v416: 97 pgs: 97 active+clean; 140 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 182 op/s 2024-04-26T01:59:06.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:06 smithi188 ceph-mon[32751]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T01:59:07.642 INFO:tasks.workunit.client.0.smithi006.stdout: 1 460791 3.50 MB/sec execute 571 sec latency 28.642 ms 2024-04-26T01:59:08.642 INFO:tasks.workunit.client.0.smithi006.stdout: 1 462099 3.51 MB/sec execute 572 sec latency 16.636 ms 2024-04-26T01:59:08.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:08 smithi006 ceph-mon[28419]: pgmap v418: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-04-26T01:59:08.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:08 smithi188 ceph-mon[32751]: pgmap v418: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-04-26T01:59:08.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:08 smithi029 ceph-mon[32775]: pgmap v418: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-04-26T01:59:09.643 INFO:tasks.workunit.client.0.smithi006.stdout: 1 462684 3.51 MB/sec execute 573 sec latency 23.007 ms 2024-04-26T01:59:10.643 INFO:tasks.workunit.client.0.smithi006.stdout: 1 463632 3.51 MB/sec execute 574 sec latency 28.796 ms 2024-04-26T01:59:10.743 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:10 smithi029 ceph-mon[32775]: pgmap v419: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-04-26T01:59:10.743 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:10 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:59:10.743 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:10 smithi029 ceph-mon[32775]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T01:59:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:10 smithi006 ceph-mon[28419]: pgmap v419: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-04-26T01:59:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:10 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:59:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:10 smithi006 ceph-mon[28419]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T01:59:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:10 smithi188 ceph-mon[32751]: pgmap v419: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 204 op/s 2024-04-26T01:59:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:10 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:59:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:10 smithi188 ceph-mon[32751]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T01:59:11.643 INFO:tasks.workunit.client.0.smithi006.stdout: 1 465075 3.52 MB/sec execute 575 sec latency 21.212 ms 2024-04-26T01:59:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.444+0000 7ff056800640 1 -- 172.21.15.6:0/156898390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0501021b0 msgr2=0x7ff050102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.444+0000 7ff056800640 1 --2- 172.21.15.6:0/156898390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0501021b0 0x7ff050102590 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff040009900 tx=0x7ff04002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.444+0000 7ff056800640 1 -- 172.21.15.6:0/156898390 shutdown_connections 2024-04-26T01:59:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.444+0000 7ff056800640 1 --2- 172.21.15.6:0/156898390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff050102b60 0x7ff050106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.444+0000 7ff056800640 1 --2- 172.21.15.6:0/156898390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff050107280 0x7ff05010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.444+0000 7ff056800640 1 --2- 172.21.15.6:0/156898390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0501021b0 0x7ff050102590 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.444+0000 7ff056800640 1 -- 172.21.15.6:0/156898390 >> 172.21.15.6:0/156898390 conn(0x7ff0500fda10 msgr2=0x7ff0500ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.444+0000 7ff056800640 1 -- 172.21.15.6:0/156898390 shutdown_connections 2024-04-26T01:59:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.444+0000 7ff056800640 1 -- 172.21.15.6:0/156898390 wait complete. 2024-04-26T01:59:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.445+0000 7ff056800640 1 Processor -- start 2024-04-26T01:59:12.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.445+0000 7ff056800640 1 -- start start 2024-04-26T01:59:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.446+0000 7ff056800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0501021b0 0x7ff0501a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.446+0000 7ff056800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff050107280 0x7ff0501a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.446+0000 7ff056800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff05019a770 0x7ff05019abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.446+0000 7ff056800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff050112210 con 0x7ff0501021b0 2024-04-26T01:59:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.446+0000 7ff056800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff050112090 con 0x7ff050107280 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.446+0000 7ff056800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff050112390 con 0x7ff05019a770 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.446+0000 7ff04f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0501021b0 0x7ff0501a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.446+0000 7ff04ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff050107280 0x7ff0501a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.446+0000 7ff04f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0501021b0 0x7ff0501a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56054/0 (socket says 172.21.15.6:56054) 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.446+0000 7ff04f400640 1 -- 172.21.15.6:0/2428548315 learned_addr learned my addr 172.21.15.6:0/2428548315 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.446+0000 7ff04fe00640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff05019a770 0x7ff05019abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.447+0000 7ff04ea00640 1 -- 172.21.15.6:0/2428548315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff05019a770 msgr2=0x7ff05019abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.447+0000 7ff04ea00640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff05019a770 0x7ff05019abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.447+0000 7ff04ea00640 1 -- 172.21.15.6:0/2428548315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0501021b0 msgr2=0x7ff0501a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.447+0000 7ff04ea00640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0501021b0 0x7ff0501a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.447+0000 7ff04ea00640 1 -- 172.21.15.6:0/2428548315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05019b370 con 0x7ff050107280 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.447+0000 7ff04ea00640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff050107280 0x7ff0501a0bf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff038005fd0 tx=0x7ff03800df40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.447+0000 7ff047e00640 1 -- 172.21.15.6:0/2428548315 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff038017030 con 0x7ff050107280 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.447+0000 7ff047e00640 1 -- 172.21.15.6:0/2428548315 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff038003e20 con 0x7ff050107280 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.447+0000 7ff047e00640 1 -- 172.21.15.6:0/2428548315 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff038010460 con 0x7ff050107280 2024-04-26T01:59:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.447+0000 7ff056800640 1 -- 172.21.15.6:0/2428548315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0501a7450 con 0x7ff050107280 2024-04-26T01:59:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.448+0000 7ff056800640 1 -- 172.21.15.6:0/2428548315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0501a78b0 con 0x7ff050107280 2024-04-26T01:59:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.451+0000 7ff047e00640 1 -- 172.21.15.6:0/2428548315 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7ff03801a070 con 0x7ff050107280 2024-04-26T01:59:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.451+0000 7ff05978f740 1 Processor -- start 2024-04-26T01:59:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.451+0000 7ff05978f740 1 -- start start 2024-04-26T01:59:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.452+0000 7ff05978f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c798820 0x562b5c798c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.452+0000 7ff05978f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c799140 0x562b5c7a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.452+0000 7ff05978f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c7a3a00 0x562b5c7a5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.452+0000 7ff05978f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x562b5c6fe900 con 0x562b5c798820 2024-04-26T01:59:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.452+0000 7ff05978f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562b5c6a4ef0 con 0x562b5c7a3a00 2024-04-26T01:59:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.452+0000 7ff05978f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562b5c6fa8c0 con 0x562b5c799140 2024-04-26T01:59:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.452+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c798820 0x562b5c798c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.452+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c798820 0x562b5c798c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56068/0 (socket says 172.21.15.6:56068) 2024-04-26T01:59:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.452+0000 7ff04fe00640 1 -- 172.21.15.6:0/2241012968 learned_addr learned my addr 172.21.15.6:0/2241012968 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.452+0000 7ff04f400640 1 --2- 172.21.15.6:0/2241012968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c799140 0x562b5c7a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.452+0000 7ff04ea00640 1 --2- 172.21.15.6:0/2241012968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c7a3a00 0x562b5c7a5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.453+0000 7ff04fe00640 1 -- 172.21.15.6:0/2241012968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c799140 msgr2=0x562b5c7a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.453+0000 7ff04fe00640 1 --2- 172.21.15.6:0/2241012968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c799140 0x562b5c7a34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.453+0000 7ff04fe00640 1 -- 172.21.15.6:0/2241012968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c7a3a00 msgr2=0x562b5c7a5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.453+0000 7ff04fe00640 1 --2- 172.21.15.6:0/2241012968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c7a3a00 0x562b5c7a5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.453+0000 7ff04fe00640 1 -- 172.21.15.6:0/2241012968 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b5c411210 con 0x562b5c798820 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.453+0000 7ff047e00640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff0340a3d90 0x7ff0340a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.453+0000 7ff047e00640 1 -- 172.21.15.6:0/2428548315 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7ff0380d0570 con 0x7ff050107280 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.453+0000 7ff04fe00640 1 --2- 172.21.15.6:0/2241012968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c798820 0x562b5c798c00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff04800b6b0 tx=0x7ff048004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25159754c4e12a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.453+0000 7ff04f400640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff0340a3d90 0x7ff0340a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.453+0000 7ff044c00640 1 -- 172.21.15.6:0/2241012968 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048003b20 con 0x562b5c798820 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.453+0000 7ff044c00640 1 -- 172.21.15.6:0/2241012968 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff048015070 con 0x562b5c798820 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.454+0000 7ff044c00640 1 -- 172.21.15.6:0/2241012968 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0480116a0 con 0x562b5c798820 2024-04-26T01:59:12.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.454+0000 7ff05978f740 1 -- 172.21.15.6:0/2241012968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c798820 msgr2=0x562b5c798c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:12.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.454+0000 7ff04f400640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff0340a3d90 0x7ff0340a6250 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x562b5c799dc0 tx=0x7ff040005a20 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:12.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.454+0000 7ff05978f740 1 --2- 172.21.15.6:0/2241012968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c798820 0x562b5c798c00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff04800b6b0 tx=0x7ff048004610 comp rx=0 tx=0).stop 2024-04-26T01:59:12.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.454+0000 7ff05978f740 1 -- 172.21.15.6:0/2241012968 shutdown_connections 2024-04-26T01:59:12.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.454+0000 7ff05978f740 1 --2- 172.21.15.6:0/2241012968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c7a3a00 0x562b5c7a5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.454+0000 7ff05978f740 1 --2- 172.21.15.6:0/2241012968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c799140 0x562b5c7a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.454+0000 7ff05978f740 1 --2- 172.21.15.6:0/2241012968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c798820 0x562b5c798c00 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.454+0000 7ff05978f740 1 -- 172.21.15.6:0/2241012968 >> 172.21.15.6:0/2241012968 conn(0x562b5c794d30 msgr2=0x562b5c795110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:12.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.454+0000 7ff05978f740 1 -- 172.21.15.6:0/2241012968 shutdown_connections 2024-04-26T01:59:12.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.455+0000 7ff05978f740 1 -- 172.21.15.6:0/2241012968 wait complete. 2024-04-26T01:59:12.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.455+0000 7ff05978f740 1 -- start start 2024-04-26T01:59:12.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.455+0000 7ff05978f740 1 Processor -- start 2024-04-26T01:59:12.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff05978f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c798820 0x562b5c7a0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff05978f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 0x562b5c7a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff05978f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c7a3a00 0x562b5c7a1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff05978f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x562b5c7a6630 con 0x562b5c798820 2024-04-26T01:59:12.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff05978f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562b5c6b8310 con 0x562b5c799140 2024-04-26T01:59:12.459 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff05978f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562b5c7a67b0 con 0x562b5c7a3a00 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff05978f740 10 client.0 _ll_register_callbacks cb 0x562b5c686cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff04ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 0x562b5c7a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff04ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 0x562b5c7a0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55244/0 (socket says 172.21.15.6:55244) 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff04ea00640 1 -- 172.21.15.6:0/2518642841 learned_addr learned my addr 172.21.15.6:0/2518642841 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c7a3a00 0x562b5c7a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff04f400640 1 --2- 172.21.15.6:0/2518642841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c798820 0x562b5c7a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff04ea00640 1 -- 172.21.15.6:0/2518642841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c7a3a00 msgr2=0x562b5c7a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff04ea00640 1 --2- 172.21.15.6:0/2518642841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c7a3a00 0x562b5c7a1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff04ea00640 1 -- 172.21.15.6:0/2518642841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c798820 msgr2=0x562b5c7a0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff04ea00640 1 --2- 172.21.15.6:0/2518642841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c798820 0x562b5c7a0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.456+0000 7ff04ea00640 1 -- 172.21.15.6:0/2518642841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b5c4dd620 con 0x562b5c799140 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff04ea00640 1 --2- 172.21.15.6:0/2518642841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 0x562b5c7a0ed0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff03809c160 tx=0x7ff038098700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8bd9d202e3a7bb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:12.460 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff03d600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff03d600640 1 -- 172.21.15.6:0/2518642841 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03809bdc0 con 0x562b5c799140 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff03d600640 1 -- 172.21.15.6:0/2518642841 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7ff038098ad0 con 0x562b5c799140 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff05978f740 1 -- 172.21.15.6:0/2518642841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff048015070 con 0x562b5c799140 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff03d600640 1 -- 172.21.15.6:0/2518642841 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7ff03809cb60 con 0x562b5c799140 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff05978f740 10 client.26768 fetch_fsmap learned FSMap version 44 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff05978f740 1 -- 172.21.15.6:0/2518642841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x562b5c7b2890 con 0x562b5c799140 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff03d600640 1 -- 172.21.15.6:0/2518642841 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff048015070 con 0x562b5c799140 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff03d600640 1 -- 172.21.15.6:0/2518642841 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7ff038024060 con 0x562b5c799140 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff05978f740 10 client.26768 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff05978f740 10 client.26768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.457+0000 7ff05978f740 20 client.26768 populate_metadata read hostname 'smithi006' 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.458+0000 7ff05978f740 1 --2- 172.21.15.6:0/2518642841 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562b5c7b3d00 0x562b5c7b60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.458+0000 7ff05978f740 4 client.26768 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.458+0000 7ff05978f740 1 -- 172.21.15.6:0/2518642841 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b5c7128b0 con 0x562b5c7b3d00 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.458+0000 7ff04f400640 1 --2- 172.21.15.6:0/2518642841 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562b5c7b3d00 0x562b5c7b60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.461 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.459+0000 7ff04f400640 1 --2- 172.21.15.6:0/2518642841 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562b5c7b3d00 0x562b5c7b60f0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aed4df3d77c7c0cd server_cookie=3aba999a768b35b3 in_seq=0 out_seq=0 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.459+0000 7ff03d600640 10 client.26768 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff03d600640 1 -- 172.21.15.6:0/2518642841 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562b5c7128b0 con 0x562b5c7b3d00 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff03d600640 10 client.26768 handle_command_reply: tid=0 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff05978f740 1 client.26768 shutdown 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff05978f740 1 -- 172.21.15.6:0/2518642841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 msgr2=0x562b5c7a0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff05978f740 1 --2- 172.21.15.6:0/2518642841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 0x562b5c7a0ed0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff03809c160 tx=0x7ff038098700 comp rx=0 tx=0).stop 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff05978f740 1 -- 172.21.15.6:0/2518642841 shutdown_connections 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff05978f740 1 --2- 172.21.15.6:0/2518642841 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562b5c7b3d00 0x562b5c7b60f0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff05978f740 1 --2- 172.21.15.6:0/2518642841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c7a3a00 0x562b5c7a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff03d600640 0 client.26768 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff05978f740 1 --2- 172.21.15.6:0/2518642841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 0x562b5c7a0ed0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff05978f740 1 --2- 172.21.15.6:0/2518642841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c798820 0x562b5c7a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff05978f740 1 -- 172.21.15.6:0/2518642841 >> 172.21.15.6:0/2518642841 conn(0x562b5c794d30 msgr2=0x562b5c79aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:12.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff05978f740 1 -- 172.21.15.6:0/2518642841 shutdown_connections 2024-04-26T01:59:12.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff05978f740 1 -- 172.21.15.6:0/2518642841 wait complete. 2024-04-26T01:59:12.463 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.460+0000 7ff05978f740 20 client.26768 trim_cache size 0 max 16384 2024-04-26T01:59:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.478+0000 7ff05978f740 1 Processor -- start 2024-04-26T01:59:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.478+0000 7ff05978f740 1 -- start start 2024-04-26T01:59:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.479+0000 7ff05978f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c798820 0x562b5c794610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.479+0000 7ff05978f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 0x562b5c7af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.479+0000 7ff05978f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c7a3a00 0x562b5c7bb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.479+0000 7ff05978f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501b28c0 con 0x562b5c7a3a00 2024-04-26T01:59:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.479+0000 7ff05978f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501b0190 con 0x562b5c799140 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.479+0000 7ff05978f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501ae000 con 0x562b5c798820 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.479+0000 7ff04f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c798820 0x562b5c794610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.479+0000 7ff04ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 0x562b5c7af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.479+0000 7ff04f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c798820 0x562b5c794610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:37078/0 (socket says 172.21.15.6:37078) 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.479+0000 7ff04f400640 1 -- 172.21.15.6:0/3939315899 learned_addr learned my addr 172.21.15.6:0/3939315899 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.479+0000 7ff04ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 0x562b5c7af4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55256/0 (socket says 172.21.15.6:55256) 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.479+0000 7ff04fe00640 1 --2- 172.21.15.6:0/3939315899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c7a3a00 0x562b5c7bb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.480+0000 7ff04ea00640 1 -- 172.21.15.6:0/3939315899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c798820 msgr2=0x562b5c794610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.480+0000 7ff04ea00640 1 --2- 172.21.15.6:0/3939315899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c798820 0x562b5c794610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.480+0000 7ff04ea00640 1 -- 172.21.15.6:0/3939315899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c7a3a00 msgr2=0x562b5c7bb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.480+0000 7ff04ea00640 1 --2- 172.21.15.6:0/3939315899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c7a3a00 0x562b5c7bb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.480+0000 7ff04ea00640 1 -- 172.21.15.6:0/3939315899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b5c799880 con 0x562b5c799140 2024-04-26T01:59:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.480+0000 7ff04f400640 1 --2- 172.21.15.6:0/3939315899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c798820 0x562b5c794610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.480+0000 7ff04ea00640 1 --2- 172.21.15.6:0/3939315899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 0x562b5c7af4f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff03800b6f0 tx=0x7ff0380092a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=444aca0302ff3a84 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.480+0000 7ff03fe00640 1 -- 172.21.15.6:0/3939315899 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03809c460 con 0x562b5c799140 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.480+0000 7ff03fe00640 1 -- 172.21.15.6:0/3939315899 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0380a50d0 con 0x562b5c799140 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.480+0000 7ff03fe00640 1 -- 172.21.15.6:0/3939315899 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0380d0040 con 0x562b5c799140 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.481+0000 7ff05978f740 1 -- 172.21.15.6:0/3939315899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 msgr2=0x562b5c7af4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.481+0000 7ff05978f740 1 --2- 172.21.15.6:0/3939315899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 0x562b5c7af4f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff03800b6f0 tx=0x7ff0380092a0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.481+0000 7ff05978f740 1 -- 172.21.15.6:0/3939315899 shutdown_connections 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.481+0000 7ff05978f740 1 --2- 172.21.15.6:0/3939315899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c7a3a00 0x562b5c7bb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.481+0000 7ff05978f740 1 --2- 172.21.15.6:0/3939315899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c799140 0x562b5c7af4f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.481+0000 7ff05978f740 1 --2- 172.21.15.6:0/3939315899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c798820 0x562b5c794610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.481+0000 7ff05978f740 1 -- 172.21.15.6:0/3939315899 >> 172.21.15.6:0/3939315899 conn(0x562b5c794d30 msgr2=0x562b5c7a11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.481+0000 7ff05978f740 1 -- 172.21.15.6:0/3939315899 shutdown_connections 2024-04-26T01:59:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.481+0000 7ff05978f740 1 -- 172.21.15.6:0/3939315899 wait complete. 2024-04-26T01:59:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.482+0000 7ff05978f740 1 -- start start 2024-04-26T01:59:12.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.482+0000 7ff05978f740 1 Processor -- start 2024-04-26T01:59:12.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.482+0000 7ff05978f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c798820 0x562b5c7abf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.482+0000 7ff05978f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c799140 0x562b5c7a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.482+0000 7ff05978f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c7a3a00 0x562b5c7a9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.482+0000 7ff05978f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501ae650 con 0x562b5c7a3a00 2024-04-26T01:59:12.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.482+0000 7ff05978f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501afac0 con 0x562b5c798820 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.482+0000 7ff05978f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501aed20 con 0x562b5c799140 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.482+0000 7ff05978f740 10 client.0 _ll_register_callbacks cb 0x7ff038004930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c798820 0x562b5c7abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c798820 0x562b5c7abf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55272/0 (socket says 172.21.15.6:55272) 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04f400640 1 -- 172.21.15.6:0/1723730839 learned_addr learned my addr 172.21.15.6:0/1723730839 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04ea00640 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c799140 0x562b5c7a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04fe00640 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c7a3a00 0x562b5c7a9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04f400640 1 -- 172.21.15.6:0/1723730839 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c799140 msgr2=0x562b5c7a92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04f400640 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c799140 0x562b5c7a92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04f400640 1 -- 172.21.15.6:0/1723730839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c7a3a00 msgr2=0x562b5c7a9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04f400640 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c7a3a00 0x562b5c7a9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04f400640 1 -- 172.21.15.6:0/1723730839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b5c819230 con 0x562b5c798820 2024-04-26T01:59:12.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04fe00640 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c7a3a00 0x562b5c7a9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04ea00640 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c799140 0x562b5c7a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff04f400640 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c798820 0x562b5c7abf40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff040006010 tx=0x7ff040009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9a3234d243ba883 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff03d600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.483+0000 7ff03d600640 1 -- 172.21.15.6:0/1723730839 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff040035040 con 0x562b5c798820 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff05978f740 1 -- 172.21.15.6:0/1723730839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0380a50d0 con 0x562b5c798820 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff03d600640 1 -- 172.21.15.6:0/1723730839 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7ff040054890 con 0x562b5c798820 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff03d600640 1 -- 172.21.15.6:0/1723730839 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7ff040064050 con 0x562b5c798820 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff05978f740 10 client.26774 fetch_fsmap learned FSMap version 44 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff05978f740 1 -- 172.21.15.6:0/1723730839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x562b5c81a440 con 0x562b5c798820 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff03d600640 1 -- 172.21.15.6:0/1723730839 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0380a50d0 con 0x562b5c798820 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff03d600640 1 -- 172.21.15.6:0/1723730839 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7ff040044070 con 0x562b5c798820 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff05978f740 10 client.26774 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff05978f740 10 client.26774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff05978f740 20 client.26774 populate_metadata read hostname 'smithi006' 2024-04-26T01:59:12.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff05978f740 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562b5c81ab40 0x562b5c81cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:12.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff05978f740 4 client.26774 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:59:12.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.484+0000 7ff05978f740 1 -- 172.21.15.6:0/1723730839 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562b5c70c320 con 0x562b5c81ab40 2024-04-26T01:59:12.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.485+0000 7ff04ea00640 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562b5c81ab40 0x562b5c81cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:12.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.485+0000 7ff04ea00640 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562b5c81ab40 0x562b5c81cf60 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bb6b6624228711 server_cookie=4a9d9ab0b7b85ba7 in_seq=0 out_seq=0 2024-04-26T01:59:12.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.485+0000 7ff03d600640 10 client.26774 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.541+0000 7ff03d600640 1 -- 172.21.15.6:0/1723730839 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562b5c70c320 con 0x562b5c81ab40 2024-04-26T01:59:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.541+0000 7ff03d600640 10 client.26774 handle_command_reply: tid=0 2024-04-26T01:59:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.541+0000 7ff05978f740 1 client.26774 shutdown 2024-04-26T01:59:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.542+0000 7ff05978f740 1 -- 172.21.15.6:0/1723730839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c798820 msgr2=0x562b5c7abf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.542+0000 7ff05978f740 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c798820 0x562b5c7abf40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff040006010 tx=0x7ff040009260 comp rx=0 tx=0).stop 2024-04-26T01:59:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.542+0000 7ff05978f740 1 -- 172.21.15.6:0/1723730839 shutdown_connections 2024-04-26T01:59:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.542+0000 7ff05978f740 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x562b5c81ab40 0x562b5c81cf60 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.542+0000 7ff05978f740 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x562b5c7a3a00 0x562b5c7a9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.542+0000 7ff05978f740 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562b5c799140 0x562b5c7a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.542+0000 7ff03d600640 0 client.26774 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.542+0000 7ff05978f740 1 --2- 172.21.15.6:0/1723730839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5c798820 0x562b5c7abf40 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.542+0000 7ff05978f740 1 -- 172.21.15.6:0/1723730839 >> 172.21.15.6:0/1723730839 conn(0x562b5c794d30 msgr2=0x562b5c79a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.542+0000 7ff05978f740 1 -- 172.21.15.6:0/1723730839 shutdown_connections 2024-04-26T01:59:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.542+0000 7ff05978f740 1 -- 172.21.15.6:0/1723730839 wait complete. 2024-04-26T01:59:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.542+0000 7ff05978f740 20 client.26774 trim_cache size 0 max 16384 2024-04-26T01:59:12.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:59:12.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:59:12.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:59:12.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:59:12.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.543+0000 7ff056800640 1 -- 172.21.15.6:0/2428548315 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff0340a3d90 msgr2=0x7ff0340a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.543+0000 7ff056800640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff0340a3d90 0x7ff0340a6250 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x562b5c799dc0 tx=0x7ff040005a20 comp rx=0 tx=0).stop 2024-04-26T01:59:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.543+0000 7ff056800640 1 -- 172.21.15.6:0/2428548315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff050107280 msgr2=0x7ff0501a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.543+0000 7ff056800640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff050107280 0x7ff0501a0bf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff038005fd0 tx=0x7ff03800df40 comp rx=0 tx=0).stop 2024-04-26T01:59:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.543+0000 7ff056800640 1 -- 172.21.15.6:0/2428548315 shutdown_connections 2024-04-26T01:59:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.543+0000 7ff056800640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff0340a3d90 0x7ff0340a6250 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.543+0000 7ff056800640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff05019a770 0x7ff05019abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.543+0000 7ff056800640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff050107280 0x7ff0501a0bf0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.543+0000 7ff056800640 1 --2- 172.21.15.6:0/2428548315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0501021b0 0x7ff0501a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.543+0000 7ff056800640 1 -- 172.21.15.6:0/2428548315 >> 172.21.15.6:0/2428548315 conn(0x7ff0500fda10 msgr2=0x7ff0500ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.544+0000 7ff056800640 1 -- 172.21.15.6:0/2428548315 shutdown_connections 2024-04-26T01:59:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:12.544+0000 7ff056800640 1 -- 172.21.15.6:0/2428548315 wait complete. 2024-04-26T01:59:12.643 INFO:tasks.workunit.client.0.smithi006.stdout: 1 465390 3.52 MB/sec execute 576 sec latency 804.559 ms 2024-04-26T01:59:12.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:12 smithi006 ceph-mon[28419]: pgmap v421: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-04-26T01:59:12.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:12 smithi188 ceph-mon[32751]: pgmap v421: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-04-26T01:59:12.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:12 smithi029 ceph-mon[32775]: pgmap v421: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-04-26T01:59:13.643 INFO:tasks.workunit.client.0.smithi006.stdout: 1 465390 3.51 MB/sec execute 577 sec latency 1804.707 ms 2024-04-26T01:59:14.643 INFO:tasks.workunit.client.0.smithi006.stdout: 1 465390 3.51 MB/sec execute 578 sec latency 2804.808 ms 2024-04-26T01:59:14.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:14 smithi006 ceph-mon[28419]: pgmap v422: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-04-26T01:59:14.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:14 smithi188 ceph-mon[32751]: pgmap v422: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-04-26T01:59:14.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:14 smithi029 ceph-mon[32775]: pgmap v422: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-04-26T01:59:15.643 INFO:tasks.workunit.client.0.smithi006.stdout: 1 465390 3.50 MB/sec execute 579 sec latency 3804.968 ms 2024-04-26T01:59:16.644 INFO:tasks.workunit.client.0.smithi006.stdout: 1 465390 3.49 MB/sec execute 580 sec latency 4805.142 ms 2024-04-26T01:59:16.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:16 smithi006 ceph-mon[28419]: pgmap v423: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-04-26T01:59:16.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:16 smithi188 ceph-mon[32751]: pgmap v423: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-04-26T01:59:16.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:16 smithi029 ceph-mon[32775]: pgmap v423: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-04-26T01:59:17.644 INFO:tasks.workunit.client.0.smithi006.stdout: 1 465390 3.49 MB/sec execute 581 sec latency 5805.282 ms 2024-04-26T01:59:18.644 INFO:tasks.workunit.client.0.smithi006.stdout: 1 465390 3.48 MB/sec execute 582 sec latency 6805.430 ms 2024-04-26T01:59:18.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:18 smithi006 ceph-mon[28419]: pgmap v424: 97 pgs: 97 active+clean; 142 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 805 KiB/s wr, 114 op/s 2024-04-26T01:59:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:18 smithi188 ceph-mon[32751]: pgmap v424: 97 pgs: 97 active+clean; 142 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 805 KiB/s wr, 114 op/s 2024-04-26T01:59:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:18 smithi029 ceph-mon[32775]: pgmap v424: 97 pgs: 97 active+clean; 142 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 805 KiB/s wr, 114 op/s 2024-04-26T01:59:19.644 INFO:tasks.workunit.client.0.smithi006.stdout: 1 465756 3.48 MB/sec execute 583 sec latency 7172.637 ms 2024-04-26T01:59:20.644 INFO:tasks.workunit.client.0.smithi006.stdout: 1 466514 3.48 MB/sec execute 584 sec latency 20.183 ms 2024-04-26T01:59:20.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:20 smithi006 ceph-mon[28419]: pgmap v425: 97 pgs: 97 active+clean; 142 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 805 KiB/s wr, 114 op/s 2024-04-26T01:59:20.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:20 smithi188 ceph-mon[32751]: pgmap v425: 97 pgs: 97 active+clean; 142 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 805 KiB/s wr, 114 op/s 2024-04-26T01:59:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:20 smithi029 ceph-mon[32775]: pgmap v425: 97 pgs: 97 active+clean; 142 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 805 KiB/s wr, 114 op/s 2024-04-26T01:59:21.644 INFO:tasks.workunit.client.0.smithi006.stdout: 1 467793 3.48 MB/sec execute 585 sec latency 27.978 ms 2024-04-26T01:59:22.644 INFO:tasks.workunit.client.0.smithi006.stdout: 1 468933 3.49 MB/sec execute 586 sec latency 37.286 ms 2024-04-26T01:59:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:22 smithi006 ceph-mon[28419]: pgmap v426: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-04-26T01:59:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:22 smithi188 ceph-mon[32751]: pgmap v426: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-04-26T01:59:22.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:22 smithi029 ceph-mon[32775]: pgmap v426: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-04-26T01:59:23.645 INFO:tasks.workunit.client.0.smithi006.stdout: 1 469433 3.49 MB/sec execute 587 sec latency 40.935 ms 2024-04-26T01:59:24.645 INFO:tasks.workunit.client.0.smithi006.stdout: 1 470512 3.49 MB/sec execute 588 sec latency 27.846 ms 2024-04-26T01:59:24.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:24 smithi006 ceph-mon[28419]: pgmap v427: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 715 KiB/s wr, 80 op/s 2024-04-26T01:59:24.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:24 smithi188 ceph-mon[32751]: pgmap v427: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 715 KiB/s wr, 80 op/s 2024-04-26T01:59:24.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:24 smithi029 ceph-mon[32775]: pgmap v427: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 715 KiB/s wr, 80 op/s 2024-04-26T01:59:25.645 INFO:tasks.workunit.client.0.smithi006.stdout: 1 471984 3.50 MB/sec execute 589 sec latency 20.550 ms 2024-04-26T01:59:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.700+0000 7f42a4400640 1 -- 172.21.15.6:0/3103049895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f429c1043c0 msgr2=0x7f429c1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.700+0000 7f42a4400640 1 --2- 172.21.15.6:0/3103049895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f429c1043c0 0x7f429c1047a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4290009900 tx=0x7f429002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.700+0000 7f42a4400640 1 -- 172.21.15.6:0/3103049895 shutdown_connections 2024-04-26T01:59:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.700+0000 7f42a4400640 1 --2- 172.21.15.6:0/3103049895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f429c109490 0x7f429c111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.700+0000 7f42a4400640 1 --2- 172.21.15.6:0/3103049895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f429c104d70 0x7f429c108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.700+0000 7f42a4400640 1 --2- 172.21.15.6:0/3103049895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f429c1043c0 0x7f429c1047a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.700+0000 7f42a4400640 1 -- 172.21.15.6:0/3103049895 >> 172.21.15.6:0/3103049895 conn(0x7f429c0ffc20 msgr2=0x7f429c102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.701+0000 7f42a4400640 1 -- 172.21.15.6:0/3103049895 shutdown_connections 2024-04-26T01:59:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.701+0000 7f42a4400640 1 -- 172.21.15.6:0/3103049895 wait complete. 2024-04-26T01:59:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.702+0000 7f42a4400640 1 Processor -- start 2024-04-26T01:59:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.702+0000 7f42a4400640 1 -- start start 2024-04-26T01:59:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.702+0000 7f42a4400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f429c1043c0 0x7f429c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.702+0000 7f42a4400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f429c109490 0x7f429c1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.702+0000 7f42a4400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f429c19a7c0 0x7f429c19ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.702+0000 7f42a4400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f429c114420 con 0x7f429c19a7c0 2024-04-26T01:59:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.702+0000 7f42a4400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f429c1142a0 con 0x7f429c109490 2024-04-26T01:59:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.702+0000 7f42a4400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f429c1145a0 con 0x7f429c1043c0 2024-04-26T01:59:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.703+0000 7f42a1a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f429c1043c0 0x7f429c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.703+0000 7f42a1000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f429c109490 0x7f429c1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.703+0000 7f42a1a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f429c1043c0 0x7f429c1a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58486/0 (socket says 172.21.15.6:58486) 2024-04-26T01:59:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.703+0000 7f42a1a00640 1 -- 172.21.15.6:0/354564790 learned_addr learned my addr 172.21.15.6:0/354564790 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.703+0000 7f42a2400640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f429c19a7c0 0x7f429c19ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.703+0000 7f42a1a00640 1 -- 172.21.15.6:0/354564790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f429c109490 msgr2=0x7f429c1a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.703+0000 7f42a1a00640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f429c109490 0x7f429c1a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.703+0000 7f42a1a00640 1 -- 172.21.15.6:0/354564790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f429c19a7c0 msgr2=0x7f429c19ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.703+0000 7f42a1a00640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f429c19a7c0 0x7f429c19ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.703+0000 7f42a1a00640 1 -- 172.21.15.6:0/354564790 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429c19b350 con 0x7f429c1043c0 2024-04-26T01:59:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.703+0000 7f42a1000640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f429c109490 0x7f429c1a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.703+0000 7f42a1a00640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f429c1043c0 0x7f429c1a0650 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4290005ab0 tx=0x7f4290003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.704+0000 7f429a000640 1 -- 172.21.15.6:0/354564790 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429003f070 con 0x7f429c1043c0 2024-04-26T01:59:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.704+0000 7f429a000640 1 -- 172.21.15.6:0/354564790 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4290031b60 con 0x7f429c1043c0 2024-04-26T01:59:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.704+0000 7f429a000640 1 -- 172.21.15.6:0/354564790 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4290035030 con 0x7f429c1043c0 2024-04-26T01:59:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.704+0000 7f42a4400640 1 -- 172.21.15.6:0/354564790 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f429c19b5e0 con 0x7f429c1043c0 2024-04-26T01:59:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.704+0000 7f42a4400640 1 -- 172.21.15.6:0/354564790 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f429c1a7740 con 0x7f429c1043c0 2024-04-26T01:59:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.708+0000 7f429a000640 1 -- 172.21.15.6:0/354564790 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f4290042070 con 0x7f429c1043c0 2024-04-26T01:59:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.708+0000 7f42a6f22740 1 Processor -- start 2024-04-26T01:59:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.708+0000 7f42a6f22740 1 -- start start 2024-04-26T01:59:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.708+0000 7f42a6f22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c4820 0x56361c6c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.708+0000 7f42a6f22740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c5140 0x56361c6cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.708+0000 7f42a6f22740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6cfa00 0x56361c6d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.708+0000 7f42a6f22740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56361c62a900 con 0x56361c6c5140 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.708+0000 7f42a6f22740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56361c5d0ef0 con 0x56361c6cfa00 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.708+0000 7f42a6f22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56361c6268c0 con 0x56361c6c4820 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.708+0000 7f42a2400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c4820 0x56361c6c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.708+0000 7f42a2400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c4820 0x56361c6c4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58492/0 (socket says 172.21.15.6:58492) 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.708+0000 7f42a2400640 1 -- 172.21.15.6:0/4228160447 learned_addr learned my addr 172.21.15.6:0/4228160447 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.709+0000 7f42a1000640 1 --2- 172.21.15.6:0/4228160447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6cfa00 0x56361c6d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.709+0000 7f42a1a00640 1 --2- 172.21.15.6:0/4228160447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c5140 0x56361c6cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.709+0000 7f42a2400640 1 -- 172.21.15.6:0/4228160447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6cfa00 msgr2=0x56361c6d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.709+0000 7f42a2400640 1 --2- 172.21.15.6:0/4228160447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6cfa00 0x56361c6d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.709+0000 7f42a2400640 1 -- 172.21.15.6:0/4228160447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c5140 msgr2=0x56361c6cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.709+0000 7f42a2400640 1 --2- 172.21.15.6:0/4228160447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c5140 0x56361c6cf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.709+0000 7f42a2400640 1 -- 172.21.15.6:0/4228160447 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56361c33d210 con 0x56361c6c4820 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.709+0000 7f42a1000640 1 --2- 172.21.15.6:0/4228160447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6cfa00 0x56361c6d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.709+0000 7f42a2400640 1 --2- 172.21.15.6:0/4228160447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c4820 0x56361c6c4c00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4294009f60 tx=0x7f4294009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c50f3ab8c176bd74 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.709+0000 7f429a000640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f42840a3d90 0x7f42840a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.710+0000 7f428ea00640 1 -- 172.21.15.6:0/4228160447 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429401e070 con 0x56361c6c4820 2024-04-26T01:59:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.710+0000 7f428ea00640 1 -- 172.21.15.6:0/4228160447 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4294003c30 con 0x56361c6c4820 2024-04-26T01:59:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.710+0000 7f428ea00640 1 -- 172.21.15.6:0/4228160447 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429400f8a0 con 0x56361c6c4820 2024-04-26T01:59:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.710+0000 7f429a000640 1 -- 172.21.15.6:0/354564790 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f42900f0ec0 con 0x7f429c1043c0 2024-04-26T01:59:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.710+0000 7f42a1a00640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f42840a3d90 0x7f42840a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.710+0000 7f42a6f22740 1 -- 172.21.15.6:0/4228160447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c4820 msgr2=0x56361c6c4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.710+0000 7f42a6f22740 1 --2- 172.21.15.6:0/4228160447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c4820 0x56361c6c4c00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4294009f60 tx=0x7f4294009f90 comp rx=0 tx=0).stop 2024-04-26T01:59:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.710+0000 7f42a6f22740 1 -- 172.21.15.6:0/4228160447 shutdown_connections 2024-04-26T01:59:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.710+0000 7f42a6f22740 1 --2- 172.21.15.6:0/4228160447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c5140 0x56361c6cf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.711+0000 7f42a6f22740 1 --2- 172.21.15.6:0/4228160447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6cfa00 0x56361c6d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.711+0000 7f42a6f22740 1 --2- 172.21.15.6:0/4228160447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c4820 0x56361c6c4c00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.711+0000 7f42a6f22740 1 -- 172.21.15.6:0/4228160447 >> 172.21.15.6:0/4228160447 conn(0x56361c6c0d30 msgr2=0x56361c6c1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.711+0000 7f42a1a00640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f42840a3d90 0x7f42840a6250 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x56361c6c5dc0 tx=0x7f4290009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.711+0000 7f42a6f22740 1 -- 172.21.15.6:0/4228160447 shutdown_connections 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.711+0000 7f42a6f22740 1 -- 172.21.15.6:0/4228160447 wait complete. 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.711+0000 7f42a6f22740 1 -- start start 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.711+0000 7f42a6f22740 1 Processor -- start 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a6f22740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6c4820 0x56361c6cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a6f22740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c5140 0x56361c6ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a6f22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6cfa00 0x56361c6cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a6f22740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56361c6d2630 con 0x56361c6c5140 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a6f22740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56361c5e4310 con 0x56361c6c4820 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a6f22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56361c6d27b0 con 0x56361c6cfa00 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a2400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6c4820 0x56361c6cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a2400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6c4820 0x56361c6cca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:38042/0 (socket says 172.21.15.6:38042) 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a2400640 1 -- 172.21.15.6:0/3775374084 learned_addr learned my addr 172.21.15.6:0/3775374084 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a1000640 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c5140 0x56361c6ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a6f22740 10 client.0 _ll_register_callbacks cb 0x56361c5b2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a2400640 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6cfa00 0x56361c6cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a2400640 1 -- 172.21.15.6:0/3775374084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6cfa00 msgr2=0x56361c6cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a2400640 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6cfa00 0x56361c6cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a2400640 1 -- 172.21.15.6:0/3775374084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c5140 msgr2=0x56361c6ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a2400640 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c5140 0x56361c6ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.712+0000 7f42a2400640 1 -- 172.21.15.6:0/3775374084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56361c409620 con 0x56361c6c4820 2024-04-26T01:59:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.713+0000 7f42a2400640 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6c4820 0x56361c6cca30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4294009aa0 tx=0x7f429400f520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f82493ca723728c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.713+0000 7f4283400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T01:59:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.713+0000 7f4283400640 1 -- 172.21.15.6:0/3775374084 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429401e030 con 0x56361c6c4820 2024-04-26T01:59:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.713+0000 7f4283400640 1 -- 172.21.15.6:0/3775374084 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f4294004490 con 0x56361c6c4820 2024-04-26T01:59:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.713+0000 7f42a6f22740 1 -- 172.21.15.6:0/3775374084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4294003c30 con 0x56361c6c4820 2024-04-26T01:59:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.713+0000 7f4283400640 1 -- 172.21.15.6:0/3775374084 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f429400f600 con 0x56361c6c4820 2024-04-26T01:59:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.713+0000 7f42a6f22740 10 client.26786 fetch_fsmap learned FSMap version 44 2024-04-26T01:59:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.713+0000 7f42a6f22740 1 -- 172.21.15.6:0/3775374084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x56361c6de8e0 con 0x56361c6c4820 2024-04-26T01:59:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.714+0000 7f4283400640 1 -- 172.21.15.6:0/3775374084 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4294003c30 con 0x56361c6c4820 2024-04-26T01:59:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.714+0000 7f4283400640 1 -- 172.21.15.6:0/3775374084 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f4294019840 con 0x56361c6c4820 2024-04-26T01:59:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.714+0000 7f42a6f22740 10 client.26786 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T01:59:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.714+0000 7f42a6f22740 10 client.26786 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:59:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.714+0000 7f42a6f22740 20 client.26786 populate_metadata read hostname 'smithi006' 2024-04-26T01:59:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.714+0000 7f42a6f22740 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56361c6dfcf0 0x56361c6e20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.714+0000 7f42a6f22740 4 client.26786 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:59:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.714+0000 7f42a6f22740 1 -- 172.21.15.6:0/3775374084 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56361c63e8b0 con 0x56361c6dfcf0 2024-04-26T01:59:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.714+0000 7f42a1000640 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56361c6dfcf0 0x56361c6e20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.715+0000 7f42a1000640 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56361c6dfcf0 0x56361c6e20e0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51cc2a891ec9c11a server_cookie=7b895d9f009f33ec in_seq=0 out_seq=0 2024-04-26T01:59:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.715+0000 7f4283400640 10 client.26786 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.715+0000 7f4283400640 1 -- 172.21.15.6:0/3775374084 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56361c63e8b0 con 0x56361c6dfcf0 2024-04-26T01:59:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.715+0000 7f4283400640 10 client.26786 handle_command_reply: tid=0 2024-04-26T01:59:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f42a6f22740 1 client.26786 shutdown 2024-04-26T01:59:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f42a6f22740 1 -- 172.21.15.6:0/3775374084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6c4820 msgr2=0x56361c6cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f42a6f22740 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6c4820 0x56361c6cca30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4294009aa0 tx=0x7f429400f520 comp rx=0 tx=0).stop 2024-04-26T01:59:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f42a6f22740 1 -- 172.21.15.6:0/3775374084 shutdown_connections 2024-04-26T01:59:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f42a6f22740 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56361c6dfcf0 0x56361c6e20e0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f42a6f22740 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c5140 0x56361c6ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f42a6f22740 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6cfa00 0x56361c6cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f42a6f22740 1 --2- 172.21.15.6:0/3775374084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6c4820 0x56361c6cca30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f4283400640 0 client.26786 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f42a6f22740 1 -- 172.21.15.6:0/3775374084 >> 172.21.15.6:0/3775374084 conn(0x56361c6c0d30 msgr2=0x56361c6c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f42a6f22740 1 -- 172.21.15.6:0/3775374084 shutdown_connections 2024-04-26T01:59:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f42a6f22740 1 -- 172.21.15.6:0/3775374084 wait complete. 2024-04-26T01:59:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.716+0000 7f42a6f22740 20 client.26786 trim_cache size 0 max 16384 2024-04-26T01:59:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.735+0000 7f42a6f22740 1 Processor -- start 2024-04-26T01:59:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.735+0000 7f42a6f22740 1 -- start start 2024-04-26T01:59:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.735+0000 7f42a6f22740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c4820 0x56361c6bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.735+0000 7f42a6f22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 0x56361c6c71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.735+0000 7f42a6f22740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6cfa00 0x56361c6cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.735+0000 7f42a6f22740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f429c1b28f0 con 0x56361c6c4820 2024-04-26T01:59:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.735+0000 7f42a6f22740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f429c1b01c0 con 0x56361c6cfa00 2024-04-26T01:59:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.735+0000 7f42a6f22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f429c1ae030 con 0x56361c6c5140 2024-04-26T01:59:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.735+0000 7f42a1000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 0x56361c6c71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.735+0000 7f42a1000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 0x56361c6c71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58514/0 (socket says 172.21.15.6:58514) 2024-04-26T01:59:25.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.735+0000 7f42a1000640 1 -- 172.21.15.6:0/1366632385 learned_addr learned my addr 172.21.15.6:0/1366632385 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:25.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.735+0000 7f42a2400640 1 --2- 172.21.15.6:0/1366632385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c4820 0x56361c6bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.736 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.736+0000 7f42a2400640 1 --2- 172.21.15.6:0/1366632385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6cfa00 0x56361c6cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.736+0000 7f42a1000640 1 -- 172.21.15.6:0/1366632385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6cfa00 msgr2=0x56361c6cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.736+0000 7f42a1000640 1 --2- 172.21.15.6:0/1366632385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6cfa00 0x56361c6cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.736+0000 7f42a1000640 1 -- 172.21.15.6:0/1366632385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c4820 msgr2=0x56361c6bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.736+0000 7f42a1000640 1 --2- 172.21.15.6:0/1366632385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c4820 0x56361c6bfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.736+0000 7f42a1000640 1 -- 172.21.15.6:0/1366632385 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56361c6c4f60 con 0x56361c6c5140 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.736+0000 7f42a2400640 1 --2- 172.21.15.6:0/1366632385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6cfa00 0x56361c6cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.736+0000 7f42a1000640 1 --2- 172.21.15.6:0/1366632385 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 0x56361c6c71e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f429c100140 tx=0x7f4288005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8defa32227c1eaf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.736+0000 7f428e000640 1 -- 172.21.15.6:0/1366632385 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4288023030 con 0x56361c6c5140 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.736+0000 7f428e000640 1 -- 172.21.15.6:0/1366632385 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42880054d0 con 0x56361c6c5140 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.736+0000 7f428e000640 1 -- 172.21.15.6:0/1366632385 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428801f370 con 0x56361c6c5140 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.737+0000 7f42a6f22740 1 -- 172.21.15.6:0/1366632385 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 msgr2=0x56361c6c71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.737+0000 7f42a6f22740 1 --2- 172.21.15.6:0/1366632385 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 0x56361c6c71e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f429c100140 tx=0x7f4288005f90 comp rx=0 tx=0).stop 2024-04-26T01:59:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.737+0000 7f42a6f22740 1 -- 172.21.15.6:0/1366632385 shutdown_connections 2024-04-26T01:59:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.737+0000 7f42a6f22740 1 --2- 172.21.15.6:0/1366632385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6cfa00 0x56361c6cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.737+0000 7f42a6f22740 1 --2- 172.21.15.6:0/1366632385 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 0x56361c6c71e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.737+0000 7f42a6f22740 1 --2- 172.21.15.6:0/1366632385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6c4820 0x56361c6bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.737+0000 7f42a6f22740 1 -- 172.21.15.6:0/1366632385 >> 172.21.15.6:0/1366632385 conn(0x56361c6c0d30 msgr2=0x56361c6cc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.737+0000 7f42a6f22740 1 -- 172.21.15.6:0/1366632385 shutdown_connections 2024-04-26T01:59:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.737+0000 7f42a6f22740 1 -- 172.21.15.6:0/1366632385 wait complete. 2024-04-26T01:59:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.738+0000 7f42a6f22740 1 -- start start 2024-04-26T01:59:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.738+0000 7f42a6f22740 1 Processor -- start 2024-04-26T01:59:25.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.738+0000 7f42a6f22740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6c4820 0x56361c6d5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.738+0000 7f42a6f22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 0x56361c6d6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a6f22740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6cfa00 0x56361c74d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a6f22740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f429c1ae680 con 0x56361c6cfa00 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a6f22740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f429c1afaf0 con 0x56361c6c4820 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a6f22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f429c1aed50 con 0x56361c6c5140 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a6f22740 10 client.0 _ll_register_callbacks cb 0x7f4294004d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a1000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 0x56361c6d6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a1000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 0x56361c6d6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:58528/0 (socket says 172.21.15.6:58528) 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a1000640 1 -- 172.21.15.6:0/2682034375 learned_addr learned my addr 172.21.15.6:0/2682034375 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a2400640 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6cfa00 0x56361c74d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a2400640 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6c4820 0x56361c6d5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a1000640 1 -- 172.21.15.6:0/2682034375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6c4820 msgr2=0x56361c6d5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a1000640 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6c4820 0x56361c6d5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a1000640 1 -- 172.21.15.6:0/2682034375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6cfa00 msgr2=0x56361c74d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a1000640 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6cfa00 0x56361c74d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.739+0000 7f42a1000640 1 -- 172.21.15.6:0/2682034375 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56361c74def0 con 0x56361c6c5140 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.740+0000 7f42a1000640 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 0x56361c6d6190 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4288012620 tx=0x7f4288004e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a66ac204bc2d00a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.740+0000 7f4283400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.740+0000 7f4283400640 1 -- 172.21.15.6:0/2682034375 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4288023030 con 0x56361c6c5140 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.740+0000 7f4283400640 1 -- 172.21.15.6:0/2682034375 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f4288004730 con 0x56361c6c5140 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.740+0000 7f42a6f22740 1 -- 172.21.15.6:0/2682034375 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42880054d0 con 0x56361c6c5140 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.740+0000 7f4283400640 1 -- 172.21.15.6:0/2682034375 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f4288004930 con 0x56361c6c5140 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.740+0000 7f42a6f22740 10 client.26833 fetch_fsmap learned FSMap version 44 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.740+0000 7f42a6f22740 1 -- 172.21.15.6:0/2682034375 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x56361c74f180 con 0x56361c6c5140 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.741+0000 7f4283400640 1 -- 172.21.15.6:0/2682034375 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42880054d0 con 0x56361c6c5140 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.741+0000 7f4283400640 1 -- 172.21.15.6:0/2682034375 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f42880104f0 con 0x56361c6c5140 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.741+0000 7f42a6f22740 10 client.26833 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T01:59:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.741+0000 7f42a6f22740 10 client.26833 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:59:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.741+0000 7f42a6f22740 20 client.26833 populate_metadata read hostname 'smithi006' 2024-04-26T01:59:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.741+0000 7f42a6f22740 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56361c74f8e0 0x56361c751d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.741+0000 7f42a6f22740 4 client.26833 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:59:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.741+0000 7f42a6f22740 1 -- 172.21.15.6:0/2682034375 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56361c638320 con 0x56361c74f8e0 2024-04-26T01:59:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.742+0000 7f42a2400640 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56361c74f8e0 0x56361c751d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.742+0000 7f42a2400640 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56361c74f8e0 0x56361c751d00 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fb5e3c6fe27cd4 server_cookie=5e23ab5276fc2482 in_seq=0 out_seq=0 2024-04-26T01:59:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.742+0000 7f4283400640 10 client.26833 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.779+0000 7f4283400640 1 -- 172.21.15.6:0/2682034375 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56361c638320 con 0x56361c74f8e0 2024-04-26T01:59:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.779+0000 7f4283400640 10 client.26833 handle_command_reply: tid=0 2024-04-26T01:59:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.779+0000 7f42a6f22740 1 client.26833 shutdown 2024-04-26T01:59:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.780+0000 7f42a6f22740 1 -- 172.21.15.6:0/2682034375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 msgr2=0x56361c6d6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.780+0000 7f42a6f22740 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 0x56361c6d6190 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4288012620 tx=0x7f4288004e50 comp rx=0 tx=0).stop 2024-04-26T01:59:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.780+0000 7f42a6f22740 1 -- 172.21.15.6:0/2682034375 shutdown_connections 2024-04-26T01:59:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.780+0000 7f42a6f22740 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56361c74f8e0 0x56361c751d00 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.780+0000 7f42a6f22740 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56361c6cfa00 0x56361c74d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.780+0000 7f42a6f22740 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56361c6c5140 0x56361c6d6190 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.780+0000 7f42a6f22740 1 --2- 172.21.15.6:0/2682034375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56361c6c4820 0x56361c6d5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.780+0000 7f4283400640 0 client.26833 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.780+0000 7f42a6f22740 1 -- 172.21.15.6:0/2682034375 >> 172.21.15.6:0/2682034375 conn(0x56361c6c0d30 msgr2=0x56361c6bfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.780+0000 7f42a6f22740 1 -- 172.21.15.6:0/2682034375 shutdown_connections 2024-04-26T01:59:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.780+0000 7f42a6f22740 1 -- 172.21.15.6:0/2682034375 wait complete. 2024-04-26T01:59:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.780+0000 7f42a6f22740 20 client.26833 trim_cache size 0 max 16384 2024-04-26T01:59:25.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:59:25.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:59:25.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:59:25.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:59:25.781 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.781+0000 7f42a4400640 1 -- 172.21.15.6:0/354564790 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f42840a3d90 msgr2=0x7f42840a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.781+0000 7f42a4400640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f42840a3d90 0x7f42840a6250 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x56361c6c5dc0 tx=0x7f4290009260 comp rx=0 tx=0).stop 2024-04-26T01:59:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.781+0000 7f42a4400640 1 -- 172.21.15.6:0/354564790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f429c1043c0 msgr2=0x7f429c1a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.781+0000 7f42a4400640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f429c1043c0 0x7f429c1a0650 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4290005ab0 tx=0x7f4290003c10 comp rx=0 tx=0).stop 2024-04-26T01:59:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.781+0000 7f42a4400640 1 -- 172.21.15.6:0/354564790 shutdown_connections 2024-04-26T01:59:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.781+0000 7f42a4400640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f42840a3d90 0x7f42840a6250 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.781+0000 7f42a4400640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f429c19a7c0 0x7f429c19ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.781+0000 7f42a4400640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f429c109490 0x7f429c1a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.781+0000 7f42a4400640 1 --2- 172.21.15.6:0/354564790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f429c1043c0 0x7f429c1a0650 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.781+0000 7f42a4400640 1 -- 172.21.15.6:0/354564790 >> 172.21.15.6:0/354564790 conn(0x7f429c0ffc20 msgr2=0x7f429c100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.781+0000 7f42a4400640 1 -- 172.21.15.6:0/354564790 shutdown_connections 2024-04-26T01:59:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:25.782+0000 7f42a4400640 1 -- 172.21.15.6:0/354564790 wait complete. 2024-04-26T01:59:26.645 INFO:tasks.workunit.client.0.smithi006.stdout: 1 472938 3.50 MB/sec execute 590 sec latency 24.663 ms 2024-04-26T01:59:26.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:26 smithi006 ceph-mon[28419]: pgmap v428: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 715 KiB/s wr, 80 op/s 2024-04-26T01:59:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:26 smithi188 ceph-mon[32751]: pgmap v428: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 715 KiB/s wr, 80 op/s 2024-04-26T01:59:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:26 smithi029 ceph-mon[32775]: pgmap v428: 97 pgs: 97 active+clean; 143 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 715 KiB/s wr, 80 op/s 2024-04-26T01:59:27.645 INFO:tasks.workunit.client.0.smithi006.stdout: 1 473974 3.51 MB/sec execute 591 sec latency 49.809 ms 2024-04-26T01:59:28.645 INFO:tasks.workunit.client.0.smithi006.stdout: 1 475467 3.51 MB/sec execute 592 sec latency 23.275 ms 2024-04-26T01:59:28.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:28 smithi006 ceph-mon[28419]: pgmap v429: 97 pgs: 97 active+clean; 144 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-04-26T01:59:28.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:28 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:59:28.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:28 smithi188 ceph-mon[32751]: pgmap v429: 97 pgs: 97 active+clean; 144 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-04-26T01:59:28.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:28 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:59:28.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:28 smithi029 ceph-mon[32775]: pgmap v429: 97 pgs: 97 active+clean; 144 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-04-26T01:59:28.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:28 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:59:29.645 INFO:tasks.workunit.client.0.smithi006.stdout: 1 476471 3.52 MB/sec execute 593 sec latency 31.104 ms 2024-04-26T01:59:29.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:29 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:59:29.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:29 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:59:29.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:29 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:59:29.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:29 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:59:29.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:29 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:59:29.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:29 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:59:29.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:29 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:59:29.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:29 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:59:29.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:29 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T01:59:30.645 INFO:tasks.workunit.client.0.smithi006.stdout: 1 477257 3.52 MB/sec execute 594 sec latency 19.644 ms 2024-04-26T01:59:30.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:30 smithi006 ceph-mon[28419]: pgmap v430: 97 pgs: 97 active+clean; 144 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.5 MiB/s wr, 121 op/s 2024-04-26T01:59:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:30 smithi188 ceph-mon[32751]: pgmap v430: 97 pgs: 97 active+clean; 144 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.5 MiB/s wr, 121 op/s 2024-04-26T01:59:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:30 smithi029 ceph-mon[32775]: pgmap v430: 97 pgs: 97 active+clean; 144 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.5 MiB/s wr, 121 op/s 2024-04-26T01:59:31.645 INFO:tasks.workunit.client.0.smithi006.stdout: 1 478512 3.52 MB/sec execute 595 sec latency 35.164 ms 2024-04-26T01:59:32.646 INFO:tasks.workunit.client.0.smithi006.stdout: 1 479682 3.53 MB/sec execute 596 sec latency 38.840 ms 2024-04-26T01:59:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:32 smithi006 ceph-mon[28419]: pgmap v431: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 2.3 MiB/s wr, 189 op/s 2024-04-26T01:59:32.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:32 smithi188 ceph-mon[32751]: pgmap v431: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 2.3 MiB/s wr, 189 op/s 2024-04-26T01:59:32.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:32 smithi029 ceph-mon[32775]: pgmap v431: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 2.3 MiB/s wr, 189 op/s 2024-04-26T01:59:33.646 INFO:tasks.workunit.client.0.smithi006.stdout: 1 480135 3.53 MB/sec execute 597 sec latency 25.084 ms 2024-04-26T01:59:34.646 INFO:tasks.workunit.client.0.smithi006.stdout: 1 481078 3.53 MB/sec execute 598 sec latency 37.645 ms 2024-04-26T01:59:34.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:34 smithi006 ceph-mon[28419]: pgmap v432: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 1.7 MiB/s wr, 150 op/s 2024-04-26T01:59:34.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:34 smithi188 ceph-mon[32751]: pgmap v432: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 1.7 MiB/s wr, 150 op/s 2024-04-26T01:59:34.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:34 smithi029 ceph-mon[32775]: pgmap v432: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 1.7 MiB/s wr, 150 op/s 2024-04-26T01:59:35.646 INFO:tasks.workunit.client.0.smithi006.stdout: 1 482266 3.53 MB/sec execute 599 sec latency 28.715 ms 2024-04-26T01:59:36.646 INFO:tasks.workunit.client.0.smithi006.stdout: 1 cleanup 600 sec 2024-04-26T01:59:36.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:36 smithi006 ceph-mon[28419]: pgmap v433: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 1.7 MiB/s wr, 150 op/s 2024-04-26T01:59:36.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:36 smithi188 ceph-mon[32751]: pgmap v433: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 1.7 MiB/s wr, 150 op/s 2024-04-26T01:59:36.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:36 smithi029 ceph-mon[32775]: pgmap v433: 97 pgs: 97 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 1.7 MiB/s wr, 150 op/s 2024-04-26T01:59:36.978 INFO:tasks.workunit.client.0.smithi006.stdout: 0 cleanup 600 sec 2024-04-26T01:59:36.978 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-26T01:59:36.978 INFO:tasks.workunit.client.0.smithi006.stdout: Operation Count AvgLat MaxLat 2024-04-26T01:59:36.978 INFO:tasks.workunit.client.0.smithi006.stdout: ---------------------------------------- 2024-04-26T01:59:36.978 INFO:tasks.workunit.client.0.smithi006.stdout: NTCreateX 67424 3.943 30786.314 2024-04-26T01:59:36.978 INFO:tasks.workunit.client.0.smithi006.stdout: Close 49579 0.013 0.457 2024-04-26T01:59:36.978 INFO:tasks.workunit.client.0.smithi006.stdout: Rename 2861 16.044 23580.504 2024-04-26T01:59:36.978 INFO:tasks.workunit.client.0.smithi006.stdout: Unlink 13580 7.379 29988.367 2024-04-26T01:59:36.978 INFO:tasks.workunit.client.0.smithi006.stdout: Deltree 2 149.234 201.006 2024-04-26T01:59:36.978 INFO:tasks.workunit.client.0.smithi006.stdout: Mkdir 1 0.020 0.020 2024-04-26T01:59:36.978 INFO:tasks.workunit.client.0.smithi006.stdout: Qpathinfo 61132 1.751 7172.615 2024-04-26T01:59:36.978 INFO:tasks.workunit.client.0.smithi006.stdout: Qfileinfo 10739 0.005 0.107 2024-04-26T01:59:36.979 INFO:tasks.workunit.client.0.smithi006.stdout: Qfsinfo 11212 0.508 46.161 2024-04-26T01:59:36.979 INFO:tasks.workunit.client.0.smithi006.stdout: Sfileinfo 5501 7.009 37678.258 2024-04-26T01:59:36.979 INFO:tasks.workunit.client.0.smithi006.stdout: Find 23650 0.451 38.366 2024-04-26T01:59:36.979 INFO:tasks.workunit.client.0.smithi006.stdout: WriteX 33727 0.173 66.378 2024-04-26T01:59:36.979 INFO:tasks.workunit.client.0.smithi006.stdout: ReadX 105872 0.077 266.290 2024-04-26T01:59:36.979 INFO:tasks.workunit.client.0.smithi006.stdout: LockX 220 0.976 2.253 2024-04-26T01:59:36.979 INFO:tasks.workunit.client.0.smithi006.stdout: UnlockX 220 0.831 3.754 2024-04-26T01:59:36.979 INFO:tasks.workunit.client.0.smithi006.stdout: Flush 4730 5.446 66.000 2024-04-26T01:59:36.979 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-26T01:59:36.979 INFO:tasks.workunit.client.0.smithi006.stdout:Throughput 3.52772 MB/sec 1 clients 1 procs max_latency=37678.270 ms 2024-04-26T01:59:36.980 INFO:tasks.workunit.client.0.smithi006.stderr:+ dbench 10 2024-04-26T01:59:36.982 INFO:tasks.workunit.client.0.smithi006.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-26T01:59:36.982 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-26T01:59:36.982 INFO:tasks.workunit.client.0.smithi006.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-26T01:59:36.986 INFO:tasks.workunit.client.0.smithi006.stdout:9 of 10 processes prepared for launch 0 sec 2024-04-26T01:59:37.086 INFO:tasks.workunit.client.0.smithi006.stdout:10 of 10 processes prepared for launch 0 sec 2024-04-26T01:59:37.086 INFO:tasks.workunit.client.0.smithi006.stdout:releasing clients 2024-04-26T01:59:38.087 INFO:tasks.workunit.client.0.smithi006.stdout: 10 332 120.26 MB/sec warmup 1 sec latency 115.892 ms 2024-04-26T01:59:38.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:38 smithi006 ceph-mon[28419]: pgmap v434: 97 pgs: 97 active+clean; 153 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 209 op/s 2024-04-26T01:59:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:38 smithi188 ceph-mon[32751]: pgmap v434: 97 pgs: 97 active+clean; 153 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 209 op/s 2024-04-26T01:59:38.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:38 smithi029 ceph-mon[32775]: pgmap v434: 97 pgs: 97 active+clean; 153 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 209 op/s 2024-04-26T01:59:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.964+0000 7f7d14a00640 1 -- 172.21.15.6:0/2733978231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d10102b50 msgr2=0x7f7d10106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.964+0000 7f7d14a00640 1 --2- 172.21.15.6:0/2733978231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d10102b50 0x7f7d10106b90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7d00009900 tx=0x7f7d0002a410 comp rx=0 tx=0).stop 2024-04-26T01:59:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.964+0000 7f7d14a00640 1 -- 172.21.15.6:0/2733978231 shutdown_connections 2024-04-26T01:59:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.964+0000 7f7d14a00640 1 --2- 172.21.15.6:0/2733978231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d10107270 0x7f7d1010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.964+0000 7f7d14a00640 1 --2- 172.21.15.6:0/2733978231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d10102b50 0x7f7d10106b90 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.964+0000 7f7d14a00640 1 --2- 172.21.15.6:0/2733978231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d101021a0 0x7f7d10102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.964+0000 7f7d14a00640 1 -- 172.21.15.6:0/2733978231 >> 172.21.15.6:0/2733978231 conn(0x7f7d100fda20 msgr2=0x7f7d100ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.965+0000 7f7d14a00640 1 -- 172.21.15.6:0/2733978231 shutdown_connections 2024-04-26T01:59:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.965+0000 7f7d14a00640 1 -- 172.21.15.6:0/2733978231 wait complete. 2024-04-26T01:59:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.966+0000 7f7d14a00640 1 Processor -- start 2024-04-26T01:59:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.966+0000 7f7d14a00640 1 -- start start 2024-04-26T01:59:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d14a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d101021a0 0x7f7d101a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d14a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d10107270 0x7f7d101a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d14a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d1019a780 0x7f7d1019ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d14a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7d10112200 con 0x7f7d1019a780 2024-04-26T01:59:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d14a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7d10112080 con 0x7f7d101021a0 2024-04-26T01:59:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d14a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7d10112380 con 0x7f7d10107270 2024-04-26T01:59:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d0de00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d101021a0 0x7f7d101a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d0de00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d101021a0 0x7f7d101a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:50040/0 (socket says 172.21.15.6:50040) 2024-04-26T01:59:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d0de00640 1 -- 172.21.15.6:0/1808494802 learned_addr learned my addr 172.21.15.6:0/1808494802 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d05400640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d10107270 0x7f7d101a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d0e800640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d1019a780 0x7f7d1019ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d0de00640 1 -- 172.21.15.6:0/1808494802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d10107270 msgr2=0x7f7d101a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d0de00640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d10107270 0x7f7d101a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d0de00640 1 -- 172.21.15.6:0/1808494802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d1019a780 msgr2=0x7f7d1019ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d0de00640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d1019a780 0x7f7d1019ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.967+0000 7f7d0de00640 1 -- 172.21.15.6:0/1808494802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1019b310 con 0x7f7d101021a0 2024-04-26T01:59:38.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.968+0000 7f7d0de00640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d101021a0 0x7f7d101a0610 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800e9f0 tx=0x7f7cf800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:38.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.968+0000 7f7d07400640 1 -- 172.21.15.6:0/1808494802 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf800cd80 con 0x7f7d101021a0 2024-04-26T01:59:38.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.968+0000 7f7d07400640 1 -- 172.21.15.6:0/1808494802 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cf80108e0 con 0x7f7d101021a0 2024-04-26T01:59:38.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.968+0000 7f7d07400640 1 -- 172.21.15.6:0/1808494802 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8018980 con 0x7f7d101021a0 2024-04-26T01:59:38.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.968+0000 7f7d14a00640 1 -- 172.21.15.6:0/1808494802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d1019b600 con 0x7f7d101021a0 2024-04-26T01:59:38.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.968+0000 7f7d14a00640 1 -- 172.21.15.6:0/1808494802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d101a7760 con 0x7f7d101021a0 2024-04-26T01:59:38.970 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.972+0000 7f7d07400640 1 -- 172.21.15.6:0/1808494802 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f7cf8010430 con 0x7f7d101021a0 2024-04-26T01:59:38.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.972+0000 7f7d173fd740 1 Processor -- start 2024-04-26T01:59:38.971 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.973+0000 7f7d173fd740 1 -- start start 2024-04-26T01:59:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.973+0000 7f7d173fd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 0x556983e63c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.973+0000 7f7d173fd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e64140 0x556983e6e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.973+0000 7f7d173fd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 0x556983e70df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.973+0000 7f7d173fd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x556983dc9900 con 0x556983e63820 2024-04-26T01:59:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.973+0000 7f7d173fd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556983d6fef0 con 0x556983e6ea00 2024-04-26T01:59:38.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.973+0000 7f7d173fd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556983dc58c0 con 0x556983e64140 2024-04-26T01:59:38.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.974+0000 7f7d0de00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e64140 0x556983e6e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:38.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.974+0000 7f7d0e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 0x556983e63c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.974+0000 7f7d0e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 0x556983e63c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55094/0 (socket says 172.21.15.6:55094) 2024-04-26T01:59:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.974+0000 7f7d0e800640 1 -- 172.21.15.6:0/1035869066 learned_addr learned my addr 172.21.15.6:0/1035869066 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.974+0000 7f7d05400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 0x556983e70df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.974+0000 7f7d0de00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e64140 0x556983e6e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59150/0 (socket says 172.21.15.6:59150) 2024-04-26T01:59:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.974+0000 7f7d0e800640 1 -- 172.21.15.6:0/1035869066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e64140 msgr2=0x556983e6e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.974+0000 7f7d0e800640 1 --2- 172.21.15.6:0/1035869066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e64140 0x556983e6e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.974+0000 7f7d0e800640 1 -- 172.21.15.6:0/1035869066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 msgr2=0x556983e70df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.974+0000 7f7d0e800640 1 --2- 172.21.15.6:0/1035869066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 0x556983e70df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.974+0000 7f7d0e800640 1 -- 172.21.15.6:0/1035869066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556983adc210 con 0x556983e63820 2024-04-26T01:59:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.975+0000 7f7d07400640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7cf40a3d90 0x7f7cf40a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.975+0000 7f7d0e800640 1 --2- 172.21.15.6:0/1035869066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 0x556983e63c00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009f40 tx=0x7f7d08010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1330560b48011729 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.975+0000 7f7d07400640 1 -- 172.21.15.6:0/1808494802 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f7cf80d0ca0 con 0x7f7d101021a0 2024-04-26T01:59:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.975+0000 7f7cff000640 1 -- 172.21.15.6:0/1035869066 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08003b70 con 0x556983e63820 2024-04-26T01:59:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.975+0000 7f7d0de00640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7cf40a3d90 0x7f7cf40a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.975+0000 7f7cff000640 1 -- 172.21.15.6:0/1035869066 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d08003d10 con 0x556983e63820 2024-04-26T01:59:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.975+0000 7f7cff000640 1 -- 172.21.15.6:0/1035869066 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08011960 con 0x556983e63820 2024-04-26T01:59:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.976+0000 7f7d173fd740 1 -- 172.21.15.6:0/1035869066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 msgr2=0x556983e63c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.976+0000 7f7d173fd740 1 --2- 172.21.15.6:0/1035869066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 0x556983e63c00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009f40 tx=0x7f7d08010860 comp rx=0 tx=0).stop 2024-04-26T01:59:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.976+0000 7f7d0de00640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7cf40a3d90 0x7f7cf40a6250 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x556983e64dc0 tx=0x7f7cf80d4750 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.976+0000 7f7d173fd740 1 -- 172.21.15.6:0/1035869066 shutdown_connections 2024-04-26T01:59:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.976+0000 7f7d173fd740 1 --2- 172.21.15.6:0/1035869066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 0x556983e70df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.976+0000 7f7d173fd740 1 --2- 172.21.15.6:0/1035869066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e64140 0x556983e6e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.976+0000 7f7d173fd740 1 --2- 172.21.15.6:0/1035869066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 0x556983e63c00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.976+0000 7f7d173fd740 1 -- 172.21.15.6:0/1035869066 >> 172.21.15.6:0/1035869066 conn(0x556983e5fd30 msgr2=0x556983e60110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.977+0000 7f7d173fd740 1 -- 172.21.15.6:0/1035869066 shutdown_connections 2024-04-26T01:59:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.977+0000 7f7d173fd740 1 -- 172.21.15.6:0/1035869066 wait complete. 2024-04-26T01:59:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.977+0000 7f7d173fd740 1 -- start start 2024-04-26T01:59:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.978+0000 7f7d173fd740 1 Processor -- start 2024-04-26T01:59:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.978+0000 7f7d173fd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 0x556983e6ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d173fd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e64140 0x556983e6bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d173fd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 0x556983e6c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d173fd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x556983e71630 con 0x556983e64140 2024-04-26T01:59:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d173fd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556983d83310 con 0x556983e6ea00 2024-04-26T01:59:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d173fd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556983e717b0 con 0x556983e63820 2024-04-26T01:59:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d0e800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 0x556983e6ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d0e800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 0x556983e6ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59160/0 (socket says 172.21.15.6:59160) 2024-04-26T01:59:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d0e800640 1 -- 172.21.15.6:0/214710085 learned_addr learned my addr 172.21.15.6:0/214710085 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d0e800640 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 0x556983e6c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d173fd740 10 client.0 _ll_register_callbacks cb 0x556983d51cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d0e800640 1 -- 172.21.15.6:0/214710085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 msgr2=0x556983e6c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d05400640 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e64140 0x556983e6bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d0e800640 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 0x556983e6c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.979+0000 7f7d0e800640 1 -- 172.21.15.6:0/214710085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e64140 msgr2=0x556983e6bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.980+0000 7f7d0e800640 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e64140 0x556983e6bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.980+0000 7f7d0e800640 1 -- 172.21.15.6:0/214710085 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556983ba8620 con 0x556983e63820 2024-04-26T01:59:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.980+0000 7f7d0e800640 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 0x556983e6ba30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800a010 tx=0x7f7d08005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ec9e1b6d4dc9731 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.980+0000 7f7cf3400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:59:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.980+0000 7f7cf3400640 1 -- 172.21.15.6:0/214710085 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08025030 con 0x556983e63820 2024-04-26T01:59:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.980+0000 7f7d173fd740 1 -- 172.21.15.6:0/214710085 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d08003d10 con 0x556983e63820 2024-04-26T01:59:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.980+0000 7f7cf3400640 1 -- 172.21.15.6:0/214710085 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f7d08011ce0 con 0x556983e63820 2024-04-26T01:59:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.981+0000 7f7cf3400640 1 -- 172.21.15.6:0/214710085 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f7d08003b60 con 0x556983e63820 2024-04-26T01:59:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.981+0000 7f7d173fd740 10 client.26845 fetch_fsmap learned FSMap version 44 2024-04-26T01:59:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.981+0000 7f7d173fd740 1 -- 172.21.15.6:0/214710085 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x556983e86110 con 0x556983e63820 2024-04-26T01:59:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.981+0000 7f7cf3400640 1 -- 172.21.15.6:0/214710085 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d08003d10 con 0x556983e63820 2024-04-26T01:59:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.981+0000 7f7cf3400640 1 -- 172.21.15.6:0/214710085 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f7d080050d0 con 0x556983e63820 2024-04-26T01:59:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.981+0000 7f7d173fd740 10 client.26845 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T01:59:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.981+0000 7f7d173fd740 10 client.26845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:59:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.981+0000 7f7d173fd740 20 client.26845 populate_metadata read hostname 'smithi006' 2024-04-26T01:59:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.981+0000 7f7d173fd740 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556983e87520 0x556983e89910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.981+0000 7f7d173fd740 4 client.26845 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:59:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.981+0000 7f7d173fd740 1 -- 172.21.15.6:0/214710085 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556983ddd8b0 con 0x556983e87520 2024-04-26T01:59:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.982+0000 7f7d05400640 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556983e87520 0x556983e89910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.982+0000 7f7d05400640 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556983e87520 0x556983e89910 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84ccfd5648e6909f server_cookie=33e2573d521268cc in_seq=0 out_seq=0 2024-04-26T01:59:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.982+0000 7f7cf3400640 10 client.26845 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7cf3400640 1 -- 172.21.15.6:0/214710085 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556983ddd8b0 con 0x556983e87520 2024-04-26T01:59:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7cf3400640 10 client.26845 handle_command_reply: tid=0 2024-04-26T01:59:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7d173fd740 1 client.26845 shutdown 2024-04-26T01:59:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7d173fd740 1 -- 172.21.15.6:0/214710085 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 msgr2=0x556983e6ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7d173fd740 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 0x556983e6ba30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800a010 tx=0x7f7d08005bd0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7d173fd740 1 -- 172.21.15.6:0/214710085 shutdown_connections 2024-04-26T01:59:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7d173fd740 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556983e87520 0x556983e89910 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7d173fd740 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e64140 0x556983e6bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7d173fd740 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 0x556983e6c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7cf3400640 0 client.26845 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7d173fd740 1 --2- 172.21.15.6:0/214710085 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 0x556983e6ba30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7d173fd740 1 -- 172.21.15.6:0/214710085 >> 172.21.15.6:0/214710085 conn(0x556983e5fd30 msgr2=0x556983e65a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7d173fd740 1 -- 172.21.15.6:0/214710085 shutdown_connections 2024-04-26T01:59:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7d173fd740 1 -- 172.21.15.6:0/214710085 wait complete. 2024-04-26T01:59:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:38.985+0000 7f7d173fd740 20 client.26845 trim_cache size 0 max 16384 2024-04-26T01:59:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.005+0000 7f7d173fd740 1 Processor -- start 2024-04-26T01:59:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.005+0000 7f7d173fd740 1 -- start start 2024-04-26T01:59:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d173fd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 0x556983e5f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d173fd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e64140 0x556983e6b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d173fd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e6ea00 0x556983e8eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d173fd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7d101ae4d0 con 0x556983e63820 2024-04-26T01:59:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d173fd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7d101abda0 con 0x556983e64140 2024-04-26T01:59:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d173fd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7d101a9c70 con 0x556983e6ea00 2024-04-26T01:59:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d0e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 0x556983e5f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d0e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 0x556983e5f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55116/0 (socket says 172.21.15.6:55116) 2024-04-26T01:59:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d0e800640 1 -- 172.21.15.6:0/2504279146 learned_addr learned my addr 172.21.15.6:0/2504279146 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d05400640 1 --2- 172.21.15.6:0/2504279146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e64140 0x556983e6b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d0e800640 1 --2- 172.21.15.6:0/2504279146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e6ea00 0x556983e8eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d05400640 1 -- 172.21.15.6:0/2504279146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e6ea00 msgr2=0x556983e8eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.006+0000 7f7d05400640 1 --2- 172.21.15.6:0/2504279146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e6ea00 0x556983e8eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.007+0000 7f7d05400640 1 -- 172.21.15.6:0/2504279146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 msgr2=0x556983e5f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.007+0000 7f7d05400640 1 --2- 172.21.15.6:0/2504279146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 0x556983e5f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.007+0000 7f7d05400640 1 -- 172.21.15.6:0/2504279146 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556983e67630 con 0x556983e64140 2024-04-26T01:59:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.007+0000 7f7d05400640 1 --2- 172.21.15.6:0/2504279146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e64140 0x556983e6b330 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7d0002a930 tx=0x7f7d00000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=271118ef0ab42d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.007+0000 7f7cfe600640 1 -- 172.21.15.6:0/2504279146 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00035040 con 0x556983e64140 2024-04-26T01:59:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.007+0000 7f7cfe600640 1 -- 172.21.15.6:0/2504279146 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d000043c0 con 0x556983e64140 2024-04-26T01:59:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.007+0000 7f7cfe600640 1 -- 172.21.15.6:0/2504279146 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00043310 con 0x556983e64140 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.008+0000 7f7d173fd740 1 -- 172.21.15.6:0/2504279146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e64140 msgr2=0x556983e6b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.008+0000 7f7d173fd740 1 --2- 172.21.15.6:0/2504279146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e64140 0x556983e6b330 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7d0002a930 tx=0x7f7d00000bc0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.008+0000 7f7d173fd740 1 -- 172.21.15.6:0/2504279146 shutdown_connections 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.008+0000 7f7d173fd740 1 --2- 172.21.15.6:0/2504279146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e6ea00 0x556983e8eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.008+0000 7f7d173fd740 1 --2- 172.21.15.6:0/2504279146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e64140 0x556983e6b330 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.008+0000 7f7d173fd740 1 --2- 172.21.15.6:0/2504279146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e63820 0x556983e5f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.008+0000 7f7d173fd740 1 -- 172.21.15.6:0/2504279146 >> 172.21.15.6:0/2504279146 conn(0x556983e5fd30 msgr2=0x556983e6c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.009+0000 7f7d173fd740 1 -- 172.21.15.6:0/2504279146 shutdown_connections 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.009+0000 7f7d173fd740 1 -- 172.21.15.6:0/2504279146 wait complete. 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.009+0000 7f7d173fd740 1 -- start start 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.009+0000 7f7d173fd740 1 Processor -- start 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d173fd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 0x556983e7cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d173fd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e64140 0x556983e7d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d173fd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 0x556983e7d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d173fd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7d101aa2c0 con 0x556983e64140 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d173fd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7d101ab6d0 con 0x556983e6ea00 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d173fd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7d101aa930 con 0x556983e63820 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d173fd740 10 client.0 _ll_register_callbacks cb 0x7f7d08004de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d0e800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 0x556983e7cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d0e800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 0x556983e7cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59184/0 (socket says 172.21.15.6:59184) 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d05400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e64140 0x556983e7d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d0e800640 1 -- 172.21.15.6:0/1434463491 learned_addr learned my addr 172.21.15.6:0/1434463491 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d0e800640 1 --2- 172.21.15.6:0/1434463491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 0x556983e7d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d0e800640 1 -- 172.21.15.6:0/1434463491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 msgr2=0x556983e7d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d0e800640 1 --2- 172.21.15.6:0/1434463491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 0x556983e7d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d0e800640 1 -- 172.21.15.6:0/1434463491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e64140 msgr2=0x556983e7d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d0e800640 1 --2- 172.21.15.6:0/1434463491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e64140 0x556983e7d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d0e800640 1 -- 172.21.15.6:0/1434463491 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556983eec800 con 0x556983e63820 2024-04-26T01:59:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.010+0000 7f7d0e800640 1 --2- 172.21.15.6:0/1434463491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 0x556983e7cd50 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7d08010ee0 tx=0x7f7d08013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d8fc62f1dfc0565 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.011+0000 7f7cf3400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.011+0000 7f7cf3400640 1 -- 172.21.15.6:0/1434463491 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08011ce0 con 0x556983e63820 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.011+0000 7f7d173fd740 1 -- 172.21.15.6:0/1434463491 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d000043c0 con 0x556983e63820 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.011+0000 7f7cf3400640 1 -- 172.21.15.6:0/1434463491 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f7d0803b430 con 0x556983e63820 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.011+0000 7f7cf3400640 1 -- 172.21.15.6:0/1434463491 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f7d0803b630 con 0x556983e63820 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.011+0000 7f7d173fd740 10 client.26851 fetch_fsmap learned FSMap version 44 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.011+0000 7f7d173fd740 1 -- 172.21.15.6:0/1434463491 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x556983eedaf0 con 0x556983e63820 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.012+0000 7f7cf3400640 1 -- 172.21.15.6:0/1434463491 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d000043c0 con 0x556983e63820 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.012+0000 7f7cf3400640 1 -- 172.21.15.6:0/1434463491 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f7d0802c650 con 0x556983e63820 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.012+0000 7f7d173fd740 10 client.26851 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.012+0000 7f7d173fd740 10 client.26851 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.012+0000 7f7d173fd740 20 client.26851 populate_metadata read hostname 'smithi006' 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.012+0000 7f7d173fd740 1 --2- 172.21.15.6:0/1434463491 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556983eee100 0x556983ef04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.012+0000 7f7d173fd740 4 client.26851 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:59:39.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.012+0000 7f7d173fd740 1 -- 172.21.15.6:0/1434463491 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556983e66170 con 0x556983eee100 2024-04-26T01:59:39.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.013+0000 7f7d05400640 1 --2- 172.21.15.6:0/1434463491 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556983eee100 0x556983ef04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:39.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.013+0000 7f7d05400640 1 --2- 172.21.15.6:0/1434463491 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556983eee100 0x556983ef04f0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2c5b080567a91ce server_cookie=4de8e09bb041d5f6 in_seq=0 out_seq=0 2024-04-26T01:59:39.017 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.013+0000 7f7cf3400640 10 client.26851 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:39.087 INFO:tasks.workunit.client.0.smithi006.stdout: 10 699 103.19 MB/sec warmup 2 sec latency 116.900 ms 2024-04-26T01:59:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.321+0000 7f7cf3400640 1 -- 172.21.15.6:0/1434463491 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556983e66170 con 0x556983eee100 2024-04-26T01:59:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7cf3400640 10 client.26851 handle_command_reply: tid=0 2024-04-26T01:59:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7d173fd740 1 client.26851 shutdown 2024-04-26T01:59:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7d173fd740 1 -- 172.21.15.6:0/1434463491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 msgr2=0x556983e7cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7d173fd740 1 --2- 172.21.15.6:0/1434463491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 0x556983e7cd50 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7d08010ee0 tx=0x7f7d08013040 comp rx=0 tx=0).stop 2024-04-26T01:59:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7d173fd740 1 -- 172.21.15.6:0/1434463491 shutdown_connections 2024-04-26T01:59:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7d173fd740 1 --2- 172.21.15.6:0/1434463491 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x556983eee100 0x556983ef04f0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7d173fd740 1 --2- 172.21.15.6:0/1434463491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556983e64140 0x556983e7d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7cf3400640 0 client.26851 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7d173fd740 1 --2- 172.21.15.6:0/1434463491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556983e6ea00 0x556983e7d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7d173fd740 1 --2- 172.21.15.6:0/1434463491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556983e63820 0x556983e7cd50 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7d173fd740 1 -- 172.21.15.6:0/1434463491 >> 172.21.15.6:0/1434463491 conn(0x556983e5fd30 msgr2=0x556983e6d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7d173fd740 1 -- 172.21.15.6:0/1434463491 shutdown_connections 2024-04-26T01:59:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.322+0000 7f7d173fd740 1 -- 172.21.15.6:0/1434463491 wait complete. 2024-04-26T01:59:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.323+0000 7f7d173fd740 20 client.26851 trim_cache size 0 max 16384 2024-04-26T01:59:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:59:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:59:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:59:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:59:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.324+0000 7f7d14a00640 1 -- 172.21.15.6:0/1808494802 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7cf40a3d90 msgr2=0x7f7cf40a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.324+0000 7f7d14a00640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7cf40a3d90 0x7f7cf40a6250 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x556983e64dc0 tx=0x7f7cf80d4750 comp rx=0 tx=0).stop 2024-04-26T01:59:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.324+0000 7f7d14a00640 1 -- 172.21.15.6:0/1808494802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d101021a0 msgr2=0x7f7d101a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.324+0000 7f7d14a00640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d101021a0 0x7f7d101a0610 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800e9f0 tx=0x7f7cf800eeb0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.324+0000 7f7d14a00640 1 -- 172.21.15.6:0/1808494802 shutdown_connections 2024-04-26T01:59:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.324+0000 7f7d14a00640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f7cf40a3d90 0x7f7cf40a6250 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.324+0000 7f7d14a00640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d1019a780 0x7f7d1019ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.324+0000 7f7d14a00640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d10107270 0x7f7d101a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.324+0000 7f7d14a00640 1 --2- 172.21.15.6:0/1808494802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d101021a0 0x7f7d101a0610 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.324+0000 7f7d14a00640 1 -- 172.21.15.6:0/1808494802 >> 172.21.15.6:0/1808494802 conn(0x7f7d100fda20 msgr2=0x7f7d100ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.324+0000 7f7d14a00640 1 -- 172.21.15.6:0/1808494802 shutdown_connections 2024-04-26T01:59:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:39.324+0000 7f7d14a00640 1 -- 172.21.15.6:0/1808494802 wait complete. 2024-04-26T01:59:40.087 INFO:tasks.workunit.client.0.smithi006.stdout: 10 1034 75.77 MB/sec warmup 3 sec latency 231.481 ms 2024-04-26T01:59:40.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:40 smithi188 ceph-mon[32751]: pgmap v435: 97 pgs: 97 active+clean; 153 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-04-26T01:59:40.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:40 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:59:40.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:40 smithi029 ceph-mon[32775]: pgmap v435: 97 pgs: 97 active+clean; 153 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-04-26T01:59:40.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:40 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:59:41.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:40 smithi006 ceph-mon[28419]: pgmap v435: 97 pgs: 97 active+clean; 153 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-04-26T01:59:41.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:40 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:59:41.087 INFO:tasks.workunit.client.0.smithi006.stdout: 10 1569 64.36 MB/sec warmup 4 sec latency 47.055 ms 2024-04-26T01:59:42.088 INFO:tasks.workunit.client.0.smithi006.stdout: 10 2284 55.02 MB/sec warmup 5 sec latency 79.751 ms 2024-04-26T01:59:42.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:42 smithi188 ceph-mon[32751]: pgmap v436: 97 pgs: 97 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.4 MiB/s wr, 380 op/s 2024-04-26T01:59:42.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:42 smithi029 ceph-mon[32775]: pgmap v436: 97 pgs: 97 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.4 MiB/s wr, 380 op/s 2024-04-26T01:59:43.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:42 smithi006 ceph-mon[28419]: pgmap v436: 97 pgs: 97 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.4 MiB/s wr, 380 op/s 2024-04-26T01:59:43.088 INFO:tasks.workunit.client.0.smithi006.stdout: 10 2806 54.37 MB/sec warmup 6 sec latency 72.500 ms 2024-04-26T01:59:44.088 INFO:tasks.workunit.client.0.smithi006.stdout: 10 3422 53.33 MB/sec warmup 7 sec latency 91.526 ms 2024-04-26T01:59:44.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:44 smithi188 ceph-mon[32751]: pgmap v437: 97 pgs: 97 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 3.6 MiB/s wr, 311 op/s 2024-04-26T01:59:44.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:44 smithi029 ceph-mon[32775]: pgmap v437: 97 pgs: 97 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 3.6 MiB/s wr, 311 op/s 2024-04-26T01:59:45.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:44 smithi006 ceph-mon[28419]: pgmap v437: 97 pgs: 97 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 3.6 MiB/s wr, 311 op/s 2024-04-26T01:59:45.088 INFO:tasks.workunit.client.0.smithi006.stdout: 10 3700 48.34 MB/sec warmup 8 sec latency 62.927 ms 2024-04-26T01:59:46.088 INFO:tasks.workunit.client.0.smithi006.stdout: 10 3892 43.78 MB/sec warmup 9 sec latency 68.199 ms 2024-04-26T01:59:46.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:46 smithi188 ceph-mon[32751]: pgmap v438: 97 pgs: 97 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 3.6 MiB/s wr, 311 op/s 2024-04-26T01:59:46.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:46 smithi029 ceph-mon[32775]: pgmap v438: 97 pgs: 97 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 3.6 MiB/s wr, 311 op/s 2024-04-26T01:59:47.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:46 smithi006 ceph-mon[28419]: pgmap v438: 97 pgs: 97 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 3.6 MiB/s wr, 311 op/s 2024-04-26T01:59:47.088 INFO:tasks.workunit.client.0.smithi006.stdout: 10 4075 39.62 MB/sec warmup 10 sec latency 129.893 ms 2024-04-26T01:59:48.088 INFO:tasks.workunit.client.0.smithi006.stdout: 10 4426 36.99 MB/sec warmup 11 sec latency 89.415 ms 2024-04-26T01:59:48.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:48 smithi188 ceph-mon[32751]: pgmap v439: 97 pgs: 97 active+clean; 165 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.0 MiB/s wr, 389 op/s 2024-04-26T01:59:48.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:48 smithi029 ceph-mon[32775]: pgmap v439: 97 pgs: 97 active+clean; 165 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.0 MiB/s wr, 389 op/s 2024-04-26T01:59:49.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:48 smithi006 ceph-mon[28419]: pgmap v439: 97 pgs: 97 active+clean; 165 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.0 MiB/s wr, 389 op/s 2024-04-26T01:59:49.088 INFO:tasks.workunit.client.0.smithi006.stdout: 10 4887 37.17 MB/sec warmup 12 sec latency 56.449 ms 2024-04-26T01:59:50.088 INFO:tasks.workunit.client.0.smithi006.stdout: 10 5453 34.77 MB/sec warmup 13 sec latency 69.862 ms 2024-04-26T01:59:50.873 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:50 smithi006 ceph-mon[28419]: pgmap v440: 97 pgs: 97 active+clean; 165 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 4.4 MiB/s wr, 330 op/s 2024-04-26T01:59:50.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:50 smithi188 ceph-mon[32751]: pgmap v440: 97 pgs: 97 active+clean; 165 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 4.4 MiB/s wr, 330 op/s 2024-04-26T01:59:50.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:50 smithi029 ceph-mon[32775]: pgmap v440: 97 pgs: 97 active+clean; 165 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 4.4 MiB/s wr, 330 op/s 2024-04-26T01:59:51.088 INFO:tasks.workunit.client.0.smithi006.stdout: 10 5909 33.70 MB/sec warmup 14 sec latency 47.793 ms 2024-04-26T01:59:52.089 INFO:tasks.workunit.client.0.smithi006.stdout: 10 6527 34.82 MB/sec warmup 15 sec latency 73.154 ms 2024-04-26T01:59:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.498+0000 7f32d2c00640 1 -- 172.21.15.6:0/1365863499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32cc104bb0 msgr2=0x7f32cc108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.498+0000 7f32d2c00640 1 --2- 172.21.15.6:0/1365863499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32cc104bb0 0x7f32cc108bf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f32b8009900 tx=0x7f32b802a3f0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.499+0000 7f32d2c00640 1 -- 172.21.15.6:0/1365863499 shutdown_connections 2024-04-26T01:59:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.499+0000 7f32d2c00640 1 --2- 172.21.15.6:0/1365863499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32cc1092d0 0x7f32cc110ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.499+0000 7f32d2c00640 1 --2- 172.21.15.6:0/1365863499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32cc104bb0 0x7f32cc108bf0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.499+0000 7f32d2c00640 1 --2- 172.21.15.6:0/1365863499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32cc104200 0x7f32cc1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.499+0000 7f32d2c00640 1 -- 172.21.15.6:0/1365863499 >> 172.21.15.6:0/1365863499 conn(0x7f32cc0ffa60 msgr2=0x7f32cc101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.499+0000 7f32d2c00640 1 -- 172.21.15.6:0/1365863499 shutdown_connections 2024-04-26T01:59:52.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.499+0000 7f32d2c00640 1 -- 172.21.15.6:0/1365863499 wait complete. 2024-04-26T01:59:52.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.500+0000 7f32d2c00640 1 Processor -- start 2024-04-26T01:59:52.499 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.500+0000 7f32d2c00640 1 -- start start 2024-04-26T01:59:52.500 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.501+0000 7f32d2c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32cc104200 0x7f32cc1a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.501+0000 7f32d2c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32cc1092d0 0x7f32cc1a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.501+0000 7f32d2c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32cc19a8e0 0x7f32cc19ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.501+0000 7f32d2c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc114260 con 0x7f32cc104200 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.501+0000 7f32d2c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc1140e0 con 0x7f32cc1092d0 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.501+0000 7f32d2c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc1143e0 con 0x7f32cc19a8e0 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.501+0000 7f32d0e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32cc1092d0 0x7f32cc1a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.501+0000 7f32d0e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32cc1092d0 0x7f32cc1a0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55794/0 (socket says 172.21.15.6:55794) 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.501+0000 7f32d0e00640 1 -- 172.21.15.6:0/1412734870 learned_addr learned my addr 172.21.15.6:0/1412734870 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.501+0000 7f32d1800640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32cc104200 0x7f32cc1a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.501+0000 7f32d2200640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32cc19a8e0 0x7f32cc19ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.502+0000 7f32d0e00640 1 -- 172.21.15.6:0/1412734870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32cc19a8e0 msgr2=0x7f32cc19ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.502+0000 7f32d0e00640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32cc19a8e0 0x7f32cc19ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.502+0000 7f32d0e00640 1 -- 172.21.15.6:0/1412734870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32cc104200 msgr2=0x7f32cc1a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.502+0000 7f32d0e00640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32cc104200 0x7f32cc1a0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.502+0000 7f32d0e00640 1 -- 172.21.15.6:0/1412734870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32cc19b400 con 0x7f32cc1092d0 2024-04-26T01:59:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.502+0000 7f32d1800640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32cc104200 0x7f32cc1a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:59:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.502+0000 7f32d0e00640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32cc1092d0 0x7f32cc1a0c70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f32b8005ab0 tx=0x7f32b8003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.502+0000 7f32ca000640 1 -- 172.21.15.6:0/1412734870 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b803f070 con 0x7f32cc1092d0 2024-04-26T01:59:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.502+0000 7f32ca000640 1 -- 172.21.15.6:0/1412734870 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32b8031b50 con 0x7f32cc1092d0 2024-04-26T01:59:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.502+0000 7f32ca000640 1 -- 172.21.15.6:0/1412734870 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8035030 con 0x7f32cc1092d0 2024-04-26T01:59:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.502+0000 7f32d2c00640 1 -- 172.21.15.6:0/1412734870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32cc19b690 con 0x7f32cc1092d0 2024-04-26T01:59:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.503+0000 7f32d2c00640 1 -- 172.21.15.6:0/1412734870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32cc1a78e0 con 0x7f32cc1092d0 2024-04-26T01:59:52.504 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.506+0000 7f32ca000640 1 -- 172.21.15.6:0/1412734870 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f32b8042070 con 0x7f32cc1092d0 2024-04-26T01:59:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.506+0000 7f32d7056740 1 Processor -- start 2024-04-26T01:59:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.506+0000 7f32d7056740 1 -- start start 2024-04-26T01:59:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d7056740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6048f0 0x5621af604cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d7056740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 0x5621af617e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d7056740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6183a0 0x5621af61a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d7056740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5621af56a900 con 0x5621af6048f0 2024-04-26T01:59:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d7056740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5621af510ef0 con 0x5621af6183a0 2024-04-26T01:59:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d7056740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5621af5668c0 con 0x5621af605210 2024-04-26T01:59:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d1800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 0x5621af617e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d2200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6048f0 0x5621af604cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d1800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 0x5621af617e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34752/0 (socket says 172.21.15.6:34752) 2024-04-26T01:59:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d1800640 1 -- 172.21.15.6:0/304180562 learned_addr learned my addr 172.21.15.6:0/304180562 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d0e00640 1 --2- 172.21.15.6:0/304180562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6183a0 0x5621af61a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d2200640 1 -- 172.21.15.6:0/304180562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 msgr2=0x5621af617e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d2200640 1 --2- 172.21.15.6:0/304180562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 0x5621af617e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d2200640 1 -- 172.21.15.6:0/304180562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6183a0 msgr2=0x5621af61a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d2200640 1 --2- 172.21.15.6:0/304180562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6183a0 0x5621af61a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d2200640 1 -- 172.21.15.6:0/304180562 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621af27d210 con 0x5621af6048f0 2024-04-26T01:59:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.507+0000 7f32d1800640 1 --2- 172.21.15.6:0/304180562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 0x5621af617e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.508+0000 7f32ca000640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f32b40a3d80 0x7f32b40a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.508+0000 7f32ca000640 1 -- 172.21.15.6:0/1412734870 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f32b80f0f60 con 0x7f32cc1092d0 2024-04-26T01:59:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.508+0000 7f32d1800640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f32b40a3d80 0x7f32b40a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.508+0000 7f32d2200640 1 --2- 172.21.15.6:0/304180562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6048f0 0x5621af604cd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f32c4009eb0 tx=0x7f32c4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a35b88f70fb214b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.508+0000 7f32bea00640 1 -- 172.21.15.6:0/304180562 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c4016070 con 0x5621af6048f0 2024-04-26T01:59:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.508+0000 7f32bea00640 1 -- 172.21.15.6:0/304180562 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32c4004d20 con 0x5621af6048f0 2024-04-26T01:59:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.508+0000 7f32bea00640 1 -- 172.21.15.6:0/304180562 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c4012720 con 0x5621af6048f0 2024-04-26T01:59:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.509+0000 7f32d7056740 1 -- 172.21.15.6:0/304180562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6048f0 msgr2=0x5621af604cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.509+0000 7f32d1800640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f32b40a3d80 0x7f32b40a6240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f32c0009fd0 tx=0x7f32c0005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.509+0000 7f32d7056740 1 --2- 172.21.15.6:0/304180562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6048f0 0x5621af604cd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f32c4009eb0 tx=0x7f32c4010740 comp rx=0 tx=0).stop 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.509+0000 7f32d7056740 1 -- 172.21.15.6:0/304180562 shutdown_connections 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.509+0000 7f32d7056740 1 --2- 172.21.15.6:0/304180562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6183a0 0x5621af61a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.509+0000 7f32d7056740 1 --2- 172.21.15.6:0/304180562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 0x5621af617e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.509+0000 7f32d7056740 1 --2- 172.21.15.6:0/304180562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6048f0 0x5621af604cd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.509+0000 7f32d7056740 1 -- 172.21.15.6:0/304180562 >> 172.21.15.6:0/304180562 conn(0x5621af600ec0 msgr2=0x5621af6012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.509+0000 7f32d7056740 1 -- 172.21.15.6:0/304180562 shutdown_connections 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.510+0000 7f32d7056740 1 -- 172.21.15.6:0/304180562 wait complete. 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.510+0000 7f32d7056740 1 -- start start 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.510+0000 7f32d7056740 1 Processor -- start 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.511+0000 7f32d7056740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6048f0 0x5621af615410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.511+0000 7f32d7056740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 0x5621af615950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.511+0000 7f32d7056740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6183a0 0x5621af615e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.511+0000 7f32d7056740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5621af61af70 con 0x5621af6183a0 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.511+0000 7f32d7056740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5621af524310 con 0x5621af6048f0 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.511+0000 7f32d7056740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5621af61b0f0 con 0x5621af605210 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.511+0000 7f32d7056740 10 client.0 _ll_register_callbacks cb 0x5621af4f2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.511+0000 7f32d0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 0x5621af615950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.511+0000 7f32d0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 0x5621af615950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34764/0 (socket says 172.21.15.6:34764) 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.511+0000 7f32d0e00640 1 -- 172.21.15.6:0/4293728325 learned_addr learned my addr 172.21.15.6:0/4293728325 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.511+0000 7f32d1800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6048f0 0x5621af615410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.511+0000 7f32d2200640 1 --2- 172.21.15.6:0/4293728325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6183a0 0x5621af615e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.512+0000 7f32d0e00640 1 -- 172.21.15.6:0/4293728325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6048f0 msgr2=0x5621af615410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.512+0000 7f32d0e00640 1 --2- 172.21.15.6:0/4293728325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6048f0 0x5621af615410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.512+0000 7f32d0e00640 1 -- 172.21.15.6:0/4293728325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6183a0 msgr2=0x5621af615e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.512+0000 7f32d0e00640 1 --2- 172.21.15.6:0/4293728325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6183a0 0x5621af615e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.512+0000 7f32d0e00640 1 -- 172.21.15.6:0/4293728325 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621af349620 con 0x5621af605210 2024-04-26T01:59:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.512+0000 7f32d0e00640 1 --2- 172.21.15.6:0/4293728325 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 0x5621af615950 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f32b8000c00 tx=0x7f32b8031f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b86861f6b85080bf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.512+0000 7f32b3400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:59:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.512+0000 7f32b3400640 1 -- 172.21.15.6:0/4293728325 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8049070 con 0x5621af605210 2024-04-26T01:59:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.512+0000 7f32b3400640 1 -- 172.21.15.6:0/4293728325 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f32b80b94c0 con 0x5621af605210 2024-04-26T01:59:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.512+0000 7f32d7056740 1 -- 172.21.15.6:0/4293728325 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32c4004d20 con 0x5621af605210 2024-04-26T01:59:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.513+0000 7f32b3400640 1 -- 172.21.15.6:0/4293728325 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f32b80bce50 con 0x5621af605210 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.513+0000 7f32d7056740 10 client.26863 fetch_fsmap learned FSMap version 45 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.513+0000 7f32d7056740 1 -- 172.21.15.6:0/4293728325 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5621af61afd0 con 0x5621af605210 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.513+0000 7f32b3400640 1 -- 172.21.15.6:0/4293728325 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32b80bf030 con 0x5621af605210 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.513+0000 7f32b3400640 1 -- 172.21.15.6:0/4293728325 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 45) ==== 8346+0+0 (secure 0 0 0) 0x7f32b80ee910 con 0x5621af605210 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.513+0000 7f32d7056740 10 client.26863 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.513+0000 7f32d7056740 10 client.26863 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.513+0000 7f32d7056740 20 client.26863 populate_metadata read hostname 'smithi006' 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.513+0000 7f32d7056740 1 --2- 172.21.15.6:0/4293728325 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621af628800 0x5621af62abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.513+0000 7f32d7056740 4 client.26863 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.513+0000 7f32d7056740 1 -- 172.21.15.6:0/4293728325 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621af57e8b0 con 0x5621af628800 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.514+0000 7f32d1800640 1 --2- 172.21.15.6:0/4293728325 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621af628800 0x5621af62abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.514+0000 7f32d1800640 1 --2- 172.21.15.6:0/4293728325 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621af628800 0x5621af62abf0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fdd21595dfedfe5 server_cookie=1cc52c1d36923831 in_seq=0 out_seq=0 2024-04-26T01:59:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.514+0000 7f32b3400640 10 client.26863 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.516+0000 7f32b3400640 1 -- 172.21.15.6:0/4293728325 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621af57e8b0 con 0x5621af628800 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.516+0000 7f32b3400640 10 client.26863 handle_command_reply: tid=0 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32d7056740 1 client.26863 shutdown 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32d7056740 1 -- 172.21.15.6:0/4293728325 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 msgr2=0x5621af615950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32d7056740 1 --2- 172.21.15.6:0/4293728325 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 0x5621af615950 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f32b8000c00 tx=0x7f32b8031f50 comp rx=0 tx=0).stop 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32d7056740 1 -- 172.21.15.6:0/4293728325 shutdown_connections 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32d7056740 1 --2- 172.21.15.6:0/4293728325 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621af628800 0x5621af62abf0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32d7056740 1 --2- 172.21.15.6:0/4293728325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6183a0 0x5621af615e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32d7056740 1 --2- 172.21.15.6:0/4293728325 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af605210 0x5621af615950 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32d7056740 1 --2- 172.21.15.6:0/4293728325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6048f0 0x5621af615410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32d7056740 1 -- 172.21.15.6:0/4293728325 >> 172.21.15.6:0/4293728325 conn(0x5621af600ec0 msgr2=0x5621af606c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32b3400640 0 client.26863 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32d7056740 1 -- 172.21.15.6:0/4293728325 shutdown_connections 2024-04-26T01:59:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32d7056740 1 -- 172.21.15.6:0/4293728325 wait complete. 2024-04-26T01:59:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.517+0000 7f32d7056740 20 client.26863 trim_cache size 0 max 16384 2024-04-26T01:59:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.537+0000 7f32d7056740 1 Processor -- start 2024-04-26T01:59:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.537+0000 7f32d7056740 1 -- start start 2024-04-26T01:59:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.537+0000 7f32d7056740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af6048f0 0x5621af623e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.537+0000 7f32d7056740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af605210 0x5621af615ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.537+0000 7f32d7056740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6183a0 0x5621af630500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.537+0000 7f32d7056740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc1ae280 con 0x5621af605210 2024-04-26T01:59:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.537+0000 7f32d7056740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc1abb20 con 0x5621af6183a0 2024-04-26T01:59:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.537+0000 7f32d7056740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc1a9990 con 0x5621af6048f0 2024-04-26T01:59:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.537+0000 7f32d2200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6183a0 0x5621af630500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.537+0000 7f32d1800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af6048f0 0x5621af623e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.537+0000 7f32d2200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6183a0 0x5621af630500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:55816/0 (socket says 172.21.15.6:55816) 2024-04-26T01:59:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.537+0000 7f32d2200640 1 -- 172.21.15.6:0/2496985280 learned_addr learned my addr 172.21.15.6:0/2496985280 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.538+0000 7f32d0e00640 1 --2- 172.21.15.6:0/2496985280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af605210 0x5621af615ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.538+0000 7f32d1800640 1 -- 172.21.15.6:0/2496985280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6183a0 msgr2=0x5621af630500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.538+0000 7f32d1800640 1 --2- 172.21.15.6:0/2496985280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6183a0 0x5621af630500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.538+0000 7f32d1800640 1 -- 172.21.15.6:0/2496985280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af605210 msgr2=0x5621af615ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.538+0000 7f32d1800640 1 --2- 172.21.15.6:0/2496985280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af605210 0x5621af615ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.538+0000 7f32d1800640 1 -- 172.21.15.6:0/2496985280 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621af605040 con 0x5621af6048f0 2024-04-26T01:59:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.538+0000 7f32d2200640 1 --2- 172.21.15.6:0/2496985280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6183a0 0x5621af630500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.538+0000 7f32d1800640 1 --2- 172.21.15.6:0/2496985280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af6048f0 0x5621af623e70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x5621af624540 tx=0x7f32c001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca5a318a256f4ff1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.539+0000 7f32be000640 1 -- 172.21.15.6:0/2496985280 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c0011e80 con 0x5621af6048f0 2024-04-26T01:59:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.539+0000 7f32be000640 1 -- 172.21.15.6:0/2496985280 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32c002a8d0 con 0x5621af6048f0 2024-04-26T01:59:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.539+0000 7f32be000640 1 -- 172.21.15.6:0/2496985280 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c0004a00 con 0x5621af6048f0 2024-04-26T01:59:52.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.539+0000 7f32d7056740 1 -- 172.21.15.6:0/2496985280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af6048f0 msgr2=0x5621af623e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:52.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.539+0000 7f32d7056740 1 --2- 172.21.15.6:0/2496985280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af6048f0 0x5621af623e70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x5621af624540 tx=0x7f32c001a000 comp rx=0 tx=0).stop 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.540+0000 7f32d7056740 1 -- 172.21.15.6:0/2496985280 shutdown_connections 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.540+0000 7f32d7056740 1 --2- 172.21.15.6:0/2496985280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af605210 0x5621af615ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.540+0000 7f32d7056740 1 --2- 172.21.15.6:0/2496985280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af6183a0 0x5621af630500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.540+0000 7f32d7056740 1 --2- 172.21.15.6:0/2496985280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af6048f0 0x5621af623e70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.540+0000 7f32d7056740 1 -- 172.21.15.6:0/2496985280 >> 172.21.15.6:0/2496985280 conn(0x5621af600ec0 msgr2=0x5621af6153d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.540+0000 7f32d7056740 1 -- 172.21.15.6:0/2496985280 shutdown_connections 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.540+0000 7f32d7056740 1 -- 172.21.15.6:0/2496985280 wait complete. 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.540+0000 7f32d7056740 1 -- start start 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.540+0000 7f32d7056740 1 Processor -- start 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.541+0000 7f32d7056740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6048f0 0x5621af620650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.541+0000 7f32d7056740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af605210 0x5621af61b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.541+0000 7f32d7056740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af6183a0 0x5621af61bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.541+0000 7f32d7056740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc1a9fe0 con 0x5621af6048f0 2024-04-26T01:59:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.541+0000 7f32d7056740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc1ab450 con 0x5621af605210 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.541+0000 7f32d7056740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc1aa6b0 con 0x5621af6183a0 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.541+0000 7f32d7056740 10 client.0 _ll_register_callbacks cb 0x7f32b80f4700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.541+0000 7f32d1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6048f0 0x5621af620650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.541+0000 7f32d1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6048f0 0x5621af620650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50112/0 (socket says 172.21.15.6:50112) 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.541+0000 7f32d1800640 1 -- 172.21.15.6:0/3648567669 learned_addr learned my addr 172.21.15.6:0/3648567669 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.541+0000 7f32d2200640 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af6183a0 0x5621af61bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.541+0000 7f32d0e00640 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af605210 0x5621af61b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.542+0000 7f32d2200640 1 -- 172.21.15.6:0/3648567669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af605210 msgr2=0x5621af61b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.542+0000 7f32d2200640 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af605210 0x5621af61b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.542+0000 7f32d2200640 1 -- 172.21.15.6:0/3648567669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6048f0 msgr2=0x5621af620650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.542+0000 7f32d2200640 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6048f0 0x5621af620650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.542+0000 7f32d2200640 1 -- 172.21.15.6:0/3648567669 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621af61c5f0 con 0x5621af6183a0 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.542+0000 7f32d2200640 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af6183a0 0x5621af61bf00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x5621af601d00 tx=0x7f32c4004bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4332f4c57ffd1d1e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.542+0000 7f32b3400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T01:59:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.542+0000 7f32b3400640 1 -- 172.21.15.6:0/3648567669 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c402b070 con 0x5621af6183a0 2024-04-26T01:59:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.542+0000 7f32b3400640 1 -- 172.21.15.6:0/3648567669 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f32c4027920 con 0x5621af6183a0 2024-04-26T01:59:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.542+0000 7f32d7056740 1 -- 172.21.15.6:0/3648567669 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32c002a8d0 con 0x5621af6183a0 2024-04-26T01:59:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.543+0000 7f32b3400640 1 -- 172.21.15.6:0/3648567669 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f32c4034050 con 0x5621af6183a0 2024-04-26T01:59:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.543+0000 7f32d7056740 10 client.26869 fetch_fsmap learned FSMap version 45 2024-04-26T01:59:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.543+0000 7f32d7056740 1 -- 172.21.15.6:0/3648567669 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5621af68ec80 con 0x5621af6183a0 2024-04-26T01:59:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.543+0000 7f32b3400640 1 -- 172.21.15.6:0/3648567669 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32c002a8d0 con 0x5621af6183a0 2024-04-26T01:59:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.543+0000 7f32b3400640 1 -- 172.21.15.6:0/3648567669 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 45) ==== 8346+0+0 (secure 0 0 0) 0x7f32c402e020 con 0x5621af6183a0 2024-04-26T01:59:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.543+0000 7f32d7056740 10 client.26869 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:59:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.543+0000 7f32d7056740 10 client.26869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T01:59:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.543+0000 7f32d7056740 20 client.26869 populate_metadata read hostname 'smithi006' 2024-04-26T01:59:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.543+0000 7f32d7056740 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621af68f060 0x5621af691450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.543+0000 7f32d7056740 4 client.26869 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:59:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.543+0000 7f32d7056740 1 -- 172.21.15.6:0/3648567669 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5621af578320 con 0x5621af68f060 2024-04-26T01:59:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.543+0000 7f32d1800640 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621af68f060 0x5621af691450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.544+0000 7f32d1800640 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621af68f060 0x5621af691450 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e530ea2c4821210 server_cookie=3bf5b74191ce7fe2 in_seq=0 out_seq=0 2024-04-26T01:59:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.544+0000 7f32b3400640 10 client.26869 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:52.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:52 smithi188 ceph-mon[32751]: pgmap v441: 97 pgs: 97 active+clean; 177 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 413 op/s 2024-04-26T01:59:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.904+0000 7f32b3400640 1 -- 172.21.15.6:0/3648567669 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5621af578320 con 0x5621af68f060 2024-04-26T01:59:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.904+0000 7f32b3400640 10 client.26869 handle_command_reply: tid=0 2024-04-26T01:59:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.904+0000 7f32d7056740 1 client.26869 shutdown 2024-04-26T01:59:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.904+0000 7f32d7056740 1 -- 172.21.15.6:0/3648567669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af6183a0 msgr2=0x5621af61bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.904+0000 7f32d7056740 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af6183a0 0x5621af61bf00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x5621af601d00 tx=0x7f32c4004bd0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.904+0000 7f32d7056740 1 -- 172.21.15.6:0/3648567669 shutdown_connections 2024-04-26T01:59:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.904+0000 7f32d7056740 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5621af68f060 0x5621af691450 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.904+0000 7f32d7056740 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5621af6183a0 0x5621af61bf00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.904+0000 7f32d7056740 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5621af605210 0x5621af61b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.904+0000 7f32d7056740 1 --2- 172.21.15.6:0/3648567669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621af6048f0 0x5621af620650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.904+0000 7f32b3400640 0 client.26869 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T01:59:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.904+0000 7f32d7056740 1 -- 172.21.15.6:0/3648567669 >> 172.21.15.6:0/3648567669 conn(0x5621af600ec0 msgr2=0x5621af606c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.905+0000 7f32d7056740 1 -- 172.21.15.6:0/3648567669 shutdown_connections 2024-04-26T01:59:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.905+0000 7f32d7056740 1 -- 172.21.15.6:0/3648567669 wait complete. 2024-04-26T01:59:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.905+0000 7f32d7056740 20 client.26869 trim_cache size 0 max 16384 2024-04-26T01:59:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T01:59:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T01:59:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T01:59:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T01:59:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.906+0000 7f32d2c00640 1 -- 172.21.15.6:0/1412734870 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f32b40a3d80 msgr2=0x7f32b40a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.906+0000 7f32d2c00640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f32b40a3d80 0x7f32b40a6240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f32c0009fd0 tx=0x7f32c0005f90 comp rx=0 tx=0).stop 2024-04-26T01:59:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.906+0000 7f32d2c00640 1 -- 172.21.15.6:0/1412734870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32cc1092d0 msgr2=0x7f32cc1a0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.906+0000 7f32d2c00640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32cc1092d0 0x7f32cc1a0c70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f32b8005ab0 tx=0x7f32b8003b90 comp rx=0 tx=0).stop 2024-04-26T01:59:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.907+0000 7f32d2c00640 1 -- 172.21.15.6:0/1412734870 shutdown_connections 2024-04-26T01:59:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.907+0000 7f32d2c00640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f32b40a3d80 0x7f32b40a6240 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.907+0000 7f32d2c00640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32cc19a8e0 0x7f32cc19ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.907+0000 7f32d2c00640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32cc1092d0 0x7f32cc1a0c70 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.907+0000 7f32d2c00640 1 --2- 172.21.15.6:0/1412734870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32cc104200 0x7f32cc1a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.907+0000 7f32d2c00640 1 -- 172.21.15.6:0/1412734870 >> 172.21.15.6:0/1412734870 conn(0x7f32cc0ffa60 msgr2=0x7f32cc100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.907+0000 7f32d2c00640 1 -- 172.21.15.6:0/1412734870 shutdown_connections 2024-04-26T01:59:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T01:59:52.907+0000 7f32d2c00640 1 -- 172.21.15.6:0/1412734870 wait complete. 2024-04-26T01:59:52.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:52 smithi029 ceph-mon[32775]: pgmap v441: 97 pgs: 97 active+clean; 177 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 413 op/s 2024-04-26T01:59:53.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:52 smithi006 ceph-mon[28419]: pgmap v441: 97 pgs: 97 active+clean; 177 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 413 op/s 2024-04-26T01:59:53.089 INFO:tasks.workunit.client.0.smithi006.stdout: 10 6908 35.06 MB/sec warmup 16 sec latency 175.752 ms 2024-04-26T01:59:53.310 INFO:tasks.quiescer.fs.[cephfs]:Total duration (49.2) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T01:59:53.310 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.6 2024-04-26T01:59:53.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:59:53.797 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:59:54.089 INFO:tasks.workunit.client.0.smithi006.stdout: 10 7156 33.83 MB/sec warmup 17 sec latency 66.172 ms 2024-04-26T01:59:54.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:54 smithi006 ceph-mon[28419]: pgmap v442: 97 pgs: 97 active+clean; 177 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 160 op/s 2024-04-26T01:59:54.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:54 smithi188 ceph-mon[32751]: pgmap v442: 97 pgs: 97 active+clean; 177 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 160 op/s 2024-04-26T01:59:54.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:54 smithi029 ceph-mon[32775]: pgmap v442: 97 pgs: 97 active+clean; 177 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 160 op/s 2024-04-26T01:59:55.089 INFO:tasks.workunit.client.0.smithi006.stdout: 10 7352 32.12 MB/sec warmup 18 sec latency 142.895 ms 2024-04-26T01:59:55.629 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:55 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:59:55.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:55 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:59:55.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:55 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:59:56.090 INFO:tasks.workunit.client.0.smithi006.stdout: 10 7779 31.05 MB/sec warmup 19 sec latency 116.927 ms 2024-04-26T01:59:56.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.316+0000 7f3fed800700 1 -- 172.21.15.6:0/220752987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fe8108a30 msgr2=0x7f3fe8108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:56.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.316+0000 7f3fed800700 1 --2- 172.21.15.6:0/220752987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fe8108a30 0x7f3fe8108e10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80098f0 tx=0x7f3fd8009c00 comp rx=0 tx=0).stop 2024-04-26T01:59:56.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.316+0000 7f3fed800700 1 -- 172.21.15.6:0/220752987 shutdown_connections 2024-04-26T01:59:56.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.316+0000 7f3fed800700 1 --2- 172.21.15.6:0/220752987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fe806b0f0 0x7f3fe810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:56.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.316+0000 7f3fed800700 1 --2- 172.21.15.6:0/220752987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fe8108a30 0x7f3fe8108e10 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:56.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.316+0000 7f3fed800700 1 --2- 172.21.15.6:0/220752987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fe806a750 0x7f3fe806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:56.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.316+0000 7f3fed800700 1 -- 172.21.15.6:0/220752987 >> 172.21.15.6:0/220752987 conn(0x7f3fe80788f0 msgr2=0x7f3fe8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:56.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.317+0000 7f3fed800700 1 -- 172.21.15.6:0/220752987 shutdown_connections 2024-04-26T01:59:56.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.317+0000 7f3fed800700 1 -- 172.21.15.6:0/220752987 wait complete. 2024-04-26T01:59:56.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.318+0000 7f3fed800700 1 Processor -- start 2024-04-26T01:59:56.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.318+0000 7f3fed800700 1 -- start start 2024-04-26T01:59:56.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.318+0000 7f3fed800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fe806a750 0x7f3fe81991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:56.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.318+0000 7f3fed800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fe806b0f0 0x7f3fe8199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:56.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.318+0000 7f3fed800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fe8108a30 0x7f3fe819dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:56.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.318+0000 7f3fed800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe8112c30 con 0x7f3fe806b0f0 2024-04-26T01:59:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.318+0000 7f3fed800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe8112ab0 con 0x7f3fe806a750 2024-04-26T01:59:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.318+0000 7f3fed800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe8112db0 con 0x7f3fe8108a30 2024-04-26T01:59:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.319+0000 7f3fe7400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fe8108a30 0x7f3fe819dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.319+0000 7f3fe7400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fe8108a30 0x7f3fe819dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34798/0 (socket says 172.21.15.6:34798) 2024-04-26T01:59:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.319+0000 7f3fe7400700 1 -- 172.21.15.6:0/144317521 learned_addr learned my addr 172.21.15.6:0/144317521 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T01:59:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.319+0000 7f3fe6a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fe806a750 0x7f3fe81991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.319+0000 7f3fe7400700 1 -- 172.21.15.6:0/144317521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fe806a750 msgr2=0x7f3fe81991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.319+0000 7f3fe7400700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fe806a750 0x7f3fe81991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.319+0000 7f3fe7400700 1 -- 172.21.15.6:0/144317521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fe806b0f0 msgr2=0x7f3fe8199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.319+0000 7f3fe6000700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fe806b0f0 0x7f3fe8199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.319+0000 7f3fe7400700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fe806b0f0 0x7f3fe8199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.319+0000 7f3fe7400700 1 -- 172.21.15.6:0/144317521 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe819e1f0 con 0x7f3fe8108a30 2024-04-26T01:59:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.319+0000 7f3fe7400700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fe8108a30 0x7f3fe819dab0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000ef30 tx=0x7f3fe000c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.320+0000 7f3fdf400700 1 -- 172.21.15.6:0/144317521 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe00092e0 con 0x7f3fe8108a30 2024-04-26T01:59:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.320+0000 7f3fdf400700 1 -- 172.21.15.6:0/144317521 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fe0004bb0 con 0x7f3fe8108a30 2024-04-26T01:59:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.320+0000 7f3fdf400700 1 -- 172.21.15.6:0/144317521 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0004280 con 0x7f3fe8108a30 2024-04-26T01:59:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.320+0000 7f3fed800700 1 -- 172.21.15.6:0/144317521 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe819e480 con 0x7f3fe8108a30 2024-04-26T01:59:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.320+0000 7f3fed800700 1 -- 172.21.15.6:0/144317521 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe8101940 con 0x7f3fe8108a30 2024-04-26T01:59:56.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.322+0000 7f3fed800700 1 -- 172.21.15.6:0/144317521 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fc8005330 con 0x7f3fe8108a30 2024-04-26T01:59:56.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.323+0000 7f3fdf400700 1 -- 172.21.15.6:0/144317521 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f3fe001b030 con 0x7f3fe8108a30 2024-04-26T01:59:56.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.325+0000 7f3fdf400700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3fcc0a3e30 0x7f3fcc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:56.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.325+0000 7f3fdf400700 1 -- 172.21.15.6:0/144317521 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f3fe00d03d0 con 0x7f3fe8108a30 2024-04-26T01:59:56.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.325+0000 7f3fe6a00700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3fcc0a3e30 0x7f3fcc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:56.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.326+0000 7f3fe6a00700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3fcc0a3e30 0x7f3fcc0a62f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8009680 tx=0x7f3fd8019040 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:56.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.329+0000 7f3fdf400700 1 -- 172.21.15.6:0/144317521 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3fe0022020 con 0x7f3fe8108a30 2024-04-26T01:59:56.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.724+0000 7f3fed800700 1 -- 172.21.15.6:0/144317521 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fc80058f0 con 0x7f3fe8108a30 2024-04-26T01:59:56.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.726+0000 7f3fdf400700 1 -- 172.21.15.6:0/144317521 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+11853 (secure 0 0 0) 0x7f3fe0010040 con 0x7f3fe8108a30 2024-04-26T01:59:56.725 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T01:59:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":46,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T01:59:54.454408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1414952991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1414952991},{"type":"v1","addr":"172.21.15.188:6835","nonce":1414952991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24512,24505]},"id":1}]} 2024-04-26T01:59:56.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.730+0000 7f3fed800700 1 -- 172.21.15.6:0/144317521 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3fcc0a3e30 msgr2=0x7f3fcc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:56.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.730+0000 7f3fed800700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3fcc0a3e30 0x7f3fcc0a62f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8009680 tx=0x7f3fd8019040 comp rx=0 tx=0).stop 2024-04-26T01:59:56.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.730+0000 7f3fed800700 1 -- 172.21.15.6:0/144317521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fe8108a30 msgr2=0x7f3fe819dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:56.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.730+0000 7f3fed800700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fe8108a30 0x7f3fe819dab0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000ef30 tx=0x7f3fe000c450 comp rx=0 tx=0).stop 2024-04-26T01:59:56.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.731+0000 7f3fed800700 1 -- 172.21.15.6:0/144317521 shutdown_connections 2024-04-26T01:59:56.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.731+0000 7f3fed800700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3fcc0a3e30 0x7f3fcc0a62f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:56.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.731+0000 7f3fed800700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fe8108a30 0x7f3fe819dab0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:56.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.731+0000 7f3fed800700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fe806a750 0x7f3fe81991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:56.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.731+0000 7f3fed800700 1 --2- 172.21.15.6:0/144317521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fe806b0f0 0x7f3fe8199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:56.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.731+0000 7f3fed800700 1 -- 172.21.15.6:0/144317521 >> 172.21.15.6:0/144317521 conn(0x7f3fe80788f0 msgr2=0x7f3fe80ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:56.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.732+0000 7f3fed800700 1 -- 172.21.15.6:0/144317521 shutdown_connections 2024-04-26T01:59:56.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T01:59:56.732+0000 7f3fed800700 1 -- 172.21.15.6:0/144317521 wait complete. 2024-04-26T01:59:56.736 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 46 2024-04-26T01:59:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:56 smithi188 ceph-mon[32751]: pgmap v443: 97 pgs: 97 active+clean; 177 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 160 op/s 2024-04-26T01:59:56.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:56 smithi029 ceph-mon[32775]: pgmap v443: 97 pgs: 97 active+clean; 177 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 160 op/s 2024-04-26T01:59:57.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:56 smithi006 ceph-mon[28419]: pgmap v443: 97 pgs: 97 active+clean; 177 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 160 op/s 2024-04-26T01:59:57.089 INFO:tasks.workunit.client.0.smithi006.stdout: 10 8569 31.29 MB/sec warmup 20 sec latency 56.449 ms 2024-04-26T01:59:57.668 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 84.6 --await-for 42.0' 2024-04-26T01:59:57.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 84.6 --await-for 42.0 2024-04-26T01:59:57.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:57 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/144317521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:59:58.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:57 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/144317521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:59:58.089 INFO:tasks.workunit.client.0.smithi006.stdout: 10 9574 32.36 MB/sec warmup 21 sec latency 51.377 ms 2024-04-26T01:59:58.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:57 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/144317521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:59:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:59:58.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 01:59:58 smithi029 ceph-mon[32775]: pgmap v444: 97 pgs: 97 active+clean; 171 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 483 op/s 2024-04-26T01:59:58.960 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 01:59:58 smithi006 ceph-mon[28419]: pgmap v444: 97 pgs: 97 active+clean; 171 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 483 op/s 2024-04-26T01:59:59.041 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 01:59:58 smithi188 ceph-mon[32751]: pgmap v444: 97 pgs: 97 active+clean; 171 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 483 op/s 2024-04-26T01:59:59.090 INFO:tasks.workunit.client.0.smithi006.stdout: 10 10167 32.91 MB/sec warmup 22 sec latency 41.803 ms 2024-04-26T02:00:00.090 INFO:tasks.workunit.client.0.smithi006.stdout: 10 10890 33.44 MB/sec warmup 23 sec latency 174.543 ms 2024-04-26T02:00:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.712+0000 7f14ba800700 1 -- 172.21.15.6:0/2333669170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14b4102af0 msgr2=0x7f14b4102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.712+0000 7f14ba800700 1 --2- 172.21.15.6:0/2333669170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14b4102af0 0x7f14b4102f70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f14a0009990 tx=0x7f14a0009ca0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.712+0000 7f14ba800700 1 -- 172.21.15.6:0/2333669170 shutdown_connections 2024-04-26T02:00:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.712+0000 7f14ba800700 1 --2- 172.21.15.6:0/2333669170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14b41034b0 0x7f14b410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.712+0000 7f14ba800700 1 --2- 172.21.15.6:0/2333669170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14b4107940 0x7f14b4107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.712+0000 7f14ba800700 1 --2- 172.21.15.6:0/2333669170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14b4102af0 0x7f14b4102f70 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.712+0000 7f14ba800700 1 -- 172.21.15.6:0/2333669170 >> 172.21.15.6:0/2333669170 conn(0x7f14b4079ba0 msgr2=0x7f14b4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.713+0000 7f14ba800700 1 -- 172.21.15.6:0/2333669170 shutdown_connections 2024-04-26T02:00:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.713+0000 7f14ba800700 1 -- 172.21.15.6:0/2333669170 wait complete. 2024-04-26T02:00:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.714+0000 7f14ba800700 1 Processor -- start 2024-04-26T02:00:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.715+0000 7f14ba800700 1 -- start start 2024-04-26T02:00:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.715+0000 7f14ba800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14b4102af0 0x7f14b41990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.715+0000 7f14ba800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14b41034b0 0x7f14b41995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.715+0000 7f14ba800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14b4107940 0x7f14b419d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.715+0000 7f14ba800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f14b4112cb0 con 0x7f14b4107940 2024-04-26T02:00:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.715+0000 7f14ba800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f14b4112b30 con 0x7f14b41034b0 2024-04-26T02:00:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.715+0000 7f14ba800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f14b4112e30 con 0x7f14b4102af0 2024-04-26T02:00:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.715+0000 7f14b3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14b4102af0 0x7f14b41990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.715+0000 7f14b2a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14b41034b0 0x7f14b41995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.715+0000 7f14b2a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14b41034b0 0x7f14b41995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36562/0 (socket says 172.21.15.6:36562) 2024-04-26T02:00:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.715+0000 7f14b2a00700 1 -- 172.21.15.6:0/1729215080 learned_addr learned my addr 172.21.15.6:0/1729215080 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.715+0000 7f14b3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14b4102af0 0x7f14b41990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36742/0 (socket says 172.21.15.6:36742) 2024-04-26T02:00:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.716+0000 7f14b2a00700 1 -- 172.21.15.6:0/1729215080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14b4102af0 msgr2=0x7f14b41990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.716+0000 7f14b2a00700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14b4102af0 0x7f14b41990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.716+0000 7f14b2a00700 1 -- 172.21.15.6:0/1729215080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14b4107940 msgr2=0x7f14b419d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.716+0000 7f14b2a00700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14b4107940 0x7f14b419d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.716+0000 7f14b2a00700 1 -- 172.21.15.6:0/1729215080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b419e0b0 con 0x7f14b41034b0 2024-04-26T02:00:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.716+0000 7f14b3400700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14b4102af0 0x7f14b41990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.717+0000 7f14b2a00700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14b41034b0 0x7f14b41995e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f14a0005910 tx=0x7f14a000bfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.718+0000 7f14abe00700 1 -- 172.21.15.6:0/1729215080 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a0027070 con 0x7f14b41034b0 2024-04-26T02:00:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.718+0000 7f14abe00700 1 -- 172.21.15.6:0/1729215080 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14a000b610 con 0x7f14b41034b0 2024-04-26T02:00:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.718+0000 7f14ba800700 1 -- 172.21.15.6:0/1729215080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b419e340 con 0x7f14b41034b0 2024-04-26T02:00:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.718+0000 7f14abe00700 1 -- 172.21.15.6:0/1729215080 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a00173f0 con 0x7f14b41034b0 2024-04-26T02:00:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.718+0000 7f14ba800700 1 -- 172.21.15.6:0/1729215080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b41a5c20 con 0x7f14b41034b0 2024-04-26T02:00:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.722+0000 7f14abe00700 1 -- 172.21.15.6:0/1729215080 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f14a002a070 con 0x7f14b41034b0 2024-04-26T02:00:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.723+0000 7f14abe00700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f14980a3e50 0x7f14980a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.724+0000 7f14abe00700 1 -- 172.21.15.6:0/1729215080 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f14a00d1750 con 0x7f14b41034b0 2024-04-26T02:00:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.724+0000 7f14b3400700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f14980a3e50 0x7f14980a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.724+0000 7f14c5188b80 1 Processor -- start 2024-04-26T02:00:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.724+0000 7f14c5188b80 1 -- start start 2024-04-26T02:00:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.725+0000 7f14c5188b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb36920 0x5561bdb34dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.725+0000 7f14c5188b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb35790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.725+0000 7f14c5188b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb48c00 0x5561bdb4aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.725+0000 7f14c5188b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5561bda88020 con 0x5561bdb48c00 2024-04-26T02:00:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.725+0000 7f14c5188b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5561bda2e8c0 con 0x5561bdb36920 2024-04-26T02:00:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.725+0000 7f14c5188b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5561bda83fe0 con 0x5561bdb35310 2024-04-26T02:00:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.725+0000 7f14b2a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb35790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.725+0000 7f14b3400700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f14980a3e50 0x7f14980a6310 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009fd0 tx=0x7f14a4009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.725+0000 7f14b2a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb35790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36752/0 (socket says 172.21.15.6:36752) 2024-04-26T02:00:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.725+0000 7f14b2a00700 1 -- 172.21.15.6:0/1086364935 learned_addr learned my addr 172.21.15.6:0/1086364935 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.725+0000 7f14b3400700 1 --2- 172.21.15.6:0/1086364935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb36920 0x5561bdb34dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.725+0000 7f14b3e00700 1 --2- 172.21.15.6:0/1086364935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb48c00 0x5561bdb4aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.726+0000 7f14b2a00700 1 -- 172.21.15.6:0/1086364935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb36920 msgr2=0x5561bdb34dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.726+0000 7f14b2a00700 1 --2- 172.21.15.6:0/1086364935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb36920 0x5561bdb34dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.726+0000 7f14b2a00700 1 -- 172.21.15.6:0/1086364935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb48c00 msgr2=0x5561bdb4aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.726+0000 7f14b2a00700 1 --2- 172.21.15.6:0/1086364935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb48c00 0x5561bdb4aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.726+0000 7f14b2a00700 1 -- 172.21.15.6:0/1086364935 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561bda24810 con 0x5561bdb35310 2024-04-26T02:00:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.726+0000 7f14b2a00700 1 --2- 172.21.15.6:0/1086364935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb35790 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f14a00a5d80 tx=0x7f14a0009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f3eeee0bc709eee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.726+0000 7f14a8c00700 1 -- 172.21.15.6:0/1086364935 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a009d6d0 con 0x5561bdb35310 2024-04-26T02:00:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.727+0000 7f14a8c00700 1 -- 172.21.15.6:0/1086364935 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14a00a1a80 con 0x5561bdb35310 2024-04-26T02:00:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.727+0000 7f14a8c00700 1 -- 172.21.15.6:0/1086364935 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a009d930 con 0x5561bdb35310 2024-04-26T02:00:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.727+0000 7f14c5188b80 1 -- 172.21.15.6:0/1086364935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 msgr2=0x5561bdb35790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.727+0000 7f14c5188b80 1 --2- 172.21.15.6:0/1086364935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb35790 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f14a00a5d80 tx=0x7f14a0009310 comp rx=0 tx=0).stop 2024-04-26T02:00:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.728+0000 7f14c5188b80 1 -- 172.21.15.6:0/1086364935 shutdown_connections 2024-04-26T02:00:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.728+0000 7f14c5188b80 1 --2- 172.21.15.6:0/1086364935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb48c00 0x5561bdb4aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.728+0000 7f14c5188b80 1 --2- 172.21.15.6:0/1086364935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb36920 0x5561bdb34dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.728+0000 7f14c5188b80 1 --2- 172.21.15.6:0/1086364935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb35790 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.728+0000 7f14c5188b80 1 -- 172.21.15.6:0/1086364935 >> 172.21.15.6:0/1086364935 conn(0x5561bda640e0 msgr2=0x5561bdb312e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.728+0000 7f14c5188b80 1 -- 172.21.15.6:0/1086364935 shutdown_connections 2024-04-26T02:00:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.728+0000 7f14c5188b80 1 -- 172.21.15.6:0/1086364935 wait complete. 2024-04-26T02:00:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.729+0000 7f14c5188b80 1 -- start start 2024-04-26T02:00:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.729+0000 7f14c5188b80 1 Processor -- start 2024-04-26T02:00:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.729+0000 7f14c5188b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb3b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14c5188b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb36920 0x5561bdb38bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14c5188b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 0x5561bdb39100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14c5188b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5561bdb4b590 con 0x5561bdb36920 2024-04-26T02:00:00.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14c5188b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5561bda419f0 con 0x5561bdb48c00 2024-04-26T02:00:00.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14c5188b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5561bdb4b710 con 0x5561bdb35310 2024-04-26T02:00:00.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14c5188b80 10 client.0 _ll_register_callbacks cb 0x5561bda208b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:00.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14b3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb3b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14b3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb3b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36764/0 (socket says 172.21.15.6:36764) 2024-04-26T02:00:00.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14b3400700 1 -- 172.21.15.6:0/329825187 learned_addr learned my addr 172.21.15.6:0/329825187 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:00.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14b2a00700 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb36920 0x5561bdb38bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14b3400700 1 -- 172.21.15.6:0/329825187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 msgr2=0x5561bdb39100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:00.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14b3400700 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 0x5561bdb39100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.730+0000 7f14b3400700 1 -- 172.21.15.6:0/329825187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb36920 msgr2=0x5561bdb38bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:00.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.731+0000 7f14b3400700 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb36920 0x5561bdb38bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.731+0000 7f14b3400700 1 -- 172.21.15.6:0/329825187 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561bd9d5a60 con 0x5561bdb35310 2024-04-26T02:00:00.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.731+0000 7f14b2a00700 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb36920 0x5561bdb38bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:00:00.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.732+0000 7f14b3400700 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb3b8c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f14a4003a00 tx=0x7f14a40047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2e243c9be1b5f32 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:00.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.732+0000 7f149d600700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:00:00.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.732+0000 7f149d600700 1 -- 172.21.15.6:0/329825187 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a401fa10 con 0x5561bdb35310 2024-04-26T02:00:00.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.732+0000 7f14c5188b80 1 -- 172.21.15.6:0/329825187 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14a00a1a80 con 0x5561bdb35310 2024-04-26T02:00:00.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.732+0000 7f149d600700 1 -- 172.21.15.6:0/329825187 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f14a401fbb0 con 0x5561bdb35310 2024-04-26T02:00:00.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.732+0000 7f149d600700 1 -- 172.21.15.6:0/329825187 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f14a401fdb0 con 0x5561bdb35310 2024-04-26T02:00:00.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.732+0000 7f14c5188b80 10 client.26887 fetch_fsmap learned FSMap version 47 2024-04-26T02:00:00.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.732+0000 7f14c5188b80 1 -- 172.21.15.6:0/329825187 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5561bda1ec80 con 0x5561bdb35310 2024-04-26T02:00:00.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.733+0000 7f149d600700 1 -- 172.21.15.6:0/329825187 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14a00a1a80 con 0x5561bdb35310 2024-04-26T02:00:00.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.733+0000 7f149d600700 1 -- 172.21.15.6:0/329825187 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 47) ==== 8354+0+0 (secure 0 0 0) 0x7f14a4026020 con 0x5561bdb35310 2024-04-26T02:00:00.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.733+0000 7f14c5188b80 10 client.26887 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T02:00:00.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.733+0000 7f14c5188b80 10 client.26887 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:00:00.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.733+0000 7f14c5188b80 20 client.26887 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:00.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.733+0000 7f14c5188b80 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5561bdb58300 0x5561bdb5a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.733+0000 7f14c5188b80 4 client.26887 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:00.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.733+0000 7f14c5188b80 1 -- 172.21.15.6:0/329825187 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561bda95810 con 0x5561bdb58300 2024-04-26T02:00:00.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.734+0000 7f14b2a00700 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5561bdb58300 0x5561bdb5a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.734+0000 7f14b2a00700 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5561bdb58300 0x5561bdb5a770 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b4e70d7886b4274 server_cookie=389caf800b19f85e in_seq=0 out_seq=0 2024-04-26T02:00:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.734+0000 7f149d600700 10 client.26887 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f149d600700 1 -- 172.21.15.6:0/329825187 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561bda95810 con 0x5561bdb58300 2024-04-26T02:00:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f149d600700 10 client.26887 handle_command_reply: tid=0 2024-04-26T02:00:00.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f14c5188b80 1 client.26887 shutdown 2024-04-26T02:00:00.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f14c5188b80 1 -- 172.21.15.6:0/329825187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 msgr2=0x5561bdb3b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:00.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f14c5188b80 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb3b8c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f14a4003a00 tx=0x7f14a40047c0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f14c5188b80 1 -- 172.21.15.6:0/329825187 shutdown_connections 2024-04-26T02:00:00.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f14c5188b80 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 0x5561bdb39100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f14c5188b80 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5561bdb58300 0x5561bdb5a770 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f14c5188b80 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb3b8c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f14c5188b80 1 --2- 172.21.15.6:0/329825187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb36920 0x5561bdb38bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f14c5188b80 1 -- 172.21.15.6:0/329825187 >> 172.21.15.6:0/329825187 conn(0x5561bda640e0 msgr2=0x5561bdb30ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f149d600700 0 client.26887 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.736+0000 7f14c5188b80 1 -- 172.21.15.6:0/329825187 shutdown_connections 2024-04-26T02:00:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.737+0000 7f14c5188b80 1 -- 172.21.15.6:0/329825187 wait complete. 2024-04-26T02:00:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.737+0000 7f14c5188b80 20 client.26887 trim_cache size 0 max 16384 2024-04-26T02:00:00.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.766+0000 7f14c5188b80 1 Processor -- start 2024-04-26T02:00:00.766 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:00 smithi006 ceph-mon[28419]: pgmap v445: 97 pgs: 97 active+clean; 171 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 405 op/s 2024-04-26T02:00:00.766 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:00 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:00:00.766 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:00 smithi006 ceph-mon[28419]: overall HEALTH_OK 2024-04-26T02:00:00.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.766+0000 7f14c5188b80 1 -- start start 2024-04-26T02:00:00.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.767+0000 7f14c5188b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb54450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.767+0000 7f14c5188b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb36920 0x5561bdb60040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.767+0000 7f14c5188b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 0x5561bdb60580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.767+0000 7f14c5188b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f14b41ada60 con 0x5561bdb36920 2024-04-26T02:00:00.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.767+0000 7f14c5188b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f14b41ab260 con 0x5561bdb48c00 2024-04-26T02:00:00.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.767+0000 7f14c5188b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f14b41a90d0 con 0x5561bdb35310 2024-04-26T02:00:00.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.767+0000 7f14b3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb54450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.767+0000 7f14b3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb54450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36772/0 (socket says 172.21.15.6:36772) 2024-04-26T02:00:00.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.767+0000 7f14b3400700 1 -- 172.21.15.6:0/338314909 learned_addr learned my addr 172.21.15.6:0/338314909 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:00.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.767+0000 7f14b3e00700 1 --2- 172.21.15.6:0/338314909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 0x5561bdb60580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.767+0000 7f14b2a00700 1 --2- 172.21.15.6:0/338314909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb36920 0x5561bdb60040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.768+0000 7f14b3400700 1 -- 172.21.15.6:0/338314909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 msgr2=0x5561bdb60580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:00.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.768+0000 7f14b3400700 1 --2- 172.21.15.6:0/338314909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 0x5561bdb60580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.768+0000 7f14b3400700 1 -- 172.21.15.6:0/338314909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb36920 msgr2=0x5561bdb60040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:00.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.768+0000 7f14b3400700 1 --2- 172.21.15.6:0/338314909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb36920 0x5561bdb60040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.768+0000 7f14b3400700 1 -- 172.21.15.6:0/338314909 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561bdb47d60 con 0x5561bdb35310 2024-04-26T02:00:00.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.768+0000 7f14b3400700 1 --2- 172.21.15.6:0/338314909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb54450 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f14a40094f0 tx=0x7f14a4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f235ed2e9bf92586 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:00.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.768+0000 7f149fe00700 1 -- 172.21.15.6:0/338314909 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a403d070 con 0x5561bdb35310 2024-04-26T02:00:00.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.768+0000 7f149fe00700 1 -- 172.21.15.6:0/338314909 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14a4030740 con 0x5561bdb35310 2024-04-26T02:00:00.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.768+0000 7f149fe00700 1 -- 172.21.15.6:0/338314909 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a40287a0 con 0x5561bdb35310 2024-04-26T02:00:00.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.769+0000 7f14c5188b80 1 -- 172.21.15.6:0/338314909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 msgr2=0x5561bdb54450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:00.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.769+0000 7f14c5188b80 1 --2- 172.21.15.6:0/338314909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb54450 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f14a40094f0 tx=0x7f14a4004730 comp rx=0 tx=0).stop 2024-04-26T02:00:00.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.769+0000 7f14c5188b80 1 -- 172.21.15.6:0/338314909 shutdown_connections 2024-04-26T02:00:00.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.769+0000 7f14c5188b80 1 --2- 172.21.15.6:0/338314909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 0x5561bdb60580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.769+0000 7f14c5188b80 1 --2- 172.21.15.6:0/338314909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb35310 0x5561bdb54450 secure :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f14a40094f0 tx=0x7f14a4004730 comp rx=0 tx=0).stop 2024-04-26T02:00:00.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.770+0000 7f14c5188b80 1 --2- 172.21.15.6:0/338314909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb36920 0x5561bdb60040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.770+0000 7f14c5188b80 1 -- 172.21.15.6:0/338314909 >> 172.21.15.6:0/338314909 conn(0x5561bda640e0 msgr2=0x5561bdb30490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:00.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.770+0000 7f14c5188b80 1 -- 172.21.15.6:0/338314909 shutdown_connections 2024-04-26T02:00:00.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.771+0000 7f14c5188b80 1 -- 172.21.15.6:0/338314909 wait complete. 2024-04-26T02:00:00.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.771+0000 7f14c5188b80 1 -- start start 2024-04-26T02:00:00.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.771+0000 7f14c5188b80 1 Processor -- start 2024-04-26T02:00:00.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.772+0000 7f14c5188b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb36920 0x5561bdb50da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.772+0000 7f14c5188b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 0x5561bdb4d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.772+0000 7f14c5188b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb4d720 0x5561bdb4db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.772+0000 7f14c5188b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f14b41a9720 con 0x5561bdb4d720 2024-04-26T02:00:00.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.772+0000 7f14c5188b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f14b41aab90 con 0x5561bdb48c00 2024-04-26T02:00:00.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.772+0000 7f14c5188b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f14b41a9df0 con 0x5561bdb36920 2024-04-26T02:00:00.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.772+0000 7f14c5188b80 10 client.0 _ll_register_callbacks cb 0x5561bda208b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:00.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.773+0000 7f14b3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb36920 0x5561bdb50da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.773+0000 7f14b3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb36920 0x5561bdb50da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36774/0 (socket says 172.21.15.6:36774) 2024-04-26T02:00:00.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.773+0000 7f14b3400700 1 -- 172.21.15.6:0/2710715199 learned_addr learned my addr 172.21.15.6:0/2710715199 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:00.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.773+0000 7f14b3e00700 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb4d720 0x5561bdb4db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.773+0000 7f14b3400700 1 -- 172.21.15.6:0/2710715199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 msgr2=0x5561bdb4d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:00.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.773+0000 7f14b2a00700 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 0x5561bdb4d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.774+0000 7f14b3400700 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 0x5561bdb4d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.774+0000 7f14b3400700 1 -- 172.21.15.6:0/2710715199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb4d720 msgr2=0x5561bdb4db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:00.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.774+0000 7f14b3400700 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb4d720 0x5561bdb4db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:00.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.774+0000 7f14b3400700 1 -- 172.21.15.6:0/2710715199 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561bdbbea40 con 0x5561bdb36920 2024-04-26T02:00:00.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.774+0000 7f14b2a00700 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 0x5561bdb4d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:00:00.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.774+0000 7f14b3400700 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb36920 0x5561bdb50da0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f14a4030640 tx=0x7f14a4030910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a605d086f769aaa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:00.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.774+0000 7f149d600700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:00:00.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.774+0000 7f149d600700 1 -- 172.21.15.6:0/2710715199 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a403d070 con 0x5561bdb36920 2024-04-26T02:00:00.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.775+0000 7f14c5188b80 1 -- 172.21.15.6:0/2710715199 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14a4030740 con 0x5561bdb36920 2024-04-26T02:00:00.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.775+0000 7f149d600700 1 -- 172.21.15.6:0/2710715199 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8600+0+0 (secure 0 0 0) 0x7f14a4005040 con 0x5561bdb36920 2024-04-26T02:00:00.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.775+0000 7f149d600700 1 -- 172.21.15.6:0/2710715199 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f14a4047050 con 0x5561bdb36920 2024-04-26T02:00:00.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.775+0000 7f14c5188b80 10 client.26893 fetch_fsmap learned FSMap version 47 2024-04-26T02:00:00.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.775+0000 7f14c5188b80 1 -- 172.21.15.6:0/2710715199 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5561bdbbfd00 con 0x5561bdb36920 2024-04-26T02:00:00.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.775+0000 7f149d600700 1 -- 172.21.15.6:0/2710715199 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14a4030740 con 0x5561bdb36920 2024-04-26T02:00:00.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.776+0000 7f149d600700 1 -- 172.21.15.6:0/2710715199 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 47) ==== 8354+0+0 (secure 0 0 0) 0x7f14a4026050 con 0x5561bdb36920 2024-04-26T02:00:00.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.776+0000 7f14c5188b80 10 client.26893 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T02:00:00.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.776+0000 7f14c5188b80 10 client.26893 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:00:00.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.776+0000 7f14c5188b80 20 client.26893 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:00.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.776+0000 7f14c5188b80 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5561bdbc03f0 0x5561bdbc2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:00.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.776+0000 7f14c5188b80 4 client.26893 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.6, "await_for": 42.0}] 2024-04-26T02:00:00.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.776+0000 7f14c5188b80 1 -- 172.21.15.6:0/2710715199 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.6, "await_for": 42.0}) -- 0x5561bda8f8b0 con 0x5561bdbc03f0 2024-04-26T02:00:00.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.777+0000 7f14b2a00700 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5561bdbc03f0 0x5561bdbc2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:00.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.778+0000 7f149d600700 10 client.26893 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:00.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:00.778+0000 7f14b2a00700 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5561bdbc03f0 0x5561bdbc2810 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d8d1e9561731055 server_cookie=42b9306d0d51118c in_seq=0 out_seq=0 2024-04-26T02:00:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:00 smithi029 ceph-mon[32775]: pgmap v445: 97 pgs: 97 active+clean; 171 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 405 op/s 2024-04-26T02:00:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:00 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:00:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:00 smithi029 ceph-mon[32775]: overall HEALTH_OK 2024-04-26T02:00:01.090 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11174 32.60 MB/sec warmup 24 sec latency 593.076 ms 2024-04-26T02:00:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:00 smithi188 ceph-mon[32751]: pgmap v445: 97 pgs: 97 active+clean; 171 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 405 op/s 2024-04-26T02:00:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:00 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:00:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:00 smithi188 ceph-mon[32751]: overall HEALTH_OK 2024-04-26T02:00:01.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.630+0000 7f149d600700 1 -- 172.21.15.6:0/2710715199 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5561bda8f8b0 con 0x5561bdbc03f0 2024-04-26T02:00:01.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.630+0000 7f149d600700 10 client.26893 handle_command_reply: tid=0 2024-04-26T02:00:01.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.630+0000 7f14c5188b80 1 client.26893 shutdown 2024-04-26T02:00:01.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.631+0000 7f14c5188b80 1 -- 172.21.15.6:0/2710715199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb36920 msgr2=0x5561bdb50da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:01.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.631+0000 7f14c5188b80 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb36920 0x5561bdb50da0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f14a4030640 tx=0x7f14a4030910 comp rx=0 tx=0).stop 2024-04-26T02:00:01.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.631+0000 7f14c5188b80 1 -- 172.21.15.6:0/2710715199 shutdown_connections 2024-04-26T02:00:01.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.631+0000 7f14c5188b80 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5561bdb4d720 0x5561bdb4db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.631+0000 7f14c5188b80 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5561bdbc03f0 0x5561bdbc2810 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.631+0000 7f14c5188b80 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5561bdb36920 0x5561bdb50da0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.631+0000 7f14c5188b80 1 --2- 172.21.15.6:0/2710715199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5561bdb48c00 0x5561bdb4d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.631+0000 7f149d600700 0 client.26893 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:01.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.631+0000 7f14c5188b80 1 -- 172.21.15.6:0/2710715199 >> 172.21.15.6:0/2710715199 conn(0x5561bda640e0 msgr2=0x5561bdb37690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:01.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.632+0000 7f14c5188b80 1 -- 172.21.15.6:0/2710715199 shutdown_connections 2024-04-26T02:00:01.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.632+0000 7f14c5188b80 1 -- 172.21.15.6:0/2710715199 wait complete. 2024-04-26T02:00:01.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.632+0000 7f14c5188b80 20 client.26893 trim_cache size 0 max 16384 2024-04-26T02:00:01.632 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:00:01.632 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 47, 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 42, 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "c6386e05": { 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 42, 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 84.6, 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T02:00:01.633 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T02:00:01.634 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:00:01.634 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T02:00:01.634 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:00:01.634 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:01.634 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:01.634 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:01.634 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:01.634 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:01.634 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:00:01.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.634+0000 7f14ba800700 1 -- 172.21.15.6:0/1729215080 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f14980a3e50 msgr2=0x7f14980a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:01.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.634+0000 7f14ba800700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f14980a3e50 0x7f14980a6310 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009fd0 tx=0x7f14a4009260 comp rx=0 tx=0).stop 2024-04-26T02:00:01.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.634+0000 7f14ba800700 1 -- 172.21.15.6:0/1729215080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14b41034b0 msgr2=0x7f14b41995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:01.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.634+0000 7f14ba800700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14b41034b0 0x7f14b41995e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f14a0005910 tx=0x7f14a000bfc0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.636+0000 7f14ba800700 1 -- 172.21.15.6:0/1729215080 shutdown_connections 2024-04-26T02:00:01.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.636+0000 7f14ba800700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f14980a3e50 0x7f14980a6310 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.636+0000 7f14ba800700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14b4107940 0x7f14b419d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.636+0000 7f14ba800700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14b4102af0 0x7f14b41990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.636+0000 7f14ba800700 1 --2- 172.21.15.6:0/1729215080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14b41034b0 0x7f14b41995e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.636+0000 7f14ba800700 1 -- 172.21.15.6:0/1729215080 >> 172.21.15.6:0/1729215080 conn(0x7f14b4079ba0 msgr2=0x7f14b4100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:01.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.636+0000 7f14ba800700 1 -- 172.21.15.6:0/1729215080 shutdown_connections 2024-04-26T02:00:01.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:01.636+0000 7f14ba800700 1 -- 172.21.15.6:0/1729215080 wait complete. 2024-04-26T02:00:02.090 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 31.30 MB/sec warmup 25 sec latency 1579.232 ms 2024-04-26T02:00:02.502 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c6386e05', quiesce took 9.2 seconds. Will release after: 24.6 2024-04-26T02:00:02.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:02 smithi006 ceph-mon[28419]: pgmap v446: 97 pgs: 97 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 9.0 MiB/s wr, 620 op/s 2024-04-26T02:00:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:02 smithi029 ceph-mon[32775]: pgmap v446: 97 pgs: 97 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 9.0 MiB/s wr, 620 op/s 2024-04-26T02:00:03.090 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 30.10 MB/sec warmup 26 sec latency 2579.372 ms 2024-04-26T02:00:03.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:02 smithi188 ceph-mon[32751]: pgmap v446: 97 pgs: 97 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 9.0 MiB/s wr, 620 op/s 2024-04-26T02:00:04.090 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 28.98 MB/sec warmup 27 sec latency 3579.509 ms 2024-04-26T02:00:04.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:04 smithi029 ceph-mon[32775]: pgmap v447: 97 pgs: 97 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.8 MiB/s wr, 537 op/s 2024-04-26T02:00:05.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:04 smithi006 ceph-mon[28419]: pgmap v447: 97 pgs: 97 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.8 MiB/s wr, 537 op/s 2024-04-26T02:00:05.090 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 27.95 MB/sec warmup 28 sec latency 4579.636 ms 2024-04-26T02:00:05.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:04 smithi188 ceph-mon[32751]: pgmap v447: 97 pgs: 97 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.8 MiB/s wr, 537 op/s 2024-04-26T02:00:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.056+0000 7f42ea000640 1 -- 172.21.15.6:0/1430568011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e4074c10 msgr2=0x7f42e4112820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.056+0000 7f42ea000640 1 --2- 172.21.15.6:0/1430568011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e4074c10 0x7f42e4112820 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009860 tx=0x7f42d002a3d0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.055 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.056+0000 7f42ea000640 1 -- 172.21.15.6:0/1430568011 shutdown_connections 2024-04-26T02:00:06.055 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.056+0000 7f42ea000640 1 --2- 172.21.15.6:0/1430568011 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e4112d60 0x7f42e4115150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.055 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.056+0000 7f42ea000640 1 --2- 172.21.15.6:0/1430568011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e4074c10 0x7f42e4112820 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.055 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.056+0000 7f42ea000640 1 --2- 172.21.15.6:0/1430568011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e40742f0 0x7f42e40746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.055 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.056+0000 7f42ea000640 1 -- 172.21.15.6:0/1430568011 >> 172.21.15.6:0/1430568011 conn(0x7f42e40fd880 msgr2=0x7f42e40ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:06.055 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.056+0000 7f42ea000640 1 -- 172.21.15.6:0/1430568011 shutdown_connections 2024-04-26T02:00:06.055 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.056+0000 7f42ea000640 1 -- 172.21.15.6:0/1430568011 wait complete. 2024-04-26T02:00:06.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.057+0000 7f42ea000640 1 Processor -- start 2024-04-26T02:00:06.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.058+0000 7f42ea000640 1 -- start start 2024-04-26T02:00:06.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.058+0000 7f42ea000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e40742f0 0x7f42e41a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.058+0000 7f42ea000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e4074c10 0x7f42e41a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.058+0000 7f42ea000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e4112d60 0x7f42e419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.058+0000 7f42ea000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f42e4118630 con 0x7f42e4074c10 2024-04-26T02:00:06.056 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.058+0000 7f42ea000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f42e41184b0 con 0x7f42e4112d60 2024-04-26T02:00:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.058+0000 7f42ea000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f42e41187b0 con 0x7f42e40742f0 2024-04-26T02:00:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.058+0000 7f42e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e4074c10 0x7f42e41a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.058+0000 7f42e9600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e4112d60 0x7f42e419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.058+0000 7f42e9600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e4112d60 0x7f42e419ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36618/0 (socket says 172.21.15.6:36618) 2024-04-26T02:00:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.058+0000 7f42e9600640 1 -- 172.21.15.6:0/2770795282 learned_addr learned my addr 172.21.15.6:0/2770795282 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.058+0000 7f42e8c00640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e40742f0 0x7f42e41a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.059+0000 7f42e9600640 1 -- 172.21.15.6:0/2770795282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e40742f0 msgr2=0x7f42e41a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.059+0000 7f42e9600640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e40742f0 0x7f42e41a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.059+0000 7f42e9600640 1 -- 172.21.15.6:0/2770795282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e4074c10 msgr2=0x7f42e41a4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.059+0000 7f42e9600640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e4074c10 0x7f42e41a4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.059+0000 7f42e9600640 1 -- 172.21.15.6:0/2770795282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e419f190 con 0x7f42e4112d60 2024-04-26T02:00:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.059+0000 7f42e8c00640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e40742f0 0x7f42e41a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.059+0000 7f42e9600640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e4112d60 0x7f42e419ea10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00eb40 tx=0x7f42dc00ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.059+0000 7f42e1600640 1 -- 172.21.15.6:0/2770795282 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc00ce40 con 0x7f42e4112d60 2024-04-26T02:00:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.059+0000 7f42ea000640 1 -- 172.21.15.6:0/2770795282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42e419f480 con 0x7f42e4112d60 2024-04-26T02:00:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.059+0000 7f42e1600640 1 -- 172.21.15.6:0/2770795282 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42dc005020 con 0x7f42e4112d60 2024-04-26T02:00:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.059+0000 7f42e1600640 1 -- 172.21.15.6:0/2770795282 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc010650 con 0x7f42e4112d60 2024-04-26T02:00:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.060+0000 7f42ea000640 1 -- 172.21.15.6:0/2770795282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42e41ab720 con 0x7f42e4112d60 2024-04-26T02:00:06.060 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.063+0000 7f42ee480740 1 Processor -- start 2024-04-26T02:00:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.063+0000 7f42e1600640 1 -- 172.21.15.6:0/2770795282 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f42dc0054b0 con 0x7f42e4112d60 2024-04-26T02:00:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.063+0000 7f42ee480740 1 -- start start 2024-04-26T02:00:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.063+0000 7f42ee480740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a589b0 0x555924a58d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.063+0000 7f42ee480740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a592d0 0x555924a63650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42ee480740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a63b90 0x555924a65f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42ee480740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5559249be900 con 0x555924a592d0 2024-04-26T02:00:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42ee480740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555924964ef0 con 0x555924a63b90 2024-04-26T02:00:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42ee480740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5559249ba8c0 con 0x555924a589b0 2024-04-26T02:00:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42e8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a592d0 0x555924a63650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42e3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a589b0 0x555924a58d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42e3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a589b0 0x555924a58d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:36810/0 (socket says 172.21.15.6:36810) 2024-04-26T02:00:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42e3e00640 1 -- 172.21.15.6:0/3806921921 learned_addr learned my addr 172.21.15.6:0/3806921921 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:06.062 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42e3e00640 1 --2- 172.21.15.6:0/3806921921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a63b90 0x555924a65f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42e8c00640 1 -- 172.21.15.6:0/3806921921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a589b0 msgr2=0x555924a58d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:06.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42e8c00640 1 --2- 172.21.15.6:0/3806921921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a589b0 0x555924a58d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42e8c00640 1 -- 172.21.15.6:0/3806921921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a63b90 msgr2=0x555924a65f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:06.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42e8c00640 1 --2- 172.21.15.6:0/3806921921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a63b90 0x555924a65f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42e8c00640 1 -- 172.21.15.6:0/3806921921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559246d1210 con 0x555924a592d0 2024-04-26T02:00:06.064 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.064+0000 7f42e3e00640 1 --2- 172.21.15.6:0/3806921921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a63b90 0x555924a65f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.065+0000 7f42e8c00640 1 --2- 172.21.15.6:0/3806921921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a592d0 0x555924a63650 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f42d8005fd0 tx=0x7f42d80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d77e7eca05f83b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.065+0000 7f42e1600640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f42cc0a3d80 0x7f42cc0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.065+0000 7f42e1600640 1 -- 172.21.15.6:0/2770795282 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8644+0+0 (secure 0 0 0) 0x7f42dc0d0f40 con 0x7f42e4112d60 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.065+0000 7f42d5c00640 1 -- 172.21.15.6:0/3806921921 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d801e070 con 0x555924a592d0 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.065+0000 7f42e3e00640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f42cc0a3d80 0x7f42cc0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.065+0000 7f42d5c00640 1 -- 172.21.15.6:0/3806921921 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42d80035f0 con 0x555924a592d0 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.065+0000 7f42d5c00640 1 -- 172.21.15.6:0/3806921921 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d800f7c0 con 0x555924a592d0 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.066+0000 7f42ee480740 1 -- 172.21.15.6:0/3806921921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a592d0 msgr2=0x555924a63650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.066+0000 7f42e3e00640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f42cc0a3d80 0x7f42cc0a6240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x555924a562e0 tx=0x7f42d0005c90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.066+0000 7f42ee480740 1 --2- 172.21.15.6:0/3806921921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a592d0 0x555924a63650 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f42d8005fd0 tx=0x7f42d80118d0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.066+0000 7f42ee480740 1 -- 172.21.15.6:0/3806921921 shutdown_connections 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.066+0000 7f42ee480740 1 --2- 172.21.15.6:0/3806921921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a592d0 0x555924a63650 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.066+0000 7f42ee480740 1 --2- 172.21.15.6:0/3806921921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a63b90 0x555924a65f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.065 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.066+0000 7f42ee480740 1 --2- 172.21.15.6:0/3806921921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a589b0 0x555924a58d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.066+0000 7f42ee480740 1 -- 172.21.15.6:0/3806921921 >> 172.21.15.6:0/3806921921 conn(0x555924a54ec0 msgr2=0x555924a552a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:06.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.066+0000 7f42ee480740 1 -- 172.21.15.6:0/3806921921 shutdown_connections 2024-04-26T02:00:06.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.066+0000 7f42ee480740 1 -- 172.21.15.6:0/3806921921 wait complete. 2024-04-26T02:00:06.066 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.067+0000 7f42ee480740 1 -- start start 2024-04-26T02:00:06.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.067+0000 7f42ee480740 1 Processor -- start 2024-04-26T02:00:06.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.067+0000 7f42ee480740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a589b0 0x555924a60af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.067 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.067+0000 7f42ee480740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a592d0 0x555924a61030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.067+0000 7f42ee480740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a63b90 0x555924a61570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.067+0000 7f42ee480740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555924a667c0 con 0x555924a592d0 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.067+0000 7f42ee480740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555924978310 con 0x555924a589b0 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.067+0000 7f42ee480740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555924a66940 con 0x555924a63b90 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.067+0000 7f42ee480740 10 client.0 _ll_register_callbacks cb 0x555924946cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.068+0000 7f42e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a592d0 0x555924a61030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.068+0000 7f42e8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a589b0 0x555924a60af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.068+0000 7f42e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a592d0 0x555924a61030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50810/0 (socket says 172.21.15.6:50810) 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.068+0000 7f42e3e00640 1 -- 172.21.15.6:0/2908680697 learned_addr learned my addr 172.21.15.6:0/2908680697 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.068+0000 7f42e9600640 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a63b90 0x555924a61570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.068+0000 7f42e8c00640 1 -- 172.21.15.6:0/2908680697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a63b90 msgr2=0x555924a61570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.068+0000 7f42e8c00640 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a63b90 0x555924a61570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.068+0000 7f42e8c00640 1 -- 172.21.15.6:0/2908680697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a592d0 msgr2=0x555924a61030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:06.068 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.068+0000 7f42e8c00640 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a592d0 0x555924a61030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.068+0000 7f42e8c00640 1 -- 172.21.15.6:0/2908680697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55592479d620 con 0x555924a589b0 2024-04-26T02:00:06.069 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.068+0000 7f42e9600640 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a63b90 0x555924a61570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:06.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:05 smithi006 ceph-mon[28419]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T02:00:06.071 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.069+0000 7f42e8c00640 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a589b0 0x555924a60af0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f42d801c040 tx=0x7f42d80042b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3c9ac746b8ee0e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:06.071 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.069+0000 7f42ca000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:00:06.071 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.069+0000 7f42ca000640 1 -- 172.21.15.6:0/2908680697 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d801e040 con 0x555924a589b0 2024-04-26T02:00:06.071 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.069+0000 7f42ca000640 1 -- 172.21.15.6:0/2908680697 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8644+0+0 (secure 0 0 0) 0x7f42d8016050 con 0x555924a589b0 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.069+0000 7f42ee480740 1 -- 172.21.15.6:0/2908680697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42d80035f0 con 0x555924a589b0 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.069+0000 7f42ca000640 1 -- 172.21.15.6:0/2908680697 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f42d8031050 con 0x555924a589b0 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.069+0000 7f42ee480740 10 client.26855 fetch_fsmap learned FSMap version 47 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.069+0000 7f42ee480740 1 -- 172.21.15.6:0/2908680697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x555924a72a20 con 0x555924a589b0 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.070+0000 7f42ca000640 1 -- 172.21.15.6:0/2908680697 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42d80035f0 con 0x555924a589b0 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.070+0000 7f42ca000640 1 -- 172.21.15.6:0/2908680697 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 47) ==== 8354+0+0 (secure 0 0 0) 0x7f42d8021030 con 0x555924a589b0 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.070+0000 7f42ee480740 10 client.26855 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.070+0000 7f42ee480740 10 client.26855 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.070+0000 7f42ee480740 20 client.26855 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.070+0000 7f42ee480740 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555924a73e20 0x555924a76210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.070+0000 7f42ee480740 4 client.26855 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.070+0000 7f42ee480740 1 -- 172.21.15.6:0/2908680697 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559249d28b0 con 0x555924a73e20 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.071+0000 7f42e3e00640 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555924a73e20 0x555924a76210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.071+0000 7f42e3e00640 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555924a73e20 0x555924a76210 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed7629099b2b7ef5 server_cookie=a8753da1306fed86 in_seq=0 out_seq=0 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.071+0000 7f42ca000640 10 client.26855 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.072+0000 7f42ca000640 1 -- 172.21.15.6:0/2908680697 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5559249d28b0 con 0x555924a73e20 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.072+0000 7f42ca000640 10 client.26855 handle_command_reply: tid=0 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.072+0000 7f42ee480740 1 client.26855 shutdown 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.073+0000 7f42ee480740 1 -- 172.21.15.6:0/2908680697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a589b0 msgr2=0x555924a60af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.073+0000 7f42ee480740 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a589b0 0x555924a60af0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f42d801c040 tx=0x7f42d80042b0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.073+0000 7f42ee480740 1 -- 172.21.15.6:0/2908680697 shutdown_connections 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.073+0000 7f42ee480740 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555924a73e20 0x555924a76210 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.073+0000 7f42ee480740 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a592d0 0x555924a61030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.073+0000 7f42ee480740 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a63b90 0x555924a61570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.073+0000 7f42ca000640 0 client.26855 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.073+0000 7f42ee480740 1 --2- 172.21.15.6:0/2908680697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a589b0 0x555924a60af0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.073+0000 7f42ee480740 1 -- 172.21.15.6:0/2908680697 >> 172.21.15.6:0/2908680697 conn(0x555924a54ec0 msgr2=0x555924a5abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.073+0000 7f42ee480740 1 -- 172.21.15.6:0/2908680697 shutdown_connections 2024-04-26T02:00:06.073 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.073+0000 7f42ee480740 1 -- 172.21.15.6:0/2908680697 wait complete. 2024-04-26T02:00:06.074 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.073+0000 7f42ee480740 20 client.26855 trim_cache size 0 max 16384 2024-04-26T02:00:06.087 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.089+0000 7f42ee480740 1 Processor -- start 2024-04-26T02:00:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.089+0000 7f42ee480740 1 -- start start 2024-04-26T02:00:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.090+0000 7f42ee480740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a589b0 0x555924a6f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.090+0000 7f42ee480740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a592d0 0x555924a53ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.090+0000 7f42ee480740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a63b90 0x555924a7ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.090+0000 7f42ee480740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f42e41b6b70 con 0x555924a589b0 2024-04-26T02:00:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.090+0000 7f42ee480740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f42e41b43b0 con 0x555924a63b90 2024-04-26T02:00:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.090+0000 7f42ee480740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f42e41b2220 con 0x555924a592d0 2024-04-26T02:00:06.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.090+0000 7f42e3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a592d0 0x555924a53ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.089 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.090+0000 7f42e8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a589b0 0x555924a6f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.090+0000 7f42e9600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a63b90 0x555924a7ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.090+0000 7f42e8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a589b0 0x555924a6f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50826/0 (socket says 172.21.15.6:50826) 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.090+0000 7f42e8c00640 1 -- 172.21.15.6:0/2045140708 learned_addr learned my addr 172.21.15.6:0/2045140708 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.091+0000 7f42e8c00640 1 -- 172.21.15.6:0/2045140708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a592d0 msgr2=0x555924a53ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.091+0000 7f42e8c00640 1 --2- 172.21.15.6:0/2045140708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a592d0 0x555924a53ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.091+0000 7f42e8c00640 1 -- 172.21.15.6:0/2045140708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a63b90 msgr2=0x555924a7ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.091+0000 7f42e8c00640 1 --2- 172.21.15.6:0/2045140708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a63b90 0x555924a7ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.091+0000 7f42e8c00640 1 -- 172.21.15.6:0/2045140708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555924a5b960 con 0x555924a589b0 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.091+0000 7f42e9600640 1 --2- 172.21.15.6:0/2045140708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a63b90 0x555924a7ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.091+0000 7f42e8c00640 1 --2- 172.21.15.6:0/2045140708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a589b0 0x555924a6f680 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f42d8000c00 tx=0x7f42d8001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629f5d438ecc2d6b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.091+0000 7f42d5200640 1 -- 172.21.15.6:0/2045140708 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d801e030 con 0x555924a589b0 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.091+0000 7f42d5200640 1 -- 172.21.15.6:0/2045140708 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42d8016030 con 0x555924a589b0 2024-04-26T02:00:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.091+0000 7f42d5200640 1 -- 172.21.15.6:0/2045140708 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d8005080 con 0x555924a589b0 2024-04-26T02:00:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.092+0000 7f42ee480740 1 -- 172.21.15.6:0/2045140708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a589b0 msgr2=0x555924a6f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.092+0000 7f42ee480740 1 --2- 172.21.15.6:0/2045140708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a589b0 0x555924a6f680 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f42d8000c00 tx=0x7f42d8001e50 comp rx=0 tx=0).stop 2024-04-26T02:00:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.092+0000 7f42ee480740 1 -- 172.21.15.6:0/2045140708 shutdown_connections 2024-04-26T02:00:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.092+0000 7f42ee480740 1 --2- 172.21.15.6:0/2045140708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a63b90 0x555924a7ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.092+0000 7f42ee480740 1 --2- 172.21.15.6:0/2045140708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a592d0 0x555924a53ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.092+0000 7f42ee480740 1 --2- 172.21.15.6:0/2045140708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a589b0 0x555924a6f680 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.092+0000 7f42ee480740 1 -- 172.21.15.6:0/2045140708 >> 172.21.15.6:0/2045140708 conn(0x555924a54ec0 msgr2=0x555924a53790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.092+0000 7f42ee480740 1 -- 172.21.15.6:0/2045140708 shutdown_connections 2024-04-26T02:00:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.092+0000 7f42ee480740 1 -- 172.21.15.6:0/2045140708 wait complete. 2024-04-26T02:00:06.094 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 26.98 MB/sec warmup 29 sec latency 5579.722 ms 2024-04-26T02:00:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.093+0000 7f42ee480740 1 -- start start 2024-04-26T02:00:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.093+0000 7f42ee480740 1 Processor -- start 2024-04-26T02:00:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.093+0000 7f42ee480740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a589b0 0x555924a68b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.093+0000 7f42ee480740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a592d0 0x555924a690a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.093+0000 7f42ee480740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a63b90 0x555924ad8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.093+0000 7f42ee480740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f42e41b2870 con 0x555924a63b90 2024-04-26T02:00:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.093+0000 7f42ee480740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f42e41b3ce0 con 0x555924a589b0 2024-04-26T02:00:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.093+0000 7f42ee480740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f42e41b2f40 con 0x555924a592d0 2024-04-26T02:00:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.093+0000 7f42ee480740 10 client.0 _ll_register_callbacks cb 0x7f42d8009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42e9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a63b90 0x555924ad8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42e9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a63b90 0x555924ad8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50840/0 (socket says 172.21.15.6:50840) 2024-04-26T02:00:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42e9600640 1 -- 172.21.15.6:0/2247266795 learned_addr learned my addr 172.21.15.6:0/2247266795 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42e8c00640 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a589b0 0x555924a68b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42e3e00640 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a592d0 0x555924a690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42e9600640 1 -- 172.21.15.6:0/2247266795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a592d0 msgr2=0x555924a690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42e9600640 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a592d0 0x555924a690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42e9600640 1 -- 172.21.15.6:0/2247266795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a589b0 msgr2=0x555924a68b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42e9600640 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a589b0 0x555924a68b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42e9600640 1 -- 172.21.15.6:0/2247266795 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555924ad87c0 con 0x555924a63b90 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42e9600640 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a63b90 0x555924ad8280 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f42dc017000 tx=0x7f42dc0a1270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0cdf843c197f654 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42ca000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42ca000640 1 -- 172.21.15.6:0/2247266795 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc0b5070 con 0x555924a63b90 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.094+0000 7f42ca000640 1 -- 172.21.15.6:0/2247266795 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8644+0+0 (secure 0 0 0) 0x7f42dc0a57f0 con 0x555924a63b90 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.095+0000 7f42ee480740 1 -- 172.21.15.6:0/2247266795 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42d8016030 con 0x555924a63b90 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.095+0000 7f42ca000640 1 -- 172.21.15.6:0/2247266795 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f42dc0b6270 con 0x555924a63b90 2024-04-26T02:00:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.095+0000 7f42ee480740 10 client.16704 fetch_fsmap learned FSMap version 47 2024-04-26T02:00:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.095+0000 7f42ee480740 1 -- 172.21.15.6:0/2247266795 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x555924ad99d0 con 0x555924a63b90 2024-04-26T02:00:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.095+0000 7f42ca000640 1 -- 172.21.15.6:0/2247266795 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42d8016030 con 0x555924a63b90 2024-04-26T02:00:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.095+0000 7f42ca000640 1 -- 172.21.15.6:0/2247266795 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 47) ==== 8354+0+0 (secure 0 0 0) 0x7f42dc081070 con 0x555924a63b90 2024-04-26T02:00:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.095+0000 7f42ee480740 10 client.16704 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T02:00:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.095+0000 7f42ee480740 10 client.16704 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:00:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.095+0000 7f42ee480740 20 client.16704 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.095+0000 7f42ee480740 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555924ada150 0x555924adc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.095+0000 7f42ee480740 4 client.16704 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:00:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.095+0000 7f42ee480740 1 -- 172.21.15.6:0/2247266795 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5559249cc320 con 0x555924ada150 2024-04-26T02:00:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.096+0000 7f42e8c00640 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555924ada150 0x555924adc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.096+0000 7f42e8c00640 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555924ada150 0x555924adc570 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef2a9f19f1c012fa server_cookie=4108c630c9f564d3 in_seq=0 out_seq=0 2024-04-26T02:00:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.096+0000 7f42ca000640 10 client.16704 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:06.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:05 smithi188 ceph-mon[32751]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T02:00:06.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:05 smithi029 ceph-mon[32775]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T02:00:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.238+0000 7f42ca000640 1 -- 172.21.15.6:0/2247266795 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5559249cc320 con 0x555924ada150 2024-04-26T02:00:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.238+0000 7f42ca000640 10 client.16704 handle_command_reply: tid=0 2024-04-26T02:00:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.238+0000 7f42ee480740 1 client.16704 shutdown 2024-04-26T02:00:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.238+0000 7f42ee480740 1 -- 172.21.15.6:0/2247266795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a63b90 msgr2=0x555924ad8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.238+0000 7f42ee480740 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a63b90 0x555924ad8280 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f42dc017000 tx=0x7f42dc0a1270 comp rx=0 tx=0).stop 2024-04-26T02:00:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.239+0000 7f42ee480740 1 -- 172.21.15.6:0/2247266795 shutdown_connections 2024-04-26T02:00:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.239+0000 7f42ee480740 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555924ada150 0x555924adc570 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.239+0000 7f42ee480740 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555924a63b90 0x555924ad8280 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.239+0000 7f42ee480740 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555924a592d0 0x555924a690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.239+0000 7f42ee480740 1 --2- 172.21.15.6:0/2247266795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555924a589b0 0x555924a68b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.239+0000 7f42ee480740 1 -- 172.21.15.6:0/2247266795 >> 172.21.15.6:0/2247266795 conn(0x555924a54ec0 msgr2=0x555924a61320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.239+0000 7f42ca000640 0 client.16704 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.239+0000 7f42ee480740 1 -- 172.21.15.6:0/2247266795 shutdown_connections 2024-04-26T02:00:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.239+0000 7f42ee480740 1 -- 172.21.15.6:0/2247266795 wait complete. 2024-04-26T02:00:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.239+0000 7f42ee480740 20 client.16704 trim_cache size 0 max 16384 2024-04-26T02:00:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:00:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:00:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:00:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:00:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.240+0000 7f42ea000640 1 -- 172.21.15.6:0/2770795282 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f42cc0a3d80 msgr2=0x7f42cc0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.240+0000 7f42ea000640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f42cc0a3d80 0x7f42cc0a6240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x555924a562e0 tx=0x7f42d0005c90 comp rx=0 tx=0).stop 2024-04-26T02:00:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.240+0000 7f42ea000640 1 -- 172.21.15.6:0/2770795282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e4112d60 msgr2=0x7f42e419ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.240+0000 7f42ea000640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e4112d60 0x7f42e419ea10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00eb40 tx=0x7f42dc00ef40 comp rx=0 tx=0).stop 2024-04-26T02:00:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.240+0000 7f42ea000640 1 -- 172.21.15.6:0/2770795282 shutdown_connections 2024-04-26T02:00:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.240+0000 7f42ea000640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f42cc0a3d80 0x7f42cc0a6240 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.240+0000 7f42ea000640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42e4074c10 0x7f42e41a4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.240+0000 7f42ea000640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e4112d60 0x7f42e419ea10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.240+0000 7f42ea000640 1 --2- 172.21.15.6:0/2770795282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e40742f0 0x7f42e41a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.240+0000 7f42ea000640 1 -- 172.21.15.6:0/2770795282 >> 172.21.15.6:0/2770795282 conn(0x7f42e40fd880 msgr2=0x7f42e40ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.240+0000 7f42ea000640 1 -- 172.21.15.6:0/2770795282 shutdown_connections 2024-04-26T02:00:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:06.240+0000 7f42ea000640 1 -- 172.21.15.6:0/2770795282 wait complete. 2024-04-26T02:00:07.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:06 smithi006 ceph-mon[28419]: pgmap v449: 97 pgs: 97 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 9.4 MiB/s wr, 645 op/s 2024-04-26T02:00:07.090 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 26.08 MB/sec warmup 30 sec latency 6579.869 ms 2024-04-26T02:00:07.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:06 smithi188 ceph-mon[32751]: pgmap v449: 97 pgs: 97 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 9.4 MiB/s wr, 645 op/s 2024-04-26T02:00:07.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:06 smithi029 ceph-mon[32775]: pgmap v449: 97 pgs: 97 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 9.4 MiB/s wr, 645 op/s 2024-04-26T02:00:08.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:07 smithi006 ceph-mon[28419]: pgmap v450: 97 pgs: 97 active+clean; 249 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 9.7 MiB/s wr, 339 op/s 2024-04-26T02:00:08.091 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 25.24 MB/sec warmup 31 sec latency 7580.032 ms 2024-04-26T02:00:08.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:07 smithi188 ceph-mon[32751]: pgmap v450: 97 pgs: 97 active+clean; 249 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 9.7 MiB/s wr, 339 op/s 2024-04-26T02:00:08.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:07 smithi029 ceph-mon[32775]: pgmap v450: 97 pgs: 97 active+clean; 249 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 9.7 MiB/s wr, 339 op/s 2024-04-26T02:00:09.091 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 24.45 MB/sec warmup 32 sec latency 8580.119 ms 2024-04-26T02:00:10.091 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 23.71 MB/sec warmup 33 sec latency 9580.276 ms 2024-04-26T02:00:10.640 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:10 smithi188 ceph-mon[32751]: pgmap v451: 97 pgs: 97 active+clean; 249 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 9.7 MiB/s wr, 339 op/s 2024-04-26T02:00:10.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:10 smithi029 ceph-mon[32775]: pgmap v451: 97 pgs: 97 active+clean; 249 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 9.7 MiB/s wr, 339 op/s 2024-04-26T02:00:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:10 smithi006 ceph-mon[28419]: pgmap v451: 97 pgs: 97 active+clean; 249 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 9.7 MiB/s wr, 339 op/s 2024-04-26T02:00:11.091 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 23.01 MB/sec warmup 34 sec latency 10580.417 ms 2024-04-26T02:00:11.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:11 smithi029 ceph-mon[32775]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T02:00:11.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:11 smithi006 ceph-mon[28419]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T02:00:11.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:11 smithi188 ceph-mon[32751]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T02:00:12.091 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 22.36 MB/sec warmup 35 sec latency 11580.562 ms 2024-04-26T02:00:12.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:12 smithi006 ceph-mon[28419]: pgmap v453: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 15 MiB/s wr, 301 op/s 2024-04-26T02:00:12.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:12 smithi188 ceph-mon[32751]: pgmap v453: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 15 MiB/s wr, 301 op/s 2024-04-26T02:00:12.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:12 smithi029 ceph-mon[32775]: pgmap v453: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 15 MiB/s wr, 301 op/s 2024-04-26T02:00:13.092 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 21.74 MB/sec warmup 36 sec latency 12580.724 ms 2024-04-26T02:00:14.091 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 21.15 MB/sec warmup 37 sec latency 13580.805 ms 2024-04-26T02:00:14.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:14 smithi006 ceph-mon[28419]: pgmap v454: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 14 MiB/s wr, 278 op/s 2024-04-26T02:00:14.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:14 smithi188 ceph-mon[32751]: pgmap v454: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 14 MiB/s wr, 278 op/s 2024-04-26T02:00:14.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:14 smithi029 ceph-mon[32775]: pgmap v454: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 14 MiB/s wr, 278 op/s 2024-04-26T02:00:15.092 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 20.59 MB/sec warmup 38 sec latency 14580.976 ms 2024-04-26T02:00:16.092 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 20.06 MB/sec warmup 39 sec latency 15581.130 ms 2024-04-26T02:00:16.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:16 smithi006 ceph-mon[28419]: pgmap v455: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 12 MiB/s wr, 240 op/s 2024-04-26T02:00:16.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:16 smithi188 ceph-mon[32751]: pgmap v455: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 12 MiB/s wr, 240 op/s 2024-04-26T02:00:16.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:16 smithi029 ceph-mon[32775]: pgmap v455: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 12 MiB/s wr, 240 op/s 2024-04-26T02:00:17.092 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 19.56 MB/sec warmup 40 sec latency 16581.293 ms 2024-04-26T02:00:18.092 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 19.09 MB/sec warmup 41 sec latency 17581.457 ms 2024-04-26T02:00:18.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:18 smithi006 ceph-mon[28419]: pgmap v456: 97 pgs: 97 active+clean; 386 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 18 MiB/s wr, 276 op/s 2024-04-26T02:00:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:18 smithi188 ceph-mon[32751]: pgmap v456: 97 pgs: 97 active+clean; 386 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 18 MiB/s wr, 276 op/s 2024-04-26T02:00:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:18 smithi029 ceph-mon[32775]: pgmap v456: 97 pgs: 97 active+clean; 386 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 18 MiB/s wr, 276 op/s 2024-04-26T02:00:19.092 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 18.63 MB/sec warmup 42 sec latency 18581.555 ms 2024-04-26T02:00:19.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.387+0000 7f3393200640 1 -- 172.21.15.6:0/1023680570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3394102f20 msgr2=0x7f3394103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.387+0000 7f3393200640 1 --2- 172.21.15.6:0/1023680570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3394102f20 0x7f3394103300 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3384009900 tx=0x7f338402a480 comp rx=0 tx=0).stop 2024-04-26T02:00:19.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.387+0000 7f3393200640 1 -- 172.21.15.6:0/1023680570 shutdown_connections 2024-04-26T02:00:19.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.387+0000 7f3393200640 1 --2- 172.21.15.6:0/1023680570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f339410c340 0x7f339410e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.387+0000 7f3393200640 1 --2- 172.21.15.6:0/1023680570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3394103840 0x7f339410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.387+0000 7f3393200640 1 --2- 172.21.15.6:0/1023680570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3394102f20 0x7f3394103300 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.387+0000 7f3393200640 1 -- 172.21.15.6:0/1023680570 >> 172.21.15.6:0/1023680570 conn(0x7f33940fc720 msgr2=0x7f33940feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:19.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.388+0000 7f3393200640 1 -- 172.21.15.6:0/1023680570 shutdown_connections 2024-04-26T02:00:19.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.388+0000 7f3393200640 1 -- 172.21.15.6:0/1023680570 wait complete. 2024-04-26T02:00:19.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.389+0000 7f3393200640 1 Processor -- start 2024-04-26T02:00:19.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.389+0000 7f3393200640 1 -- start start 2024-04-26T02:00:19.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.389+0000 7f3393200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3394102f20 0x7f3394100b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.389+0000 7f3393200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3394103840 0x7f33941010b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.389+0000 7f3393200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339410c340 0x7f33941015f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.389+0000 7f3393200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3394112000 con 0x7f3394102f20 2024-04-26T02:00:19.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.389+0000 7f3393200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3394111e80 con 0x7f3394103840 2024-04-26T02:00:19.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.389+0000 7f3393200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3394112180 con 0x7f339410c340 2024-04-26T02:00:19.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.390+0000 7f3391e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3394102f20 0x7f3394100b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.390+0000 7f3391e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3394102f20 0x7f3394100b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54374/0 (socket says 172.21.15.6:54374) 2024-04-26T02:00:19.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.390+0000 7f3391e00640 1 -- 172.21.15.6:0/2928291625 learned_addr learned my addr 172.21.15.6:0/2928291625 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:19.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.390+0000 7f338be00640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3394103840 0x7f33941010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.390+0000 7f3392800640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339410c340 0x7f33941015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.390+0000 7f3391e00640 1 -- 172.21.15.6:0/2928291625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339410c340 msgr2=0x7f33941015f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.390+0000 7f3391e00640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339410c340 0x7f33941015f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.390+0000 7f3391e00640 1 -- 172.21.15.6:0/2928291625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3394103840 msgr2=0x7f33941010b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.390+0000 7f3391e00640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3394103840 0x7f33941010b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.390+0000 7f3391e00640 1 -- 172.21.15.6:0/2928291625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33941a6aa0 con 0x7f3394102f20 2024-04-26T02:00:19.389 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.390+0000 7f338be00640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3394103840 0x7f33941010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:19.390 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.390+0000 7f3391e00640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3394102f20 0x7f3394100b70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3384005ab0 tx=0x7f3384003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:19.390 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.391+0000 7f338aa00640 1 -- 172.21.15.6:0/2928291625 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338403f070 con 0x7f3394102f20 2024-04-26T02:00:19.390 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.391+0000 7f338aa00640 1 -- 172.21.15.6:0/2928291625 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3384031b60 con 0x7f3394102f20 2024-04-26T02:00:19.390 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.391+0000 7f3393200640 1 -- 172.21.15.6:0/2928291625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33941a6d30 con 0x7f3394102f20 2024-04-26T02:00:19.390 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.391+0000 7f338aa00640 1 -- 172.21.15.6:0/2928291625 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3384049060 con 0x7f3394102f20 2024-04-26T02:00:19.390 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.391+0000 7f3393200640 1 -- 172.21.15.6:0/2928291625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33941a71d0 con 0x7f3394102f20 2024-04-26T02:00:19.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.394+0000 7f339bb76740 1 Processor -- start 2024-04-26T02:00:19.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.394+0000 7f338aa00640 1 -- 172.21.15.6:0/2928291625 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f3384042070 con 0x7f3394102f20 2024-04-26T02:00:19.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.394+0000 7f339bb76740 1 -- start start 2024-04-26T02:00:19.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f339bb76740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1b39b0 0x56368a1b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f339bb76740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 0x56368a1be650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f339bb76740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1beb90 0x56368a1c0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f339bb76740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56368a119900 con 0x56368a1beb90 2024-04-26T02:00:19.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f339bb76740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56368a0bfef0 con 0x56368a1b39b0 2024-04-26T02:00:19.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f339bb76740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56368a1158c0 con 0x56368a1b42d0 2024-04-26T02:00:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f3391e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 0x56368a1be650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f3391e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 0x56368a1be650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50050/0 (socket says 172.21.15.6:50050) 2024-04-26T02:00:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f3391e00640 1 -- 172.21.15.6:0/2443040065 learned_addr learned my addr 172.21.15.6:0/2443040065 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f338be00640 1 --2- 172.21.15.6:0/2443040065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1beb90 0x56368a1c0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f3392800640 1 --2- 172.21.15.6:0/2443040065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1b39b0 0x56368a1b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f3391e00640 1 -- 172.21.15.6:0/2443040065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1b39b0 msgr2=0x56368a1b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f3391e00640 1 --2- 172.21.15.6:0/2443040065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1b39b0 0x56368a1b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f3391e00640 1 -- 172.21.15.6:0/2443040065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1beb90 msgr2=0x56368a1c0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f3391e00640 1 --2- 172.21.15.6:0/2443040065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1beb90 0x56368a1c0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.395+0000 7f3391e00640 1 -- 172.21.15.6:0/2443040065 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563689e2c210 con 0x56368a1b42d0 2024-04-26T02:00:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.396+0000 7f3391e00640 1 --2- 172.21.15.6:0/2443040065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 0x56368a1be650 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f33840ee880 tx=0x7f33840f9540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a126c9766e505099 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.396+0000 7f338aa00640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f33780a3d00 0x7f33780a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.396+0000 7f3383000640 1 -- 172.21.15.6:0/2443040065 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33840b94e0 con 0x56368a1b42d0 2024-04-26T02:00:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.396+0000 7f3383000640 1 -- 172.21.15.6:0/2443040065 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33840fe030 con 0x56368a1b42d0 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.396+0000 7f3392800640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f33780a3d00 0x7f33780a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.396+0000 7f338aa00640 1 -- 172.21.15.6:0/2928291625 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8600+0+0 (secure 0 0 0) 0x7f33840f12a0 con 0x7f3394102f20 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.396+0000 7f3383000640 1 -- 172.21.15.6:0/2443040065 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33840bc690 con 0x56368a1b42d0 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.397+0000 7f339bb76740 1 -- 172.21.15.6:0/2443040065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 msgr2=0x56368a1be650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.397+0000 7f339bb76740 1 --2- 172.21.15.6:0/2443040065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 0x56368a1be650 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f33840ee880 tx=0x7f33840f9540 comp rx=0 tx=0).stop 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.397+0000 7f339bb76740 1 -- 172.21.15.6:0/2443040065 shutdown_connections 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.397+0000 7f339bb76740 1 --2- 172.21.15.6:0/2443040065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1beb90 0x56368a1c0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.397+0000 7f339bb76740 1 --2- 172.21.15.6:0/2443040065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 0x56368a1be650 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.397+0000 7f339bb76740 1 --2- 172.21.15.6:0/2443040065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1b39b0 0x56368a1b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.397+0000 7f339bb76740 1 -- 172.21.15.6:0/2443040065 >> 172.21.15.6:0/2443040065 conn(0x56368a1afec0 msgr2=0x56368a1b02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.397+0000 7f3392800640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f33780a3d00 0x7f33780a61c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f338c005fd0 tx=0x7f338c009500 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.397+0000 7f339bb76740 1 -- 172.21.15.6:0/2443040065 shutdown_connections 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.397+0000 7f339bb76740 1 -- 172.21.15.6:0/2443040065 wait complete. 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.398+0000 7f339bb76740 1 -- start start 2024-04-26T02:00:19.400 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.398+0000 7f339bb76740 1 Processor -- start 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.398+0000 7f339bb76740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 0x56368a1bba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.398+0000 7f339bb76740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1b42d0 0x56368a1bbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.398+0000 7f339bb76740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1beb90 0x56368a1bc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.398+0000 7f339bb76740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56368a1c1580 con 0x56368a1b39b0 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.398+0000 7f339bb76740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56368a0d3310 con 0x56368a1b42d0 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.398+0000 7f339bb76740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56368a1c1700 con 0x56368a1beb90 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.398+0000 7f339bb76740 10 client.0 _ll_register_callbacks cb 0x56368a0a1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f3391e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 0x56368a1bba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f3391e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 0x56368a1bba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54386/0 (socket says 172.21.15.6:54386) 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f3391e00640 1 -- 172.21.15.6:0/4018425974 learned_addr learned my addr 172.21.15.6:0/4018425974 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f338be00640 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1b42d0 0x56368a1bbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f3392800640 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1beb90 0x56368a1bc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f3391e00640 1 -- 172.21.15.6:0/4018425974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1beb90 msgr2=0x56368a1bc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f3391e00640 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1beb90 0x56368a1bc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f3391e00640 1 -- 172.21.15.6:0/4018425974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1b42d0 msgr2=0x56368a1bbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f3391e00640 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1b42d0 0x56368a1bbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f3391e00640 1 -- 172.21.15.6:0/4018425974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563689ef8620 con 0x56368a1b39b0 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f338be00640 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1b42d0 0x56368a1bbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f3391e00640 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 0x56368a1bba70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3384036040 tx=0x7f338402a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a20468a116d354eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.399+0000 7f3377400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.400+0000 7f3377400640 1 -- 172.21.15.6:0/4018425974 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33840bc9c0 con 0x56368a1b39b0 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.400+0000 7f3377400640 1 -- 172.21.15.6:0/4018425974 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8600+0+0 (secure 0 0 0) 0x7f33840bb540 con 0x56368a1b39b0 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.400+0000 7f339bb76740 1 -- 172.21.15.6:0/4018425974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33840fe030 con 0x56368a1b39b0 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.400+0000 7f3377400640 1 -- 172.21.15.6:0/4018425974 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f33840bcba0 con 0x56368a1b39b0 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.400+0000 7f339bb76740 10 client.16716 fetch_fsmap learned FSMap version 47 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.400+0000 7f339bb76740 1 -- 172.21.15.6:0/4018425974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56368a1c1580 con 0x56368a1b39b0 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.400+0000 7f3377400640 1 -- 172.21.15.6:0/4018425974 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33840fe030 con 0x56368a1b39b0 2024-04-26T02:00:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.400+0000 7f3377400640 1 -- 172.21.15.6:0/4018425974 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 47) ==== 8354+0+0 (secure 0 0 0) 0x7f33840bc690 con 0x56368a1b39b0 2024-04-26T02:00:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.401+0000 7f339bb76740 10 client.16716 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T02:00:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.401+0000 7f339bb76740 10 client.16716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:00:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.401+0000 7f339bb76740 20 client.16716 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.401+0000 7f339bb76740 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56368a1ce960 0x56368a1d0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.401+0000 7f339bb76740 4 client.16716 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.401+0000 7f339bb76740 1 -- 172.21.15.6:0/4018425974 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56368a12d8b0 con 0x56368a1ce960 2024-04-26T02:00:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.401+0000 7f338be00640 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56368a1ce960 0x56368a1d0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.401+0000 7f338be00640 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56368a1ce960 0x56368a1d0d70 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d009b4bb1a62df7 server_cookie=54d755cf37295a1b in_seq=0 out_seq=0 2024-04-26T02:00:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.401+0000 7f3377400640 10 client.16716 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.402+0000 7f3377400640 1 -- 172.21.15.6:0/4018425974 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56368a12d8b0 con 0x56368a1ce960 2024-04-26T02:00:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.402+0000 7f3377400640 10 client.16716 handle_command_reply: tid=0 2024-04-26T02:00:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.402+0000 7f339bb76740 1 client.16716 shutdown 2024-04-26T02:00:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.402+0000 7f339bb76740 1 -- 172.21.15.6:0/4018425974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 msgr2=0x56368a1bba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.402+0000 7f339bb76740 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 0x56368a1bba70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3384036040 tx=0x7f338402a480 comp rx=0 tx=0).stop 2024-04-26T02:00:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.403+0000 7f339bb76740 1 -- 172.21.15.6:0/4018425974 shutdown_connections 2024-04-26T02:00:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.403+0000 7f339bb76740 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56368a1ce960 0x56368a1d0d70 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.403+0000 7f339bb76740 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1beb90 0x56368a1bc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.403+0000 7f339bb76740 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1b42d0 0x56368a1bbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.403+0000 7f339bb76740 1 --2- 172.21.15.6:0/4018425974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 0x56368a1bba70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.403+0000 7f3377400640 0 client.16716 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.403+0000 7f339bb76740 1 -- 172.21.15.6:0/4018425974 >> 172.21.15.6:0/4018425974 conn(0x56368a1afec0 msgr2=0x56368a1b5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.403+0000 7f339bb76740 1 -- 172.21.15.6:0/4018425974 shutdown_connections 2024-04-26T02:00:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.403+0000 7f339bb76740 1 -- 172.21.15.6:0/4018425974 wait complete. 2024-04-26T02:00:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.403+0000 7f339bb76740 20 client.16716 trim_cache size 0 max 16384 2024-04-26T02:00:19.417 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.419+0000 7f339bb76740 1 Processor -- start 2024-04-26T02:00:19.417 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.419+0000 7f339bb76740 1 -- start start 2024-04-26T02:00:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f339bb76740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b39b0 0x56368a1ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f339bb76740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b42d0 0x56368a1b6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f339bb76740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1beb90 0x56368a1aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f339bb76740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f33941b26f0 con 0x56368a1b42d0 2024-04-26T02:00:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f339bb76740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f33941aff30 con 0x56368a1beb90 2024-04-26T02:00:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f339bb76740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f33941adda0 con 0x56368a1b39b0 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f3391e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b39b0 0x56368a1ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f3391e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b39b0 0x56368a1ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50062/0 (socket says 172.21.15.6:50062) 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f3391e00640 1 -- 172.21.15.6:0/1576030419 learned_addr learned my addr 172.21.15.6:0/1576030419 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f338be00640 1 --2- 172.21.15.6:0/1576030419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b42d0 0x56368a1b6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f3392800640 1 --2- 172.21.15.6:0/1576030419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1beb90 0x56368a1aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f3391e00640 1 -- 172.21.15.6:0/1576030419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1beb90 msgr2=0x56368a1aeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f3391e00640 1 --2- 172.21.15.6:0/1576030419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1beb90 0x56368a1aeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f3391e00640 1 -- 172.21.15.6:0/1576030419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b42d0 msgr2=0x56368a1b6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f3391e00640 1 --2- 172.21.15.6:0/1576030419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b42d0 0x56368a1b6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f3391e00640 1 -- 172.21.15.6:0/1576030419 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56368a1afcf0 con 0x56368a1b39b0 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f3392800640 1 --2- 172.21.15.6:0/1576030419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1beb90 0x56368a1aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.420+0000 7f338be00640 1 --2- 172.21.15.6:0/1576030419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b42d0 0x56368a1b6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:00:19.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.421+0000 7f3391e00640 1 --2- 172.21.15.6:0/1576030419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b39b0 0x56368a1ca3b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f33840b9c10 tx=0x7f3384009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7e5e6e84da08288 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.421+0000 7f3382600640 1 -- 172.21.15.6:0/1576030419 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33840bcd80 con 0x56368a1b39b0 2024-04-26T02:00:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.421+0000 7f3382600640 1 -- 172.21.15.6:0/1576030419 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33840fe070 con 0x56368a1b39b0 2024-04-26T02:00:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.421+0000 7f3382600640 1 -- 172.21.15.6:0/1576030419 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33840bd010 con 0x56368a1b39b0 2024-04-26T02:00:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.421+0000 7f339bb76740 1 -- 172.21.15.6:0/1576030419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b39b0 msgr2=0x56368a1ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.421+0000 7f339bb76740 1 --2- 172.21.15.6:0/1576030419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b39b0 0x56368a1ca3b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f33840b9c10 tx=0x7f3384009680 comp rx=0 tx=0).stop 2024-04-26T02:00:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.422+0000 7f339bb76740 1 -- 172.21.15.6:0/1576030419 shutdown_connections 2024-04-26T02:00:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.422+0000 7f339bb76740 1 --2- 172.21.15.6:0/1576030419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b42d0 0x56368a1b6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.422+0000 7f339bb76740 1 --2- 172.21.15.6:0/1576030419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1beb90 0x56368a1aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.422+0000 7f339bb76740 1 --2- 172.21.15.6:0/1576030419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b39b0 0x56368a1ca3b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.422+0000 7f339bb76740 1 -- 172.21.15.6:0/1576030419 >> 172.21.15.6:0/1576030419 conn(0x56368a1afec0 msgr2=0x56368a1ae790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.422+0000 7f339bb76740 1 -- 172.21.15.6:0/1576030419 shutdown_connections 2024-04-26T02:00:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.422+0000 7f339bb76740 1 -- 172.21.15.6:0/1576030419 wait complete. 2024-04-26T02:00:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.422+0000 7f339bb76740 1 -- start start 2024-04-26T02:00:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.422+0000 7f339bb76740 1 Processor -- start 2024-04-26T02:00:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f339bb76740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 0x56368a1c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f339bb76740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 0x56368a1c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f339bb76740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1beb90 0x56368a1c4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f339bb76740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f33941ae3f0 con 0x56368a1b39b0 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f339bb76740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f33941af860 con 0x56368a1beb90 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f339bb76740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f33941aeac0 con 0x56368a1b42d0 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f339bb76740 10 client.0 _ll_register_callbacks cb 0x7f33840f9300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f3391e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 0x56368a1c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f338be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 0x56368a1c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f3391e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 0x56368a1c37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54404/0 (socket says 172.21.15.6:54404) 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f3391e00640 1 -- 172.21.15.6:0/4207336449 learned_addr learned my addr 172.21.15.6:0/4207336449 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f3392800640 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1beb90 0x56368a1c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f3391e00640 1 -- 172.21.15.6:0/4207336449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 msgr2=0x56368a1c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f3391e00640 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 0x56368a1c3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f3391e00640 1 -- 172.21.15.6:0/4207336449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1beb90 msgr2=0x56368a1c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f3391e00640 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1beb90 0x56368a1c4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f3391e00640 1 -- 172.21.15.6:0/4207336449 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56368a233190 con 0x56368a1b39b0 2024-04-26T02:00:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f3392800640 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1beb90 0x56368a1c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.423+0000 7f338be00640 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 0x56368a1c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.424+0000 7f3391e00640 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 0x56368a1c37c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f33840c58d0 tx=0x7f33840f9540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a210f15c846acdab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.424+0000 7f3377400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:00:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.424+0000 7f3377400640 1 -- 172.21.15.6:0/4207336449 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33840b8f90 con 0x56368a1b39b0 2024-04-26T02:00:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.424+0000 7f3377400640 1 -- 172.21.15.6:0/4207336449 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8600+0+0 (secure 0 0 0) 0x7f33840b98b0 con 0x56368a1b39b0 2024-04-26T02:00:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.424+0000 7f339bb76740 1 -- 172.21.15.6:0/4207336449 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33840fe070 con 0x56368a1b39b0 2024-04-26T02:00:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.424+0000 7f3377400640 1 -- 172.21.15.6:0/4207336449 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f33840b9310 con 0x56368a1b39b0 2024-04-26T02:00:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.424+0000 7f339bb76740 10 client.16719 fetch_fsmap learned FSMap version 47 2024-04-26T02:00:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.424+0000 7f339bb76740 1 -- 172.21.15.6:0/4207336449 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56368a233190 con 0x56368a1b39b0 2024-04-26T02:00:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.424+0000 7f3377400640 1 -- 172.21.15.6:0/4207336449 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33840fe070 con 0x56368a1b39b0 2024-04-26T02:00:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.425+0000 7f3377400640 1 -- 172.21.15.6:0/4207336449 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 47) ==== 8354+0+0 (secure 0 0 0) 0x7f338403b460 con 0x56368a1b39b0 2024-04-26T02:00:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.425+0000 7f339bb76740 10 client.16719 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T02:00:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.425+0000 7f339bb76740 10 client.16719 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:00:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.425+0000 7f339bb76740 20 client.16719 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.425+0000 7f339bb76740 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56368a2346a0 0x56368a236ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.425+0000 7f339bb76740 4 client.16719 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:00:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.425+0000 7f339bb76740 1 -- 172.21.15.6:0/4207336449 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56368a127320 con 0x56368a2346a0 2024-04-26T02:00:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.425+0000 7f338be00640 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56368a2346a0 0x56368a236ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.426+0000 7f338be00640 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56368a2346a0 0x56368a236ac0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ceadde3ab71463a server_cookie=10e3f60988cf621 in_seq=0 out_seq=0 2024-04-26T02:00:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.426+0000 7f3377400640 10 client.16719 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.519+0000 7f3377400640 1 -- 172.21.15.6:0/4207336449 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56368a127320 con 0x56368a2346a0 2024-04-26T02:00:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.519+0000 7f3377400640 10 client.16719 handle_command_reply: tid=0 2024-04-26T02:00:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.519+0000 7f339bb76740 1 client.16719 shutdown 2024-04-26T02:00:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.519+0000 7f339bb76740 1 -- 172.21.15.6:0/4207336449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 msgr2=0x56368a1c37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.519+0000 7f339bb76740 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 0x56368a1c37c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f33840c58d0 tx=0x7f33840f9540 comp rx=0 tx=0).stop 2024-04-26T02:00:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.520+0000 7f339bb76740 1 -- 172.21.15.6:0/4207336449 shutdown_connections 2024-04-26T02:00:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.520+0000 7f339bb76740 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56368a2346a0 0x56368a236ac0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.520+0000 7f339bb76740 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56368a1beb90 0x56368a1c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.520+0000 7f339bb76740 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56368a1b42d0 0x56368a1c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.520+0000 7f339bb76740 1 --2- 172.21.15.6:0/4207336449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56368a1b39b0 0x56368a1c37c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.520+0000 7f3377400640 0 client.16719 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.520+0000 7f339bb76740 1 -- 172.21.15.6:0/4207336449 >> 172.21.15.6:0/4207336449 conn(0x56368a1afec0 msgr2=0x56368a1bc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.520+0000 7f339bb76740 1 -- 172.21.15.6:0/4207336449 shutdown_connections 2024-04-26T02:00:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.521+0000 7f339bb76740 1 -- 172.21.15.6:0/4207336449 wait complete. 2024-04-26T02:00:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.521+0000 7f339bb76740 20 client.16719 trim_cache size 0 max 16384 2024-04-26T02:00:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:00:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:00:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:00:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:00:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.522+0000 7f3393200640 1 -- 172.21.15.6:0/2928291625 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f33780a3d00 msgr2=0x7f33780a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.522+0000 7f3393200640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f33780a3d00 0x7f33780a61c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f338c005fd0 tx=0x7f338c009500 comp rx=0 tx=0).stop 2024-04-26T02:00:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.522+0000 7f3393200640 1 -- 172.21.15.6:0/2928291625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3394102f20 msgr2=0x7f3394100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.522+0000 7f3393200640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3394102f20 0x7f3394100b70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3384005ab0 tx=0x7f3384003c10 comp rx=0 tx=0).stop 2024-04-26T02:00:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.523+0000 7f3393200640 1 -- 172.21.15.6:0/2928291625 shutdown_connections 2024-04-26T02:00:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.523+0000 7f3393200640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f33780a3d00 0x7f33780a61c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.523+0000 7f3393200640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339410c340 0x7f33941015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.523+0000 7f3393200640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3394103840 0x7f33941010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.523+0000 7f3393200640 1 --2- 172.21.15.6:0/2928291625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3394102f20 0x7f3394100b70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.523+0000 7f3393200640 1 -- 172.21.15.6:0/2928291625 >> 172.21.15.6:0/2928291625 conn(0x7f33940fc720 msgr2=0x7f33940fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.523+0000 7f3393200640 1 -- 172.21.15.6:0/2928291625 shutdown_connections 2024-04-26T02:00:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:19.523+0000 7f3393200640 1 -- 172.21.15.6:0/2928291625 wait complete. 2024-04-26T02:00:20.093 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 18.20 MB/sec warmup 43 sec latency 19581.692 ms 2024-04-26T02:00:20.393 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:00:19 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: ignoring --setuser ceph since I am not root 2024-04-26T02:00:20.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:00:19 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: ignoring --setgroup ceph since I am not root 2024-04-26T02:00:20.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:00:19 smithi188 ceph-mds[68153]: -- 172.21.15.188:0/3149323318 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562db3d2da40 con 0x562db3d00c00 2024-04-26T02:00:20.394 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:00:19 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: starting mds.f at 2024-04-26T02:00:20.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:20 smithi006 ceph-mon[28419]: pgmap v457: 97 pgs: 97 active+clean; 386 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 18 MiB/s wr, 276 op/s 2024-04-26T02:00:20.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:20 smithi006 ceph-mon[28419]: daemon mds.f restarted 2024-04-26T02:00:20.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:20 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T02:00:20.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:20 smithi006 ceph-mon[28419]: osdmap e119: 12 total, 12 up, 12 in 2024-04-26T02:00:20.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:20 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6834/1750728695,v1:172.21.15.188:6835/1750728695] up:boot 2024-04-26T02:00:20.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:20 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:00:20.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:20 smithi188 ceph-mon[32751]: pgmap v457: 97 pgs: 97 active+clean; 386 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 18 MiB/s wr, 276 op/s 2024-04-26T02:00:20.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:20 smithi188 ceph-mon[32751]: daemon mds.f restarted 2024-04-26T02:00:20.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:20 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T02:00:20.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:20 smithi188 ceph-mon[32751]: osdmap e119: 12 total, 12 up, 12 in 2024-04-26T02:00:20.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:20 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6834/1750728695,v1:172.21.15.188:6835/1750728695] up:boot 2024-04-26T02:00:20.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:20 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:00:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:20 smithi029 ceph-mon[32775]: pgmap v457: 97 pgs: 97 active+clean; 386 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 18 MiB/s wr, 276 op/s 2024-04-26T02:00:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:20 smithi029 ceph-mon[32775]: daemon mds.f restarted 2024-04-26T02:00:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:20 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T02:00:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:20 smithi029 ceph-mon[32775]: osdmap e119: 12 total, 12 up, 12 in 2024-04-26T02:00:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:20 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6834/1750728695,v1:172.21.15.188:6835/1750728695] up:boot 2024-04-26T02:00:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:20 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:00:21.093 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 17.78 MB/sec warmup 44 sec latency 20581.881 ms 2024-04-26T02:00:22.093 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 17.39 MB/sec warmup 45 sec latency 21582.013 ms 2024-04-26T02:00:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:22 smithi006 ceph-mon[28419]: pgmap v459: 97 pgs: 97 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 8.6 MiB/s wr, 173 op/s 2024-04-26T02:00:22.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:22 smithi188 ceph-mon[32751]: pgmap v459: 97 pgs: 97 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 8.6 MiB/s wr, 173 op/s 2024-04-26T02:00:22.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:22 smithi029 ceph-mon[32775]: pgmap v459: 97 pgs: 97 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 8.6 MiB/s wr, 173 op/s 2024-04-26T02:00:23.093 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 17.01 MB/sec warmup 46 sec latency 22582.104 ms 2024-04-26T02:00:24.093 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 16.65 MB/sec warmup 47 sec latency 23582.218 ms 2024-04-26T02:00:24.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:24 smithi006 ceph-mon[28419]: pgmap v460: 97 pgs: 97 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 8.6 MiB/s wr, 173 op/s 2024-04-26T02:00:24.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:24 smithi188 ceph-mon[32751]: pgmap v460: 97 pgs: 97 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 8.6 MiB/s wr, 173 op/s 2024-04-26T02:00:24.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:24 smithi029 ceph-mon[32775]: pgmap v460: 97 pgs: 97 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 8.6 MiB/s wr, 173 op/s 2024-04-26T02:00:25.093 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 16.30 MB/sec warmup 48 sec latency 24582.370 ms 2024-04-26T02:00:26.093 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 15.97 MB/sec warmup 49 sec latency 25582.553 ms 2024-04-26T02:00:26.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:26 smithi006 ceph-mon[28419]: pgmap v461: 97 pgs: 97 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 8.6 MiB/s wr, 173 op/s 2024-04-26T02:00:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:26 smithi188 ceph-mon[32751]: pgmap v461: 97 pgs: 97 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 8.6 MiB/s wr, 173 op/s 2024-04-26T02:00:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:26 smithi029 ceph-mon[32775]: pgmap v461: 97 pgs: 97 active+clean; 390 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 8.6 MiB/s wr, 173 op/s 2024-04-26T02:00:27.093 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 15.65 MB/sec warmup 50 sec latency 26582.690 ms 2024-04-26T02:00:27.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:00:27.590 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:00:28.094 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 15.34 MB/sec warmup 51 sec latency 27582.878 ms 2024-04-26T02:00:28.798 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:28 smithi006 ceph-mon[28419]: pgmap v462: 97 pgs: 97 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 3.3 MiB/s wr, 116 op/s 2024-04-26T02:00:28.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:28 smithi188 ceph-mon[32751]: pgmap v462: 97 pgs: 97 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 3.3 MiB/s wr, 116 op/s 2024-04-26T02:00:28.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:28 smithi029 ceph-mon[32775]: pgmap v462: 97 pgs: 97 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 3.3 MiB/s wr, 116 op/s 2024-04-26T02:00:29.094 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 15.05 MB/sec warmup 52 sec latency 28582.975 ms 2024-04-26T02:00:29.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.711+0000 7fccf6000700 1 -- 172.21.15.6:0/2017739032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccf0075590 msgr2=0x7fccf010ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:29.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.711+0000 7fccf6000700 1 --2- 172.21.15.6:0/2017739032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccf0075590 0x7fccf010ef00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fccdc009990 tx=0x7fccdc009ca0 comp rx=0 tx=0).stop 2024-04-26T02:00:29.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.711+0000 7fccf6000700 1 -- 172.21.15.6:0/2017739032 shutdown_connections 2024-04-26T02:00:29.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.711+0000 7fccf6000700 1 --2- 172.21.15.6:0/2017739032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccf010f440 0x7fccf0111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:29.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.711+0000 7fccf6000700 1 --2- 172.21.15.6:0/2017739032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf0074c70 0x7fccf0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:29.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.711+0000 7fccf6000700 1 --2- 172.21.15.6:0/2017739032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccf0075590 0x7fccf010ef00 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:29.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.711+0000 7fccf6000700 1 -- 172.21.15.6:0/2017739032 >> 172.21.15.6:0/2017739032 conn(0x7fccf0079ba0 msgr2=0x7fccf0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:29.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.711+0000 7fccf6000700 1 -- 172.21.15.6:0/2017739032 shutdown_connections 2024-04-26T02:00:29.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.711+0000 7fccf6000700 1 -- 172.21.15.6:0/2017739032 wait complete. 2024-04-26T02:00:29.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.712+0000 7fccf6000700 1 Processor -- start 2024-04-26T02:00:29.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.713+0000 7fccf6000700 1 -- start start 2024-04-26T02:00:29.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.713+0000 7fccf6000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf0074c70 0x7fccf019b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:29.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.713+0000 7fccf6000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccf0075590 0x7fccf019b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:29.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.713+0000 7fccf6000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccf010f440 0x7fccf019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:29.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.713+0000 7fccf6000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fccf0114e70 con 0x7fccf0075590 2024-04-26T02:00:29.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.713+0000 7fccf6000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fccf0114cf0 con 0x7fccf0074c70 2024-04-26T02:00:29.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.713+0000 7fccf6000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fccf0114ff0 con 0x7fccf010f440 2024-04-26T02:00:29.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.713+0000 7fccef400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf0074c70 0x7fccf019b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:29.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.713+0000 7fccef400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf0074c70 0x7fccf019b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:42886/0 (socket says 172.21.15.6:42886) 2024-04-26T02:00:29.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.713+0000 7fccef400700 1 -- 172.21.15.6:0/146449623 learned_addr learned my addr 172.21.15.6:0/146449623 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:29.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.713+0000 7fcceea00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccf0075590 0x7fccf019b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:29.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.714+0000 7fccefe00700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccf010f440 0x7fccf019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:29.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.714+0000 7fccef400700 1 -- 172.21.15.6:0/146449623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccf010f440 msgr2=0x7fccf019fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:29.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.714+0000 7fccef400700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccf010f440 0x7fccf019fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:29.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.714+0000 7fccef400700 1 -- 172.21.15.6:0/146449623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccf0075590 msgr2=0x7fccf019b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:29.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.714+0000 7fccef400700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccf0075590 0x7fccf019b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:29.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.714+0000 7fccef400700 1 -- 172.21.15.6:0/146449623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf01a0360 con 0x7fccf0074c70 2024-04-26T02:00:29.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.714+0000 7fccef400700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf0074c70 0x7fccf019b3a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcce000e9c0 tx=0x7fcce000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:29.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.714+0000 7fcce7e00700 1 -- 172.21.15.6:0/146449623 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce000cbd0 con 0x7fccf0074c70 2024-04-26T02:00:29.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.715+0000 7fccf6000700 1 -- 172.21.15.6:0/146449623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf01a0650 con 0x7fccf0074c70 2024-04-26T02:00:29.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.715+0000 7fcce7e00700 1 -- 172.21.15.6:0/146449623 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcce000cd70 con 0x7fccf0074c70 2024-04-26T02:00:29.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.715+0000 7fcce7e00700 1 -- 172.21.15.6:0/146449623 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce0005610 con 0x7fccf0074c70 2024-04-26T02:00:29.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.715+0000 7fccf6000700 1 -- 172.21.15.6:0/146449623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf01a7ed0 con 0x7fccf0074c70 2024-04-26T02:00:29.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.716+0000 7fccf6000700 1 -- 172.21.15.6:0/146449623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccf0069730 con 0x7fccf0074c70 2024-04-26T02:00:29.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.718+0000 7fcce7e00700 1 -- 172.21.15.6:0/146449623 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fcce0004750 con 0x7fccf0074c70 2024-04-26T02:00:29.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.719+0000 7fcce7e00700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fccd40a3db0 0x7fccd40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:29.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.719+0000 7fcce7e00700 1 -- 172.21.15.6:0/146449623 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7fcce00d0b70 con 0x7fccf0074c70 2024-04-26T02:00:29.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.720+0000 7fcceea00700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fccd40a3db0 0x7fccd40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:29.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.720+0000 7fcceea00700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fccd40a3db0 0x7fccd40a6270 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fccf0102640 tx=0x7fccdc00b260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:29.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:29.722+0000 7fcce7e00700 1 -- 172.21.15.6:0/146449623 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcce009d190 con 0x7fccf0074c70 2024-04-26T02:00:30.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.006+0000 7fccf6000700 1 -- 172.21.15.6:0/146449623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fccf019c6e0 con 0x7fccf0074c70 2024-04-26T02:00:30.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.007+0000 7fcce7e00700 1 -- 172.21.15.6:0/146449623 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11855 (secure 0 0 0) 0x7fccf019c6e0 con 0x7fccf0074c70 2024-04-26T02:00:30.008 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:00:30.008 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T02:00:20.436443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26935":{"gid":26935,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1750728695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1750728695},{"type":"v1","addr":"172.21.15.188:6835","nonce":1750728695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24512,24505]},"id":1}]} 2024-04-26T02:00:30.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.010+0000 7fccf6000700 1 -- 172.21.15.6:0/146449623 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fccd40a3db0 msgr2=0x7fccd40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:30.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.010+0000 7fccf6000700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fccd40a3db0 0x7fccd40a6270 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fccf0102640 tx=0x7fccdc00b260 comp rx=0 tx=0).stop 2024-04-26T02:00:30.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.010+0000 7fccf6000700 1 -- 172.21.15.6:0/146449623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf0074c70 msgr2=0x7fccf019b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:30.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.010+0000 7fccf6000700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf0074c70 0x7fccf019b3a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcce000e9c0 tx=0x7fcce000ecd0 comp rx=0 tx=0).stop 2024-04-26T02:00:30.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.010+0000 7fccf6000700 1 -- 172.21.15.6:0/146449623 shutdown_connections 2024-04-26T02:00:30.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.010+0000 7fccf6000700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fccd40a3db0 0x7fccd40a6270 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:30.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.010+0000 7fccf6000700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fccf010f440 0x7fccf019fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:30.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.010+0000 7fccf6000700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccf0074c70 0x7fccf019b3a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:30.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.010+0000 7fccf6000700 1 --2- 172.21.15.6:0/146449623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccf0075590 0x7fccf019b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:30.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.010+0000 7fccf6000700 1 -- 172.21.15.6:0/146449623 >> 172.21.15.6:0/146449623 conn(0x7fccf0079ba0 msgr2=0x7fccf00ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:30.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.011+0000 7fccf6000700 1 -- 172.21.15.6:0/146449623 shutdown_connections 2024-04-26T02:00:30.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:30.011+0000 7fccf6000700 1 -- 172.21.15.6:0/146449623 wait complete. 2024-04-26T02:00:30.010 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 48 2024-04-26T02:00:30.094 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 14.76 MB/sec warmup 53 sec latency 29583.058 ms 2024-04-26T02:00:30.779 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:30 smithi006 ceph-mon[28419]: pgmap v463: 97 pgs: 97 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 3.3 MiB/s wr, 116 op/s 2024-04-26T02:00:30.779 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:30 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:00:30.779 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:30 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/146449623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:30.857 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id c6386e05 --release --await' 2024-04-26T02:00:30.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id c6386e05 --release --await 2024-04-26T02:00:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:30 smithi188 ceph-mon[32751]: pgmap v463: 97 pgs: 97 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 3.3 MiB/s wr, 116 op/s 2024-04-26T02:00:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:30 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:00:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:30 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/146449623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:30 smithi029 ceph-mon[32775]: pgmap v463: 97 pgs: 97 active+clean; 405 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 3.3 MiB/s wr, 116 op/s 2024-04-26T02:00:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:30 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:00:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:30 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/146449623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:31.094 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 14.49 MB/sec warmup 54 sec latency 30583.136 ms 2024-04-26T02:00:31.326 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:00:31.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:31 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:00:31.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:31 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:00:31.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:31 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:00:31.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:31 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:00:31.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:31 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:00:31.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:31 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:00:31.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:31 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:00:31.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:31 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:00:31.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:31 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:00:32.094 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 14.23 MB/sec warmup 55 sec latency 31583.256 ms 2024-04-26T02:00:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.681+0000 7f30efe00640 1 -- 172.21.15.6:0/371912976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f010a560 msgr2=0x7f30f0110f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.681+0000 7f30efe00640 1 --2- 172.21.15.6:0/371912976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f010a560 0x7f30f0110f40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009ca0 tx=0x7f30e802a8c0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.681+0000 7f30efe00640 1 -- 172.21.15.6:0/371912976 shutdown_connections 2024-04-26T02:00:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.681+0000 7f30efe00640 1 --2- 172.21.15.6:0/371912976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30f0101290 0x7f30f0109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.681+0000 7f30efe00640 1 --2- 172.21.15.6:0/371912976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f010a560 0x7f30f0110f40 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.681+0000 7f30efe00640 1 --2- 172.21.15.6:0/371912976 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30f01008e0 0x7f30f0100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.681+0000 7f30efe00640 1 -- 172.21.15.6:0/371912976 >> 172.21.15.6:0/371912976 conn(0x7f30f0075f90 msgr2=0x7f30f00763a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.681+0000 7f30efe00640 1 -- 172.21.15.6:0/371912976 shutdown_connections 2024-04-26T02:00:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.681+0000 7f30efe00640 1 -- 172.21.15.6:0/371912976 wait complete. 2024-04-26T02:00:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.682+0000 7f30efe00640 1 Processor -- start 2024-04-26T02:00:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.682+0000 7f30efe00640 1 -- start start 2024-04-26T02:00:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.682+0000 7f30efe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f01008e0 0x7f30f01a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.682+0000 7f30efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30f010a560 0x7f30f01a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.682+0000 7f30efe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30f019a680 0x7f30f019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.682+0000 7f30efe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f30f01142a0 con 0x7f30f010a560 2024-04-26T02:00:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.682+0000 7f30efe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f30f0114120 con 0x7f30f01008e0 2024-04-26T02:00:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.682+0000 7f30efe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f30f0114420 con 0x7f30f019a680 2024-04-26T02:00:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.683+0000 7f30e6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30f010a560 0x7f30f01a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.683+0000 7f30e6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30f010a560 0x7f30f01a0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43560/0 (socket says 172.21.15.6:43560) 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.683+0000 7f30e6000640 1 -- 172.21.15.6:0/3227540023 learned_addr learned my addr 172.21.15.6:0/3227540023 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.683+0000 7f30eea00640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f01008e0 0x7f30f01a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.683+0000 7f30ef400640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30f019a680 0x7f30f019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.683+0000 7f30e6000640 1 -- 172.21.15.6:0/3227540023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30f019a680 msgr2=0x7f30f019ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.683+0000 7f30e6000640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30f019a680 0x7f30f019ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.683+0000 7f30e6000640 1 -- 172.21.15.6:0/3227540023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f01008e0 msgr2=0x7f30f01a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.683+0000 7f30e6000640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f01008e0 0x7f30f01a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.683+0000 7f30e6000640 1 -- 172.21.15.6:0/3227540023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f019b280 con 0x7f30f010a560 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.683+0000 7f30eea00640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f01008e0 0x7f30f01a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.684+0000 7f30e6000640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30f010a560 0x7f30f01a0ab0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f30d800e9f0 tx=0x7f30d800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.684+0000 7f30e7e00640 1 -- 172.21.15.6:0/3227540023 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d800ccc0 con 0x7f30f010a560 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.684+0000 7f30e7e00640 1 -- 172.21.15.6:0/3227540023 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30d80108e0 con 0x7f30f010a560 2024-04-26T02:00:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.684+0000 7f30efe00640 1 -- 172.21.15.6:0/3227540023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30f01a7310 con 0x7f30f010a560 2024-04-26T02:00:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.684+0000 7f30e7e00640 1 -- 172.21.15.6:0/3227540023 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d8018860 con 0x7f30f010a560 2024-04-26T02:00:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.684+0000 7f30efe00640 1 -- 172.21.15.6:0/3227540023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30f01a7710 con 0x7f30f010a560 2024-04-26T02:00:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.687+0000 7f30f8413740 1 Processor -- start 2024-04-26T02:00:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.687+0000 7f30e7e00640 1 -- 172.21.15.6:0/3227540023 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f30d8010430 con 0x7f30f010a560 2024-04-26T02:00:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.687+0000 7f30f8413740 1 -- start start 2024-04-26T02:00:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30f8413740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba772d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30f8413740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba77d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30f8413740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 0x559eba77ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30f8413740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x559eba6d8900 con 0x559eba77db90 2024-04-26T02:00:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30f8413740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559eba67eef0 con 0x559eba7729b0 2024-04-26T02:00:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30f8413740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559eba6d48c0 con 0x559eba7732d0 2024-04-26T02:00:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30e6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 0x559eba77ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30eea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba77d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30ef400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba772d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30eea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba77d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44224/0 (socket says 172.21.15.6:44224) 2024-04-26T02:00:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30e6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 0x559eba77ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43562/0 (socket says 172.21.15.6:43562) 2024-04-26T02:00:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30eea00640 1 -- 172.21.15.6:0/1650041920 learned_addr learned my addr 172.21.15.6:0/1650041920 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.688+0000 7f30ef400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba772d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:42910/0 (socket says 172.21.15.6:42910) 2024-04-26T02:00:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.689+0000 7f30ef400640 1 -- 172.21.15.6:0/1650041920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 msgr2=0x559eba77d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.689+0000 7f30ef400640 1 --2- 172.21.15.6:0/1650041920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba77d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.689+0000 7f30ef400640 1 -- 172.21.15.6:0/1650041920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 msgr2=0x559eba77ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.689+0000 7f30ef400640 1 --2- 172.21.15.6:0/1650041920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 0x559eba77ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.689+0000 7f30ef400640 1 -- 172.21.15.6:0/1650041920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559eba3eb210 con 0x559eba7729b0 2024-04-26T02:00:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.689+0000 7f30e7e00640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f30d00a3d00 0x7f30d00a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.689+0000 7f30ef400640 1 --2- 172.21.15.6:0/1650041920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba772d90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f30e8005e30 tx=0x7f30e80093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cf8919ed25f36cd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.689+0000 7f30e7e00640 1 -- 172.21.15.6:0/3227540023 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7f30d80d0a70 con 0x7f30f010a560 2024-04-26T02:00:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.689+0000 7f30eea00640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f30d00a3d00 0x7f30d00a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.689+0000 7f30df400640 1 -- 172.21.15.6:0/1650041920 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8031cf0 con 0x559eba7729b0 2024-04-26T02:00:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.689+0000 7f30df400640 1 -- 172.21.15.6:0/1650041920 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30e8036030 con 0x559eba7729b0 2024-04-26T02:00:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.689+0000 7f30df400640 1 -- 172.21.15.6:0/1650041920 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e80042a0 con 0x559eba7729b0 2024-04-26T02:00:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.690+0000 7f30eea00640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f30d00a3d00 0x7f30d00a61c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x559eba773f50 tx=0x7f30e0005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.690+0000 7f30f8413740 1 -- 172.21.15.6:0/1650041920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 msgr2=0x559eba772d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.690+0000 7f30f8413740 1 --2- 172.21.15.6:0/1650041920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba772d90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f30e8005e30 tx=0x7f30e80093e0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.690+0000 7f30f8413740 1 -- 172.21.15.6:0/1650041920 shutdown_connections 2024-04-26T02:00:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.690+0000 7f30f8413740 1 --2- 172.21.15.6:0/1650041920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 0x559eba77ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.690+0000 7f30f8413740 1 --2- 172.21.15.6:0/1650041920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba77d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.690+0000 7f30f8413740 1 --2- 172.21.15.6:0/1650041920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba772d90 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.690+0000 7f30f8413740 1 -- 172.21.15.6:0/1650041920 >> 172.21.15.6:0/1650041920 conn(0x559eba76eec0 msgr2=0x559eba76f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.690+0000 7f30f8413740 1 -- 172.21.15.6:0/1650041920 shutdown_connections 2024-04-26T02:00:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.690+0000 7f30f8413740 1 -- 172.21.15.6:0/1650041920 wait complete. 2024-04-26T02:00:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.691+0000 7f30f8413740 1 -- start start 2024-04-26T02:00:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.691+0000 7f30f8413740 1 Processor -- start 2024-04-26T02:00:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30f8413740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba77ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30f8413740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7732d0 0x559eba77b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30f8413740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba77db90 0x559eba77b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30f8413740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x559eba7805e0 con 0x559eba7732d0 2024-04-26T02:00:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30f8413740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559eba692310 con 0x559eba7729b0 2024-04-26T02:00:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30f8413740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559eba780760 con 0x559eba77db90 2024-04-26T02:00:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30f8413740 10 client.0 _ll_register_callbacks cb 0x559eba660cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30e6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7732d0 0x559eba77b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30e6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7732d0 0x559eba77b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43572/0 (socket says 172.21.15.6:43572) 2024-04-26T02:00:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30e6000640 1 -- 172.21.15.6:0/2856357857 learned_addr learned my addr 172.21.15.6:0/2856357857 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30ef400640 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba77db90 0x559eba77b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30eea00640 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba77ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30e6000640 1 -- 172.21.15.6:0/2856357857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba77db90 msgr2=0x559eba77b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30e6000640 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba77db90 0x559eba77b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30e6000640 1 -- 172.21.15.6:0/2856357857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 msgr2=0x559eba77ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30e6000640 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba77ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.692+0000 7f30e6000640 1 -- 172.21.15.6:0/2856357857 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559eba4b7620 con 0x559eba7732d0 2024-04-26T02:00:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.693+0000 7f30e6000640 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7732d0 0x559eba77b0c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f30d800e9c0 tx=0x7f30d800cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0c96ded8da129c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.693+0000 7f30d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:00:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.693+0000 7f30d7e00640 1 -- 172.21.15.6:0/2856357857 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d80d0e60 con 0x559eba7732d0 2024-04-26T02:00:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.693+0000 7f30d7e00640 1 -- 172.21.15.6:0/2856357857 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7f30d80bbb50 con 0x559eba7732d0 2024-04-26T02:00:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.693+0000 7f30f8413740 1 -- 172.21.15.6:0/2856357857 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30e8036030 con 0x559eba7732d0 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.693+0000 7f30d7e00640 1 -- 172.21.15.6:0/2856357857 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f30d80ce5c0 con 0x559eba7732d0 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.693+0000 7f30f8413740 10 client.16737 fetch_fsmap learned FSMap version 48 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.693+0000 7f30f8413740 1 -- 172.21.15.6:0/2856357857 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x559eba78c900 con 0x559eba7732d0 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.693+0000 7f30d7e00640 1 -- 172.21.15.6:0/2856357857 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30e8036030 con 0x559eba7732d0 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.694+0000 7f30d7e00640 1 -- 172.21.15.6:0/2856357857 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7f30d80870d0 con 0x559eba7732d0 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.694+0000 7f30f8413740 10 client.16737 fetch_fsmap finished waiting for FSMap version 48 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.694+0000 7f30f8413740 10 client.16737 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.694+0000 7f30f8413740 20 client.16737 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.694+0000 7f30f8413740 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559eba78de60 0x559eba790290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.694+0000 7f30f8413740 4 client.16737 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.694+0000 7f30f8413740 1 -- 172.21.15.6:0/2856357857 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559eba6ec8b0 con 0x559eba78de60 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.694+0000 7f30eea00640 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559eba78de60 0x559eba790290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.695+0000 7f30eea00640 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559eba78de60 0x559eba790290 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6570c1e085cf9cb1 server_cookie=26cc188c278f6790 in_seq=0 out_seq=0 2024-04-26T02:00:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.695+0000 7f30d7e00640 10 client.16737 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30d7e00640 1 -- 172.21.15.6:0/2856357857 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559eba6ec8b0 con 0x559eba78de60 2024-04-26T02:00:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30d7e00640 10 client.16737 handle_command_reply: tid=0 2024-04-26T02:00:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30f8413740 1 client.16737 shutdown 2024-04-26T02:00:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30f8413740 1 -- 172.21.15.6:0/2856357857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7732d0 msgr2=0x559eba77b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30f8413740 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7732d0 0x559eba77b0c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f30d800e9c0 tx=0x7f30d800cef0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30f8413740 1 -- 172.21.15.6:0/2856357857 shutdown_connections 2024-04-26T02:00:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30f8413740 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559eba78de60 0x559eba790290 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30f8413740 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7732d0 0x559eba77b0c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30f8413740 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba77db90 0x559eba77b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30f8413740 1 --2- 172.21.15.6:0/2856357857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba77ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30d7e00640 0 client.16737 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30f8413740 1 -- 172.21.15.6:0/2856357857 >> 172.21.15.6:0/2856357857 conn(0x559eba76eec0 msgr2=0x559eba774bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30f8413740 1 -- 172.21.15.6:0/2856357857 shutdown_connections 2024-04-26T02:00:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30f8413740 1 -- 172.21.15.6:0/2856357857 wait complete. 2024-04-26T02:00:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.696+0000 7f30f8413740 20 client.16737 trim_cache size 0 max 16384 2024-04-26T02:00:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.713+0000 7f30f8413740 1 Processor -- start 2024-04-26T02:00:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.713+0000 7f30f8413740 1 -- start start 2024-04-26T02:00:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.714+0000 7f30f8413740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7729b0 0x559eba76e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.714+0000 7f30f8413740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba775370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.714+0000 7f30f8413740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba77db90 0x559eba795b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.714+0000 7f30f8413740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f30f01b26b0 con 0x559eba7729b0 2024-04-26T02:00:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.714+0000 7f30f8413740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f30f01afef0 con 0x559eba77db90 2024-04-26T02:00:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.714+0000 7f30f8413740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f30f01add40 con 0x559eba7732d0 2024-04-26T02:00:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.714+0000 7f30eea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7729b0 0x559eba76e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.714+0000 7f30eea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7729b0 0x559eba76e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43584/0 (socket says 172.21.15.6:43584) 2024-04-26T02:00:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.714+0000 7f30eea00640 1 -- 172.21.15.6:0/3876343794 learned_addr learned my addr 172.21.15.6:0/3876343794 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.714+0000 7f30ef400640 1 --2- 172.21.15.6:0/3876343794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba77db90 0x559eba795b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.714+0000 7f30e6000640 1 --2- 172.21.15.6:0/3876343794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba775370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.715+0000 7f30eea00640 1 -- 172.21.15.6:0/3876343794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 msgr2=0x559eba775370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.715+0000 7f30eea00640 1 --2- 172.21.15.6:0/3876343794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba775370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.715+0000 7f30eea00640 1 -- 172.21.15.6:0/3876343794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba77db90 msgr2=0x559eba795b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.715+0000 7f30eea00640 1 --2- 172.21.15.6:0/3876343794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba77db90 0x559eba795b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.715+0000 7f30eea00640 1 -- 172.21.15.6:0/3876343794 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559eba773c60 con 0x559eba7729b0 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.715+0000 7f30e6000640 1 --2- 172.21.15.6:0/3876343794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba775370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.715+0000 7f30eea00640 1 --2- 172.21.15.6:0/3876343794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7729b0 0x559eba76e7a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x559eba790b00 tx=0x7f30e001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139da02454834dd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.715+0000 7f30dea00640 1 -- 172.21.15.6:0/3876343794 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e0030070 con 0x559eba7729b0 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.715+0000 7f30dea00640 1 -- 172.21.15.6:0/3876343794 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30e00043d0 con 0x559eba7729b0 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.715+0000 7f30dea00640 1 -- 172.21.15.6:0/3876343794 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e0003850 con 0x559eba7729b0 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.716+0000 7f30f8413740 1 -- 172.21.15.6:0/3876343794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7729b0 msgr2=0x559eba76e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.716+0000 7f30f8413740 1 --2- 172.21.15.6:0/3876343794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7729b0 0x559eba76e7a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x559eba790b00 tx=0x7f30e001a000 comp rx=0 tx=0).stop 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.716+0000 7f30f8413740 1 -- 172.21.15.6:0/3876343794 shutdown_connections 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.716+0000 7f30f8413740 1 --2- 172.21.15.6:0/3876343794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba77db90 0x559eba795b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.716+0000 7f30f8413740 1 --2- 172.21.15.6:0/3876343794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba775370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.716+0000 7f30f8413740 1 --2- 172.21.15.6:0/3876343794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba7729b0 0x559eba76e7a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.716+0000 7f30f8413740 1 -- 172.21.15.6:0/3876343794 >> 172.21.15.6:0/3876343794 conn(0x559eba76eec0 msgr2=0x559eba77cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.716+0000 7f30f8413740 1 -- 172.21.15.6:0/3876343794 shutdown_connections 2024-04-26T02:00:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.716+0000 7f30f8413740 1 -- 172.21.15.6:0/3876343794 wait complete. 2024-04-26T02:00:32.718 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.717+0000 7f30f8413740 1 -- start start 2024-04-26T02:00:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.717+0000 7f30f8413740 1 Processor -- start 2024-04-26T02:00:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.717+0000 7f30f8413740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba785e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.717+0000 7f30f8413740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba7831a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.717+0000 7f30f8413740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 0x559eba783720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.717+0000 7f30f8413740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f30f01ae3b0 con 0x559eba77db90 2024-04-26T02:00:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.717+0000 7f30f8413740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f30f01af820 con 0x559eba7729b0 2024-04-26T02:00:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.717+0000 7f30f8413740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f30f01aea80 con 0x559eba7732d0 2024-04-26T02:00:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.717+0000 7f30f8413740 10 client.0 _ll_register_callbacks cb 0x7f30d809c810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.718+0000 7f30ef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 0x559eba783720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.718+0000 7f30ef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 0x559eba783720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43600/0 (socket says 172.21.15.6:43600) 2024-04-26T02:00:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.718+0000 7f30ef400640 1 -- 172.21.15.6:0/2394122693 learned_addr learned my addr 172.21.15.6:0/2394122693 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.718+0000 7f30eea00640 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba785e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.718+0000 7f30e6000640 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba7831a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.718+0000 7f30ef400640 1 -- 172.21.15.6:0/2394122693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 msgr2=0x559eba7831a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.718+0000 7f30ef400640 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba7831a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.718+0000 7f30ef400640 1 -- 172.21.15.6:0/2394122693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 msgr2=0x559eba785e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.718+0000 7f30ef400640 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba785e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.718+0000 7f30ef400640 1 -- 172.21.15.6:0/2394122693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559eba783f50 con 0x559eba77db90 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.718+0000 7f30eea00640 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba785e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.719+0000 7f30ef400640 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 0x559eba783720 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009bf0 tx=0x7f30e8009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437373fa93ba01b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.719+0000 7f30d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.719+0000 7f30d7e00640 1 -- 172.21.15.6:0/2394122693 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8055070 con 0x559eba77db90 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.719+0000 7f30d7e00640 1 -- 172.21.15.6:0/2394122693 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7f30e8031cf0 con 0x559eba77db90 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.719+0000 7f30f8413740 1 -- 172.21.15.6:0/2394122693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30e00043d0 con 0x559eba77db90 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.719+0000 7f30d7e00640 1 -- 172.21.15.6:0/2394122693 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f30e804a040 con 0x559eba77db90 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.719+0000 7f30f8413740 10 client.16743 fetch_fsmap learned FSMap version 48 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.719+0000 7f30f8413740 1 -- 172.21.15.6:0/2394122693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x559eba7f4490 con 0x559eba77db90 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.719+0000 7f30d7e00640 1 -- 172.21.15.6:0/2394122693 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30e00043d0 con 0x559eba77db90 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.720+0000 7f30d7e00640 1 -- 172.21.15.6:0/2394122693 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7f30e8043070 con 0x559eba77db90 2024-04-26T02:00:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.720+0000 7f30f8413740 10 client.16743 fetch_fsmap finished waiting for FSMap version 48 2024-04-26T02:00:32.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.720+0000 7f30f8413740 10 client.16743 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:00:32.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.720+0000 7f30f8413740 20 client.16743 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:32.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.720+0000 7f30f8413740 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559eba7f49d0 0x559eba7f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:32.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.720+0000 7f30f8413740 4 client.16743 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:00:32.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.720+0000 7f30f8413740 1 -- 172.21.15.6:0/2394122693 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559eba6e6320 con 0x559eba7f49d0 2024-04-26T02:00:32.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.720+0000 7f30eea00640 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559eba7f49d0 0x559eba7f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:32.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.721+0000 7f30eea00640 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559eba7f49d0 0x559eba7f6df0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=deae26d3fc392a2f server_cookie=8d9b4a1965d0d4ab in_seq=0 out_seq=0 2024-04-26T02:00:32.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.721+0000 7f30d7e00640 10 client.16743 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.796+0000 7f30d7e00640 1 -- 172.21.15.6:0/2394122693 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559eba6e6320 con 0x559eba7f49d0 2024-04-26T02:00:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.796+0000 7f30d7e00640 10 client.16743 handle_command_reply: tid=0 2024-04-26T02:00:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.796+0000 7f30f8413740 1 client.16743 shutdown 2024-04-26T02:00:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.797+0000 7f30f8413740 1 -- 172.21.15.6:0/2394122693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 msgr2=0x559eba783720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.797+0000 7f30f8413740 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 0x559eba783720 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009bf0 tx=0x7f30e8009540 comp rx=0 tx=0).stop 2024-04-26T02:00:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.797+0000 7f30f8413740 1 -- 172.21.15.6:0/2394122693 shutdown_connections 2024-04-26T02:00:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.797+0000 7f30f8413740 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x559eba7f49d0 0x559eba7f6df0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.797+0000 7f30f8413740 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x559eba77db90 0x559eba783720 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.797+0000 7f30f8413740 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559eba7732d0 0x559eba7831a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.797+0000 7f30f8413740 1 --2- 172.21.15.6:0/2394122693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559eba7729b0 0x559eba785e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.797+0000 7f30f8413740 1 -- 172.21.15.6:0/2394122693 >> 172.21.15.6:0/2394122693 conn(0x559eba76eec0 msgr2=0x559eba77b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.797+0000 7f30d7e00640 0 client.16743 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.797+0000 7f30f8413740 1 -- 172.21.15.6:0/2394122693 shutdown_connections 2024-04-26T02:00:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.797+0000 7f30f8413740 1 -- 172.21.15.6:0/2394122693 wait complete. 2024-04-26T02:00:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.797+0000 7f30f8413740 20 client.16743 trim_cache size 0 max 16384 2024-04-26T02:00:32.797 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:00:32.797 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:00:32.797 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:00:32.797 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:00:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.798+0000 7f30efe00640 1 -- 172.21.15.6:0/3227540023 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f30d00a3d00 msgr2=0x7f30d00a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.798+0000 7f30efe00640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f30d00a3d00 0x7f30d00a61c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x559eba773f50 tx=0x7f30e0005f90 comp rx=0 tx=0).stop 2024-04-26T02:00:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.798+0000 7f30efe00640 1 -- 172.21.15.6:0/3227540023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30f010a560 msgr2=0x7f30f01a0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.798+0000 7f30efe00640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30f010a560 0x7f30f01a0ab0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f30d800e9f0 tx=0x7f30d800eeb0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.798+0000 7f30efe00640 1 -- 172.21.15.6:0/3227540023 shutdown_connections 2024-04-26T02:00:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.798+0000 7f30efe00640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f30d00a3d00 0x7f30d00a61c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.798+0000 7f30efe00640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30f010a560 0x7f30f01a0ab0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.798+0000 7f30efe00640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30f019a680 0x7f30f019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.798+0000 7f30efe00640 1 --2- 172.21.15.6:0/3227540023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f01008e0 0x7f30f01a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.798+0000 7f30efe00640 1 -- 172.21.15.6:0/3227540023 >> 172.21.15.6:0/3227540023 conn(0x7f30f0075f90 msgr2=0x7f30f0107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.799+0000 7f30efe00640 1 -- 172.21.15.6:0/3227540023 shutdown_connections 2024-04-26T02:00:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:32.799+0000 7f30efe00640 1 -- 172.21.15.6:0/3227540023 wait complete. 2024-04-26T02:00:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:32 smithi006 ceph-mon[28419]: pgmap v464: 97 pgs: 97 active+clean; 390 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 4.2 MiB/s wr, 176 op/s 2024-04-26T02:00:32.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:32 smithi188 ceph-mon[32751]: pgmap v464: 97 pgs: 97 active+clean; 390 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 4.2 MiB/s wr, 176 op/s 2024-04-26T02:00:32.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:32 smithi029 ceph-mon[32775]: pgmap v464: 97 pgs: 97 active+clean; 390 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 4.2 MiB/s wr, 176 op/s 2024-04-26T02:00:33.094 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11183 13.97 MB/sec warmup 56 sec latency 32583.344 ms 2024-04-26T02:00:33.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.893+0000 7f8230600700 1 -- 172.21.15.6:0/2367325181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82281034b0 msgr2=0x7f822810f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:33.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.893+0000 7f8230600700 1 --2- 172.21.15.6:0/2367325181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82281034b0 0x7f822810f920 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f822000b0d0 tx=0x7f822000b3e0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.893+0000 7f8230600700 1 -- 172.21.15.6:0/2367325181 shutdown_connections 2024-04-26T02:00:33.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.893+0000 7f8230600700 1 --2- 172.21.15.6:0/2367325181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82281034b0 0x7f822810f920 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.893+0000 7f8230600700 1 --2- 172.21.15.6:0/2367325181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8228107940 0x7f8228107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.893+0000 7f8230600700 1 --2- 172.21.15.6:0/2367325181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8228102af0 0x7f8228102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.893+0000 7f8230600700 1 -- 172.21.15.6:0/2367325181 >> 172.21.15.6:0/2367325181 conn(0x7f8228079ba0 msgr2=0x7f8228079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:33.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.893+0000 7f8230600700 1 -- 172.21.15.6:0/2367325181 shutdown_connections 2024-04-26T02:00:33.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.893+0000 7f8230600700 1 -- 172.21.15.6:0/2367325181 wait complete. 2024-04-26T02:00:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.895+0000 7f8230600700 1 Processor -- start 2024-04-26T02:00:33.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.895+0000 7f8230600700 1 -- start start 2024-04-26T02:00:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.896+0000 7f8230600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8228102af0 0x7f822819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.896+0000 7f8230600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82281034b0 0x7f822819b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.896+0000 7f8230600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228107940 0x7f822819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.896+0000 7f8230600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8228112ca0 con 0x7f82281034b0 2024-04-26T02:00:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.896+0000 7f8230600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8228112b20 con 0x7f8228107940 2024-04-26T02:00:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.896+0000 7f8230600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8228112e20 con 0x7f8228102af0 2024-04-26T02:00:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.896+0000 7f822e600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228107940 0x7f822819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.896+0000 7f822e600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228107940 0x7f822819fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:42946/0 (socket says 172.21.15.6:42946) 2024-04-26T02:00:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.896+0000 7f822e600700 1 -- 172.21.15.6:0/3331704583 learned_addr learned my addr 172.21.15.6:0/3331704583 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.896+0000 7f822d200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82281034b0 0x7f822819b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.897+0000 7f822dc00700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8228102af0 0x7f822819b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.897+0000 7f822e600700 1 -- 172.21.15.6:0/3331704583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8228102af0 msgr2=0x7f822819b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.897+0000 7f822e600700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8228102af0 0x7f822819b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.897+0000 7f822e600700 1 -- 172.21.15.6:0/3331704583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82281034b0 msgr2=0x7f822819b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.897+0000 7f822e600700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82281034b0 0x7f822819b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.897+0000 7f822e600700 1 -- 172.21.15.6:0/3331704583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82281a03c0 con 0x7f8228107940 2024-04-26T02:00:33.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.897+0000 7f822e600700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228107940 0x7f822819fc80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f82200079a0 tx=0x7f82200079d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:33.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.898+0000 7f8226000700 1 -- 172.21.15.6:0/3331704583 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822001f070 con 0x7f8228107940 2024-04-26T02:00:33.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.898+0000 7f8230600700 1 -- 172.21.15.6:0/3331704583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82281a0650 con 0x7f8228107940 2024-04-26T02:00:33.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.898+0000 7f8226000700 1 -- 172.21.15.6:0/3331704583 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8220003c00 con 0x7f8228107940 2024-04-26T02:00:33.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.898+0000 7f8226000700 1 -- 172.21.15.6:0/3331704583 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822001b430 con 0x7f8228107940 2024-04-26T02:00:33.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.898+0000 7f8230600700 1 -- 172.21.15.6:0/3331704583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82281a7f30 con 0x7f8228107940 2024-04-26T02:00:33.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.901+0000 7f8226000700 1 -- 172.21.15.6:0/3331704583 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f8220022070 con 0x7f8228107940 2024-04-26T02:00:33.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.903+0000 7f8226000700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f82100a3e30 0x7f82100a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.903+0000 7f823aec3b80 1 Processor -- start 2024-04-26T02:00:33.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.903+0000 7f8226000700 1 -- 172.21.15.6:0/3331704583 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7f822000f020 con 0x7f8228107940 2024-04-26T02:00:33.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.903+0000 7f823aec3b80 1 -- start start 2024-04-26T02:00:33.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.903+0000 7f822dc00700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f82100a3e30 0x7f82100a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.903+0000 7f823aec3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92e1920 0x5599a92e1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.903+0000 7f823aec3b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e2240 0x5599a92ecbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.903+0000 7f823aec3b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92ed0f0 0x5599a92ef4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f823aec3b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5599a9234f90 con 0x5599a92ed0f0 2024-04-26T02:00:33.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f823aec3b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5599a91db830 con 0x5599a92e2240 2024-04-26T02:00:33.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f823aec3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5599a9230f50 con 0x5599a92e1920 2024-04-26T02:00:33.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f822e600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92ed0f0 0x5599a92ef4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f822e600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92ed0f0 0x5599a92ef4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43630/0 (socket says 172.21.15.6:43630) 2024-04-26T02:00:33.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f822e600700 1 -- 172.21.15.6:0/367882032 learned_addr learned my addr 172.21.15.6:0/367882032 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:33.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f822dc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92e1920 0x5599a92e1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f822d200700 1 --2- 172.21.15.6:0/367882032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e2240 0x5599a92ecbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f822dc00700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f82100a3e30 0x7f82100a62f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f821c009c30 tx=0x7f821c009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:33.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f822e600700 1 -- 172.21.15.6:0/367882032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92e1920 msgr2=0x5599a92e1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:33.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f822e600700 1 --2- 172.21.15.6:0/367882032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92e1920 0x5599a92e1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f822e600700 1 -- 172.21.15.6:0/367882032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e2240 msgr2=0x5599a92ecbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:33.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f822e600700 1 --2- 172.21.15.6:0/367882032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e2240 0x5599a92ecbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.904+0000 7f822e600700 1 -- 172.21.15.6:0/367882032 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599a91d1780 con 0x5599a92ed0f0 2024-04-26T02:00:33.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.905+0000 7f822e600700 1 --2- 172.21.15.6:0/367882032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92ed0f0 0x5599a92ef4e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8220099b60 tx=0x7f82200055b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f0955b29f9146f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:33.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.905+0000 7f821aa00700 1 -- 172.21.15.6:0/367882032 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8220020260 con 0x5599a92ed0f0 2024-04-26T02:00:33.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.905+0000 7f821aa00700 1 -- 172.21.15.6:0/367882032 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f822009de00 con 0x5599a92ed0f0 2024-04-26T02:00:33.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.905+0000 7f821aa00700 1 -- 172.21.15.6:0/367882032 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822009ba00 con 0x5599a92ed0f0 2024-04-26T02:00:33.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.906+0000 7f823aec3b80 1 -- 172.21.15.6:0/367882032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92ed0f0 msgr2=0x5599a92ef4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:33.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.906+0000 7f823aec3b80 1 --2- 172.21.15.6:0/367882032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92ed0f0 0x5599a92ef4e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8220099b60 tx=0x7f82200055b0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.906+0000 7f823aec3b80 1 -- 172.21.15.6:0/367882032 shutdown_connections 2024-04-26T02:00:33.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.906+0000 7f823aec3b80 1 --2- 172.21.15.6:0/367882032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92ed0f0 0x5599a92ef4e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.906+0000 7f823aec3b80 1 --2- 172.21.15.6:0/367882032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92e1920 0x5599a92e1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.906+0000 7f823aec3b80 1 --2- 172.21.15.6:0/367882032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e2240 0x5599a92ecbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.906+0000 7f823aec3b80 1 -- 172.21.15.6:0/367882032 >> 172.21.15.6:0/367882032 conn(0x5599a92106a0 msgr2=0x5599a92dddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:33.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.906+0000 7f823aec3b80 1 -- 172.21.15.6:0/367882032 shutdown_connections 2024-04-26T02:00:33.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.906+0000 7f823aec3b80 1 -- 172.21.15.6:0/367882032 wait complete. 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.907+0000 7f823aec3b80 1 -- start start 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.907+0000 7f823aec3b80 1 Processor -- start 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.907+0000 7f823aec3b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e1920 0x5599a92ec770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.907+0000 7f823aec3b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e2240 0x5599a92e9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.907+0000 7f823aec3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92ed0f0 0x5599a92ea050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.907+0000 7f823aec3b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5599a92efa20 con 0x5599a92e1920 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.907+0000 7f823aec3b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5599a91ee960 con 0x5599a92e2240 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.907+0000 7f823aec3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5599a92efba0 con 0x5599a92ed0f0 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.908+0000 7f823aec3b80 10 client.0 _ll_register_callbacks cb 0x5599a91cd820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.908+0000 7f822dc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e1920 0x5599a92ec770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.908+0000 7f822dc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e1920 0x5599a92ec770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43636/0 (socket says 172.21.15.6:43636) 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.908+0000 7f822dc00700 1 -- 172.21.15.6:0/3882463348 learned_addr learned my addr 172.21.15.6:0/3882463348 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.908+0000 7f822d200700 1 --2- 172.21.15.6:0/3882463348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e2240 0x5599a92e9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.908+0000 7f822dc00700 1 -- 172.21.15.6:0/3882463348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92ed0f0 msgr2=0x5599a92ea050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:33.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.908+0000 7f822dc00700 1 --2- 172.21.15.6:0/3882463348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92ed0f0 0x5599a92ea050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.908+0000 7f822dc00700 1 -- 172.21.15.6:0/3882463348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e2240 msgr2=0x5599a92e9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:33.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.908+0000 7f822dc00700 1 --2- 172.21.15.6:0/3882463348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e2240 0x5599a92e9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.908+0000 7f822dc00700 1 -- 172.21.15.6:0/3882463348 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599a91829d0 con 0x5599a92e1920 2024-04-26T02:00:33.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.909+0000 7f822dc00700 1 --2- 172.21.15.6:0/3882463348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e1920 0x5599a92ec770 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f821c01c5a0 tx=0x7f821c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52cf8a72240c06d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:33.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.909+0000 7f820f400700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:00:33.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.909+0000 7f820f400700 1 -- 172.21.15.6:0/3882463348 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c004830 con 0x5599a92e1920 2024-04-26T02:00:33.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.909+0000 7f820f400700 1 -- 172.21.15.6:0/3882463348 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7f821c022070 con 0x5599a92e1920 2024-04-26T02:00:33.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.909+0000 7f823aec3b80 1 -- 172.21.15.6:0/3882463348 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f822009de00 con 0x5599a92e1920 2024-04-26T02:00:33.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.910+0000 7f820f400700 1 -- 172.21.15.6:0/3882463348 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f821c0049d0 con 0x5599a92e1920 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.910+0000 7f823aec3b80 10 client.16755 fetch_fsmap learned FSMap version 48 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.910+0000 7f823aec3b80 1 -- 172.21.15.6:0/3882463348 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5599a91cbbf0 con 0x5599a92e1920 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.910+0000 7f820f400700 1 -- 172.21.15.6:0/3882463348 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f822009de00 con 0x5599a92e1920 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.910+0000 7f820f400700 1 -- 172.21.15.6:0/3882463348 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7f821c01a050 con 0x5599a92e1920 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.910+0000 7f823aec3b80 10 client.16755 fetch_fsmap finished waiting for FSMap version 48 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.910+0000 7f823aec3b80 10 client.16755 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.910+0000 7f823aec3b80 20 client.16755 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.910+0000 7f823aec3b80 1 --2- 172.21.15.6:0/3882463348 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5599a92fc710 0x5599a92feb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.910+0000 7f823aec3b80 4 client.16755 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.910+0000 7f823aec3b80 1 -- 172.21.15.6:0/3882463348 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599a9242780 con 0x5599a92fc710 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.911+0000 7f822d200700 1 --2- 172.21.15.6:0/3882463348 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5599a92fc710 0x5599a92feb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.911+0000 7f822d200700 1 --2- 172.21.15.6:0/3882463348 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5599a92fc710 0x5599a92feb40 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12ad5cfd9cb2ee96 server_cookie=6427f965682e752c in_seq=0 out_seq=0 2024-04-26T02:00:33.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.911+0000 7f820f400700 10 client.16755 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:33.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.912+0000 7f820f400700 1 -- 172.21.15.6:0/3882463348 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5599a9242780 con 0x5599a92fc710 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.912+0000 7f820f400700 10 client.16755 handle_command_reply: tid=0 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.912+0000 7f823aec3b80 1 client.16755 shutdown 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.913+0000 7f823aec3b80 1 -- 172.21.15.6:0/3882463348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e1920 msgr2=0x5599a92ec770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.913+0000 7f823aec3b80 1 --2- 172.21.15.6:0/3882463348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e1920 0x5599a92ec770 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f821c01c5a0 tx=0x7f821c0047c0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.913+0000 7f823aec3b80 1 -- 172.21.15.6:0/3882463348 shutdown_connections 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.913+0000 7f823aec3b80 1 --2- 172.21.15.6:0/3882463348 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5599a92fc710 0x5599a92feb40 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.913+0000 7f823aec3b80 1 --2- 172.21.15.6:0/3882463348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92ed0f0 0x5599a92ea050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.913+0000 7f823aec3b80 1 --2- 172.21.15.6:0/3882463348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e1920 0x5599a92ec770 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.913+0000 7f823aec3b80 1 --2- 172.21.15.6:0/3882463348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e2240 0x5599a92e9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.913+0000 7f823aec3b80 1 -- 172.21.15.6:0/3882463348 >> 172.21.15.6:0/3882463348 conn(0x5599a92106a0 msgr2=0x5599a92dd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.913+0000 7f820f400700 0 client.16755 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.913+0000 7f823aec3b80 1 -- 172.21.15.6:0/3882463348 shutdown_connections 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.913+0000 7f823aec3b80 1 -- 172.21.15.6:0/3882463348 wait complete. 2024-04-26T02:00:33.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.913+0000 7f823aec3b80 20 client.16755 trim_cache size 0 max 16384 2024-04-26T02:00:33.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.935+0000 7f823aec3b80 1 Processor -- start 2024-04-26T02:00:33.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.935+0000 7f823aec3b80 1 -- start start 2024-04-26T02:00:33.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.936+0000 7f823aec3b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e1920 0x5599a92e8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.936+0000 7f823aec3b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e2240 0x5599a9304410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.936+0000 7f823aec3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92ed0f0 0x5599a9304950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.936+0000 7f823aec3b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f82281b1e20 con 0x5599a92e2240 2024-04-26T02:00:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.936+0000 7f823aec3b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f82281af620 con 0x5599a92e1920 2024-04-26T02:00:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.936+0000 7f823aec3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f82281ad470 con 0x5599a92ed0f0 2024-04-26T02:00:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.936+0000 7f822dc00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e1920 0x5599a92e8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.936+0000 7f822e600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92ed0f0 0x5599a9304950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.936+0000 7f822dc00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e1920 0x5599a92e8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:42984/0 (socket says 172.21.15.6:42984) 2024-04-26T02:00:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.936+0000 7f822dc00700 1 -- 172.21.15.6:0/2441457884 learned_addr learned my addr 172.21.15.6:0/2441457884 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.936+0000 7f822d200700 1 --2- 172.21.15.6:0/2441457884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e2240 0x5599a9304410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.937+0000 7f822dc00700 1 -- 172.21.15.6:0/2441457884 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92ed0f0 msgr2=0x5599a9304950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:33.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.937+0000 7f822dc00700 1 --2- 172.21.15.6:0/2441457884 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92ed0f0 0x5599a9304950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.937+0000 7f822dc00700 1 -- 172.21.15.6:0/2441457884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e2240 msgr2=0x5599a9304410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:33.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.937+0000 7f822dc00700 1 --2- 172.21.15.6:0/2441457884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e2240 0x5599a9304410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.937+0000 7f822dc00700 1 -- 172.21.15.6:0/2441457884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599a92eaa70 con 0x5599a92e1920 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.937+0000 7f822dc00700 1 --2- 172.21.15.6:0/2441457884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e1920 0x5599a92e8a70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f821c00f620 tx=0x7f821c009350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a999ff3a54212fec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.937+0000 7f821a000700 1 -- 172.21.15.6:0/2441457884 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c01cc40 con 0x5599a92e1920 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.937+0000 7f821a000700 1 -- 172.21.15.6:0/2441457884 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f821c0044d0 con 0x5599a92e1920 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.937+0000 7f821a000700 1 -- 172.21.15.6:0/2441457884 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c003680 con 0x5599a92e1920 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.938+0000 7f823aec3b80 1 -- 172.21.15.6:0/2441457884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e1920 msgr2=0x5599a92e8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.938+0000 7f823aec3b80 1 --2- 172.21.15.6:0/2441457884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e1920 0x5599a92e8a70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f821c00f620 tx=0x7f821c009350 comp rx=0 tx=0).stop 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.938+0000 7f823aec3b80 1 -- 172.21.15.6:0/2441457884 shutdown_connections 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.938+0000 7f823aec3b80 1 --2- 172.21.15.6:0/2441457884 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92ed0f0 0x5599a9304950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.938+0000 7f823aec3b80 1 --2- 172.21.15.6:0/2441457884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92e1920 0x5599a92e8a70 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.938+0000 7f823aec3b80 1 --2- 172.21.15.6:0/2441457884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e2240 0x5599a9304410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.938+0000 7f823aec3b80 1 -- 172.21.15.6:0/2441457884 >> 172.21.15.6:0/2441457884 conn(0x5599a92106a0 msgr2=0x5599a92eb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.938+0000 7f823aec3b80 1 -- 172.21.15.6:0/2441457884 shutdown_connections 2024-04-26T02:00:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.938+0000 7f823aec3b80 1 -- 172.21.15.6:0/2441457884 wait complete. 2024-04-26T02:00:33.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.939+0000 7f823aec3b80 1 -- start start 2024-04-26T02:00:33.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.939+0000 7f823aec3b80 1 Processor -- start 2024-04-26T02:00:33.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f823aec3b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e1920 0x5599a92f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f823aec3b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92e2240 0x5599a92f2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f823aec3b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92ed0f0 0x5599a92f2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f823aec3b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f82281adae0 con 0x5599a92e1920 2024-04-26T02:00:33.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f823aec3b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f82281aef50 con 0x5599a92ed0f0 2024-04-26T02:00:33.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f823aec3b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f82281ae1b0 con 0x5599a92e2240 2024-04-26T02:00:33.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f823aec3b80 10 client.0 _ll_register_callbacks cb 0x5599a91cd820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f822e600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92ed0f0 0x5599a92f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f822d200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92e2240 0x5599a92f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f822dc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e1920 0x5599a92f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f822d200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92e2240 0x5599a92f2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44302/0 (socket says 172.21.15.6:44302) 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f822d200700 1 -- 172.21.15.6:0/3814332125 learned_addr learned my addr 172.21.15.6:0/3814332125 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.940+0000 7f822e600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92ed0f0 0x5599a92f2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:42992/0 (socket says 172.21.15.6:42992) 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.941+0000 7f822d200700 1 -- 172.21.15.6:0/3814332125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92ed0f0 msgr2=0x5599a92f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.941+0000 7f822d200700 1 --2- 172.21.15.6:0/3814332125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92ed0f0 0x5599a92f2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.941+0000 7f822d200700 1 -- 172.21.15.6:0/3814332125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e1920 msgr2=0x5599a92f1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.941+0000 7f822d200700 1 --2- 172.21.15.6:0/3814332125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e1920 0x5599a92f1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.941+0000 7f822d200700 1 -- 172.21.15.6:0/3814332125 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599a9362cc0 con 0x5599a92e2240 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.941+0000 7f822e600700 1 --2- 172.21.15.6:0/3814332125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92ed0f0 0x5599a92f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.941+0000 7f822d200700 1 --2- 172.21.15.6:0/3814332125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92e2240 0x5599a92f2300 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x5599a92ff3b0 tx=0x7f82140094b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2923b078279f182f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:33.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.941+0000 7f820f400700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:00:33.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.941+0000 7f820f400700 1 -- 172.21.15.6:0/3814332125 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8214004440 con 0x5599a92e2240 2024-04-26T02:00:33.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f823aec3b80 1 -- 172.21.15.6:0/3814332125 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f821c0044d0 con 0x5599a92e2240 2024-04-26T02:00:33.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f820f400700 1 -- 172.21.15.6:0/3814332125 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7f8214003680 con 0x5599a92e2240 2024-04-26T02:00:33.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f820f400700 1 -- 172.21.15.6:0/3814332125 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f8214003880 con 0x5599a92e2240 2024-04-26T02:00:33.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f823aec3b80 10 client.26974 fetch_fsmap learned FSMap version 48 2024-04-26T02:00:33.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f823aec3b80 1 -- 172.21.15.6:0/3814332125 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5599a9364040 con 0x5599a92e2240 2024-04-26T02:00:33.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f820f400700 1 -- 172.21.15.6:0/3814332125 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f821c0044d0 con 0x5599a92e2240 2024-04-26T02:00:33.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f820f400700 1 -- 172.21.15.6:0/3814332125 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7f8214010a10 con 0x5599a92e2240 2024-04-26T02:00:33.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f823aec3b80 10 client.26974 fetch_fsmap finished waiting for FSMap version 48 2024-04-26T02:00:33.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f823aec3b80 10 client.26974 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:00:33.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f823aec3b80 20 client.26974 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:33.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f823aec3b80 1 --2- 172.21.15.6:0/3814332125 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5599a93647e0 0x5599a9366c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:33.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f823aec3b80 4 client.26974 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c6386e05", "release": true, "await": true}] 2024-04-26T02:00:33.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.942+0000 7f823aec3b80 1 -- 172.21.15.6:0/3814332125 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c6386e05", "release": true, "await": true}) -- 0x5599a923c820 con 0x5599a93647e0 2024-04-26T02:00:33.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.943+0000 7f822dc00700 1 --2- 172.21.15.6:0/3814332125 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5599a93647e0 0x5599a9366c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:33.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.943+0000 7f822dc00700 1 --2- 172.21.15.6:0/3814332125 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5599a93647e0 0x5599a9366c00 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c40f95cca64c5a5 server_cookie=a61e262a72417f5f in_seq=0 out_seq=0 2024-04-26T02:00:33.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.943+0000 7f820f400700 10 client.26974 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:33.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.968+0000 7f820f400700 1 -- 172.21.15.6:0/3814332125 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5599a923c820 con 0x5599a93647e0 2024-04-26T02:00:33.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.968+0000 7f820f400700 10 client.26974 handle_command_reply: tid=0 2024-04-26T02:00:33.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.968+0000 7f823aec3b80 1 client.26974 shutdown 2024-04-26T02:00:33.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.969+0000 7f823aec3b80 1 -- 172.21.15.6:0/3814332125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92e2240 msgr2=0x5599a92f2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:33.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.969+0000 7f823aec3b80 1 --2- 172.21.15.6:0/3814332125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92e2240 0x5599a92f2300 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x5599a92ff3b0 tx=0x7f82140094b0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.969+0000 7f823aec3b80 1 -- 172.21.15.6:0/3814332125 shutdown_connections 2024-04-26T02:00:33.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.969+0000 7f823aec3b80 1 --2- 172.21.15.6:0/3814332125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599a92ed0f0 0x5599a92f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.969+0000 7f823aec3b80 1 --2- 172.21.15.6:0/3814332125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5599a92e1920 0x5599a92f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.969+0000 7f823aec3b80 1 --2- 172.21.15.6:0/3814332125 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5599a93647e0 0x5599a9366c00 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.969+0000 7f823aec3b80 1 --2- 172.21.15.6:0/3814332125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5599a92e2240 0x5599a92f2300 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.969+0000 7f823aec3b80 1 -- 172.21.15.6:0/3814332125 >> 172.21.15.6:0/3814332125 conn(0x5599a92106a0 msgr2=0x5599a92dcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:33.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.969+0000 7f820f400700 0 client.26974 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:33.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.969+0000 7f823aec3b80 1 -- 172.21.15.6:0/3814332125 shutdown_connections 2024-04-26T02:00:33.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.969+0000 7f823aec3b80 1 -- 172.21.15.6:0/3814332125 wait complete. 2024-04-26T02:00:33.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.969+0000 7f823aec3b80 20 client.26974 trim_cache size 0 max 16384 2024-04-26T02:00:33.968 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:00:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 48, 2024-04-26T02:00:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 44, 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "c6386e05": { 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 44, 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 84.6, 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T02:00:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:00:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T02:00:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:00:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:33.970 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:00:33.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.970+0000 7f8230600700 1 -- 172.21.15.6:0/3331704583 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f82100a3e30 msgr2=0x7f82100a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:33.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.970+0000 7f8230600700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f82100a3e30 0x7f82100a62f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f821c009c30 tx=0x7f821c009260 comp rx=0 tx=0).stop 2024-04-26T02:00:33.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.970+0000 7f8230600700 1 -- 172.21.15.6:0/3331704583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228107940 msgr2=0x7f822819fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:33.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.970+0000 7f8230600700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228107940 0x7f822819fc80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f82200079a0 tx=0x7f82200079d0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.970+0000 7f8230600700 1 -- 172.21.15.6:0/3331704583 shutdown_connections 2024-04-26T02:00:33.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.970+0000 7f8230600700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f82100a3e30 0x7f82100a62f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.970+0000 7f8230600700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8228107940 0x7f822819fc80 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.970+0000 7f8230600700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8228102af0 0x7f822819b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.970+0000 7f8230600700 1 --2- 172.21.15.6:0/3331704583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82281034b0 0x7f822819b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:33.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.970+0000 7f8230600700 1 -- 172.21.15.6:0/3331704583 >> 172.21.15.6:0/3331704583 conn(0x7f8228079ba0 msgr2=0x7f8228100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:33.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.971+0000 7f8230600700 1 -- 172.21.15.6:0/3331704583 shutdown_connections 2024-04-26T02:00:33.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:33.971+0000 7f8230600700 1 -- 172.21.15.6:0/3331704583 wait complete. 2024-04-26T02:00:34.094 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11219 13.75 MB/sec warmup 57 sec latency 33489.331 ms 2024-04-26T02:00:34.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:34 smithi006 ceph-mon[28419]: pgmap v465: 97 pgs: 97 active+clean; 390 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-04-26T02:00:34.854 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c6386e05', seconds elapsed: 41.5 2024-04-26T02:00:34.854 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.6 2024-04-26T02:00:34.854 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:00:34.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:34 smithi188 ceph-mon[32751]: pgmap v465: 97 pgs: 97 active+clean; 390 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-04-26T02:00:34.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:34 smithi029 ceph-mon[32775]: pgmap v465: 97 pgs: 97 active+clean; 390 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-04-26T02:00:35.094 INFO:tasks.workunit.client.0.smithi006.stdout: 10 11795 14.03 MB/sec warmup 58 sec latency 35.026 ms 2024-04-26T02:00:35.425 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:00:36.095 INFO:tasks.workunit.client.0.smithi006.stdout: 10 12351 14.24 MB/sec warmup 59 sec latency 104.700 ms 2024-04-26T02:00:36.532 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:36 smithi006 ceph-mon[28419]: pgmap v466: 97 pgs: 97 active+clean; 390 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-04-26T02:00:36.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:36 smithi188 ceph-mon[32751]: pgmap v466: 97 pgs: 97 active+clean; 390 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-04-26T02:00:36.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:36 smithi029 ceph-mon[32775]: pgmap v466: 97 pgs: 97 active+clean; 390 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-04-26T02:00:37.094 INFO:tasks.workunit.client.0.smithi006.stdout: 10 13139 14.60 MB/sec warmup 60 sec latency 91.151 ms 2024-04-26T02:00:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.986+0000 7f1e86000700 1 -- 172.21.15.6:0/3600727954 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7800bbb0 con 0x7f1e88105890 2024-04-26T02:00:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.987+0000 7f1e8e800700 1 -- 172.21.15.6:0/3600727954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e88105890 msgr2=0x7f1e881098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.987+0000 7f1e8e800700 1 --2- 172.21.15.6:0/3600727954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e88105890 0x7f1e881098e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1e78009990 tx=0x7f1e78009ca0 comp rx=0 tx=0).stop 2024-04-26T02:00:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.987+0000 7f1e8e800700 1 -- 172.21.15.6:0/3600727954 shutdown_connections 2024-04-26T02:00:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.987+0000 7f1e8e800700 1 --2- 172.21.15.6:0/3600727954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e88109fc0 0x7f1e88111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.987+0000 7f1e8e800700 1 --2- 172.21.15.6:0/3600727954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e88104ee0 0x7f1e881052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.987+0000 7f1e8e800700 1 --2- 172.21.15.6:0/3600727954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e88105890 0x7f1e881098e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.987+0000 7f1e8e800700 1 -- 172.21.15.6:0/3600727954 >> 172.21.15.6:0/3600727954 conn(0x7f1e88100770 msgr2=0x7f1e88102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.988+0000 7f1e8e800700 1 -- 172.21.15.6:0/3600727954 shutdown_connections 2024-04-26T02:00:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.988+0000 7f1e8e800700 1 -- 172.21.15.6:0/3600727954 wait complete. 2024-04-26T02:00:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.989+0000 7f1e8e800700 1 Processor -- start 2024-04-26T02:00:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.989+0000 7f1e8e800700 1 -- start start 2024-04-26T02:00:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e8e800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e88104ee0 0x7f1e8819b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e8e800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e88105890 0x7f1e8819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e8e800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e88109fc0 0x7f1e8819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e8e800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1e88114e50 con 0x7f1e88109fc0 2024-04-26T02:00:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e8e800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1e88114cd0 con 0x7f1e88105890 2024-04-26T02:00:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e8e800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1e88114fd0 con 0x7f1e88104ee0 2024-04-26T02:00:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e87400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e88104ee0 0x7f1e8819b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e87400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e88104ee0 0x7f1e8819b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50188/0 (socket says 172.21.15.6:50188) 2024-04-26T02:00:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e87400700 1 -- 172.21.15.6:0/2711798459 learned_addr learned my addr 172.21.15.6:0/2711798459 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e86a00700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e88105890 0x7f1e8819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e87e00700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e88109fc0 0x7f1e8819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e87400700 1 -- 172.21.15.6:0/2711798459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e88105890 msgr2=0x7f1e8819b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.990+0000 7f1e87400700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e88105890 0x7f1e8819b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.991+0000 7f1e87400700 1 -- 172.21.15.6:0/2711798459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e88109fc0 msgr2=0x7f1e8819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.991+0000 7f1e87400700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e88109fc0 0x7f1e8819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.991+0000 7f1e87400700 1 -- 172.21.15.6:0/2711798459 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e881a0320 con 0x7f1e88104ee0 2024-04-26T02:00:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.991+0000 7f1e86a00700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e88105890 0x7f1e8819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.991+0000 7f1e87e00700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e88109fc0 0x7f1e8819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.991+0000 7f1e87400700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e88104ee0 0x7f1e8819b360 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1e7400e9c0 tx=0x7f1e7400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.992+0000 7f1e7fe00700 1 -- 172.21.15.6:0/2711798459 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7400cbd0 con 0x7f1e88104ee0 2024-04-26T02:00:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.992+0000 7f1e7fe00700 1 -- 172.21.15.6:0/2711798459 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e7400cd70 con 0x7f1e88104ee0 2024-04-26T02:00:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.992+0000 7f1e7fe00700 1 -- 172.21.15.6:0/2711798459 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e74005610 con 0x7f1e88104ee0 2024-04-26T02:00:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.992+0000 7f1e8e800700 1 -- 172.21.15.6:0/2711798459 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e881a0610 con 0x7f1e88104ee0 2024-04-26T02:00:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.992+0000 7f1e8e800700 1 -- 172.21.15.6:0/2711798459 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e881a7e90 con 0x7f1e88104ee0 2024-04-26T02:00:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.994+0000 7f1e8e800700 1 -- 172.21.15.6:0/2711798459 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e88069730 con 0x7f1e88104ee0 2024-04-26T02:00:37.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.996+0000 7f1e7fe00700 1 -- 172.21.15.6:0/2711798459 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f1e74004750 con 0x7f1e88104ee0 2024-04-26T02:00:37.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.998+0000 7f1e7fe00700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1e6c0a3eb0 0x7f1e6c0a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:37.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.998+0000 7f1e7fe00700 1 -- 172.21.15.6:0/2711798459 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7f1e740d0b40 con 0x7f1e88104ee0 2024-04-26T02:00:37.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.998+0000 7f1e86a00700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1e6c0a3eb0 0x7f1e6c0a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:37.999+0000 7f1e86a00700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1e6c0a3eb0 0x7f1e6c0a6370 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e78000c00 tx=0x7f1e78005960 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.004+0000 7f1e7fe00700 1 -- 172.21.15.6:0/2711798459 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e7409d160 con 0x7f1e88104ee0 2024-04-26T02:00:38.094 INFO:tasks.workunit.client.0.smithi006.stdout: 10 14034 15.22 MB/sec warmup 61 sec latency 37.631 ms 2024-04-26T02:00:38.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.393+0000 7f1e8e800700 1 -- 172.21.15.6:0/2711798459 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e8819c670 con 0x7f1e88104ee0 2024-04-26T02:00:38.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.394+0000 7f1e7fe00700 1 -- 172.21.15.6:0/2711798459 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11855 (secure 0 0 0) 0x7f1e7409ca60 con 0x7f1e88104ee0 2024-04-26T02:00:38.394 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:00:38.394 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T02:00:20.436443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26935":{"gid":26935,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1750728695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1750728695},{"type":"v1","addr":"172.21.15.188:6835","nonce":1750728695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24512,24505]},"id":1}]} 2024-04-26T02:00:38.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.397+0000 7f1e8e800700 1 -- 172.21.15.6:0/2711798459 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1e6c0a3eb0 msgr2=0x7f1e6c0a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:38.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.397+0000 7f1e8e800700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1e6c0a3eb0 0x7f1e6c0a6370 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e78000c00 tx=0x7f1e78005960 comp rx=0 tx=0).stop 2024-04-26T02:00:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.397+0000 7f1e8e800700 1 -- 172.21.15.6:0/2711798459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e88104ee0 msgr2=0x7f1e8819b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.397+0000 7f1e8e800700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e88104ee0 0x7f1e8819b360 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1e7400e9c0 tx=0x7f1e7400ecd0 comp rx=0 tx=0).stop 2024-04-26T02:00:38.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.398+0000 7f1e8e800700 1 -- 172.21.15.6:0/2711798459 shutdown_connections 2024-04-26T02:00:38.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.398+0000 7f1e8e800700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1e6c0a3eb0 0x7f1e6c0a6370 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:38.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.398+0000 7f1e8e800700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e88109fc0 0x7f1e8819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:38.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.398+0000 7f1e8e800700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e88104ee0 0x7f1e8819b360 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:38.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.398+0000 7f1e8e800700 1 --2- 172.21.15.6:0/2711798459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e88105890 0x7f1e8819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:38.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.399+0000 7f1e8e800700 1 -- 172.21.15.6:0/2711798459 >> 172.21.15.6:0/2711798459 conn(0x7f1e88100770 msgr2=0x7f1e88102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.399+0000 7f1e8e800700 1 -- 172.21.15.6:0/2711798459 shutdown_connections 2024-04-26T02:00:38.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:38.399+0000 7f1e8e800700 1 -- 172.21.15.6:0/2711798459 wait complete. 2024-04-26T02:00:38.400 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 48 2024-04-26T02:00:38.802 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:38 smithi006 ceph-mon[28419]: pgmap v467: 97 pgs: 97 active+clean; 388 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 258 op/s 2024-04-26T02:00:38.802 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:38 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2711798459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:38 smithi188 ceph-mon[32751]: pgmap v467: 97 pgs: 97 active+clean; 388 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 258 op/s 2024-04-26T02:00:38.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:38 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2711798459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:38.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:38 smithi029 ceph-mon[32775]: pgmap v467: 97 pgs: 97 active+clean; 388 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 258 op/s 2024-04-26T02:00:38.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:38 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2711798459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:39.095 INFO:tasks.workunit.client.0.smithi006.stdout: 10 14859 15.82 MB/sec warmup 62 sec latency 44.406 ms 2024-04-26T02:00:39.328 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 84.6 --await-for 42.0' 2024-04-26T02:00:39.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 84.6 --await-for 42.0 2024-04-26T02:00:39.883 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:00:40.095 INFO:tasks.workunit.client.0.smithi006.stdout: 10 15384 16.08 MB/sec warmup 63 sec latency 77.486 ms 2024-04-26T02:00:40.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:40 smithi006 ceph-mon[28419]: pgmap v468: 97 pgs: 97 active+clean; 388 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 208 op/s 2024-04-26T02:00:40.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:40 smithi188 ceph-mon[32751]: pgmap v468: 97 pgs: 97 active+clean; 388 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 208 op/s 2024-04-26T02:00:40.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:40 smithi029 ceph-mon[32775]: pgmap v468: 97 pgs: 97 active+clean; 388 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 208 op/s 2024-04-26T02:00:41.095 INFO:tasks.workunit.client.0.smithi006.stdout: 10 15942 16.09 MB/sec warmup 64 sec latency 45.983 ms 2024-04-26T02:00:42.095 INFO:tasks.workunit.client.0.smithi006.stdout: 10 16766 16.56 MB/sec warmup 65 sec latency 46.559 ms 2024-04-26T02:00:42.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.332+0000 7f708e800700 1 -- 172.21.15.6:0/1110907070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7088107930 msgr2=0x7f7088107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:42.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.332+0000 7f708e800700 1 --2- 172.21.15.6:0/1110907070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7088107930 0x7f7088107d10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f708000b0a0 tx=0x7f708000b3b0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.333+0000 7f708e800700 1 -- 172.21.15.6:0/1110907070 shutdown_connections 2024-04-26T02:00:42.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.333+0000 7f708e800700 1 --2- 172.21.15.6:0/1110907070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7088102300 0x7f708810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.333+0000 7f708e800700 1 --2- 172.21.15.6:0/1110907070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7088107930 0x7f7088107d10 secure :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f708000b0a0 tx=0x7f708000b3b0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.333+0000 7f708e800700 1 --2- 172.21.15.6:0/1110907070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7088101940 0x7f7088101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.333+0000 7f708e800700 1 -- 172.21.15.6:0/1110907070 >> 172.21.15.6:0/1110907070 conn(0x7f70880788e0 msgr2=0x7f7088078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:42.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.334+0000 7f708e800700 1 -- 172.21.15.6:0/1110907070 shutdown_connections 2024-04-26T02:00:42.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.334+0000 7f708e800700 1 -- 172.21.15.6:0/1110907070 wait complete. 2024-04-26T02:00:42.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.336+0000 7f708e800700 1 Processor -- start 2024-04-26T02:00:42.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.338+0000 7f708e800700 1 -- start start 2024-04-26T02:00:42.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.339+0000 7f708e800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7088101940 0x7f7088074730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.339+0000 7f708e800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7088102300 0x7f70880717e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.339+0000 7f708e800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7088071e30 0x7f70881b0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.339+0000 7f708e800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f708806cd20 con 0x7f7088102300 2024-04-26T02:00:42.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.339+0000 7f708e800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f708806cba0 con 0x7f7088101940 2024-04-26T02:00:42.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.339+0000 7f708e800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f708806cea0 con 0x7f7088071e30 2024-04-26T02:00:42.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.339+0000 7f7087e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7088071e30 0x7f70881b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.340+0000 7f7087e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7088071e30 0x7f70881b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50208/0 (socket says 172.21.15.6:50208) 2024-04-26T02:00:42.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.340+0000 7f7087e00700 1 -- 172.21.15.6:0/1533386506 learned_addr learned my addr 172.21.15.6:0/1533386506 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:42.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.340+0000 7f7087400700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7088101940 0x7f7088074730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.340+0000 7f7086a00700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7088102300 0x7f70880717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.340+0000 7f7087400700 1 -- 172.21.15.6:0/1533386506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7088071e30 msgr2=0x7f70881b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:42.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.340+0000 7f7087400700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7088071e30 0x7f70881b0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.340+0000 7f7087400700 1 -- 172.21.15.6:0/1533386506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7088102300 msgr2=0x7f70880717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:42.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.341+0000 7f7087400700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7088102300 0x7f70880717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.341+0000 7f7087400700 1 -- 172.21.15.6:0/1533386506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7088108790 con 0x7f7088101940 2024-04-26T02:00:42.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.341+0000 7f7086a00700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7088102300 0x7f70880717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:00:42.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.341+0000 7f7087400700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7088101940 0x7f7088074730 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7080008000 tx=0x7f70800078a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:42.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.341+0000 7f707fe00700 1 -- 172.21.15.6:0/1533386506 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080021030 con 0x7f7088101940 2024-04-26T02:00:42.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.342+0000 7f707fe00700 1 -- 172.21.15.6:0/1533386506 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7080007d90 con 0x7f7088101940 2024-04-26T02:00:42.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.342+0000 7f707fe00700 1 -- 172.21.15.6:0/1533386506 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080005040 con 0x7f7088101940 2024-04-26T02:00:42.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.342+0000 7f708e800700 1 -- 172.21.15.6:0/1533386506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7088108a20 con 0x7f7088101940 2024-04-26T02:00:42.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.342+0000 7f708e800700 1 -- 172.21.15.6:0/1533386506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7088108ee0 con 0x7f7088101940 2024-04-26T02:00:42.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.346+0000 7f707fe00700 1 -- 172.21.15.6:0/1533386506 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f7080024070 con 0x7f7088101940 2024-04-26T02:00:42.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.348+0000 7f707fe00700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f706c0a3ed0 0x7f706c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.349+0000 7f707fe00700 1 -- 172.21.15.6:0/1533386506 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7f70800d1570 con 0x7f7088101940 2024-04-26T02:00:42.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.349+0000 7f7086a00700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f706c0a3ed0 0x7f706c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.351+0000 7f7086a00700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f706c0a3ed0 0x7f706c0a6390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7070006fd0 tx=0x7f7070008040 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:42.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.354+0000 7f709910ab80 1 Processor -- start 2024-04-26T02:00:42.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.354+0000 7f709910ab80 1 -- start start 2024-04-26T02:00:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.355+0000 7f709910ab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d1211b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.355+0000 7f709910ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d1216f0 0x558f5d12c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.355+0000 7f709910ab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d12c560 0x558f5d12e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.355+0000 7f709910ab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x558f5d074020 con 0x558f5d12c560 2024-04-26T02:00:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.355+0000 7f709910ab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558f5d01a8c0 con 0x558f5d120dd0 2024-04-26T02:00:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.355+0000 7f709910ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558f5d06ffe0 con 0x558f5d1216f0 2024-04-26T02:00:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.355+0000 7f7087400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d1211b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.355+0000 7f7087400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d1211b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:58768/0 (socket says 172.21.15.6:58768) 2024-04-26T02:00:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.355+0000 7f7087400700 1 -- 172.21.15.6:0/2858726373 learned_addr learned my addr 172.21.15.6:0/2858726373 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.355+0000 7f7087e00700 1 --2- 172.21.15.6:0/2858726373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d12c560 0x558f5d12e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.356+0000 7f7086a00700 1 --2- 172.21.15.6:0/2858726373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d1216f0 0x558f5d12c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.356+0000 7f7086a00700 1 -- 172.21.15.6:0/2858726373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 msgr2=0x558f5d1211b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.356+0000 7f7086a00700 1 --2- 172.21.15.6:0/2858726373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d1211b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.356+0000 7f7086a00700 1 -- 172.21.15.6:0/2858726373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d12c560 msgr2=0x558f5d12e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.356+0000 7f7086a00700 1 --2- 172.21.15.6:0/2858726373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d12c560 0x558f5d12e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.356+0000 7f7086a00700 1 -- 172.21.15.6:0/2858726373 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f5d010810 con 0x558f5d1216f0 2024-04-26T02:00:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.356+0000 7f7087e00700 1 --2- 172.21.15.6:0/2858726373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d12c560 0x558f5d12e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.356+0000 7f7086a00700 1 --2- 172.21.15.6:0/2858726373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d1216f0 0x558f5d12c020 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7088073030 tx=0x7f707000f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34d0bcb094e595fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.357+0000 7f707cc00700 1 -- 172.21.15.6:0/2858726373 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070019040 con 0x558f5d1216f0 2024-04-26T02:00:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.357+0000 7f707cc00700 1 -- 172.21.15.6:0/2858726373 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70700044d0 con 0x558f5d1216f0 2024-04-26T02:00:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.357+0000 7f707cc00700 1 -- 172.21.15.6:0/2858726373 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070013660 con 0x558f5d1216f0 2024-04-26T02:00:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.358+0000 7f709910ab80 1 -- 172.21.15.6:0/2858726373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d1216f0 msgr2=0x558f5d12c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.358+0000 7f709910ab80 1 --2- 172.21.15.6:0/2858726373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d1216f0 0x558f5d12c020 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7088073030 tx=0x7f707000f000 comp rx=0 tx=0).stop 2024-04-26T02:00:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.358+0000 7f709910ab80 1 -- 172.21.15.6:0/2858726373 shutdown_connections 2024-04-26T02:00:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.358+0000 7f709910ab80 1 --2- 172.21.15.6:0/2858726373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d12c560 0x558f5d12e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.358+0000 7f709910ab80 1 --2- 172.21.15.6:0/2858726373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d1211b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.358+0000 7f709910ab80 1 --2- 172.21.15.6:0/2858726373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d1216f0 0x558f5d12c020 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.358+0000 7f709910ab80 1 -- 172.21.15.6:0/2858726373 >> 172.21.15.6:0/2858726373 conn(0x558f5d0500e0 msgr2=0x558f5d11d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.358+0000 7f709910ab80 1 -- 172.21.15.6:0/2858726373 shutdown_connections 2024-04-26T02:00:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.358+0000 7f709910ab80 1 -- 172.21.15.6:0/2858726373 wait complete. 2024-04-26T02:00:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.359+0000 7f709910ab80 1 -- start start 2024-04-26T02:00:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.359+0000 7f709910ab80 1 Processor -- start 2024-04-26T02:00:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f709910ab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d12bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f709910ab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d1216f0 0x558f5d128f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f709910ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 0x558f5d1294d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f709910ab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x558f5d12eff0 con 0x558f5d1216f0 2024-04-26T02:00:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f709910ab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558f5d02d9f0 con 0x558f5d120dd0 2024-04-26T02:00:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f709910ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558f5d12f170 con 0x558f5d12c560 2024-04-26T02:00:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f709910ab80 10 client.0 _ll_register_callbacks cb 0x558f5d00c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f7087e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 0x558f5d1294d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f7087e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 0x558f5d1294d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50234/0 (socket says 172.21.15.6:50234) 2024-04-26T02:00:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f7087e00700 1 -- 172.21.15.6:0/1774777540 learned_addr learned my addr 172.21.15.6:0/1774777540 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f7086a00700 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d1216f0 0x558f5d128f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f7087400700 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d12bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f7087e00700 1 -- 172.21.15.6:0/1774777540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 msgr2=0x558f5d12bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:42.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.360+0000 7f7087e00700 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d12bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.361+0000 7f7087e00700 1 -- 172.21.15.6:0/1774777540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d1216f0 msgr2=0x558f5d128f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.361+0000 7f7087e00700 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d1216f0 0x558f5d128f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.361+0000 7f7087e00700 1 -- 172.21.15.6:0/1774777540 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f5cfc1a60 con 0x558f5d12c560 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.361+0000 7f7086a00700 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d1216f0 0x558f5d128f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.361+0000 7f7087e00700 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 0x558f5d1294d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7078011a60 tx=0x7f7078011d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c08f580303a243e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.361+0000 7f7075600700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.361+0000 7f7075600700 1 -- 172.21.15.6:0/1774777540 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707801e060 con 0x558f5d12c560 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.362+0000 7f7075600700 1 -- 172.21.15.6:0/1774777540 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7f707800fcc0 con 0x558f5d12c560 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.362+0000 7f709910ab80 1 -- 172.21.15.6:0/1774777540 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70700044d0 con 0x558f5d12c560 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.362+0000 7f7075600700 1 -- 172.21.15.6:0/1774777540 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f7078022430 con 0x558f5d12c560 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.362+0000 7f709910ab80 10 client.26989 fetch_fsmap learned FSMap version 48 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.362+0000 7f709910ab80 1 -- 172.21.15.6:0/1774777540 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x558f5d00ac80 con 0x558f5d12c560 2024-04-26T02:00:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.363+0000 7f7075600700 1 -- 172.21.15.6:0/1774777540 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70700044d0 con 0x558f5d12c560 2024-04-26T02:00:42.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.363+0000 7f7075600700 1 -- 172.21.15.6:0/1774777540 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7f7078018030 con 0x558f5d12c560 2024-04-26T02:00:42.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.363+0000 7f709910ab80 10 client.26989 fetch_fsmap finished waiting for FSMap version 48 2024-04-26T02:00:42.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.363+0000 7f709910ab80 10 client.26989 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:00:42.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.363+0000 7f709910ab80 20 client.26989 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:42.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.363+0000 7f709910ab80 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558f5d13bda0 0x558f5d13e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.364+0000 7f709910ab80 4 client.26989 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:42.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.364+0000 7f709910ab80 1 -- 172.21.15.6:0/1774777540 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f5d081810 con 0x558f5d13bda0 2024-04-26T02:00:42.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.364+0000 7f7087400700 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558f5d13bda0 0x558f5d13e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.364+0000 7f7087400700 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558f5d13bda0 0x558f5d13e190 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a2791908eeecad0 server_cookie=89f81776d5bac4d1 in_seq=0 out_seq=0 2024-04-26T02:00:42.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.364+0000 7f7075600700 10 client.26989 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.367+0000 7f7075600700 1 -- 172.21.15.6:0/1774777540 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f5d081810 con 0x558f5d13bda0 2024-04-26T02:00:42.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.367+0000 7f7075600700 10 client.26989 handle_command_reply: tid=0 2024-04-26T02:00:42.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.367+0000 7f709910ab80 1 client.26989 shutdown 2024-04-26T02:00:42.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.367+0000 7f709910ab80 1 -- 172.21.15.6:0/1774777540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 msgr2=0x558f5d1294d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.367+0000 7f709910ab80 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 0x558f5d1294d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7078011a60 tx=0x7f7078011d70 comp rx=0 tx=0).stop 2024-04-26T02:00:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.368+0000 7f709910ab80 1 -- 172.21.15.6:0/1774777540 shutdown_connections 2024-04-26T02:00:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.368+0000 7f709910ab80 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558f5d13bda0 0x558f5d13e190 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.368+0000 7f709910ab80 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 0x558f5d1294d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.368+0000 7f709910ab80 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d12bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.368+0000 7f709910ab80 1 --2- 172.21.15.6:0/1774777540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d1216f0 0x558f5d128f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.368+0000 7f709910ab80 1 -- 172.21.15.6:0/1774777540 >> 172.21.15.6:0/1774777540 conn(0x558f5d0500e0 msgr2=0x558f5d11ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.368+0000 7f7075600700 0 client.26989 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.368+0000 7f709910ab80 1 -- 172.21.15.6:0/1774777540 shutdown_connections 2024-04-26T02:00:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.369+0000 7f709910ab80 1 -- 172.21.15.6:0/1774777540 wait complete. 2024-04-26T02:00:42.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.370+0000 7f709910ab80 20 client.26989 trim_cache size 0 max 16384 2024-04-26T02:00:42.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.401+0000 7f709910ab80 1 Processor -- start 2024-04-26T02:00:42.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.401+0000 7f709910ab80 1 -- start start 2024-04-26T02:00:42.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.402+0000 7f709910ab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d143a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.402+0000 7f709910ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d1216f0 0x558f5d143f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.402+0000 7f709910ab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d12c560 0x558f5d134420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.402+0000 7f709910ab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f70881bf000 con 0x558f5d12c560 2024-04-26T02:00:42.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.402+0000 7f709910ab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70881bc800 con 0x558f5d120dd0 2024-04-26T02:00:42.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.402+0000 7f709910ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f708810c900 con 0x558f5d1216f0 2024-04-26T02:00:42.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.402+0000 7f7087400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.402+0000 7f7086a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d1216f0 0x558f5d143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.402+0000 7f7087400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d143a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:58784/0 (socket says 172.21.15.6:58784) 2024-04-26T02:00:42.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.402+0000 7f7087400700 1 -- 172.21.15.6:0/736864927 learned_addr learned my addr 172.21.15.6:0/736864927 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:42.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.402+0000 7f7087e00700 1 --2- 172.21.15.6:0/736864927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d12c560 0x558f5d134420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.403+0000 7f7086a00700 1 -- 172.21.15.6:0/736864927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 msgr2=0x558f5d143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:42.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.403+0000 7f7086a00700 1 --2- 172.21.15.6:0/736864927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d143a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.403+0000 7f7086a00700 1 -- 172.21.15.6:0/736864927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d12c560 msgr2=0x558f5d134420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:42.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.403+0000 7f7086a00700 1 --2- 172.21.15.6:0/736864927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d12c560 0x558f5d134420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.403+0000 7f7086a00700 1 -- 172.21.15.6:0/736864927 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f5d122c40 con 0x558f5d1216f0 2024-04-26T02:00:42.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.403+0000 7f7086a00700 1 --2- 172.21.15.6:0/736864927 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d1216f0 0x558f5d143f70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x558f5d12a560 tx=0x7f707000a720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46a30f805c124a0d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:42.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.403+0000 7f7077e00700 1 -- 172.21.15.6:0/736864927 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707002b030 con 0x558f5d1216f0 2024-04-26T02:00:42.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.404+0000 7f7077e00700 1 -- 172.21.15.6:0/736864927 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7070003fa0 con 0x558f5d1216f0 2024-04-26T02:00:42.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.404+0000 7f7077e00700 1 -- 172.21.15.6:0/736864927 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070027440 con 0x558f5d1216f0 2024-04-26T02:00:42.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.405+0000 7f709910ab80 1 -- 172.21.15.6:0/736864927 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d1216f0 msgr2=0x558f5d143f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:42.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.405+0000 7f709910ab80 1 --2- 172.21.15.6:0/736864927 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d1216f0 0x558f5d143f70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x558f5d12a560 tx=0x7f707000a720 comp rx=0 tx=0).stop 2024-04-26T02:00:42.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.405+0000 7f709910ab80 1 -- 172.21.15.6:0/736864927 shutdown_connections 2024-04-26T02:00:42.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.405+0000 7f709910ab80 1 --2- 172.21.15.6:0/736864927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d12c560 0x558f5d134420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.405+0000 7f709910ab80 1 --2- 172.21.15.6:0/736864927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d120dd0 0x558f5d143a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.405+0000 7f709910ab80 1 --2- 172.21.15.6:0/736864927 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d1216f0 0x558f5d143f70 secure :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0x558f5d12a560 tx=0x7f707000a720 comp rx=0 tx=0).stop 2024-04-26T02:00:42.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.405+0000 7f709910ab80 1 -- 172.21.15.6:0/736864927 >> 172.21.15.6:0/736864927 conn(0x558f5d0500e0 msgr2=0x558f5d12b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:42.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.405+0000 7f709910ab80 1 -- 172.21.15.6:0/736864927 shutdown_connections 2024-04-26T02:00:42.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.405+0000 7f709910ab80 1 -- 172.21.15.6:0/736864927 wait complete. 2024-04-26T02:00:42.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.406+0000 7f709910ab80 1 -- start start 2024-04-26T02:00:42.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.406+0000 7f709910ab80 1 Processor -- start 2024-04-26T02:00:42.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.407+0000 7f709910ab80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d120dd0 0x558f5d134810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.407+0000 7f709910ab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 0x558f5d130b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.407+0000 7f709910ab80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d131040 0x558f5d1314c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.407+0000 7f709910ab80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f70881bacc0 con 0x558f5d120dd0 2024-04-26T02:00:42.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.407+0000 7f709910ab80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70881bc130 con 0x558f5d131040 2024-04-26T02:00:42.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.407+0000 7f709910ab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f70881bb390 con 0x558f5d12c560 2024-04-26T02:00:42.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.407+0000 7f709910ab80 10 client.0 _ll_register_callbacks cb 0x558f5d00c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:42.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.407+0000 7f7087e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d131040 0x558f5d1314c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.407+0000 7f7087e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d131040 0x558f5d1314c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:58788/0 (socket says 172.21.15.6:58788) 2024-04-26T02:00:42.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.407+0000 7f7087e00700 1 -- 172.21.15.6:0/3869825732 learned_addr learned my addr 172.21.15.6:0/3869825732 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:42.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.408+0000 7f7086a00700 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 0x558f5d130b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.408+0000 7f7087e00700 1 -- 172.21.15.6:0/3869825732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 msgr2=0x558f5d130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:42.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.408+0000 7f7087e00700 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 0x558f5d130b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.408+0000 7f7087e00700 1 -- 172.21.15.6:0/3869825732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d120dd0 msgr2=0x558f5d134810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:00:42.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.408+0000 7f7087e00700 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d120dd0 0x558f5d134810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.408+0000 7f7087e00700 1 -- 172.21.15.6:0/3869825732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f5d121d70 con 0x558f5d131040 2024-04-26T02:00:42.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.409+0000 7f7087e00700 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d131040 0x558f5d1314c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7078000c00 tx=0x7f707800c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2867622895211b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:42.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.409+0000 7f7086a00700 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 0x558f5d130b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:42.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.409+0000 7f7075600700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:00:42.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.409+0000 7f7075600700 1 -- 172.21.15.6:0/3869825732 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7078005c50 con 0x558f5d131040 2024-04-26T02:00:42.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.409+0000 7f7075600700 1 -- 172.21.15.6:0/3869825732 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7f7078004020 con 0x558f5d131040 2024-04-26T02:00:42.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.409+0000 7f709910ab80 1 -- 172.21.15.6:0/3869825732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7070003fa0 con 0x558f5d131040 2024-04-26T02:00:42.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.410+0000 7f7075600700 1 -- 172.21.15.6:0/3869825732 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f707801e030 con 0x558f5d131040 2024-04-26T02:00:42.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.412+0000 7f709910ab80 10 client.26945 fetch_fsmap learned FSMap version 48 2024-04-26T02:00:42.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.412+0000 7f709910ab80 1 -- 172.21.15.6:0/3869825732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x558f5d1a34f0 con 0x558f5d131040 2024-04-26T02:00:42.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.412+0000 7f7075600700 1 -- 172.21.15.6:0/3869825732 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7070003fa0 con 0x558f5d131040 2024-04-26T02:00:42.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.412+0000 7f7075600700 1 -- 172.21.15.6:0/3869825732 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7f7078018040 con 0x558f5d131040 2024-04-26T02:00:42.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.412+0000 7f709910ab80 10 client.26945 fetch_fsmap finished waiting for FSMap version 48 2024-04-26T02:00:42.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.412+0000 7f709910ab80 10 client.26945 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:00:42.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.412+0000 7f709910ab80 20 client.26945 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:42.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.412+0000 7f709910ab80 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558f5d1a3ca0 0x558f5d1a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:42.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.412+0000 7f709910ab80 4 client.26945 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.6, "await_for": 42.0}] 2024-04-26T02:00:42.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.412+0000 7f709910ab80 1 -- 172.21.15.6:0/3869825732 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.6, "await_for": 42.0}) -- 0x558f5d07b8b0 con 0x558f5d1a3ca0 2024-04-26T02:00:42.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.413+0000 7f7087400700 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558f5d1a3ca0 0x558f5d1a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:42.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.414+0000 7f7087400700 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558f5d1a3ca0 0x558f5d1a60e0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44ccc4faabfe6d14 server_cookie=d387eb00328e991d in_seq=0 out_seq=0 2024-04-26T02:00:42.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.414+0000 7f7075600700 10 client.26945 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:42.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:42 smithi006 ceph-mon[28419]: pgmap v469: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.7 MiB/s wr, 429 op/s 2024-04-26T02:00:42.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:42 smithi188 ceph-mon[32751]: pgmap v469: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.7 MiB/s wr, 429 op/s 2024-04-26T02:00:42.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.905+0000 7f7075600700 1 -- 172.21.15.6:0/3869825732 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x558f5d07b8b0 con 0x558f5d1a3ca0 2024-04-26T02:00:42.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.905+0000 7f7075600700 10 client.26945 handle_command_reply: tid=0 2024-04-26T02:00:42.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.905+0000 7f709910ab80 1 client.26945 shutdown 2024-04-26T02:00:42.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.905+0000 7f709910ab80 1 -- 172.21.15.6:0/3869825732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d131040 msgr2=0x558f5d1314c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:42.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.905+0000 7f709910ab80 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d131040 0x558f5d1314c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7078000c00 tx=0x7f707800c7a0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.905+0000 7f709910ab80 1 -- 172.21.15.6:0/3869825732 shutdown_connections 2024-04-26T02:00:42.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.905+0000 7f709910ab80 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f5d131040 0x558f5d1314c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.905+0000 7f709910ab80 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558f5d120dd0 0x558f5d134810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.905+0000 7f709910ab80 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558f5d1a3ca0 0x558f5d1a60e0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.905+0000 7f709910ab80 1 --2- 172.21.15.6:0/3869825732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558f5d12c560 0x558f5d130b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.905+0000 7f7075600700 0 client.26945 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:42.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.905+0000 7f709910ab80 1 -- 172.21.15.6:0/3869825732 >> 172.21.15.6:0/3869825732 conn(0x558f5d0500e0 msgr2=0x558f5d11c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:42.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.906+0000 7f709910ab80 1 -- 172.21.15.6:0/3869825732 shutdown_connections 2024-04-26T02:00:42.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.906+0000 7f709910ab80 1 -- 172.21.15.6:0/3869825732 wait complete. 2024-04-26T02:00:42.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.906+0000 7f709910ab80 20 client.26945 trim_cache size 0 max 16384 2024-04-26T02:00:42.906 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:00:42.906 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 48, 2024-04-26T02:00:42.906 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T02:00:42.906 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 46, 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "a6261f3e": { 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 46, 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 84.6, 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T02:00:42.907 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:00:42.908 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T02:00:42.908 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:00:42.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:42.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:42.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:42.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:42.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:00:42.908 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:00:42.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.907+0000 7f708e800700 1 -- 172.21.15.6:0/1533386506 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f706c0a3ed0 msgr2=0x7f706c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:42.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.907+0000 7f708e800700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f706c0a3ed0 0x7f706c0a6390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7070006fd0 tx=0x7f7070008040 comp rx=0 tx=0).stop 2024-04-26T02:00:42.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.907+0000 7f708e800700 1 -- 172.21.15.6:0/1533386506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7088101940 msgr2=0x7f7088074730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:42.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.907+0000 7f708e800700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7088101940 0x7f7088074730 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7080008000 tx=0x7f70800078a0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.907+0000 7f708e800700 1 -- 172.21.15.6:0/1533386506 shutdown_connections 2024-04-26T02:00:42.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.907+0000 7f708e800700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f706c0a3ed0 0x7f706c0a6390 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.907+0000 7f708e800700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7088071e30 0x7f70881b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.907+0000 7f708e800700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7088101940 0x7f7088074730 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.907+0000 7f708e800700 1 --2- 172.21.15.6:0/1533386506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7088102300 0x7f70880717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:42.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.907+0000 7f708e800700 1 -- 172.21.15.6:0/1533386506 >> 172.21.15.6:0/1533386506 conn(0x7f70880788e0 msgr2=0x7f70880ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:42.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.907+0000 7f708e800700 1 -- 172.21.15.6:0/1533386506 shutdown_connections 2024-04-26T02:00:42.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:00:42.907+0000 7f708e800700 1 -- 172.21.15.6:0/1533386506 wait complete. 2024-04-26T02:00:42.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:42 smithi029 ceph-mon[32775]: pgmap v469: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.7 MiB/s wr, 429 op/s 2024-04-26T02:00:43.097 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 16.49 MB/sec warmup 66 sec latency 641.913 ms 2024-04-26T02:00:43.643 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a6261f3e', quiesce took 8.8 seconds. Will release after: 24.6 2024-04-26T02:00:44.095 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 16.25 MB/sec warmup 67 sec latency 1642.052 ms 2024-04-26T02:00:44.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:44 smithi006 ceph-mon[28419]: pgmap v470: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.7 MiB/s wr, 366 op/s 2024-04-26T02:00:44.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:44 smithi188 ceph-mon[32751]: pgmap v470: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.7 MiB/s wr, 366 op/s 2024-04-26T02:00:44.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:44 smithi029 ceph-mon[32775]: pgmap v470: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.7 MiB/s wr, 366 op/s 2024-04-26T02:00:45.095 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 16.01 MB/sec warmup 68 sec latency 2642.224 ms 2024-04-26T02:00:45.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:45 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:00:45.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:45 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:00:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.946+0000 7ff8b1e00640 1 -- 172.21.15.6:0/5418488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ac074c40 msgr2=0x7ff8ac10e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.946+0000 7ff8b1e00640 1 --2- 172.21.15.6:0/5418488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ac074c40 0x7ff8ac10e2a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff898009ca0 tx=0x7ff89802a950 comp rx=0 tx=0).stop 2024-04-26T02:00:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.946+0000 7ff8b1e00640 1 -- 172.21.15.6:0/5418488 shutdown_connections 2024-04-26T02:00:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.946+0000 7ff8b1e00640 1 --2- 172.21.15.6:0/5418488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac10e7e0 0x7ff8ac110bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.946+0000 7ff8b1e00640 1 --2- 172.21.15.6:0/5418488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ac074c40 0x7ff8ac10e2a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.947 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.946+0000 7ff8b1e00640 1 --2- 172.21.15.6:0/5418488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8ac074320 0x7ff8ac074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.947 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.946+0000 7ff8b1e00640 1 -- 172.21.15.6:0/5418488 >> 172.21.15.6:0/5418488 conn(0x7ff8ac0fda10 msgr2=0x7ff8ac0ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:45.947 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.946+0000 7ff8b1e00640 1 -- 172.21.15.6:0/5418488 shutdown_connections 2024-04-26T02:00:45.947 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.946+0000 7ff8b1e00640 1 -- 172.21.15.6:0/5418488 wait complete. 2024-04-26T02:00:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.947+0000 7ff8b1e00640 1 Processor -- start 2024-04-26T02:00:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.948+0000 7ff8b1e00640 1 -- start start 2024-04-26T02:00:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.948+0000 7ff8b1e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac074320 0x7ff8ac1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.948+0000 7ff8b1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8ac074c40 0x7ff8ac1a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.948+0000 7ff8b1e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ac10e7e0 0x7ff8ac19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.948+0000 7ff8b1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac114460 con 0x7ff8ac074c40 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.948+0000 7ff8b1e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac1142e0 con 0x7ff8ac10e7e0 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.948+0000 7ff8b1e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac1145e0 con 0x7ff8ac074320 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.949+0000 7ff8aaa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac074320 0x7ff8ac1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.949+0000 7ff8aa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8ac074c40 0x7ff8ac1a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.949+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ac10e7e0 0x7ff8ac19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.949+0000 7ff8aaa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac074320 0x7ff8ac1a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50270/0 (socket says 172.21.15.6:50270) 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.949+0000 7ff8aaa00640 1 -- 172.21.15.6:0/1702042748 learned_addr learned my addr 172.21.15.6:0/1702042748 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.949+0000 7ff8aaa00640 1 -- 172.21.15.6:0/1702042748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ac10e7e0 msgr2=0x7ff8ac19a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.949+0000 7ff8aaa00640 1 --2- 172.21.15.6:0/1702042748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ac10e7e0 0x7ff8ac19a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.949+0000 7ff8aaa00640 1 -- 172.21.15.6:0/1702042748 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8ac074c40 msgr2=0x7ff8ac1a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.949+0000 7ff8aaa00640 1 --2- 172.21.15.6:0/1702042748 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8ac074c40 0x7ff8ac1a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.949+0000 7ff8aaa00640 1 -- 172.21.15.6:0/1702042748 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ac19afb0 con 0x7ff8ac074320 2024-04-26T02:00:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.949+0000 7ff8aaa00640 1 --2- 172.21.15.6:0/1702042748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac074320 0x7ff8ac1a0670 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00e9f0 tx=0x7ff89c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.950+0000 7ff8b0a00640 1 -- 172.21.15.6:0/1702042748 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c00cd80 con 0x7ff8ac074320 2024-04-26T02:00:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.950+0000 7ff8b0a00640 1 -- 172.21.15.6:0/1702042748 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff89c0108e0 con 0x7ff8ac074320 2024-04-26T02:00:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.950+0000 7ff8b1e00640 1 -- 172.21.15.6:0/1702042748 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8ac19b2a0 con 0x7ff8ac074320 2024-04-26T02:00:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.950+0000 7ff8b0a00640 1 -- 172.21.15.6:0/1702042748 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c018980 con 0x7ff8ac074320 2024-04-26T02:00:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.950+0000 7ff8b1e00640 1 -- 172.21.15.6:0/1702042748 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8ac1a74c0 con 0x7ff8ac074320 2024-04-26T02:00:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.953+0000 7ff8b4c12740 1 Processor -- start 2024-04-26T02:00:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.953+0000 7ff8b4c12740 1 -- start start 2024-04-26T02:00:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.953+0000 7ff8b4c12740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd72461820 0x55cd72461c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.953+0000 7ff8b4c12740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd7246c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8b4c12740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd7246ca00 0x55cd7246edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8b4c12740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55cd723c7900 con 0x55cd7246ca00 2024-04-26T02:00:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8b4c12740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cd7236def0 con 0x55cd72462140 2024-04-26T02:00:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8b4c12740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55cd723c38c0 con 0x55cd72461820 2024-04-26T02:00:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8b0a00640 1 -- 172.21.15.6:0/1702042748 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7ff89c010430 con 0x7ff8ac074320 2024-04-26T02:00:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd72461820 0x55cd72461c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd72461820 0x55cd72461c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50272/0 (socket says 172.21.15.6:50272) 2024-04-26T02:00:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8ab400640 1 -- 172.21.15.6:0/4043570283 learned_addr learned my addr 172.21.15.6:0/4043570283 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8aa000640 1 --2- 172.21.15.6:0/4043570283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd7246ca00 0x55cd7246edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8aaa00640 1 --2- 172.21.15.6:0/4043570283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd7246c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8ab400640 1 -- 172.21.15.6:0/4043570283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 msgr2=0x55cd7246c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8ab400640 1 --2- 172.21.15.6:0/4043570283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd7246c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8ab400640 1 -- 172.21.15.6:0/4043570283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd7246ca00 msgr2=0x55cd7246edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8ab400640 1 --2- 172.21.15.6:0/4043570283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd7246ca00 0x55cd7246edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8ab400640 1 -- 172.21.15.6:0/4043570283 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd720da210 con 0x55cd72461820 2024-04-26T02:00:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.954+0000 7ff8aaa00640 1 --2- 172.21.15.6:0/4043570283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd7246c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.955+0000 7ff8ab400640 1 --2- 172.21.15.6:0/4043570283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd72461820 0x55cd72461c00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009b80 tx=0x7ff8a4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b0d454ffc52a23 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.955+0000 7ff897e00640 1 -- 172.21.15.6:0/4043570283 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4004030 con 0x55cd72461820 2024-04-26T02:00:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.955+0000 7ff897e00640 1 -- 172.21.15.6:0/4043570283 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8a40035f0 con 0x55cd72461820 2024-04-26T02:00:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.955+0000 7ff897e00640 1 -- 172.21.15.6:0/4043570283 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4019600 con 0x55cd72461820 2024-04-26T02:00:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.956+0000 7ff8b0a00640 1 --2- 172.21.15.6:0/1702042748 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff8900a3d10 0x7ff8900a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.956+0000 7ff8b4c12740 1 -- 172.21.15.6:0/4043570283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd72461820 msgr2=0x55cd72461c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.956+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/4043570283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd72461820 0x55cd72461c00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009b80 tx=0x7ff8a4011810 comp rx=0 tx=0).stop 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.956+0000 7ff8b4c12740 1 -- 172.21.15.6:0/4043570283 shutdown_connections 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.956+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/4043570283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd7246ca00 0x55cd7246edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.956+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/4043570283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd7246c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.956+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/4043570283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd72461820 0x55cd72461c00 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.956+0000 7ff8b4c12740 1 -- 172.21.15.6:0/4043570283 >> 172.21.15.6:0/4043570283 conn(0x55cd7245dd30 msgr2=0x55cd7245e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.956+0000 7ff8b4c12740 1 -- 172.21.15.6:0/4043570283 shutdown_connections 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.956+0000 7ff8aaa00640 1 --2- 172.21.15.6:0/1702042748 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff8900a3d10 0x7ff8900a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.956+0000 7ff8b0a00640 1 -- 172.21.15.6:0/1702042748 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7ff89c0d0b00 con 0x7ff8ac074320 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.956+0000 7ff8b4c12740 1 -- 172.21.15.6:0/4043570283 wait complete. 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.957+0000 7ff8b4c12740 1 -- start start 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.957+0000 7ff8aaa00640 1 --2- 172.21.15.6:0/1702042748 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff8900a3d10 0x7ff8900a61d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff89c0ce2e0 tx=0x7ff89c000bc0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.957+0000 7ff8b4c12740 1 Processor -- start 2024-04-26T02:00:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.957+0000 7ff8b4c12740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd72461820 0x55cd72469960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.957+0000 7ff8b4c12740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd72469ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.957+0000 7ff8b4c12740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd7246ca00 0x55cd7246a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.957+0000 7ff8b4c12740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55cd7246f630 con 0x55cd7246ca00 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.957+0000 7ff8b4c12740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cd72381310 con 0x55cd72462140 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.957+0000 7ff8b4c12740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55cd7246f7b0 con 0x55cd72461820 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.957+0000 7ff8b4c12740 10 client.0 _ll_register_callbacks cb 0x55cd7234fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.958+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd72461820 0x55cd72469960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.958+0000 7ff8aa000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd72469ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.958+0000 7ff8aa000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd72469ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:58824/0 (socket says 172.21.15.6:58824) 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.958+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd7246ca00 0x55cd7246a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.958+0000 7ff8aa000640 1 -- 172.21.15.6:0/2561557463 learned_addr learned my addr 172.21.15.6:0/2561557463 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.958+0000 7ff8aa000640 1 -- 172.21.15.6:0/2561557463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd72461820 msgr2=0x55cd72469960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.958+0000 7ff8aa000640 1 --2- 172.21.15.6:0/2561557463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd72461820 0x55cd72469960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.958+0000 7ff8aa000640 1 -- 172.21.15.6:0/2561557463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd7246ca00 msgr2=0x55cd7246a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.958+0000 7ff8aa000640 1 --2- 172.21.15.6:0/2561557463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd7246ca00 0x55cd7246a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.958+0000 7ff8aa000640 1 -- 172.21.15.6:0/2561557463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd721a6620 con 0x55cd72462140 2024-04-26T02:00:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.959+0000 7ff8aa000640 1 --2- 172.21.15.6:0/2561557463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd72469ea0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff89802ae70 tx=0x7ff898003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d1ea3405865f423 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.959+0000 7ff894c00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:00:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.959+0000 7ff894c00640 1 -- 172.21.15.6:0/2561557463 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898049060 con 0x55cd72462140 2024-04-26T02:00:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.959+0000 7ff8b4c12740 1 -- 172.21.15.6:0/2561557463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8a40035f0 con 0x55cd72462140 2024-04-26T02:00:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.959+0000 7ff894c00640 1 -- 172.21.15.6:0/2561557463 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7ff89802fa70 con 0x55cd72462140 2024-04-26T02:00:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.959+0000 7ff894c00640 1 -- 172.21.15.6:0/2561557463 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ff898034040 con 0x55cd72462140 2024-04-26T02:00:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.959+0000 7ff8b4c12740 10 client.26957 fetch_fsmap learned FSMap version 49 2024-04-26T02:00:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.959+0000 7ff8b4c12740 1 -- 172.21.15.6:0/2561557463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55cd7247b8a0 con 0x55cd72462140 2024-04-26T02:00:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.960+0000 7ff894c00640 1 -- 172.21.15.6:0/2561557463 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8a40035f0 con 0x55cd72462140 2024-04-26T02:00:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.960+0000 7ff894c00640 1 -- 172.21.15.6:0/2561557463 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7ff898030440 con 0x55cd72462140 2024-04-26T02:00:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.960+0000 7ff8b4c12740 10 client.26957 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:00:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.960+0000 7ff8b4c12740 10 client.26957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:00:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.960+0000 7ff8b4c12740 20 client.26957 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.960+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/2561557463 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cd7247cdb0 0x55cd7247f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.960+0000 7ff8b4c12740 4 client.26957 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.960+0000 7ff8b4c12740 1 -- 172.21.15.6:0/2561557463 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd723db8b0 con 0x55cd7247cdb0 2024-04-26T02:00:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.960+0000 7ff8ab400640 1 --2- 172.21.15.6:0/2561557463 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cd7247cdb0 0x55cd7247f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.961+0000 7ff8ab400640 1 --2- 172.21.15.6:0/2561557463 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cd7247cdb0 0x55cd7247f1c0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=953167236811e2b server_cookie=50e1e3e378eaa3d3 in_seq=0 out_seq=0 2024-04-26T02:00:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.961+0000 7ff894c00640 10 client.26957 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff894c00640 1 -- 172.21.15.6:0/2561557463 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd723db8b0 con 0x55cd7247cdb0 2024-04-26T02:00:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff894c00640 10 client.26957 handle_command_reply: tid=0 2024-04-26T02:00:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff8b4c12740 1 client.26957 shutdown 2024-04-26T02:00:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff8b4c12740 1 -- 172.21.15.6:0/2561557463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 msgr2=0x55cd72469ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/2561557463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd72469ea0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff89802ae70 tx=0x7ff898003ba0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff8b4c12740 1 -- 172.21.15.6:0/2561557463 shutdown_connections 2024-04-26T02:00:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/2561557463 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cd7247cdb0 0x55cd7247f1c0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/2561557463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd7246ca00 0x55cd7246a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/2561557463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd72469ea0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/2561557463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd72461820 0x55cd72469960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff894c00640 0 client.26957 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff8b4c12740 1 -- 172.21.15.6:0/2561557463 >> 172.21.15.6:0/2561557463 conn(0x55cd7245dd30 msgr2=0x55cd72463a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.962+0000 7ff8b4c12740 1 -- 172.21.15.6:0/2561557463 shutdown_connections 2024-04-26T02:00:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.963+0000 7ff8b4c12740 1 -- 172.21.15.6:0/2561557463 wait complete. 2024-04-26T02:00:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.963+0000 7ff8b4c12740 20 client.26957 trim_cache size 0 max 16384 2024-04-26T02:00:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.979+0000 7ff8b4c12740 1 Processor -- start 2024-04-26T02:00:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.979+0000 7ff8b4c12740 1 -- start start 2024-04-26T02:00:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8b4c12740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd72461820 0x55cd7246a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8b4c12740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd72484a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8b4c12740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd7246ca00 0x55cd72484fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8b4c12740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac1b2820 con 0x55cd72461820 2024-04-26T02:00:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8b4c12740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac1b0060 con 0x55cd72462140 2024-04-26T02:00:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8b4c12740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac1adeb0 con 0x55cd7246ca00 2024-04-26T02:00:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd72461820 0x55cd7246a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8aa000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd72484a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8aa000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd72484a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:58840/0 (socket says 172.21.15.6:58840) 2024-04-26T02:00:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8aa000640 1 -- 172.21.15.6:0/3473715802 learned_addr learned my addr 172.21.15.6:0/3473715802 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8ab400640 1 --2- 172.21.15.6:0/3473715802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd7246ca00 0x55cd72484fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8aa000640 1 -- 172.21.15.6:0/3473715802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd7246ca00 msgr2=0x55cd72484fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8aa000640 1 --2- 172.21.15.6:0/3473715802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd7246ca00 0x55cd72484fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8aa000640 1 -- 172.21.15.6:0/3473715802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd72461820 msgr2=0x55cd7246a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8aa000640 1 --2- 172.21.15.6:0/3473715802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd72461820 0x55cd7246a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.980+0000 7ff8aa000640 1 -- 172.21.15.6:0/3473715802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd724628d0 con 0x55cd72462140 2024-04-26T02:00:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.981+0000 7ff8aa000640 1 --2- 172.21.15.6:0/3473715802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd72484a90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff898002230 tx=0x7ff89802f780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c38bbc1507afb18 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:45.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.981+0000 7ff897400640 1 -- 172.21.15.6:0/3473715802 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89805a070 con 0x55cd72462140 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.981+0000 7ff897400640 1 -- 172.21.15.6:0/3473715802 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff89802fd20 con 0x55cd72462140 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.981+0000 7ff897400640 1 -- 172.21.15.6:0/3473715802 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898049040 con 0x55cd72462140 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.982+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3473715802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 msgr2=0x55cd72484a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.982+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/3473715802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd72484a90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff898002230 tx=0x7ff89802f780 comp rx=0 tx=0).stop 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.982+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3473715802 shutdown_connections 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.982+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/3473715802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd7246ca00 0x55cd72484fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.982+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/3473715802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72462140 0x55cd72484a90 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.982+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/3473715802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd72461820 0x55cd7246a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.982+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3473715802 >> 172.21.15.6:0/3473715802 conn(0x55cd7245dd30 msgr2=0x55cd7245c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.982+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3473715802 shutdown_connections 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.982+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3473715802 wait complete. 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.982+0000 7ff8b4c12740 1 -- start start 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.982+0000 7ff8b4c12740 1 Processor -- start 2024-04-26T02:00:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.983+0000 7ff8b4c12740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72461820 0x55cd72474cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.983+0000 7ff8b4c12740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd72462140 0x55cd72472050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.983+0000 7ff8b4c12740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd7246ca00 0x55cd72472590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.983+0000 7ff8b4c12740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac1ae520 con 0x55cd72462140 2024-04-26T02:00:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.983+0000 7ff8b4c12740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac1af990 con 0x55cd72461820 2024-04-26T02:00:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.983+0000 7ff8b4c12740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac1aebf0 con 0x55cd7246ca00 2024-04-26T02:00:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.983+0000 7ff8b4c12740 10 client.0 _ll_register_callbacks cb 0x7ff8980098a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.983+0000 7ff8aa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd72462140 0x55cd72472050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.983+0000 7ff8ab400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72461820 0x55cd72474cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.983+0000 7ff8aa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd72462140 0x55cd72472050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52324/0 (socket says 172.21.15.6:52324) 2024-04-26T02:00:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.983+0000 7ff8aa000640 1 -- 172.21.15.6:0/3258532161 learned_addr learned my addr 172.21.15.6:0/3258532161 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.983+0000 7ff8ab400640 1 --2- 172.21.15.6:0/3258532161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd7246ca00 0x55cd72472590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.984+0000 7ff8aa000640 1 -- 172.21.15.6:0/3258532161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd7246ca00 msgr2=0x55cd72472590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.984+0000 7ff8aa000640 1 --2- 172.21.15.6:0/3258532161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd7246ca00 0x55cd72472590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.984+0000 7ff8aa000640 1 -- 172.21.15.6:0/3258532161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72461820 msgr2=0x55cd72474cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.984+0000 7ff8aa000640 1 --2- 172.21.15.6:0/3258532161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72461820 0x55cd72474cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.984+0000 7ff8aa000640 1 -- 172.21.15.6:0/3258532161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd72472c80 con 0x55cd72462140 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.984+0000 7ff8aa000640 1 --2- 172.21.15.6:0/3258532161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd72462140 0x55cd72472050 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff89802f810 tx=0x7ff898000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e5e095345f74d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.984+0000 7ff894c00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.984+0000 7ff894c00640 1 -- 172.21.15.6:0/3258532161 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89805a070 con 0x55cd72462140 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.984+0000 7ff894c00640 1 -- 172.21.15.6:0/3258532161 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7ff898003ea0 con 0x55cd72462140 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.984+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3258532161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff89802fd20 con 0x55cd72462140 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.985+0000 7ff894c00640 1 -- 172.21.15.6:0/3258532161 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ff89802fed0 con 0x55cd72462140 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.985+0000 7ff8b4c12740 10 client.16794 fetch_fsmap learned FSMap version 49 2024-04-26T02:00:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.985+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3258532161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55cd724e3220 con 0x55cd72462140 2024-04-26T02:00:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.985+0000 7ff894c00640 1 -- 172.21.15.6:0/3258532161 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff89802fd20 con 0x55cd72462140 2024-04-26T02:00:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.985+0000 7ff894c00640 1 -- 172.21.15.6:0/3258532161 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7ff89804d530 con 0x55cd72462140 2024-04-26T02:00:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.985+0000 7ff8b4c12740 10 client.16794 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:00:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.985+0000 7ff8b4c12740 10 client.16794 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:00:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.985+0000 7ff8b4c12740 20 client.16794 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.985+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/3258532161 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cd724e3830 0x55cd724e5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.985+0000 7ff8b4c12740 4 client.16794 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:00:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.985+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3258532161 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cd723d5320 con 0x55cd724e3830 2024-04-26T02:00:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.986+0000 7ff8ab400640 1 --2- 172.21.15.6:0/3258532161 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cd724e3830 0x55cd724e5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.986+0000 7ff8ab400640 1 --2- 172.21.15.6:0/3258532161 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cd724e3830 0x55cd724e5c50 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8cc92f73a5a3edb server_cookie=8ca3b486ecb59fd4 in_seq=0 out_seq=0 2024-04-26T02:00:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:45.986+0000 7ff894c00640 10 client.16794 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:46.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:45 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:00:46.095 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 15.77 MB/sec warmup 69 sec latency 3642.316 ms 2024-04-26T02:00:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.111+0000 7ff894c00640 1 -- 172.21.15.6:0/3258532161 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cd723d5320 con 0x55cd724e3830 2024-04-26T02:00:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.111+0000 7ff894c00640 10 client.16794 handle_command_reply: tid=0 2024-04-26T02:00:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.111+0000 7ff8b4c12740 1 client.16794 shutdown 2024-04-26T02:00:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.111+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3258532161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd72462140 msgr2=0x55cd72472050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.111+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/3258532161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd72462140 0x55cd72472050 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff89802f810 tx=0x7ff898000f40 comp rx=0 tx=0).stop 2024-04-26T02:00:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.111+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3258532161 shutdown_connections 2024-04-26T02:00:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.111+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/3258532161 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55cd724e3830 0x55cd724e5c50 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.111+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/3258532161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55cd72462140 0x55cd72472050 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.111+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/3258532161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cd7246ca00 0x55cd72472590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.111+0000 7ff8b4c12740 1 --2- 172.21.15.6:0/3258532161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd72461820 0x55cd72474cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.111+0000 7ff894c00640 0 client.16794 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.111+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3258532161 >> 172.21.15.6:0/3258532161 conn(0x55cd7245dd30 msgr2=0x55cd7245ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.112+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3258532161 shutdown_connections 2024-04-26T02:00:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.112+0000 7ff8b4c12740 1 -- 172.21.15.6:0/3258532161 wait complete. 2024-04-26T02:00:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.112+0000 7ff8b4c12740 20 client.16794 trim_cache size 0 max 16384 2024-04-26T02:00:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:00:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:00:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:00:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:00:46.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.113+0000 7ff8b1e00640 1 -- 172.21.15.6:0/1702042748 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff8900a3d10 msgr2=0x7ff8900a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:46.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.113+0000 7ff8b1e00640 1 --2- 172.21.15.6:0/1702042748 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff8900a3d10 0x7ff8900a61d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff89c0ce2e0 tx=0x7ff89c000bc0 comp rx=0 tx=0).stop 2024-04-26T02:00:46.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.113+0000 7ff8b1e00640 1 -- 172.21.15.6:0/1702042748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac074320 msgr2=0x7ff8ac1a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:46.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.113+0000 7ff8b1e00640 1 --2- 172.21.15.6:0/1702042748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac074320 0x7ff8ac1a0670 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00e9f0 tx=0x7ff89c00eeb0 comp rx=0 tx=0).stop 2024-04-26T02:00:46.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.113+0000 7ff8b1e00640 1 -- 172.21.15.6:0/1702042748 shutdown_connections 2024-04-26T02:00:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.113+0000 7ff8b1e00640 1 --2- 172.21.15.6:0/1702042748 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff8900a3d10 0x7ff8900a61d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.113+0000 7ff8b1e00640 1 --2- 172.21.15.6:0/1702042748 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8ac074c40 0x7ff8ac1a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.113+0000 7ff8b1e00640 1 --2- 172.21.15.6:0/1702042748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8ac10e7e0 0x7ff8ac19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.113+0000 7ff8b1e00640 1 --2- 172.21.15.6:0/1702042748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac074320 0x7ff8ac1a0670 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.113+0000 7ff8b1e00640 1 -- 172.21.15.6:0/1702042748 >> 172.21.15.6:0/1702042748 conn(0x7ff8ac0fda10 msgr2=0x7ff8ac0ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.113+0000 7ff8b1e00640 1 -- 172.21.15.6:0/1702042748 shutdown_connections 2024-04-26T02:00:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:46.113+0000 7ff8b1e00640 1 -- 172.21.15.6:0/1702042748 wait complete. 2024-04-26T02:00:46.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:46 smithi188 ceph-mon[32751]: pgmap v471: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.7 MiB/s wr, 366 op/s 2024-04-26T02:00:46.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:46 smithi029 ceph-mon[32775]: pgmap v471: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.7 MiB/s wr, 366 op/s 2024-04-26T02:00:47.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:46 smithi006 ceph-mon[28419]: pgmap v471: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.7 MiB/s wr, 366 op/s 2024-04-26T02:00:47.095 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 15.55 MB/sec warmup 70 sec latency 4642.477 ms 2024-04-26T02:00:48.096 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 15.33 MB/sec warmup 71 sec latency 5642.650 ms 2024-04-26T02:00:48.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:48 smithi188 ceph-mon[32751]: pgmap v472: 97 pgs: 97 active+clean; 422 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.7 MiB/s wr, 512 op/s 2024-04-26T02:00:48.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:48 smithi029 ceph-mon[32775]: pgmap v472: 97 pgs: 97 active+clean; 422 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.7 MiB/s wr, 512 op/s 2024-04-26T02:00:49.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:48 smithi006 ceph-mon[28419]: pgmap v472: 97 pgs: 97 active+clean; 422 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.7 MiB/s wr, 512 op/s 2024-04-26T02:00:49.096 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 15.12 MB/sec warmup 72 sec latency 6642.750 ms 2024-04-26T02:00:50.096 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 14.91 MB/sec warmup 73 sec latency 7642.851 ms 2024-04-26T02:00:50.870 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:50 smithi006 ceph-mon[28419]: pgmap v473: 97 pgs: 97 active+clean; 422 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.5 MiB/s wr, 367 op/s 2024-04-26T02:00:50.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:50 smithi188 ceph-mon[32751]: pgmap v473: 97 pgs: 97 active+clean; 422 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.5 MiB/s wr, 367 op/s 2024-04-26T02:00:50.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:50 smithi029 ceph-mon[32775]: pgmap v473: 97 pgs: 97 active+clean; 422 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.5 MiB/s wr, 367 op/s 2024-04-26T02:00:51.096 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 14.71 MB/sec warmup 74 sec latency 8642.979 ms 2024-04-26T02:00:52.096 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 14.51 MB/sec warmup 75 sec latency 9643.105 ms 2024-04-26T02:00:52.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:52 smithi006 ceph-mon[28419]: pgmap v474: 97 pgs: 97 active+clean; 415 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.5 MiB/s wr, 390 op/s 2024-04-26T02:00:52.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:52 smithi188 ceph-mon[32751]: pgmap v474: 97 pgs: 97 active+clean; 415 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.5 MiB/s wr, 390 op/s 2024-04-26T02:00:52.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:52 smithi029 ceph-mon[32775]: pgmap v474: 97 pgs: 97 active+clean; 415 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.5 MiB/s wr, 390 op/s 2024-04-26T02:00:53.096 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 14.32 MB/sec warmup 76 sec latency 10643.264 ms 2024-04-26T02:00:54.096 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 14.14 MB/sec warmup 77 sec latency 11643.357 ms 2024-04-26T02:00:54.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:54 smithi188 ceph-mon[32751]: pgmap v475: 97 pgs: 97 active+clean; 415 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 169 op/s 2024-04-26T02:00:54.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:54 smithi029 ceph-mon[32775]: pgmap v475: 97 pgs: 97 active+clean; 415 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 169 op/s 2024-04-26T02:00:55.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:54 smithi006 ceph-mon[28419]: pgmap v475: 97 pgs: 97 active+clean; 415 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 169 op/s 2024-04-26T02:00:55.096 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 13.95 MB/sec warmup 78 sec latency 12643.457 ms 2024-04-26T02:00:56.097 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 13.78 MB/sec warmup 79 sec latency 13643.647 ms 2024-04-26T02:00:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:56 smithi188 ceph-mon[32751]: pgmap v476: 97 pgs: 97 active+clean; 415 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 169 op/s 2024-04-26T02:00:56.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:56 smithi029 ceph-mon[32775]: pgmap v476: 97 pgs: 97 active+clean; 415 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 169 op/s 2024-04-26T02:00:57.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:56 smithi006 ceph-mon[28419]: pgmap v476: 97 pgs: 97 active+clean; 415 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 169 op/s 2024-04-26T02:00:57.097 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 13.61 MB/sec warmup 80 sec latency 14643.741 ms 2024-04-26T02:00:58.097 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 13.44 MB/sec warmup 81 sec latency 15643.831 ms 2024-04-26T02:00:58.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:00:58 smithi188 ceph-mon[32751]: pgmap v477: 97 pgs: 97 active+clean; 431 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 192 op/s 2024-04-26T02:00:58.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:00:58 smithi029 ceph-mon[32775]: pgmap v477: 97 pgs: 97 active+clean; 431 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 192 op/s 2024-04-26T02:00:59.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:00:58 smithi006 ceph-mon[28419]: pgmap v477: 97 pgs: 97 active+clean; 431 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 192 op/s 2024-04-26T02:00:59.097 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 13.27 MB/sec warmup 82 sec latency 16644.004 ms 2024-04-26T02:00:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.271+0000 7fed72a00640 1 -- 172.21.15.6:0/2176226137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c108d50 msgr2=0x7fed6c115210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.271+0000 7fed72a00640 1 --2- 172.21.15.6:0/2176226137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c108d50 0x7fed6c115210 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fed64009ca0 tx=0x7fed6402a950 comp rx=0 tx=0).stop 2024-04-26T02:00:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.271+0000 7fed72a00640 1 -- 172.21.15.6:0/2176226137 shutdown_connections 2024-04-26T02:00:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.271+0000 7fed72a00640 1 --2- 172.21.15.6:0/2176226137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c108d50 0x7fed6c115210 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.271+0000 7fed72a00640 1 --2- 172.21.15.6:0/2176226137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6c1083b0 0x7fed6c108810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.271+0000 7fed72a00640 1 --2- 172.21.15.6:0/2176226137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed6c10e470 0x7fed6c10e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.271+0000 7fed72a00640 1 -- 172.21.15.6:0/2176226137 >> 172.21.15.6:0/2176226137 conn(0x7fed6c077b00 msgr2=0x7fed6c079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.271+0000 7fed72a00640 1 -- 172.21.15.6:0/2176226137 shutdown_connections 2024-04-26T02:00:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.271+0000 7fed72a00640 1 -- 172.21.15.6:0/2176226137 wait complete. 2024-04-26T02:00:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.272+0000 7fed72a00640 1 Processor -- start 2024-04-26T02:00:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.272+0000 7fed72a00640 1 -- start start 2024-04-26T02:00:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.272+0000 7fed72a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6c1083b0 0x7fed6c1a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.272+0000 7fed72a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c108d50 0x7fed6c1a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.272+0000 7fed72a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed6c10e470 0x7fed6c19e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.272+0000 7fed72a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c118650 con 0x7fed6c108d50 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.272+0000 7fed72a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c1184d0 con 0x7fed6c1083b0 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.272+0000 7fed72a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c1187d0 con 0x7fed6c10e470 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.273+0000 7fed71600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6c1083b0 0x7fed6c1a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.273+0000 7fed70c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c108d50 0x7fed6c1a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.273+0000 7fed71600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6c1083b0 0x7fed6c1a4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:50874/0 (socket says 172.21.15.6:50874) 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.273+0000 7fed71600640 1 -- 172.21.15.6:0/1801015102 learned_addr learned my addr 172.21.15.6:0/1801015102 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.273+0000 7fed72000640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed6c10e470 0x7fed6c19e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.273+0000 7fed71600640 1 -- 172.21.15.6:0/1801015102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed6c10e470 msgr2=0x7fed6c19e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.273+0000 7fed71600640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed6c10e470 0x7fed6c19e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.273+0000 7fed71600640 1 -- 172.21.15.6:0/1801015102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c108d50 msgr2=0x7fed6c1a4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.273+0000 7fed71600640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c108d50 0x7fed6c1a4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.273+0000 7fed71600640 1 -- 172.21.15.6:0/1801015102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed6c19f090 con 0x7fed6c1083b0 2024-04-26T02:00:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.274+0000 7fed71600640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6c1083b0 0x7fed6c1a4760 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fed6000e9f0 tx=0x7fed6000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.274+0000 7fed6a000640 1 -- 172.21.15.6:0/1801015102 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6000cd80 con 0x7fed6c1083b0 2024-04-26T02:00:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.274+0000 7fed6a000640 1 -- 172.21.15.6:0/1801015102 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed600054d0 con 0x7fed6c1083b0 2024-04-26T02:00:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.274+0000 7fed72a00640 1 -- 172.21.15.6:0/1801015102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed6c19f380 con 0x7fed6c1083b0 2024-04-26T02:00:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.274+0000 7fed6a000640 1 -- 172.21.15.6:0/1801015102 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed60010690 con 0x7fed6c1083b0 2024-04-26T02:00:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.274+0000 7fed72a00640 1 -- 172.21.15.6:0/1801015102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed6c1ab620 con 0x7fed6c1083b0 2024-04-26T02:00:59.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.277+0000 7fed76dea740 1 Processor -- start 2024-04-26T02:00:59.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.277+0000 7fed76dea740 1 -- start start 2024-04-26T02:00:59.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.277+0000 7fed6a000640 1 -- 172.21.15.6:0/1801015102 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fed60004e90 con 0x7fed6c1083b0 2024-04-26T02:00:59.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.278+0000 7fed76dea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e444d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.278+0000 7fed76dea740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4452d0 0x55c18e44f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.278+0000 7fed76dea740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e451f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.278+0000 7fed76dea740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55c18e3aa900 con 0x55c18e44fb90 2024-04-26T02:00:59.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.278+0000 7fed76dea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c18e350ef0 con 0x55c18e4449b0 2024-04-26T02:00:59.277 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.278+0000 7fed76dea740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c18e3a68c0 con 0x55c18e4452d0 2024-04-26T02:00:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.278+0000 7fed71600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4452d0 0x55c18e44f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.278+0000 7fed70c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e451f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.278+0000 7fed71600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4452d0 0x55c18e44f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51630/0 (socket says 172.21.15.6:51630) 2024-04-26T02:00:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.278+0000 7fed71600640 1 -- 172.21.15.6:0/1737371343 learned_addr learned my addr 172.21.15.6:0/1737371343 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.278+0000 7fed72000640 1 --2- 172.21.15.6:0/1737371343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e444d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.279+0000 7fed71600640 1 -- 172.21.15.6:0/1737371343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 msgr2=0x55c18e444d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.279+0000 7fed71600640 1 --2- 172.21.15.6:0/1737371343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e444d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.279+0000 7fed71600640 1 -- 172.21.15.6:0/1737371343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 msgr2=0x55c18e451f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.279+0000 7fed71600640 1 --2- 172.21.15.6:0/1737371343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e451f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.279+0000 7fed71600640 1 -- 172.21.15.6:0/1737371343 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c18e0bd210 con 0x55c18e4452d0 2024-04-26T02:00:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.279+0000 7fed72000640 1 --2- 172.21.15.6:0/1737371343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e444d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.279+0000 7fed70c00640 1 --2- 172.21.15.6:0/1737371343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e451f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.279+0000 7fed71600640 1 --2- 172.21.15.6:0/1737371343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4452d0 0x55c18e44f650 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fed600ce200 tx=0x7fed600d4c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd83c4a6b5bb1cdc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.279+0000 7fed5a600640 1 -- 172.21.15.6:0/1737371343 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed600e0020 con 0x55c18e4452d0 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.279+0000 7fed6a000640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fed540a3d80 0x7fed540a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.279+0000 7fed5a600640 1 -- 172.21.15.6:0/1737371343 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed600dc460 con 0x55c18e4452d0 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.279+0000 7fed5a600640 1 -- 172.21.15.6:0/1737371343 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed600db620 con 0x55c18e4452d0 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.280+0000 7fed6a000640 1 -- 172.21.15.6:0/1801015102 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7fed600d0bf0 con 0x7fed6c1083b0 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.280+0000 7fed72000640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fed540a3d80 0x7fed540a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.280+0000 7fed76dea740 1 -- 172.21.15.6:0/1737371343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4452d0 msgr2=0x55c18e44f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.280+0000 7fed76dea740 1 --2- 172.21.15.6:0/1737371343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4452d0 0x55c18e44f650 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fed600ce200 tx=0x7fed600d4c90 comp rx=0 tx=0).stop 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.280+0000 7fed76dea740 1 -- 172.21.15.6:0/1737371343 shutdown_connections 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.280+0000 7fed76dea740 1 --2- 172.21.15.6:0/1737371343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e451f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.280+0000 7fed76dea740 1 --2- 172.21.15.6:0/1737371343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4452d0 0x55c18e44f650 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.280+0000 7fed76dea740 1 --2- 172.21.15.6:0/1737371343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e444d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.280+0000 7fed76dea740 1 -- 172.21.15.6:0/1737371343 >> 172.21.15.6:0/1737371343 conn(0x55c18e440ec0 msgr2=0x55c18e4412a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.281+0000 7fed76dea740 1 -- 172.21.15.6:0/1737371343 shutdown_connections 2024-04-26T02:00:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.281+0000 7fed72000640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fed540a3d80 0x7fed540a6240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fed64006010 tx=0x7fed64005d80 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.281+0000 7fed76dea740 1 -- 172.21.15.6:0/1737371343 wait complete. 2024-04-26T02:00:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.281+0000 7fed76dea740 1 -- start start 2024-04-26T02:00:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.281+0000 7fed76dea740 1 Processor -- start 2024-04-26T02:00:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed76dea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e44cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed76dea740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4452d0 0x55c18e44d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed76dea740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e44d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed76dea740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55c18e4527c0 con 0x55c18e44fb90 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed76dea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c18e364310 con 0x55c18e4449b0 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed76dea740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c18e452940 con 0x55c18e4452d0 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed76dea740 10 client.0 _ll_register_callbacks cb 0x55c18e332cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed71600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e44cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed71600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e44cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:50894/0 (socket says 172.21.15.6:50894) 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed71600640 1 -- 172.21.15.6:0/1812217144 learned_addr learned my addr 172.21.15.6:0/1812217144 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed70c00640 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4452d0 0x55c18e44d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed72000640 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e44d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed71600640 1 -- 172.21.15.6:0/1812217144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4452d0 msgr2=0x55c18e44d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed71600640 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4452d0 0x55c18e44d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.286 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed71600640 1 -- 172.21.15.6:0/1812217144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 msgr2=0x55c18e44d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed71600640 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e44d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed71600640 1 -- 172.21.15.6:0/1812217144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c18e189620 con 0x55c18e4449b0 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.282+0000 7fed72000640 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e44d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.283+0000 7fed71600640 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e44cac0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fed60098f70 tx=0x7fed60009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a45c952ae33c2b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.283+0000 7fed4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.283+0000 7fed4ea00640 1 -- 172.21.15.6:0/1812217144 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed600e0070 con 0x55c18e4449b0 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.283+0000 7fed4ea00640 1 -- 172.21.15.6:0/1812217144 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7fed600046c0 con 0x55c18e4449b0 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.283+0000 7fed76dea740 1 -- 172.21.15.6:0/1812217144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed600dc460 con 0x55c18e4449b0 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.283+0000 7fed4ea00640 1 -- 172.21.15.6:0/1812217144 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fed60098db0 con 0x55c18e4449b0 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.283+0000 7fed76dea740 10 client.26975 fetch_fsmap learned FSMap version 49 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.283+0000 7fed76dea740 1 -- 172.21.15.6:0/1812217144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55c18e45ea30 con 0x55c18e4449b0 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.283+0000 7fed4ea00640 1 -- 172.21.15.6:0/1812217144 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed600dc460 con 0x55c18e4449b0 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.284+0000 7fed4ea00640 1 -- 172.21.15.6:0/1812217144 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fed60099180 con 0x55c18e4449b0 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.284+0000 7fed76dea740 10 client.26975 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:00:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.284+0000 7fed76dea740 10 client.26975 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.284+0000 7fed76dea740 20 client.26975 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.284+0000 7fed76dea740 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c18e45fd80 0x55c18e462170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.284+0000 7fed76dea740 4 client.26975 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.284+0000 7fed76dea740 1 -- 172.21.15.6:0/1812217144 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c18e3be8b0 con 0x55c18e45fd80 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.284+0000 7fed70c00640 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c18e45fd80 0x55c18e462170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.285+0000 7fed70c00640 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c18e45fd80 0x55c18e462170 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97a9e6b2482c6e36 server_cookie=9d572ec09822b3ba in_seq=0 out_seq=0 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.285+0000 7fed4ea00640 10 client.26975 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed4ea00640 1 -- 172.21.15.6:0/1812217144 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c18e3be8b0 con 0x55c18e45fd80 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed4ea00640 10 client.26975 handle_command_reply: tid=0 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed76dea740 1 client.26975 shutdown 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed76dea740 1 -- 172.21.15.6:0/1812217144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 msgr2=0x55c18e44cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed76dea740 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e44cac0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fed60098f70 tx=0x7fed60009720 comp rx=0 tx=0).stop 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed76dea740 1 -- 172.21.15.6:0/1812217144 shutdown_connections 2024-04-26T02:00:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed76dea740 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c18e45fd80 0x55c18e462170 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed76dea740 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e44d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed76dea740 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4452d0 0x55c18e44d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed76dea740 1 --2- 172.21.15.6:0/1812217144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e44cac0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed76dea740 1 -- 172.21.15.6:0/1812217144 >> 172.21.15.6:0/1812217144 conn(0x55c18e440ec0 msgr2=0x55c18e446bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed4ea00640 0 client.26975 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed76dea740 1 -- 172.21.15.6:0/1812217144 shutdown_connections 2024-04-26T02:00:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed76dea740 1 -- 172.21.15.6:0/1812217144 wait complete. 2024-04-26T02:00:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.286+0000 7fed76dea740 20 client.26975 trim_cache size 0 max 16384 2024-04-26T02:00:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.309+0000 7fed76dea740 1 Processor -- start 2024-04-26T02:00:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.310+0000 7fed76dea740 1 -- start start 2024-04-26T02:00:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.310+0000 7fed76dea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e45b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.310+0000 7fed76dea740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e4452d0 0x55c18e467710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.310+0000 7fed76dea740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e44fb90 0x55c18e467c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.310+0000 7fed76dea740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c1b6a30 con 0x55c18e4452d0 2024-04-26T02:00:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.310+0000 7fed76dea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c1b4270 con 0x55c18e4449b0 2024-04-26T02:00:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.310+0000 7fed76dea740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c1b20c0 con 0x55c18e44fb90 2024-04-26T02:00:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.310+0000 7fed71600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e45b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.311+0000 7fed71600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e45b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:50906/0 (socket says 172.21.15.6:50906) 2024-04-26T02:00:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.311+0000 7fed71600640 1 -- 172.21.15.6:0/909308375 learned_addr learned my addr 172.21.15.6:0/909308375 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.311+0000 7fed72000640 1 --2- 172.21.15.6:0/909308375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e44fb90 0x55c18e467c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.311+0000 7fed70c00640 1 --2- 172.21.15.6:0/909308375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e4452d0 0x55c18e467710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.311+0000 7fed71600640 1 -- 172.21.15.6:0/909308375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e44fb90 msgr2=0x55c18e467c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.311+0000 7fed71600640 1 --2- 172.21.15.6:0/909308375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e44fb90 0x55c18e467c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.311+0000 7fed71600640 1 -- 172.21.15.6:0/909308375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e4452d0 msgr2=0x55c18e467710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.311+0000 7fed71600640 1 --2- 172.21.15.6:0/909308375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e4452d0 0x55c18e467710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.311+0000 7fed71600640 1 -- 172.21.15.6:0/909308375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c18e445100 con 0x55c18e4449b0 2024-04-26T02:00:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.311+0000 7fed70c00640 1 --2- 172.21.15.6:0/909308375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e4452d0 0x55c18e467710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.311+0000 7fed71600640 1 --2- 172.21.15.6:0/909308375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e45b6b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fed600994e0 tx=0x7fed60009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=811e183bb91bc3af server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:59.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.312+0000 7fed59c00640 1 -- 172.21.15.6:0/909308375 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed600e0070 con 0x55c18e4449b0 2024-04-26T02:00:59.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.312+0000 7fed59c00640 1 -- 172.21.15.6:0/909308375 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed600d96b0 con 0x55c18e4449b0 2024-04-26T02:00:59.312 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.312+0000 7fed59c00640 1 -- 172.21.15.6:0/909308375 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed60005180 con 0x55c18e4449b0 2024-04-26T02:00:59.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.312+0000 7fed76dea740 1 -- 172.21.15.6:0/909308375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 msgr2=0x55c18e45b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:59.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.312+0000 7fed76dea740 1 --2- 172.21.15.6:0/909308375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e45b6b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fed600994e0 tx=0x7fed60009790 comp rx=0 tx=0).stop 2024-04-26T02:00:59.314 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.312+0000 7fed76dea740 1 -- 172.21.15.6:0/909308375 shutdown_connections 2024-04-26T02:00:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.312+0000 7fed76dea740 1 --2- 172.21.15.6:0/909308375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e4452d0 0x55c18e467710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.312+0000 7fed76dea740 1 --2- 172.21.15.6:0/909308375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e44fb90 0x55c18e467c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.312+0000 7fed76dea740 1 --2- 172.21.15.6:0/909308375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4449b0 0x55c18e45b6b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.312+0000 7fed76dea740 1 -- 172.21.15.6:0/909308375 >> 172.21.15.6:0/909308375 conn(0x55c18e440ec0 msgr2=0x55c18e4412a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.313+0000 7fed76dea740 1 -- 172.21.15.6:0/909308375 shutdown_connections 2024-04-26T02:00:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.313+0000 7fed76dea740 1 -- 172.21.15.6:0/909308375 wait complete. 2024-04-26T02:00:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.313+0000 7fed76dea740 1 -- start start 2024-04-26T02:00:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.313+0000 7fed76dea740 1 Processor -- start 2024-04-26T02:00:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.314+0000 7fed76dea740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4449b0 0x55c18e457f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.314+0000 7fed76dea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4452d0 0x55c18e4552c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.314+0000 7fed76dea740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e455800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.314+0000 7fed76dea740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c1b2730 con 0x55c18e44fb90 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.314+0000 7fed76dea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c1b3ba0 con 0x55c18e4452d0 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.314+0000 7fed76dea740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c1b2e00 con 0x55c18e4449b0 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.314+0000 7fed76dea740 10 client.0 _ll_register_callbacks cb 0x7fed60005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.314+0000 7fed71600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4449b0 0x55c18e457f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.314+0000 7fed72000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e455800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.314+0000 7fed71600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4449b0 0x55c18e457f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51660/0 (socket says 172.21.15.6:51660) 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.314+0000 7fed71600640 1 -- 172.21.15.6:0/2384703027 learned_addr learned my addr 172.21.15.6:0/2384703027 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.314+0000 7fed70c00640 1 --2- 172.21.15.6:0/2384703027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4452d0 0x55c18e4552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed71600640 1 -- 172.21.15.6:0/2384703027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4452d0 msgr2=0x55c18e4552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed71600640 1 --2- 172.21.15.6:0/2384703027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4452d0 0x55c18e4552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed71600640 1 -- 172.21.15.6:0/2384703027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 msgr2=0x55c18e455800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed71600640 1 --2- 172.21.15.6:0/2384703027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e455800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed71600640 1 -- 172.21.15.6:0/2384703027 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c18e456030 con 0x55c18e4449b0 2024-04-26T02:00:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed71600640 1 --2- 172.21.15.6:0/2384703027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4449b0 0x55c18e457f30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fed60098f70 tx=0x7fed6009b020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cb887b3bacac255 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:00:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed4ea00640 1 -- 172.21.15.6:0/2384703027 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed600e0070 con 0x55c18e4449b0 2024-04-26T02:00:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed4ea00640 1 -- 172.21.15.6:0/2384703027 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8556+0+0 (secure 0 0 0) 0x7fed60099180 con 0x55c18e4449b0 2024-04-26T02:00:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed76dea740 1 -- 172.21.15.6:0/2384703027 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed600d96b0 con 0x55c18e4449b0 2024-04-26T02:00:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed4ea00640 1 -- 172.21.15.6:0/2384703027 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fed6009c470 con 0x55c18e4449b0 2024-04-26T02:00:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed76dea740 10 client.27031 fetch_fsmap learned FSMap version 49 2024-04-26T02:00:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.315+0000 7fed76dea740 1 -- 172.21.15.6:0/2384703027 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55c18e456030 con 0x55c18e4449b0 2024-04-26T02:00:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.316+0000 7fed4ea00640 1 -- 172.21.15.6:0/2384703027 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed600d96b0 con 0x55c18e4449b0 2024-04-26T02:00:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.316+0000 7fed4ea00640 1 -- 172.21.15.6:0/2384703027 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fed600ce540 con 0x55c18e4449b0 2024-04-26T02:00:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.316+0000 7fed76dea740 10 client.27031 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:00:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.316+0000 7fed76dea740 10 client.27031 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:00:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.316+0000 7fed76dea740 20 client.27031 populate_metadata read hostname 'smithi006' 2024-04-26T02:00:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.316+0000 7fed76dea740 1 --2- 172.21.15.6:0/2384703027 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c18e4c62c0 0x55c18e4c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.316+0000 7fed76dea740 4 client.27031 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:00:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.316+0000 7fed76dea740 1 -- 172.21.15.6:0/2384703027 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c18e3b8320 con 0x55c18e4c62c0 2024-04-26T02:00:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.316+0000 7fed70c00640 1 --2- 172.21.15.6:0/2384703027 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c18e4c62c0 0x55c18e4c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.317+0000 7fed70c00640 1 --2- 172.21.15.6:0/2384703027 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c18e4c62c0 0x55c18e4c86b0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cac83d811564526 server_cookie=47ad54beea798c55 in_seq=0 out_seq=0 2024-04-26T02:00:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.317+0000 7fed4ea00640 10 client.27031 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.380+0000 7fed4ea00640 1 -- 172.21.15.6:0/2384703027 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c18e3b8320 con 0x55c18e4c62c0 2024-04-26T02:00:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.380+0000 7fed4ea00640 10 client.27031 handle_command_reply: tid=0 2024-04-26T02:00:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.380+0000 7fed76dea740 1 client.27031 shutdown 2024-04-26T02:00:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.381+0000 7fed76dea740 1 -- 172.21.15.6:0/2384703027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4449b0 msgr2=0x55c18e457f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.381+0000 7fed76dea740 1 --2- 172.21.15.6:0/2384703027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4449b0 0x55c18e457f30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fed60098f70 tx=0x7fed6009b020 comp rx=0 tx=0).stop 2024-04-26T02:00:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.381+0000 7fed76dea740 1 -- 172.21.15.6:0/2384703027 shutdown_connections 2024-04-26T02:00:59.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.381+0000 7fed76dea740 1 --2- 172.21.15.6:0/2384703027 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c18e4c62c0 0x55c18e4c86b0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.381+0000 7fed76dea740 1 --2- 172.21.15.6:0/2384703027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c18e44fb90 0x55c18e455800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.381+0000 7fed76dea740 1 --2- 172.21.15.6:0/2384703027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c18e4452d0 0x55c18e4552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.381+0000 7fed76dea740 1 --2- 172.21.15.6:0/2384703027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c18e4449b0 0x55c18e457f30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.381+0000 7fed4ea00640 0 client.27031 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:00:59.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.381+0000 7fed76dea740 1 -- 172.21.15.6:0/2384703027 >> 172.21.15.6:0/2384703027 conn(0x55c18e440ec0 msgr2=0x55c18e446360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:59.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.381+0000 7fed76dea740 1 -- 172.21.15.6:0/2384703027 shutdown_connections 2024-04-26T02:00:59.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.381+0000 7fed76dea740 1 -- 172.21.15.6:0/2384703027 wait complete. 2024-04-26T02:00:59.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.381+0000 7fed76dea740 20 client.27031 trim_cache size 0 max 16384 2024-04-26T02:00:59.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:00:59.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.382+0000 7fed72a00640 1 -- 172.21.15.6:0/1801015102 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fed540a3d80 msgr2=0x7fed540a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.382+0000 7fed72a00640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fed540a3d80 0x7fed540a6240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fed64006010 tx=0x7fed64005d80 comp rx=0 tx=0).stop 2024-04-26T02:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.382+0000 7fed72a00640 1 -- 172.21.15.6:0/1801015102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6c1083b0 msgr2=0x7fed6c1a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.382+0000 7fed72a00640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6c1083b0 0x7fed6c1a4760 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fed6000e9f0 tx=0x7fed6000edf0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.382+0000 7fed72a00640 1 -- 172.21.15.6:0/1801015102 shutdown_connections 2024-04-26T02:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.382+0000 7fed72a00640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fed540a3d80 0x7fed540a6240 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.382+0000 7fed72a00640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6c108d50 0x7fed6c1a4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.382+0000 7fed72a00640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed6c10e470 0x7fed6c19e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.382+0000 7fed72a00640 1 --2- 172.21.15.6:0/1801015102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed6c1083b0 0x7fed6c1a4760 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.382+0000 7fed72a00640 1 -- 172.21.15.6:0/1801015102 >> 172.21.15.6:0/1801015102 conn(0x7fed6c077b00 msgr2=0x7fed6c079260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.382+0000 7fed72a00640 1 -- 172.21.15.6:0/1801015102 shutdown_connections 2024-04-26T02:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:00:59.382+0000 7fed72a00640 1 -- 172.21.15.6:0/1801015102 wait complete. 2024-04-26T02:01:00.097 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 13.11 MB/sec warmup 83 sec latency 17644.147 ms 2024-04-26T02:01:00.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:00 smithi188 ceph-mon[32751]: pgmap v478: 97 pgs: 97 active+clean; 431 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.4 MiB/s wr, 45 op/s 2024-04-26T02:01:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:00 smithi029 ceph-mon[32775]: pgmap v478: 97 pgs: 97 active+clean; 431 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.4 MiB/s wr, 45 op/s 2024-04-26T02:01:01.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:00 smithi006 ceph-mon[28419]: pgmap v478: 97 pgs: 97 active+clean; 431 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.4 MiB/s wr, 45 op/s 2024-04-26T02:01:01.097 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 12.96 MB/sec warmup 84 sec latency 18644.241 ms 2024-04-26T02:01:02.098 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 12.81 MB/sec warmup 85 sec latency 19644.367 ms 2024-04-26T02:01:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:02 smithi188 ceph-mon[32751]: pgmap v479: 97 pgs: 97 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 3.5 MiB/s wr, 135 op/s 2024-04-26T02:01:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:02 smithi029 ceph-mon[32775]: pgmap v479: 97 pgs: 97 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 3.5 MiB/s wr, 135 op/s 2024-04-26T02:01:03.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:02 smithi006 ceph-mon[28419]: pgmap v479: 97 pgs: 97 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 3.5 MiB/s wr, 135 op/s 2024-04-26T02:01:03.098 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 12.66 MB/sec warmup 86 sec latency 20645.001 ms 2024-04-26T02:01:04.098 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 12.51 MB/sec warmup 87 sec latency 21645.146 ms 2024-04-26T02:01:04.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:04 smithi188 ceph-mon[32751]: pgmap v480: 97 pgs: 97 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 3.5 MiB/s wr, 111 op/s 2024-04-26T02:01:04.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:04 smithi029 ceph-mon[32775]: pgmap v480: 97 pgs: 97 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 3.5 MiB/s wr, 111 op/s 2024-04-26T02:01:05.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:04 smithi006 ceph-mon[28419]: pgmap v480: 97 pgs: 97 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 3.5 MiB/s wr, 111 op/s 2024-04-26T02:01:05.098 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 12.37 MB/sec warmup 88 sec latency 22645.227 ms 2024-04-26T02:01:05.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:05 smithi188 ceph-mon[32751]: osdmap e120: 12 total, 12 up, 12 in 2024-04-26T02:01:05.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:05 smithi029 ceph-mon[32775]: osdmap e120: 12 total, 12 up, 12 in 2024-04-26T02:01:06.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:05 smithi006 ceph-mon[28419]: osdmap e120: 12 total, 12 up, 12 in 2024-04-26T02:01:06.098 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 12.23 MB/sec warmup 89 sec latency 23645.355 ms 2024-04-26T02:01:06.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:06 smithi029 ceph-mon[32775]: pgmap v482: 97 pgs: 97 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 4.2 MiB/s wr, 134 op/s 2024-04-26T02:01:07.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:06 smithi006 ceph-mon[28419]: pgmap v482: 97 pgs: 97 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 4.2 MiB/s wr, 134 op/s 2024-04-26T02:01:07.098 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 12.09 MB/sec warmup 90 sec latency 24645.518 ms 2024-04-26T02:01:07.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:06 smithi188 ceph-mon[32751]: pgmap v482: 97 pgs: 97 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 4.2 MiB/s wr, 134 op/s 2024-04-26T02:01:08.099 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 11.96 MB/sec warmup 91 sec latency 25645.685 ms 2024-04-26T02:01:08.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:01:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:01:08.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:08 smithi029 ceph-mon[32775]: pgmap v483: 97 pgs: 97 active+clean; 420 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 3.9 MiB/s wr, 196 op/s 2024-04-26T02:01:09.009 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:08 smithi006 ceph-mon[28419]: pgmap v483: 97 pgs: 97 active+clean; 420 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 3.9 MiB/s wr, 196 op/s 2024-04-26T02:01:09.099 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 11.83 MB/sec warmup 92 sec latency 26645.787 ms 2024-04-26T02:01:09.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:08 smithi188 ceph-mon[32751]: pgmap v483: 97 pgs: 97 active+clean; 420 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 3.9 MiB/s wr, 196 op/s 2024-04-26T02:01:10.099 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 11.70 MB/sec warmup 93 sec latency 27645.893 ms 2024-04-26T02:01:10.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.926+0000 7fe766400700 1 -- 172.21.15.6:0/3045466115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7601092d0 msgr2=0x7fe7601096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:10.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.926+0000 7fe766400700 1 --2- 172.21.15.6:0/3045466115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7601092d0 0x7fe7601096b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe750009990 tx=0x7fe750009ca0 comp rx=0 tx=0).stop 2024-04-26T02:01:10.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.926+0000 7fe766400700 1 -- 172.21.15.6:0/3045466115 shutdown_connections 2024-04-26T02:01:10.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.926+0000 7fe766400700 1 --2- 172.21.15.6:0/3045466115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe76010e3b0 0x7fe760115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:10.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.926+0000 7fe766400700 1 --2- 172.21.15.6:0/3045466115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7601092d0 0x7fe7601096b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:10.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.926+0000 7fe766400700 1 --2- 172.21.15.6:0/3045466115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe760109c80 0x7fe76010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:10.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.926+0000 7fe766400700 1 -- 172.21.15.6:0/3045466115 >> 172.21.15.6:0/3045466115 conn(0x7fe760076210 msgr2=0x7fe760107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:10.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.926+0000 7fe766400700 1 -- 172.21.15.6:0/3045466115 shutdown_connections 2024-04-26T02:01:10.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.926+0000 7fe766400700 1 -- 172.21.15.6:0/3045466115 wait complete. 2024-04-26T02:01:10.927 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:10 smithi006 ceph-mon[28419]: pgmap v484: 97 pgs: 97 active+clean; 420 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 3.9 MiB/s wr, 196 op/s 2024-04-26T02:01:10.927 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:10 smithi006 ceph-mon[28419]: osdmap e121: 12 total, 12 up, 12 in 2024-04-26T02:01:10.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.928+0000 7fe766400700 1 Processor -- start 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.928+0000 7fe766400700 1 -- start start 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.928+0000 7fe766400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7601092d0 0x7fe7601a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.928+0000 7fe766400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe760109c80 0x7fe7601a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.928+0000 7fe766400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe76010e3b0 0x7fe76019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.928+0000 7fe766400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe760119290 con 0x7fe76010e3b0 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.928+0000 7fe766400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe760119110 con 0x7fe7601092d0 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.928+0000 7fe766400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe760119410 con 0x7fe760109c80 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.929+0000 7fe75ea00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe760109c80 0x7fe7601a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.929+0000 7fe75ea00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe760109c80 0x7fe7601a5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:60480/0 (socket says 172.21.15.6:60480) 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.929+0000 7fe75ea00700 1 -- 172.21.15.6:0/2710511617 learned_addr learned my addr 172.21.15.6:0/2710511617 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.929+0000 7fe75f400700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7601092d0 0x7fe7601a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.929+0000 7fe75fe00700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe76010e3b0 0x7fe76019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:10.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.929+0000 7fe75ea00700 1 -- 172.21.15.6:0/2710511617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7601092d0 msgr2=0x7fe7601a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:10.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.929+0000 7fe75ea00700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7601092d0 0x7fe7601a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:10.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.929+0000 7fe75ea00700 1 -- 172.21.15.6:0/2710511617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe76010e3b0 msgr2=0x7fe76019f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:10.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.929+0000 7fe75ea00700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe76010e3b0 0x7fe76019f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:10.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.929+0000 7fe75ea00700 1 -- 172.21.15.6:0/2710511617 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe76019fd50 con 0x7fe760109c80 2024-04-26T02:01:10.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.929+0000 7fe75fe00700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe76010e3b0 0x7fe76019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:01:10.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.930+0000 7fe75ea00700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe760109c80 0x7fe7601a5920 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00e970 tx=0x7fe74c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:10.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.930+0000 7fe757e00700 1 -- 172.21.15.6:0/2710511617 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c00cbd0 con 0x7fe760109c80 2024-04-26T02:01:10.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.930+0000 7fe766400700 1 -- 172.21.15.6:0/2710511617 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7601a0040 con 0x7fe760109c80 2024-04-26T02:01:10.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.930+0000 7fe757e00700 1 -- 172.21.15.6:0/2710511617 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe74c00cd70 con 0x7fe760109c80 2024-04-26T02:01:10.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.930+0000 7fe757e00700 1 -- 172.21.15.6:0/2710511617 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c005680 con 0x7fe760109c80 2024-04-26T02:01:10.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.930+0000 7fe766400700 1 -- 172.21.15.6:0/2710511617 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7601ac2b0 con 0x7fe760109c80 2024-04-26T02:01:10.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.931+0000 7fe766400700 1 -- 172.21.15.6:0/2710511617 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe760069730 con 0x7fe760109c80 2024-04-26T02:01:10.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.934+0000 7fe757e00700 1 -- 172.21.15.6:0/2710511617 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fe74c004750 con 0x7fe760109c80 2024-04-26T02:01:10.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.935+0000 7fe757e00700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe7440a3db0 0x7fe7440a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:10.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.935+0000 7fe757e00700 1 -- 172.21.15.6:0/2710511617 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7fe74c0d0aa0 con 0x7fe760109c80 2024-04-26T02:01:10.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.935+0000 7fe75f400700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe7440a3db0 0x7fe7440a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:10.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.936+0000 7fe75f400700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe7440a3db0 0x7fe7440a6270 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe75000b790 tx=0x7fe75000b260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:10.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:10.937+0000 7fe757e00700 1 -- 172.21.15.6:0/2710511617 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe74c09d090 con 0x7fe760109c80 2024-04-26T02:01:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:10 smithi029 ceph-mon[32775]: pgmap v484: 97 pgs: 97 active+clean; 420 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 3.9 MiB/s wr, 196 op/s 2024-04-26T02:01:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:10 smithi029 ceph-mon[32775]: osdmap e121: 12 total, 12 up, 12 in 2024-04-26T02:01:11.099 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 11.58 MB/sec warmup 94 sec latency 28646.002 ms 2024-04-26T02:01:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:10 smithi188 ceph-mon[32751]: pgmap v484: 97 pgs: 97 active+clean; 420 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 3.9 MiB/s wr, 196 op/s 2024-04-26T02:01:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:10 smithi188 ceph-mon[32751]: osdmap e121: 12 total, 12 up, 12 in 2024-04-26T02:01:11.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.220+0000 7fe766400700 1 -- 172.21.15.6:0/2710511617 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe76002df80 con 0x7fe760109c80 2024-04-26T02:01:11.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.222+0000 7fe757e00700 1 -- 172.21.15.6:0/2710511617 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11857 (secure 0 0 0) 0x7fe74c09c990 con 0x7fe760109c80 2024-04-26T02:01:11.221 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:01:11.222 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T02:00:44.001061+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26935":{"gid":26935,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1750728695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1750728695},{"type":"v1","addr":"172.21.15.188:6835","nonce":1750728695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T02:01:11.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.225+0000 7fe766400700 1 -- 172.21.15.6:0/2710511617 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe7440a3db0 msgr2=0x7fe7440a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:11.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.225+0000 7fe766400700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe7440a3db0 0x7fe7440a6270 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe75000b790 tx=0x7fe75000b260 comp rx=0 tx=0).stop 2024-04-26T02:01:11.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.225+0000 7fe766400700 1 -- 172.21.15.6:0/2710511617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe760109c80 msgr2=0x7fe7601a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:11.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.225+0000 7fe766400700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe760109c80 0x7fe7601a5920 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00e970 tx=0x7fe74c00ec80 comp rx=0 tx=0).stop 2024-04-26T02:01:11.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.225+0000 7fe766400700 1 -- 172.21.15.6:0/2710511617 shutdown_connections 2024-04-26T02:01:11.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.225+0000 7fe766400700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe7440a3db0 0x7fe7440a6270 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:11.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.225+0000 7fe766400700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe76010e3b0 0x7fe76019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:11.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.225+0000 7fe766400700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7601092d0 0x7fe7601a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:11.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.225+0000 7fe766400700 1 --2- 172.21.15.6:0/2710511617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe760109c80 0x7fe7601a5920 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:11.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.225+0000 7fe766400700 1 -- 172.21.15.6:0/2710511617 >> 172.21.15.6:0/2710511617 conn(0x7fe760076210 msgr2=0x7fe760077e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:11.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.225+0000 7fe766400700 1 -- 172.21.15.6:0/2710511617 shutdown_connections 2024-04-26T02:01:11.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:11.226+0000 7fe766400700 1 -- 172.21.15.6:0/2710511617 wait complete. 2024-04-26T02:01:11.226 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-26T02:01:11.935 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:11 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2710511617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:01:11.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:11 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2710511617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:01:11.982 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id a6261f3e --release --await' 2024-04-26T02:01:11.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id a6261f3e --release --await 2024-04-26T02:01:12.099 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 11.46 MB/sec warmup 95 sec latency 29646.110 ms 2024-04-26T02:01:12.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:11 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2710511617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:01:12.436 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.519+0000 7f77b7e00640 1 -- 172.21.15.6:0/2083147311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77b0108f60 msgr2=0x7f77b0115420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.519+0000 7f77b7e00640 1 --2- 172.21.15.6:0/2083147311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77b0108f60 0x7f77b0115420 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f77a8009ca0 tx=0x7f77a802a8c0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.519+0000 7f77b7e00640 1 -- 172.21.15.6:0/2083147311 shutdown_connections 2024-04-26T02:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.519+0000 7f77b7e00640 1 --2- 172.21.15.6:0/2083147311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77b0108f60 0x7f77b0115420 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.519+0000 7f77b7e00640 1 --2- 172.21.15.6:0/2083147311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77b01085a0 0x7f77b0108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.519+0000 7f77b7e00640 1 --2- 172.21.15.6:0/2083147311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77b010e660 0x7f77b010ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.519+0000 7f77b7e00640 1 -- 172.21.15.6:0/2083147311 >> 172.21.15.6:0/2083147311 conn(0x7f77b0076460 msgr2=0x7f77b0078880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.519+0000 7f77b7e00640 1 -- 172.21.15.6:0/2083147311 shutdown_connections 2024-04-26T02:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.519+0000 7f77b7e00640 1 -- 172.21.15.6:0/2083147311 wait complete. 2024-04-26T02:01:12.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.520+0000 7f77b7e00640 1 Processor -- start 2024-04-26T02:01:12.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.521+0000 7f77b7e00640 1 -- start start 2024-04-26T02:01:12.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.521+0000 7f77b7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77b01085a0 0x7f77b01a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.521+0000 7f77b7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77b0108f60 0x7f77b01a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.521 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.521+0000 7f77b7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77b010e660 0x7f77b019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.521+0000 7f77b7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f77b01188a0 con 0x7f77b010e660 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.521+0000 7f77b7e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f77b0118720 con 0x7f77b0108f60 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.521+0000 7f77b7e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f77b0118a20 con 0x7f77b01085a0 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.521+0000 7f77b4a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77b0108f60 0x7f77b01a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.521+0000 7f77b4a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77b0108f60 0x7f77b01a4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:33968/0 (socket says 172.21.15.6:33968) 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.521+0000 7f77b4a00640 1 -- 172.21.15.6:0/3332505647 learned_addr learned my addr 172.21.15.6:0/3332505647 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.521+0000 7f77b5e00640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77b010e660 0x7f77b019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.521+0000 7f77b5400640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77b01085a0 0x7f77b01a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.522+0000 7f77b4a00640 1 -- 172.21.15.6:0/3332505647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77b01085a0 msgr2=0x7f77b01a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.522+0000 7f77b4a00640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77b01085a0 0x7f77b01a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.522+0000 7f77b4a00640 1 -- 172.21.15.6:0/3332505647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77b010e660 msgr2=0x7f77b019eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.522+0000 7f77b4a00640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77b010e660 0x7f77b019eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.522+0000 7f77b4a00640 1 -- 172.21.15.6:0/3332505647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b019f2c0 con 0x7f77b0108f60 2024-04-26T02:01:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.522+0000 7f77b4a00640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77b0108f60 0x7f77b01a4e80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f779c00ea60 tx=0x7f779c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:12.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.522+0000 7f77ae000640 1 -- 172.21.15.6:0/3332505647 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c00cde0 con 0x7f77b0108f60 2024-04-26T02:01:12.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.522+0000 7f77b7e00640 1 -- 172.21.15.6:0/3332505647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b019f5b0 con 0x7f77b0108f60 2024-04-26T02:01:12.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.522+0000 7f77ae000640 1 -- 172.21.15.6:0/3332505647 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f779c0035f0 con 0x7f77b0108f60 2024-04-26T02:01:12.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.522+0000 7f77ae000640 1 -- 172.21.15.6:0/3332505647 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c010690 con 0x7f77b0108f60 2024-04-26T02:01:12.523 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.522+0000 7f77b7e00640 1 -- 172.21.15.6:0/3332505647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b01ab800 con 0x7f77b0108f60 2024-04-26T02:01:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.526+0000 7f77bad96740 1 Processor -- start 2024-04-26T02:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77bad96740 1 -- start start 2024-04-26T02:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77bad96740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358b9820 0x55c0358b9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77bad96740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77bad96740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358c4a00 0x55c0358c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77bad96740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55c03581f900 con 0x55c0358ba140 2024-04-26T02:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77bad96740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c0357c5ef0 con 0x55c0358c4a00 2024-04-26T02:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77bad96740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c03581b8c0 con 0x55c0358b9820 2024-04-26T02:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77ae000640 1 -- 172.21.15.6:0/3332505647 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f779c0037a0 con 0x7f77b0108f60 2024-04-26T02:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77b5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77b5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358c44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45708/0 (socket says 172.21.15.6:45708) 2024-04-26T02:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77b5e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358b9820 0x55c0358b9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77b5400640 1 -- 172.21.15.6:0/2212941476 learned_addr learned my addr 172.21.15.6:0/2212941476 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.527+0000 7f77b4a00640 1 --2- 172.21.15.6:0/2212941476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358c4a00 0x55c0358c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.528+0000 7f77b5e00640 1 -- 172.21.15.6:0/2212941476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358c4a00 msgr2=0x55c0358c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.528+0000 7f77b5e00640 1 --2- 172.21.15.6:0/2212941476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358c4a00 0x55c0358c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.528+0000 7f77b5e00640 1 -- 172.21.15.6:0/2212941476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 msgr2=0x55c0358c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.528+0000 7f77b5e00640 1 --2- 172.21.15.6:0/2212941476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.528+0000 7f77b5e00640 1 -- 172.21.15.6:0/2212941476 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c035532210 con 0x55c0358b9820 2024-04-26T02:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.528+0000 7f77b4a00640 1 --2- 172.21.15.6:0/2212941476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358c4a00 0x55c0358c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:01:12.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.528+0000 7f77b5e00640 1 --2- 172.21.15.6:0/2212941476 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358b9820 0x55c0358b9c00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f77a8005e30 tx=0x7f77a80093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a18c24796930b2d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:12.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.529+0000 7f77a2600640 1 -- 172.21.15.6:0/2212941476 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a8031ce0 con 0x55c0358b9820 2024-04-26T02:01:12.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.529+0000 7f77ae000640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f77980a3d90 0x7f77980a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.529+0000 7f77a2600640 1 -- 172.21.15.6:0/2212941476 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77a8036030 con 0x55c0358b9820 2024-04-26T02:01:12.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.529+0000 7f77a2600640 1 -- 172.21.15.6:0/2212941476 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a80042a0 con 0x55c0358b9820 2024-04-26T02:01:12.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.529+0000 7f77ae000640 1 -- 172.21.15.6:0/3332505647 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7f779c0d0aa0 con 0x7f77b0108f60 2024-04-26T02:01:12.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.529+0000 7f77b5400640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f77980a3d90 0x7f77980a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.529+0000 7f77bad96740 1 -- 172.21.15.6:0/2212941476 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358b9820 msgr2=0x55c0358b9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:12.532 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.529+0000 7f77bad96740 1 --2- 172.21.15.6:0/2212941476 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358b9820 0x55c0358b9c00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f77a8005e30 tx=0x7f77a80093e0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.530+0000 7f77bad96740 1 -- 172.21.15.6:0/2212941476 shutdown_connections 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.530+0000 7f77bad96740 1 --2- 172.21.15.6:0/2212941476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.530+0000 7f77bad96740 1 --2- 172.21.15.6:0/2212941476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358c4a00 0x55c0358c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.530+0000 7f77bad96740 1 --2- 172.21.15.6:0/2212941476 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358b9820 0x55c0358b9c00 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.530+0000 7f77bad96740 1 -- 172.21.15.6:0/2212941476 >> 172.21.15.6:0/2212941476 conn(0x55c0358b5d30 msgr2=0x55c0358b6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.530+0000 7f77bad96740 1 -- 172.21.15.6:0/2212941476 shutdown_connections 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.530+0000 7f77bad96740 1 -- 172.21.15.6:0/2212941476 wait complete. 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.530+0000 7f77b5400640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f77980a3d90 0x7f77980a6250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55c0358badc0 tx=0x7f77a4005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.530+0000 7f77bad96740 1 -- start start 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.530+0000 7f77bad96740 1 Processor -- start 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.531+0000 7f77bad96740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358b9820 0x55c0358c1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.531+0000 7f77bad96740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358c1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.531+0000 7f77bad96740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358c4a00 0x55c0358c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.531+0000 7f77bad96740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55c0358c7450 con 0x55c0358ba140 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.531+0000 7f77bad96740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c0357d9310 con 0x55c0358b9820 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.531+0000 7f77bad96740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c0358c75d0 con 0x55c0358c4a00 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.531+0000 7f77bad96740 10 client.0 _ll_register_callbacks cb 0x55c0357a7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.531+0000 7f77b4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358c1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.531+0000 7f77b5400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358b9820 0x55c0358c1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.531+0000 7f77b4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358c1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45714/0 (socket says 172.21.15.6:45714) 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.531+0000 7f77b4a00640 1 -- 172.21.15.6:0/1412601229 learned_addr learned my addr 172.21.15.6:0/1412601229 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.531+0000 7f77b5e00640 1 --2- 172.21.15.6:0/1412601229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358c4a00 0x55c0358c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f77b4a00640 1 -- 172.21.15.6:0/1412601229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358c4a00 msgr2=0x55c0358c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f77b4a00640 1 --2- 172.21.15.6:0/1412601229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358c4a00 0x55c0358c23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f77b4a00640 1 -- 172.21.15.6:0/1412601229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358b9820 msgr2=0x55c0358c1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f77b4a00640 1 --2- 172.21.15.6:0/1412601229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358b9820 0x55c0358c1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f77b4a00640 1 -- 172.21.15.6:0/1412601229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0355fe620 con 0x55c0358ba140 2024-04-26T02:01:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f77b4a00640 1 --2- 172.21.15.6:0/1412601229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358c1e60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f779c09c980 tx=0x7f779c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55412380fa34716 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f7796a00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f7796a00640 1 -- 172.21.15.6:0/1412601229 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c0d6030 con 0x55c0358ba140 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f7796a00640 1 -- 172.21.15.6:0/1412601229 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7f779c0ce420 con 0x55c0358ba140 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f77bad96740 1 -- 172.21.15.6:0/1412601229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77a8036030 con 0x55c0358ba140 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f7796a00640 1 -- 172.21.15.6:0/1412601229 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f779c09c350 con 0x55c0358ba140 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f77bad96740 10 client.16830 fetch_fsmap learned FSMap version 49 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.532+0000 7f77bad96740 1 -- 172.21.15.6:0/1412601229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55c0358d3650 con 0x55c0358ba140 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.533+0000 7f7796a00640 1 -- 172.21.15.6:0/1412601229 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77a8036030 con 0x55c0358ba140 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.533+0000 7f7796a00640 1 -- 172.21.15.6:0/1412601229 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f779c099060 con 0x55c0358ba140 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.533+0000 7f77bad96740 10 client.16830 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.533+0000 7f77bad96740 10 client.16830 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.533+0000 7f77bad96740 20 client.16830 populate_metadata read hostname 'smithi006' 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.533+0000 7f77bad96740 1 --2- 172.21.15.6:0/1412601229 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c0358d4b80 0x55c0358d6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.533+0000 7f77bad96740 4 client.16830 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:01:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.533+0000 7f77bad96740 1 -- 172.21.15.6:0/1412601229 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0358338b0 con 0x55c0358d4b80 2024-04-26T02:01:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.534+0000 7f77b5400640 1 --2- 172.21.15.6:0/1412601229 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c0358d4b80 0x55c0358d6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.534+0000 7f77b5400640 1 --2- 172.21.15.6:0/1412601229 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c0358d4b80 0x55c0358d6fb0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95135f2c014face server_cookie=e2fe5dbcc78248f6 in_seq=0 out_seq=0 2024-04-26T02:01:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.534+0000 7f7796a00640 10 client.16830 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.535+0000 7f7796a00640 1 -- 172.21.15.6:0/1412601229 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0358338b0 con 0x55c0358d4b80 2024-04-26T02:01:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.535+0000 7f7796a00640 10 client.16830 handle_command_reply: tid=0 2024-04-26T02:01:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.535+0000 7f77bad96740 1 client.16830 shutdown 2024-04-26T02:01:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.535+0000 7f77bad96740 1 -- 172.21.15.6:0/1412601229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 msgr2=0x55c0358c1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.535+0000 7f77bad96740 1 --2- 172.21.15.6:0/1412601229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358c1e60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f779c09c980 tx=0x7f779c009580 comp rx=0 tx=0).stop 2024-04-26T02:01:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.536+0000 7f77bad96740 1 -- 172.21.15.6:0/1412601229 shutdown_connections 2024-04-26T02:01:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.536+0000 7f77bad96740 1 --2- 172.21.15.6:0/1412601229 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c0358d4b80 0x55c0358d6fb0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.536+0000 7f77bad96740 1 --2- 172.21.15.6:0/1412601229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358c1e60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.536+0000 7f77bad96740 1 --2- 172.21.15.6:0/1412601229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358c4a00 0x55c0358c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.536+0000 7f77bad96740 1 --2- 172.21.15.6:0/1412601229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358b9820 0x55c0358c1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.536+0000 7f77bad96740 1 -- 172.21.15.6:0/1412601229 >> 172.21.15.6:0/1412601229 conn(0x55c0358b5d30 msgr2=0x55c0358bbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:12.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.536+0000 7f7796a00640 0 client.16830 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:12.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.536+0000 7f77bad96740 1 -- 172.21.15.6:0/1412601229 shutdown_connections 2024-04-26T02:01:12.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.536+0000 7f77bad96740 1 -- 172.21.15.6:0/1412601229 wait complete. 2024-04-26T02:01:12.538 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.536+0000 7f77bad96740 20 client.16830 trim_cache size 0 max 16384 2024-04-26T02:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77bad96740 1 Processor -- start 2024-04-26T02:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77bad96740 1 -- start start 2024-04-26T02:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77bad96740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358b9820 0x55c0358d0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77bad96740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358bc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77bad96740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358c4a00 0x55c0358dc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77bad96740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f77b01b6af0 con 0x55c0358ba140 2024-04-26T02:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77bad96740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f77b01b4330 con 0x55c0358c4a00 2024-04-26T02:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77bad96740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f77b01b21a0 con 0x55c0358b9820 2024-04-26T02:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77b4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358bc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77b4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358bc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45718/0 (socket says 172.21.15.6:45718) 2024-04-26T02:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77b4a00640 1 -- 172.21.15.6:0/2562313932 learned_addr learned my addr 172.21.15.6:0/2562313932 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77b5e00640 1 --2- 172.21.15.6:0/2562313932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358c4a00 0x55c0358dc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.552+0000 7f77b5400640 1 --2- 172.21.15.6:0/2562313932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358b9820 0x55c0358d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.553 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.553+0000 7f77b4a00640 1 -- 172.21.15.6:0/2562313932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358b9820 msgr2=0x55c0358d0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:12.553 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.553+0000 7f77b4a00640 1 --2- 172.21.15.6:0/2562313932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358b9820 0x55c0358d0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.553+0000 7f77b4a00640 1 -- 172.21.15.6:0/2562313932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358c4a00 msgr2=0x55c0358dc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.553+0000 7f77b4a00640 1 --2- 172.21.15.6:0/2562313932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358c4a00 0x55c0358dc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.553+0000 7f77b4a00640 1 -- 172.21.15.6:0/2562313932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0358c08d0 con 0x55c0358ba140 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.553+0000 7f77b4a00640 1 --2- 172.21.15.6:0/2562313932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358bc1e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f779c09c6f0 tx=0x7f779c098620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed1773a088f6533a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.553+0000 7f77a1c00640 1 -- 172.21.15.6:0/2562313932 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c0d6020 con 0x55c0358ba140 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.553+0000 7f77a1c00640 1 -- 172.21.15.6:0/2562313932 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f779c09d0f0 con 0x55c0358ba140 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.553+0000 7f77a1c00640 1 -- 172.21.15.6:0/2562313932 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c0b7070 con 0x55c0358ba140 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.554+0000 7f77bad96740 1 -- 172.21.15.6:0/2562313932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 msgr2=0x55c0358bc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.554+0000 7f77bad96740 1 --2- 172.21.15.6:0/2562313932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358bc1e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f779c09c6f0 tx=0x7f779c098620 comp rx=0 tx=0).stop 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.554+0000 7f77bad96740 1 -- 172.21.15.6:0/2562313932 shutdown_connections 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.554+0000 7f77bad96740 1 --2- 172.21.15.6:0/2562313932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358bc1e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.554+0000 7f77bad96740 1 --2- 172.21.15.6:0/2562313932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358c4a00 0x55c0358dc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.554+0000 7f77bad96740 1 --2- 172.21.15.6:0/2562313932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358b9820 0x55c0358d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.555 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.554+0000 7f77bad96740 1 -- 172.21.15.6:0/2562313932 >> 172.21.15.6:0/2562313932 conn(0x55c0358b5d30 msgr2=0x55c0358b6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:12.555 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.554+0000 7f77bad96740 1 -- 172.21.15.6:0/2562313932 shutdown_connections 2024-04-26T02:01:12.555 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.554+0000 7f77bad96740 1 -- 172.21.15.6:0/2562313932 wait complete. 2024-04-26T02:01:12.555 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.554+0000 7f77bad96740 1 -- start start 2024-04-26T02:01:12.555 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.555+0000 7f77bad96740 1 Processor -- start 2024-04-26T02:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.555+0000 7f77bad96740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358b9820 0x55c0358cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.555+0000 7f77bad96740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.555+0000 7f77bad96740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358c4a00 0x55c0358c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.555+0000 7f77bad96740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f77b01b27f0 con 0x55c0358ba140 2024-04-26T02:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.555+0000 7f77bad96740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f77b01b3c60 con 0x55c0358b9820 2024-04-26T02:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.555+0000 7f77bad96740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f77b01b2ec0 con 0x55c0358c4a00 2024-04-26T02:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.555+0000 7f77bad96740 10 client.0 _ll_register_callbacks cb 0x7f779c098570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.555+0000 7f77b4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.555+0000 7f77b4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45732/0 (socket says 172.21.15.6:45732) 2024-04-26T02:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.555+0000 7f77b4a00640 1 -- 172.21.15.6:0/1379697727 learned_addr learned my addr 172.21.15.6:0/1379697727 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f77b5400640 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358b9820 0x55c0358cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f77b5e00640 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358c4a00 0x55c0358c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f77b4a00640 1 -- 172.21.15.6:0/1379697727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358c4a00 msgr2=0x55c0358c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f77b4a00640 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358c4a00 0x55c0358c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f77b4a00640 1 -- 172.21.15.6:0/1379697727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358b9820 msgr2=0x55c0358cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f77b4a00640 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358b9820 0x55c0358cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f77b4a00640 1 -- 172.21.15.6:0/1379697727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0358ca4c0 con 0x55c0358ba140 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f77b5e00640 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358c4a00 0x55c0358c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f77b4a00640 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358ccd10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f779c0ce7a0 tx=0x7f779c099310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=849b89506c3bf4b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f7796a00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f7796a00640 1 -- 172.21.15.6:0/1379697727 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c0d6030 con 0x55c0358ba140 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f7796a00640 1 -- 172.21.15.6:0/1379697727 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7f779c004200 con 0x55c0358ba140 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.556+0000 7f77bad96740 1 -- 172.21.15.6:0/1379697727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f779c09d0f0 con 0x55c0358ba140 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.557+0000 7f7796a00640 1 -- 172.21.15.6:0/1379697727 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f779c0b7070 con 0x55c0358ba140 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.557+0000 7f77bad96740 10 client.16836 fetch_fsmap learned FSMap version 49 2024-04-26T02:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.557+0000 7f77bad96740 1 -- 172.21.15.6:0/1379697727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55c03593ada0 con 0x55c0358ba140 2024-04-26T02:01:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.557+0000 7f7796a00640 1 -- 172.21.15.6:0/1379697727 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f779c09d0f0 con 0x55c0358ba140 2024-04-26T02:01:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.557+0000 7f7796a00640 1 -- 172.21.15.6:0/1379697727 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f779c0871d0 con 0x55c0358ba140 2024-04-26T02:01:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.557+0000 7f77bad96740 10 client.16836 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:01:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.557+0000 7f77bad96740 10 client.16836 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:01:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.557+0000 7f77bad96740 20 client.16836 populate_metadata read hostname 'smithi006' 2024-04-26T02:01:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.557+0000 7f77bad96740 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c03593b1c0 0x55c03593d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.557+0000 7f77bad96740 4 client.16836 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:01:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.557+0000 7f77bad96740 1 -- 172.21.15.6:0/1379697727 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c03582d320 con 0x55c03593b1c0 2024-04-26T02:01:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.558+0000 7f77b5400640 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c03593b1c0 0x55c03593d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.558+0000 7f77b5400640 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c03593b1c0 0x55c03593d5e0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9b50bfab379612d server_cookie=c1ce51ef1994535f in_seq=0 out_seq=0 2024-04-26T02:01:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.558+0000 7f7796a00640 10 client.16836 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.596+0000 7f7796a00640 1 -- 172.21.15.6:0/1379697727 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c03582d320 con 0x55c03593b1c0 2024-04-26T02:01:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.596+0000 7f7796a00640 10 client.16836 handle_command_reply: tid=0 2024-04-26T02:01:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.596+0000 7f77bad96740 1 client.16836 shutdown 2024-04-26T02:01:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.596+0000 7f77bad96740 1 -- 172.21.15.6:0/1379697727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 msgr2=0x55c0358ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.596+0000 7f77bad96740 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358ccd10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f779c0ce7a0 tx=0x7f779c099310 comp rx=0 tx=0).stop 2024-04-26T02:01:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.596+0000 7f77bad96740 1 -- 172.21.15.6:0/1379697727 shutdown_connections 2024-04-26T02:01:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.596+0000 7f77bad96740 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c03593b1c0 0x55c03593d5e0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.596+0000 7f77bad96740 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c0358ba140 0x55c0358ccd10 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.596+0000 7f77bad96740 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c0358c4a00 0x55c0358c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.596+0000 7f77bad96740 1 --2- 172.21.15.6:0/1379697727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c0358b9820 0x55c0358cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.596+0000 7f7796a00640 0 client.16836 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:12.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.596+0000 7f77bad96740 1 -- 172.21.15.6:0/1379697727 >> 172.21.15.6:0/1379697727 conn(0x55c0358b5d30 msgr2=0x55c0358b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:12.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.597+0000 7f77bad96740 1 -- 172.21.15.6:0/1379697727 shutdown_connections 2024-04-26T02:01:12.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.597+0000 7f77bad96740 1 -- 172.21.15.6:0/1379697727 wait complete. 2024-04-26T02:01:12.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.597+0000 7f77bad96740 20 client.16836 trim_cache size 0 max 16384 2024-04-26T02:01:12.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:01:12.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:01:12.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:01:12.597 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:01:12.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.598+0000 7f77b7e00640 1 -- 172.21.15.6:0/3332505647 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f77980a3d90 msgr2=0x7f77980a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:12.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.598+0000 7f77b7e00640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f77980a3d90 0x7f77980a6250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55c0358badc0 tx=0x7f77a4005f90 comp rx=0 tx=0).stop 2024-04-26T02:01:12.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.598+0000 7f77b7e00640 1 -- 172.21.15.6:0/3332505647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77b0108f60 msgr2=0x7f77b01a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:12.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.598+0000 7f77b7e00640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77b0108f60 0x7f77b01a4e80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f779c00ea60 tx=0x7f779c00ef20 comp rx=0 tx=0).stop 2024-04-26T02:01:12.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.598+0000 7f77b7e00640 1 -- 172.21.15.6:0/3332505647 shutdown_connections 2024-04-26T02:01:12.598 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.598+0000 7f77b7e00640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f77980a3d90 0x7f77980a6250 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.598+0000 7f77b7e00640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77b010e660 0x7f77b019eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.598+0000 7f77b7e00640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77b0108f60 0x7f77b01a4e80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.598+0000 7f77b7e00640 1 --2- 172.21.15.6:0/3332505647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77b01085a0 0x7f77b01a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:12.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.598+0000 7f77b7e00640 1 -- 172.21.15.6:0/3332505647 >> 172.21.15.6:0/3332505647 conn(0x7f77b0076460 msgr2=0x7f77b0077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:12.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.598+0000 7f77b7e00640 1 -- 172.21.15.6:0/3332505647 shutdown_connections 2024-04-26T02:01:12.599 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:12.598+0000 7f77b7e00640 1 -- 172.21.15.6:0/3332505647 wait complete. 2024-04-26T02:01:12.922 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:12 smithi006 ceph-mon[28419]: pgmap v486: 97 pgs: 97 active+clean; 421 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.8 MiB/s wr, 117 op/s 2024-04-26T02:01:12.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:12 smithi029 ceph-mon[32775]: pgmap v486: 97 pgs: 97 active+clean; 421 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.8 MiB/s wr, 117 op/s 2024-04-26T02:01:13.099 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 11.34 MB/sec warmup 96 sec latency 30646.207 ms 2024-04-26T02:01:13.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:12 smithi188 ceph-mon[32751]: pgmap v486: 97 pgs: 97 active+clean; 421 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.8 MiB/s wr, 117 op/s 2024-04-26T02:01:14.099 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17097 11.22 MB/sec warmup 97 sec latency 31646.336 ms 2024-04-26T02:01:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.561+0000 7feab3400700 1 -- 172.21.15.6:0/1814944381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaac108a30 msgr2=0x7feaac108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.561+0000 7feab3400700 1 --2- 172.21.15.6:0/1814944381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaac108a30 0x7feaac108e10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009990 tx=0x7feaa0009ca0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.561+0000 7feab3400700 1 -- 172.21.15.6:0/1814944381 shutdown_connections 2024-04-26T02:01:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.561+0000 7feab3400700 1 --2- 172.21.15.6:0/1814944381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feaac06b0f0 0x7feaac10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.561+0000 7feab3400700 1 --2- 172.21.15.6:0/1814944381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaac108a30 0x7feaac108e10 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.561+0000 7feab3400700 1 --2- 172.21.15.6:0/1814944381 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feaac06a750 0x7feaac06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.561+0000 7feab3400700 1 -- 172.21.15.6:0/1814944381 >> 172.21.15.6:0/1814944381 conn(0x7feaac0788f0 msgr2=0x7feaac078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.561+0000 7feab3400700 1 -- 172.21.15.6:0/1814944381 shutdown_connections 2024-04-26T02:01:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.562+0000 7feab3400700 1 -- 172.21.15.6:0/1814944381 wait complete. 2024-04-26T02:01:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.563+0000 7feab3400700 1 Processor -- start 2024-04-26T02:01:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.563+0000 7feab3400700 1 -- start start 2024-04-26T02:01:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.563+0000 7feab3400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feaac06a750 0x7feaac199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.563+0000 7feab3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feaac06b0f0 0x7feaac199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.563+0000 7feab3400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaac108a30 0x7feaac19d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.563+0000 7feab3400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feaac112bd0 con 0x7feaac06a750 2024-04-26T02:01:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.563+0000 7feab3400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feaac112a50 con 0x7feaac108a30 2024-04-26T02:01:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.563+0000 7feab3400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7feaac112d50 con 0x7feaac06b0f0 2024-04-26T02:01:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.563+0000 7feab0a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feaac06a750 0x7feaac199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.564+0000 7feab0a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feaac06a750 0x7feaac199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45756/0 (socket says 172.21.15.6:45756) 2024-04-26T02:01:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.564+0000 7feab0a00700 1 -- 172.21.15.6:0/2181767843 learned_addr learned my addr 172.21.15.6:0/2181767843 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.564+0000 7feab1400700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaac108a30 0x7feaac19d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.564+0000 7feaabe00700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feaac06b0f0 0x7feaac199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.564+0000 7feab0a00700 1 -- 172.21.15.6:0/2181767843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feaac06b0f0 msgr2=0x7feaac199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.564+0000 7feab0a00700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feaac06b0f0 0x7feaac199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.564+0000 7feab0a00700 1 -- 172.21.15.6:0/2181767843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaac108a30 msgr2=0x7feaac19d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.564+0000 7feab0a00700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaac108a30 0x7feaac19d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.564+0000 7feab0a00700 1 -- 172.21.15.6:0/2181767843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac19e130 con 0x7feaac06a750 2024-04-26T02:01:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.564+0000 7feab0a00700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feaac06a750 0x7feaac199120 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7feaa0000c00 tx=0x7feaa000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.565+0000 7feaa9600700 1 -- 172.21.15.6:0/2181767843 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0027070 con 0x7feaac06a750 2024-04-26T02:01:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.565+0000 7feaa9600700 1 -- 172.21.15.6:0/2181767843 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaa0022470 con 0x7feaac06a750 2024-04-26T02:01:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.565+0000 7feaa9600700 1 -- 172.21.15.6:0/2181767843 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0017650 con 0x7feaac06a750 2024-04-26T02:01:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.565+0000 7feab3400700 1 -- 172.21.15.6:0/2181767843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaac19e3c0 con 0x7feaac06a750 2024-04-26T02:01:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.565+0000 7feab3400700 1 -- 172.21.15.6:0/2181767843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaac1a5ca0 con 0x7feaac06a750 2024-04-26T02:01:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.568+0000 7feaa9600700 1 -- 172.21.15.6:0/2181767843 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7feaa002a070 con 0x7feaac06a750 2024-04-26T02:01:14.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.570+0000 7feaa9600700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fea940a3e30 0x7fea940a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.570+0000 7feabddfbb80 1 Processor -- start 2024-04-26T02:01:14.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.570+0000 7feaa9600700 1 -- 172.21.15.6:0/2181767843 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7feaa00d1470 con 0x7feaac06a750 2024-04-26T02:01:14.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.570+0000 7feabddfbb80 1 -- start start 2024-04-26T02:01:14.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.570+0000 7feaabe00700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fea940a3e30 0x7fea940a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.570+0000 7feabddfbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf7d40 0x56154bcf8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feabddfbb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf8660 0x56154bd073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feabddfbb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd09cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feabddfbb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56154bc4af90 con 0x56154bcf8660 2024-04-26T02:01:14.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feabddfbb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56154bbf1830 con 0x56154bd078e0 2024-04-26T02:01:14.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feabddfbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56154bc46f50 con 0x56154bcf7d40 2024-04-26T02:01:14.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feaabe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf8660 0x56154bd073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feab0a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf7d40 0x56154bcf8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feaabe00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf8660 0x56154bd073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45768/0 (socket says 172.21.15.6:45768) 2024-04-26T02:01:14.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feaabe00700 1 -- 172.21.15.6:0/144968166 learned_addr learned my addr 172.21.15.6:0/144968166 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:14.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feab1400700 1 --2- 172.21.15.6:0/144968166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd09cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feaabe00700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fea940a3e30 0x7fea940a62f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feaac079030 tx=0x7fea98011720 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:14.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feaabe00700 1 -- 172.21.15.6:0/144968166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf7d40 msgr2=0x56154bcf8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:14.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feaabe00700 1 --2- 172.21.15.6:0/144968166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf7d40 0x56154bcf8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feaabe00700 1 -- 172.21.15.6:0/144968166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 msgr2=0x56154bd09cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:14.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feaabe00700 1 --2- 172.21.15.6:0/144968166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd09cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feaabe00700 1 -- 172.21.15.6:0/144968166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56154bbe7780 con 0x56154bcf8660 2024-04-26T02:01:14.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.571+0000 7feab0a00700 1 --2- 172.21.15.6:0/144968166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf7d40 0x56154bcf8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:01:14.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.572+0000 7feab1400700 1 --2- 172.21.15.6:0/144968166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd09cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:01:14.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.572+0000 7feaabe00700 1 --2- 172.21.15.6:0/144968166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf8660 0x56154bd073a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fea98011f90 tx=0x7fea9801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f978cd26ee03778 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:14.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.572+0000 7fea9e000700 1 -- 172.21.15.6:0/144968166 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea98017040 con 0x56154bcf8660 2024-04-26T02:01:14.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.572+0000 7fea9e000700 1 -- 172.21.15.6:0/144968166 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea98019050 con 0x56154bcf8660 2024-04-26T02:01:14.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.572+0000 7fea9e000700 1 -- 172.21.15.6:0/144968166 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea980253f0 con 0x56154bcf8660 2024-04-26T02:01:14.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.573+0000 7feabddfbb80 1 -- 172.21.15.6:0/144968166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf8660 msgr2=0x56154bd073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:14.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.573+0000 7feabddfbb80 1 --2- 172.21.15.6:0/144968166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf8660 0x56154bd073a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fea98011f90 tx=0x7fea9801c000 comp rx=0 tx=0).stop 2024-04-26T02:01:14.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.573+0000 7feabddfbb80 1 -- 172.21.15.6:0/144968166 shutdown_connections 2024-04-26T02:01:14.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.573+0000 7feabddfbb80 1 --2- 172.21.15.6:0/144968166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd09cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.573+0000 7feabddfbb80 1 --2- 172.21.15.6:0/144968166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf7d40 0x56154bcf8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.573+0000 7feabddfbb80 1 --2- 172.21.15.6:0/144968166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf8660 0x56154bd073a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.573+0000 7feabddfbb80 1 -- 172.21.15.6:0/144968166 >> 172.21.15.6:0/144968166 conn(0x56154bc27050 msgr2=0x56154bcf4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:14.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.573+0000 7feabddfbb80 1 -- 172.21.15.6:0/144968166 shutdown_connections 2024-04-26T02:01:14.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.573+0000 7feabddfbb80 1 -- 172.21.15.6:0/144968166 wait complete. 2024-04-26T02:01:14.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.574+0000 7feabddfbb80 1 -- start start 2024-04-26T02:01:14.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.574+0000 7feabddfbb80 1 Processor -- start 2024-04-26T02:01:14.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feabddfbb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf7d40 0x56154bd07040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feabddfbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd04340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feabddfbb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd04880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feabddfbb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56154bd0a370 con 0x56154bcf7d40 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feabddfbb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56154bc04960 con 0x56154bd078e0 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feabddfbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56154bd0a4f0 con 0x56154bcf8660 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feabddfbb80 10 client.0 _ll_register_callbacks cb 0x56154bbe3820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feab0a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf7d40 0x56154bd07040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feab1400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd04880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feaabe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd04340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feab0a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf7d40 0x56154bd07040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45772/0 (socket says 172.21.15.6:45772) 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feab0a00700 1 -- 172.21.15.6:0/477600033 learned_addr learned my addr 172.21.15.6:0/477600033 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feab1400700 1 --2- 172.21.15.6:0/477600033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd04880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34032/0 (socket says 172.21.15.6:34032) 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feaabe00700 1 -- 172.21.15.6:0/477600033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 msgr2=0x56154bd04880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feaabe00700 1 --2- 172.21.15.6:0/477600033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd04880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feaabe00700 1 -- 172.21.15.6:0/477600033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf7d40 msgr2=0x56154bd07040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:14.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feaabe00700 1 --2- 172.21.15.6:0/477600033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf7d40 0x56154bd07040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.575+0000 7feaabe00700 1 -- 172.21.15.6:0/477600033 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56154bb989d0 con 0x56154bcf8660 2024-04-26T02:01:14.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.576+0000 7feaabe00700 1 --2- 172.21.15.6:0/477600033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd04340 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fea98011b20 tx=0x7fea98009740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f141645fb347abb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:14.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.576+0000 7fea92a00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:01:14.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.576+0000 7fea92a00700 1 -- 172.21.15.6:0/477600033 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea98017040 con 0x56154bcf8660 2024-04-26T02:01:14.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.576+0000 7fea92a00700 1 -- 172.21.15.6:0/477600033 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7fea98004910 con 0x56154bcf8660 2024-04-26T02:01:14.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.576+0000 7feabddfbb80 1 -- 172.21.15.6:0/477600033 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea98019050 con 0x56154bcf8660 2024-04-26T02:01:14.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.576+0000 7fea92a00700 1 -- 172.21.15.6:0/477600033 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fea9803b050 con 0x56154bcf8660 2024-04-26T02:01:14.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.576+0000 7feabddfbb80 10 client.27067 fetch_fsmap learned FSMap version 49 2024-04-26T02:01:14.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.577+0000 7feabddfbb80 1 -- 172.21.15.6:0/477600033 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56154bbe1bf0 con 0x56154bcf8660 2024-04-26T02:01:14.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.577+0000 7fea92a00700 1 -- 172.21.15.6:0/477600033 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea98019050 con 0x56154bcf8660 2024-04-26T02:01:14.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.577+0000 7fea92a00700 1 -- 172.21.15.6:0/477600033 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fea98024030 con 0x56154bcf8660 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.577+0000 7feabddfbb80 10 client.27067 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.577+0000 7feabddfbb80 10 client.27067 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.577+0000 7feabddfbb80 20 client.27067 populate_metadata read hostname 'smithi006' 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.577+0000 7feabddfbb80 1 --2- 172.21.15.6:0/477600033 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56154bd16fb0 0x56154bd193a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.577+0000 7feabddfbb80 4 client.27067 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.577+0000 7feabddfbb80 1 -- 172.21.15.6:0/477600033 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56154bc58780 con 0x56154bd16fb0 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.578+0000 7feab0a00700 1 --2- 172.21.15.6:0/477600033 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56154bd16fb0 0x56154bd193a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.578+0000 7feab0a00700 1 --2- 172.21.15.6:0/477600033 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56154bd16fb0 0x56154bd193a0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a25ec103ec2b4940 server_cookie=4ed03ff858effec6 in_seq=0 out_seq=0 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.578+0000 7fea92a00700 10 client.27067 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.579+0000 7fea92a00700 1 -- 172.21.15.6:0/477600033 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56154bc58780 con 0x56154bd16fb0 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.579+0000 7fea92a00700 10 client.27067 handle_command_reply: tid=0 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.579+0000 7feabddfbb80 1 client.27067 shutdown 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.579+0000 7feabddfbb80 1 -- 172.21.15.6:0/477600033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 msgr2=0x56154bd04340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:14.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.579+0000 7feabddfbb80 1 --2- 172.21.15.6:0/477600033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd04340 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fea98011b20 tx=0x7fea98009740 comp rx=0 tx=0).stop 2024-04-26T02:01:14.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.579+0000 7feabddfbb80 1 -- 172.21.15.6:0/477600033 shutdown_connections 2024-04-26T02:01:14.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.579+0000 7feabddfbb80 1 --2- 172.21.15.6:0/477600033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd04880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.579+0000 7feabddfbb80 1 --2- 172.21.15.6:0/477600033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf7d40 0x56154bd07040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.579+0000 7feabddfbb80 1 --2- 172.21.15.6:0/477600033 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56154bd16fb0 0x56154bd193a0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.580+0000 7feabddfbb80 1 --2- 172.21.15.6:0/477600033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd04340 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.580+0000 7feabddfbb80 1 -- 172.21.15.6:0/477600033 >> 172.21.15.6:0/477600033 conn(0x56154bc27050 msgr2=0x56154bcf3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:14.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.580+0000 7fea92a00700 0 client.27067 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:14.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.580+0000 7feabddfbb80 1 -- 172.21.15.6:0/477600033 shutdown_connections 2024-04-26T02:01:14.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.580+0000 7feabddfbb80 1 -- 172.21.15.6:0/477600033 wait complete. 2024-04-26T02:01:14.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.580+0000 7feabddfbb80 20 client.27067 trim_cache size 0 max 16384 2024-04-26T02:01:14.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.601+0000 7feabddfbb80 1 Processor -- start 2024-04-26T02:01:14.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.602+0000 7feabddfbb80 1 -- start start 2024-04-26T02:01:14.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.602+0000 7feabddfbb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf7d40 0x56154bd13270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.602+0000 7feabddfbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd1eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.602+0000 7feabddfbb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd1f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.602+0000 7feabddfbb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feaac1adb70 con 0x56154bcf7d40 2024-04-26T02:01:14.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.602+0000 7feabddfbb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feaac1ab370 con 0x56154bd078e0 2024-04-26T02:01:14.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.602+0000 7feabddfbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7feaac1a9200 con 0x56154bcf8660 2024-04-26T02:01:14.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.602+0000 7feab0a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf7d40 0x56154bd13270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.602+0000 7feaabe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd1eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.602+0000 7feab0a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf7d40 0x56154bd13270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45784/0 (socket says 172.21.15.6:45784) 2024-04-26T02:01:14.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.602+0000 7feab0a00700 1 -- 172.21.15.6:0/3180946558 learned_addr learned my addr 172.21.15.6:0/3180946558 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:14.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.602+0000 7feab1400700 1 --2- 172.21.15.6:0/3180946558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd1f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.603+0000 7feaabe00700 1 -- 172.21.15.6:0/3180946558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 msgr2=0x56154bd1f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:14.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.603+0000 7feaabe00700 1 --2- 172.21.15.6:0/3180946558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd1f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.603+0000 7feaabe00700 1 -- 172.21.15.6:0/3180946558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf7d40 msgr2=0x56154bd13270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:14.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.603+0000 7feaabe00700 1 --2- 172.21.15.6:0/3180946558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf7d40 0x56154bd13270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.603+0000 7feaabe00700 1 -- 172.21.15.6:0/3180946558 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56154bcf8a80 con 0x56154bcf8660 2024-04-26T02:01:14.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.603+0000 7feaabe00700 1 --2- 172.21.15.6:0/3180946558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd1eae0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fea98000c00 tx=0x7fea9801dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45c5aa495147e244 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:14.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.603+0000 7fea9d600700 1 -- 172.21.15.6:0/3180946558 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea98021020 con 0x56154bcf8660 2024-04-26T02:01:14.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.604+0000 7fea9d600700 1 -- 172.21.15.6:0/3180946558 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea98032070 con 0x56154bcf8660 2024-04-26T02:01:14.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.604+0000 7fea9d600700 1 -- 172.21.15.6:0/3180946558 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea98015050 con 0x56154bcf8660 2024-04-26T02:01:14.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.604+0000 7feabddfbb80 1 -- 172.21.15.6:0/3180946558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 msgr2=0x56154bd1eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:14.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.604+0000 7feabddfbb80 1 --2- 172.21.15.6:0/3180946558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd1eae0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fea98000c00 tx=0x7fea9801dab0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.604+0000 7feabddfbb80 1 -- 172.21.15.6:0/3180946558 shutdown_connections 2024-04-26T02:01:14.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.604+0000 7feabddfbb80 1 --2- 172.21.15.6:0/3180946558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bd078e0 0x56154bd1f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.604+0000 7feabddfbb80 1 --2- 172.21.15.6:0/3180946558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bcf7d40 0x56154bd13270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.604+0000 7feabddfbb80 1 --2- 172.21.15.6:0/3180946558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd1eae0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.604+0000 7feabddfbb80 1 -- 172.21.15.6:0/3180946558 >> 172.21.15.6:0/3180946558 conn(0x56154bc27050 msgr2=0x56154bd06400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:14.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.604+0000 7feabddfbb80 1 -- 172.21.15.6:0/3180946558 shutdown_connections 2024-04-26T02:01:14.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.604+0000 7feabddfbb80 1 -- 172.21.15.6:0/3180946558 wait complete. 2024-04-26T02:01:14.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.605+0000 7feabddfbb80 1 -- start start 2024-04-26T02:01:14.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.605+0000 7feabddfbb80 1 Processor -- start 2024-04-26T02:01:14.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feabddfbb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bcf7d40 0x56154bd0c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feabddfbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd0ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feabddfbb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bd078e0 0x56154bd7d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feabddfbb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feaac1a9850 con 0x56154bd078e0 2024-04-26T02:01:14.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feabddfbb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feaac1aaca0 con 0x56154bcf7d40 2024-04-26T02:01:14.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feabddfbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7feaac1a9f00 con 0x56154bcf8660 2024-04-26T02:01:14.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feabddfbb80 10 client.0 _ll_register_callbacks cb 0x56154bbe3820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:14.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feab1400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bd078e0 0x56154bd7d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feab0a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bcf7d40 0x56154bd0c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feaabe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd0ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feab1400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bd078e0 0x56154bd7d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45792/0 (socket says 172.21.15.6:45792) 2024-04-26T02:01:14.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feab0a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bcf7d40 0x56154bd0c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34050/0 (socket says 172.21.15.6:34050) 2024-04-26T02:01:14.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feab1400700 1 -- 172.21.15.6:0/712022488 learned_addr learned my addr 172.21.15.6:0/712022488 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:14.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feaabe00700 1 -- 172.21.15.6:0/712022488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bcf7d40 msgr2=0x56154bd0c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:14.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.606+0000 7feaabe00700 1 --2- 172.21.15.6:0/712022488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bcf7d40 0x56154bd0c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.607+0000 7feaabe00700 1 -- 172.21.15.6:0/712022488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bd078e0 msgr2=0x56154bd7d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:14.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.607+0000 7feaabe00700 1 --2- 172.21.15.6:0/712022488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bd078e0 0x56154bd7d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.607+0000 7feaabe00700 1 -- 172.21.15.6:0/712022488 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56154bd7d8a0 con 0x56154bcf8660 2024-04-26T02:01:14.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.607+0000 7feaabe00700 1 --2- 172.21.15.6:0/712022488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd0ca50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x56154bcf2c20 tx=0x7fea9802d6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9679e2595e11a044 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:14.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.607+0000 7fea92a00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.607+0000 7fea92a00700 1 -- 172.21.15.6:0/712022488 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea98017030 con 0x56154bcf8660 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.607+0000 7fea92a00700 1 -- 172.21.15.6:0/712022488 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7fea98048670 con 0x56154bcf8660 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.607+0000 7feabddfbb80 1 -- 172.21.15.6:0/712022488 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea98032070 con 0x56154bcf8660 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.608+0000 7fea92a00700 1 -- 172.21.15.6:0/712022488 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fea98015070 con 0x56154bcf8660 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.608+0000 7feabddfbb80 10 client.27073 fetch_fsmap learned FSMap version 49 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.608+0000 7feabddfbb80 1 -- 172.21.15.6:0/712022488 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56154bd7ea20 con 0x56154bcf8660 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.608+0000 7fea92a00700 1 -- 172.21.15.6:0/712022488 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea98032070 con 0x56154bcf8660 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.608+0000 7fea92a00700 1 -- 172.21.15.6:0/712022488 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fea98024040 con 0x56154bcf8660 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.608+0000 7feabddfbb80 10 client.27073 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.608+0000 7feabddfbb80 10 client.27073 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.608+0000 7feabddfbb80 20 client.27073 populate_metadata read hostname 'smithi006' 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.608+0000 7feabddfbb80 1 --2- 172.21.15.6:0/712022488 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56154bd7efc0 0x56154bd813e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.608+0000 7feabddfbb80 4 client.27073 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a6261f3e", "release": true, "await": true}] 2024-04-26T02:01:14.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.608+0000 7feabddfbb80 1 -- 172.21.15.6:0/712022488 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a6261f3e", "release": true, "await": true}) -- 0x56154bc52820 con 0x56154bd7efc0 2024-04-26T02:01:14.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.609+0000 7feab0a00700 1 --2- 172.21.15.6:0/712022488 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56154bd7efc0 0x56154bd813e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:14.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.609+0000 7feab0a00700 1 --2- 172.21.15.6:0/712022488 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56154bd7efc0 0x56154bd813e0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f490030863499ea server_cookie=61215178a053e235 in_seq=0 out_seq=0 2024-04-26T02:01:14.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.609+0000 7fea92a00700 10 client.27073 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:14.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.660+0000 7fea92a00700 1 -- 172.21.15.6:0/712022488 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56154bc52820 con 0x56154bd7efc0 2024-04-26T02:01:14.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.660+0000 7fea92a00700 10 client.27073 handle_command_reply: tid=0 2024-04-26T02:01:14.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.660+0000 7feabddfbb80 1 client.27073 shutdown 2024-04-26T02:01:14.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.661+0000 7feabddfbb80 1 -- 172.21.15.6:0/712022488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 msgr2=0x56154bd0ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:14.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.661+0000 7feabddfbb80 1 --2- 172.21.15.6:0/712022488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd0ca50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x56154bcf2c20 tx=0x7fea9802d6b0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.661+0000 7feabddfbb80 1 -- 172.21.15.6:0/712022488 shutdown_connections 2024-04-26T02:01:14.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.661+0000 7feabddfbb80 1 --2- 172.21.15.6:0/712022488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56154bd078e0 0x56154bd7d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.661+0000 7feabddfbb80 1 --2- 172.21.15.6:0/712022488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56154bcf7d40 0x56154bd0c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.661+0000 7feabddfbb80 1 --2- 172.21.15.6:0/712022488 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56154bd7efc0 0x56154bd813e0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.661+0000 7feabddfbb80 1 --2- 172.21.15.6:0/712022488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56154bcf8660 0x56154bd0ca50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.661+0000 7feabddfbb80 1 -- 172.21.15.6:0/712022488 >> 172.21.15.6:0/712022488 conn(0x56154bc27050 msgr2=0x56154bcf9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:14.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.661+0000 7fea92a00700 0 client.27073 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:14.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.662+0000 7feabddfbb80 1 -- 172.21.15.6:0/712022488 shutdown_connections 2024-04-26T02:01:14.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.662+0000 7feabddfbb80 1 -- 172.21.15.6:0/712022488 wait complete. 2024-04-26T02:01:14.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.662+0000 7feabddfbb80 20 client.27073 trim_cache size 0 max 16384 2024-04-26T02:01:14.663 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:01:14.663 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 49, 2024-04-26T02:01:14.663 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T02:01:14.663 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 48, 2024-04-26T02:01:14.663 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T02:01:14.663 INFO:teuthology.orchestra.run.smithi006.stdout: "a6261f3e": { 2024-04-26T02:01:14.663 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 48, 2024-04-26T02:01:14.663 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T02:01:14.663 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:01:14.663 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 84.6, 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:01:14.664 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:01:14.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.663+0000 7feab3400700 1 -- 172.21.15.6:0/2181767843 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fea940a3e30 msgr2=0x7fea940a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:14.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.663+0000 7feab3400700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fea940a3e30 0x7fea940a62f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feaac079030 tx=0x7fea98011720 comp rx=0 tx=0).stop 2024-04-26T02:01:14.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.663+0000 7feab3400700 1 -- 172.21.15.6:0/2181767843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feaac06a750 msgr2=0x7feaac199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:14.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.663+0000 7feab3400700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feaac06a750 0x7feaac199120 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7feaa0000c00 tx=0x7feaa000bfd0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.664+0000 7feab3400700 1 -- 172.21.15.6:0/2181767843 shutdown_connections 2024-04-26T02:01:14.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.664+0000 7feab3400700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fea940a3e30 0x7fea940a62f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.664+0000 7feab3400700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feaac108a30 0x7feaac19d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.664+0000 7feab3400700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feaac06a750 0x7feaac199120 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.664+0000 7feab3400700 1 --2- 172.21.15.6:0/2181767843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feaac06b0f0 0x7feaac199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:14.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.664+0000 7feab3400700 1 -- 172.21.15.6:0/2181767843 >> 172.21.15.6:0/2181767843 conn(0x7feaac0788f0 msgr2=0x7feaac0ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:14.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.664+0000 7feab3400700 1 -- 172.21.15.6:0/2181767843 shutdown_connections 2024-04-26T02:01:14.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:01:14.664+0000 7feab3400700 1 -- 172.21.15.6:0/2181767843 wait complete. 2024-04-26T02:01:14.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:14 smithi029 ceph-mon[32775]: pgmap v487: 97 pgs: 97 active+clean; 421 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.7 MiB/s wr, 107 op/s 2024-04-26T02:01:15.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:14 smithi006 ceph-mon[28419]: pgmap v487: 97 pgs: 97 active+clean; 421 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.7 MiB/s wr, 107 op/s 2024-04-26T02:01:15.099 INFO:tasks.workunit.client.0.smithi006.stdout: 10 17441 11.31 MB/sec warmup 98 sec latency 32222.828 ms 2024-04-26T02:01:15.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:14 smithi188 ceph-mon[32751]: pgmap v487: 97 pgs: 97 active+clean; 421 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.7 MiB/s wr, 107 op/s 2024-04-26T02:01:15.534 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a6261f3e', seconds elapsed: 40.7 2024-04-26T02:01:16.100 INFO:tasks.workunit.client.0.smithi006.stdout: 10 18231 11.70 MB/sec warmup 99 sec latency 55.401 ms 2024-04-26T02:01:16.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:16 smithi029 ceph-mon[32775]: pgmap v488: 97 pgs: 97 active+clean; 421 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 1.5 MiB/s wr, 94 op/s 2024-04-26T02:01:17.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:16 smithi006 ceph-mon[28419]: pgmap v488: 97 pgs: 97 active+clean; 421 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 1.5 MiB/s wr, 94 op/s 2024-04-26T02:01:17.099 INFO:tasks.workunit.client.0.smithi006.stdout: 10 18846 11.97 MB/sec warmup 100 sec latency 49.826 ms 2024-04-26T02:01:17.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:16 smithi188 ceph-mon[32751]: pgmap v488: 97 pgs: 97 active+clean; 421 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 1.5 MiB/s wr, 94 op/s 2024-04-26T02:01:18.100 INFO:tasks.workunit.client.0.smithi006.stdout: 10 19416 12.05 MB/sec warmup 101 sec latency 68.598 ms 2024-04-26T02:01:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:18 smithi029 ceph-mon[32775]: pgmap v489: 97 pgs: 97 active+clean; 425 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 963 KiB/s wr, 98 op/s 2024-04-26T02:01:19.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:18 smithi006 ceph-mon[28419]: pgmap v489: 97 pgs: 97 active+clean; 425 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 963 KiB/s wr, 98 op/s 2024-04-26T02:01:19.101 INFO:tasks.workunit.client.0.smithi006.stdout: 10 20191 12.34 MB/sec warmup 102 sec latency 36.671 ms 2024-04-26T02:01:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:18 smithi188 ceph-mon[32751]: pgmap v489: 97 pgs: 97 active+clean; 425 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 963 KiB/s wr, 98 op/s 2024-04-26T02:01:20.100 INFO:tasks.workunit.client.0.smithi006.stdout: 10 20964 12.64 MB/sec warmup 103 sec latency 70.544 ms 2024-04-26T02:01:21.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:20 smithi006 ceph-mon[28419]: pgmap v490: 97 pgs: 97 active+clean; 425 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 963 KiB/s wr, 98 op/s 2024-04-26T02:01:21.100 INFO:tasks.workunit.client.0.smithi006.stdout: 10 21603 12.87 MB/sec warmup 104 sec latency 54.482 ms 2024-04-26T02:01:21.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:20 smithi188 ceph-mon[32751]: pgmap v490: 97 pgs: 97 active+clean; 425 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 963 KiB/s wr, 98 op/s 2024-04-26T02:01:21.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:20 smithi029 ceph-mon[32775]: pgmap v490: 97 pgs: 97 active+clean; 425 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 963 KiB/s wr, 98 op/s 2024-04-26T02:01:22.100 INFO:tasks.workunit.client.0.smithi006.stdout: 10 22302 13.12 MB/sec warmup 105 sec latency 52.643 ms 2024-04-26T02:01:23.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:22 smithi006 ceph-mon[28419]: pgmap v491: 97 pgs: 97 active+clean; 454 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.7 MiB/s wr, 330 op/s 2024-04-26T02:01:23.100 INFO:tasks.workunit.client.0.smithi006.stdout: 10 23063 13.29 MB/sec warmup 106 sec latency 66.818 ms 2024-04-26T02:01:23.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:22 smithi188 ceph-mon[32751]: pgmap v491: 97 pgs: 97 active+clean; 454 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.7 MiB/s wr, 330 op/s 2024-04-26T02:01:23.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:22 smithi029 ceph-mon[32775]: pgmap v491: 97 pgs: 97 active+clean; 454 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.7 MiB/s wr, 330 op/s 2024-04-26T02:01:24.100 INFO:tasks.workunit.client.0.smithi006.stdout: 10 23784 13.60 MB/sec warmup 107 sec latency 108.061 ms 2024-04-26T02:01:25.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:24 smithi006 ceph-mon[28419]: pgmap v492: 97 pgs: 97 active+clean; 454 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3.4 MiB/s wr, 296 op/s 2024-04-26T02:01:25.100 INFO:tasks.workunit.client.0.smithi006.stdout: 10 24474 13.81 MB/sec warmup 108 sec latency 49.317 ms 2024-04-26T02:01:25.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:24 smithi188 ceph-mon[32751]: pgmap v492: 97 pgs: 97 active+clean; 454 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3.4 MiB/s wr, 296 op/s 2024-04-26T02:01:25.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:24 smithi029 ceph-mon[32775]: pgmap v492: 97 pgs: 97 active+clean; 454 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3.4 MiB/s wr, 296 op/s 2024-04-26T02:01:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.770+0000 7f84ffe00640 1 -- 172.21.15.6:0/2238470084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500108d50 msgr2=0x7f8500115210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.770+0000 7f84ffe00640 1 --2- 172.21.15.6:0/2238470084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500108d50 0x7f8500115210 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009ca0 tx=0x7f84f802a950 comp rx=0 tx=0).stop 2024-04-26T02:01:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.770+0000 7f84ffe00640 1 -- 172.21.15.6:0/2238470084 shutdown_connections 2024-04-26T02:01:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.770+0000 7f84ffe00640 1 --2- 172.21.15.6:0/2238470084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500108d50 0x7f8500115210 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.770+0000 7f84ffe00640 1 --2- 172.21.15.6:0/2238470084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85001083b0 0x7f8500108810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.770+0000 7f84ffe00640 1 --2- 172.21.15.6:0/2238470084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f850010e470 0x7f850010e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.770+0000 7f84ffe00640 1 -- 172.21.15.6:0/2238470084 >> 172.21.15.6:0/2238470084 conn(0x7f8500077b00 msgr2=0x7f8500079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.771+0000 7f84ffe00640 1 -- 172.21.15.6:0/2238470084 shutdown_connections 2024-04-26T02:01:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.771+0000 7f84ffe00640 1 -- 172.21.15.6:0/2238470084 wait complete. 2024-04-26T02:01:25.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.772+0000 7f84ffe00640 1 Processor -- start 2024-04-26T02:01:25.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.772+0000 7f84ffe00640 1 -- start start 2024-04-26T02:01:25.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.772+0000 7f84ffe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85001083b0 0x7f85001a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.772+0000 7f84ffe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500108d50 0x7f85001a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.772+0000 7f84ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f850010e470 0x7f850019e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.772+0000 7f84ffe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8500118650 con 0x7f850010e470 2024-04-26T02:01:25.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.772+0000 7f84ffe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f85001184d0 con 0x7f8500108d50 2024-04-26T02:01:25.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.772+0000 7f84ffe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f85001187d0 con 0x7f85001083b0 2024-04-26T02:01:25.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.773+0000 7f84fea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85001083b0 0x7f85001a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.773+0000 7f84fea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85001083b0 0x7f85001a4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59654/0 (socket says 172.21.15.6:59654) 2024-04-26T02:01:25.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.773+0000 7f84fea00640 1 -- 172.21.15.6:0/1324851188 learned_addr learned my addr 172.21.15.6:0/1324851188 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:25.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.773+0000 7f84f7e00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500108d50 0x7f85001a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.773+0000 7f84ff400640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f850010e470 0x7f850019e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.773+0000 7f84fea00640 1 -- 172.21.15.6:0/1324851188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500108d50 msgr2=0x7f85001a4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:25.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.773+0000 7f84fea00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500108d50 0x7f85001a4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.773+0000 7f84fea00640 1 -- 172.21.15.6:0/1324851188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f850010e470 msgr2=0x7f850019e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:25.774 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.773+0000 7f84fea00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f850010e470 0x7f850019e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.773+0000 7f84fea00640 1 -- 172.21.15.6:0/1324851188 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850019f050 con 0x7f85001083b0 2024-04-26T02:01:25.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.773+0000 7f84ff400640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f850010e470 0x7f850019e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:01:25.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.773+0000 7f84f7e00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500108d50 0x7f85001a4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:01:25.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.774+0000 7f84fea00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85001083b0 0x7f85001a4720 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f84f000e9f0 tx=0x7f84f000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:25.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.774+0000 7f84f7400640 1 -- 172.21.15.6:0/1324851188 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f000cd80 con 0x7f85001083b0 2024-04-26T02:01:25.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.774+0000 7f84ffe00640 1 -- 172.21.15.6:0/1324851188 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850019f340 con 0x7f85001083b0 2024-04-26T02:01:25.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.774+0000 7f84f7400640 1 -- 172.21.15.6:0/1324851188 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84f00108e0 con 0x7f85001083b0 2024-04-26T02:01:25.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.774+0000 7f84f7400640 1 -- 172.21.15.6:0/1324851188 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0018980 con 0x7f85001083b0 2024-04-26T02:01:25.775 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.774+0000 7f84ffe00640 1 -- 172.21.15.6:0/1324851188 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85001ab5e0 con 0x7f85001083b0 2024-04-26T02:01:25.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.777+0000 7f850862f740 1 Processor -- start 2024-04-26T02:01:25.776 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.777+0000 7f850862f740 1 -- start start 2024-04-26T02:01:25.778 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f84f7400640 1 -- 172.21.15.6:0/1324851188 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f84f0010430 con 0x7f85001083b0 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f850862f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 0x563f3dc60d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f850862f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 0x563f3dc6b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f850862f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc6bb90 0x563f3dc6df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f850862f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x563f3dbc6900 con 0x563f3dc609b0 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f850862f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563f3db6cef0 con 0x563f3dc612d0 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f850862f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563f3dbc28c0 con 0x563f3dc6bb90 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f84fea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 0x563f3dc6b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f84ff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 0x563f3dc60d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f84ff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 0x563f3dc60d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51924/0 (socket says 172.21.15.6:51924) 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f84ff400640 1 -- 172.21.15.6:0/1094247059 learned_addr learned my addr 172.21.15.6:0/1094247059 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f84fea00640 1 --2- 172.21.15.6:0/1094247059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 0x563f3dc6b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34032/0 (socket says 172.21.15.6:34032) 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.778+0000 7f84f7e00640 1 --2- 172.21.15.6:0/1094247059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc6bb90 0x563f3dc6df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.779 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.779+0000 7f84ff400640 1 -- 172.21.15.6:0/1094247059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc6bb90 msgr2=0x563f3dc6df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.779+0000 7f84ff400640 1 --2- 172.21.15.6:0/1094247059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc6bb90 0x563f3dc6df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.779+0000 7f84ff400640 1 -- 172.21.15.6:0/1094247059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 msgr2=0x563f3dc6b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.779+0000 7f84ff400640 1 --2- 172.21.15.6:0/1094247059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 0x563f3dc6b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.779+0000 7f84ff400640 1 -- 172.21.15.6:0/1094247059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f3d8d9210 con 0x563f3dc609b0 2024-04-26T02:01:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.779+0000 7f84fea00640 1 --2- 172.21.15.6:0/1094247059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 0x563f3dc6b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:01:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.779+0000 7f84ff400640 1 --2- 172.21.15.6:0/1094247059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 0x563f3dc60d90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009a40 tx=0x7f84f802f3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b57baf54c08092a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.780+0000 7f84efe00640 1 -- 172.21.15.6:0/1094247059 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8034050 con 0x563f3dc609b0 2024-04-26T02:01:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.780+0000 7f84efe00640 1 -- 172.21.15.6:0/1094247059 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84f8003e20 con 0x563f3dc609b0 2024-04-26T02:01:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.780+0000 7f84efe00640 1 -- 172.21.15.6:0/1094247059 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f80043a0 con 0x563f3dc609b0 2024-04-26T02:01:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.780+0000 7f84f7400640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f84e40a3d00 0x7f84e40a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.780+0000 7f84f7400640 1 -- 172.21.15.6:0/1324851188 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7f84f00d0c20 con 0x7f85001083b0 2024-04-26T02:01:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.780+0000 7f84fea00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f84e40a3d00 0x7f84e40a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.780+0000 7f850862f740 1 -- 172.21.15.6:0/1094247059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 msgr2=0x563f3dc60d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.780+0000 7f850862f740 1 --2- 172.21.15.6:0/1094247059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 0x563f3dc60d90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009a40 tx=0x7f84f802f3a0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.781+0000 7f850862f740 1 -- 172.21.15.6:0/1094247059 shutdown_connections 2024-04-26T02:01:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.781+0000 7f850862f740 1 --2- 172.21.15.6:0/1094247059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc6bb90 0x563f3dc6df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.781+0000 7f850862f740 1 --2- 172.21.15.6:0/1094247059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 0x563f3dc6b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.781+0000 7f850862f740 1 --2- 172.21.15.6:0/1094247059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 0x563f3dc60d90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.781+0000 7f850862f740 1 -- 172.21.15.6:0/1094247059 >> 172.21.15.6:0/1094247059 conn(0x563f3dc5cec0 msgr2=0x563f3dc5d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.781+0000 7f850862f740 1 -- 172.21.15.6:0/1094247059 shutdown_connections 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.781+0000 7f850862f740 1 -- 172.21.15.6:0/1094247059 wait complete. 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.781+0000 7f84fea00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f84e40a3d00 0x7f84e40a61c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x563f3dc61f50 tx=0x7f84f00d46d0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.782+0000 7f850862f740 1 -- start start 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.782+0000 7f850862f740 1 Processor -- start 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f850862f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc609b0 0x563f3dc68ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f850862f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc612d0 0x563f3dc69000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f850862f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc6bb90 0x563f3dc69540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f850862f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x563f3dc6e760 con 0x563f3dc6bb90 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f850862f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563f3db80310 con 0x563f3dc609b0 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f850862f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563f3dc6e8e0 con 0x563f3dc612d0 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f850862f740 10 client.0 _ll_register_callbacks cb 0x563f3db4ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f84f7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc612d0 0x563f3dc69000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f84f7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc612d0 0x563f3dc69000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59672/0 (socket says 172.21.15.6:59672) 2024-04-26T02:01:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f84f7e00640 1 -- 172.21.15.6:0/2239891159 learned_addr learned my addr 172.21.15.6:0/2239891159 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f84ff400640 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc609b0 0x563f3dc68ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f84f7e00640 1 -- 172.21.15.6:0/2239891159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc609b0 msgr2=0x563f3dc68ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f84f7e00640 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc609b0 0x563f3dc68ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f84f7e00640 1 -- 172.21.15.6:0/2239891159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc6bb90 msgr2=0x563f3dc69540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f84f7e00640 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc6bb90 0x563f3dc69540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f84f7e00640 1 -- 172.21.15.6:0/2239891159 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f3d9a5620 con 0x563f3dc612d0 2024-04-26T02:01:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.783+0000 7f84ff400640 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc609b0 0x563f3dc68ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:01:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.784+0000 7f84f7e00640 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc612d0 0x563f3dc69000 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f84e8012d20 tx=0x7f84e800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b00367791ef195b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:25.786 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.784+0000 7f84ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:01:25.786 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.784+0000 7f84ecc00640 1 -- 172.21.15.6:0/2239891159 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8012110 con 0x563f3dc612d0 2024-04-26T02:01:25.786 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.784+0000 7f84ecc00640 1 -- 172.21.15.6:0/2239891159 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7f84e8004050 con 0x563f3dc612d0 2024-04-26T02:01:25.786 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.784+0000 7f850862f740 1 -- 172.21.15.6:0/2239891159 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84f8003e20 con 0x563f3dc612d0 2024-04-26T02:01:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.784+0000 7f84ecc00640 1 -- 172.21.15.6:0/2239891159 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f84e80122b0 con 0x563f3dc612d0 2024-04-26T02:01:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.785+0000 7f850862f740 10 client.27085 fetch_fsmap learned FSMap version 49 2024-04-26T02:01:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.785+0000 7f850862f740 1 -- 172.21.15.6:0/2239891159 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x563f3dc7aa80 con 0x563f3dc612d0 2024-04-26T02:01:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.786+0000 7f84ecc00640 1 -- 172.21.15.6:0/2239891159 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84f8003e20 con 0x563f3dc612d0 2024-04-26T02:01:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.786+0000 7f84ecc00640 1 -- 172.21.15.6:0/2239891159 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f84e8010430 con 0x563f3dc612d0 2024-04-26T02:01:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.786+0000 7f850862f740 10 client.27085 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:01:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.786+0000 7f850862f740 10 client.27085 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:01:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.786+0000 7f850862f740 20 client.27085 populate_metadata read hostname 'smithi006' 2024-04-26T02:01:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.786+0000 7f850862f740 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563f3dc7bf90 0x563f3dc7e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.786+0000 7f850862f740 4 client.27085 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:01:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.786+0000 7f850862f740 1 -- 172.21.15.6:0/2239891159 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f3dbda8b0 con 0x563f3dc7bf90 2024-04-26T02:01:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.786+0000 7f84ff400640 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563f3dc7bf90 0x563f3dc7e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.788+0000 7f84ff400640 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563f3dc7bf90 0x563f3dc7e3c0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=197dcb173d4a1718 server_cookie=604e1328ca60ee3c in_seq=0 out_seq=0 2024-04-26T02:01:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.788+0000 7f84ecc00640 10 client.27085 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:25.789 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.789+0000 7f84ecc00640 1 -- 172.21.15.6:0/2239891159 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f3dbda8b0 con 0x563f3dc7bf90 2024-04-26T02:01:25.789 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.789+0000 7f84ecc00640 10 client.27085 handle_command_reply: tid=0 2024-04-26T02:01:25.789 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.789+0000 7f850862f740 1 client.27085 shutdown 2024-04-26T02:01:25.789 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.789+0000 7f850862f740 1 -- 172.21.15.6:0/2239891159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc612d0 msgr2=0x563f3dc69000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:25.789 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.789+0000 7f850862f740 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc612d0 0x563f3dc69000 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f84e8012d20 tx=0x7f84e800f790 comp rx=0 tx=0).stop 2024-04-26T02:01:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.789+0000 7f850862f740 1 -- 172.21.15.6:0/2239891159 shutdown_connections 2024-04-26T02:01:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.789+0000 7f850862f740 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563f3dc7bf90 0x563f3dc7e3c0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.789+0000 7f850862f740 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc6bb90 0x563f3dc69540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.789+0000 7f850862f740 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc612d0 0x563f3dc69000 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.789+0000 7f850862f740 1 --2- 172.21.15.6:0/2239891159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc609b0 0x563f3dc68ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.789+0000 7f84ecc00640 0 client.27085 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.789+0000 7f850862f740 1 -- 172.21.15.6:0/2239891159 >> 172.21.15.6:0/2239891159 conn(0x563f3dc5cec0 msgr2=0x563f3dc62c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.790+0000 7f850862f740 1 -- 172.21.15.6:0/2239891159 shutdown_connections 2024-04-26T02:01:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.790+0000 7f850862f740 1 -- 172.21.15.6:0/2239891159 wait complete. 2024-04-26T02:01:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.790+0000 7f850862f740 20 client.27085 trim_cache size 0 max 16384 2024-04-26T02:01:25.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.809+0000 7f850862f740 1 Processor -- start 2024-04-26T02:01:25.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.809+0000 7f850862f740 1 -- start start 2024-04-26T02:01:25.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.809+0000 7f850862f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 0x563f3dc77650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.809+0000 7f850862f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 0x563f3dc63370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.809+0000 7f850862f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc6bb90 0x563f3dc83cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.808 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.809+0000 7f850862f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f85001b6a20 con 0x563f3dc609b0 2024-04-26T02:01:25.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.809+0000 7f850862f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f85001b4260 con 0x563f3dc612d0 2024-04-26T02:01:25.809 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.809+0000 7f850862f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f85001b20d0 con 0x563f3dc6bb90 2024-04-26T02:01:25.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.810+0000 7f84f7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 0x563f3dc63370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.810+0000 7f84ff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 0x563f3dc77650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.810+0000 7f84f7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 0x563f3dc63370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34048/0 (socket says 172.21.15.6:34048) 2024-04-26T02:01:25.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.810+0000 7f84f7e00640 1 -- 172.21.15.6:0/787677162 learned_addr learned my addr 172.21.15.6:0/787677162 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:25.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.810+0000 7f84ff400640 1 --2- 172.21.15.6:0/787677162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc6bb90 0x563f3dc83cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.810+0000 7f84f7e00640 1 -- 172.21.15.6:0/787677162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc6bb90 msgr2=0x563f3dc83cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:25.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.810+0000 7f84f7e00640 1 --2- 172.21.15.6:0/787677162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc6bb90 0x563f3dc83cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.810 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.810+0000 7f84f7e00640 1 -- 172.21.15.6:0/787677162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 msgr2=0x563f3dc77650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.810+0000 7f84f7e00640 1 --2- 172.21.15.6:0/787677162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 0x563f3dc77650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.810+0000 7f84f7e00640 1 -- 172.21.15.6:0/787677162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f3dc67010 con 0x563f3dc612d0 2024-04-26T02:01:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.810+0000 7f84ff400640 1 --2- 172.21.15.6:0/787677162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 0x563f3dc77650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:01:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.810+0000 7f84f7e00640 1 --2- 172.21.15.6:0/787677162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 0x563f3dc63370 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f84e8001eb0 tx=0x7f84e800fc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e841774622f90c9b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.811+0000 7f84ef400640 1 -- 172.21.15.6:0/787677162 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8028040 con 0x563f3dc612d0 2024-04-26T02:01:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.811+0000 7f84ef400640 1 -- 172.21.15.6:0/787677162 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84e8010430 con 0x563f3dc612d0 2024-04-26T02:01:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.811+0000 7f84ef400640 1 -- 172.21.15.6:0/787677162 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e80035f0 con 0x563f3dc612d0 2024-04-26T02:01:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.812+0000 7f850862f740 1 -- 172.21.15.6:0/787677162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 msgr2=0x563f3dc63370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.812+0000 7f850862f740 1 --2- 172.21.15.6:0/787677162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 0x563f3dc63370 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f84e8001eb0 tx=0x7f84e800fc90 comp rx=0 tx=0).stop 2024-04-26T02:01:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.812+0000 7f850862f740 1 -- 172.21.15.6:0/787677162 shutdown_connections 2024-04-26T02:01:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.812+0000 7f850862f740 1 --2- 172.21.15.6:0/787677162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc6bb90 0x563f3dc83cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.812+0000 7f850862f740 1 --2- 172.21.15.6:0/787677162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc612d0 0x563f3dc63370 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.812+0000 7f850862f740 1 --2- 172.21.15.6:0/787677162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc609b0 0x563f3dc77650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.812+0000 7f850862f740 1 -- 172.21.15.6:0/787677162 >> 172.21.15.6:0/787677162 conn(0x563f3dc5cec0 msgr2=0x563f3dc69310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.812+0000 7f850862f740 1 -- 172.21.15.6:0/787677162 shutdown_connections 2024-04-26T02:01:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.812+0000 7f850862f740 1 -- 172.21.15.6:0/787677162 wait complete. 2024-04-26T02:01:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.813+0000 7f850862f740 1 -- start start 2024-04-26T02:01:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.813+0000 7f850862f740 1 Processor -- start 2024-04-26T02:01:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.813+0000 7f850862f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc609b0 0x563f3dc71660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.813+0000 7f850862f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc612d0 0x563f3dc71ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.813+0000 7f850862f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc6bb90 0x563f3dc720e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.813+0000 7f850862f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f85001b2720 con 0x563f3dc612d0 2024-04-26T02:01:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.813+0000 7f850862f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f85001b3b90 con 0x563f3dc6bb90 2024-04-26T02:01:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.813+0000 7f850862f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f85001b2df0 con 0x563f3dc609b0 2024-04-26T02:01:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.814+0000 7f850862f740 10 client.0 _ll_register_callbacks cb 0x7f84e8012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.814+0000 7f84f7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc612d0 0x563f3dc71ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.814+0000 7f84ff400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc609b0 0x563f3dc71660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.814+0000 7f84ff400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc609b0 0x563f3dc71660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59688/0 (socket says 172.21.15.6:59688) 2024-04-26T02:01:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.814+0000 7f84ff400640 1 -- 172.21.15.6:0/2577420342 learned_addr learned my addr 172.21.15.6:0/2577420342 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.814+0000 7f84ff400640 1 --2- 172.21.15.6:0/2577420342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc6bb90 0x563f3dc720e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.814+0000 7f84ff400640 1 -- 172.21.15.6:0/2577420342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc6bb90 msgr2=0x563f3dc720e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.814+0000 7f84ff400640 1 --2- 172.21.15.6:0/2577420342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc6bb90 0x563f3dc720e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.814+0000 7f84ff400640 1 -- 172.21.15.6:0/2577420342 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc612d0 msgr2=0x563f3dc71ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.814+0000 7f84ff400640 1 --2- 172.21.15.6:0/2577420342 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc612d0 0x563f3dc71ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.814+0000 7f84ff400640 1 -- 172.21.15.6:0/2577420342 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f3dce1620 con 0x563f3dc609b0 2024-04-26T02:01:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.815+0000 7f84ff400640 1 --2- 172.21.15.6:0/2577420342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc609b0 0x563f3dc71660 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009ca0 tx=0x7f84f8009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1a9c2ed19d5591e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.815+0000 7f84ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:01:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.816+0000 7f84ecc00640 1 -- 172.21.15.6:0/2577420342 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8051740 con 0x563f3dc609b0 2024-04-26T02:01:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.816+0000 7f850862f740 1 -- 172.21.15.6:0/2577420342 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84e8010430 con 0x563f3dc609b0 2024-04-26T02:01:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.816+0000 7f84ecc00640 1 -- 172.21.15.6:0/2577420342 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7f84f8048040 con 0x563f3dc609b0 2024-04-26T02:01:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.816+0000 7f84ecc00640 1 -- 172.21.15.6:0/2577420342 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f84f80518e0 con 0x563f3dc609b0 2024-04-26T02:01:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.816+0000 7f850862f740 10 client.27091 fetch_fsmap learned FSMap version 49 2024-04-26T02:01:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.816+0000 7f850862f740 1 -- 172.21.15.6:0/2577420342 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x563f3dce29c0 con 0x563f3dc609b0 2024-04-26T02:01:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.817+0000 7f84ecc00640 1 -- 172.21.15.6:0/2577420342 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84e8010430 con 0x563f3dc609b0 2024-04-26T02:01:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.817+0000 7f84ecc00640 1 -- 172.21.15.6:0/2577420342 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f84f8041070 con 0x563f3dc609b0 2024-04-26T02:01:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.817+0000 7f850862f740 10 client.27091 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:01:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.817+0000 7f850862f740 10 client.27091 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:01:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.817+0000 7f850862f740 20 client.27091 populate_metadata read hostname 'smithi006' 2024-04-26T02:01:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.817+0000 7f850862f740 1 --2- 172.21.15.6:0/2577420342 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563f3dce2d00 0x563f3dce5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.817+0000 7f850862f740 4 client.27091 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:01:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.817+0000 7f850862f740 1 -- 172.21.15.6:0/2577420342 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563f3dbd4320 con 0x563f3dce2d00 2024-04-26T02:01:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.817+0000 7f84f7e00640 1 --2- 172.21.15.6:0/2577420342 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563f3dce2d00 0x563f3dce5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.818+0000 7f84f7e00640 1 --2- 172.21.15.6:0/2577420342 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563f3dce2d00 0x563f3dce5120 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b35b401b8a315f9d server_cookie=b390e37a0c4bf4a5 in_seq=0 out_seq=0 2024-04-26T02:01:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:25.818+0000 7f84ecc00640 10 client.27091 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:26.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.003+0000 7f84ecc00640 1 -- 172.21.15.6:0/2577420342 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563f3dbd4320 con 0x563f3dce2d00 2024-04-26T02:01:26.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.003+0000 7f84ecc00640 10 client.27091 handle_command_reply: tid=0 2024-04-26T02:01:26.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.003+0000 7f850862f740 1 client.27091 shutdown 2024-04-26T02:01:26.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.004+0000 7f850862f740 1 -- 172.21.15.6:0/2577420342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc609b0 msgr2=0x563f3dc71660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:26.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.004+0000 7f850862f740 1 --2- 172.21.15.6:0/2577420342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc609b0 0x563f3dc71660 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009ca0 tx=0x7f84f8009360 comp rx=0 tx=0).stop 2024-04-26T02:01:26.004 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.004+0000 7f850862f740 1 -- 172.21.15.6:0/2577420342 shutdown_connections 2024-04-26T02:01:26.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.004+0000 7f850862f740 1 --2- 172.21.15.6:0/2577420342 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x563f3dce2d00 0x563f3dce5120 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:26.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.004+0000 7f850862f740 1 --2- 172.21.15.6:0/2577420342 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563f3dc612d0 0x563f3dc71ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:26.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.004+0000 7f850862f740 1 --2- 172.21.15.6:0/2577420342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563f3dc6bb90 0x563f3dc720e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:26.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.004+0000 7f850862f740 1 --2- 172.21.15.6:0/2577420342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f3dc609b0 0x563f3dc71660 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:26.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.004+0000 7f84ecc00640 0 client.27091 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:26.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.004+0000 7f850862f740 1 -- 172.21.15.6:0/2577420342 >> 172.21.15.6:0/2577420342 conn(0x563f3dc5cec0 msgr2=0x563f3dc5b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:26.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.005+0000 7f850862f740 1 -- 172.21.15.6:0/2577420342 shutdown_connections 2024-04-26T02:01:26.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.005+0000 7f850862f740 1 -- 172.21.15.6:0/2577420342 wait complete. 2024-04-26T02:01:26.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.005+0000 7f850862f740 20 client.27091 trim_cache size 0 max 16384 2024-04-26T02:01:26.005 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:01:26.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:01:26.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:01:26.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:01:26.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.006+0000 7f84ffe00640 1 -- 172.21.15.6:0/1324851188 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f84e40a3d00 msgr2=0x7f84e40a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:26.006 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.006+0000 7f84ffe00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f84e40a3d00 0x7f84e40a61c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x563f3dc61f50 tx=0x7f84f00d46d0 comp rx=0 tx=0).stop 2024-04-26T02:01:26.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.007+0000 7f84ffe00640 1 -- 172.21.15.6:0/1324851188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85001083b0 msgr2=0x7f85001a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:26.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.007+0000 7f84ffe00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85001083b0 0x7f85001a4720 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f84f000e9f0 tx=0x7f84f000eeb0 comp rx=0 tx=0).stop 2024-04-26T02:01:26.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.007+0000 7f84ffe00640 1 -- 172.21.15.6:0/1324851188 shutdown_connections 2024-04-26T02:01:26.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.007+0000 7f84ffe00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f84e40a3d00 0x7f84e40a61c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:26.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.007+0000 7f84ffe00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f850010e470 0x7f850019e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:26.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.007+0000 7f84ffe00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500108d50 0x7f85001a4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:26.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.007+0000 7f84ffe00640 1 --2- 172.21.15.6:0/1324851188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f85001083b0 0x7f85001a4720 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:26.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.007+0000 7f84ffe00640 1 -- 172.21.15.6:0/1324851188 >> 172.21.15.6:0/1324851188 conn(0x7f8500077b00 msgr2=0x7f8500079190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:26.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.007+0000 7f84ffe00640 1 -- 172.21.15.6:0/1324851188 shutdown_connections 2024-04-26T02:01:26.007 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:26.007+0000 7f84ffe00640 1 -- 172.21.15.6:0/1324851188 wait complete. 2024-04-26T02:01:26.100 INFO:tasks.workunit.client.0.smithi006.stdout: 10 25004 13.94 MB/sec warmup 109 sec latency 90.817 ms 2024-04-26T02:01:27.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:26 smithi006 ceph-mon[28419]: pgmap v493: 97 pgs: 97 active+clean; 454 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3.4 MiB/s wr, 296 op/s 2024-04-26T02:01:27.100 INFO:tasks.workunit.client.0.smithi006.stdout: 10 25762 14.24 MB/sec warmup 110 sec latency 51.911 ms 2024-04-26T02:01:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:26 smithi188 ceph-mon[32751]: pgmap v493: 97 pgs: 97 active+clean; 454 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3.4 MiB/s wr, 296 op/s 2024-04-26T02:01:27.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:26 smithi029 ceph-mon[32775]: pgmap v493: 97 pgs: 97 active+clean; 454 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3.4 MiB/s wr, 296 op/s 2024-04-26T02:01:28.101 INFO:tasks.workunit.client.0.smithi006.stdout: 10 26470 14.34 MB/sec warmup 111 sec latency 95.263 ms 2024-04-26T02:01:29.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:28 smithi006 ceph-mon[28419]: pgmap v494: 97 pgs: 97 active+clean; 469 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 6.0 MiB/s wr, 575 op/s 2024-04-26T02:01:29.100 INFO:tasks.workunit.client.0.smithi006.stdout: 10 27122 14.65 MB/sec warmup 112 sec latency 123.103 ms 2024-04-26T02:01:29.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:28 smithi188 ceph-mon[32751]: pgmap v494: 97 pgs: 97 active+clean; 469 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 6.0 MiB/s wr, 575 op/s 2024-04-26T02:01:29.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:28 smithi029 ceph-mon[32775]: pgmap v494: 97 pgs: 97 active+clean; 469 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 6.0 MiB/s wr, 575 op/s 2024-04-26T02:01:30.101 INFO:tasks.workunit.client.0.smithi006.stdout: 10 27640 14.71 MB/sec warmup 113 sec latency 175.358 ms 2024-04-26T02:01:31.031 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:30 smithi006 ceph-mon[28419]: pgmap v495: 97 pgs: 97 active+clean; 469 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.2 MiB/s wr, 496 op/s 2024-04-26T02:01:31.101 INFO:tasks.workunit.client.0.smithi006.stdout: 10 28244 14.82 MB/sec warmup 114 sec latency 313.774 ms 2024-04-26T02:01:31.127 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:30 smithi188 ceph-mon[32751]: pgmap v495: 97 pgs: 97 active+clean; 469 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.2 MiB/s wr, 496 op/s 2024-04-26T02:01:31.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:30 smithi029 ceph-mon[32775]: pgmap v495: 97 pgs: 97 active+clean; 469 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.2 MiB/s wr, 496 op/s 2024-04-26T02:01:31.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:31 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:01:32.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:31 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:01:32.101 INFO:tasks.workunit.client.0.smithi006.stdout: 10 29027 15.05 MB/sec warmup 115 sec latency 60.837 ms 2024-04-26T02:01:32.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:31 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:01:33.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:32 smithi006 ceph-mon[28419]: pgmap v496: 97 pgs: 97 active+clean; 503 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 8.4 MiB/s wr, 706 op/s 2024-04-26T02:01:33.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:32 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:01:33.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:32 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:01:33.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:32 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:01:33.104 INFO:tasks.workunit.client.0.smithi006.stdout: 10 29771 15.34 MB/sec warmup 116 sec latency 42.677 ms 2024-04-26T02:01:33.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:32 smithi188 ceph-mon[32751]: pgmap v496: 97 pgs: 97 active+clean; 503 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 8.4 MiB/s wr, 706 op/s 2024-04-26T02:01:33.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:32 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:01:33.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:32 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:01:33.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:32 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:01:33.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:32 smithi029 ceph-mon[32775]: pgmap v496: 97 pgs: 97 active+clean; 503 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 8.4 MiB/s wr, 706 op/s 2024-04-26T02:01:33.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:32 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:01:33.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:32 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:01:33.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:32 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:01:34.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:33 smithi006 ceph-mon[28419]: pgmap v497: 97 pgs: 97 active+clean; 503 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 5.8 MiB/s wr, 488 op/s 2024-04-26T02:01:34.104 INFO:tasks.workunit.client.0.smithi006.stdout: 10 30343 15.53 MB/sec warmup 117 sec latency 102.890 ms 2024-04-26T02:01:34.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:33 smithi188 ceph-mon[32751]: pgmap v497: 97 pgs: 97 active+clean; 503 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 5.8 MiB/s wr, 488 op/s 2024-04-26T02:01:34.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:33 smithi029 ceph-mon[32775]: pgmap v497: 97 pgs: 97 active+clean; 503 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 5.8 MiB/s wr, 488 op/s 2024-04-26T02:01:35.104 INFO:tasks.workunit.client.0.smithi006.stdout: 10 30797 15.55 MB/sec warmup 118 sec latency 322.833 ms 2024-04-26T02:01:36.104 INFO:tasks.workunit.client.0.smithi006.stdout: 10 31463 15.66 MB/sec warmup 119 sec latency 144.115 ms 2024-04-26T02:01:36.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:36 smithi029 ceph-mon[32775]: pgmap v498: 97 pgs: 97 active+clean; 503 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 5.8 MiB/s wr, 488 op/s 2024-04-26T02:01:36.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:36 smithi006 ceph-mon[28419]: pgmap v498: 97 pgs: 97 active+clean; 503 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 5.8 MiB/s wr, 488 op/s 2024-04-26T02:01:36.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:36 smithi188 ceph-mon[32751]: pgmap v498: 97 pgs: 97 active+clean; 503 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 5.8 MiB/s wr, 488 op/s 2024-04-26T02:01:38.104 INFO:tasks.workunit.client.0.smithi006.stdout: 10 33148 50.55 MB/sec execute 1 sec latency 69.389 ms 2024-04-26T02:01:38.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:38 smithi029 ceph-mon[32775]: pgmap v499: 97 pgs: 97 active+clean; 527 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.2 MiB/s wr, 674 op/s 2024-04-26T02:01:38.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:38 smithi006 ceph-mon[28419]: pgmap v499: 97 pgs: 97 active+clean; 527 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.2 MiB/s wr, 674 op/s 2024-04-26T02:01:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:38 smithi188 ceph-mon[32751]: pgmap v499: 97 pgs: 97 active+clean; 527 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.2 MiB/s wr, 674 op/s 2024-04-26T02:01:39.104 INFO:tasks.workunit.client.0.smithi006.stdout: 10 33737 43.46 MB/sec execute 2 sec latency 42.854 ms 2024-04-26T02:01:39.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.183+0000 7fd03f000640 1 -- 172.21.15.6:0/3412143257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd038104d70 msgr2=0x7fd038108db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:39.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.183+0000 7fd03f000640 1 --2- 172.21.15.6:0/3412143257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd038104d70 0x7fd038108db0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd028009900 tx=0x7fd02802a410 comp rx=0 tx=0).stop 2024-04-26T02:01:39.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.184+0000 7fd03f000640 1 -- 172.21.15.6:0/3412143257 shutdown_connections 2024-04-26T02:01:39.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.184+0000 7fd03f000640 1 --2- 172.21.15.6:0/3412143257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd038109490 0x7fd038111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.184+0000 7fd03f000640 1 --2- 172.21.15.6:0/3412143257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd038104d70 0x7fd038108db0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.184+0000 7fd03f000640 1 --2- 172.21.15.6:0/3412143257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0381043c0 0x7fd0381047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.184+0000 7fd03f000640 1 -- 172.21.15.6:0/3412143257 >> 172.21.15.6:0/3412143257 conn(0x7fd0380ffc20 msgr2=0x7fd038102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.184+0000 7fd03f000640 1 -- 172.21.15.6:0/3412143257 shutdown_connections 2024-04-26T02:01:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.184+0000 7fd03f000640 1 -- 172.21.15.6:0/3412143257 wait complete. 2024-04-26T02:01:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.185+0000 7fd03f000640 1 Processor -- start 2024-04-26T02:01:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.185+0000 7fd03f000640 1 -- start start 2024-04-26T02:01:39.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd03f000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0381043c0 0x7fd03819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd03f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd038109490 0x7fd03819aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd03f000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd03819f3a0 0x7fd03819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd03f000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd038114420 con 0x7fd038109490 2024-04-26T02:01:39.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd03f000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0381142a0 con 0x7fd0381043c0 2024-04-26T02:01:39.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd03f000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd0381145a0 con 0x7fd03819f3a0 2024-04-26T02:01:39.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd037e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0381043c0 0x7fd03819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd037e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0381043c0 0x7fd03819aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56510/0 (socket says 172.21.15.6:56510) 2024-04-26T02:01:39.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd037e00640 1 -- 172.21.15.6:0/1079369427 learned_addr learned my addr 172.21.15.6:0/1079369427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:39.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd03d000640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd03819f3a0 0x7fd03819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd037e00640 1 -- 172.21.15.6:0/1079369427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd03819f3a0 msgr2=0x7fd03819f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd037e00640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd03819f3a0 0x7fd03819f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.186+0000 7fd037e00640 1 -- 172.21.15.6:0/1079369427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd038109490 msgr2=0x7fd03819aff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T02:01:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.187+0000 7fd037e00640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd038109490 0x7fd03819aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.187+0000 7fd037e00640 1 -- 172.21.15.6:0/1079369427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03819fe90 con 0x7fd0381043c0 2024-04-26T02:01:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.187+0000 7fd037e00640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0381043c0 0x7fd03819aab0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd02000e9f0 tx=0x7fd02000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.188+0000 7fd034c00640 1 -- 172.21.15.6:0/1079369427 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02000cd80 con 0x7fd0381043c0 2024-04-26T02:01:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.188+0000 7fd034c00640 1 -- 172.21.15.6:0/1079369427 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0200054d0 con 0x7fd0381043c0 2024-04-26T02:01:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.188+0000 7fd034c00640 1 -- 172.21.15.6:0/1079369427 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd020010430 con 0x7fd0381043c0 2024-04-26T02:01:39.188 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.188+0000 7fd03f000640 1 -- 172.21.15.6:0/1079369427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0381a0180 con 0x7fd0381043c0 2024-04-26T02:01:39.188 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.188+0000 7fd03f000640 1 -- 172.21.15.6:0/1079369427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0381a7940 con 0x7fd0381043c0 2024-04-26T02:01:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.193+0000 7fd034c00640 1 -- 172.21.15.6:0/1079369427 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fd020005000 con 0x7fd0381043c0 2024-04-26T02:01:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.193+0000 7fd041fdc740 1 Processor -- start 2024-04-26T02:01:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.194+0000 7fd041fdc740 1 -- start start 2024-04-26T02:01:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.195+0000 7fd041fdc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684880760 0x561684880b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.195+0000 7fd034c00640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd01c0a3d90 0x7fd01c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.195+0000 7fd041fdc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684881080 0x561684893cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.195+0000 7fd041fdc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684894210 0x561684896600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.195+0000 7fd041fdc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5616847e6900 con 0x561684894210 2024-04-26T02:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.195+0000 7fd041fdc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56168478cef0 con 0x561684881080 2024-04-26T02:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.195+0000 7fd041fdc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5616847e28c0 con 0x561684880760 2024-04-26T02:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.195+0000 7fd034c00640 1 -- 172.21.15.6:0/1079369427 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7fd0200d0d00 con 0x7fd0381043c0 2024-04-26T02:01:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.196+0000 7fd03d000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684880760 0x561684880b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.196+0000 7fd037400640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd01c0a3d90 0x7fd01c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.196+0000 7fd03d000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684880760 0x561684880b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:33406/0 (socket says 172.21.15.6:33406) 2024-04-26T02:01:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.196+0000 7fd03d000640 1 -- 172.21.15.6:0/3595530272 learned_addr learned my addr 172.21.15.6:0/3595530272 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.197+0000 7fd037e00640 1 --2- 172.21.15.6:0/3595530272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684881080 0x561684893cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.197+0000 7fd037400640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd01c0a3d90 0x7fd01c0a6250 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd028009230 tx=0x7fd028031400 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.197+0000 7fd03d000640 1 -- 172.21.15.6:0/3595530272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684881080 msgr2=0x561684893cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.197+0000 7fd03d000640 1 --2- 172.21.15.6:0/3595530272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684881080 0x561684893cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.197+0000 7fd03d000640 1 -- 172.21.15.6:0/3595530272 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684894210 msgr2=0x561684896600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.197+0000 7fd03d000640 1 --2- 172.21.15.6:0/3595530272 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684894210 0x561684896600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.197+0000 7fd03d000640 1 -- 172.21.15.6:0/3595530272 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616844f9210 con 0x561684880760 2024-04-26T02:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.197+0000 7fd03d000640 1 --2- 172.21.15.6:0/3595530272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684880760 0x561684880b40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd030005c40 tx=0x7fd0300139f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d841f244420c839e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.198+0000 7fd02d600640 1 -- 172.21.15.6:0/3595530272 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03001c470 con 0x561684880760 2024-04-26T02:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.198+0000 7fd02d600640 1 -- 172.21.15.6:0/3595530272 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd030017070 con 0x561684880760 2024-04-26T02:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.198+0000 7fd02d600640 1 -- 172.21.15.6:0/3595530272 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03000fac0 con 0x561684880760 2024-04-26T02:01:39.199 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.201+0000 7fd041fdc740 1 -- 172.21.15.6:0/3595530272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684880760 msgr2=0x561684880b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:39.199 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.201+0000 7fd041fdc740 1 --2- 172.21.15.6:0/3595530272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684880760 0x561684880b40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd030005c40 tx=0x7fd0300139f0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.202+0000 7fd041fdc740 1 -- 172.21.15.6:0/3595530272 shutdown_connections 2024-04-26T02:01:39.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.202+0000 7fd041fdc740 1 --2- 172.21.15.6:0/3595530272 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684894210 0x561684896600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.202+0000 7fd041fdc740 1 --2- 172.21.15.6:0/3595530272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684881080 0x561684893cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.202+0000 7fd041fdc740 1 --2- 172.21.15.6:0/3595530272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684880760 0x561684880b40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.202+0000 7fd041fdc740 1 -- 172.21.15.6:0/3595530272 >> 172.21.15.6:0/3595530272 conn(0x56168487cd30 msgr2=0x56168487d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:39.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.202+0000 7fd041fdc740 1 -- 172.21.15.6:0/3595530272 shutdown_connections 2024-04-26T02:01:39.200 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.202+0000 7fd041fdc740 1 -- 172.21.15.6:0/3595530272 wait complete. 2024-04-26T02:01:39.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.202+0000 7fd041fdc740 1 -- start start 2024-04-26T02:01:39.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.203+0000 7fd041fdc740 1 Processor -- start 2024-04-26T02:01:39.201 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.204+0000 7fd041fdc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684880760 0x561684891230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.206+0000 7fd041fdc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684881080 0x561684891770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.206+0000 7fd041fdc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684894210 0x561684891cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.206+0000 7fd041fdc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x561684896de0 con 0x561684880760 2024-04-26T02:01:39.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.206+0000 7fd037e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684880760 0x561684891230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.206+0000 7fd037400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684881080 0x561684891770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.204 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.206+0000 7fd037400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684881080 0x561684891770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56536/0 (socket says 172.21.15.6:56536) 2024-04-26T02:01:39.205 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.206+0000 7fd037400640 1 -- 172.21.15.6:0/1270693737 learned_addr learned my addr 172.21.15.6:0/1270693737 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:39.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.208+0000 7fd03d000640 1 --2- 172.21.15.6:0/1270693737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684894210 0x561684891cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.206+0000 7fd041fdc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5616847a0310 con 0x561684881080 2024-04-26T02:01:39.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.208+0000 7fd041fdc740 1 -- 172.21.15.6:0/1270693737 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561684896f60 con 0x561684894210 2024-04-26T02:01:39.206 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.208+0000 7fd041fdc740 10 client.0 _ll_register_callbacks cb 0x56168476ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:39.207 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.209+0000 7fd03d000640 1 -- 172.21.15.6:0/1270693737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684881080 msgr2=0x561684891770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:39.207 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.209+0000 7fd03d000640 1 --2- 172.21.15.6:0/1270693737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684881080 0x561684891770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.207 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.209+0000 7fd03d000640 1 -- 172.21.15.6:0/1270693737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684880760 msgr2=0x561684891230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:39.207 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.209+0000 7fd03d000640 1 --2- 172.21.15.6:0/1270693737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684880760 0x561684891230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.207 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.209+0000 7fd03d000640 1 -- 172.21.15.6:0/1270693737 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616845c5620 con 0x561684894210 2024-04-26T02:01:39.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.210+0000 7fd03d000640 1 --2- 172.21.15.6:0/1270693737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684894210 0x561684891cb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd030006010 tx=0x7fd030009dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42eae223899ed400 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:39.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.210+0000 7fd026000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:01:39.208 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.210+0000 7fd026000640 1 -- 172.21.15.6:0/1270693737 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd030021070 con 0x561684894210 2024-04-26T02:01:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.210+0000 7fd041fdc740 1 -- 172.21.15.6:0/1270693737 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd030017070 con 0x561684894210 2024-04-26T02:01:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.210+0000 7fd026000640 1 -- 172.21.15.6:0/1270693737 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7fd03001c680 con 0x561684894210 2024-04-26T02:01:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.210+0000 7fd026000640 1 -- 172.21.15.6:0/1270693737 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fd03000fac0 con 0x561684894210 2024-04-26T02:01:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.210+0000 7fd041fdc740 10 client.27103 fetch_fsmap learned FSMap version 49 2024-04-26T02:01:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.211+0000 7fd041fdc740 1 -- 172.21.15.6:0/1270693737 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5616848a3150 con 0x561684894210 2024-04-26T02:01:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.211+0000 7fd026000640 1 -- 172.21.15.6:0/1270693737 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd030017070 con 0x561684894210 2024-04-26T02:01:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.211+0000 7fd026000640 1 -- 172.21.15.6:0/1270693737 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fd0300267b0 con 0x561684894210 2024-04-26T02:01:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.211+0000 7fd041fdc740 10 client.27103 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:01:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.211+0000 7fd041fdc740 10 client.27103 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:01:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.211+0000 7fd041fdc740 20 client.27103 populate_metadata read hostname 'smithi006' 2024-04-26T02:01:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.211+0000 7fd041fdc740 1 --2- 172.21.15.6:0/1270693737 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5616848a4560 0x5616848a6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.211+0000 7fd041fdc740 4 client.27103 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:01:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.211+0000 7fd041fdc740 1 -- 172.21.15.6:0/1270693737 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616847fa8b0 con 0x5616848a4560 2024-04-26T02:01:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.212+0000 7fd037e00640 1 --2- 172.21.15.6:0/1270693737 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5616848a4560 0x5616848a6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.213+0000 7fd037e00640 1 --2- 172.21.15.6:0/1270693737 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5616848a4560 0x5616848a6950 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b172d1b30ca8714 server_cookie=3569d40c981a6b33 in_seq=0 out_seq=0 2024-04-26T02:01:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.213+0000 7fd026000640 10 client.27103 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:39.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.216+0000 7fd026000640 1 -- 172.21.15.6:0/1270693737 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616847fa8b0 con 0x5616848a4560 2024-04-26T02:01:39.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.216+0000 7fd026000640 10 client.27103 handle_command_reply: tid=0 2024-04-26T02:01:39.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.216+0000 7fd041fdc740 1 client.27103 shutdown 2024-04-26T02:01:39.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.217+0000 7fd041fdc740 1 -- 172.21.15.6:0/1270693737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684894210 msgr2=0x561684891cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:39.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.217+0000 7fd041fdc740 1 --2- 172.21.15.6:0/1270693737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684894210 0x561684891cb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd030006010 tx=0x7fd030009dc0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.217+0000 7fd041fdc740 1 -- 172.21.15.6:0/1270693737 shutdown_connections 2024-04-26T02:01:39.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.217+0000 7fd041fdc740 1 --2- 172.21.15.6:0/1270693737 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5616848a4560 0x5616848a6950 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.217+0000 7fd041fdc740 1 --2- 172.21.15.6:0/1270693737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684894210 0x561684891cb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.217+0000 7fd041fdc740 1 --2- 172.21.15.6:0/1270693737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684881080 0x561684891770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.217+0000 7fd041fdc740 1 --2- 172.21.15.6:0/1270693737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684880760 0x561684891230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.217+0000 7fd041fdc740 1 -- 172.21.15.6:0/1270693737 >> 172.21.15.6:0/1270693737 conn(0x56168487cd30 msgr2=0x561684882ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.217+0000 7fd026000640 0 client.27103 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.218+0000 7fd041fdc740 1 -- 172.21.15.6:0/1270693737 shutdown_connections 2024-04-26T02:01:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.218+0000 7fd041fdc740 1 -- 172.21.15.6:0/1270693737 wait complete. 2024-04-26T02:01:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.218+0000 7fd041fdc740 20 client.27103 trim_cache size 0 max 16384 2024-04-26T02:01:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.244+0000 7fd041fdc740 1 Processor -- start 2024-04-26T02:01:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.244+0000 7fd041fdc740 1 -- start start 2024-04-26T02:01:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.244+0000 7fd041fdc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 0x56168489fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.244+0000 7fd041fdc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684881080 0x56168488c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.244+0000 7fd041fdc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684894210 0x5616848abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.244+0000 7fd041fdc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd0381ae5a0 con 0x561684881080 2024-04-26T02:01:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.244+0000 7fd041fdc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0381abde0 con 0x561684880760 2024-04-26T02:01:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.244+0000 7fd041fdc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd0381a9c50 con 0x561684894210 2024-04-26T02:01:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.244+0000 7fd037e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 0x56168489fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.245+0000 7fd037e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 0x56168489fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56540/0 (socket says 172.21.15.6:56540) 2024-04-26T02:01:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.245+0000 7fd037e00640 1 -- 172.21.15.6:0/50388311 learned_addr learned my addr 172.21.15.6:0/50388311 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.245+0000 7fd03d000640 1 --2- 172.21.15.6:0/50388311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684894210 0x5616848abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.245+0000 7fd037400640 1 --2- 172.21.15.6:0/50388311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684881080 0x56168488c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.245+0000 7fd037e00640 1 -- 172.21.15.6:0/50388311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684894210 msgr2=0x5616848abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.245+0000 7fd037e00640 1 --2- 172.21.15.6:0/50388311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684894210 0x5616848abfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.245+0000 7fd037e00640 1 -- 172.21.15.6:0/50388311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684881080 msgr2=0x56168488c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.245+0000 7fd037e00640 1 --2- 172.21.15.6:0/50388311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684881080 0x56168488c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.245+0000 7fd037e00640 1 -- 172.21.15.6:0/50388311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561684896f40 con 0x561684880760 2024-04-26T02:01:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.245+0000 7fd037e00640 1 --2- 172.21.15.6:0/50388311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 0x56168489fce0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd0200995c0 tx=0x7fd020004a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96d7ba187e98bada server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:39.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.245+0000 7fd02cc00640 1 -- 172.21.15.6:0/50388311 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02009cbf0 con 0x561684880760 2024-04-26T02:01:39.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.246+0000 7fd02cc00640 1 -- 172.21.15.6:0/50388311 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0200a0fa0 con 0x561684880760 2024-04-26T02:01:39.245 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.246+0000 7fd02cc00640 1 -- 172.21.15.6:0/50388311 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd020099260 con 0x561684880760 2024-04-26T02:01:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.247+0000 7fd041fdc740 1 -- 172.21.15.6:0/50388311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 msgr2=0x56168489fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.247+0000 7fd041fdc740 1 --2- 172.21.15.6:0/50388311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 0x56168489fce0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd0200995c0 tx=0x7fd020004a20 comp rx=0 tx=0).stop 2024-04-26T02:01:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.247+0000 7fd041fdc740 1 -- 172.21.15.6:0/50388311 shutdown_connections 2024-04-26T02:01:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.247+0000 7fd041fdc740 1 --2- 172.21.15.6:0/50388311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684881080 0x56168488c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.247+0000 7fd041fdc740 1 --2- 172.21.15.6:0/50388311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684894210 0x5616848abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.247+0000 7fd041fdc740 1 --2- 172.21.15.6:0/50388311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 0x56168489fce0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.247+0000 7fd041fdc740 1 -- 172.21.15.6:0/50388311 >> 172.21.15.6:0/50388311 conn(0x56168487cd30 msgr2=0x56168487be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.247+0000 7fd041fdc740 1 -- 172.21.15.6:0/50388311 shutdown_connections 2024-04-26T02:01:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.247+0000 7fd041fdc740 1 -- 172.21.15.6:0/50388311 wait complete. 2024-04-26T02:01:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.248+0000 7fd041fdc740 1 -- start start 2024-04-26T02:01:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.248+0000 7fd041fdc740 1 Processor -- start 2024-04-26T02:01:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.248+0000 7fd041fdc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 0x56168489c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.248+0000 7fd041fdc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684881080 0x5616848999d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.248+0000 7fd041fdc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684894210 0x561684899f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.249+0000 7fd041fdc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd0381aa2a0 con 0x561684894210 2024-04-26T02:01:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.249+0000 7fd041fdc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0381ab710 con 0x561684880760 2024-04-26T02:01:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.249+0000 7fd041fdc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd0381aa970 con 0x561684881080 2024-04-26T02:01:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.249+0000 7fd041fdc740 10 client.0 _ll_register_callbacks cb 0x7fd030005a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.249+0000 7fd037e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 0x56168489c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.249+0000 7fd037400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684881080 0x5616848999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.249+0000 7fd037400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684881080 0x5616848999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:33438/0 (socket says 172.21.15.6:33438) 2024-04-26T02:01:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.249+0000 7fd037400640 1 -- 172.21.15.6:0/3032172438 learned_addr learned my addr 172.21.15.6:0/3032172438 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.249+0000 7fd037e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 0x56168489c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56556/0 (socket says 172.21.15.6:56556) 2024-04-26T02:01:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.249+0000 7fd03d000640 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684894210 0x561684899f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.250+0000 7fd037e00640 1 -- 172.21.15.6:0/3032172438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684881080 msgr2=0x5616848999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.250+0000 7fd037e00640 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684881080 0x5616848999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.250+0000 7fd037e00640 1 -- 172.21.15.6:0/3032172438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684894210 msgr2=0x561684899f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.250+0000 7fd037e00640 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684894210 0x561684899f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.250+0000 7fd037e00640 1 -- 172.21.15.6:0/3032172438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56168489a740 con 0x561684880760 2024-04-26T02:01:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.251+0000 7fd03d000640 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684894210 0x561684899f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:01:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.252+0000 7fd037e00640 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 0x56168489c5f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd0200995c0 tx=0x7fd0200ce390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dab4827f16f8682f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.252+0000 7fd026000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:01:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.252+0000 7fd026000640 1 -- 172.21.15.6:0/3032172438 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd020051070 con 0x561684880760 2024-04-26T02:01:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.252+0000 7fd026000640 1 -- 172.21.15.6:0/3032172438 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7fd020059df0 con 0x561684880760 2024-04-26T02:01:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.253+0000 7fd041fdc740 1 -- 172.21.15.6:0/3032172438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0200a0fa0 con 0x561684880760 2024-04-26T02:01:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.253+0000 7fd026000640 1 -- 172.21.15.6:0/3032172438 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fd02009cbf0 con 0x561684880760 2024-04-26T02:01:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.253+0000 7fd041fdc740 10 client.27056 fetch_fsmap learned FSMap version 49 2024-04-26T02:01:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.253+0000 7fd041fdc740 1 -- 172.21.15.6:0/3032172438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56168490a8d0 con 0x561684880760 2024-04-26T02:01:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.253+0000 7fd026000640 1 -- 172.21.15.6:0/3032172438 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0200a0fa0 con 0x561684880760 2024-04-26T02:01:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.254+0000 7fd026000640 1 -- 172.21.15.6:0/3032172438 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fd0200ce620 con 0x561684880760 2024-04-26T02:01:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.254+0000 7fd041fdc740 10 client.27056 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:01:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.254+0000 7fd041fdc740 10 client.27056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:01:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.255+0000 7fd041fdc740 20 client.27056 populate_metadata read hostname 'smithi006' 2024-04-26T02:01:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.255+0000 7fd041fdc740 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56168490af00 0x56168490d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.255+0000 7fd041fdc740 4 client.27056 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:01:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.255+0000 7fd041fdc740 1 -- 172.21.15.6:0/3032172438 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5616847f4320 con 0x56168490af00 2024-04-26T02:01:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.255+0000 7fd037400640 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56168490af00 0x56168490d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.256+0000 7fd037400640 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56168490af00 0x56168490d2f0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60ff3e40a6a4abe2 server_cookie=af3055e45fef8271 in_seq=0 out_seq=0 2024-04-26T02:01:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.256+0000 7fd026000640 10 client.27056 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.542+0000 7fd026000640 1 -- 172.21.15.6:0/3032172438 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5616847f4320 con 0x56168490af00 2024-04-26T02:01:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.542+0000 7fd026000640 10 client.27056 handle_command_reply: tid=0 2024-04-26T02:01:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.542+0000 7fd041fdc740 1 client.27056 shutdown 2024-04-26T02:01:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.543+0000 7fd041fdc740 1 -- 172.21.15.6:0/3032172438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 msgr2=0x56168489c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.543+0000 7fd041fdc740 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 0x56168489c5f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd0200995c0 tx=0x7fd0200ce390 comp rx=0 tx=0).stop 2024-04-26T02:01:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.543+0000 7fd041fdc740 1 -- 172.21.15.6:0/3032172438 shutdown_connections 2024-04-26T02:01:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.543+0000 7fd041fdc740 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56168490af00 0x56168490d2f0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.543+0000 7fd041fdc740 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561684894210 0x561684899f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.543+0000 7fd041fdc740 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561684881080 0x5616848999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.543+0000 7fd041fdc740 1 --2- 172.21.15.6:0/3032172438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561684880760 0x56168489c5f0 secure :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd0200995c0 tx=0x7fd0200ce390 comp rx=0 tx=0).stop 2024-04-26T02:01:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.543+0000 7fd041fdc740 1 -- 172.21.15.6:0/3032172438 >> 172.21.15.6:0/3032172438 conn(0x56168487cd30 msgr2=0x5616848911f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.543+0000 7fd026000640 0 client.27056 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.544+0000 7fd041fdc740 1 -- 172.21.15.6:0/3032172438 shutdown_connections 2024-04-26T02:01:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.544+0000 7fd041fdc740 1 -- 172.21.15.6:0/3032172438 wait complete. 2024-04-26T02:01:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.544+0000 7fd041fdc740 20 client.27056 trim_cache size 0 max 16384 2024-04-26T02:01:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:01:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:01:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:01:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:01:39.543 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.545+0000 7fd03f000640 1 -- 172.21.15.6:0/1079369427 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd01c0a3d90 msgr2=0x7fd01c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.545+0000 7fd03f000640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd01c0a3d90 0x7fd01c0a6250 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd028009230 tx=0x7fd028031400 comp rx=0 tx=0).stop 2024-04-26T02:01:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.545+0000 7fd03f000640 1 -- 172.21.15.6:0/1079369427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0381043c0 msgr2=0x7fd03819aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.545+0000 7fd03f000640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0381043c0 0x7fd03819aab0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd02000e9f0 tx=0x7fd02000edf0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.546+0000 7fd03f000640 1 -- 172.21.15.6:0/1079369427 shutdown_connections 2024-04-26T02:01:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.546+0000 7fd03f000640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd01c0a3d90 0x7fd01c0a6250 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.546+0000 7fd03f000640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd038109490 0x7fd03819aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.546+0000 7fd03f000640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd03819f3a0 0x7fd03819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.546+0000 7fd03f000640 1 --2- 172.21.15.6:0/1079369427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0381043c0 0x7fd03819aab0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.546+0000 7fd03f000640 1 -- 172.21.15.6:0/1079369427 >> 172.21.15.6:0/1079369427 conn(0x7fd0380ffc20 msgr2=0x7fd038100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.546+0000 7fd03f000640 1 -- 172.21.15.6:0/1079369427 shutdown_connections 2024-04-26T02:01:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:39.546+0000 7fd03f000640 1 -- 172.21.15.6:0/1079369427 wait complete. 2024-04-26T02:01:40.104 INFO:tasks.workunit.client.0.smithi006.stdout: 10 34232 34.88 MB/sec execute 3 sec latency 117.473 ms 2024-04-26T02:01:40.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:40 smithi029 ceph-mon[32775]: pgmap v500: 97 pgs: 97 active+clean; 527 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 5.6 MiB/s wr, 395 op/s 2024-04-26T02:01:40.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:40 smithi006 ceph-mon[28419]: pgmap v500: 97 pgs: 97 active+clean; 527 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 5.6 MiB/s wr, 395 op/s 2024-04-26T02:01:40.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:40 smithi188 ceph-mon[32751]: pgmap v500: 97 pgs: 97 active+clean; 527 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 5.6 MiB/s wr, 395 op/s 2024-04-26T02:01:41.105 INFO:tasks.workunit.client.0.smithi006.stdout: 10 34953 35.83 MB/sec execute 4 sec latency 110.840 ms 2024-04-26T02:01:42.105 INFO:tasks.workunit.client.0.smithi006.stdout: 10 35805 37.63 MB/sec execute 5 sec latency 39.817 ms 2024-04-26T02:01:42.691 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:42 smithi029 ceph-mon[32775]: pgmap v501: 97 pgs: 97 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.1 MiB/s wr, 695 op/s 2024-04-26T02:01:42.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:42 smithi006 ceph-mon[28419]: pgmap v501: 97 pgs: 97 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.1 MiB/s wr, 695 op/s 2024-04-26T02:01:42.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:42 smithi188 ceph-mon[32751]: pgmap v501: 97 pgs: 97 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.1 MiB/s wr, 695 op/s 2024-04-26T02:01:43.105 INFO:tasks.workunit.client.0.smithi006.stdout: 10 36572 39.09 MB/sec execute 6 sec latency 57.293 ms 2024-04-26T02:01:44.105 INFO:tasks.workunit.client.0.smithi006.stdout: 10 37085 37.68 MB/sec execute 7 sec latency 64.582 ms 2024-04-26T02:01:44.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:44 smithi006 ceph-mon[28419]: pgmap v502: 97 pgs: 97 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.9 MiB/s wr, 486 op/s 2024-04-26T02:01:44.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:44 smithi188 ceph-mon[32751]: pgmap v502: 97 pgs: 97 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.9 MiB/s wr, 486 op/s 2024-04-26T02:01:44.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:44 smithi029 ceph-mon[32775]: pgmap v502: 97 pgs: 97 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.9 MiB/s wr, 486 op/s 2024-04-26T02:01:45.105 INFO:tasks.workunit.client.0.smithi006.stdout: 10 37741 37.30 MB/sec execute 8 sec latency 77.871 ms 2024-04-26T02:01:46.105 INFO:tasks.workunit.client.0.smithi006.stdout: 10 38457 36.21 MB/sec execute 9 sec latency 110.364 ms 2024-04-26T02:01:46.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:46 smithi006 ceph-mon[28419]: pgmap v503: 97 pgs: 97 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.9 MiB/s wr, 486 op/s 2024-04-26T02:01:46.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:46 smithi188 ceph-mon[32751]: pgmap v503: 97 pgs: 97 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.9 MiB/s wr, 486 op/s 2024-04-26T02:01:46.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:46 smithi029 ceph-mon[32775]: pgmap v503: 97 pgs: 97 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.9 MiB/s wr, 486 op/s 2024-04-26T02:01:47.106 INFO:tasks.workunit.client.0.smithi006.stdout: 10 39192 37.14 MB/sec execute 10 sec latency 72.318 ms 2024-04-26T02:01:48.106 INFO:tasks.workunit.client.0.smithi006.stdout: 10 39899 37.97 MB/sec execute 11 sec latency 43.149 ms 2024-04-26T02:01:48.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:48 smithi006 ceph-mon[28419]: pgmap v504: 97 pgs: 97 active+clean; 577 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.7 MiB/s wr, 696 op/s 2024-04-26T02:01:48.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:48 smithi188 ceph-mon[32751]: pgmap v504: 97 pgs: 97 active+clean; 577 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.7 MiB/s wr, 696 op/s 2024-04-26T02:01:48.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:48 smithi029 ceph-mon[32775]: pgmap v504: 97 pgs: 97 active+clean; 577 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.7 MiB/s wr, 696 op/s 2024-04-26T02:01:49.106 INFO:tasks.workunit.client.0.smithi006.stdout: 10 40506 37.02 MB/sec execute 12 sec latency 97.276 ms 2024-04-26T02:01:50.106 INFO:tasks.workunit.client.0.smithi006.stdout: 10 41316 37.10 MB/sec execute 13 sec latency 43.961 ms 2024-04-26T02:01:50.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:50 smithi006 ceph-mon[28419]: pgmap v505: 97 pgs: 97 active+clean; 577 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 510 op/s 2024-04-26T02:01:50.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:50 smithi188 ceph-mon[32751]: pgmap v505: 97 pgs: 97 active+clean; 577 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 510 op/s 2024-04-26T02:01:50.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:50 smithi029 ceph-mon[32775]: pgmap v505: 97 pgs: 97 active+clean; 577 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 510 op/s 2024-04-26T02:01:51.106 INFO:tasks.workunit.client.0.smithi006.stdout: 10 41772 36.48 MB/sec execute 14 sec latency 287.157 ms 2024-04-26T02:01:52.106 INFO:tasks.workunit.client.0.smithi006.stdout: 10 42434 36.96 MB/sec execute 15 sec latency 89.420 ms 2024-04-26T02:01:52.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.724+0000 7fd19e600640 1 -- 172.21.15.6:0/3681072849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd198112df0 msgr2=0x7fd1981151e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:52.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.724+0000 7fd19e600640 1 --2- 172.21.15.6:0/3681072849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd198112df0 0x7fd1981151e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd190009cd0 tx=0x7fd19002a8f0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.724+0000 7fd19e600640 1 -- 172.21.15.6:0/3681072849 shutdown_connections 2024-04-26T02:01:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.724+0000 7fd19e600640 1 --2- 172.21.15.6:0/3681072849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd198112df0 0x7fd1981151e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.724+0000 7fd19e600640 1 --2- 172.21.15.6:0/3681072849 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd198076da0 0x7fd1981128b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.724+0000 7fd19e600640 1 --2- 172.21.15.6:0/3681072849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd198076480 0x7fd198076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.724+0000 7fd19e600640 1 -- 172.21.15.6:0/3681072849 >> 172.21.15.6:0/3681072849 conn(0x7fd1980ffc20 msgr2=0x7fd198102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.724+0000 7fd19e600640 1 -- 172.21.15.6:0/3681072849 shutdown_connections 2024-04-26T02:01:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.724+0000 7fd19e600640 1 -- 172.21.15.6:0/3681072849 wait complete. 2024-04-26T02:01:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.725+0000 7fd19e600640 1 Processor -- start 2024-04-26T02:01:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.725+0000 7fd19e600640 1 -- start start 2024-04-26T02:01:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.726+0000 7fd19e600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd198076480 0x7fd19819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.726+0000 7fd19e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd198076da0 0x7fd19819f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.726+0000 7fd19e600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd198112df0 0x7fd1981a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.726+0000 7fd19e600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd198118890 con 0x7fd198076da0 2024-04-26T02:01:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.726+0000 7fd19e600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd198118710 con 0x7fd198076480 2024-04-26T02:01:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.726+0000 7fd19e600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd198118a10 con 0x7fd198112df0 2024-04-26T02:01:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.726+0000 7fd197400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd198076480 0x7fd19819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.726+0000 7fd197400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd198076480 0x7fd19819edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56924/0 (socket says 172.21.15.6:56924) 2024-04-26T02:01:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.726+0000 7fd197400640 1 -- 172.21.15.6:0/101342585 learned_addr learned my addr 172.21.15.6:0/101342585 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.726+0000 7fd197e00640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd198112df0 0x7fd1981a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.726+0000 7fd196a00640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd198076da0 0x7fd19819f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.727+0000 7fd197400640 1 -- 172.21.15.6:0/101342585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd198112df0 msgr2=0x7fd1981a3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.727+0000 7fd197400640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd198112df0 0x7fd1981a3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.727+0000 7fd197400640 1 -- 172.21.15.6:0/101342585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd198076da0 msgr2=0x7fd19819f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.727+0000 7fd197400640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd198076da0 0x7fd19819f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.727+0000 7fd197400640 1 -- 172.21.15.6:0/101342585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1981a3da0 con 0x7fd198076480 2024-04-26T02:01:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.727+0000 7fd196a00640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd198076da0 0x7fd19819f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:01:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.727+0000 7fd197400640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd198076480 0x7fd19819edb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c920 tx=0x7fd18800cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.727+0000 7fd18fe00640 1 -- 172.21.15.6:0/101342585 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd188007c70 con 0x7fd198076480 2024-04-26T02:01:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.727+0000 7fd18fe00640 1 -- 172.21.15.6:0/101342585 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd188007e10 con 0x7fd198076480 2024-04-26T02:01:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.727+0000 7fd18fe00640 1 -- 172.21.15.6:0/101342585 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd188018820 con 0x7fd198076480 2024-04-26T02:01:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.727+0000 7fd19e600640 1 -- 172.21.15.6:0/101342585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1981a4090 con 0x7fd198076480 2024-04-26T02:01:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.728+0000 7fd19e600640 1 -- 172.21.15.6:0/101342585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1981ab920 con 0x7fd198076480 2024-04-26T02:01:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.731+0000 7fd18fe00640 1 -- 172.21.15.6:0/101342585 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fd18800f450 con 0x7fd198076480 2024-04-26T02:01:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.732+0000 7fd1a13ef740 1 Processor -- start 2024-04-26T02:01:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.732+0000 7fd1a13ef740 1 -- start start 2024-04-26T02:01:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.732+0000 7fd1a13ef740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 0x55e1729f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.732+0000 7fd1a13ef740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 0x55e1729ff4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.732+0000 7fd1a13ef740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e172a01df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.732+0000 7fd1a13ef740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55e17295a900 con 0x55e1729f5140 2024-04-26T02:01:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.732+0000 7fd1a13ef740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e172900ef0 con 0x55e1729f4820 2024-04-26T02:01:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.732+0000 7fd1a13ef740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e1729568c0 con 0x55e1729ffa00 2024-04-26T02:01:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.733+0000 7fd197e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 0x55e1729f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.733+0000 7fd197e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 0x55e1729f4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56936/0 (socket says 172.21.15.6:56936) 2024-04-26T02:01:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.733+0000 7fd196a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e172a01df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.733+0000 7fd197e00640 1 -- 172.21.15.6:0/1937675128 learned_addr learned my addr 172.21.15.6:0/1937675128 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.733+0000 7fd197400640 1 --2- 172.21.15.6:0/1937675128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 0x55e1729ff4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.733+0000 7fd196a00640 1 -- 172.21.15.6:0/1937675128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 msgr2=0x55e1729f4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.733+0000 7fd196a00640 1 --2- 172.21.15.6:0/1937675128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 0x55e1729f4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.733+0000 7fd196a00640 1 -- 172.21.15.6:0/1937675128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 msgr2=0x55e1729ff4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.733+0000 7fd196a00640 1 --2- 172.21.15.6:0/1937675128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 0x55e1729ff4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.733+0000 7fd196a00640 1 -- 172.21.15.6:0/1937675128 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e17266d210 con 0x55e1729ffa00 2024-04-26T02:01:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.734+0000 7fd197e00640 1 --2- 172.21.15.6:0/1937675128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 0x55e1729f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:01:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.734+0000 7fd18fe00640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd17c0a3d90 0x7fd17c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.734+0000 7fd196a00640 1 --2- 172.21.15.6:0/1937675128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e172a01df0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd184007730 tx=0x7fd184007bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d468520f48291159 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.734+0000 7fd18fe00640 1 -- 172.21.15.6:0/101342585 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7fd1880d0970 con 0x7fd198076480 2024-04-26T02:01:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.734+0000 7fd197e00640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd17c0a3d90 0x7fd17c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.734+0000 7fd18cc00640 1 -- 172.21.15.6:0/1937675128 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd184007e40 con 0x55e1729ffa00 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.734+0000 7fd18cc00640 1 -- 172.21.15.6:0/1937675128 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd184018070 con 0x55e1729ffa00 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.734+0000 7fd18cc00640 1 -- 172.21.15.6:0/1937675128 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd184013cf0 con 0x55e1729ffa00 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.735+0000 7fd1a13ef740 1 -- 172.21.15.6:0/1937675128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 msgr2=0x55e172a01df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.735+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/1937675128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e172a01df0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd184007730 tx=0x7fd184007bf0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.735+0000 7fd197e00640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd17c0a3d90 0x7fd17c0a6250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd190009430 tx=0x7fd190009800 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.735+0000 7fd1a13ef740 1 -- 172.21.15.6:0/1937675128 shutdown_connections 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.735+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/1937675128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 0x55e1729ff4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.735+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/1937675128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e172a01df0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.735+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/1937675128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 0x55e1729f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.735+0000 7fd1a13ef740 1 -- 172.21.15.6:0/1937675128 >> 172.21.15.6:0/1937675128 conn(0x55e1729f0d30 msgr2=0x55e1729f1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.735+0000 7fd1a13ef740 1 -- 172.21.15.6:0/1937675128 shutdown_connections 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.735+0000 7fd1a13ef740 1 -- 172.21.15.6:0/1937675128 wait complete. 2024-04-26T02:01:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.736+0000 7fd1a13ef740 1 -- start start 2024-04-26T02:01:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.736+0000 7fd1a13ef740 1 Processor -- start 2024-04-26T02:01:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.737+0000 7fd1a13ef740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f4820 0x55e1729fc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.737+0000 7fd1a13ef740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f5140 0x55e1729fcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.737+0000 7fd1a13ef740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e1729fd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.737+0000 7fd1a13ef740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55e172a02630 con 0x55e1729f4820 2024-04-26T02:01:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.737+0000 7fd1a13ef740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e172914310 con 0x55e1729f5140 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.737+0000 7fd1a13ef740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e172a027b0 con 0x55e1729ffa00 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.737+0000 7fd1a13ef740 10 client.0 _ll_register_callbacks cb 0x55e1728e2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.737+0000 7fd196a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f5140 0x55e1729fcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.737+0000 7fd196a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f5140 0x55e1729fcda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56948/0 (socket says 172.21.15.6:56948) 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.737+0000 7fd196a00640 1 -- 172.21.15.6:0/2051540153 learned_addr learned my addr 172.21.15.6:0/2051540153 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.737+0000 7fd197400640 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f4820 0x55e1729fc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.737+0000 7fd197e00640 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e1729fd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.738+0000 7fd196a00640 1 -- 172.21.15.6:0/2051540153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 msgr2=0x55e1729fd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.738+0000 7fd196a00640 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e1729fd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.738+0000 7fd196a00640 1 -- 172.21.15.6:0/2051540153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f4820 msgr2=0x55e1729fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.738+0000 7fd196a00640 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f4820 0x55e1729fc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.738+0000 7fd196a00640 1 -- 172.21.15.6:0/2051540153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e172739620 con 0x55e1729f5140 2024-04-26T02:01:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.738+0000 7fd197400640 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f4820 0x55e1729fc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.738+0000 7fd196a00640 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f5140 0x55e1729fcda0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd184013730 tx=0x7fd184013760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2062c6e968575ede server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.738+0000 7fd181600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.738+0000 7fd181600640 1 -- 172.21.15.6:0/2051540153 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18400d950 con 0x55e1729f5140 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.738+0000 7fd181600640 1 -- 172.21.15.6:0/2051540153 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7fd18400daf0 con 0x55e1729f5140 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.738+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2051540153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd184018070 con 0x55e1729f5140 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.739+0000 7fd181600640 1 -- 172.21.15.6:0/2051540153 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fd18400dcf0 con 0x55e1729f5140 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.739+0000 7fd1a13ef740 10 client.27068 fetch_fsmap learned FSMap version 49 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.739+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2051540153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e172a0e7a0 con 0x55e1729f5140 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.739+0000 7fd181600640 1 -- 172.21.15.6:0/2051540153 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd184018070 con 0x55e1729f5140 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.739+0000 7fd181600640 1 -- 172.21.15.6:0/2051540153 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fd184024420 con 0x55e1729f5140 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.739+0000 7fd1a13ef740 10 client.27068 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.739+0000 7fd1a13ef740 10 client.27068 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.739+0000 7fd1a13ef740 20 client.27068 populate_metadata read hostname 'smithi006' 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.740+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e172a0fbb0 0x55e172a11fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.740+0000 7fd1a13ef740 4 client.27068 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:01:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.740+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2051540153 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e17296e8b0 con 0x55e172a0fbb0 2024-04-26T02:01:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.740+0000 7fd197400640 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e172a0fbb0 0x55e172a11fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.740+0000 7fd197400640 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e172a0fbb0 0x55e172a11fa0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c4452bfe218ec9e server_cookie=57442c2e1c51af68 in_seq=0 out_seq=0 2024-04-26T02:01:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.740+0000 7fd181600640 10 client.27068 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd181600640 1 -- 172.21.15.6:0/2051540153 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e17296e8b0 con 0x55e172a0fbb0 2024-04-26T02:01:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd181600640 10 client.27068 handle_command_reply: tid=0 2024-04-26T02:01:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd1a13ef740 1 client.27068 shutdown 2024-04-26T02:01:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2051540153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f5140 msgr2=0x55e1729fcda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f5140 0x55e1729fcda0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd184013730 tx=0x7fd184013760 comp rx=0 tx=0).stop 2024-04-26T02:01:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2051540153 shutdown_connections 2024-04-26T02:01:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e172a0fbb0 0x55e172a11fa0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e1729fd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f5140 0x55e1729fcda0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2051540153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f4820 0x55e1729fc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2051540153 >> 172.21.15.6:0/2051540153 conn(0x55e1729f0d30 msgr2=0x55e1729f6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd181600640 0 client.27068 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2051540153 shutdown_connections 2024-04-26T02:01:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.741+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2051540153 wait complete. 2024-04-26T02:01:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.742+0000 7fd1a13ef740 20 client.27068 trim_cache size 0 max 16384 2024-04-26T02:01:52.760 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.761+0000 7fd1a13ef740 1 Processor -- start 2024-04-26T02:01:52.760 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.761+0000 7fd1a13ef740 1 -- start start 2024-04-26T02:01:52.760 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.761+0000 7fd1a13ef740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 0x55e172a0b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.760 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.761+0000 7fd1a13ef740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 0x55e1729fb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.760 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.761+0000 7fd1a13ef740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e172a17690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.760 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.761+0000 7fd1a13ef740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd1981b6d60 con 0x55e1729f5140 2024-04-26T02:01:52.760 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.761+0000 7fd1a13ef740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd1981b45a0 con 0x55e1729f4820 2024-04-26T02:01:52.760 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.761+0000 7fd1a13ef740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd1981b2410 con 0x55e1729ffa00 2024-04-26T02:01:52.761 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.762+0000 7fd197400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 0x55e172a0b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.761 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.762+0000 7fd197400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 0x55e172a0b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:56964/0 (socket says 172.21.15.6:56964) 2024-04-26T02:01:52.761 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.762+0000 7fd197400640 1 -- 172.21.15.6:0/2082056114 learned_addr learned my addr 172.21.15.6:0/2082056114 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:52.761 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.762+0000 7fd197e00640 1 --2- 172.21.15.6:0/2082056114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e172a17690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.761 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.762+0000 7fd196a00640 1 --2- 172.21.15.6:0/2082056114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 0x55e1729fb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.761 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.762+0000 7fd197400640 1 -- 172.21.15.6:0/2082056114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 msgr2=0x55e172a17690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:52.761 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.762+0000 7fd197400640 1 --2- 172.21.15.6:0/2082056114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e172a17690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.761 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.762+0000 7fd197400640 1 -- 172.21.15.6:0/2082056114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 msgr2=0x55e1729fb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:52.761 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.762+0000 7fd197400640 1 --2- 172.21.15.6:0/2082056114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 0x55e1729fb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.761 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.762+0000 7fd197400640 1 -- 172.21.15.6:0/2082056114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1729f4f70 con 0x55e1729f4820 2024-04-26T02:01:52.762 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.762+0000 7fd197400640 1 --2- 172.21.15.6:0/2082056114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 0x55e172a0b4f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd188099230 tx=0x7fd188087040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b841a302d966720f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:52.762 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.763+0000 7fd183e00640 1 -- 172.21.15.6:0/2082056114 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1880d6020 con 0x55e1729f4820 2024-04-26T02:01:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.763+0000 7fd183e00640 1 -- 172.21.15.6:0/2082056114 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd18809cf60 con 0x55e1729f4820 2024-04-26T02:01:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.763+0000 7fd183e00640 1 -- 172.21.15.6:0/2082056114 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18800fa90 con 0x55e1729f4820 2024-04-26T02:01:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.763+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2082056114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 msgr2=0x55e172a0b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.764+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2082056114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 0x55e172a0b4f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd188099230 tx=0x7fd188087040 comp rx=0 tx=0).stop 2024-04-26T02:01:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.764+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2082056114 shutdown_connections 2024-04-26T02:01:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.764+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2082056114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 0x55e1729fb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.764+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2082056114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729ffa00 0x55e172a17690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.764+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2082056114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729f4820 0x55e172a0b4f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.764+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2082056114 >> 172.21.15.6:0/2082056114 conn(0x55e1729f0d30 msgr2=0x55e1729f1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.764+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2082056114 shutdown_connections 2024-04-26T02:01:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.764+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2082056114 wait complete. 2024-04-26T02:01:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.765+0000 7fd1a13ef740 1 -- start start 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.765+0000 7fd1a13ef740 1 Processor -- start 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.765+0000 7fd1a13ef740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729f4820 0x55e172a04bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.765+0000 7fd1a13ef740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 0x55e172a05100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.765+0000 7fd1a13ef740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729ffa00 0x55e172a73f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd1a13ef740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd1981b2a60 con 0x55e1729f5140 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd1a13ef740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd1981b3ed0 con 0x55e1729ffa00 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd1a13ef740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd1981b3130 con 0x55e1729f4820 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd1a13ef740 10 client.0 _ll_register_callbacks cb 0x7fd1840075b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd197400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729f4820 0x55e172a04bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd197e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729ffa00 0x55e172a73f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd197400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729f4820 0x55e172a04bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59018/0 (socket says 172.21.15.6:59018) 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd197400640 1 -- 172.21.15.6:0/2868819516 learned_addr learned my addr 172.21.15.6:0/2868819516 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd196a00640 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 0x55e172a05100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd197400640 1 -- 172.21.15.6:0/2868819516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729ffa00 msgr2=0x55e172a73f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd197400640 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729ffa00 0x55e172a73f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd197400640 1 -- 172.21.15.6:0/2868819516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 msgr2=0x55e172a05100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd197400640 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 0x55e172a05100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd197400640 1 -- 172.21.15.6:0/2868819516 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e172a74440 con 0x55e1729f4820 2024-04-26T02:01:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.766+0000 7fd197e00640 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729ffa00 0x55e172a73f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:01:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.767+0000 7fd197400640 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729f4820 0x55e172a04bc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd18809c560 tx=0x7fd188000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d69d8d1e28c4d2f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.767+0000 7fd181600640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:01:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.767+0000 7fd181600640 1 -- 172.21.15.6:0/2868819516 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1880d6020 con 0x55e1729f4820 2024-04-26T02:01:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.767+0000 7fd181600640 1 -- 172.21.15.6:0/2868819516 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8600+0+0 (secure 0 0 0) 0x7fd188098ed0 con 0x55e1729f4820 2024-04-26T02:01:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.767+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2868819516 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd18809cf60 con 0x55e1729f4820 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.768+0000 7fd181600640 1 -- 172.21.15.6:0/2868819516 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fd1880ba070 con 0x55e1729f4820 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.768+0000 7fd1a13ef740 10 client.27124 fetch_fsmap learned FSMap version 49 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.768+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2868819516 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e172a756c0 con 0x55e1729f4820 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.768+0000 7fd181600640 1 -- 172.21.15.6:0/2868819516 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd18809cf60 con 0x55e1729f4820 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.768+0000 7fd181600640 1 -- 172.21.15.6:0/2868819516 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fd1880ce320 con 0x55e1729f4820 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.768+0000 7fd1a13ef740 10 client.27124 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.768+0000 7fd1a13ef740 10 client.27124 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.768+0000 7fd1a13ef740 20 client.27124 populate_metadata read hostname 'smithi006' 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.768+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e172a75df0 0x55e172a78210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.768+0000 7fd1a13ef740 4 client.27124 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.768+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2868819516 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e172968320 con 0x55e172a75df0 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.769+0000 7fd196a00640 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e172a75df0 0x55e172a78210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.769+0000 7fd196a00640 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e172a75df0 0x55e172a78210 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f4892a70e0837ca server_cookie=5eba0f20178768a9 in_seq=0 out_seq=0 2024-04-26T02:01:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.769+0000 7fd181600640 10 client.27124 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:52.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:52 smithi006 ceph-mon[28419]: pgmap v506: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 11 MiB/s wr, 763 op/s 2024-04-26T02:01:52.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:52 smithi188 ceph-mon[32751]: pgmap v506: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 11 MiB/s wr, 763 op/s 2024-04-26T02:01:52.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:52 smithi029 ceph-mon[32775]: pgmap v506: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 11 MiB/s wr, 763 op/s 2024-04-26T02:01:52.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd181600640 1 -- 172.21.15.6:0/2868819516 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e172968320 con 0x55e172a75df0 2024-04-26T02:01:52.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd181600640 10 client.27124 handle_command_reply: tid=0 2024-04-26T02:01:52.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd1a13ef740 1 client.27124 shutdown 2024-04-26T02:01:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2868819516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729f4820 msgr2=0x55e172a04bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729f4820 0x55e172a04bc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd18809c560 tx=0x7fd188000bc0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2868819516 shutdown_connections 2024-04-26T02:01:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55e172a75df0 0x55e172a78210 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e1729f5140 0x55e172a05100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1729ffa00 0x55e172a73f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd1a13ef740 1 --2- 172.21.15.6:0/2868819516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1729f4820 0x55e172a04bc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd181600640 0 client.27124 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:01:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2868819516 >> 172.21.15.6:0/2868819516 conn(0x55e1729f0d30 msgr2=0x55e1729fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2868819516 shutdown_connections 2024-04-26T02:01:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.962+0000 7fd1a13ef740 1 -- 172.21.15.6:0/2868819516 wait complete. 2024-04-26T02:01:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.963+0000 7fd1a13ef740 20 client.27124 trim_cache size 0 max 16384 2024-04-26T02:01:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:01:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:01:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:01:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:01:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.963+0000 7fd19e600640 1 -- 172.21.15.6:0/101342585 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd17c0a3d90 msgr2=0x7fd17c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.963+0000 7fd19e600640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd17c0a3d90 0x7fd17c0a6250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd190009430 tx=0x7fd190009800 comp rx=0 tx=0).stop 2024-04-26T02:01:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.964+0000 7fd19e600640 1 -- 172.21.15.6:0/101342585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd198076480 msgr2=0x7fd19819edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.964+0000 7fd19e600640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd198076480 0x7fd19819edb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c920 tx=0x7fd18800cde0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.964+0000 7fd19e600640 1 -- 172.21.15.6:0/101342585 shutdown_connections 2024-04-26T02:01:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.964+0000 7fd19e600640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd17c0a3d90 0x7fd17c0a6250 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.964+0000 7fd19e600640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd198076da0 0x7fd19819f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.964+0000 7fd19e600640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd198112df0 0x7fd1981a3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.964+0000 7fd19e600640 1 --2- 172.21.15.6:0/101342585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd198076480 0x7fd19819edb0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.964+0000 7fd19e600640 1 -- 172.21.15.6:0/101342585 >> 172.21.15.6:0/101342585 conn(0x7fd1980ffc20 msgr2=0x7fd198114440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.964+0000 7fd19e600640 1 -- 172.21.15.6:0/101342585 shutdown_connections 2024-04-26T02:01:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:01:52.964+0000 7fd19e600640 1 -- 172.21.15.6:0/101342585 wait complete. 2024-04-26T02:01:53.106 INFO:tasks.workunit.client.0.smithi006.stdout: 10 42981 36.17 MB/sec execute 16 sec latency 124.764 ms 2024-04-26T02:01:54.106 INFO:tasks.workunit.client.0.smithi006.stdout: 10 43612 36.06 MB/sec execute 17 sec latency 44.170 ms 2024-04-26T02:01:54.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:54 smithi006 ceph-mon[28419]: pgmap v507: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.0 MiB/s wr, 463 op/s 2024-04-26T02:01:54.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:54 smithi188 ceph-mon[32751]: pgmap v507: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.0 MiB/s wr, 463 op/s 2024-04-26T02:01:54.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:54 smithi029 ceph-mon[32775]: pgmap v507: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.0 MiB/s wr, 463 op/s 2024-04-26T02:01:55.106 INFO:tasks.workunit.client.0.smithi006.stdout: 10 44342 36.11 MB/sec execute 18 sec latency 126.222 ms 2024-04-26T02:01:56.107 INFO:tasks.workunit.client.0.smithi006.stdout: 10 45086 36.31 MB/sec execute 19 sec latency 50.775 ms 2024-04-26T02:01:56.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:56 smithi006 ceph-mon[28419]: pgmap v508: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.0 MiB/s wr, 463 op/s 2024-04-26T02:01:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:56 smithi188 ceph-mon[32751]: pgmap v508: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.0 MiB/s wr, 463 op/s 2024-04-26T02:01:56.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:56 smithi029 ceph-mon[32775]: pgmap v508: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.0 MiB/s wr, 463 op/s 2024-04-26T02:01:57.107 INFO:tasks.workunit.client.0.smithi006.stdout: 10 45678 36.49 MB/sec execute 20 sec latency 187.118 ms 2024-04-26T02:01:58.107 INFO:tasks.workunit.client.0.smithi006.stdout: 10 46216 35.86 MB/sec execute 21 sec latency 81.952 ms 2024-04-26T02:01:58.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:01:58 smithi006 ceph-mon[28419]: pgmap v509: 97 pgs: 97 active+clean; 599 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 11 MiB/s wr, 801 op/s 2024-04-26T02:01:58.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:01:58 smithi188 ceph-mon[32751]: pgmap v509: 97 pgs: 97 active+clean; 599 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 11 MiB/s wr, 801 op/s 2024-04-26T02:01:58.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:01:58 smithi029 ceph-mon[32775]: pgmap v509: 97 pgs: 97 active+clean; 599 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 11 MiB/s wr, 801 op/s 2024-04-26T02:01:59.107 INFO:tasks.workunit.client.0.smithi006.stdout: 10 46826 35.72 MB/sec execute 22 sec latency 103.800 ms 2024-04-26T02:02:00.107 INFO:tasks.workunit.client.0.smithi006.stdout: 10 47536 35.59 MB/sec execute 23 sec latency 107.343 ms 2024-04-26T02:02:00.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:00 smithi006 ceph-mon[28419]: pgmap v510: 97 pgs: 97 active+clean; 599 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.6 MiB/s wr, 591 op/s 2024-04-26T02:02:00.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:00 smithi188 ceph-mon[32751]: pgmap v510: 97 pgs: 97 active+clean; 599 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.6 MiB/s wr, 591 op/s 2024-04-26T02:02:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:00 smithi029 ceph-mon[32775]: pgmap v510: 97 pgs: 97 active+clean; 599 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.6 MiB/s wr, 591 op/s 2024-04-26T02:02:01.107 INFO:tasks.workunit.client.0.smithi006.stdout: 10 47956 34.78 MB/sec execute 24 sec latency 377.525 ms 2024-04-26T02:02:02.107 INFO:tasks.workunit.client.0.smithi006.stdout: 10 48021 33.70 MB/sec execute 25 sec latency 931.810 ms 2024-04-26T02:02:02.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:02 smithi006 ceph-mon[28419]: pgmap v511: 97 pgs: 97 active+clean; 616 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 849 op/s 2024-04-26T02:02:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:02 smithi188 ceph-mon[32751]: pgmap v511: 97 pgs: 97 active+clean; 616 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 849 op/s 2024-04-26T02:02:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:02 smithi029 ceph-mon[32775]: pgmap v511: 97 pgs: 97 active+clean; 616 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 849 op/s 2024-04-26T02:02:03.107 INFO:tasks.workunit.client.0.smithi006.stdout: 10 48021 32.41 MB/sec execute 26 sec latency 1327.186 ms 2024-04-26T02:02:04.109 INFO:tasks.workunit.client.0.smithi006.stdout: 10 48103 31.36 MB/sec execute 27 sec latency 2182.695 ms 2024-04-26T02:02:04.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:04 smithi006 ceph-mon[28419]: pgmap v512: 97 pgs: 97 active+clean; 616 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 7.4 MiB/s wr, 595 op/s 2024-04-26T02:02:04.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:04 smithi188 ceph-mon[32751]: pgmap v512: 97 pgs: 97 active+clean; 616 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 7.4 MiB/s wr, 595 op/s 2024-04-26T02:02:04.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:04 smithi029 ceph-mon[32775]: pgmap v512: 97 pgs: 97 active+clean; 616 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 7.4 MiB/s wr, 595 op/s 2024-04-26T02:02:05.107 INFO:tasks.workunit.client.0.smithi006.stdout: 10 48526 31.18 MB/sec execute 28 sec latency 342.393 ms 2024-04-26T02:02:05.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:05 smithi006 ceph-mon[28419]: osdmap e122: 12 total, 12 up, 12 in 2024-04-26T02:02:05.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:05 smithi188 ceph-mon[32751]: osdmap e122: 12 total, 12 up, 12 in 2024-04-26T02:02:05.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:05 smithi029 ceph-mon[32775]: osdmap e122: 12 total, 12 up, 12 in 2024-04-26T02:02:06.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 48820 30.66 MB/sec execute 29 sec latency 440.972 ms 2024-04-26T02:02:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.151+0000 7f922c600640 1 -- 172.21.15.6:0/1749497193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f922410e980 msgr2=0x7f922410ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.151+0000 7f922c600640 1 --2- 172.21.15.6:0/1749497193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f922410e980 0x7f922410ed60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9218009890 tx=0x7f921802a360 comp rx=0 tx=0).stop 2024-04-26T02:02:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.152+0000 7f922c600640 1 -- 172.21.15.6:0/1749497193 shutdown_connections 2024-04-26T02:02:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.152+0000 7f922c600640 1 --2- 172.21.15.6:0/1749497193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92241098f0 0x7f9224115840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.152+0000 7f922c600640 1 --2- 172.21.15.6:0/1749497193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9224108f50 0x7f92241093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.152+0000 7f922c600640 1 --2- 172.21.15.6:0/1749497193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f922410e980 0x7f922410ed60 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.152+0000 7f922c600640 1 -- 172.21.15.6:0/1749497193 >> 172.21.15.6:0/1749497193 conn(0x7f922406d030 msgr2=0x7f922406d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.152+0000 7f922c600640 1 -- 172.21.15.6:0/1749497193 shutdown_connections 2024-04-26T02:02:06.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.152+0000 7f922c600640 1 -- 172.21.15.6:0/1749497193 wait complete. 2024-04-26T02:02:06.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.153+0000 7f922c600640 1 Processor -- start 2024-04-26T02:02:06.152 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.154+0000 7f922c600640 1 -- start start 2024-04-26T02:02:06.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.155+0000 7f922c600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9224108f50 0x7f92241a7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.155+0000 7f922c600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92241098f0 0x7f92241a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.155+0000 7f922c600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f922410e980 0x7f92241abe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.154 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.155+0000 7f9229c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9224108f50 0x7f92241a7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.155+0000 7f9229c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9224108f50 0x7f92241a7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:43618/0 (socket says 172.21.15.6:43618) 2024-04-26T02:02:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.156+0000 7f922c600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9224121030 con 0x7f922410e980 2024-04-26T02:02:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.156+0000 7f922c600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9224120eb0 con 0x7f92241098f0 2024-04-26T02:02:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.156+0000 7f922c600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f92241211b0 con 0x7f9224108f50 2024-04-26T02:02:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.156+0000 7f9229c00640 1 -- 172.21.15.6:0/3618210566 learned_addr learned my addr 172.21.15.6:0/3618210566 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.156+0000 7f9229200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92241098f0 0x7f92241a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.156+0000 7f922a600640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f922410e980 0x7f92241abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.156+0000 7f9229200640 1 -- 172.21.15.6:0/3618210566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9224108f50 msgr2=0x7f92241a7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:06.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.156+0000 7f9229200640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9224108f50 0x7f92241a7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.156+0000 7f9229200640 1 -- 172.21.15.6:0/3618210566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f922410e980 msgr2=0x7f92241abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:06.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.156+0000 7f9229200640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f922410e980 0x7f92241abe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.156+0000 7f9229200640 1 -- 172.21.15.6:0/3618210566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92241ac580 con 0x7f92241098f0 2024-04-26T02:02:06.156 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.157+0000 7f9229200640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92241098f0 0x7f92241a7ad0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f92140095a0 tx=0x7f9214009a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.158+0000 7f9222000640 1 -- 172.21.15.6:0/3618210566 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9214018020 con 0x7f92241098f0 2024-04-26T02:02:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.158+0000 7f9222000640 1 -- 172.21.15.6:0/3618210566 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9214014cd0 con 0x7f92241098f0 2024-04-26T02:02:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.158+0000 7f9222000640 1 -- 172.21.15.6:0/3618210566 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9214013d60 con 0x7f92241098f0 2024-04-26T02:02:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.158+0000 7f922c600640 1 -- 172.21.15.6:0/3618210566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92241ac870 con 0x7f92241098f0 2024-04-26T02:02:06.158 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.158+0000 7f922c600640 1 -- 172.21.15.6:0/3618210566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92241b4100 con 0x7f92241098f0 2024-04-26T02:02:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.165+0000 7f9222000640 1 -- 172.21.15.6:0/3618210566 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f921401b020 con 0x7f92241098f0 2024-04-26T02:02:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.168+0000 7f9222000640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f920c0a3d90 0x7f920c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.168+0000 7f9222000640 1 -- 172.21.15.6:0/3618210566 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8644+0+0 (secure 0 0 0) 0x7f92140d0410 con 0x7f92241098f0 2024-04-26T02:02:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.169+0000 7f9229c00640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f920c0a3d90 0x7f920c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.169+0000 7f922f417740 1 Processor -- start 2024-04-26T02:02:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.170+0000 7f922f417740 1 -- start start 2024-04-26T02:02:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.171+0000 7f9229c00640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f920c0a3d90 0x7f920c0a6250 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f92180095f0 tx=0x7f92180056d0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.172+0000 7f922f417740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c8820 0x5558dc8c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.172+0000 7f922f417740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8c9140 0x5558dc8d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.172+0000 7f922f417740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8d3a00 0x5558dc8d5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f9229200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8c9140 0x5558dc8d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f9229200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8c9140 0x5558dc8d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:58862/0 (socket says 172.21.15.6:58862) 2024-04-26T02:02:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f9229c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c8820 0x5558dc8c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f9229c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c8820 0x5558dc8c8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41068/0 (socket says 172.21.15.6:41068) 2024-04-26T02:02:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f922f417740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5558dc82e900 con 0x5558dc8c8820 2024-04-26T02:02:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f9229200640 1 -- 172.21.15.6:0/656650546 learned_addr learned my addr 172.21.15.6:0/656650546 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f922f417740 1 -- 172.21.15.6:0/656650546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5558dc7d4ef0 con 0x5558dc8c9140 2024-04-26T02:02:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f922f417740 1 -- 172.21.15.6:0/656650546 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5558dc82a8c0 con 0x5558dc8d3a00 2024-04-26T02:02:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f9229200640 1 -- 172.21.15.6:0/656650546 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8d3a00 msgr2=0x5558dc8d5df0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T02:02:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f9229200640 1 --2- 172.21.15.6:0/656650546 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8d3a00 0x5558dc8d5df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f9229200640 1 -- 172.21.15.6:0/656650546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c8820 msgr2=0x5558dc8c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f9229200640 1 --2- 172.21.15.6:0/656650546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c8820 0x5558dc8c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.173+0000 7f9229200640 1 -- 172.21.15.6:0/656650546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558dc541210 con 0x5558dc8c9140 2024-04-26T02:02:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.174+0000 7f9229200640 1 --2- 172.21.15.6:0/656650546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8c9140 0x5558dc8d34c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9214098790 tx=0x7f921409ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13709b87be23e2e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.174+0000 7f9212a00640 1 -- 172.21.15.6:0/656650546 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92140a06e0 con 0x5558dc8c9140 2024-04-26T02:02:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.174+0000 7f9212a00640 1 -- 172.21.15.6:0/656650546 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92140a4fa0 con 0x5558dc8c9140 2024-04-26T02:02:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.174+0000 7f9212a00640 1 -- 172.21.15.6:0/656650546 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921409b9e0 con 0x5558dc8c9140 2024-04-26T02:02:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.177+0000 7f922f417740 1 -- 172.21.15.6:0/656650546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8c9140 msgr2=0x5558dc8d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.177+0000 7f922f417740 1 --2- 172.21.15.6:0/656650546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8c9140 0x5558dc8d34c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9214098790 tx=0x7f921409ca30 comp rx=0 tx=0).stop 2024-04-26T02:02:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.177+0000 7f922f417740 1 -- 172.21.15.6:0/656650546 shutdown_connections 2024-04-26T02:02:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.177+0000 7f922f417740 1 --2- 172.21.15.6:0/656650546 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8d3a00 0x5558dc8d5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.177+0000 7f922f417740 1 --2- 172.21.15.6:0/656650546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8c9140 0x5558dc8d34c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.177+0000 7f922f417740 1 --2- 172.21.15.6:0/656650546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c8820 0x5558dc8c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.177+0000 7f922f417740 1 -- 172.21.15.6:0/656650546 >> 172.21.15.6:0/656650546 conn(0x5558dc8c4d30 msgr2=0x5558dc8c5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:06.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.177+0000 7f922f417740 1 -- 172.21.15.6:0/656650546 shutdown_connections 2024-04-26T02:02:06.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.177+0000 7f922f417740 1 -- 172.21.15.6:0/656650546 wait complete. 2024-04-26T02:02:06.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.178+0000 7f922f417740 1 -- start start 2024-04-26T02:02:06.177 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.178+0000 7f922f417740 1 Processor -- start 2024-04-26T02:02:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.179+0000 7f922f417740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8c8820 0x5558dc8d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.179+0000 7f922f417740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c9140 0x5558dc8d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.179+0000 7f922f417740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8d3a00 0x5558dc8d1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.179+0000 7f922f417740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5558dc8d6630 con 0x5558dc8c9140 2024-04-26T02:02:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.179+0000 7f922f417740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5558dc7e8310 con 0x5558dc8d3a00 2024-04-26T02:02:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.179+0000 7f922f417740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5558dc8d67b0 con 0x5558dc8c8820 2024-04-26T02:02:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.179+0000 7f922f417740 10 client.0 _ll_register_callbacks cb 0x5558dc7b6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.179+0000 7f9229200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c9140 0x5558dc8d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.179+0000 7f9229200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c9140 0x5558dc8d0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41072/0 (socket says 172.21.15.6:41072) 2024-04-26T02:02:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.179+0000 7f9229200640 1 -- 172.21.15.6:0/146492631 learned_addr learned my addr 172.21.15.6:0/146492631 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.179+0000 7f9229c00640 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8c8820 0x5558dc8d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.180+0000 7f9229200640 1 -- 172.21.15.6:0/146492631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8c8820 msgr2=0x5558dc8d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.180+0000 7f9229200640 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8c8820 0x5558dc8d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.180+0000 7f9229200640 1 -- 172.21.15.6:0/146492631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8d3a00 msgr2=0x5558dc8d1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.180+0000 7f922a600640 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8d3a00 0x5558dc8d1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.180+0000 7f9229200640 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8d3a00 0x5558dc8d1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.180+0000 7f9229200640 1 -- 172.21.15.6:0/146492631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558dc60d620 con 0x5558dc8c9140 2024-04-26T02:02:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.180+0000 7f9229c00640 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8c8820 0x5558dc8d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:02:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.180+0000 7f9229200640 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c9140 0x5558dc8d0f40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f92140ce0e0 tx=0x7f92140cdac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe06c240639727e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.180+0000 7f9207400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:02:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.181+0000 7f9207400640 1 -- 172.21.15.6:0/146492631 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92140bb2c0 con 0x5558dc8c9140 2024-04-26T02:02:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.181+0000 7f922f417740 1 -- 172.21.15.6:0/146492631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92140a4fa0 con 0x5558dc8c9140 2024-04-26T02:02:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.181+0000 7f9207400640 1 -- 172.21.15.6:0/146492631 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8644+0+0 (secure 0 0 0) 0x7f9214014820 con 0x5558dc8c9140 2024-04-26T02:02:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.181+0000 7f9207400640 1 -- 172.21.15.6:0/146492631 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f9214014a20 con 0x5558dc8c9140 2024-04-26T02:02:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.181+0000 7f922f417740 10 client.16905 fetch_fsmap learned FSMap version 49 2024-04-26T02:02:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.181+0000 7f922f417740 1 -- 172.21.15.6:0/146492631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5558dc8f36f0 con 0x5558dc8c9140 2024-04-26T02:02:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.181+0000 7f9207400640 1 -- 172.21.15.6:0/146492631 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92140a4fa0 con 0x5558dc8c9140 2024-04-26T02:02:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.182+0000 7f9207400640 1 -- 172.21.15.6:0/146492631 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f92140981a0 con 0x5558dc8c9140 2024-04-26T02:02:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.182+0000 7f922f417740 10 client.16905 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:02:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.182+0000 7f922f417740 10 client.16905 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:02:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.182+0000 7f922f417740 20 client.16905 populate_metadata read hostname 'smithi006' 2024-04-26T02:02:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.182+0000 7f922f417740 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558dc8f4a40 0x5558dc8f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.182+0000 7f922f417740 4 client.16905 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:02:06.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.182+0000 7f922f417740 1 -- 172.21.15.6:0/146492631 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558dc8428b0 con 0x5558dc8f4a40 2024-04-26T02:02:06.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.183+0000 7f9229c00640 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558dc8f4a40 0x5558dc8f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.183+0000 7f9207400640 10 client.16905 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:06.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.183+0000 7f9229c00640 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558dc8f4a40 0x5558dc8f6e30 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69f8fc1d8afdadf9 server_cookie=d71efb7e89a9db37 in_seq=0 out_seq=0 2024-04-26T02:02:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.185+0000 7f9207400640 1 -- 172.21.15.6:0/146492631 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5558dc8428b0 con 0x5558dc8f4a40 2024-04-26T02:02:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.185+0000 7f9207400640 10 client.16905 handle_command_reply: tid=0 2024-04-26T02:02:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.185+0000 7f922f417740 1 client.16905 shutdown 2024-04-26T02:02:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.185+0000 7f922f417740 1 -- 172.21.15.6:0/146492631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c9140 msgr2=0x5558dc8d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.185+0000 7f922f417740 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c9140 0x5558dc8d0f40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f92140ce0e0 tx=0x7f92140cdac0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.186+0000 7f922f417740 1 -- 172.21.15.6:0/146492631 shutdown_connections 2024-04-26T02:02:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.186+0000 7f922f417740 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558dc8f4a40 0x5558dc8f6e30 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.186+0000 7f922f417740 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c9140 0x5558dc8d0f40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.186 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.186+0000 7f922f417740 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8d3a00 0x5558dc8d1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.186 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.186+0000 7f922f417740 1 --2- 172.21.15.6:0/146492631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8c8820 0x5558dc8d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.186 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.186+0000 7f9207400640 0 client.16905 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:06.186 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.186+0000 7f922f417740 1 -- 172.21.15.6:0/146492631 >> 172.21.15.6:0/146492631 conn(0x5558dc8c4d30 msgr2=0x5558dc8caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:06.186 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.186+0000 7f922f417740 1 -- 172.21.15.6:0/146492631 shutdown_connections 2024-04-26T02:02:06.186 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.186+0000 7f922f417740 1 -- 172.21.15.6:0/146492631 wait complete. 2024-04-26T02:02:06.186 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.186+0000 7f922f417740 20 client.16905 trim_cache size 0 max 16384 2024-04-26T02:02:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.212+0000 7f922f417740 1 Processor -- start 2024-04-26T02:02:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.212+0000 7f922f417740 1 -- start start 2024-04-26T02:02:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.212+0000 7f922f417740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8c8820 0x5558dc8f9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.212+0000 7f922f417740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c9140 0x5558dc8fa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.212+0000 7f922f417740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8d3a00 0x5558dc8d09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.212+0000 7f922f417740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f92241b6ed0 con 0x5558dc8c9140 2024-04-26T02:02:06.211 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.212+0000 7f922f417740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f92241b4710 con 0x5558dc8d3a00 2024-04-26T02:02:06.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.212+0000 7f922f417740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f922411d6a0 con 0x5558dc8c8820 2024-04-26T02:02:06.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.213+0000 7f922a600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8d3a00 0x5558dc8d09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.213+0000 7f9229c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8c8820 0x5558dc8f9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.213+0000 7f9229200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c9140 0x5558dc8fa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.213+0000 7f922a600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8d3a00 0x5558dc8d09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:58876/0 (socket says 172.21.15.6:58876) 2024-04-26T02:02:06.212 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.213+0000 7f922a600640 1 -- 172.21.15.6:0/1815708631 learned_addr learned my addr 172.21.15.6:0/1815708631 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:06.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.213+0000 7f922a600640 1 -- 172.21.15.6:0/1815708631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8c8820 msgr2=0x5558dc8f9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:06.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.213+0000 7f922a600640 1 --2- 172.21.15.6:0/1815708631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8c8820 0x5558dc8f9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.213+0000 7f922a600640 1 -- 172.21.15.6:0/1815708631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c9140 msgr2=0x5558dc8fa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:06.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.213+0000 7f922a600640 1 --2- 172.21.15.6:0/1815708631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c9140 0x5558dc8fa390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.213+0000 7f922a600640 1 -- 172.21.15.6:0/1815708631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558dc8c0690 con 0x5558dc8d3a00 2024-04-26T02:02:06.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.214+0000 7f922a600640 1 --2- 172.21.15.6:0/1815708631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8d3a00 0x5558dc8d09c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f921c009d30 tx=0x7f921c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df4816b56e78e67c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:06.213 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.214+0000 7f9212000640 1 -- 172.21.15.6:0/1815708631 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c004d20 con 0x5558dc8d3a00 2024-04-26T02:02:06.214 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.215+0000 7f9212000640 1 -- 172.21.15.6:0/1815708631 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f921c009e20 con 0x5558dc8d3a00 2024-04-26T02:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.215+0000 7f9212000640 1 -- 172.21.15.6:0/1815708631 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c004860 con 0x5558dc8d3a00 2024-04-26T02:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.215+0000 7f922f417740 1 -- 172.21.15.6:0/1815708631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8d3a00 msgr2=0x5558dc8d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.215+0000 7f922f417740 1 --2- 172.21.15.6:0/1815708631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8d3a00 0x5558dc8d09c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f921c009d30 tx=0x7f921c0117b0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.215+0000 7f922f417740 1 -- 172.21.15.6:0/1815708631 shutdown_connections 2024-04-26T02:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.215+0000 7f922f417740 1 --2- 172.21.15.6:0/1815708631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c9140 0x5558dc8fa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.216+0000 7f922f417740 1 --2- 172.21.15.6:0/1815708631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8d3a00 0x5558dc8d09c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.216+0000 7f922f417740 1 --2- 172.21.15.6:0/1815708631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8c8820 0x5558dc8f9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.216+0000 7f922f417740 1 -- 172.21.15.6:0/1815708631 >> 172.21.15.6:0/1815708631 conn(0x5558dc8c4d30 msgr2=0x5558dc8bfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:06.215 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.216+0000 7f922f417740 1 -- 172.21.15.6:0/1815708631 shutdown_connections 2024-04-26T02:02:06.216 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.217+0000 7f922f417740 1 -- 172.21.15.6:0/1815708631 wait complete. 2024-04-26T02:02:06.216 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.217+0000 7f922f417740 1 -- start start 2024-04-26T02:02:06.216 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.217+0000 7f922f417740 1 Processor -- start 2024-04-26T02:02:06.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.218+0000 7f922f417740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c8820 0x5558dc9534a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.218+0000 7f922f417740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8c9140 0x5558dc9539e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.217 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.218+0000 7f922f417740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8d3a00 0x5558dc8f95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.218+0000 7f922f417740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f922411dcf0 con 0x5558dc8c8820 2024-04-26T02:02:06.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.218+0000 7f922f417740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f922411f160 con 0x5558dc8c9140 2024-04-26T02:02:06.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.218+0000 7f922f417740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f922411e3c0 con 0x5558dc8d3a00 2024-04-26T02:02:06.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.218+0000 7f922f417740 10 client.0 _ll_register_callbacks cb 0x7f9214013370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:06.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.218+0000 7f922a600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8d3a00 0x5558dc8f95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.218+0000 7f922a600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8d3a00 0x5558dc8f95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:43656/0 (socket says 172.21.15.6:43656) 2024-04-26T02:02:06.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.218+0000 7f922a600640 1 -- 172.21.15.6:0/679009656 learned_addr learned my addr 172.21.15.6:0/679009656 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:06.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.218+0000 7f9229c00640 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c8820 0x5558dc9534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.218+0000 7f9229200640 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8c9140 0x5558dc9539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.218 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.219+0000 7f922a600640 1 -- 172.21.15.6:0/679009656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8c9140 msgr2=0x5558dc9539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:06.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.219+0000 7f922a600640 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8c9140 0x5558dc9539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.219+0000 7f922a600640 1 -- 172.21.15.6:0/679009656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c8820 msgr2=0x5558dc9534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:06.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.219+0000 7f922a600640 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c8820 0x5558dc9534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.219+0000 7f922a600640 1 -- 172.21.15.6:0/679009656 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558dc8f9e10 con 0x5558dc8d3a00 2024-04-26T02:02:06.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.220+0000 7f922a600640 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8d3a00 0x5558dc8f95e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f921c004690 tx=0x7f921c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32ef4cb0ad6b3410 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:06.219 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.220+0000 7f9207400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:02:06.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.221+0000 7f9207400640 1 -- 172.21.15.6:0/679009656 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c02a060 con 0x5558dc8d3a00 2024-04-26T02:02:06.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.221+0000 7f922f417740 1 -- 172.21.15.6:0/679009656 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f921c009e20 con 0x5558dc8d3a00 2024-04-26T02:02:06.220 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.221+0000 7f9207400640 1 -- 172.21.15.6:0/679009656 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8644+0+0 (secure 0 0 0) 0x7f921c003d80 con 0x5558dc8d3a00 2024-04-26T02:02:06.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.221+0000 7f9207400640 1 -- 172.21.15.6:0/679009656 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f921c009a50 con 0x5558dc8d3a00 2024-04-26T02:02:06.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.221+0000 7f922f417740 10 client.27139 fetch_fsmap learned FSMap version 49 2024-04-26T02:02:06.221 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.221+0000 7f922f417740 1 -- 172.21.15.6:0/679009656 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5558dc95ad30 con 0x5558dc8d3a00 2024-04-26T02:02:06.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.222+0000 7f9207400640 1 -- 172.21.15.6:0/679009656 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f921c009e20 con 0x5558dc8d3a00 2024-04-26T02:02:06.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.222+0000 7f9207400640 1 -- 172.21.15.6:0/679009656 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f921c005810 con 0x5558dc8d3a00 2024-04-26T02:02:06.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.222+0000 7f922f417740 10 client.27139 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T02:02:06.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.222+0000 7f922f417740 10 client.27139 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:02:06.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.222+0000 7f922f417740 20 client.27139 populate_metadata read hostname 'smithi006' 2024-04-26T02:02:06.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.222+0000 7f922f417740 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558dc95b3d0 0x5558dc95d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:06.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.222+0000 7f922f417740 4 client.27139 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:02:06.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.222+0000 7f922f417740 1 -- 172.21.15.6:0/679009656 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5558dc83c320 con 0x5558dc95b3d0 2024-04-26T02:02:06.222 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.223+0000 7f9229c00640 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558dc95b3d0 0x5558dc95d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:06.223 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.223+0000 7f9229c00640 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558dc95b3d0 0x5558dc95d810 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a593608847023657 server_cookie=a7cbf9dfe1d6940f in_seq=0 out_seq=0 2024-04-26T02:02:06.223 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.224+0000 7f9207400640 10 client.27139 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.452+0000 7f9207400640 1 -- 172.21.15.6:0/679009656 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5558dc83c320 con 0x5558dc95b3d0 2024-04-26T02:02:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f9207400640 10 client.27139 handle_command_reply: tid=0 2024-04-26T02:02:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f922f417740 1 client.27139 shutdown 2024-04-26T02:02:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f922f417740 1 -- 172.21.15.6:0/679009656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8d3a00 msgr2=0x5558dc8f95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f922f417740 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8d3a00 0x5558dc8f95e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f921c004690 tx=0x7f921c0046c0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f922f417740 1 -- 172.21.15.6:0/679009656 shutdown_connections 2024-04-26T02:02:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f922f417740 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5558dc95b3d0 0x5558dc95d810 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f922f417740 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558dc8d3a00 0x5558dc8f95e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f922f417740 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558dc8c9140 0x5558dc9539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f9207400640 0 client.27139 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f922f417740 1 --2- 172.21.15.6:0/679009656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5558dc8c8820 0x5558dc9534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f922f417740 1 -- 172.21.15.6:0/679009656 >> 172.21.15.6:0/679009656 conn(0x5558dc8c4d30 msgr2=0x5558dc8dea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f922f417740 1 -- 172.21.15.6:0/679009656 shutdown_connections 2024-04-26T02:02:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f922f417740 1 -- 172.21.15.6:0/679009656 wait complete. 2024-04-26T02:02:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.453+0000 7f922f417740 20 client.27139 trim_cache size 0 max 16384 2024-04-26T02:02:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:02:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:02:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:02:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:02:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.454+0000 7f922c600640 1 -- 172.21.15.6:0/3618210566 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f920c0a3d90 msgr2=0x7f920c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.454+0000 7f922c600640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f920c0a3d90 0x7f920c0a6250 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f92180095f0 tx=0x7f92180056d0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.454+0000 7f922c600640 1 -- 172.21.15.6:0/3618210566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92241098f0 msgr2=0x7f92241a7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.454+0000 7f922c600640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92241098f0 0x7f92241a7ad0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f92140095a0 tx=0x7f9214009a60 comp rx=0 tx=0).stop 2024-04-26T02:02:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.454+0000 7f922c600640 1 -- 172.21.15.6:0/3618210566 shutdown_connections 2024-04-26T02:02:06.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.454+0000 7f922c600640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f920c0a3d90 0x7f920c0a6250 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.454+0000 7f922c600640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f922410e980 0x7f92241abe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.454+0000 7f922c600640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92241098f0 0x7f92241a7ad0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.454+0000 7f922c600640 1 --2- 172.21.15.6:0/3618210566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9224108f50 0x7f92241a7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:06.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.454+0000 7f922c600640 1 -- 172.21.15.6:0/3618210566 >> 172.21.15.6:0/3618210566 conn(0x7f922406d030 msgr2=0x7f9224072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:06.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.454+0000 7f922c600640 1 -- 172.21.15.6:0/3618210566 shutdown_connections 2024-04-26T02:02:06.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:06.455+0000 7f922c600640 1 -- 172.21.15.6:0/3618210566 wait complete. 2024-04-26T02:02:06.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:06 smithi006 ceph-mon[28419]: pgmap v514: 97 pgs: 97 active+clean; 616 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.9 MiB/s wr, 714 op/s 2024-04-26T02:02:06.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:06 smithi188 ceph-mon[32751]: pgmap v514: 97 pgs: 97 active+clean; 616 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.9 MiB/s wr, 714 op/s 2024-04-26T02:02:06.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:06 smithi029 ceph-mon[32775]: pgmap v514: 97 pgs: 97 active+clean; 616 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.9 MiB/s wr, 714 op/s 2024-04-26T02:02:07.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 49263 30.76 MB/sec execute 30 sec latency 151.066 ms 2024-04-26T02:02:08.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 49675 30.38 MB/sec execute 31 sec latency 235.133 ms 2024-04-26T02:02:08.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:08 smithi006 ceph-mon[28419]: pgmap v515: 97 pgs: 97 active+clean; 620 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 10 MiB/s wr, 679 op/s 2024-04-26T02:02:08.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:08 smithi188 ceph-mon[32751]: pgmap v515: 97 pgs: 97 active+clean; 620 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 10 MiB/s wr, 679 op/s 2024-04-26T02:02:08.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:08 smithi029 ceph-mon[32775]: pgmap v515: 97 pgs: 97 active+clean; 620 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 10 MiB/s wr, 679 op/s 2024-04-26T02:02:09.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 50228 30.31 MB/sec execute 32 sec latency 40.256 ms 2024-04-26T02:02:10.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 50945 30.56 MB/sec execute 33 sec latency 75.060 ms 2024-04-26T02:02:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:10 smithi006 ceph-mon[28419]: pgmap v516: 97 pgs: 97 active+clean; 620 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 10 MiB/s wr, 679 op/s 2024-04-26T02:02:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:10 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:02:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:10 smithi006 ceph-mon[28419]: osdmap e123: 12 total, 12 up, 12 in 2024-04-26T02:02:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:10 smithi188 ceph-mon[32751]: pgmap v516: 97 pgs: 97 active+clean; 620 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 10 MiB/s wr, 679 op/s 2024-04-26T02:02:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:10 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:02:10.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:10 smithi188 ceph-mon[32751]: osdmap e123: 12 total, 12 up, 12 in 2024-04-26T02:02:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:10 smithi029 ceph-mon[32775]: pgmap v516: 97 pgs: 97 active+clean; 620 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 10 MiB/s wr, 679 op/s 2024-04-26T02:02:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:10 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:02:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:10 smithi029 ceph-mon[32775]: osdmap e123: 12 total, 12 up, 12 in 2024-04-26T02:02:11.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 51684 30.81 MB/sec execute 34 sec latency 70.200 ms 2024-04-26T02:02:12.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 52392 30.97 MB/sec execute 35 sec latency 77.730 ms 2024-04-26T02:02:12.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:12 smithi006 ceph-mon[28419]: pgmap v518: 97 pgs: 97 active+clean; 636 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 13 MiB/s wr, 1.03k op/s 2024-04-26T02:02:12.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:12 smithi188 ceph-mon[32751]: pgmap v518: 97 pgs: 97 active+clean; 636 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 13 MiB/s wr, 1.03k op/s 2024-04-26T02:02:12.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:12 smithi029 ceph-mon[32775]: pgmap v518: 97 pgs: 97 active+clean; 636 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 13 MiB/s wr, 1.03k op/s 2024-04-26T02:02:13.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 52911 30.97 MB/sec execute 36 sec latency 268.514 ms 2024-04-26T02:02:14.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 53505 30.81 MB/sec execute 37 sec latency 89.889 ms 2024-04-26T02:02:14.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:14 smithi006 ceph-mon[28419]: pgmap v519: 97 pgs: 97 active+clean; 636 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 12 MiB/s wr, 931 op/s 2024-04-26T02:02:14.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:14 smithi188 ceph-mon[32751]: pgmap v519: 97 pgs: 97 active+clean; 636 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 12 MiB/s wr, 931 op/s 2024-04-26T02:02:14.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:14 smithi029 ceph-mon[32775]: pgmap v519: 97 pgs: 97 active+clean; 636 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 12 MiB/s wr, 931 op/s 2024-04-26T02:02:15.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 54134 30.98 MB/sec execute 38 sec latency 167.160 ms 2024-04-26T02:02:16.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 54829 31.28 MB/sec execute 39 sec latency 59.572 ms 2024-04-26T02:02:16.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:16 smithi006 ceph-mon[28419]: pgmap v520: 97 pgs: 97 active+clean; 636 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 11 MiB/s wr, 827 op/s 2024-04-26T02:02:16.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:16 smithi188 ceph-mon[32751]: pgmap v520: 97 pgs: 97 active+clean; 636 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 11 MiB/s wr, 827 op/s 2024-04-26T02:02:16.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:16 smithi029 ceph-mon[32775]: pgmap v520: 97 pgs: 97 active+clean; 636 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 11 MiB/s wr, 827 op/s 2024-04-26T02:02:17.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 55513 31.39 MB/sec execute 40 sec latency 81.026 ms 2024-04-26T02:02:18.108 INFO:tasks.workunit.client.0.smithi006.stdout: 10 55936 31.20 MB/sec execute 41 sec latency 213.458 ms 2024-04-26T02:02:18.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:18 smithi006 ceph-mon[28419]: pgmap v521: 97 pgs: 97 active+clean; 663 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 798 op/s 2024-04-26T02:02:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:18 smithi188 ceph-mon[32751]: pgmap v521: 97 pgs: 97 active+clean; 663 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 798 op/s 2024-04-26T02:02:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:18 smithi029 ceph-mon[32775]: pgmap v521: 97 pgs: 97 active+clean; 663 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 798 op/s 2024-04-26T02:02:19.109 INFO:tasks.workunit.client.0.smithi006.stdout: 10 56557 31.15 MB/sec execute 42 sec latency 82.882 ms 2024-04-26T02:02:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.627+0000 7f420a200640 1 -- 172.21.15.6:0/2935879937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4204107280 msgr2=0x7f420410ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.627+0000 7f420a200640 1 --2- 172.21.15.6:0/2935879937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4204107280 0x7f420410ee50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f41fc009be0 tx=0x7f41fc02a6f0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.627+0000 7f420a200640 1 -- 172.21.15.6:0/2935879937 shutdown_connections 2024-04-26T02:02:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.627+0000 7f420a200640 1 --2- 172.21.15.6:0/2935879937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4204102b60 0x7f4204106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.627+0000 7f420a200640 1 --2- 172.21.15.6:0/2935879937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4204107280 0x7f420410ee50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.627+0000 7f420a200640 1 --2- 172.21.15.6:0/2935879937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42041021b0 0x7f4204102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.627+0000 7f420a200640 1 -- 172.21.15.6:0/2935879937 >> 172.21.15.6:0/2935879937 conn(0x7f42040fda10 msgr2=0x7f42040ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.628+0000 7f420a200640 1 -- 172.21.15.6:0/2935879937 shutdown_connections 2024-04-26T02:02:19.628 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.629+0000 7f420a200640 1 -- 172.21.15.6:0/2935879937 wait complete. 2024-04-26T02:02:19.629 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.630+0000 7f420a200640 1 Processor -- start 2024-04-26T02:02:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.630+0000 7f420a200640 1 -- start start 2024-04-26T02:02:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.630+0000 7f420a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42041021b0 0x7f4204198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.630+0000 7f420a200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4204107280 0x7f4204198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.630+0000 7f420a200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f420419ce40 0x7f420419d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.631+0000 7f420a200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4204112240 con 0x7f42041021b0 2024-04-26T02:02:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.631+0000 7f420a200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f42041120c0 con 0x7f4204107280 2024-04-26T02:02:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.631+0000 7f420a200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f42041123c0 con 0x7f420419ce40 2024-04-26T02:02:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.631+0000 7f4203e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f420419ce40 0x7f420419d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.631+0000 7f4203400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42041021b0 0x7f4204198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.631+0000 7f4203e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f420419ce40 0x7f420419d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45342/0 (socket says 172.21.15.6:45342) 2024-04-26T02:02:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.631+0000 7f4203e00640 1 -- 172.21.15.6:0/931109396 learned_addr learned my addr 172.21.15.6:0/931109396 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.632+0000 7f4203e00640 1 -- 172.21.15.6:0/931109396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4204107280 msgr2=0x7f4204198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.632+0000 7f4202a00640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4204107280 0x7f4204198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.632+0000 7f4203e00640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4204107280 0x7f4204198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.632+0000 7f4203e00640 1 -- 172.21.15.6:0/931109396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42041021b0 msgr2=0x7f4204198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.632+0000 7f4203e00640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42041021b0 0x7f4204198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.632+0000 7f4203e00640 1 -- 172.21.15.6:0/931109396 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420419d930 con 0x7f420419ce40 2024-04-26T02:02:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.632+0000 7f4202a00640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4204107280 0x7f4204198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.633+0000 7f4203e00640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f420419ce40 0x7f420419d2f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f41fc02ac10 tx=0x7f41fc0316e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.634+0000 7f41fbe00640 1 -- 172.21.15.6:0/931109396 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc041020 con 0x7f420419ce40 2024-04-26T02:02:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.634+0000 7f41fbe00640 1 -- 172.21.15.6:0/931109396 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41fc03c920 con 0x7f420419ce40 2024-04-26T02:02:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.634+0000 7f41fbe00640 1 -- 172.21.15.6:0/931109396 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc036040 con 0x7f420419ce40 2024-04-26T02:02:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.635+0000 7f420a200640 1 -- 172.21.15.6:0/931109396 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420419dbc0 con 0x7f420419ce40 2024-04-26T02:02:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.635+0000 7f420a200640 1 -- 172.21.15.6:0/931109396 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f420406bbb0 con 0x7f420419ce40 2024-04-26T02:02:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.643+0000 7f41fbe00640 1 -- 172.21.15.6:0/931109396 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f41fc044020 con 0x7f420419ce40 2024-04-26T02:02:19.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.645+0000 7f420d02f740 1 Processor -- start 2024-04-26T02:02:19.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.645+0000 7f420d02f740 1 -- start start 2024-04-26T02:02:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.647+0000 7f41fbe00640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f41e80a3e60 0x7f41e80a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.648+0000 7f41fbe00640 1 -- 172.21.15.6:0/931109396 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7f41fc03c470 con 0x7f420419ce40 2024-04-26T02:02:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.648+0000 7f420d02f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a4820 0x56019f2a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.648+0000 7f420d02f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a5140 0x56019f2af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.649+0000 7f420d02f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.649+0000 7f4203400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a4820 0x56019f2a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.649+0000 7f4203400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a4820 0x56019f2a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41240/0 (socket says 172.21.15.6:41240) 2024-04-26T02:02:19.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.649+0000 7f4203400640 1 -- 172.21.15.6:0/3514937999 learned_addr learned my addr 172.21.15.6:0/3514937999 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:19.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.649+0000 7f4203e00640 1 --2- 172.21.15.6:0/3514937999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.649+0000 7f420d02f740 1 -- 172.21.15.6:0/3514937999 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56019f20a900 con 0x56019f2a5140 2024-04-26T02:02:19.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.649+0000 7f420d02f740 1 -- 172.21.15.6:0/3514937999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56019f1b0ef0 con 0x56019f2a4820 2024-04-26T02:02:19.649 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.649+0000 7f420d02f740 1 -- 172.21.15.6:0/3514937999 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56019f2068c0 con 0x56019f2afa00 2024-04-26T02:02:19.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.650+0000 7f4203e00640 1 -- 172.21.15.6:0/3514937999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a4820 msgr2=0x56019f2a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:19.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.650+0000 7f4203e00640 1 --2- 172.21.15.6:0/3514937999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a4820 0x56019f2a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.650+0000 7f4202a00640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f41e80a3e60 0x7f41e80a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.650+0000 7f4203e00640 1 -- 172.21.15.6:0/3514937999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a5140 msgr2=0x56019f2af4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:19.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.650+0000 7f4203e00640 1 --2- 172.21.15.6:0/3514937999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a5140 0x56019f2af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.650+0000 7f4203e00640 1 -- 172.21.15.6:0/3514937999 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56019ef1d210 con 0x56019f2afa00 2024-04-26T02:02:19.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.651+0000 7f4203e00640 1 --2- 172.21.15.6:0/3514937999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2b1df0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0f01e0 tx=0x7f41fc03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9480e7d3e3e6e327 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:19.650 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.651+0000 7f4202a00640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f41e80a3e60 0x7f41e80a6320 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x56019f2a5dc0 tx=0x7f41f4006e90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.652+0000 7f41f3e00640 1 -- 172.21.15.6:0/3514937999 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0bf130 con 0x56019f2afa00 2024-04-26T02:02:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.652+0000 7f41f3e00640 1 -- 172.21.15.6:0/3514937999 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41fc0df070 con 0x56019f2afa00 2024-04-26T02:02:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.652+0000 7f41f3e00640 1 -- 172.21.15.6:0/3514937999 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0f0460 con 0x56019f2afa00 2024-04-26T02:02:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.653+0000 7f420d02f740 1 -- 172.21.15.6:0/3514937999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 msgr2=0x56019f2b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.653+0000 7f420d02f740 1 --2- 172.21.15.6:0/3514937999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2b1df0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0f01e0 tx=0x7f41fc03d000 comp rx=0 tx=0).stop 2024-04-26T02:02:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.654+0000 7f420d02f740 1 -- 172.21.15.6:0/3514937999 shutdown_connections 2024-04-26T02:02:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.654+0000 7f420d02f740 1 --2- 172.21.15.6:0/3514937999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a5140 0x56019f2af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.654+0000 7f420d02f740 1 --2- 172.21.15.6:0/3514937999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2b1df0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.654+0000 7f420d02f740 1 --2- 172.21.15.6:0/3514937999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a4820 0x56019f2a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.654+0000 7f420d02f740 1 -- 172.21.15.6:0/3514937999 >> 172.21.15.6:0/3514937999 conn(0x56019f2a0d30 msgr2=0x56019f2a1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:19.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.655+0000 7f420d02f740 1 -- 172.21.15.6:0/3514937999 shutdown_connections 2024-04-26T02:02:19.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.655+0000 7f420d02f740 1 -- 172.21.15.6:0/3514937999 wait complete. 2024-04-26T02:02:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.656+0000 7f420d02f740 1 -- start start 2024-04-26T02:02:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.656+0000 7f420d02f740 1 Processor -- start 2024-04-26T02:02:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.657+0000 7f420d02f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 0x56019f2aca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.658+0000 7f420d02f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.658+0000 7f420d02f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2ad4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.658+0000 7f420d02f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56019f2b23f0 con 0x56019f2a4820 2024-04-26T02:02:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.658+0000 7f420d02f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56019f1c4310 con 0x56019f2a5140 2024-04-26T02:02:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.658+0000 7f420d02f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56019f2b2570 con 0x56019f2afa00 2024-04-26T02:02:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.658+0000 7f420d02f740 10 client.0 _ll_register_callbacks cb 0x56019f192cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.658+0000 7f4202a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.658+0000 7f4203e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2ad4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.658+0000 7f4202a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2acfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41254/0 (socket says 172.21.15.6:41254) 2024-04-26T02:02:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.658+0000 7f4202a00640 1 -- 172.21.15.6:0/2589187964 learned_addr learned my addr 172.21.15.6:0/2589187964 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.659+0000 7f4203e00640 1 -- 172.21.15.6:0/2589187964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 msgr2=0x56019f2acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.659+0000 7f4203e00640 1 --2- 172.21.15.6:0/2589187964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2acfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.659+0000 7f4203e00640 1 -- 172.21.15.6:0/2589187964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 msgr2=0x56019f2aca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.659+0000 7f4203e00640 1 --2- 172.21.15.6:0/2589187964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 0x56019f2aca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.659+0000 7f4203e00640 1 -- 172.21.15.6:0/2589187964 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56019efe9620 con 0x56019f2afa00 2024-04-26T02:02:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.661+0000 7f4203e00640 1 --2- 172.21.15.6:0/2589187964 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2ad4e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0df220 tx=0x7f41fc0bf130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cda2714690ef354c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.661+0000 7f41f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:02:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.663+0000 7f41f0c00640 1 -- 172.21.15.6:0/2589187964 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0c76a0 con 0x56019f2afa00 2024-04-26T02:02:19.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.663+0000 7f41f0c00640 1 -- 172.21.15.6:0/2589187964 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7f41fc03cdb0 con 0x56019f2afa00 2024-04-26T02:02:19.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.663+0000 7f420d02f740 1 -- 172.21.15.6:0/2589187964 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41fc0df070 con 0x56019f2afa00 2024-04-26T02:02:19.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.663+0000 7f41f0c00640 1 -- 172.21.15.6:0/2589187964 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f41fc0c2de0 con 0x56019f2afa00 2024-04-26T02:02:19.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.665+0000 7f420d02f740 10 client.27151 fetch_fsmap learned FSMap version 50 2024-04-26T02:02:19.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.665+0000 7f420d02f740 1 -- 172.21.15.6:0/2589187964 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x56019f2b23f0 con 0x56019f2afa00 2024-04-26T02:02:19.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.665+0000 7f41f0c00640 1 -- 172.21.15.6:0/2589187964 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41fc0df070 con 0x56019f2afa00 2024-04-26T02:02:19.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.665+0000 7f41f0c00640 1 -- 172.21.15.6:0/2589187964 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 50) ==== 8354+0+0 (secure 0 0 0) 0x7f41fc0db070 con 0x56019f2afa00 2024-04-26T02:02:19.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.665+0000 7f420d02f740 10 client.27151 fetch_fsmap finished waiting for FSMap version 50 2024-04-26T02:02:19.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.665+0000 7f420d02f740 10 client.27151 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:02:19.665 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.665+0000 7f420d02f740 20 client.27151 populate_metadata read hostname 'smithi006' 2024-04-26T02:02:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.665+0000 7f420d02f740 1 --2- 172.21.15.6:0/2589187964 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56019f2bf610 0x56019f2c1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.665+0000 7f420d02f740 4 client.27151 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:02:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.665+0000 7f420d02f740 1 -- 172.21.15.6:0/2589187964 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56019f21e8b0 con 0x56019f2bf610 2024-04-26T02:02:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.666+0000 7f4203400640 1 --2- 172.21.15.6:0/2589187964 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56019f2bf610 0x56019f2c1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.667+0000 7f4203400640 1 --2- 172.21.15.6:0/2589187964 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56019f2bf610 0x56019f2c1a00 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e51a6fb340d09fe1 server_cookie=e891c5ded267ce82 in_seq=0 out_seq=0 2024-04-26T02:02:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.667+0000 7f41f0c00640 10 client.27151 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:19.673 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.674+0000 7f41f0c00640 1 -- 172.21.15.6:0/2589187964 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56019f21e8b0 con 0x56019f2bf610 2024-04-26T02:02:19.673 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.674+0000 7f41f0c00640 10 client.27151 handle_command_reply: tid=0 2024-04-26T02:02:19.673 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.674+0000 7f420d02f740 1 client.27151 shutdown 2024-04-26T02:02:19.673 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.674+0000 7f420d02f740 1 -- 172.21.15.6:0/2589187964 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 msgr2=0x56019f2ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:19.673 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.674+0000 7f420d02f740 1 --2- 172.21.15.6:0/2589187964 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2ad4e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0df220 tx=0x7f41fc0bf130 comp rx=0 tx=0).stop 2024-04-26T02:02:19.674 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.675+0000 7f420d02f740 1 -- 172.21.15.6:0/2589187964 shutdown_connections 2024-04-26T02:02:19.674 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.675+0000 7f420d02f740 1 --2- 172.21.15.6:0/2589187964 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56019f2bf610 0x56019f2c1a00 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.674 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.675+0000 7f420d02f740 1 --2- 172.21.15.6:0/2589187964 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2ad4e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.674 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.675+0000 7f420d02f740 1 --2- 172.21.15.6:0/2589187964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2acfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.674 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.675+0000 7f420d02f740 1 --2- 172.21.15.6:0/2589187964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 0x56019f2aca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.674 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.675+0000 7f420d02f740 1 -- 172.21.15.6:0/2589187964 >> 172.21.15.6:0/2589187964 conn(0x56019f2a0d30 msgr2=0x56019f2a6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:19.674 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.675+0000 7f41f0c00640 0 client.27151 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:19.675 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.675+0000 7f420d02f740 1 -- 172.21.15.6:0/2589187964 shutdown_connections 2024-04-26T02:02:19.675 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.675+0000 7f420d02f740 1 -- 172.21.15.6:0/2589187964 wait complete. 2024-04-26T02:02:19.676 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.675+0000 7f420d02f740 20 client.27151 trim_cache size 0 max 16384 2024-04-26T02:02:19.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.700+0000 7f420d02f740 1 Processor -- start 2024-04-26T02:02:19.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.700+0000 7f420d02f740 1 -- start start 2024-04-26T02:02:19.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.700+0000 7f420d02f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 0x56019f2af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.700+0000 7f420d02f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2a0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.700+0000 7f420d02f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2ad300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.700+0000 7f420d02f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f42041b69c0 con 0x56019f2a4820 2024-04-26T02:02:19.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.700+0000 7f420d02f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f42040744c0 con 0x56019f2a5140 2024-04-26T02:02:19.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.700+0000 7f420d02f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4204199910 con 0x56019f2afa00 2024-04-26T02:02:19.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.700+0000 7f4202a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.701+0000 7f4202a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:41270/0 (socket says 172.21.15.6:41270) 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.701+0000 7f4202a00640 1 -- 172.21.15.6:0/918354770 learned_addr learned my addr 172.21.15.6:0/918354770 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.701+0000 7f4203e00640 1 --2- 172.21.15.6:0/918354770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.701+0000 7f4203400640 1 --2- 172.21.15.6:0/918354770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 0x56019f2af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.701+0000 7f4202a00640 1 -- 172.21.15.6:0/918354770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 msgr2=0x56019f2ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.701+0000 7f4202a00640 1 --2- 172.21.15.6:0/918354770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2ad300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.701+0000 7f4202a00640 1 -- 172.21.15.6:0/918354770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 msgr2=0x56019f2af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.701+0000 7f4202a00640 1 --2- 172.21.15.6:0/918354770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 0x56019f2af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.701+0000 7f4202a00640 1 -- 172.21.15.6:0/918354770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56019f29c690 con 0x56019f2a5140 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.701+0000 7f4203400640 1 --2- 172.21.15.6:0/918354770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 0x56019f2af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.701+0000 7f4202a00640 1 --2- 172.21.15.6:0/918354770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2a0610 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009fd0 tx=0x7f41f4011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ac0e25b6f98ab3b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.702+0000 7f41f3400640 1 -- 172.21.15.6:0/918354770 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4022020 con 0x56019f2a5140 2024-04-26T02:02:19.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.702+0000 7f41f3400640 1 -- 172.21.15.6:0/918354770 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41f4016070 con 0x56019f2a5140 2024-04-26T02:02:19.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.702+0000 7f41f3400640 1 -- 172.21.15.6:0/918354770 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4018040 con 0x56019f2a5140 2024-04-26T02:02:19.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.703+0000 7f420d02f740 1 -- 172.21.15.6:0/918354770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 msgr2=0x56019f2a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:19.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.703+0000 7f420d02f740 1 --2- 172.21.15.6:0/918354770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2a0610 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009fd0 tx=0x7f41f4011000 comp rx=0 tx=0).stop 2024-04-26T02:02:19.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.703+0000 7f420d02f740 1 -- 172.21.15.6:0/918354770 shutdown_connections 2024-04-26T02:02:19.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.703+0000 7f420d02f740 1 --2- 172.21.15.6:0/918354770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.703+0000 7f420d02f740 1 --2- 172.21.15.6:0/918354770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2a0610 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.703+0000 7f420d02f740 1 --2- 172.21.15.6:0/918354770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 0x56019f2af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.703+0000 7f420d02f740 1 -- 172.21.15.6:0/918354770 >> 172.21.15.6:0/918354770 conn(0x56019f2a0d30 msgr2=0x56019f2ba210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:19.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.703+0000 7f420d02f740 1 -- 172.21.15.6:0/918354770 shutdown_connections 2024-04-26T02:02:19.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.703+0000 7f420d02f740 1 -- 172.21.15.6:0/918354770 wait complete. 2024-04-26T02:02:19.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.704+0000 7f420d02f740 1 -- start start 2024-04-26T02:02:19.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.704+0000 7f420d02f740 1 Processor -- start 2024-04-26T02:02:19.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f420d02f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 0x56019f2b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f420d02f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f420d02f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f420d02f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4204072980 con 0x56019f2a4820 2024-04-26T02:02:19.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f420d02f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4204073df0 con 0x56019f2a5140 2024-04-26T02:02:19.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f420d02f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4204073050 con 0x56019f2afa00 2024-04-26T02:02:19.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f420d02f740 10 client.0 _ll_register_callbacks cb 0x7f41fc0be800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:19.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f4203e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.704 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f4203e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45360/0 (socket says 172.21.15.6:45360) 2024-04-26T02:02:19.705 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f4203e00640 1 -- 172.21.15.6:0/2546704941 learned_addr learned my addr 172.21.15.6:0/2546704941 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f4203400640 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 0x56019f2b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f4202a00640 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f4203e00640 1 -- 172.21.15.6:0/2546704941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 msgr2=0x56019f2b4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f4203e00640 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2b4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.705+0000 7f4203e00640 1 -- 172.21.15.6:0/2546704941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 msgr2=0x56019f2b7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.706+0000 7f4203e00640 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 0x56019f2b7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.706+0000 7f4203e00640 1 -- 172.21.15.6:0/2546704941 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56019f324a80 con 0x56019f2afa00 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.706+0000 7f4202a00640 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2b4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.706+0000 7f4203e00640 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2b5700 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f41fc03c040 tx=0x7f41fc02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a9195418d76fc85 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.706+0000 7f41f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.706+0000 7f41f0c00640 1 -- 172.21.15.6:0/2546704941 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0bf130 con 0x56019f2afa00 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.706+0000 7f420d02f740 1 -- 172.21.15.6:0/2546704941 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41f4016070 con 0x56019f2afa00 2024-04-26T02:02:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.706+0000 7f41f0c00640 1 -- 172.21.15.6:0/2546704941 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7f41fc0ba780 con 0x56019f2afa00 2024-04-26T02:02:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.706+0000 7f41f0c00640 1 -- 172.21.15.6:0/2546704941 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f41fc0c76a0 con 0x56019f2afa00 2024-04-26T02:02:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.707+0000 7f420d02f740 10 client.27157 fetch_fsmap learned FSMap version 50 2024-04-26T02:02:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.707+0000 7f420d02f740 1 -- 172.21.15.6:0/2546704941 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x56019f324a80 con 0x56019f2afa00 2024-04-26T02:02:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.707+0000 7f41f0c00640 1 -- 172.21.15.6:0/2546704941 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41f4016070 con 0x56019f2afa00 2024-04-26T02:02:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.707+0000 7f41f0c00640 1 -- 172.21.15.6:0/2546704941 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 50) ==== 8354+0+0 (secure 0 0 0) 0x7f41fc0db040 con 0x56019f2afa00 2024-04-26T02:02:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.707+0000 7f420d02f740 10 client.27157 fetch_fsmap finished waiting for FSMap version 50 2024-04-26T02:02:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.707+0000 7f420d02f740 10 client.27157 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:02:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.707+0000 7f420d02f740 20 client.27157 populate_metadata read hostname 'smithi006' 2024-04-26T02:02:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.707+0000 7f420d02f740 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56019f325ed0 0x56019f3282f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.707+0000 7f420d02f740 4 client.27157 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:02:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.707+0000 7f420d02f740 1 -- 172.21.15.6:0/2546704941 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56019f218320 con 0x56019f325ed0 2024-04-26T02:02:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.707+0000 7f4203400640 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56019f325ed0 0x56019f3282f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.708+0000 7f41f0c00640 10 client.27157 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.708+0000 7f4203400640 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56019f325ed0 0x56019f3282f0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ba8bce53771a0c server_cookie=3a3cfd260b2c5e05 in_seq=0 out_seq=0 2024-04-26T02:02:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.933+0000 7f41f0c00640 1 -- 172.21.15.6:0/2546704941 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56019f218320 con 0x56019f325ed0 2024-04-26T02:02:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.933+0000 7f41f0c00640 10 client.27157 handle_command_reply: tid=0 2024-04-26T02:02:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.933+0000 7f420d02f740 1 client.27157 shutdown 2024-04-26T02:02:19.933 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.934+0000 7f420d02f740 1 -- 172.21.15.6:0/2546704941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 msgr2=0x56019f2b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:19.933 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.934+0000 7f420d02f740 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2b5700 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f41fc03c040 tx=0x7f41fc02a6f0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.933 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.934+0000 7f420d02f740 1 -- 172.21.15.6:0/2546704941 shutdown_connections 2024-04-26T02:02:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.934+0000 7f420d02f740 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56019f325ed0 0x56019f3282f0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.934+0000 7f420d02f740 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56019f2afa00 0x56019f2b5700 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.934+0000 7f420d02f740 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56019f2a5140 0x56019f2b4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.934+0000 7f420d02f740 1 --2- 172.21.15.6:0/2546704941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56019f2a4820 0x56019f2b7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.934+0000 7f41f0c00640 0 client.27157 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.934+0000 7f420d02f740 1 -- 172.21.15.6:0/2546704941 >> 172.21.15.6:0/2546704941 conn(0x56019f2a0d30 msgr2=0x56019f2baaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.934+0000 7f420d02f740 1 -- 172.21.15.6:0/2546704941 shutdown_connections 2024-04-26T02:02:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.934+0000 7f420d02f740 1 -- 172.21.15.6:0/2546704941 wait complete. 2024-04-26T02:02:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.934+0000 7f420d02f740 20 client.27157 trim_cache size 0 max 16384 2024-04-26T02:02:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:02:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:02:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:02:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:02:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.935+0000 7f420a200640 1 -- 172.21.15.6:0/931109396 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f41e80a3e60 msgr2=0x7f41e80a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.935+0000 7f420a200640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f41e80a3e60 0x7f41e80a6320 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x56019f2a5dc0 tx=0x7f41f4006e90 comp rx=0 tx=0).stop 2024-04-26T02:02:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.935+0000 7f420a200640 1 -- 172.21.15.6:0/931109396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f420419ce40 msgr2=0x7f420419d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.935+0000 7f420a200640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f420419ce40 0x7f420419d2f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f41fc02ac10 tx=0x7f41fc0316e0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.936+0000 7f420a200640 1 -- 172.21.15.6:0/931109396 shutdown_connections 2024-04-26T02:02:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.936+0000 7f420a200640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f41e80a3e60 0x7f41e80a6320 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.936+0000 7f420a200640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f420419ce40 0x7f420419d2f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.936+0000 7f420a200640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4204107280 0x7f4204198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.936+0000 7f420a200640 1 --2- 172.21.15.6:0/931109396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42041021b0 0x7f4204198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.936+0000 7f420a200640 1 -- 172.21.15.6:0/931109396 >> 172.21.15.6:0/931109396 conn(0x7f42040fda10 msgr2=0x7f42040fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.936+0000 7f420a200640 1 -- 172.21.15.6:0/931109396 shutdown_connections 2024-04-26T02:02:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:19.936+0000 7f420a200640 1 -- 172.21.15.6:0/931109396 wait complete. 2024-04-26T02:02:20.110 INFO:tasks.workunit.client.0.smithi006.stdout: 10 57136 31.16 MB/sec execute 43 sec latency 126.579 ms 2024-04-26T02:02:20.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:20 smithi006 ceph-mon[28419]: pgmap v522: 97 pgs: 97 active+clean; 663 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 798 op/s 2024-04-26T02:02:20.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:20 smithi188 ceph-mon[32751]: pgmap v522: 97 pgs: 97 active+clean; 663 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 798 op/s 2024-04-26T02:02:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:20 smithi029 ceph-mon[32775]: pgmap v522: 97 pgs: 97 active+clean; 663 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 798 op/s 2024-04-26T02:02:21.109 INFO:tasks.workunit.client.0.smithi006.stdout: 10 57771 31.27 MB/sec execute 44 sec latency 111.902 ms 2024-04-26T02:02:22.109 INFO:tasks.workunit.client.0.smithi006.stdout: 10 58387 31.38 MB/sec execute 45 sec latency 182.103 ms 2024-04-26T02:02:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:22 smithi006 ceph-mon[28419]: pgmap v523: 97 pgs: 97 active+clean; 680 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 19 MiB/s wr, 1.14k op/s 2024-04-26T02:02:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:22 smithi188 ceph-mon[32751]: pgmap v523: 97 pgs: 97 active+clean; 680 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 19 MiB/s wr, 1.14k op/s 2024-04-26T02:02:22.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:22 smithi029 ceph-mon[32775]: pgmap v523: 97 pgs: 97 active+clean; 680 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 19 MiB/s wr, 1.14k op/s 2024-04-26T02:02:23.109 INFO:tasks.workunit.client.0.smithi006.stdout: 10 58978 31.41 MB/sec execute 46 sec latency 33.164 ms 2024-04-26T02:02:24.109 INFO:tasks.workunit.client.0.smithi006.stdout: 10 59457 31.09 MB/sec execute 47 sec latency 250.160 ms 2024-04-26T02:02:24.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:24 smithi006 ceph-mon[28419]: pgmap v524: 97 pgs: 97 active+clean; 680 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 656 op/s 2024-04-26T02:02:24.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:24 smithi188 ceph-mon[32751]: pgmap v524: 97 pgs: 97 active+clean; 680 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 656 op/s 2024-04-26T02:02:24.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:24 smithi029 ceph-mon[32775]: pgmap v524: 97 pgs: 97 active+clean; 680 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 656 op/s 2024-04-26T02:02:25.109 INFO:tasks.workunit.client.0.smithi006.stdout: 10 60054 31.28 MB/sec execute 48 sec latency 114.104 ms 2024-04-26T02:02:26.109 INFO:tasks.workunit.client.0.smithi006.stdout: 10 60841 31.45 MB/sec execute 49 sec latency 99.838 ms 2024-04-26T02:02:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:26 smithi188 ceph-mon[32751]: pgmap v525: 97 pgs: 97 active+clean; 680 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 656 op/s 2024-04-26T02:02:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:26 smithi029 ceph-mon[32775]: pgmap v525: 97 pgs: 97 active+clean; 680 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 656 op/s 2024-04-26T02:02:27.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:26 smithi006 ceph-mon[28419]: pgmap v525: 97 pgs: 97 active+clean; 680 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 656 op/s 2024-04-26T02:02:27.109 INFO:tasks.workunit.client.0.smithi006.stdout: 10 61361 31.52 MB/sec execute 50 sec latency 205.493 ms 2024-04-26T02:02:28.109 INFO:tasks.workunit.client.0.smithi006.stdout: 10 61914 31.42 MB/sec execute 51 sec latency 61.419 ms 2024-04-26T02:02:28.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:28 smithi188 ceph-mon[32751]: pgmap v526: 97 pgs: 97 active+clean; 706 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 17 MiB/s wr, 917 op/s 2024-04-26T02:02:28.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:28 smithi029 ceph-mon[32775]: pgmap v526: 97 pgs: 97 active+clean; 706 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 17 MiB/s wr, 917 op/s 2024-04-26T02:02:29.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:28 smithi006 ceph-mon[28419]: pgmap v526: 97 pgs: 97 active+clean; 706 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 17 MiB/s wr, 917 op/s 2024-04-26T02:02:29.109 INFO:tasks.workunit.client.0.smithi006.stdout: 10 62333 31.17 MB/sec execute 52 sec latency 306.658 ms 2024-04-26T02:02:30.110 INFO:tasks.workunit.client.0.smithi006.stdout: 10 63053 31.21 MB/sec execute 53 sec latency 118.902 ms 2024-04-26T02:02:30.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:30 smithi188 ceph-mon[32751]: pgmap v527: 97 pgs: 97 active+clean; 706 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 632 op/s 2024-04-26T02:02:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:30 smithi029 ceph-mon[32775]: pgmap v527: 97 pgs: 97 active+clean; 706 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 632 op/s 2024-04-26T02:02:31.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:30 smithi006 ceph-mon[28419]: pgmap v527: 97 pgs: 97 active+clean; 706 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 632 op/s 2024-04-26T02:02:31.110 INFO:tasks.workunit.client.0.smithi006.stdout: 10 63770 31.40 MB/sec execute 54 sec latency 187.133 ms 2024-04-26T02:02:32.110 INFO:tasks.workunit.client.0.smithi006.stdout: 10 64525 31.71 MB/sec execute 55 sec latency 55.523 ms 2024-04-26T02:02:32.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:32 smithi188 ceph-mon[32751]: pgmap v528: 97 pgs: 97 active+clean; 729 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 16 MiB/s wr, 895 op/s 2024-04-26T02:02:32.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:32 smithi029 ceph-mon[32775]: pgmap v528: 97 pgs: 97 active+clean; 729 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 16 MiB/s wr, 895 op/s 2024-04-26T02:02:33.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:32 smithi006 ceph-mon[28419]: pgmap v528: 97 pgs: 97 active+clean; 729 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 16 MiB/s wr, 895 op/s 2024-04-26T02:02:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.097+0000 7f70af000640 1 -- 172.21.15.6:0/1835605818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70a810c360 msgr2=0x7f70a810e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.097+0000 7f70af000640 1 --2- 172.21.15.6:0/1835605818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70a810c360 0x7f70a810e7e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009ca0 tx=0x7f70a002a940 comp rx=0 tx=0).stop 2024-04-26T02:02:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.097+0000 7f70af000640 1 -- 172.21.15.6:0/1835605818 shutdown_connections 2024-04-26T02:02:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.097+0000 7f70af000640 1 --2- 172.21.15.6:0/1835605818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70a810c360 0x7f70a810e7e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.097+0000 7f70af000640 1 --2- 172.21.15.6:0/1835605818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a81038b0 0x7f70a810bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.097+0000 7f70af000640 1 --2- 172.21.15.6:0/1835605818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a8102f90 0x7f70a8103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.097+0000 7f70af000640 1 -- 172.21.15.6:0/1835605818 >> 172.21.15.6:0/1835605818 conn(0x7f70a80fc730 msgr2=0x7f70a80feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.097+0000 7f70af000640 1 -- 172.21.15.6:0/1835605818 shutdown_connections 2024-04-26T02:02:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.097+0000 7f70af000640 1 -- 172.21.15.6:0/1835605818 wait complete. 2024-04-26T02:02:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.098+0000 7f70af000640 1 Processor -- start 2024-04-26T02:02:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.099+0000 7f70af000640 1 -- start start 2024-04-26T02:02:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.099+0000 7f70af000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a8102f90 0x7f70a8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.099+0000 7f70af000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a81038b0 0x7f70a8198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.099+0000 7f70af000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70a810c360 0x7f70a819cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.099+0000 7f70af000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f70a8112000 con 0x7f70a8102f90 2024-04-26T02:02:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.099+0000 7f70af000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70a8111e80 con 0x7f70a81038b0 2024-04-26T02:02:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.099+0000 7f70af000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f70a8112180 con 0x7f70a810c360 2024-04-26T02:02:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.099+0000 7f70ad200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a81038b0 0x7f70a8198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.100+0000 7f70ad200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a81038b0 0x7f70a8198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36790/0 (socket says 172.21.15.6:36790) 2024-04-26T02:02:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.100+0000 7f70ad200640 1 -- 172.21.15.6:0/3085851683 learned_addr learned my addr 172.21.15.6:0/3085851683 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.100+0000 7f70adc00640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a8102f90 0x7f70a8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.100+0000 7f70ae600640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70a810c360 0x7f70a819cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.100+0000 7f70ad200640 1 -- 172.21.15.6:0/3085851683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70a810c360 msgr2=0x7f70a819cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.100+0000 7f70ad200640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70a810c360 0x7f70a819cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.100+0000 7f70ad200640 1 -- 172.21.15.6:0/3085851683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a8102f90 msgr2=0x7f70a8198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.100+0000 7f70ad200640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a8102f90 0x7f70a8198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.100+0000 7f70ad200640 1 -- 172.21.15.6:0/3085851683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a819d4c0 con 0x7f70a81038b0 2024-04-26T02:02:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.100+0000 7f70ad200640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a81038b0 0x7f70a8198a60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7094009de0 tx=0x7f709400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.101+0000 7f70a6000640 1 -- 172.21.15.6:0/3085851683 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709401f070 con 0x7f70a81038b0 2024-04-26T02:02:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.101+0000 7f70a6000640 1 -- 172.21.15.6:0/3085851683 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7094011750 con 0x7f70a81038b0 2024-04-26T02:02:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.101+0000 7f70af000640 1 -- 172.21.15.6:0/3085851683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a819d7b0 con 0x7f70a81038b0 2024-04-26T02:02:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.101+0000 7f70a6000640 1 -- 172.21.15.6:0/3085851683 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709401a740 con 0x7f70a81038b0 2024-04-26T02:02:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.101+0000 7f70af000640 1 -- 172.21.15.6:0/3085851683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a8100b10 con 0x7f70a81038b0 2024-04-26T02:02:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.105+0000 7f70a6000640 1 -- 172.21.15.6:0/3085851683 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f7094022070 con 0x7f70a81038b0 2024-04-26T02:02:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.105+0000 7f70b3431740 1 Processor -- start 2024-04-26T02:02:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.105+0000 7f70b3431740 1 -- start start 2024-04-26T02:02:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.106+0000 7f70b3431740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a0931c0 0x55df1a0935a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.106+0000 7f70b3431740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a093ae0 0x55df1a09de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.106+0000 7f70b3431740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a09e3a0 0x55df1a0a0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.106+0000 7f70b3431740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55df19ff0900 con 0x55df1a093ae0 2024-04-26T02:02:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.106+0000 7f70b3431740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55df19f96ef0 con 0x55df1a09e3a0 2024-04-26T02:02:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.106+0000 7f70b3431740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55df19fec8c0 con 0x55df1a0931c0 2024-04-26T02:02:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.106+0000 7f70ae600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a0931c0 0x55df1a0935a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.106+0000 7f70ae600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a0931c0 0x55df1a0935a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:60502/0 (socket says 172.21.15.6:60502) 2024-04-26T02:02:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.106+0000 7f70ae600640 1 -- 172.21.15.6:0/384374952 learned_addr learned my addr 172.21.15.6:0/384374952 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.106+0000 7f70ad200640 1 --2- 172.21.15.6:0/384374952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a09e3a0 0x55df1a0a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.106+0000 7f70adc00640 1 --2- 172.21.15.6:0/384374952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a093ae0 0x55df1a09de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.107+0000 7f70ae600640 1 -- 172.21.15.6:0/384374952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a09e3a0 msgr2=0x55df1a0a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.107+0000 7f70ae600640 1 --2- 172.21.15.6:0/384374952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a09e3a0 0x55df1a0a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.107+0000 7f70ae600640 1 -- 172.21.15.6:0/384374952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a093ae0 msgr2=0x55df1a09de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.107+0000 7f70ae600640 1 --2- 172.21.15.6:0/384374952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a093ae0 0x55df1a09de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.107+0000 7f70ae600640 1 -- 172.21.15.6:0/384374952 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df19d03210 con 0x55df1a0931c0 2024-04-26T02:02:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.107+0000 7f70a6000640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f70900a3d80 0x7f70900a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.107+0000 7f70a6000640 1 -- 172.21.15.6:0/3085851683 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7f70940d4500 con 0x7f70a81038b0 2024-04-26T02:02:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.108+0000 7f70ae600640 1 --2- 172.21.15.6:0/384374952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a0931c0 0x55df1a0935a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009ca0 tx=0x7f70a0005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21063a14bb85ec99 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.108+0000 7f70adc00640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f70900a3d80 0x7f70900a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.108+0000 7f709aa00640 1 -- 172.21.15.6:0/384374952 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0034040 con 0x55df1a0931c0 2024-04-26T02:02:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.108+0000 7f709aa00640 1 -- 172.21.15.6:0/384374952 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70a0003e20 con 0x55df1a0931c0 2024-04-26T02:02:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.108+0000 7f709aa00640 1 -- 172.21.15.6:0/384374952 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a002fe30 con 0x55df1a0931c0 2024-04-26T02:02:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.109+0000 7f70b3431740 1 -- 172.21.15.6:0/384374952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a0931c0 msgr2=0x55df1a0935a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.109+0000 7f70b3431740 1 --2- 172.21.15.6:0/384374952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a0931c0 0x55df1a0935a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009ca0 tx=0x7f70a0005f80 comp rx=0 tx=0).stop 2024-04-26T02:02:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.109+0000 7f70adc00640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f70900a3d80 0x7f70900a6240 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x55df1a094760 tx=0x7f709c005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.109+0000 7f70b3431740 1 -- 172.21.15.6:0/384374952 shutdown_connections 2024-04-26T02:02:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.109+0000 7f70b3431740 1 --2- 172.21.15.6:0/384374952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a093ae0 0x55df1a09de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.109+0000 7f70b3431740 1 --2- 172.21.15.6:0/384374952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a09e3a0 0x55df1a0a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.109+0000 7f70b3431740 1 --2- 172.21.15.6:0/384374952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a0931c0 0x55df1a0935a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.109+0000 7f70b3431740 1 -- 172.21.15.6:0/384374952 >> 172.21.15.6:0/384374952 conn(0x55df1a08f6d0 msgr2=0x55df1a08fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.110+0000 7f70b3431740 1 -- 172.21.15.6:0/384374952 shutdown_connections 2024-04-26T02:02:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.110+0000 7f70b3431740 1 -- 172.21.15.6:0/384374952 wait complete. 2024-04-26T02:02:33.110 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.110+0000 7f70b3431740 1 -- start start 2024-04-26T02:02:33.110 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.110+0000 7f70b3431740 1 Processor -- start 2024-04-26T02:02:33.110 INFO:tasks.workunit.client.0.smithi006.stdout: 10 64975 31.66 MB/sec execute 56 sec latency 151.921 ms 2024-04-26T02:02:33.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.111+0000 7f70b3431740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 0x55df1a09b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.111+0000 7f70b3431740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a093ae0 0x55df1a09b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.111+0000 7f70b3431740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a09e3a0 0x55df1a09bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.112 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.111+0000 7f70b3431740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55df1a0a0f70 con 0x55df1a093ae0 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.111+0000 7f70b3431740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55df19faa310 con 0x55df1a0931c0 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.111+0000 7f70b3431740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55df1a0a10f0 con 0x55df1a09e3a0 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.111+0000 7f70b3431740 10 client.0 _ll_register_callbacks cb 0x55df19f78cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.111+0000 7f70adc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 0x55df1a09b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.111+0000 7f70adc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 0x55df1a09b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36810/0 (socket says 172.21.15.6:36810) 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.111+0000 7f70ad200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a093ae0 0x55df1a09b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.111+0000 7f70adc00640 1 -- 172.21.15.6:0/4066490532 learned_addr learned my addr 172.21.15.6:0/4066490532 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.111+0000 7f70ae600640 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a09e3a0 0x55df1a09bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.112+0000 7f70adc00640 1 -- 172.21.15.6:0/4066490532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a09e3a0 msgr2=0x55df1a09bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.112+0000 7f70adc00640 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a09e3a0 0x55df1a09bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.112+0000 7f70adc00640 1 -- 172.21.15.6:0/4066490532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a093ae0 msgr2=0x55df1a09b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.112+0000 7f70adc00640 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a093ae0 0x55df1a09b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.112+0000 7f70adc00640 1 -- 172.21.15.6:0/4066490532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df19dcf620 con 0x55df1a0931c0 2024-04-26T02:02:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.112+0000 7f70ad200640 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a093ae0 0x55df1a09b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:02:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.112+0000 7f70ae600640 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a09e3a0 0x55df1a09bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:02:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.113+0000 7f70adc00640 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 0x55df1a09b340 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f709c009730 tx=0x7f709c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcc97331e831556a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.113+0000 7f708f400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:02:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.113+0000 7f708f400640 1 -- 172.21.15.6:0/4066490532 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c01f640 con 0x55df1a0931c0 2024-04-26T02:02:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.113+0000 7f708f400640 1 -- 172.21.15.6:0/4066490532 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7f709c01f7e0 con 0x55df1a0931c0 2024-04-26T02:02:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.113+0000 7f70b3431740 1 -- 172.21.15.6:0/4066490532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70a0003e20 con 0x55df1a0931c0 2024-04-26T02:02:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.113+0000 7f708f400640 1 -- 172.21.15.6:0/4066490532 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f709c01f9e0 con 0x55df1a0931c0 2024-04-26T02:02:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.113+0000 7f70b3431740 10 client.27113 fetch_fsmap learned FSMap version 50 2024-04-26T02:02:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.113+0000 7f70b3431740 1 -- 172.21.15.6:0/4066490532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55df1a0ad250 con 0x55df1a0931c0 2024-04-26T02:02:33.115 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.114+0000 7f708f400640 1 -- 172.21.15.6:0/4066490532 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70a0003e20 con 0x55df1a0931c0 2024-04-26T02:02:33.115 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.114+0000 7f708f400640 1 -- 172.21.15.6:0/4066490532 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 50) ==== 8354+0+0 (secure 0 0 0) 0x7f709c027070 con 0x55df1a0931c0 2024-04-26T02:02:33.115 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.114+0000 7f70b3431740 10 client.27113 fetch_fsmap finished waiting for FSMap version 50 2024-04-26T02:02:33.115 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.114+0000 7f70b3431740 10 client.27113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:02:33.115 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.114+0000 7f70b3431740 20 client.27113 populate_metadata read hostname 'smithi006' 2024-04-26T02:02:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.114+0000 7f70b3431740 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55df1a0ae7d0 0x55df1a0b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.114+0000 7f70b3431740 4 client.27113 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:02:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.114+0000 7f70b3431740 1 -- 172.21.15.6:0/4066490532 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df1a0048b0 con 0x55df1a0ae7d0 2024-04-26T02:02:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.115+0000 7f70ad200640 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55df1a0ae7d0 0x55df1a0b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.115+0000 7f70ad200640 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55df1a0ae7d0 0x55df1a0b0c40 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85048fc648e63538 server_cookie=5b6d741f701113fc in_seq=0 out_seq=0 2024-04-26T02:02:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.115+0000 7f708f400640 10 client.27113 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:33.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.118+0000 7f708f400640 1 -- 172.21.15.6:0/4066490532 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55df1a0048b0 con 0x55df1a0ae7d0 2024-04-26T02:02:33.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.118+0000 7f708f400640 10 client.27113 handle_command_reply: tid=0 2024-04-26T02:02:33.117 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.118+0000 7f70b3431740 1 client.27113 shutdown 2024-04-26T02:02:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.118+0000 7f70b3431740 1 -- 172.21.15.6:0/4066490532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 msgr2=0x55df1a09b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.118+0000 7f70b3431740 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 0x55df1a09b340 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f709c009730 tx=0x7f709c004730 comp rx=0 tx=0).stop 2024-04-26T02:02:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.119+0000 7f70b3431740 1 -- 172.21.15.6:0/4066490532 shutdown_connections 2024-04-26T02:02:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.119+0000 7f70b3431740 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55df1a0ae7d0 0x55df1a0b0c40 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.119+0000 7f70b3431740 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a093ae0 0x55df1a09b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.119+0000 7f70b3431740 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a09e3a0 0x55df1a09bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.119+0000 7f70b3431740 1 --2- 172.21.15.6:0/4066490532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 0x55df1a09b340 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.119+0000 7f70b3431740 1 -- 172.21.15.6:0/4066490532 >> 172.21.15.6:0/4066490532 conn(0x55df1a08f6d0 msgr2=0x55df1a095450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.119+0000 7f708f400640 0 client.27113 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.119+0000 7f70b3431740 1 -- 172.21.15.6:0/4066490532 shutdown_connections 2024-04-26T02:02:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.119+0000 7f70b3431740 1 -- 172.21.15.6:0/4066490532 wait complete. 2024-04-26T02:02:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.120+0000 7f70b3431740 20 client.27113 trim_cache size 0 max 16384 2024-04-26T02:02:33.139 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.140+0000 7f70b3431740 1 Processor -- start 2024-04-26T02:02:33.139 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.140+0000 7f70b3431740 1 -- start start 2024-04-26T02:02:33.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70b3431740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a0931c0 0x55df1a0a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70b3431740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a093ae0 0x55df1a0b6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70b3431740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a09e3a0 0x55df1a0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70b3431740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f70a81ae170 con 0x55df1a0931c0 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70b3431740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70a81ab9e0 con 0x55df1a093ae0 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70b3431740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f70a81a9850 con 0x55df1a09e3a0 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70ad200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a093ae0 0x55df1a0b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70adc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a0931c0 0x55df1a0a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70ad200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a093ae0 0x55df1a0b6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36826/0 (socket says 172.21.15.6:36826) 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70ad200640 1 -- 172.21.15.6:0/1170865905 learned_addr learned my addr 172.21.15.6:0/1170865905 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70ae600640 1 --2- 172.21.15.6:0/1170865905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a09e3a0 0x55df1a0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70ad200640 1 -- 172.21.15.6:0/1170865905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a09e3a0 msgr2=0x55df1a0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70ad200640 1 --2- 172.21.15.6:0/1170865905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a09e3a0 0x55df1a0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70ad200640 1 -- 172.21.15.6:0/1170865905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a0931c0 msgr2=0x55df1a0a9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:33.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70ad200640 1 --2- 172.21.15.6:0/1170865905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a0931c0 0x55df1a0a9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70ad200640 1 -- 172.21.15.6:0/1170865905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df1a0a0b00 con 0x55df1a093ae0 2024-04-26T02:02:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70ae600640 1 --2- 172.21.15.6:0/1170865905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a09e3a0 0x55df1a0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:02:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.141+0000 7f70adc00640 1 --2- 172.21.15.6:0/1170865905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a0931c0 0x55df1a0a9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:02:33.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.142+0000 7f70ad200640 1 --2- 172.21.15.6:0/1170865905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a093ae0 0x55df1a0b6550 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f709409ce20 tx=0x7f70940d8040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93db0b6dd42cfb01 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.142+0000 7f709a000640 1 -- 172.21.15.6:0/1170865905 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709409fe00 con 0x55df1a093ae0 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.142+0000 7f709a000640 1 -- 172.21.15.6:0/1170865905 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70940a04a0 con 0x55df1a093ae0 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.142+0000 7f709a000640 1 -- 172.21.15.6:0/1170865905 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709409e830 con 0x55df1a093ae0 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.143+0000 7f70b3431740 1 -- 172.21.15.6:0/1170865905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a093ae0 msgr2=0x55df1a0b6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.143+0000 7f70b3431740 1 --2- 172.21.15.6:0/1170865905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a093ae0 0x55df1a0b6550 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f709409ce20 tx=0x7f70940d8040 comp rx=0 tx=0).stop 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.143+0000 7f70b3431740 1 -- 172.21.15.6:0/1170865905 shutdown_connections 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.143+0000 7f70b3431740 1 --2- 172.21.15.6:0/1170865905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a09e3a0 0x55df1a0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.143+0000 7f70b3431740 1 --2- 172.21.15.6:0/1170865905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a093ae0 0x55df1a0b6550 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.143+0000 7f70b3431740 1 --2- 172.21.15.6:0/1170865905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a0931c0 0x55df1a0a9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.143+0000 7f70b3431740 1 -- 172.21.15.6:0/1170865905 >> 172.21.15.6:0/1170865905 conn(0x55df1a08f6d0 msgr2=0x55df1a099c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.143+0000 7f70b3431740 1 -- 172.21.15.6:0/1170865905 shutdown_connections 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.143+0000 7f70b3431740 1 -- 172.21.15.6:0/1170865905 wait complete. 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.144+0000 7f70b3431740 1 -- start start 2024-04-26T02:02:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.144+0000 7f70b3431740 1 Processor -- start 2024-04-26T02:02:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.144+0000 7f70b3431740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 0x55df1a0a3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.144+0000 7f70b3431740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a093ae0 0x55df1a0a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.144+0000 7f70b3431740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a09e3a0 0x55df1a112dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.144+0000 7f70b3431740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f70a81a9ea0 con 0x55df1a09e3a0 2024-04-26T02:02:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.144+0000 7f70b3431740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70a81ab310 con 0x55df1a0931c0 2024-04-26T02:02:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.144+0000 7f70b3431740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f70a81aa570 con 0x55df1a093ae0 2024-04-26T02:02:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.145+0000 7f70b3431740 10 client.0 _ll_register_callbacks cb 0x7f709c0044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.145+0000 7f70adc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 0x55df1a0a3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.145+0000 7f70ad200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a093ae0 0x55df1a0a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.145+0000 7f70adc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 0x55df1a0a3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36842/0 (socket says 172.21.15.6:36842) 2024-04-26T02:02:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.145+0000 7f70adc00640 1 -- 172.21.15.6:0/1276155616 learned_addr learned my addr 172.21.15.6:0/1276155616 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.145+0000 7f70ae600640 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a09e3a0 0x55df1a112dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.145+0000 7f70adc00640 1 -- 172.21.15.6:0/1276155616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a093ae0 msgr2=0x55df1a0a3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.145+0000 7f70adc00640 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a093ae0 0x55df1a0a3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.145+0000 7f70adc00640 1 -- 172.21.15.6:0/1276155616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a09e3a0 msgr2=0x55df1a112dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.145+0000 7f70adc00640 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a09e3a0 0x55df1a112dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.145+0000 7f70adc00640 1 -- 172.21.15.6:0/1276155616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df1a113300 con 0x55df1a0931c0 2024-04-26T02:02:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.145+0000 7f70ae600640 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a09e3a0 0x55df1a112dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.146+0000 7f70adc00640 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 0x55df1a0a3500 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f709c012cb0 tx=0x7f709c029650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1c6ef09afd5b8ca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.146+0000 7f708f400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.146+0000 7f708f400640 1 -- 172.21.15.6:0/1276155616 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c003e50 con 0x55df1a0931c0 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.146+0000 7f708f400640 1 -- 172.21.15.6:0/1276155616 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7f709c029840 con 0x55df1a0931c0 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.146+0000 7f70b3431740 1 -- 172.21.15.6:0/1276155616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70940a04a0 con 0x55df1a0931c0 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.146+0000 7f708f400640 1 -- 172.21.15.6:0/1276155616 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f709c029a40 con 0x55df1a0931c0 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.146+0000 7f70b3431740 10 client.27119 fetch_fsmap learned FSMap version 50 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.146+0000 7f70b3431740 1 -- 172.21.15.6:0/1276155616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55df1a114580 con 0x55df1a0931c0 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.147+0000 7f708f400640 1 -- 172.21.15.6:0/1276155616 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70940a04a0 con 0x55df1a0931c0 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.147+0000 7f708f400640 1 -- 172.21.15.6:0/1276155616 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 50) ==== 8354+0+0 (secure 0 0 0) 0x7f709c038030 con 0x55df1a0931c0 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.147+0000 7f70b3431740 10 client.27119 fetch_fsmap finished waiting for FSMap version 50 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.147+0000 7f70b3431740 10 client.27119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.147+0000 7f70b3431740 20 client.27119 populate_metadata read hostname 'smithi006' 2024-04-26T02:02:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.147+0000 7f70b3431740 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55df1a114cd0 0x55df1a1170f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:33.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.147+0000 7f70b3431740 4 client.27119 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:02:33.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.147+0000 7f70b3431740 1 -- 172.21.15.6:0/1276155616 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55df19ffe320 con 0x55df1a114cd0 2024-04-26T02:02:33.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.148+0000 7f70ad200640 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55df1a114cd0 0x55df1a1170f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:33.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.148+0000 7f70ad200640 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55df1a114cd0 0x55df1a1170f0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59e86d26ed0be518 server_cookie=62d5bb6e70b3256b in_seq=0 out_seq=0 2024-04-26T02:02:33.149 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.149+0000 7f708f400640 10 client.27119 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:33.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.402+0000 7f708f400640 1 -- 172.21.15.6:0/1276155616 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55df19ffe320 con 0x55df1a114cd0 2024-04-26T02:02:33.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.402+0000 7f708f400640 10 client.27119 handle_command_reply: tid=0 2024-04-26T02:02:33.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.402+0000 7f70b3431740 1 client.27119 shutdown 2024-04-26T02:02:33.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.403+0000 7f70b3431740 1 -- 172.21.15.6:0/1276155616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 msgr2=0x55df1a0a3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:33.402 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.403+0000 7f70b3431740 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 0x55df1a0a3500 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f709c012cb0 tx=0x7f709c029650 comp rx=0 tx=0).stop 2024-04-26T02:02:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.403+0000 7f70b3431740 1 -- 172.21.15.6:0/1276155616 shutdown_connections 2024-04-26T02:02:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.403+0000 7f70b3431740 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55df1a114cd0 0x55df1a1170f0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.403+0000 7f70b3431740 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55df1a09e3a0 0x55df1a112dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.403+0000 7f70b3431740 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df1a093ae0 0x55df1a0a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.403+0000 7f70b3431740 1 --2- 172.21.15.6:0/1276155616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55df1a0931c0 0x55df1a0a3500 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.403+0000 7f70b3431740 1 -- 172.21.15.6:0/1276155616 >> 172.21.15.6:0/1276155616 conn(0x55df1a08f6d0 msgr2=0x55df1a09a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.403+0000 7f708f400640 0 client.27119 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.403+0000 7f70b3431740 1 -- 172.21.15.6:0/1276155616 shutdown_connections 2024-04-26T02:02:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.403+0000 7f70b3431740 1 -- 172.21.15.6:0/1276155616 wait complete. 2024-04-26T02:02:33.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.404+0000 7f70b3431740 20 client.27119 trim_cache size 0 max 16384 2024-04-26T02:02:33.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:02:33.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:02:33.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:02:33.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:02:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.405+0000 7f70af000640 1 -- 172.21.15.6:0/3085851683 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f70900a3d80 msgr2=0x7f70900a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.405+0000 7f70af000640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f70900a3d80 0x7f70900a6240 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x55df1a094760 tx=0x7f709c005f90 comp rx=0 tx=0).stop 2024-04-26T02:02:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.405+0000 7f70af000640 1 -- 172.21.15.6:0/3085851683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a81038b0 msgr2=0x7f70a8198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.405+0000 7f70af000640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a81038b0 0x7f70a8198a60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7094009de0 tx=0x7f709400c6a0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.405+0000 7f70af000640 1 -- 172.21.15.6:0/3085851683 shutdown_connections 2024-04-26T02:02:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.405+0000 7f70af000640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f70900a3d80 0x7f70900a6240 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.405+0000 7f70af000640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70a810c360 0x7f70a819cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.405+0000 7f70af000640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a81038b0 0x7f70a8198a60 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.405+0000 7f70af000640 1 --2- 172.21.15.6:0/3085851683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a8102f90 0x7f70a8198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.405+0000 7f70af000640 1 -- 172.21.15.6:0/3085851683 >> 172.21.15.6:0/3085851683 conn(0x7f70a80fc730 msgr2=0x7f70a80fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.407+0000 7f70af000640 1 -- 172.21.15.6:0/3085851683 shutdown_connections 2024-04-26T02:02:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:33.407+0000 7f70af000640 1 -- 172.21.15.6:0/3085851683 wait complete. 2024-04-26T02:02:33.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:33 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:02:33.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:33 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:02:34.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:33 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:02:34.110 INFO:tasks.workunit.client.0.smithi006.stdout: 10 65304 31.38 MB/sec execute 57 sec latency 161.772 ms 2024-04-26T02:02:34.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:34 smithi188 ceph-mon[32751]: pgmap v529: 97 pgs: 97 active+clean; 729 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 10 MiB/s wr, 524 op/s 2024-04-26T02:02:34.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:34 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:02:34.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:34 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:02:34.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:34 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:02:34.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:34 smithi029 ceph-mon[32775]: pgmap v529: 97 pgs: 97 active+clean; 729 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 10 MiB/s wr, 524 op/s 2024-04-26T02:02:34.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:34 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:02:34.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:34 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:02:34.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:34 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:02:35.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:34 smithi006 ceph-mon[28419]: pgmap v529: 97 pgs: 97 active+clean; 729 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 10 MiB/s wr, 524 op/s 2024-04-26T02:02:35.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:34 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:02:35.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:34 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:02:35.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:34 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:02:35.110 INFO:tasks.workunit.client.0.smithi006.stdout: 10 65833 31.14 MB/sec execute 58 sec latency 123.055 ms 2024-04-26T02:02:36.110 INFO:tasks.workunit.client.0.smithi006.stdout: 10 66663 31.34 MB/sec execute 59 sec latency 65.887 ms 2024-04-26T02:02:36.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:36 smithi188 ceph-mon[32751]: pgmap v530: 97 pgs: 97 active+clean; 729 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 10 MiB/s wr, 524 op/s 2024-04-26T02:02:36.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:36 smithi029 ceph-mon[32775]: pgmap v530: 97 pgs: 97 active+clean; 729 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 10 MiB/s wr, 524 op/s 2024-04-26T02:02:37.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:36 smithi006 ceph-mon[28419]: pgmap v530: 97 pgs: 97 active+clean; 729 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 10 MiB/s wr, 524 op/s 2024-04-26T02:02:37.110 INFO:tasks.workunit.client.0.smithi006.stdout: 10 67520 31.68 MB/sec execute 60 sec latency 42.370 ms 2024-04-26T02:02:38.110 INFO:tasks.workunit.client.0.smithi006.stdout: 10 68127 31.75 MB/sec execute 61 sec latency 113.519 ms 2024-04-26T02:02:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:38 smithi188 ceph-mon[32751]: pgmap v531: 97 pgs: 97 active+clean; 737 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 826 op/s 2024-04-26T02:02:38.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:38 smithi029 ceph-mon[32775]: pgmap v531: 97 pgs: 97 active+clean; 737 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 826 op/s 2024-04-26T02:02:39.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:38 smithi006 ceph-mon[28419]: pgmap v531: 97 pgs: 97 active+clean; 737 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 826 op/s 2024-04-26T02:02:39.110 INFO:tasks.workunit.client.0.smithi006.stdout: 10 68614 31.72 MB/sec execute 62 sec latency 137.298 ms 2024-04-26T02:02:40.111 INFO:tasks.workunit.client.0.smithi006.stdout: 10 69213 31.60 MB/sec execute 63 sec latency 76.891 ms 2024-04-26T02:02:40.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:40 smithi188 ceph-mon[32751]: pgmap v532: 97 pgs: 97 active+clean; 737 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.2 MiB/s wr, 564 op/s 2024-04-26T02:02:40.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:40 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:02:40.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:40 smithi029 ceph-mon[32775]: pgmap v532: 97 pgs: 97 active+clean; 737 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.2 MiB/s wr, 564 op/s 2024-04-26T02:02:40.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:40 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:02:41.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:40 smithi006 ceph-mon[28419]: pgmap v532: 97 pgs: 97 active+clean; 737 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.2 MiB/s wr, 564 op/s 2024-04-26T02:02:41.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:40 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:02:41.111 INFO:tasks.workunit.client.0.smithi006.stdout: 10 69905 31.68 MB/sec execute 64 sec latency 52.209 ms 2024-04-26T02:02:42.111 INFO:tasks.workunit.client.0.smithi006.stdout: 10 70631 31.84 MB/sec execute 65 sec latency 62.590 ms 2024-04-26T02:02:42.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:42 smithi188 ceph-mon[32751]: pgmap v533: 97 pgs: 97 active+clean; 763 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 802 op/s 2024-04-26T02:02:42.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:42 smithi029 ceph-mon[32775]: pgmap v533: 97 pgs: 97 active+clean; 763 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 802 op/s 2024-04-26T02:02:43.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:42 smithi006 ceph-mon[28419]: pgmap v533: 97 pgs: 97 active+clean; 763 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 802 op/s 2024-04-26T02:02:43.111 INFO:tasks.workunit.client.0.smithi006.stdout: 10 71176 31.76 MB/sec execute 66 sec latency 201.155 ms 2024-04-26T02:02:44.111 INFO:tasks.workunit.client.0.smithi006.stdout: 10 71774 31.81 MB/sec execute 67 sec latency 234.474 ms 2024-04-26T02:02:44.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:44 smithi188 ceph-mon[32751]: pgmap v534: 97 pgs: 97 active+clean; 763 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.4 MiB/s wr, 539 op/s 2024-04-26T02:02:44.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:44 smithi029 ceph-mon[32775]: pgmap v534: 97 pgs: 97 active+clean; 763 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.4 MiB/s wr, 539 op/s 2024-04-26T02:02:45.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:44 smithi006 ceph-mon[28419]: pgmap v534: 97 pgs: 97 active+clean; 763 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.4 MiB/s wr, 539 op/s 2024-04-26T02:02:45.111 INFO:tasks.workunit.client.0.smithi006.stdout: 10 72489 31.86 MB/sec execute 68 sec latency 139.980 ms 2024-04-26T02:02:45.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:45 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:02:45.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:45 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:02:46.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:45 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:02:46.111 INFO:tasks.workunit.client.0.smithi006.stdout: 10 73155 32.02 MB/sec execute 69 sec latency 47.738 ms 2024-04-26T02:02:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.601+0000 7f6de0600640 1 -- 172.21.15.6:0/3489164556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd8102b80 msgr2=0x7f6dd8106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.601+0000 7f6de0600640 1 --2- 172.21.15.6:0/3489164556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd8102b80 0x7f6dd8106bc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4009900 tx=0x7f6dc402a3f0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.601+0000 7f6de0600640 1 -- 172.21.15.6:0/3489164556 shutdown_connections 2024-04-26T02:02:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.601+0000 7f6de0600640 1 --2- 172.21.15.6:0/3489164556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dd81072a0 0x7f6dd810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.601+0000 7f6de0600640 1 --2- 172.21.15.6:0/3489164556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd8102b80 0x7f6dd8106bc0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.601+0000 7f6de0600640 1 --2- 172.21.15.6:0/3489164556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6dd81021d0 0x7f6dd81025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.601+0000 7f6de0600640 1 -- 172.21.15.6:0/3489164556 >> 172.21.15.6:0/3489164556 conn(0x7f6dd80fda30 msgr2=0x7f6dd80ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.602+0000 7f6de0600640 1 -- 172.21.15.6:0/3489164556 shutdown_connections 2024-04-26T02:02:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.602+0000 7f6de0600640 1 -- 172.21.15.6:0/3489164556 wait complete. 2024-04-26T02:02:46.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.603+0000 7f6de0600640 1 Processor -- start 2024-04-26T02:02:46.602 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.603+0000 7f6de0600640 1 -- start start 2024-04-26T02:02:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.604+0000 7f6de0600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6dd81021d0 0x7f6dd81a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.604+0000 7f6de0600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dd81072a0 0x7f6dd81a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.604+0000 7f6de0600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd819a770 0x7f6dd819abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.604+0000 7f6de0600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd8112230 con 0x7f6dd81021d0 2024-04-26T02:02:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.604+0000 7f6de0600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81120b0 con 0x7f6dd819a770 2024-04-26T02:02:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.604+0000 7f6de0600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81123b0 con 0x7f6dd81072a0 2024-04-26T02:02:46.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.604+0000 7f6dddc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6dd81021d0 0x7f6dd81a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.604+0000 7f6dddc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6dd81021d0 0x7f6dd81a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51930/0 (socket says 172.21.15.6:51930) 2024-04-26T02:02:46.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.604+0000 7f6dddc00640 1 -- 172.21.15.6:0/464310695 learned_addr learned my addr 172.21.15.6:0/464310695 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:46.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.605+0000 7f6dde600640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd819a770 0x7f6dd819abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.605+0000 7f6ddd200640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dd81072a0 0x7f6dd81a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.605+0000 7f6dde600640 1 -- 172.21.15.6:0/464310695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dd81072a0 msgr2=0x7f6dd81a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:46.604 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.605+0000 7f6dde600640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dd81072a0 0x7f6dd81a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.605+0000 7f6dde600640 1 -- 172.21.15.6:0/464310695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6dd81021d0 msgr2=0x7f6dd81a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:46.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.605+0000 7f6dde600640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6dd81021d0 0x7f6dd81a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.605+0000 7f6dde600640 1 -- 172.21.15.6:0/464310695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd819b370 con 0x7f6dd819a770 2024-04-26T02:02:46.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.606+0000 7f6dde600640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd819a770 0x7f6dd819abf0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009fd0 tx=0x7f6dd000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:46.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.606+0000 7f6dd6000640 1 -- 172.21.15.6:0/464310695 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd000ce20 con 0x7f6dd819a770 2024-04-26T02:02:46.605 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.606+0000 7f6dd6000640 1 -- 172.21.15.6:0/464310695 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dd0005020 con 0x7f6dd819a770 2024-04-26T02:02:46.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.606+0000 7f6de0600640 1 -- 172.21.15.6:0/464310695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd81a7450 con 0x7f6dd819a770 2024-04-26T02:02:46.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.606+0000 7f6de0600640 1 -- 172.21.15.6:0/464310695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dd81a7800 con 0x7f6dd819a770 2024-04-26T02:02:46.606 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.606+0000 7f6dd6000640 1 -- 172.21.15.6:0/464310695 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd0010690 con 0x7f6dd819a770 2024-04-26T02:02:46.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.611+0000 7f6de33fc740 1 Processor -- start 2024-04-26T02:02:46.610 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.611+0000 7f6dd6000640 1 -- 172.21.15.6:0/464310695 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f6dd0004a40 con 0x7f6dd819a770 2024-04-26T02:02:46.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.612+0000 7f6de33fc740 1 -- start start 2024-04-26T02:02:46.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.612+0000 7f6de33fc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed496f820 0x555ed496fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.612+0000 7f6de33fc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed4970140 0x555ed497a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.612+0000 7f6de33fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed497aa00 0x555ed497cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.611 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.612+0000 7f6de33fc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555ed48d5900 con 0x555ed496f820 2024-04-26T02:02:46.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.612+0000 7f6de33fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555ed487bef0 con 0x555ed497aa00 2024-04-26T02:02:46.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.612+0000 7f6de33fc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555ed48d18c0 con 0x555ed4970140 2024-04-26T02:02:46.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.613+0000 7f6dddc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed4970140 0x555ed497a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.613+0000 7f6dddc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed4970140 0x555ed497a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50850/0 (socket says 172.21.15.6:50850) 2024-04-26T02:02:46.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.613+0000 7f6dddc00640 1 -- 172.21.15.6:0/2195887374 learned_addr learned my addr 172.21.15.6:0/2195887374 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:46.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.613+0000 7f6ddd200640 1 --2- 172.21.15.6:0/2195887374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed497aa00 0x555ed497cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.612 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.613+0000 7f6ddd200640 1 --2- 172.21.15.6:0/2195887374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed496f820 0x555ed496fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.614+0000 7f6dd6000640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f6dc00a3d10 0x7f6dc00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.614+0000 7f6dddc00640 1 -- 172.21.15.6:0/2195887374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed497aa00 msgr2=0x555ed497cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.614+0000 7f6dddc00640 1 --2- 172.21.15.6:0/2195887374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed497aa00 0x555ed497cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.614+0000 7f6dddc00640 1 -- 172.21.15.6:0/2195887374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed496f820 msgr2=0x555ed496fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.614+0000 7f6dddc00640 1 --2- 172.21.15.6:0/2195887374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed496f820 0x555ed496fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.614+0000 7f6dddc00640 1 -- 172.21.15.6:0/2195887374 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ed45e8210 con 0x555ed4970140 2024-04-26T02:02:46.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.614+0000 7f6dd6000640 1 -- 172.21.15.6:0/464310695 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7f6dd00d0d40 con 0x7f6dd819a770 2024-04-26T02:02:46.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.614+0000 7f6dde600640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f6dc00a3d10 0x7f6dc00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.614+0000 7f6dddc00640 1 --2- 172.21.15.6:0/2195887374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed4970140 0x555ed497a4c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009e10 tx=0x7f6dcc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ac8db500649b240 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:46.614 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.615+0000 7f6ddd200640 1 --2- 172.21.15.6:0/2195887374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed496f820 0x555ed496fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.615+0000 7f6dcaa00640 1 -- 172.21.15.6:0/2195887374 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc01e070 con 0x555ed4970140 2024-04-26T02:02:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.615+0000 7f6dcaa00640 1 -- 172.21.15.6:0/2195887374 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dcc019ae0 con 0x555ed4970140 2024-04-26T02:02:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.615+0000 7f6dcaa00640 1 -- 172.21.15.6:0/2195887374 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc00fb00 con 0x555ed4970140 2024-04-26T02:02:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.615+0000 7f6dde600640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f6dc00a3d10 0x7f6dc00a61d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0099660 tx=0x7f6dd0005f70 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.616+0000 7f6de33fc740 1 -- 172.21.15.6:0/2195887374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed4970140 msgr2=0x555ed497a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.616+0000 7f6de33fc740 1 --2- 172.21.15.6:0/2195887374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed4970140 0x555ed497a4c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009e10 tx=0x7f6dcc011810 comp rx=0 tx=0).stop 2024-04-26T02:02:46.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.617+0000 7f6de33fc740 1 -- 172.21.15.6:0/2195887374 shutdown_connections 2024-04-26T02:02:46.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.617+0000 7f6de33fc740 1 --2- 172.21.15.6:0/2195887374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed497aa00 0x555ed497cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.617+0000 7f6de33fc740 1 --2- 172.21.15.6:0/2195887374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed4970140 0x555ed497a4c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.617+0000 7f6de33fc740 1 --2- 172.21.15.6:0/2195887374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed496f820 0x555ed496fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.617+0000 7f6de33fc740 1 -- 172.21.15.6:0/2195887374 >> 172.21.15.6:0/2195887374 conn(0x555ed496bd30 msgr2=0x555ed496c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:46.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.617+0000 7f6de33fc740 1 -- 172.21.15.6:0/2195887374 shutdown_connections 2024-04-26T02:02:46.617 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.617+0000 7f6de33fc740 1 -- 172.21.15.6:0/2195887374 wait complete. 2024-04-26T02:02:46.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.617+0000 7f6de33fc740 1 -- start start 2024-04-26T02:02:46.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.618+0000 7f6de33fc740 1 Processor -- start 2024-04-26T02:02:46.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.618+0000 7f6de33fc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed496f820 0x555ed49778a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.618+0000 7f6de33fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed4970140 0x555ed4977de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.618+0000 7f6de33fc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed497aa00 0x555ed4978320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.618+0000 7f6de33fc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555ed497d630 con 0x555ed496f820 2024-04-26T02:02:46.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.618+0000 7f6de33fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555ed488f310 con 0x555ed4970140 2024-04-26T02:02:46.618 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.618+0000 7f6de33fc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555ed497d7b0 con 0x555ed497aa00 2024-04-26T02:02:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.619+0000 7f6de33fc740 10 client.0 _ll_register_callbacks cb 0x555ed485dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.619+0000 7f6dddc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed4970140 0x555ed4977de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.619+0000 7f6dddc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed4970140 0x555ed4977de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:37978/0 (socket says 172.21.15.6:37978) 2024-04-26T02:02:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.619+0000 7f6dddc00640 1 -- 172.21.15.6:0/336727417 learned_addr learned my addr 172.21.15.6:0/336727417 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.619+0000 7f6ddd200640 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed496f820 0x555ed49778a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.619+0000 7f6ddd200640 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed497aa00 0x555ed4978320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.619+0000 7f6dddc00640 1 -- 172.21.15.6:0/336727417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed497aa00 msgr2=0x555ed4978320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.619+0000 7f6dddc00640 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed497aa00 0x555ed4978320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.619+0000 7f6dddc00640 1 -- 172.21.15.6:0/336727417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed496f820 msgr2=0x555ed49778a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.619+0000 7f6dddc00640 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed496f820 0x555ed49778a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.619+0000 7f6dddc00640 1 -- 172.21.15.6:0/336727417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ed46b4620 con 0x555ed4970140 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.619+0000 7f6ddd200640 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed497aa00 0x555ed4978320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.620+0000 7f6dddc00640 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed4970140 0x555ed4977de0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009870 tx=0x7f6dcc009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f6e430ca23081a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.620+0000 7f6dbf400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.620+0000 7f6dbf400640 1 -- 172.21.15.6:0/336727417 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc01e040 con 0x555ed4970140 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.620+0000 7f6dbf400640 1 -- 172.21.15.6:0/336727417 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7f6dcc004450 con 0x555ed4970140 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.620+0000 7f6de33fc740 1 -- 172.21.15.6:0/336727417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dcc019ae0 con 0x555ed4970140 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.621+0000 7f6dbf400640 1 -- 172.21.15.6:0/336727417 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f6dcc019430 con 0x555ed4970140 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.621+0000 7f6de33fc740 10 client.27128 fetch_fsmap learned FSMap version 52 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.621+0000 7f6de33fc740 1 -- 172.21.15.6:0/336727417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x555ed4991f70 con 0x555ed4970140 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.621+0000 7f6dbf400640 1 -- 172.21.15.6:0/336727417 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dcc019ae0 con 0x555ed4970140 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.621+0000 7f6dbf400640 1 -- 172.21.15.6:0/336727417 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 52) ==== 8342+0+0 (secure 0 0 0) 0x7f6dcc0035f0 con 0x555ed4970140 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.621+0000 7f6de33fc740 10 client.27128 fetch_fsmap finished waiting for FSMap version 52 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.621+0000 7f6de33fc740 10 client.27128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:02:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.621+0000 7f6de33fc740 20 client.27128 populate_metadata read hostname 'smithi006' 2024-04-26T02:02:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.621+0000 7f6de33fc740 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ed4993320 0x555ed4995710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.621+0000 7f6de33fc740 4 client.27128 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:02:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.621+0000 7f6de33fc740 1 -- 172.21.15.6:0/336727417 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ed48e98b0 con 0x555ed4993320 2024-04-26T02:02:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.622+0000 7f6ddd200640 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ed4993320 0x555ed4995710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.623+0000 7f6ddd200640 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ed4993320 0x555ed4995710 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c074137107a0803 server_cookie=f0b709b13ee26adc in_seq=0 out_seq=0 2024-04-26T02:02:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.623+0000 7f6dbf400640 10 client.27128 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.625+0000 7f6dbf400640 1 -- 172.21.15.6:0/336727417 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555ed48e98b0 con 0x555ed4993320 2024-04-26T02:02:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.625+0000 7f6dbf400640 10 client.27128 handle_command_reply: tid=0 2024-04-26T02:02:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.625+0000 7f6de33fc740 1 client.27128 shutdown 2024-04-26T02:02:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.625+0000 7f6de33fc740 1 -- 172.21.15.6:0/336727417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed4970140 msgr2=0x555ed4977de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.625+0000 7f6de33fc740 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed4970140 0x555ed4977de0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009870 tx=0x7f6dcc009b90 comp rx=0 tx=0).stop 2024-04-26T02:02:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.626+0000 7f6de33fc740 1 -- 172.21.15.6:0/336727417 shutdown_connections 2024-04-26T02:02:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.626+0000 7f6de33fc740 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ed4993320 0x555ed4995710 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.626+0000 7f6de33fc740 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed497aa00 0x555ed4978320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.626+0000 7f6dbf400640 0 client.27128 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.626+0000 7f6de33fc740 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed4970140 0x555ed4977de0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.626+0000 7f6de33fc740 1 --2- 172.21.15.6:0/336727417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed496f820 0x555ed49778a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.626+0000 7f6de33fc740 1 -- 172.21.15.6:0/336727417 >> 172.21.15.6:0/336727417 conn(0x555ed496bd30 msgr2=0x555ed4971ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.626+0000 7f6de33fc740 1 -- 172.21.15.6:0/336727417 shutdown_connections 2024-04-26T02:02:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.626+0000 7f6de33fc740 1 -- 172.21.15.6:0/336727417 wait complete. 2024-04-26T02:02:46.626 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.626+0000 7f6de33fc740 20 client.27128 trim_cache size 0 max 16384 2024-04-26T02:02:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.651+0000 7f6de33fc740 1 Processor -- start 2024-04-26T02:02:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.651+0000 7f6de33fc740 1 -- start start 2024-04-26T02:02:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.651+0000 7f6de33fc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed496f820 0x555ed4976a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.651+0000 7f6de33fc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed4970140 0x555ed499af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.651+0000 7f6de33fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed497aa00 0x555ed499b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6de33fc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6db8011190 con 0x555ed4970140 2024-04-26T02:02:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6de33fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6db800ea60 con 0x555ed497aa00 2024-04-26T02:02:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6de33fc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6db800c8b0 con 0x555ed496f820 2024-04-26T02:02:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6ddd200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed496f820 0x555ed4976a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6ddd200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed496f820 0x555ed4976a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50870/0 (socket says 172.21.15.6:50870) 2024-04-26T02:02:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6ddd200640 1 -- 172.21.15.6:0/1978090932 learned_addr learned my addr 172.21.15.6:0/1978090932 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6dddc00640 1 --2- 172.21.15.6:0/1978090932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed4970140 0x555ed499af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6ddd200640 1 --2- 172.21.15.6:0/1978090932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed497aa00 0x555ed499b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6ddd200640 1 -- 172.21.15.6:0/1978090932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed497aa00 msgr2=0x555ed499b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6ddd200640 1 --2- 172.21.15.6:0/1978090932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed497aa00 0x555ed499b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6ddd200640 1 -- 172.21.15.6:0/1978090932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed4970140 msgr2=0x555ed499af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6ddd200640 1 --2- 172.21.15.6:0/1978090932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed4970140 0x555ed499af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6ddd200640 1 -- 172.21.15.6:0/1978090932 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ed496ff70 con 0x555ed496f820 2024-04-26T02:02:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.652+0000 7f6ddd200640 1 --2- 172.21.15.6:0/1978090932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed496f820 0x555ed4976a40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4005ab0 tx=0x7f6dc402a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dba93e7719f8dbf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.655+0000 7f6dca000640 1 -- 172.21.15.6:0/1978090932 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc4046040 con 0x555ed496f820 2024-04-26T02:02:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.655+0000 7f6dca000640 1 -- 172.21.15.6:0/1978090932 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dc4046790 con 0x555ed496f820 2024-04-26T02:02:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.655+0000 7f6dca000640 1 -- 172.21.15.6:0/1978090932 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc4035040 con 0x555ed496f820 2024-04-26T02:02:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.656+0000 7f6de33fc740 1 -- 172.21.15.6:0/1978090932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed496f820 msgr2=0x555ed4976a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.656+0000 7f6de33fc740 1 --2- 172.21.15.6:0/1978090932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed496f820 0x555ed4976a40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4005ab0 tx=0x7f6dc402a3f0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.656+0000 7f6de33fc740 1 -- 172.21.15.6:0/1978090932 shutdown_connections 2024-04-26T02:02:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.656+0000 7f6de33fc740 1 --2- 172.21.15.6:0/1978090932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed4970140 0x555ed499af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.656+0000 7f6de33fc740 1 --2- 172.21.15.6:0/1978090932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed497aa00 0x555ed499b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.656+0000 7f6de33fc740 1 --2- 172.21.15.6:0/1978090932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed496f820 0x555ed4976a40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.656+0000 7f6de33fc740 1 -- 172.21.15.6:0/1978090932 >> 172.21.15.6:0/1978090932 conn(0x555ed496bd30 msgr2=0x555ed496c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.656+0000 7f6de33fc740 1 -- 172.21.15.6:0/1978090932 shutdown_connections 2024-04-26T02:02:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.657+0000 7f6de33fc740 1 -- 172.21.15.6:0/1978090932 wait complete. 2024-04-26T02:02:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.657+0000 7f6de33fc740 1 -- start start 2024-04-26T02:02:46.657 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.657+0000 7f6de33fc740 1 Processor -- start 2024-04-26T02:02:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.659+0000 7f6de33fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed496f820 0x555ed4988450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.659+0000 7f6de33fc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed4970140 0x555ed4988990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.659+0000 7f6de33fc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed497aa00 0x555ed49f7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.659+0000 7f6de33fc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6db800cf20 con 0x555ed4970140 2024-04-26T02:02:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.659+0000 7f6de33fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6db800e390 con 0x555ed496f820 2024-04-26T02:02:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.659+0000 7f6de33fc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6db800d5f0 con 0x555ed497aa00 2024-04-26T02:02:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.659+0000 7f6de33fc740 10 client.0 _ll_register_callbacks cb 0x7f6dcc009a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.659+0000 7f6ddd200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed496f820 0x555ed4988450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.659+0000 7f6ddd200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed496f820 0x555ed4988450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:37986/0 (socket says 172.21.15.6:37986) 2024-04-26T02:02:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.659+0000 7f6ddd200640 1 -- 172.21.15.6:0/1060261902 learned_addr learned my addr 172.21.15.6:0/1060261902 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.660+0000 7f6ddd200640 1 --2- 172.21.15.6:0/1060261902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed497aa00 0x555ed49f7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.660+0000 7f6ddd200640 1 -- 172.21.15.6:0/1060261902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed497aa00 msgr2=0x555ed49f7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.660+0000 7f6ddd200640 1 --2- 172.21.15.6:0/1060261902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed497aa00 0x555ed49f7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.660+0000 7f6ddd200640 1 -- 172.21.15.6:0/1060261902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed4970140 msgr2=0x555ed4988990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.660+0000 7f6ddd200640 1 --2- 172.21.15.6:0/1060261902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed4970140 0x555ed4988990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.660+0000 7f6ddd200640 1 -- 172.21.15.6:0/1060261902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ed49f7d40 con 0x555ed496f820 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.660+0000 7f6ddd200640 1 --2- 172.21.15.6:0/1060261902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed496f820 0x555ed4988450 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4004370 tx=0x7f6dc40045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a02b326467486e63 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.660+0000 7f6dbf400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.661+0000 7f6dbf400640 1 -- 172.21.15.6:0/1060261902 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc4053060 con 0x555ed496f820 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.661+0000 7f6dbf400640 1 -- 172.21.15.6:0/1060261902 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7f6dc40462e0 con 0x555ed496f820 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.661+0000 7f6de33fc740 1 -- 172.21.15.6:0/1060261902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dc4046790 con 0x555ed496f820 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.661+0000 7f6dbf400640 1 -- 172.21.15.6:0/1060261902 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f6dc4035040 con 0x555ed496f820 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.661+0000 7f6de33fc740 10 client.27134 fetch_fsmap learned FSMap version 52 2024-04-26T02:02:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.661+0000 7f6de33fc740 1 -- 172.21.15.6:0/1060261902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x555ed49f8ff0 con 0x555ed496f820 2024-04-26T02:02:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.662+0000 7f6dbf400640 1 -- 172.21.15.6:0/1060261902 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dc4046790 con 0x555ed496f820 2024-04-26T02:02:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.662+0000 7f6dbf400640 1 -- 172.21.15.6:0/1060261902 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 52) ==== 8342+0+0 (secure 0 0 0) 0x7f6dc4003990 con 0x555ed496f820 2024-04-26T02:02:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.662+0000 7f6de33fc740 10 client.27134 fetch_fsmap finished waiting for FSMap version 52 2024-04-26T02:02:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.662+0000 7f6de33fc740 10 client.27134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:02:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.662+0000 7f6de33fc740 20 client.27134 populate_metadata read hostname 'smithi006' 2024-04-26T02:02:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.663+0000 7f6de33fc740 1 --2- 172.21.15.6:0/1060261902 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ed49f95f0 0x555ed49fba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.663+0000 7f6de33fc740 4 client.27134 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:02:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.663+0000 7f6de33fc740 1 -- 172.21.15.6:0/1060261902 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555ed48e3320 con 0x555ed49f95f0 2024-04-26T02:02:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.663+0000 7f6dddc00640 1 --2- 172.21.15.6:0/1060261902 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ed49f95f0 0x555ed49fba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.664+0000 7f6dbf400640 10 client.27134 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:46.664 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.664+0000 7f6dddc00640 1 --2- 172.21.15.6:0/1060261902 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ed49f95f0 0x555ed49fba30 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81d288e342ea0b53 server_cookie=d61e226f505a3a7 in_seq=0 out_seq=0 2024-04-26T02:02:46.906 INFO:tasks.quiescer.fs.[cephfs]:Total duration (45.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T02:02:46.906 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.55 2024-04-26T02:02:46.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:02:46.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:46 smithi029 ceph-mon[32775]: pgmap v535: 97 pgs: 97 active+clean; 763 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.4 MiB/s wr, 539 op/s 2024-04-26T02:02:46.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.962+0000 7f6dbf400640 1 -- 172.21.15.6:0/1060261902 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555ed48e3320 con 0x555ed49f95f0 2024-04-26T02:02:46.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.962+0000 7f6dbf400640 10 client.27134 handle_command_reply: tid=0 2024-04-26T02:02:46.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.962+0000 7f6de33fc740 1 client.27134 shutdown 2024-04-26T02:02:46.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.963+0000 7f6de33fc740 1 -- 172.21.15.6:0/1060261902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed496f820 msgr2=0x555ed4988450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:46.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.963+0000 7f6de33fc740 1 --2- 172.21.15.6:0/1060261902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed496f820 0x555ed4988450 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4004370 tx=0x7f6dc40045e0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.963+0000 7f6de33fc740 1 -- 172.21.15.6:0/1060261902 shutdown_connections 2024-04-26T02:02:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.963+0000 7f6de33fc740 1 --2- 172.21.15.6:0/1060261902 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555ed49f95f0 0x555ed49fba30 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.963+0000 7f6dbf400640 0 client.27134 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.963+0000 7f6de33fc740 1 --2- 172.21.15.6:0/1060261902 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555ed4970140 0x555ed4988990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.963+0000 7f6de33fc740 1 --2- 172.21.15.6:0/1060261902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555ed497aa00 0x555ed49f7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.963+0000 7f6de33fc740 1 --2- 172.21.15.6:0/1060261902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555ed496f820 0x555ed4988450 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.963+0000 7f6de33fc740 1 -- 172.21.15.6:0/1060261902 >> 172.21.15.6:0/1060261902 conn(0x555ed496bd30 msgr2=0x555ed4977840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.964+0000 7f6de33fc740 1 -- 172.21.15.6:0/1060261902 shutdown_connections 2024-04-26T02:02:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.964+0000 7f6de33fc740 1 -- 172.21.15.6:0/1060261902 wait complete. 2024-04-26T02:02:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.964+0000 7f6de33fc740 20 client.27134 trim_cache size 0 max 16384 2024-04-26T02:02:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:02:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:02:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:02:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:02:46.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.965+0000 7f6de0600640 1 -- 172.21.15.6:0/464310695 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f6dc00a3d10 msgr2=0x7f6dc00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:46.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.965+0000 7f6de0600640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f6dc00a3d10 0x7f6dc00a61d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0099660 tx=0x7f6dd0005f70 comp rx=0 tx=0).stop 2024-04-26T02:02:46.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.965+0000 7f6de0600640 1 -- 172.21.15.6:0/464310695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd819a770 msgr2=0x7f6dd819abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:46.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.965+0000 7f6de0600640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd819a770 0x7f6dd819abf0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009fd0 tx=0x7f6dd000ef10 comp rx=0 tx=0).stop 2024-04-26T02:02:46.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.965+0000 7f6de0600640 1 -- 172.21.15.6:0/464310695 shutdown_connections 2024-04-26T02:02:46.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.965+0000 7f6de0600640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f6dc00a3d10 0x7f6dc00a61d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.965+0000 7f6de0600640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd819a770 0x7f6dd819abf0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.965+0000 7f6de0600640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dd81072a0 0x7f6dd81a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.965+0000 7f6de0600640 1 --2- 172.21.15.6:0/464310695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6dd81021d0 0x7f6dd81a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:46.965 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.965+0000 7f6de0600640 1 -- 172.21.15.6:0/464310695 >> 172.21.15.6:0/464310695 conn(0x7f6dd80fda30 msgr2=0x7f6dd80ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:46.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.966+0000 7f6de0600640 1 -- 172.21.15.6:0/464310695 shutdown_connections 2024-04-26T02:02:46.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:02:46.966+0000 7f6de0600640 1 -- 172.21.15.6:0/464310695 wait complete. 2024-04-26T02:02:46.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:46 smithi006 ceph-mon[28419]: pgmap v535: 97 pgs: 97 active+clean; 763 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.4 MiB/s wr, 539 op/s 2024-04-26T02:02:47.111 INFO:tasks.workunit.client.0.smithi006.stdout: 10 73637 32.00 MB/sec execute 70 sec latency 154.823 ms 2024-04-26T02:02:47.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:46 smithi188 ceph-mon[32751]: pgmap v535: 97 pgs: 97 active+clean; 763 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.4 MiB/s wr, 539 op/s 2024-04-26T02:02:47.476 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:02:48.111 INFO:tasks.workunit.client.0.smithi006.stdout: 10 74190 31.81 MB/sec execute 71 sec latency 126.957 ms 2024-04-26T02:02:48.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:48 smithi029 ceph-mon[32775]: pgmap v536: 97 pgs: 97 active+clean; 786 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 810 op/s 2024-04-26T02:02:49.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:48 smithi006 ceph-mon[28419]: pgmap v536: 97 pgs: 97 active+clean; 786 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 810 op/s 2024-04-26T02:02:49.112 INFO:tasks.workunit.client.0.smithi006.stdout: 10 74815 31.77 MB/sec execute 72 sec latency 107.643 ms 2024-04-26T02:02:49.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:48 smithi188 ceph-mon[32751]: pgmap v536: 97 pgs: 97 active+clean; 786 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 810 op/s 2024-04-26T02:02:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.111+0000 7ff244c00700 1 -- 172.21.15.6:0/1071588593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240075590 msgr2=0x7ff24010ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.111+0000 7ff244c00700 1 --2- 172.21.15.6:0/1071588593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240075590 0x7ff24010ef60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009990 tx=0x7ff22c009ca0 comp rx=0 tx=0).stop 2024-04-26T02:02:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.111+0000 7ff244c00700 1 -- 172.21.15.6:0/1071588593 shutdown_connections 2024-04-26T02:02:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.111+0000 7ff244c00700 1 --2- 172.21.15.6:0/1071588593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff24010f4a0 0x7ff240111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.111+0000 7ff244c00700 1 --2- 172.21.15.6:0/1071588593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff240074c70 0x7ff240075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.111+0000 7ff244c00700 1 --2- 172.21.15.6:0/1071588593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240075590 0x7ff24010ef60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.111+0000 7ff244c00700 1 -- 172.21.15.6:0/1071588593 >> 172.21.15.6:0/1071588593 conn(0x7ff240079ba0 msgr2=0x7ff240079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:50.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.111+0000 7ff244c00700 1 -- 172.21.15.6:0/1071588593 shutdown_connections 2024-04-26T02:02:50.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.112+0000 7ff244c00700 1 -- 172.21.15.6:0/1071588593 wait complete. 2024-04-26T02:02:50.112 INFO:tasks.workunit.client.0.smithi006.stdout: 10 75431 31.83 MB/sec execute 73 sec latency 170.214 ms 2024-04-26T02:02:50.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.114+0000 7ff244c00700 1 Processor -- start 2024-04-26T02:02:50.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.114+0000 7ff244c00700 1 -- start start 2024-04-26T02:02:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.115+0000 7ff244c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff240074c70 0x7ff24019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.115+0000 7ff244c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240075590 0x7ff24019b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.115+0000 7ff244c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff24010f4a0 0x7ff24019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.115+0000 7ff244c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff240114e40 con 0x7ff24010f4a0 2024-04-26T02:02:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.115+0000 7ff244c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff240114cc0 con 0x7ff240074c70 2024-04-26T02:02:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.115+0000 7ff244c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff240114fc0 con 0x7ff240075590 2024-04-26T02:02:50.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.115+0000 7ff23d400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240075590 0x7ff24019b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:50.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.115+0000 7ff23d400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240075590 0x7ff24019b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44772/0 (socket says 172.21.15.6:44772) 2024-04-26T02:02:50.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.115+0000 7ff23d400700 1 -- 172.21.15.6:0/588065699 learned_addr learned my addr 172.21.15.6:0/588065699 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:50.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.116+0000 7ff23d400700 1 -- 172.21.15.6:0/588065699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff240074c70 msgr2=0x7ff24019b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:50.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.116+0000 7ff23de00700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff240074c70 0x7ff24019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:50.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.116+0000 7ff23d400700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff240074c70 0x7ff24019b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:50.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.116+0000 7ff23e800700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff24010f4a0 0x7ff24019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:50.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.116+0000 7ff23d400700 1 -- 172.21.15.6:0/588065699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff24010f4a0 msgr2=0x7ff24019fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:50.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.116+0000 7ff23d400700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff24010f4a0 0x7ff24019fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:50.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.116+0000 7ff23d400700 1 -- 172.21.15.6:0/588065699 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2401a0270 con 0x7ff240075590 2024-04-26T02:02:50.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.116+0000 7ff23e800700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff24010f4a0 0x7ff24019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:50.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.116+0000 7ff23d400700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240075590 0x7ff24019b7f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0057c0 tx=0x7ff22c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:50.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.116+0000 7ff236a00700 1 -- 172.21.15.6:0/588065699 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c026070 con 0x7ff240075590 2024-04-26T02:02:50.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.117+0000 7ff244c00700 1 -- 172.21.15.6:0/588065699 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2401a0500 con 0x7ff240075590 2024-04-26T02:02:50.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.117+0000 7ff244c00700 1 -- 172.21.15.6:0/588065699 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2401a7de0 con 0x7ff240075590 2024-04-26T02:02:50.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.117+0000 7ff236a00700 1 -- 172.21.15.6:0/588065699 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff22c00fbf0 con 0x7ff240075590 2024-04-26T02:02:50.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.117+0000 7ff236a00700 1 -- 172.21.15.6:0/588065699 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0173f0 con 0x7ff240075590 2024-04-26T02:02:50.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.119+0000 7ff244c00700 1 -- 172.21.15.6:0/588065699 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff220005330 con 0x7ff240075590 2024-04-26T02:02:50.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.121+0000 7ff236a00700 1 -- 172.21.15.6:0/588065699 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7ff22c029070 con 0x7ff240075590 2024-04-26T02:02:50.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.123+0000 7ff236a00700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff2240a3e30 0x7ff2240a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:50.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.123+0000 7ff236a00700 1 -- 172.21.15.6:0/588065699 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7ff22c0d1430 con 0x7ff240075590 2024-04-26T02:02:50.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.124+0000 7ff23de00700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff2240a3e30 0x7ff2240a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:50.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.125+0000 7ff23de00700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff2240a3e30 0x7ff2240a62f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff230009b30 tx=0x7ff230009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:50.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.130+0000 7ff236a00700 1 -- 172.21.15.6:0/588065699 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff22c013070 con 0x7ff240075590 2024-04-26T02:02:50.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.505+0000 7ff244c00700 1 -- 172.21.15.6:0/588065699 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2200058f0 con 0x7ff240075590 2024-04-26T02:02:50.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.506+0000 7ff236a00700 1 -- 172.21.15.6:0/588065699 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+11851 (secure 0 0 0) 0x7ff22c0b1e20 con 0x7ff240075590 2024-04-26T02:02:50.506 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:02:50.506 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":52,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T02:02:44.469956+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26935":{"gid":26935,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1750728695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1750728695},{"type":"v1","addr":"172.21.15.188:6835","nonce":1750728695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24512,24505]},"id":1}]} 2024-04-26T02:02:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.510+0000 7ff244c00700 1 -- 172.21.15.6:0/588065699 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff2240a3e30 msgr2=0x7ff2240a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.510+0000 7ff244c00700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff2240a3e30 0x7ff2240a62f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff230009b30 tx=0x7ff230009260 comp rx=0 tx=0).stop 2024-04-26T02:02:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.510+0000 7ff244c00700 1 -- 172.21.15.6:0/588065699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240075590 msgr2=0x7ff24019b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.510+0000 7ff244c00700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240075590 0x7ff24019b7f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0057c0 tx=0x7ff22c009580 comp rx=0 tx=0).stop 2024-04-26T02:02:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.510+0000 7ff244c00700 1 -- 172.21.15.6:0/588065699 shutdown_connections 2024-04-26T02:02:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.510+0000 7ff244c00700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff2240a3e30 0x7ff2240a62f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.510+0000 7ff244c00700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff24010f4a0 0x7ff24019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.510+0000 7ff244c00700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff240074c70 0x7ff24019b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.510+0000 7ff244c00700 1 --2- 172.21.15.6:0/588065699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240075590 0x7ff24019b7f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.510+0000 7ff244c00700 1 -- 172.21.15.6:0/588065699 >> 172.21.15.6:0/588065699 conn(0x7ff240079ba0 msgr2=0x7ff2400ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.511+0000 7ff244c00700 1 -- 172.21.15.6:0/588065699 shutdown_connections 2024-04-26T02:02:50.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:50.511+0000 7ff244c00700 1 -- 172.21.15.6:0/588065699 wait complete. 2024-04-26T02:02:50.512 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 52 2024-04-26T02:02:50.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:50 smithi006 ceph-mon[28419]: pgmap v537: 97 pgs: 97 active+clean; 786 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.0 MiB/s wr, 508 op/s 2024-04-26T02:02:50.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:50 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/588065699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:02:50.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:50 smithi029 ceph-mon[32775]: pgmap v537: 97 pgs: 97 active+clean; 786 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.0 MiB/s wr, 508 op/s 2024-04-26T02:02:50.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:50 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/588065699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:02:51.112 INFO:tasks.workunit.client.0.smithi006.stdout: 10 76192 32.04 MB/sec execute 74 sec latency 81.828 ms 2024-04-26T02:02:51.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:50 smithi188 ceph-mon[32751]: pgmap v537: 97 pgs: 97 active+clean; 786 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.0 MiB/s wr, 508 op/s 2024-04-26T02:02:51.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:50 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/588065699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:02:51.490 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 82.55 --await-for 42.0' 2024-04-26T02:02:51.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 82.55 --await-for 42.0 2024-04-26T02:02:52.097 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:02:52.112 INFO:tasks.workunit.client.0.smithi006.stdout: 10 76768 32.04 MB/sec execute 75 sec latency 62.066 ms 2024-04-26T02:02:52.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:52 smithi006 ceph-mon[28419]: pgmap v538: 97 pgs: 97 active+clean; 796 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 868 op/s 2024-04-26T02:02:52.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:52 smithi029 ceph-mon[32775]: pgmap v538: 97 pgs: 97 active+clean; 796 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 868 op/s 2024-04-26T02:02:53.113 INFO:tasks.workunit.client.0.smithi006.stdout: 10 77275 31.89 MB/sec execute 76 sec latency 48.836 ms 2024-04-26T02:02:53.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:52 smithi188 ceph-mon[32751]: pgmap v538: 97 pgs: 97 active+clean; 796 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 868 op/s 2024-04-26T02:02:54.113 INFO:tasks.workunit.client.0.smithi006.stdout: 10 77991 31.97 MB/sec execute 77 sec latency 83.642 ms 2024-04-26T02:02:54.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.407+0000 7fdc3c600700 1 -- 172.21.15.6:0/4031292027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc3406a740 msgr2=0x7fdc3406abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:54.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.407+0000 7fdc3c600700 1 --2- 172.21.15.6:0/4031292027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc3406a740 0x7fdc3406abc0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009990 tx=0x7fdc20009ca0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.407+0000 7fdc3c600700 1 -- 172.21.15.6:0/4031292027 shutdown_connections 2024-04-26T02:02:54.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.407+0000 7fdc3c600700 1 --2- 172.21.15.6:0/4031292027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc3406b100 0x7fdc3410f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.407+0000 7fdc3c600700 1 --2- 172.21.15.6:0/4031292027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc34108a60 0x7fdc34108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.407+0000 7fdc3c600700 1 --2- 172.21.15.6:0/4031292027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc3406a740 0x7fdc3406abc0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.407+0000 7fdc3c600700 1 -- 172.21.15.6:0/4031292027 >> 172.21.15.6:0/4031292027 conn(0x7fdc340788e0 msgr2=0x7fdc34078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:54.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.407+0000 7fdc3c600700 1 -- 172.21.15.6:0/4031292027 shutdown_connections 2024-04-26T02:02:54.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.407+0000 7fdc3c600700 1 -- 172.21.15.6:0/4031292027 wait complete. 2024-04-26T02:02:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.409+0000 7fdc3c600700 1 Processor -- start 2024-04-26T02:02:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.409+0000 7fdc3c600700 1 -- start start 2024-04-26T02:02:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.409+0000 7fdc3c600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc3406a740 0x7fdc34199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.409+0000 7fdc3c600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc3406b100 0x7fdc341995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.409+0000 7fdc3c600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc34108a60 0x7fdc3419d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.409+0000 7fdc3c600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdc34112c60 con 0x7fdc3406b100 2024-04-26T02:02:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.409+0000 7fdc3c600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdc34112ae0 con 0x7fdc3406a740 2024-04-26T02:02:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.410+0000 7fdc3c600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdc34112de0 con 0x7fdc34108a60 2024-04-26T02:02:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.410+0000 7fdc39200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc3406b100 0x7fdc341995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.410+0000 7fdc39200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc3406b100 0x7fdc341995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36670/0 (socket says 172.21.15.6:36670) 2024-04-26T02:02:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.410+0000 7fdc39200700 1 -- 172.21.15.6:0/3938309856 learned_addr learned my addr 172.21.15.6:0/3938309856 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.411+0000 7fdc39200700 1 -- 172.21.15.6:0/3938309856 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc34108a60 msgr2=0x7fdc3419d960 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T02:02:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.411+0000 7fdc39200700 1 --2- 172.21.15.6:0/3938309856 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc34108a60 0x7fdc3419d960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.411+0000 7fdc39200700 1 -- 172.21.15.6:0/3938309856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc3406a740 msgr2=0x7fdc34199090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T02:02:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.411+0000 7fdc39200700 1 --2- 172.21.15.6:0/3938309856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc3406a740 0x7fdc34199090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.411+0000 7fdc39200700 1 -- 172.21.15.6:0/3938309856 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3419e0a0 con 0x7fdc3406b100 2024-04-26T02:02:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.411+0000 7fdc39200700 1 --2- 172.21.15.6:0/3938309856 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc3406b100 0x7fdc341995d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdc20000c00 tx=0x7fdc20009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.412+0000 7fdc32000700 1 -- 172.21.15.6:0/3938309856 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20027070 con 0x7fdc3406b100 2024-04-26T02:02:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.412+0000 7fdc32000700 1 -- 172.21.15.6:0/3938309856 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc20022470 con 0x7fdc3406b100 2024-04-26T02:02:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.412+0000 7fdc32000700 1 -- 172.21.15.6:0/3938309856 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20017650 con 0x7fdc3406b100 2024-04-26T02:02:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.412+0000 7fdc3c600700 1 -- 172.21.15.6:0/3938309856 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3419e330 con 0x7fdc3406b100 2024-04-26T02:02:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.412+0000 7fdc3c600700 1 -- 172.21.15.6:0/3938309856 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc34101940 con 0x7fdc3406b100 2024-04-26T02:02:54.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.416+0000 7fdc32000700 1 -- 172.21.15.6:0/3938309856 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fdc20009d10 con 0x7fdc3406b100 2024-04-26T02:02:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.417+0000 7fdc46eb2b80 1 Processor -- start 2024-04-26T02:02:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.417+0000 7fdc46eb2b80 1 -- start start 2024-04-26T02:02:54.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.417+0000 7fdc46eb2b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7bdd0 0x56417dd7c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.417+0000 7fdc46eb2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd7c6f0 0x56417dd87020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.417+0000 7fdc46eb2b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 0x56417dd89950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.417+0000 7fdc46eb2b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56417dccf020 con 0x56417dd7bdd0 2024-04-26T02:02:54.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.417+0000 7fdc46eb2b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56417dc758c0 con 0x56417dd87560 2024-04-26T02:02:54.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.417+0000 7fdc46eb2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56417dccafe0 con 0x56417dd7c6f0 2024-04-26T02:02:54.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc32000700 1 --2- 172.21.15.6:0/3938309856 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdc1c0a3fb0 0x7fdc1c0a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc39200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 0x56417dd89950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc39200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 0x56417dd89950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:49296/0 (socket says 172.21.15.6:49296) 2024-04-26T02:02:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc39200700 1 -- 172.21.15.6:0/1700348129 learned_addr learned my addr 172.21.15.6:0/1700348129 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc32000700 1 -- 172.21.15.6:0/3938309856 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7fdc200d1200 con 0x7fdc3406b100 2024-04-26T02:02:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc3a600700 1 --2- 172.21.15.6:0/1700348129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7bdd0 0x56417dd7c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc39c00700 1 --2- 172.21.15.6:0/1700348129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd7c6f0 0x56417dd87020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc39200700 1 -- 172.21.15.6:0/1700348129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd7c6f0 msgr2=0x56417dd87020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc39200700 1 --2- 172.21.15.6:0/1700348129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd7c6f0 0x56417dd87020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc39200700 1 -- 172.21.15.6:0/1700348129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7bdd0 msgr2=0x56417dd7c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc39200700 1 --2- 172.21.15.6:0/1700348129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7bdd0 0x56417dd7c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc39200700 1 -- 172.21.15.6:0/1700348129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56417dc6b810 con 0x56417dd87560 2024-04-26T02:02:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.418+0000 7fdc39200700 1 --2- 172.21.15.6:0/1700348129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 0x56417dd89950 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdc2009d170 tx=0x7fdc20009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28ba245886687828 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.419+0000 7fdc26a00700 1 -- 172.21.15.6:0/1700348129 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2009cd20 con 0x56417dd87560 2024-04-26T02:02:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.419+0000 7fdc3a600700 1 --2- 172.21.15.6:0/3938309856 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdc1c0a3fb0 0x7fdc1c0a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.419+0000 7fdc26a00700 1 -- 172.21.15.6:0/1700348129 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc2009d8e0 con 0x56417dd87560 2024-04-26T02:02:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.419+0000 7fdc26a00700 1 -- 172.21.15.6:0/1700348129 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc200b4f40 con 0x56417dd87560 2024-04-26T02:02:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.420+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1700348129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 msgr2=0x56417dd89950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.420+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/1700348129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 0x56417dd89950 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdc2009d170 tx=0x7fdc20009310 comp rx=0 tx=0).stop 2024-04-26T02:02:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.420+0000 7fdc3a600700 1 --2- 172.21.15.6:0/3938309856 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdc1c0a3fb0 0x7fdc1c0a6470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0093f0 tx=0x7fdc2c009040 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.420+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1700348129 shutdown_connections 2024-04-26T02:02:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.420+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/1700348129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 0x56417dd89950 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.420+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/1700348129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7bdd0 0x56417dd7c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.420+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/1700348129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd7c6f0 0x56417dd87020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.420+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1700348129 >> 172.21.15.6:0/1700348129 conn(0x56417dcab0e0 msgr2=0x56417dd782e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:54.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.420+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1700348129 shutdown_connections 2024-04-26T02:02:54.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.420+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1700348129 wait complete. 2024-04-26T02:02:54.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.421+0000 7fdc46eb2b80 1 -- start start 2024-04-26T02:02:54.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.421+0000 7fdc46eb2b80 1 Processor -- start 2024-04-26T02:02:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.422+0000 7fdc46eb2b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7bdd0 0x56417dd86e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.422+0000 7fdc46eb2b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd7c6f0 0x56417dd84100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.422+0000 7fdc46eb2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd87560 0x56417dd84640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.422+0000 7fdc39c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7bdd0 0x56417dd86e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.422+0000 7fdc39c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7bdd0 0x56417dd86e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36692/0 (socket says 172.21.15.6:36692) 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.422+0000 7fdc39c00700 1 -- 172.21.15.6:0/1170611340 learned_addr learned my addr 172.21.15.6:0/1170611340 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.422+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1170611340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56417dccf020 con 0x56417dd7bdd0 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.422+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1170611340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56417dc889f0 con 0x56417dd7c6f0 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.422+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1170611340 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56417dc758c0 con 0x56417dd87560 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.422+0000 7fdc39200700 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd7c6f0 0x56417dd84100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.423+0000 7fdc3a600700 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd87560 0x56417dd84640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.423+0000 7fdc39c00700 1 -- 172.21.15.6:0/1170611340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd87560 msgr2=0x56417dd84640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.423+0000 7fdc39c00700 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd87560 0x56417dd84640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.423+0000 7fdc46eb2b80 10 client.0 _ll_register_callbacks cb 0x56417dc678b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.423+0000 7fdc39c00700 1 -- 172.21.15.6:0/1170611340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd7c6f0 msgr2=0x56417dd84100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.423+0000 7fdc39c00700 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd7c6f0 0x56417dd84100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.423+0000 7fdc39c00700 1 -- 172.21.15.6:0/1170611340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56417dc1ca60 con 0x56417dd7bdd0 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.423+0000 7fdc3a600700 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd87560 0x56417dd84640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.423+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1170611340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc2009d8e0 con 0x56417dd7bdd0 2024-04-26T02:02:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.424+0000 7fdc39c00700 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7bdd0 0x56417dd86e00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fdc28011460 tx=0x7fdc28011770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6034e0ab5e421d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.424+0000 7fdc1b400700 10 client.16962 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:02:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.424+0000 7fdc1b400700 1 -- 172.21.15.6:0/1170611340 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc28012650 con 0x56417dd7bdd0 2024-04-26T02:02:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.424+0000 7fdc1b400700 1 -- 172.21.15.6:0/1170611340 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7fdc280127f0 con 0x56417dd7bdd0 2024-04-26T02:02:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.424+0000 7fdc1b400700 1 -- 172.21.15.6:0/1170611340 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fdc280129f0 con 0x56417dd7bdd0 2024-04-26T02:02:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.424+0000 7fdc46eb2b80 10 client.16962 fetch_fsmap learned FSMap version 52 2024-04-26T02:02:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.424+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1170611340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({fsmap=52}) -- 0x56417dc65c80 con 0x56417dd7bdd0 2024-04-26T02:02:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.425+0000 7fdc1b400700 1 -- 172.21.15.6:0/1170611340 <== mon.0 v2:172.21.15.6:3300/0 4 ==== fsmap(e 52) ==== 8342+0+0 (secure 0 0 0) 0x7fdc2800f440 con 0x56417dd7bdd0 2024-04-26T02:02:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.425+0000 7fdc46eb2b80 10 client.16962 fetch_fsmap finished waiting for FSMap version 52 2024-04-26T02:02:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.425+0000 7fdc46eb2b80 10 client.16962 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:02:54.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.425+0000 7fdc46eb2b80 20 client.16962 populate_metadata read hostname 'smithi006' 2024-04-26T02:02:54.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.425+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56417dd9f060 0x56417dda1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.425+0000 7fdc46eb2b80 4 client.16962 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:02:54.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.425+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1170611340 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56417dcdc810 con 0x56417dd9f060 2024-04-26T02:02:54.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.425+0000 7fdc39200700 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56417dd9f060 0x56417dda1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.426+0000 7fdc39200700 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56417dd9f060 0x56417dda1450 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41648333d632910f server_cookie=60445b19ce620cdc in_seq=0 out_seq=0 2024-04-26T02:02:54.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.426+0000 7fdc1b400700 10 client.16962 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:54.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.427+0000 7fdc1b400700 1 -- 172.21.15.6:0/1170611340 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56417dcdc810 con 0x56417dd9f060 2024-04-26T02:02:54.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.427+0000 7fdc1b400700 10 client.16962 handle_command_reply: tid=0 2024-04-26T02:02:54.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.427+0000 7fdc46eb2b80 1 client.16962 shutdown 2024-04-26T02:02:54.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.427+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1170611340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7bdd0 msgr2=0x56417dd86e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:54.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.427+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7bdd0 0x56417dd86e00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fdc28011460 tx=0x7fdc28011770 comp rx=0 tx=0).stop 2024-04-26T02:02:54.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.427+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1170611340 shutdown_connections 2024-04-26T02:02:54.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.427+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56417dd9f060 0x56417dda1450 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.427+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd87560 0x56417dd84640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.427+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7bdd0 0x56417dd86e00 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.427+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/1170611340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd7c6f0 0x56417dd84100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.428+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1170611340 >> 172.21.15.6:0/1170611340 conn(0x56417dcab0e0 msgr2=0x56417dd77ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:54.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.428+0000 7fdc1b400700 0 client.16962 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:54.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.428+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1170611340 shutdown_connections 2024-04-26T02:02:54.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.428+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/1170611340 wait complete. 2024-04-26T02:02:54.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.428+0000 7fdc46eb2b80 20 client.16962 trim_cache size 0 max 16384 2024-04-26T02:02:54.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.455+0000 7fdc46eb2b80 1 Processor -- start 2024-04-26T02:02:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.455+0000 7fdc46eb2b80 1 -- start start 2024-04-26T02:02:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.455+0000 7fdc46eb2b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd7bdd0 0x56417dd78430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.455+0000 7fdc46eb2b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7c6f0 0x56417dd9b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.455+0000 7fdc46eb2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd87560 0x56417dd86040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.455+0000 7fdc46eb2b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdc341ad9a0 con 0x56417dd7c6f0 2024-04-26T02:02:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.456+0000 7fdc46eb2b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdc341ab230 con 0x56417dd7bdd0 2024-04-26T02:02:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.456+0000 7fdc46eb2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdc341a90a0 con 0x56417dd87560 2024-04-26T02:02:54.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.456+0000 7fdc39c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd7bdd0 0x56417dd78430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.456+0000 7fdc39c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd7bdd0 0x56417dd78430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:49306/0 (socket says 172.21.15.6:49306) 2024-04-26T02:02:54.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.456+0000 7fdc39c00700 1 -- 172.21.15.6:0/841340626 learned_addr learned my addr 172.21.15.6:0/841340626 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:54.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.456+0000 7fdc3a600700 1 --2- 172.21.15.6:0/841340626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd87560 0x56417dd86040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.456+0000 7fdc39200700 1 --2- 172.21.15.6:0/841340626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7c6f0 0x56417dd9b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.456+0000 7fdc39c00700 1 -- 172.21.15.6:0/841340626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd87560 msgr2=0x56417dd86040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:54.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.456+0000 7fdc39c00700 1 --2- 172.21.15.6:0/841340626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd87560 0x56417dd86040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.456+0000 7fdc39c00700 1 -- 172.21.15.6:0/841340626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7c6f0 msgr2=0x56417dd9b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:54.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.456+0000 7fdc39c00700 1 --2- 172.21.15.6:0/841340626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7c6f0 0x56417dd9b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.456+0000 7fdc39c00700 1 -- 172.21.15.6:0/841340626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56417dd7d7e0 con 0x56417dd7bdd0 2024-04-26T02:02:54.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.457+0000 7fdc39c00700 1 --2- 172.21.15.6:0/841340626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd7bdd0 0x56417dd78430 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fdc280117b0 tx=0x7fdc28014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3c4fe96e53bf44d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:54.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.457+0000 7fdc26000700 1 -- 172.21.15.6:0/841340626 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2800f440 con 0x56417dd7bdd0 2024-04-26T02:02:54.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.457+0000 7fdc26000700 1 -- 172.21.15.6:0/841340626 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc2009d8e0 con 0x56417dd7bdd0 2024-04-26T02:02:54.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.457+0000 7fdc26000700 1 -- 172.21.15.6:0/841340626 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc28004a90 con 0x56417dd7bdd0 2024-04-26T02:02:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.458+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/841340626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd7bdd0 msgr2=0x56417dd78430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.458+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/841340626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd7bdd0 0x56417dd78430 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fdc280117b0 tx=0x7fdc28014000 comp rx=0 tx=0).stop 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.458+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/841340626 shutdown_connections 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.458+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/841340626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd87560 0x56417dd86040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.458+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/841340626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd7bdd0 0x56417dd78430 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.458+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/841340626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7c6f0 0x56417dd9b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.458+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/841340626 >> 172.21.15.6:0/841340626 conn(0x56417dcab0e0 msgr2=0x56417dd85930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.458+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/841340626 shutdown_connections 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.458+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/841340626 wait complete. 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.458+0000 7fdc46eb2b80 1 -- start start 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.459+0000 7fdc46eb2b80 1 Processor -- start 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.459+0000 7fdc46eb2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd7bdd0 0x56417dd964a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.459+0000 7fdc46eb2b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7c6f0 0x56417dd969e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.459+0000 7fdc46eb2b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 0x56417dd96f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.459+0000 7fdc46eb2b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdc341a96f0 con 0x56417dd7c6f0 2024-04-26T02:02:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.459+0000 7fdc46eb2b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdc341aab60 con 0x56417dd87560 2024-04-26T02:02:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.459+0000 7fdc46eb2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdc341a9dc0 con 0x56417dd7bdd0 2024-04-26T02:02:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.459+0000 7fdc46eb2b80 10 client.0 _ll_register_callbacks cb 0x56417dc678b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.459+0000 7fdc39c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd7bdd0 0x56417dd964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.460+0000 7fdc39200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7c6f0 0x56417dd969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.460+0000 7fdc39c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd7bdd0 0x56417dd964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44826/0 (socket says 172.21.15.6:44826) 2024-04-26T02:02:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.460+0000 7fdc39c00700 1 -- 172.21.15.6:0/490587772 learned_addr learned my addr 172.21.15.6:0/490587772 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:02:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.460+0000 7fdc3a600700 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 0x56417dd96f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.460+0000 7fdc3a600700 1 -- 172.21.15.6:0/490587772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd7bdd0 msgr2=0x56417dd964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.460+0000 7fdc3a600700 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd7bdd0 0x56417dd964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.460+0000 7fdc3a600700 1 -- 172.21.15.6:0/490587772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7c6f0 msgr2=0x56417dd969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.460+0000 7fdc3a600700 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7c6f0 0x56417dd969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.460+0000 7fdc3a600700 1 -- 172.21.15.6:0/490587772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56417de05770 con 0x56417dd87560 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.460+0000 7fdc39200700 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7c6f0 0x56417dd969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.461+0000 7fdc3a600700 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 0x56417dd96f20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c007b70 tx=0x7fdc2c004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e874db657e101c6b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.461+0000 7fdc1b400700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.461+0000 7fdc1b400700 1 -- 172.21.15.6:0/490587772 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c027020 con 0x56417dd87560 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.461+0000 7fdc1b400700 1 -- 172.21.15.6:0/490587772 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7fdc2c014d00 con 0x56417dd87560 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.461+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/490587772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc2009d8e0 con 0x56417dd87560 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.462+0000 7fdc1b400700 1 -- 172.21.15.6:0/490587772 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fdc2c032050 con 0x56417dd87560 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.462+0000 7fdc46eb2b80 10 client.27152 fetch_fsmap learned FSMap version 52 2024-04-26T02:02:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.462+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/490587772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x56417de06ab0 con 0x56417dd87560 2024-04-26T02:02:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.462+0000 7fdc1b400700 1 -- 172.21.15.6:0/490587772 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc2009d8e0 con 0x56417dd87560 2024-04-26T02:02:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.462+0000 7fdc1b400700 1 -- 172.21.15.6:0/490587772 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 52) ==== 8342+0+0 (secure 0 0 0) 0x7fdc2c0223c0 con 0x56417dd87560 2024-04-26T02:02:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.462+0000 7fdc46eb2b80 10 client.27152 fetch_fsmap finished waiting for FSMap version 52 2024-04-26T02:02:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.462+0000 7fdc46eb2b80 10 client.27152 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:02:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.462+0000 7fdc46eb2b80 20 client.27152 populate_metadata read hostname 'smithi006' 2024-04-26T02:02:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.462+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56417de07220 0x56417de09610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.462+0000 7fdc46eb2b80 4 client.27152 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.55, "await_for": 42.0}] 2024-04-26T02:02:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.462+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/490587772 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.55, "await_for": 42.0}) -- 0x56417dcd68b0 con 0x56417de07220 2024-04-26T02:02:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.463+0000 7fdc39c00700 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56417de07220 0x56417de09610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.463+0000 7fdc39c00700 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56417de07220 0x56417de09610 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d739e800d879008c server_cookie=70e972c79939003b in_seq=0 out_seq=0 2024-04-26T02:02:54.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:54.463+0000 7fdc1b400700 10 client.27152 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:54.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:54 smithi029 ceph-mon[32775]: pgmap v539: 97 pgs: 97 active+clean; 796 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.6 MiB/s wr, 630 op/s 2024-04-26T02:02:55.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:54 smithi006 ceph-mon[28419]: pgmap v539: 97 pgs: 97 active+clean; 796 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.6 MiB/s wr, 630 op/s 2024-04-26T02:02:55.113 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78166 31.73 MB/sec execute 78 sec latency 490.851 ms 2024-04-26T02:02:55.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:54 smithi188 ceph-mon[32751]: pgmap v539: 97 pgs: 97 active+clean; 796 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.6 MiB/s wr, 630 op/s 2024-04-26T02:02:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.159+0000 7fdc1b400700 1 -- 172.21.15.6:0/490587772 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56417dcd68b0 con 0x56417de07220 2024-04-26T02:02:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.159+0000 7fdc1b400700 10 client.27152 handle_command_reply: tid=0 2024-04-26T02:02:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.159+0000 7fdc46eb2b80 1 client.27152 shutdown 2024-04-26T02:02:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.160+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/490587772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 msgr2=0x56417dd96f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.160+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 0x56417dd96f20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c007b70 tx=0x7fdc2c004b40 comp rx=0 tx=0).stop 2024-04-26T02:02:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.160+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/490587772 shutdown_connections 2024-04-26T02:02:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.160+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56417dd87560 0x56417dd96f20 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.160+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56417de07220 0x56417de09610 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.160+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417dd7bdd0 0x56417dd964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.160+0000 7fdc46eb2b80 1 --2- 172.21.15.6:0/490587772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56417dd7c6f0 0x56417dd969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.160+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/490587772 >> 172.21.15.6:0/490587772 conn(0x56417dcab0e0 msgr2=0x56417dd861c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.160+0000 7fdc1b400700 0 client.27152 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:02:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.160+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/490587772 shutdown_connections 2024-04-26T02:02:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.160+0000 7fdc46eb2b80 1 -- 172.21.15.6:0/490587772 wait complete. 2024-04-26T02:02:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.160+0000 7fdc46eb2b80 20 client.27152 trim_cache size 0 max 16384 2024-04-26T02:02:55.161 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:02:55.161 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 52, 2024-04-26T02:02:55.161 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T02:02:55.161 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 50, 2024-04-26T02:02:55.161 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T02:02:55.161 INFO:teuthology.orchestra.run.smithi006.stdout: "33ab2393": { 2024-04-26T02:02:55.161 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 50, 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 82.5, 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCED", 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:02:55.162 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:02:55.163 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:02:55.163 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:02:55.163 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:02:55.163 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:02:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.162+0000 7fdc3c600700 1 -- 172.21.15.6:0/3938309856 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdc1c0a3fb0 msgr2=0x7fdc1c0a6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.162+0000 7fdc3c600700 1 --2- 172.21.15.6:0/3938309856 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdc1c0a3fb0 0x7fdc1c0a6470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0093f0 tx=0x7fdc2c009040 comp rx=0 tx=0).stop 2024-04-26T02:02:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.162+0000 7fdc3c600700 1 -- 172.21.15.6:0/3938309856 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc3406b100 msgr2=0x7fdc341995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.162+0000 7fdc3c600700 1 --2- 172.21.15.6:0/3938309856 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc3406b100 0x7fdc341995d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdc20000c00 tx=0x7fdc20009510 comp rx=0 tx=0).stop 2024-04-26T02:02:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.162+0000 7fdc3c600700 1 -- 172.21.15.6:0/3938309856 shutdown_connections 2024-04-26T02:02:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.162+0000 7fdc3c600700 1 --2- 172.21.15.6:0/3938309856 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdc1c0a3fb0 0x7fdc1c0a6470 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.162+0000 7fdc3c600700 1 --2- 172.21.15.6:0/3938309856 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc34108a60 0x7fdc3419d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.162+0000 7fdc3c600700 1 --2- 172.21.15.6:0/3938309856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc3406a740 0x7fdc34199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.162+0000 7fdc3c600700 1 --2- 172.21.15.6:0/3938309856 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc3406b100 0x7fdc341995d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.162+0000 7fdc3c600700 1 -- 172.21.15.6:0/3938309856 >> 172.21.15.6:0/3938309856 conn(0x7fdc340788e0 msgr2=0x7fdc34100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.162+0000 7fdc3c600700 1 -- 172.21.15.6:0/3938309856 shutdown_connections 2024-04-26T02:02:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:02:55.162+0000 7fdc3c600700 1 -- 172.21.15.6:0/3938309856 wait complete. 2024-04-26T02:02:55.923 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '33ab2393', quiesce took 9.0 seconds. Will release after: 22.55 2024-04-26T02:02:56.113 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 31.33 MB/sec execute 79 sec latency 1491.025 ms 2024-04-26T02:02:56.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:56 smithi029 ceph-mon[32775]: pgmap v540: 97 pgs: 97 active+clean; 796 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.6 MiB/s wr, 630 op/s 2024-04-26T02:02:57.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:56 smithi006 ceph-mon[28419]: pgmap v540: 97 pgs: 97 active+clean; 796 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.6 MiB/s wr, 630 op/s 2024-04-26T02:02:57.113 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 30.94 MB/sec execute 80 sec latency 2491.189 ms 2024-04-26T02:02:57.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:56 smithi188 ceph-mon[32751]: pgmap v540: 97 pgs: 97 active+clean; 796 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.6 MiB/s wr, 630 op/s 2024-04-26T02:02:58.114 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 30.56 MB/sec execute 81 sec latency 3491.284 ms 2024-04-26T02:02:58.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:02:58 smithi029 ceph-mon[32775]: pgmap v541: 97 pgs: 97 active+clean; 814 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 866 op/s 2024-04-26T02:02:59.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:02:58 smithi006 ceph-mon[28419]: pgmap v541: 97 pgs: 97 active+clean; 814 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 866 op/s 2024-04-26T02:02:59.113 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 30.18 MB/sec execute 82 sec latency 4491.435 ms 2024-04-26T02:02:59.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:02:58 smithi188 ceph-mon[32751]: pgmap v541: 97 pgs: 97 active+clean; 814 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 866 op/s 2024-04-26T02:03:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.106+0000 7f9960000640 1 -- 172.21.15.6:0/4205377589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99581018d0 msgr2=0x7f9958105d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.106+0000 7f9960000640 1 --2- 172.21.15.6:0/4205377589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99581018d0 0x7f9958105d90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9944009930 tx=0x7f994402a4d0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.106+0000 7f9960000640 1 -- 172.21.15.6:0/4205377589 shutdown_connections 2024-04-26T02:03:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.106+0000 7f9960000640 1 --2- 172.21.15.6:0/4205377589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99581063b0 0x7f9958106860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.106+0000 7f9960000640 1 --2- 172.21.15.6:0/4205377589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99581018d0 0x7f9958105d90 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.106+0000 7f9960000640 1 --2- 172.21.15.6:0/4205377589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9958100fb0 0x7f9958101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.106+0000 7f9960000640 1 -- 172.21.15.6:0/4205377589 >> 172.21.15.6:0/4205377589 conn(0x7f9958077f70 msgr2=0x7f9958078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.106+0000 7f9960000640 1 -- 172.21.15.6:0/4205377589 shutdown_connections 2024-04-26T02:03:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.106+0000 7f9960000640 1 -- 172.21.15.6:0/4205377589 wait complete. 2024-04-26T02:03:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.107+0000 7f9960000640 1 Processor -- start 2024-04-26T02:03:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.107+0000 7f9960000640 1 -- start start 2024-04-26T02:03:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.108+0000 7f9960000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9958100fb0 0x7f995819ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.108+0000 7f9960000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99581018d0 0x7f995819f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.108+0000 7f9960000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99581063b0 0x7f99581a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.108+0000 7f9960000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f99581187e0 con 0x7f99581063b0 2024-04-26T02:03:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.108+0000 7f9960000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9958118660 con 0x7f99581018d0 2024-04-26T02:03:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.108+0000 7f9960000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9958118960 con 0x7f9958100fb0 2024-04-26T02:03:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.108+0000 7f995cc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99581018d0 0x7f995819f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.108+0000 7f995cc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99581018d0 0x7f995819f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52004/0 (socket says 172.21.15.6:52004) 2024-04-26T02:03:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.108+0000 7f995cc00640 1 -- 172.21.15.6:0/3911214238 learned_addr learned my addr 172.21.15.6:0/3911214238 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.108+0000 7f995d600640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9958100fb0 0x7f995819ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.108+0000 7f995e000640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99581063b0 0x7f99581a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.109+0000 7f995cc00640 1 -- 172.21.15.6:0/3911214238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9958100fb0 msgr2=0x7f995819ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.109+0000 7f995cc00640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9958100fb0 0x7f995819ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.109+0000 7f995cc00640 1 -- 172.21.15.6:0/3911214238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99581063b0 msgr2=0x7f99581a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.109+0000 7f995cc00640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99581063b0 0x7f99581a3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.109+0000 7f995cc00640 1 -- 172.21.15.6:0/3911214238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99581a3e30 con 0x7f99581018d0 2024-04-26T02:03:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.109+0000 7f995e000640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99581063b0 0x7f99581a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.109+0000 7f995cc00640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99581018d0 0x7f995819f380 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9944005ab0 tx=0x7f9944003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.109+0000 7f9956000640 1 -- 172.21.15.6:0/3911214238 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994403f070 con 0x7f99581018d0 2024-04-26T02:03:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.109+0000 7f9960000640 1 -- 172.21.15.6:0/3911214238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99581a40c0 con 0x7f99581018d0 2024-04-26T02:03:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.109+0000 7f9956000640 1 -- 172.21.15.6:0/3911214238 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9944031b50 con 0x7f99581018d0 2024-04-26T02:03:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.109+0000 7f9956000640 1 -- 172.21.15.6:0/3911214238 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944035030 con 0x7f99581018d0 2024-04-26T02:03:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.109+0000 7f9960000640 1 -- 172.21.15.6:0/3911214238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99581ab9b0 con 0x7f99581018d0 2024-04-26T02:03:00.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.112+0000 7f9962e69740 1 Processor -- start 2024-04-26T02:03:00.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.112+0000 7f9956000640 1 -- 172.21.15.6:0/3911214238 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f9944042070 con 0x7f99581018d0 2024-04-26T02:03:00.111 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.113+0000 7f9962e69740 1 -- start start 2024-04-26T02:03:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.113+0000 7f9962e69740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba4820 0x558509ba4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.113+0000 7f9962e69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba5140 0x558509baf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.113+0000 7f9962e69740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509bafa00 0x558509bb1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.113+0000 7f9962e69740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x558509b0a900 con 0x558509bafa00 2024-04-26T02:03:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.113+0000 7f9962e69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558509ab0ef0 con 0x558509ba5140 2024-04-26T02:03:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.113+0000 7f9962e69740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558509b068c0 con 0x558509ba4820 2024-04-26T02:03:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f995e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba4820 0x558509ba4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f995cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509bafa00 0x558509bb1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f995e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba4820 0x558509ba4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56748/0 (socket says 172.21.15.6:56748) 2024-04-26T02:03:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f995e000640 1 -- 172.21.15.6:0/3036900741 learned_addr learned my addr 172.21.15.6:0/3036900741 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f995d600640 1 --2- 172.21.15.6:0/3036900741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba5140 0x558509baf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f995e000640 1 -- 172.21.15.6:0/3036900741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba5140 msgr2=0x558509baf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f995e000640 1 --2- 172.21.15.6:0/3036900741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba5140 0x558509baf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f995e000640 1 -- 172.21.15.6:0/3036900741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509bafa00 msgr2=0x558509bb1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f995e000640 1 --2- 172.21.15.6:0/3036900741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509bafa00 0x558509bb1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f995e000640 1 -- 172.21.15.6:0/3036900741 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55850981d210 con 0x558509ba4820 2024-04-26T02:03:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f995d600640 1 --2- 172.21.15.6:0/3036900741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba5140 0x558509baf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f9956000640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f99400a3d90 0x7f99400a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.114+0000 7f995e000640 1 --2- 172.21.15.6:0/3036900741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba4820 0x558509ba4c00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9950005f40 tx=0x7f9950010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a318387a8a062e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.115+0000 7f9956000640 1 -- 172.21.15.6:0/3911214238 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7f99440f0dd0 con 0x7f99581018d0 2024-04-26T02:03:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.115+0000 7f994a600640 1 -- 172.21.15.6:0/3036900741 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950011700 con 0x558509ba4820 2024-04-26T02:03:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.115+0000 7f994a600640 1 -- 172.21.15.6:0/3036900741 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9950011d50 con 0x558509ba4820 2024-04-26T02:03:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.115+0000 7f994a600640 1 -- 172.21.15.6:0/3036900741 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995000e3c0 con 0x558509ba4820 2024-04-26T02:03:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.115+0000 7f995d600640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f99400a3d90 0x7f99400a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.115 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 29.82 MB/sec execute 83 sec latency 5491.513 ms 2024-04-26T02:03:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.115+0000 7f9962e69740 1 -- 172.21.15.6:0/3036900741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba4820 msgr2=0x558509ba4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.115+0000 7f9962e69740 1 --2- 172.21.15.6:0/3036900741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba4820 0x558509ba4c00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9950005f40 tx=0x7f9950010860 comp rx=0 tx=0).stop 2024-04-26T02:03:00.118 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.116+0000 7f995d600640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f99400a3d90 0x7f99400a6250 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x558509ba5dc0 tx=0x7f994c005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.116+0000 7f9962e69740 1 -- 172.21.15.6:0/3036900741 shutdown_connections 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.116+0000 7f9962e69740 1 --2- 172.21.15.6:0/3036900741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509bafa00 0x558509bb1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.116+0000 7f9962e69740 1 --2- 172.21.15.6:0/3036900741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba5140 0x558509baf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.116+0000 7f9962e69740 1 --2- 172.21.15.6:0/3036900741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba4820 0x558509ba4c00 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.116+0000 7f9962e69740 1 -- 172.21.15.6:0/3036900741 >> 172.21.15.6:0/3036900741 conn(0x558509ba0d30 msgr2=0x558509ba1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.116+0000 7f9962e69740 1 -- 172.21.15.6:0/3036900741 shutdown_connections 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.116+0000 7f9962e69740 1 -- 172.21.15.6:0/3036900741 wait complete. 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.116+0000 7f9962e69740 1 -- start start 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f9962e69740 1 Processor -- start 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f9962e69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba4820 0x558509baca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f9962e69740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba5140 0x558509bacf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f9962e69740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509bafa00 0x558509bad4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.119 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f9962e69740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x558509bb2630 con 0x558509bafa00 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f9962e69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558509ac4310 con 0x558509ba4820 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f9962e69740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558509bb27b0 con 0x558509ba5140 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f9962e69740 10 client.0 _ll_register_callbacks cb 0x558509a92cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f995e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509bafa00 0x558509bad4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f995d600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba4820 0x558509baca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f995cc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba5140 0x558509bacf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f995d600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba4820 0x558509baca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52030/0 (socket says 172.21.15.6:52030) 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.117+0000 7f995d600640 1 -- 172.21.15.6:0/3757811317 learned_addr learned my addr 172.21.15.6:0/3757811317 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.118+0000 7f995cc00640 1 -- 172.21.15.6:0/3757811317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba4820 msgr2=0x558509baca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.118+0000 7f995cc00640 1 --2- 172.21.15.6:0/3757811317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba4820 0x558509baca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.118+0000 7f995cc00640 1 -- 172.21.15.6:0/3757811317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509bafa00 msgr2=0x558509bad4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.118+0000 7f995cc00640 1 --2- 172.21.15.6:0/3757811317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509bafa00 0x558509bad4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.118+0000 7f995cc00640 1 -- 172.21.15.6:0/3757811317 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585098e9620 con 0x558509ba5140 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.118+0000 7f995cc00640 1 --2- 172.21.15.6:0/3757811317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba5140 0x558509bacf60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f99440ee4d0 tx=0x7f9944031f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9352d05976906908 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.118+0000 7f993ea00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.118+0000 7f993ea00640 1 -- 172.21.15.6:0/3757811317 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944049050 con 0x558509ba5140 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.118+0000 7f9962e69740 1 -- 172.21.15.6:0/3757811317 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9950011d50 con 0x558509ba5140 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.118+0000 7f993ea00640 1 -- 172.21.15.6:0/3757811317 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7f99440ee750 con 0x558509ba5140 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.119+0000 7f993ea00640 1 -- 172.21.15.6:0/3757811317 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f99440b8fc0 con 0x558509ba5140 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.119+0000 7f9962e69740 10 client.27220 fetch_fsmap learned FSMap version 52 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.119+0000 7f9962e69740 1 -- 172.21.15.6:0/3757811317 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x558509bb26e0 con 0x558509ba5140 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.119+0000 7f993ea00640 1 -- 172.21.15.6:0/3757811317 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9950011d50 con 0x558509ba5140 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.119+0000 7f993ea00640 1 -- 172.21.15.6:0/3757811317 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 52) ==== 8342+0+0 (secure 0 0 0) 0x7f99440b9390 con 0x558509ba5140 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.119+0000 7f9962e69740 10 client.27220 fetch_fsmap finished waiting for FSMap version 52 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.119+0000 7f9962e69740 10 client.27220 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.119+0000 7f9962e69740 20 client.27220 populate_metadata read hostname 'smithi006' 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.119+0000 7f9962e69740 1 --2- 172.21.15.6:0/3757811317 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558509bbfdb0 0x558509bc21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.119+0000 7f9962e69740 4 client.27220 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.119+0000 7f9962e69740 1 -- 172.21.15.6:0/3757811317 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558509b1e8b0 con 0x558509bbfdb0 2024-04-26T02:03:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.119+0000 7f995d600640 1 --2- 172.21.15.6:0/3757811317 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558509bbfdb0 0x558509bc21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.122 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.120+0000 7f995d600640 1 --2- 172.21.15.6:0/3757811317 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558509bbfdb0 0x558509bc21c0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c578a707d2801ee server_cookie=72b92189e5af0dd0 in_seq=0 out_seq=0 2024-04-26T02:03:00.122 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.120+0000 7f993ea00640 10 client.27220 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f993ea00640 1 -- 172.21.15.6:0/3757811317 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558509b1e8b0 con 0x558509bbfdb0 2024-04-26T02:03:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f993ea00640 10 client.27220 handle_command_reply: tid=0 2024-04-26T02:03:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f9962e69740 1 client.27220 shutdown 2024-04-26T02:03:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f9962e69740 1 -- 172.21.15.6:0/3757811317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba5140 msgr2=0x558509bacf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f9962e69740 1 --2- 172.21.15.6:0/3757811317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba5140 0x558509bacf60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f99440ee4d0 tx=0x7f9944031f50 comp rx=0 tx=0).stop 2024-04-26T02:03:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f9962e69740 1 -- 172.21.15.6:0/3757811317 shutdown_connections 2024-04-26T02:03:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f9962e69740 1 --2- 172.21.15.6:0/3757811317 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558509bbfdb0 0x558509bc21c0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f9962e69740 1 --2- 172.21.15.6:0/3757811317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509bafa00 0x558509bad4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f9962e69740 1 --2- 172.21.15.6:0/3757811317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba5140 0x558509bacf60 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f9962e69740 1 --2- 172.21.15.6:0/3757811317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba4820 0x558509baca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f9962e69740 1 -- 172.21.15.6:0/3757811317 >> 172.21.15.6:0/3757811317 conn(0x558509ba0d30 msgr2=0x558509ba6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f993ea00640 0 client.27220 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f9962e69740 1 -- 172.21.15.6:0/3757811317 shutdown_connections 2024-04-26T02:03:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f9962e69740 1 -- 172.21.15.6:0/3757811317 wait complete. 2024-04-26T02:03:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.121+0000 7f9962e69740 20 client.27220 trim_cache size 0 max 16384 2024-04-26T02:03:00.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f9962e69740 1 Processor -- start 2024-04-26T02:03:00.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f9962e69740 1 -- start start 2024-04-26T02:03:00.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f9962e69740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba4820 0x558509bbb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.136 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f9962e69740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509ba5140 0x558509bc7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f9962e69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509bafa00 0x558509bc7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f9962e69740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f99581b6c60 con 0x558509ba5140 2024-04-26T02:03:00.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f9962e69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f99581b4530 con 0x558509bafa00 2024-04-26T02:03:00.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f9962e69740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f99581b23a0 con 0x558509ba4820 2024-04-26T02:03:00.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f995cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509ba5140 0x558509bc7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f995cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509ba5140 0x558509bc7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56122/0 (socket says 172.21.15.6:56122) 2024-04-26T02:03:00.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f995cc00640 1 -- 172.21.15.6:0/1948853983 learned_addr learned my addr 172.21.15.6:0/1948853983 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:00.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f995d600640 1 --2- 172.21.15.6:0/1948853983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba4820 0x558509bbb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.137 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.138+0000 7f995e000640 1 --2- 172.21.15.6:0/1948853983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509bafa00 0x558509bc7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.139 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.139+0000 7f995cc00640 1 -- 172.21.15.6:0/1948853983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba4820 msgr2=0x558509bbb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:00.139 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.139+0000 7f995cc00640 1 --2- 172.21.15.6:0/1948853983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba4820 0x558509bbb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.139 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.139+0000 7f995cc00640 1 -- 172.21.15.6:0/1948853983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509bafa00 msgr2=0x558509bc7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:00.139 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.139+0000 7f995cc00640 1 --2- 172.21.15.6:0/1948853983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509bafa00 0x558509bc7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.139+0000 7f995cc00640 1 -- 172.21.15.6:0/1948853983 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558509ba77d0 con 0x558509ba5140 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.139+0000 7f995cc00640 1 --2- 172.21.15.6:0/1948853983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509ba5140 0x558509bc7aa0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f99440eead0 tx=0x7f994402a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc91fbe6d61d681a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.139+0000 7f9949c00640 1 -- 172.21.15.6:0/1948853983 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944049030 con 0x558509ba5140 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.139+0000 7f9949c00640 1 -- 172.21.15.6:0/1948853983 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99440bd420 con 0x558509ba5140 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.139+0000 7f9949c00640 1 -- 172.21.15.6:0/1948853983 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99440bc680 con 0x558509ba5140 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.140+0000 7f9962e69740 1 -- 172.21.15.6:0/1948853983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509ba5140 msgr2=0x558509bc7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.140+0000 7f9962e69740 1 --2- 172.21.15.6:0/1948853983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509ba5140 0x558509bc7aa0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f99440eead0 tx=0x7f994402a4d0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.140+0000 7f9962e69740 1 -- 172.21.15.6:0/1948853983 shutdown_connections 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.140+0000 7f9962e69740 1 --2- 172.21.15.6:0/1948853983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509ba5140 0x558509bc7aa0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.140+0000 7f9962e69740 1 --2- 172.21.15.6:0/1948853983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509bafa00 0x558509bc7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.140+0000 7f9962e69740 1 --2- 172.21.15.6:0/1948853983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509ba4820 0x558509bbb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.140+0000 7f9962e69740 1 -- 172.21.15.6:0/1948853983 >> 172.21.15.6:0/1948853983 conn(0x558509ba0d30 msgr2=0x558509ba1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:00.140 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.140+0000 7f9962e69740 1 -- 172.21.15.6:0/1948853983 shutdown_connections 2024-04-26T02:03:00.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.140+0000 7f9962e69740 1 -- 172.21.15.6:0/1948853983 wait complete. 2024-04-26T02:03:00.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.140+0000 7f9962e69740 1 -- start start 2024-04-26T02:03:00.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f9962e69740 1 Processor -- start 2024-04-26T02:03:00.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f9962e69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba4820 0x558509bb7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f9962e69740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509ba5140 0x558509bb51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f9962e69740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509bafa00 0x558509bb5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f9962e69740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f99581b29f0 con 0x558509ba5140 2024-04-26T02:03:00.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f9962e69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f99581b3e60 con 0x558509ba4820 2024-04-26T02:03:00.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f9962e69740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f99581b30c0 con 0x558509bafa00 2024-04-26T02:03:00.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f9962e69740 10 client.0 _ll_register_callbacks cb 0x7f9944005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:00.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f995e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509bafa00 0x558509bb5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.141 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f995e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509bafa00 0x558509bb5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:56772/0 (socket says 172.21.15.6:56772) 2024-04-26T02:03:00.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f995e000640 1 -- 172.21.15.6:0/3154595156 learned_addr learned my addr 172.21.15.6:0/3154595156 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:00.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f995cc00640 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509ba5140 0x558509bb51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.142 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.141+0000 7f995d600640 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba4820 0x558509bb7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.143 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.142+0000 7f995e000640 1 -- 172.21.15.6:0/3154595156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba4820 msgr2=0x558509bb7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:00.143 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.142+0000 7f995e000640 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba4820 0x558509bb7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.143 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.142+0000 7f995e000640 1 -- 172.21.15.6:0/3154595156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509ba5140 msgr2=0x558509bb51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:00.143 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.142+0000 7f995e000640 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509ba5140 0x558509bb51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.143 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.142+0000 7f995e000640 1 -- 172.21.15.6:0/3154595156 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558509bb5fa0 con 0x558509bafa00 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.142+0000 7f995e000640 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509bafa00 0x558509bb5770 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f995001ba30 tx=0x7f995001b660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67e5138dc13ac237 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.142+0000 7f993ea00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.142+0000 7f993ea00640 1 -- 172.21.15.6:0/3154595156 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995002d060 con 0x558509bafa00 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.142+0000 7f9962e69740 1 -- 172.21.15.6:0/3154595156 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99440bd420 con 0x558509bafa00 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.142+0000 7f993ea00640 1 -- 172.21.15.6:0/3154595156 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8600+0+0 (secure 0 0 0) 0x7f9950004c00 con 0x558509bafa00 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.142+0000 7f993ea00640 1 -- 172.21.15.6:0/3154595156 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f9950009c00 con 0x558509bafa00 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.143+0000 7f9962e69740 10 client.27226 fetch_fsmap learned FSMap version 52 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.143+0000 7f9962e69740 1 -- 172.21.15.6:0/3154595156 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x558509c263d0 con 0x558509bafa00 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.143+0000 7f993ea00640 1 -- 172.21.15.6:0/3154595156 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99440bd420 con 0x558509bafa00 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.143+0000 7f993ea00640 1 -- 172.21.15.6:0/3154595156 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 52) ==== 8342+0+0 (secure 0 0 0) 0x7f9950005040 con 0x558509bafa00 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.143+0000 7f9962e69740 10 client.27226 fetch_fsmap finished waiting for FSMap version 52 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.143+0000 7f9962e69740 10 client.27226 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.143+0000 7f9962e69740 20 client.27226 populate_metadata read hostname 'smithi006' 2024-04-26T02:03:00.144 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.143+0000 7f9962e69740 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558509c267c0 0x558509c28bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.143+0000 7f9962e69740 4 client.27226 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:03:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.143+0000 7f9962e69740 1 -- 172.21.15.6:0/3154595156 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558509b18320 con 0x558509c267c0 2024-04-26T02:03:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.143+0000 7f995d600640 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558509c267c0 0x558509c28bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.144+0000 7f995d600640 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558509c267c0 0x558509c28bb0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cba46146fb4ed4e6 server_cookie=6396ced87572427b in_seq=0 out_seq=0 2024-04-26T02:03:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.144+0000 7f993ea00640 10 client.27226 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.321+0000 7f993ea00640 1 -- 172.21.15.6:0/3154595156 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558509b18320 con 0x558509c267c0 2024-04-26T02:03:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.321+0000 7f993ea00640 10 client.27226 handle_command_reply: tid=0 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.321+0000 7f9962e69740 1 client.27226 shutdown 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.322+0000 7f9962e69740 1 -- 172.21.15.6:0/3154595156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509bafa00 msgr2=0x558509bb5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.322+0000 7f9962e69740 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509bafa00 0x558509bb5770 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f995001ba30 tx=0x7f995001b660 comp rx=0 tx=0).stop 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.322+0000 7f9962e69740 1 -- 172.21.15.6:0/3154595156 shutdown_connections 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.322+0000 7f9962e69740 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558509c267c0 0x558509c28bb0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.322+0000 7f9962e69740 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558509ba5140 0x558509bb51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.322+0000 7f9962e69740 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558509bafa00 0x558509bb5770 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.322+0000 7f993ea00640 0 client.27226 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.322+0000 7f9962e69740 1 --2- 172.21.15.6:0/3154595156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558509ba4820 0x558509bb7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.322+0000 7f9962e69740 1 -- 172.21.15.6:0/3154595156 >> 172.21.15.6:0/3154595156 conn(0x558509ba0d30 msgr2=0x558509b9f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.322+0000 7f9962e69740 1 -- 172.21.15.6:0/3154595156 shutdown_connections 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.322+0000 7f9962e69740 1 -- 172.21.15.6:0/3154595156 wait complete. 2024-04-26T02:03:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.322+0000 7f9962e69740 20 client.27226 trim_cache size 0 max 16384 2024-04-26T02:03:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:03:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:03:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:03:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:03:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.323+0000 7f9960000640 1 -- 172.21.15.6:0/3911214238 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f99400a3d90 msgr2=0x7f99400a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.323+0000 7f9960000640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f99400a3d90 0x7f99400a6250 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x558509ba5dc0 tx=0x7f994c005f90 comp rx=0 tx=0).stop 2024-04-26T02:03:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.323+0000 7f9960000640 1 -- 172.21.15.6:0/3911214238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99581018d0 msgr2=0x7f995819f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.323+0000 7f9960000640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99581018d0 0x7f995819f380 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9944005ab0 tx=0x7f9944003b90 comp rx=0 tx=0).stop 2024-04-26T02:03:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.323+0000 7f9960000640 1 -- 172.21.15.6:0/3911214238 shutdown_connections 2024-04-26T02:03:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.323+0000 7f9960000640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f99400a3d90 0x7f99400a6250 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.323+0000 7f9960000640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99581063b0 0x7f99581a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.323+0000 7f9960000640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99581018d0 0x7f995819f380 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.323+0000 7f9960000640 1 --2- 172.21.15.6:0/3911214238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9958100fb0 0x7f995819ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.323+0000 7f9960000640 1 -- 172.21.15.6:0/3911214238 >> 172.21.15.6:0/3911214238 conn(0x7f9958077f70 msgr2=0x7f9958103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.324+0000 7f9960000640 1 -- 172.21.15.6:0/3911214238 shutdown_connections 2024-04-26T02:03:00.324 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:00.324+0000 7f9960000640 1 -- 172.21.15.6:0/3911214238 wait complete. 2024-04-26T02:03:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:00 smithi029 ceph-mon[32775]: pgmap v542: 97 pgs: 97 active+clean; 814 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.3 MiB/s wr, 596 op/s 2024-04-26T02:03:00.942 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:00 smithi188 ceph-mon[32751]: pgmap v542: 97 pgs: 97 active+clean; 814 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.3 MiB/s wr, 596 op/s 2024-04-26T02:03:00.996 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:00 smithi006 ceph-mon[28419]: pgmap v542: 97 pgs: 97 active+clean; 814 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.3 MiB/s wr, 596 op/s 2024-04-26T02:03:01.114 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 29.46 MB/sec execute 84 sec latency 6491.636 ms 2024-04-26T02:03:02.114 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 29.12 MB/sec execute 85 sec latency 7491.817 ms 2024-04-26T02:03:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:02 smithi029 ceph-mon[32775]: pgmap v543: 97 pgs: 97 active+clean; 808 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 765 op/s 2024-04-26T02:03:03.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:02 smithi006 ceph-mon[28419]: pgmap v543: 97 pgs: 97 active+clean; 808 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 765 op/s 2024-04-26T02:03:03.114 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 28.78 MB/sec execute 86 sec latency 8491.991 ms 2024-04-26T02:03:03.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:02 smithi188 ceph-mon[32751]: pgmap v543: 97 pgs: 97 active+clean; 808 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 765 op/s 2024-04-26T02:03:04.115 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 28.45 MB/sec execute 87 sec latency 9492.157 ms 2024-04-26T02:03:05.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:04 smithi006 ceph-mon[28419]: pgmap v544: 97 pgs: 97 active+clean; 808 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.9 MiB/s wr, 405 op/s 2024-04-26T02:03:05.114 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 28.13 MB/sec execute 88 sec latency 10492.278 ms 2024-04-26T02:03:05.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:04 smithi188 ceph-mon[32751]: pgmap v544: 97 pgs: 97 active+clean; 808 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.9 MiB/s wr, 405 op/s 2024-04-26T02:03:05.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:04 smithi029 ceph-mon[32775]: pgmap v544: 97 pgs: 97 active+clean; 808 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.9 MiB/s wr, 405 op/s 2024-04-26T02:03:06.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:05 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:06.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:05 smithi006 ceph-mon[28419]: osdmap e124: 12 total, 12 up, 12 in 2024-04-26T02:03:06.114 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 27.81 MB/sec execute 89 sec latency 11492.384 ms 2024-04-26T02:03:06.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:05 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:06.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:05 smithi188 ceph-mon[32751]: osdmap e124: 12 total, 12 up, 12 in 2024-04-26T02:03:06.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:05 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:06.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:05 smithi029 ceph-mon[32775]: osdmap e124: 12 total, 12 up, 12 in 2024-04-26T02:03:07.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:06 smithi006 ceph-mon[28419]: pgmap v546: 97 pgs: 97 active+clean; 808 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.2 MiB/s wr, 486 op/s 2024-04-26T02:03:07.114 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 27.50 MB/sec execute 90 sec latency 12492.526 ms 2024-04-26T02:03:07.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:06 smithi188 ceph-mon[32751]: pgmap v546: 97 pgs: 97 active+clean; 808 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.2 MiB/s wr, 486 op/s 2024-04-26T02:03:07.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:06 smithi029 ceph-mon[32775]: pgmap v546: 97 pgs: 97 active+clean; 808 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.2 MiB/s wr, 486 op/s 2024-04-26T02:03:08.115 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 27.20 MB/sec execute 91 sec latency 13492.661 ms 2024-04-26T02:03:09.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:08 smithi006 ceph-mon[28419]: pgmap v547: 97 pgs: 97 active+clean; 822 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 5.8 MiB/s wr, 279 op/s 2024-04-26T02:03:09.115 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 26.90 MB/sec execute 92 sec latency 14492.797 ms 2024-04-26T02:03:09.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:08 smithi188 ceph-mon[32751]: pgmap v547: 97 pgs: 97 active+clean; 822 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 5.8 MiB/s wr, 279 op/s 2024-04-26T02:03:09.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:08 smithi029 ceph-mon[32775]: pgmap v547: 97 pgs: 97 active+clean; 822 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 5.8 MiB/s wr, 279 op/s 2024-04-26T02:03:10.115 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 26.61 MB/sec execute 93 sec latency 15492.968 ms 2024-04-26T02:03:10.993 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:10 smithi029 ceph-mon[32775]: pgmap v548: 97 pgs: 97 active+clean; 822 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 5.8 MiB/s wr, 279 op/s 2024-04-26T02:03:10.994 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:10 smithi029 ceph-mon[32775]: osdmap e125: 12 total, 12 up, 12 in 2024-04-26T02:03:11.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:10 smithi006 ceph-mon[28419]: pgmap v548: 97 pgs: 97 active+clean; 822 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 5.8 MiB/s wr, 279 op/s 2024-04-26T02:03:11.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:10 smithi006 ceph-mon[28419]: osdmap e125: 12 total, 12 up, 12 in 2024-04-26T02:03:11.115 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 26.33 MB/sec execute 94 sec latency 16493.076 ms 2024-04-26T02:03:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:10 smithi188 ceph-mon[32751]: pgmap v548: 97 pgs: 97 active+clean; 822 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 5.8 MiB/s wr, 279 op/s 2024-04-26T02:03:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:10 smithi188 ceph-mon[32751]: osdmap e125: 12 total, 12 up, 12 in 2024-04-26T02:03:12.115 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 26.05 MB/sec execute 95 sec latency 17493.214 ms 2024-04-26T02:03:13.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:12 smithi006 ceph-mon[28419]: pgmap v550: 97 pgs: 97 active+clean; 822 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 7.2 MiB/s wr, 226 op/s 2024-04-26T02:03:13.115 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 25.78 MB/sec execute 96 sec latency 18493.372 ms 2024-04-26T02:03:13.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:12 smithi188 ceph-mon[32751]: pgmap v550: 97 pgs: 97 active+clean; 822 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 7.2 MiB/s wr, 226 op/s 2024-04-26T02:03:13.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:12 smithi029 ceph-mon[32775]: pgmap v550: 97 pgs: 97 active+clean; 822 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 7.2 MiB/s wr, 226 op/s 2024-04-26T02:03:13.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.473+0000 7ff8aa800640 1 -- 172.21.15.6:0/1222510196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a4108f60 msgr2=0x7ff8a4115420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:13.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.473+0000 7ff8aa800640 1 --2- 172.21.15.6:0/1222510196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a4108f60 0x7ff8a4115420 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009930 tx=0x7ff89c02a440 comp rx=0 tx=0).stop 2024-04-26T02:03:13.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.473+0000 7ff8aa800640 1 -- 172.21.15.6:0/1222510196 shutdown_connections 2024-04-26T02:03:13.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.473+0000 7ff8aa800640 1 --2- 172.21.15.6:0/1222510196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a4108f60 0x7ff8a4115420 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.473+0000 7ff8aa800640 1 --2- 172.21.15.6:0/1222510196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8a41085a0 0x7ff8a4108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.471 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.473+0000 7ff8aa800640 1 --2- 172.21.15.6:0/1222510196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8a410e660 0x7ff8a410ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.473+0000 7ff8aa800640 1 -- 172.21.15.6:0/1222510196 >> 172.21.15.6:0/1222510196 conn(0x7ff8a4076460 msgr2=0x7ff8a4078880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.473+0000 7ff8aa800640 1 -- 172.21.15.6:0/1222510196 shutdown_connections 2024-04-26T02:03:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.473+0000 7ff8aa800640 1 -- 172.21.15.6:0/1222510196 wait complete. 2024-04-26T02:03:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.474+0000 7ff8aa800640 1 Processor -- start 2024-04-26T02:03:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.474+0000 7ff8aa800640 1 -- start start 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.475+0000 7ff8aa800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8a41085a0 0x7ff8a41a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.475+0000 7ff8aa800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8a4108f60 0x7ff8a41a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.475+0000 7ff8aa800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a410e660 0x7ff8a419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.475+0000 7ff8aa800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a4118810 con 0x7ff8a41085a0 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.475+0000 7ff8aa800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a4118690 con 0x7ff8a410e660 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.475+0000 7ff8aa800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a4118990 con 0x7ff8a4108f60 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.475+0000 7ff8a3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8a41085a0 0x7ff8a41a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.475+0000 7ff8a3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8a41085a0 0x7ff8a41a4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45294/0 (socket says 172.21.15.6:45294) 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.475+0000 7ff8a3400640 1 -- 172.21.15.6:0/2102901887 learned_addr learned my addr 172.21.15.6:0/2102901887 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.475+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a410e660 0x7ff8a419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.476+0000 7ff8a2a00640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8a4108f60 0x7ff8a41a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.476+0000 7ff8a3400640 1 -- 172.21.15.6:0/2102901887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8a4108f60 msgr2=0x7ff8a41a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.476+0000 7ff8a3400640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8a4108f60 0x7ff8a41a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.476+0000 7ff8a3400640 1 -- 172.21.15.6:0/2102901887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a410e660 msgr2=0x7ff8a419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.476+0000 7ff8a3400640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a410e660 0x7ff8a419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.476+0000 7ff8a3400640 1 -- 172.21.15.6:0/2102901887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a419f250 con 0x7ff8a41085a0 2024-04-26T02:03:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.476+0000 7ff8a3400640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8a41085a0 0x7ff8a41a4920 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff894009d50 tx=0x7ff89400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.476+0000 7ff89be00640 1 -- 172.21.15.6:0/2102901887 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff894003b70 con 0x7ff8a41085a0 2024-04-26T02:03:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.476+0000 7ff89be00640 1 -- 172.21.15.6:0/2102901887 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff894003d10 con 0x7ff8a41085a0 2024-04-26T02:03:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.476+0000 7ff89be00640 1 -- 172.21.15.6:0/2102901887 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8940108a0 con 0x7ff8a41085a0 2024-04-26T02:03:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.476+0000 7ff8aa800640 1 -- 172.21.15.6:0/2102901887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a419f540 con 0x7ff8a41085a0 2024-04-26T02:03:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.477+0000 7ff8aa800640 1 -- 172.21.15.6:0/2102901887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a41ab7e0 con 0x7ff8a41085a0 2024-04-26T02:03:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.480+0000 7ff8ad582740 1 Processor -- start 2024-04-26T02:03:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.480+0000 7ff89be00640 1 -- 172.21.15.6:0/2102901887 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7ff894004000 con 0x7ff8a41085a0 2024-04-26T02:03:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.480+0000 7ff8ad582740 1 -- start start 2024-04-26T02:03:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.481+0000 7ff8ad582740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5585214f6820 0x5585214f6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.481+0000 7ff8ad582740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 0x5585215014c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.481+0000 7ff8ad582740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558521501a00 0x558521503df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.481+0000 7ff8ad582740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55852145c900 con 0x5585214f6820 2024-04-26T02:03:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.481+0000 7ff8ad582740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558521402ef0 con 0x5585214f7140 2024-04-26T02:03:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.481+0000 7ff8ad582740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5585214588c0 con 0x558521501a00 2024-04-26T02:03:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.481+0000 7ff8a3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5585214f6820 0x5585214f6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.481+0000 7ff8a3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 0x5585215014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.481+0000 7ff8a3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5585214f6820 0x5585214f6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45298/0 (socket says 172.21.15.6:45298) 2024-04-26T02:03:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.481+0000 7ff8a3e00640 1 -- 172.21.15.6:0/328066569 learned_addr learned my addr 172.21.15.6:0/328066569 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.481+0000 7ff8a2a00640 1 --2- 172.21.15.6:0/328066569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558521501a00 0x558521503df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff8a3e00640 1 -- 172.21.15.6:0/328066569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558521501a00 msgr2=0x558521503df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/328066569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558521501a00 0x558521503df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff8a3e00640 1 -- 172.21.15.6:0/328066569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 msgr2=0x5585215014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/328066569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 0x5585215014c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff8a3e00640 1 -- 172.21.15.6:0/328066569 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55852116f210 con 0x5585214f6820 2024-04-26T02:03:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff8a3400640 1 --2- 172.21.15.6:0/328066569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 0x5585215014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/328066569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5585214f6820 0x5585214f6c00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff89c031440 tx=0x7ff89c032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5f7e64c1484d186 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff89be00640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff8880a3d90 0x7ff8880a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff898c00640 1 -- 172.21.15.6:0/328066569 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c040070 con 0x5585214f6820 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff89be00640 1 -- 172.21.15.6:0/2102901887 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7ff8940d0f70 con 0x7ff8a41085a0 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff898c00640 1 -- 172.21.15.6:0/328066569 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff89c004030 con 0x5585214f6820 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff898c00640 1 -- 172.21.15.6:0/328066569 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c035040 con 0x5585214f6820 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.482+0000 7ff8a3400640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff8880a3d90 0x7ff8880a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.483+0000 7ff8ad582740 1 -- 172.21.15.6:0/328066569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5585214f6820 msgr2=0x5585214f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.483+0000 7ff8ad582740 1 --2- 172.21.15.6:0/328066569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5585214f6820 0x5585214f6c00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff89c031440 tx=0x7ff89c032b20 comp rx=0 tx=0).stop 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.483+0000 7ff8a3400640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff8880a3d90 0x7ff8880a6250 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x5585214f7dc0 tx=0x7ff894004930 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.483+0000 7ff8ad582740 1 -- 172.21.15.6:0/328066569 shutdown_connections 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.483+0000 7ff8ad582740 1 --2- 172.21.15.6:0/328066569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558521501a00 0x558521503df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.483+0000 7ff8ad582740 1 --2- 172.21.15.6:0/328066569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 0x5585215014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.483+0000 7ff8ad582740 1 --2- 172.21.15.6:0/328066569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5585214f6820 0x5585214f6c00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.483+0000 7ff8ad582740 1 -- 172.21.15.6:0/328066569 >> 172.21.15.6:0/328066569 conn(0x5585214f2d30 msgr2=0x5585214f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.483+0000 7ff8ad582740 1 -- 172.21.15.6:0/328066569 shutdown_connections 2024-04-26T02:03:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.483+0000 7ff8ad582740 1 -- 172.21.15.6:0/328066569 wait complete. 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.484+0000 7ff8ad582740 1 -- start start 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.484+0000 7ff8ad582740 1 Processor -- start 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.484+0000 7ff8ad582740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 0x5585214fe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.484+0000 7ff8ad582740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 0x5585214feef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.484+0000 7ff8ad582740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 0x5585214ff430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.484+0000 7ff8ad582740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5585215045d0 con 0x558521501a00 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.484+0000 7ff8ad582740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558521416310 con 0x5585214f7140 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.484+0000 7ff8ad582740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558521504750 con 0x5585214f6820 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8ad582740 10 client.0 _ll_register_callbacks cb 0x5585213e4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8a2a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 0x5585214feef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8a3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 0x5585214ff430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8a2a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 0x5585214feef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:60130/0 (socket says 172.21.15.6:60130) 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8a2a00640 1 -- 172.21.15.6:0/31376623 learned_addr learned my addr 172.21.15.6:0/31376623 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8a3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 0x5585214ff430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45304/0 (socket says 172.21.15.6:45304) 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 0x5585214fe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8a2a00640 1 -- 172.21.15.6:0/31376623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 msgr2=0x5585214fe9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8a2a00640 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 0x5585214fe9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8a2a00640 1 -- 172.21.15.6:0/31376623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 msgr2=0x5585214ff430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8a2a00640 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 0x5585214ff430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8a2a00640 1 -- 172.21.15.6:0/31376623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55852123b620 con 0x5585214f7140 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.485+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 0x5585214fe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.486+0000 7ff8a2a00640 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 0x5585214feef0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff8900124d0 tx=0x7ff890012990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da5f7068761f6831 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.486+0000 7ff88d600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.486+0000 7ff88d600640 1 -- 172.21.15.6:0/31376623 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff890011890 con 0x5585214f7140 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.486+0000 7ff88d600640 1 -- 172.21.15.6:0/31376623 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7ff890011a30 con 0x5585214f7140 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.486+0000 7ff8ad582740 1 -- 172.21.15.6:0/31376623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff89c004030 con 0x5585214f7140 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.486+0000 7ff88d600640 1 -- 172.21.15.6:0/31376623 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7ff890011c30 con 0x5585214f7140 2024-04-26T02:03:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.486+0000 7ff8ad582740 10 client.27182 fetch_fsmap learned FSMap version 53 2024-04-26T02:03:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.486+0000 7ff8ad582740 1 -- 172.21.15.6:0/31376623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5585215108f0 con 0x5585214f7140 2024-04-26T02:03:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.487+0000 7ff88d600640 1 -- 172.21.15.6:0/31376623 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff89c004030 con 0x5585214f7140 2024-04-26T02:03:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.487+0000 7ff88d600640 1 -- 172.21.15.6:0/31376623 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7ff89000f460 con 0x5585214f7140 2024-04-26T02:03:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.487+0000 7ff8ad582740 10 client.27182 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T02:03:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.487+0000 7ff8ad582740 10 client.27182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:03:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.487+0000 7ff8ad582740 20 client.27182 populate_metadata read hostname 'smithi006' 2024-04-26T02:03:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.487+0000 7ff8ad582740 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558521511e00 0x5585215141f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.487+0000 7ff8ad582740 4 client.27182 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:03:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.487+0000 7ff8ad582740 1 -- 172.21.15.6:0/31376623 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585214708b0 con 0x558521511e00 2024-04-26T02:03:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.488+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558521511e00 0x5585215141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.488+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558521511e00 0x5585215141f0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d717aebccb46a6c1 server_cookie=479d7edccb72d14b in_seq=0 out_seq=0 2024-04-26T02:03:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.488+0000 7ff88d600640 10 client.27182 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.489+0000 7ff88d600640 1 -- 172.21.15.6:0/31376623 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5585214708b0 con 0x558521511e00 2024-04-26T02:03:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.489+0000 7ff88d600640 10 client.27182 handle_command_reply: tid=0 2024-04-26T02:03:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.489+0000 7ff8ad582740 1 client.27182 shutdown 2024-04-26T02:03:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.489+0000 7ff8ad582740 1 -- 172.21.15.6:0/31376623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 msgr2=0x5585214feef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.489+0000 7ff8ad582740 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 0x5585214feef0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff8900124d0 tx=0x7ff890012990 comp rx=0 tx=0).stop 2024-04-26T02:03:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.489+0000 7ff8ad582740 1 -- 172.21.15.6:0/31376623 shutdown_connections 2024-04-26T02:03:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.489+0000 7ff8ad582740 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x558521511e00 0x5585215141f0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.489+0000 7ff8ad582740 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 0x5585214ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.489+0000 7ff8ad582740 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f7140 0x5585214feef0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.489+0000 7ff88d600640 0 client.27182 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.489+0000 7ff8ad582740 1 --2- 172.21.15.6:0/31376623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 0x5585214fe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.489+0000 7ff8ad582740 1 -- 172.21.15.6:0/31376623 >> 172.21.15.6:0/31376623 conn(0x5585214f2d30 msgr2=0x5585214f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.490+0000 7ff8ad582740 1 -- 172.21.15.6:0/31376623 shutdown_connections 2024-04-26T02:03:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.490+0000 7ff8ad582740 1 -- 172.21.15.6:0/31376623 wait complete. 2024-04-26T02:03:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.490+0000 7ff8ad582740 20 client.27182 trim_cache size 0 max 16384 2024-04-26T02:03:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.506+0000 7ff8ad582740 1 Processor -- start 2024-04-26T02:03:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.506+0000 7ff8ad582740 1 -- start start 2024-04-26T02:03:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.506+0000 7ff8ad582740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f6820 0x55852150d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.506+0000 7ff8ad582740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f7140 0x5585214edf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.506+0000 7ff8ad582740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 0x558521519b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.506+0000 7ff8ad582740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a41b6c40 con 0x558521501a00 2024-04-26T02:03:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.506+0000 7ff8ad582740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a41b4510 con 0x5585214f6820 2024-04-26T02:03:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.506+0000 7ff8ad582740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a41b2380 con 0x5585214f7140 2024-04-26T02:03:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.507+0000 7ff8a3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 0x558521519b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.507+0000 7ff8a3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 0x558521519b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45314/0 (socket says 172.21.15.6:45314) 2024-04-26T02:03:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.507+0000 7ff8a3e00640 1 -- 172.21.15.6:0/1761143721 learned_addr learned my addr 172.21.15.6:0/1761143721 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.507+0000 7ff8a2a00640 1 --2- 172.21.15.6:0/1761143721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f7140 0x5585214edf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.507+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/1761143721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f6820 0x55852150d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.507+0000 7ff8a3e00640 1 -- 172.21.15.6:0/1761143721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f7140 msgr2=0x5585214edf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.507+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/1761143721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f7140 0x5585214edf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.507+0000 7ff8a3e00640 1 -- 172.21.15.6:0/1761143721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f6820 msgr2=0x55852150d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.507+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/1761143721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f6820 0x55852150d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.507+0000 7ff8a3e00640 1 -- 172.21.15.6:0/1761143721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585214fd960 con 0x558521501a00 2024-04-26T02:03:13.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.507+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/1761143721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 0x558521519b00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009230 tx=0x7ff89c005150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448aea906630ca87 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:13.508 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.508+0000 7ff88fe00640 1 -- 172.21.15.6:0/1761143721 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c035040 con 0x558521501a00 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.508+0000 7ff88fe00640 1 -- 172.21.15.6:0/1761143721 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff89c05a070 con 0x558521501a00 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.508+0000 7ff88fe00640 1 -- 172.21.15.6:0/1761143721 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c040040 con 0x558521501a00 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.508+0000 7ff8ad582740 1 -- 172.21.15.6:0/1761143721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 msgr2=0x558521519b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.508+0000 7ff8ad582740 1 --2- 172.21.15.6:0/1761143721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 0x558521519b00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009230 tx=0x7ff89c005150 comp rx=0 tx=0).stop 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.508+0000 7ff8ad582740 1 -- 172.21.15.6:0/1761143721 shutdown_connections 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.509+0000 7ff8ad582740 1 --2- 172.21.15.6:0/1761143721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x558521501a00 0x558521519b00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.509+0000 7ff8ad582740 1 --2- 172.21.15.6:0/1761143721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f7140 0x5585214edf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.509+0000 7ff8ad582740 1 --2- 172.21.15.6:0/1761143721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5585214f6820 0x55852150d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.509+0000 7ff8ad582740 1 -- 172.21.15.6:0/1761143721 >> 172.21.15.6:0/1761143721 conn(0x5585214f2d30 msgr2=0x5585214ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.509+0000 7ff8ad582740 1 -- 172.21.15.6:0/1761143721 shutdown_connections 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.509+0000 7ff8ad582740 1 -- 172.21.15.6:0/1761143721 wait complete. 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.509+0000 7ff8ad582740 1 -- start start 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.509+0000 7ff8ad582740 1 Processor -- start 2024-04-26T02:03:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8ad582740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 0x5585215072f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8ad582740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5585214f7140 0x558521507830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8ad582740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558521501a00 0x558521507d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8ad582740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a41b29d0 con 0x5585214f7140 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8ad582740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a41b3e40 con 0x558521501a00 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8ad582740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a41b30a0 con 0x5585214f6820 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8ad582740 10 client.0 _ll_register_callbacks cb 0x7ff890009c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8a3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 0x5585215072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8a3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 0x5585215072f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:40508/0 (socket says 172.21.15.6:40508) 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8a3e00640 1 -- 172.21.15.6:0/3014819808 learned_addr learned my addr 172.21.15.6:0/3014819808 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8a2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5585214f7140 0x558521507830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/3014819808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558521501a00 0x558521507d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8a3e00640 1 -- 172.21.15.6:0/3014819808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558521501a00 msgr2=0x558521507d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/3014819808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558521501a00 0x558521507d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8a3e00640 1 -- 172.21.15.6:0/3014819808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5585214f7140 msgr2=0x558521507830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:13.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.510+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/3014819808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5585214f7140 0x558521507830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.511 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.511+0000 7ff8a3e00640 1 -- 172.21.15.6:0/3014819808 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585214f97d0 con 0x5585214f6820 2024-04-26T02:03:13.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.511+0000 7ff8a3e00640 1 --2- 172.21.15.6:0/3014819808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 0x5585215072f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff89c005ec0 tx=0x7ff89c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2973e0be47b8c62 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:13.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.511+0000 7ff88d600640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:03:13.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.511+0000 7ff88d600640 1 -- 172.21.15.6:0/3014819808 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c040070 con 0x5585214f6820 2024-04-26T02:03:13.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.511+0000 7ff8ad582740 1 -- 172.21.15.6:0/3014819808 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff89c05a070 con 0x5585214f6820 2024-04-26T02:03:13.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.511+0000 7ff88d600640 1 -- 172.21.15.6:0/3014819808 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7ff89c031d10 con 0x5585214f6820 2024-04-26T02:03:13.512 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.511+0000 7ff88d600640 1 -- 172.21.15.6:0/3014819808 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7ff89c035040 con 0x5585214f6820 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.511+0000 7ff8ad582740 10 client.27244 fetch_fsmap learned FSMap version 53 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.511+0000 7ff8ad582740 1 -- 172.21.15.6:0/3014819808 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5585215783c0 con 0x5585214f6820 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.512+0000 7ff88d600640 1 -- 172.21.15.6:0/3014819808 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff89c05a070 con 0x5585214f6820 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.512+0000 7ff88d600640 1 -- 172.21.15.6:0/3014819808 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7ff89c054960 con 0x5585214f6820 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.512+0000 7ff8ad582740 10 client.27244 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.512+0000 7ff8ad582740 10 client.27244 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.512+0000 7ff8ad582740 20 client.27244 populate_metadata read hostname 'smithi006' 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.512+0000 7ff8ad582740 1 --2- 172.21.15.6:0/3014819808 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5585215785e0 0x55852157a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.512+0000 7ff8ad582740 4 client.27244 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.512+0000 7ff8ad582740 1 -- 172.21.15.6:0/3014819808 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55852146a320 con 0x5585215785e0 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.512+0000 7ff8a2a00640 1 --2- 172.21.15.6:0/3014819808 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5585215785e0 0x55852157a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.513+0000 7ff8a2a00640 1 --2- 172.21.15.6:0/3014819808 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5585215785e0 0x55852157a9d0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8613dac5eab2e44f server_cookie=5ce86b9f4a432081 in_seq=0 out_seq=0 2024-04-26T02:03:13.513 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.513+0000 7ff88d600640 10 client.27244 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.632+0000 7ff88d600640 1 -- 172.21.15.6:0/3014819808 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55852146a320 con 0x5585215785e0 2024-04-26T02:03:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.632+0000 7ff88d600640 10 client.27244 handle_command_reply: tid=0 2024-04-26T02:03:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.632+0000 7ff8ad582740 1 client.27244 shutdown 2024-04-26T02:03:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.632+0000 7ff8ad582740 1 -- 172.21.15.6:0/3014819808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 msgr2=0x5585215072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.632+0000 7ff8ad582740 1 --2- 172.21.15.6:0/3014819808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 0x5585215072f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff89c005ec0 tx=0x7ff89c005fd0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.632+0000 7ff8ad582740 1 -- 172.21.15.6:0/3014819808 shutdown_connections 2024-04-26T02:03:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.632+0000 7ff8ad582740 1 --2- 172.21.15.6:0/3014819808 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5585215785e0 0x55852157a9d0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.632+0000 7ff8ad582740 1 --2- 172.21.15.6:0/3014819808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5585214f7140 0x558521507830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.632+0000 7ff8ad582740 1 --2- 172.21.15.6:0/3014819808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558521501a00 0x558521507d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.632+0000 7ff8ad582740 1 --2- 172.21.15.6:0/3014819808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5585214f6820 0x5585215072f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.632+0000 7ff8ad582740 1 -- 172.21.15.6:0/3014819808 >> 172.21.15.6:0/3014819808 conn(0x5585214f2d30 msgr2=0x5585214f81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.632+0000 7ff88d600640 0 client.27244 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.633+0000 7ff8ad582740 1 -- 172.21.15.6:0/3014819808 shutdown_connections 2024-04-26T02:03:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.633+0000 7ff8ad582740 1 -- 172.21.15.6:0/3014819808 wait complete. 2024-04-26T02:03:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.633+0000 7ff8ad582740 20 client.27244 trim_cache size 0 max 16384 2024-04-26T02:03:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:03:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:03:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:03:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:03:13.633 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.634+0000 7ff8aa800640 1 -- 172.21.15.6:0/2102901887 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff8880a3d90 msgr2=0x7ff8880a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.634+0000 7ff8aa800640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff8880a3d90 0x7ff8880a6250 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x5585214f7dc0 tx=0x7ff894004930 comp rx=0 tx=0).stop 2024-04-26T02:03:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.634+0000 7ff8aa800640 1 -- 172.21.15.6:0/2102901887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8a41085a0 msgr2=0x7ff8a41a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.634+0000 7ff8aa800640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8a41085a0 0x7ff8a41a4920 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff894009d50 tx=0x7ff89400c6a0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.634+0000 7ff8aa800640 1 -- 172.21.15.6:0/2102901887 shutdown_connections 2024-04-26T02:03:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.634+0000 7ff8aa800640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7ff8880a3d90 0x7ff8880a6250 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.634+0000 7ff8aa800640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a410e660 0x7ff8a419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.634+0000 7ff8aa800640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8a4108f60 0x7ff8a41a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.634+0000 7ff8aa800640 1 --2- 172.21.15.6:0/2102901887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8a41085a0 0x7ff8a41a4920 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.634+0000 7ff8aa800640 1 -- 172.21.15.6:0/2102901887 >> 172.21.15.6:0/2102901887 conn(0x7ff8a4076460 msgr2=0x7ff8a4077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.634+0000 7ff8aa800640 1 -- 172.21.15.6:0/2102901887 shutdown_connections 2024-04-26T02:03:13.634 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:13.634+0000 7ff8aa800640 1 -- 172.21.15.6:0/2102901887 wait complete. 2024-04-26T02:03:14.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:13 smithi006 ceph-mon[28419]: pgmap v551: 97 pgs: 97 active+clean; 822 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 6.6 MiB/s wr, 207 op/s 2024-04-26T02:03:14.115 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 25.52 MB/sec execute 97 sec latency 19493.510 ms 2024-04-26T02:03:14.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:13 smithi188 ceph-mon[32751]: pgmap v551: 97 pgs: 97 active+clean; 822 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 6.6 MiB/s wr, 207 op/s 2024-04-26T02:03:14.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:13 smithi029 ceph-mon[32775]: pgmap v551: 97 pgs: 97 active+clean; 822 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 6.6 MiB/s wr, 207 op/s 2024-04-26T02:03:15.116 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 25.26 MB/sec execute 98 sec latency 20493.646 ms 2024-04-26T02:03:16.116 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 25.00 MB/sec execute 99 sec latency 21493.826 ms 2024-04-26T02:03:16.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:16 smithi006 ceph-mon[28419]: pgmap v552: 97 pgs: 97 active+clean; 822 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 5.8 MiB/s wr, 180 op/s 2024-04-26T02:03:16.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:16 smithi188 ceph-mon[32751]: pgmap v552: 97 pgs: 97 active+clean; 822 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 5.8 MiB/s wr, 180 op/s 2024-04-26T02:03:16.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:16 smithi029 ceph-mon[32775]: pgmap v552: 97 pgs: 97 active+clean; 822 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 5.8 MiB/s wr, 180 op/s 2024-04-26T02:03:17.116 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 24.75 MB/sec execute 100 sec latency 22493.999 ms 2024-04-26T02:03:18.116 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 24.51 MB/sec execute 101 sec latency 23494.092 ms 2024-04-26T02:03:18.474 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:03:18.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:18 smithi006 ceph-mon[28419]: pgmap v553: 97 pgs: 97 active+clean; 800 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 4.0 MiB/s wr, 203 op/s 2024-04-26T02:03:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:18 smithi188 ceph-mon[32751]: pgmap v553: 97 pgs: 97 active+clean; 800 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 4.0 MiB/s wr, 203 op/s 2024-04-26T02:03:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:18 smithi029 ceph-mon[32775]: pgmap v553: 97 pgs: 97 active+clean; 800 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 4.0 MiB/s wr, 203 op/s 2024-04-26T02:03:18.971 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:03:19.116 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 24.27 MB/sec execute 102 sec latency 24494.186 ms 2024-04-26T02:03:20.116 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 24.03 MB/sec execute 103 sec latency 25494.296 ms 2024-04-26T02:03:20.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:20 smithi006 ceph-mon[28419]: pgmap v554: 97 pgs: 97 active+clean; 800 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 4.0 MiB/s wr, 203 op/s 2024-04-26T02:03:20.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:20 smithi188 ceph-mon[32751]: pgmap v554: 97 pgs: 97 active+clean; 800 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 4.0 MiB/s wr, 203 op/s 2024-04-26T02:03:20.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:20 smithi029 ceph-mon[32775]: pgmap v554: 97 pgs: 97 active+clean; 800 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 4.0 MiB/s wr, 203 op/s 2024-04-26T02:03:21.116 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 23.80 MB/sec execute 104 sec latency 26494.462 ms 2024-04-26T02:03:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.450+0000 7f5115200700 1 -- 172.21.15.6:0/1259854154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f511006b0f0 msgr2=0x7f511010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.450+0000 7f5115200700 1 --2- 172.21.15.6:0/1259854154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f511006b0f0 0x7f511010f900 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f510800b0a0 tx=0x7f510800b3b0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.450+0000 7f5115200700 1 -- 172.21.15.6:0/1259854154 shutdown_connections 2024-04-26T02:03:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.450+0000 7f5115200700 1 --2- 172.21.15.6:0/1259854154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f511006b0f0 0x7f511010f900 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.450+0000 7f5115200700 1 --2- 172.21.15.6:0/1259854154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5110108a30 0x7f5110108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.450+0000 7f5115200700 1 --2- 172.21.15.6:0/1259854154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f511006a750 0x7f511006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.450+0000 7f5115200700 1 -- 172.21.15.6:0/1259854154 >> 172.21.15.6:0/1259854154 conn(0x7f51100788f0 msgr2=0x7f5110078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.451+0000 7f5115200700 1 -- 172.21.15.6:0/1259854154 shutdown_connections 2024-04-26T02:03:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.451+0000 7f5115200700 1 -- 172.21.15.6:0/1259854154 wait complete. 2024-04-26T02:03:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.452+0000 7f5115200700 1 Processor -- start 2024-04-26T02:03:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.452+0000 7f5115200700 1 -- start start 2024-04-26T02:03:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.452+0000 7f5115200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f511006a750 0x7f5110101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.452+0000 7f5115200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f511006b0f0 0x7f5110101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.452+0000 7f5115200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5110108a30 0x7f5110102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.453+0000 7f5115200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5110112bd0 con 0x7f511006b0f0 2024-04-26T02:03:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.453+0000 7f5115200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5110112a50 con 0x7f5110108a30 2024-04-26T02:03:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.453+0000 7f5115200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5110112d50 con 0x7f511006a750 2024-04-26T02:03:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.453+0000 7f510d400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f511006b0f0 0x7f5110101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.453+0000 7f510de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f511006a750 0x7f5110101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.453+0000 7f510d400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f511006b0f0 0x7f5110101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45042/0 (socket says 172.21.15.6:45042) 2024-04-26T02:03:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.453+0000 7f510d400700 1 -- 172.21.15.6:0/1374953176 learned_addr learned my addr 172.21.15.6:0/1374953176 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.453+0000 7f510e800700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5110108a30 0x7f5110102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:21.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.453+0000 7f510d400700 1 -- 172.21.15.6:0/1374953176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f511006a750 msgr2=0x7f5110101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:21.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.454+0000 7f510d400700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f511006a750 0x7f5110101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.454+0000 7f510d400700 1 -- 172.21.15.6:0/1374953176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5110108a30 msgr2=0x7f5110102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:21.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.454+0000 7f510d400700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5110108a30 0x7f5110102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.454+0000 7f510d400700 1 -- 172.21.15.6:0/1374953176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51101a7600 con 0x7f511006b0f0 2024-04-26T02:03:21.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.454+0000 7f510de00700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f511006a750 0x7f5110101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:21.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.454+0000 7f510e800700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5110108a30 0x7f5110102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:21.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.454+0000 7f510d400700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f511006b0f0 0x7f5110101f50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f50fc00e9a0 tx=0x7f50fc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:21.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.454+0000 7f5106a00700 1 -- 172.21.15.6:0/1374953176 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50fc00ca10 con 0x7f511006b0f0 2024-04-26T02:03:21.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.455+0000 7f5106a00700 1 -- 172.21.15.6:0/1374953176 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50fc00cbb0 con 0x7f511006b0f0 2024-04-26T02:03:21.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.455+0000 7f5115200700 1 -- 172.21.15.6:0/1374953176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51101a78f0 con 0x7f511006b0f0 2024-04-26T02:03:21.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.455+0000 7f5115200700 1 -- 172.21.15.6:0/1374953176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51101a7e30 con 0x7f511006b0f0 2024-04-26T02:03:21.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.455+0000 7f5106a00700 1 -- 172.21.15.6:0/1374953176 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50fc018680 con 0x7f511006b0f0 2024-04-26T02:03:21.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.456+0000 7f5115200700 1 -- 172.21.15.6:0/1374953176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5110069730 con 0x7f511006b0f0 2024-04-26T02:03:21.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.458+0000 7f5106a00700 1 -- 172.21.15.6:0/1374953176 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f50fc010430 con 0x7f511006b0f0 2024-04-26T02:03:21.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.460+0000 7f5106a00700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f50f40a3db0 0x7f50f40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:21.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.460+0000 7f5106a00700 1 -- 172.21.15.6:0/1374953176 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7f50fc0d0cd0 con 0x7f511006b0f0 2024-04-26T02:03:21.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.460+0000 7f510de00700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f50f40a3db0 0x7f50f40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:21.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.461+0000 7f510de00700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f50f40a3db0 0x7f50f40a6270 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5100009fd0 tx=0x7f5100009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:21.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.463+0000 7f5106a00700 1 -- 172.21.15.6:0/1374953176 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f50fc09d2c0 con 0x7f511006b0f0 2024-04-26T02:03:21.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.746+0000 7f5115200700 1 -- 172.21.15.6:0/1374953176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5110102ef0 con 0x7f511006b0f0 2024-04-26T02:03:21.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.748+0000 7f5106a00700 1 -- 172.21.15.6:0/1374953176 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11850 (secure 0 0 0) 0x7f50fc09cbc0 con 0x7f511006b0f0 2024-04-26T02:03:21.747 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:03:21.748 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T02:03:04.019513+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26935":{"gid":26935,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1750728695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1750728695},{"type":"v1","addr":"172.21.15.188:6835","nonce":1750728695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T02:03:21.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.751+0000 7f5115200700 1 -- 172.21.15.6:0/1374953176 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f50f40a3db0 msgr2=0x7f50f40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:21.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.751+0000 7f5115200700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f50f40a3db0 0x7f50f40a6270 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5100009fd0 tx=0x7f5100009260 comp rx=0 tx=0).stop 2024-04-26T02:03:21.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.751+0000 7f5115200700 1 -- 172.21.15.6:0/1374953176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f511006b0f0 msgr2=0x7f5110101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:21.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.751+0000 7f5115200700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f511006b0f0 0x7f5110101f50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f50fc00e9a0 tx=0x7f50fc00ecb0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.751+0000 7f5115200700 1 -- 172.21.15.6:0/1374953176 shutdown_connections 2024-04-26T02:03:21.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.751+0000 7f5115200700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f50f40a3db0 0x7f50f40a6270 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.751+0000 7f5115200700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5110108a30 0x7f5110102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.751+0000 7f5115200700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f511006a750 0x7f5110101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.751+0000 7f5115200700 1 --2- 172.21.15.6:0/1374953176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f511006b0f0 0x7f5110101f50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.751+0000 7f5115200700 1 -- 172.21.15.6:0/1374953176 >> 172.21.15.6:0/1374953176 conn(0x7f51100788f0 msgr2=0x7f51100ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:21.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.752+0000 7f5115200700 1 -- 172.21.15.6:0/1374953176 shutdown_connections 2024-04-26T02:03:21.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:21.752+0000 7f5115200700 1 -- 172.21.15.6:0/1374953176 wait complete. 2024-04-26T02:03:21.751 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 53 2024-04-26T02:03:22.116 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 23.57 MB/sec execute 105 sec latency 27494.530 ms 2024-04-26T02:03:22.572 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id 33ab2393 --release --await' 2024-04-26T02:03:22.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id 33ab2393 --release --await 2024-04-26T02:03:22.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:22 smithi006 ceph-mon[28419]: pgmap v555: 97 pgs: 97 active+clean; 791 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 6.3 MiB/s wr, 276 op/s 2024-04-26T02:03:22.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:22 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1374953176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:03:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:22 smithi188 ceph-mon[32751]: pgmap v555: 97 pgs: 97 active+clean; 791 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 6.3 MiB/s wr, 276 op/s 2024-04-26T02:03:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:22 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1374953176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:03:22.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:22 smithi029 ceph-mon[32775]: pgmap v555: 97 pgs: 97 active+clean; 791 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 6.3 MiB/s wr, 276 op/s 2024-04-26T02:03:22.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:22 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1374953176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:03:23.000 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:03:23.117 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 23.35 MB/sec execute 106 sec latency 28494.658 ms 2024-04-26T02:03:24.117 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 23.13 MB/sec execute 107 sec latency 29494.803 ms 2024-04-26T02:03:24.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:24 smithi006 ceph-mon[28419]: pgmap v556: 97 pgs: 97 active+clean; 791 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 3.2 MiB/s wr, 163 op/s 2024-04-26T02:03:24.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:24 smithi188 ceph-mon[32751]: pgmap v556: 97 pgs: 97 active+clean; 791 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 3.2 MiB/s wr, 163 op/s 2024-04-26T02:03:24.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:24 smithi029 ceph-mon[32775]: pgmap v556: 97 pgs: 97 active+clean; 791 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 3.2 MiB/s wr, 163 op/s 2024-04-26T02:03:25.117 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78169 22.92 MB/sec execute 108 sec latency 30494.911 ms 2024-04-26T02:03:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.434+0000 7fc468000700 1 -- 172.21.15.6:0/1989668022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc460101360 msgr2=0x7fc460101740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.434+0000 7fc468000700 1 --2- 172.21.15.6:0/1989668022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc460101360 0x7fc460101740 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc45400aaa0 tx=0x7fc45400adb0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.435+0000 7fc468000700 1 -- 172.21.15.6:0/1989668022 shutdown_connections 2024-04-26T02:03:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.435+0000 7fc468000700 1 --2- 172.21.15.6:0/1989668022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4601063f0 0x7fc46010dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.435+0000 7fc468000700 1 --2- 172.21.15.6:0/1989668022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc460101360 0x7fc460101740 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.435+0000 7fc468000700 1 --2- 172.21.15.6:0/1989668022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc460101d10 0x7fc460105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.435+0000 7fc468000700 1 -- 172.21.15.6:0/1989668022 >> 172.21.15.6:0/1989668022 conn(0x7fc460077fb0 msgr2=0x7fc4600ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.435+0000 7fc468000700 1 -- 172.21.15.6:0/1989668022 shutdown_connections 2024-04-26T02:03:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.435+0000 7fc468000700 1 -- 172.21.15.6:0/1989668022 wait complete. 2024-04-26T02:03:25.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.436+0000 7fc468000700 1 Processor -- start 2024-04-26T02:03:25.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.437+0000 7fc468000700 1 -- start start 2024-04-26T02:03:25.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.437+0000 7fc468000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc460101360 0x7fc4601991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.437+0000 7fc468000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc460101d10 0x7fc460199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.437+0000 7fc468000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4601063f0 0x7fc46019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.437+0000 7fc468000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc460110a70 con 0x7fc460101360 2024-04-26T02:03:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.437+0000 7fc468000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc4601108f0 con 0x7fc460101d10 2024-04-26T02:03:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.437+0000 7fc468000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc460110bf0 con 0x7fc4601063f0 2024-04-26T02:03:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.438+0000 7fc465600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc460101360 0x7fc4601991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.438+0000 7fc464c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc460101d10 0x7fc460199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.438+0000 7fc464c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc460101d10 0x7fc460199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34822/0 (socket says 172.21.15.6:34822) 2024-04-26T02:03:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.438+0000 7fc465600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc460101360 0x7fc4601991e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45056/0 (socket says 172.21.15.6:45056) 2024-04-26T02:03:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.438+0000 7fc464c00700 1 -- 172.21.15.6:0/1101242332 learned_addr learned my addr 172.21.15.6:0/1101242332 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.438+0000 7fc466000700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4601063f0 0x7fc46019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.438+0000 7fc464c00700 1 -- 172.21.15.6:0/1101242332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4601063f0 msgr2=0x7fc46019dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.438+0000 7fc464c00700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4601063f0 0x7fc46019dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.438+0000 7fc464c00700 1 -- 172.21.15.6:0/1101242332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc460101360 msgr2=0x7fc4601991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.438+0000 7fc464c00700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc460101360 0x7fc4601991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.438+0000 7fc464c00700 1 -- 172.21.15.6:0/1101242332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc46019e1f0 con 0x7fc460101d10 2024-04-26T02:03:25.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.439+0000 7fc464c00700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc460101d10 0x7fc460199720 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00e970 tx=0x7fc44c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:25.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.439+0000 7fc45e000700 1 -- 172.21.15.6:0/1101242332 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c004a30 con 0x7fc460101d10 2024-04-26T02:03:25.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.439+0000 7fc45e000700 1 -- 172.21.15.6:0/1101242332 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc44c00cc90 con 0x7fc460101d10 2024-04-26T02:03:25.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.439+0000 7fc45e000700 1 -- 172.21.15.6:0/1101242332 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c005610 con 0x7fc460101d10 2024-04-26T02:03:25.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.439+0000 7fc468000700 1 -- 172.21.15.6:0/1101242332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc46019e4e0 con 0x7fc460101d10 2024-04-26T02:03:25.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.439+0000 7fc468000700 1 -- 172.21.15.6:0/1101242332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4601a5d60 con 0x7fc460101d10 2024-04-26T02:03:25.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.443+0000 7fc45e000700 1 -- 172.21.15.6:0/1101242332 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fc44c00c7e0 con 0x7fc460101d10 2024-04-26T02:03:25.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.445+0000 7fc4728f5b80 1 Processor -- start 2024-04-26T02:03:25.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.445+0000 7fc4728f5b80 1 -- start start 2024-04-26T02:03:25.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.445+0000 7fc4728f5b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db7d40 0x560355db8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.445+0000 7fc45e000700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc4480a3e30 0x7fc4480a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc4728f5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db8660 0x560355dc73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc4728f5b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 0x560355dc9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc4728f5b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560355d0af90 con 0x560355db7d40 2024-04-26T02:03:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc4728f5b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560355cb1830 con 0x560355dc78e0 2024-04-26T02:03:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc4728f5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560355d06f50 con 0x560355db8660 2024-04-26T02:03:25.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc45e000700 1 -- 172.21.15.6:0/1101242332 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7fc44c0d09d0 con 0x7fc460101d10 2024-04-26T02:03:25.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc464c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db8660 0x560355dc73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc464c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db8660 0x560355dc73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59694/0 (socket says 172.21.15.6:59694) 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc464c00700 1 -- 172.21.15.6:0/3307261520 learned_addr learned my addr 172.21.15.6:0/3307261520 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc465600700 1 --2- 172.21.15.6:0/3307261520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db7d40 0x560355db8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc466000700 1 --2- 172.21.15.6:0/3307261520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 0x560355dc9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc466000700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc4480a3e30 0x7fc4480a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc464c00700 1 -- 172.21.15.6:0/3307261520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 msgr2=0x560355dc9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc464c00700 1 --2- 172.21.15.6:0/3307261520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 0x560355dc9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc464c00700 1 -- 172.21.15.6:0/3307261520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db7d40 msgr2=0x560355db8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc464c00700 1 --2- 172.21.15.6:0/3307261520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db7d40 0x560355db8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc464c00700 1 -- 172.21.15.6:0/3307261520 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560355ca7780 con 0x560355db8660 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.446+0000 7fc465600700 1 --2- 172.21.15.6:0/3307261520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db7d40 0x560355db8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.447+0000 7fc466000700 1 --2- 172.21.15.6:0/3307261520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 0x560355dc9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.447+0000 7fc464c00700 1 --2- 172.21.15.6:0/3307261520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db8660 0x560355dc73a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0a5000 tx=0x7fc44c098c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81799b11a78317e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.447+0000 7fc452a00700 1 -- 172.21.15.6:0/3307261520 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c00cad0 con 0x560355db8660 2024-04-26T02:03:25.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.447+0000 7fc452a00700 1 -- 172.21.15.6:0/3307261520 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc44c09cfc0 con 0x560355db8660 2024-04-26T02:03:25.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.447+0000 7fc452a00700 1 -- 172.21.15.6:0/3307261520 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c016df0 con 0x560355db8660 2024-04-26T02:03:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.448+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3307261520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db8660 msgr2=0x560355dc73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.448+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3307261520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db8660 0x560355dc73a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0a5000 tx=0x7fc44c098c90 comp rx=0 tx=0).stop 2024-04-26T02:03:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.448+0000 7fc466000700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc4480a3e30 0x7fc4480a62f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x560355db9710 tx=0x7fc45800b040 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.448+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3307261520 shutdown_connections 2024-04-26T02:03:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.448+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3307261520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 0x560355dc9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.448+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3307261520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db7d40 0x560355db8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.448+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3307261520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db8660 0x560355dc73a0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.448+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3307261520 >> 172.21.15.6:0/3307261520 conn(0x560355ce7050 msgr2=0x560355db4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.448+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3307261520 shutdown_connections 2024-04-26T02:03:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.448+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3307261520 wait complete. 2024-04-26T02:03:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.449+0000 7fc4728f5b80 1 -- start start 2024-04-26T02:03:25.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.449+0000 7fc4728f5b80 1 Processor -- start 2024-04-26T02:03:25.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc4728f5b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db7d40 0x560355dc7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc4728f5b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355db8660 0x560355dc4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc4728f5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355dc78e0 0x560355dc4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc4728f5b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560355dca3d0 con 0x560355db7d40 2024-04-26T02:03:25.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc4728f5b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560355cc4960 con 0x560355db8660 2024-04-26T02:03:25.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc4728f5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560355dca550 con 0x560355dc78e0 2024-04-26T02:03:25.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc4728f5b80 10 client.0 _ll_register_callbacks cb 0x560355ca3820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:25.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc466000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355dc78e0 0x560355dc4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc465600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db7d40 0x560355dc7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc464c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355db8660 0x560355dc4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc466000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355dc78e0 0x560355dc4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59710/0 (socket says 172.21.15.6:59710) 2024-04-26T02:03:25.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc466000700 1 -- 172.21.15.6:0/292099366 learned_addr learned my addr 172.21.15.6:0/292099366 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:25.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc466000700 1 -- 172.21.15.6:0/292099366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355db8660 msgr2=0x560355dc4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:25.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc466000700 1 --2- 172.21.15.6:0/292099366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355db8660 0x560355dc4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc466000700 1 -- 172.21.15.6:0/292099366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db7d40 msgr2=0x560355dc7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:25.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.450+0000 7fc466000700 1 --2- 172.21.15.6:0/292099366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db7d40 0x560355dc7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.451+0000 7fc466000700 1 -- 172.21.15.6:0/292099366 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560355c589d0 con 0x560355dc78e0 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.451+0000 7fc466000700 1 --2- 172.21.15.6:0/292099366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355dc78e0 0x560355dc4880 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc458018b00 tx=0x7fc458004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13544ae338b065be server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.451+0000 7fc447400700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.451+0000 7fc447400700 1 -- 172.21.15.6:0/292099366 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc458003c10 con 0x560355dc78e0 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.451+0000 7fc447400700 1 -- 172.21.15.6:0/292099366 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7fc458003db0 con 0x560355dc78e0 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.451+0000 7fc4728f5b80 1 -- 172.21.15.6:0/292099366 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc44c09cfc0 con 0x560355dc78e0 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.452+0000 7fc447400700 1 -- 172.21.15.6:0/292099366 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fc458016800 con 0x560355dc78e0 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.452+0000 7fc4728f5b80 10 client.27262 fetch_fsmap learned FSMap version 53 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.452+0000 7fc4728f5b80 1 -- 172.21.15.6:0/292099366 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x560355ca1bf0 con 0x560355dc78e0 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.452+0000 7fc447400700 1 -- 172.21.15.6:0/292099366 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc44c09cfc0 con 0x560355dc78e0 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.452+0000 7fc447400700 1 -- 172.21.15.6:0/292099366 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7fc458004620 con 0x560355dc78e0 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.452+0000 7fc4728f5b80 10 client.27262 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.452+0000 7fc4728f5b80 10 client.27262 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.452+0000 7fc4728f5b80 20 client.27262 populate_metadata read hostname 'smithi006' 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.452+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/292099366 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560355dd7080 0x560355dd9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.453+0000 7fc4728f5b80 4 client.27262 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:03:25.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.453+0000 7fc4728f5b80 1 -- 172.21.15.6:0/292099366 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560355d18780 con 0x560355dd7080 2024-04-26T02:03:25.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.453+0000 7fc465600700 1 --2- 172.21.15.6:0/292099366 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560355dd7080 0x560355dd9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.453+0000 7fc465600700 1 --2- 172.21.15.6:0/292099366 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560355dd7080 0x560355dd9470 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aae511b89e63a1f server_cookie=3754d310e68771f8 in_seq=0 out_seq=0 2024-04-26T02:03:25.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.453+0000 7fc447400700 10 client.27262 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:25.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.454+0000 7fc447400700 1 -- 172.21.15.6:0/292099366 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560355d18780 con 0x560355dd7080 2024-04-26T02:03:25.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.454+0000 7fc447400700 10 client.27262 handle_command_reply: tid=0 2024-04-26T02:03:25.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.454+0000 7fc4728f5b80 1 client.27262 shutdown 2024-04-26T02:03:25.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.454+0000 7fc4728f5b80 1 -- 172.21.15.6:0/292099366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355dc78e0 msgr2=0x560355dc4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.455+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/292099366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355dc78e0 0x560355dc4880 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc458018b00 tx=0x7fc458004b40 comp rx=0 tx=0).stop 2024-04-26T02:03:25.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.455+0000 7fc4728f5b80 1 -- 172.21.15.6:0/292099366 shutdown_connections 2024-04-26T02:03:25.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.455+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/292099366 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560355dd7080 0x560355dd9470 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.455+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/292099366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355dc78e0 0x560355dc4880 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.455+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/292099366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db7d40 0x560355dc7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.455+0000 7fc447400700 0 client.27262 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:25.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.455+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/292099366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355db8660 0x560355dc4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.455+0000 7fc4728f5b80 1 -- 172.21.15.6:0/292099366 >> 172.21.15.6:0/292099366 conn(0x560355ce7050 msgr2=0x560355db3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:25.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.455+0000 7fc4728f5b80 1 -- 172.21.15.6:0/292099366 shutdown_connections 2024-04-26T02:03:25.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.455+0000 7fc4728f5b80 1 -- 172.21.15.6:0/292099366 wait complete. 2024-04-26T02:03:25.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.455+0000 7fc4728f5b80 20 client.27262 trim_cache size 0 max 16384 2024-04-26T02:03:25.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.477+0000 7fc4728f5b80 1 Processor -- start 2024-04-26T02:03:25.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.478+0000 7fc4728f5b80 1 -- start start 2024-04-26T02:03:25.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.478+0000 7fc4728f5b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355db7d40 0x560355dc3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.478+0000 7fc4728f5b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db8660 0x560355dd32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.478+0000 7fc4728f5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355dc78e0 0x560355dded40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.478+0000 7fc4728f5b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc4601adca0 con 0x560355db8660 2024-04-26T02:03:25.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.478+0000 7fc4728f5b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc4601ab530 con 0x560355db7d40 2024-04-26T02:03:25.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.478+0000 7fc4728f5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc4601a93a0 con 0x560355dc78e0 2024-04-26T02:03:25.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.478+0000 7fc465600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355db7d40 0x560355dc3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.478+0000 7fc465600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355db7d40 0x560355dc3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34854/0 (socket says 172.21.15.6:34854) 2024-04-26T02:03:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.478+0000 7fc465600700 1 -- 172.21.15.6:0/3729160656 learned_addr learned my addr 172.21.15.6:0/3729160656 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.478+0000 7fc466000700 1 --2- 172.21.15.6:0/3729160656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355dc78e0 0x560355dded40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.479+0000 7fc464c00700 1 --2- 172.21.15.6:0/3729160656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db8660 0x560355dd32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.479+0000 7fc465600700 1 -- 172.21.15.6:0/3729160656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355dc78e0 msgr2=0x560355dded40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.479+0000 7fc465600700 1 --2- 172.21.15.6:0/3729160656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355dc78e0 0x560355dded40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.479+0000 7fc465600700 1 -- 172.21.15.6:0/3729160656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db8660 msgr2=0x560355dd32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.479+0000 7fc465600700 1 --2- 172.21.15.6:0/3729160656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db8660 0x560355dd32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.479+0000 7fc465600700 1 -- 172.21.15.6:0/3729160656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560355db9bb0 con 0x560355db7d40 2024-04-26T02:03:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.479+0000 7fc464c00700 1 --2- 172.21.15.6:0/3729160656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db8660 0x560355dd32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:03:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.479+0000 7fc465600700 1 --2- 172.21.15.6:0/3729160656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355db7d40 0x560355dc3670 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc45400a860 tx=0x7fc45400a420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f52c3b14810b72e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.479+0000 7fc452000700 1 -- 172.21.15.6:0/3729160656 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454005990 con 0x560355db7d40 2024-04-26T02:03:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.479+0000 7fc452000700 1 -- 172.21.15.6:0/3729160656 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc454003680 con 0x560355db7d40 2024-04-26T02:03:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.479+0000 7fc452000700 1 -- 172.21.15.6:0/3729160656 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454005070 con 0x560355db7d40 2024-04-26T02:03:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.480+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3729160656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355db7d40 msgr2=0x560355dc3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.480+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3729160656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355db7d40 0x560355dc3670 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc45400a860 tx=0x7fc45400a420 comp rx=0 tx=0).stop 2024-04-26T02:03:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.480+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3729160656 shutdown_connections 2024-04-26T02:03:25.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.480+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3729160656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355dc78e0 0x560355dded40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.480+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3729160656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355db7d40 0x560355dc3670 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.480+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3729160656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db8660 0x560355dd32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.480+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3729160656 >> 172.21.15.6:0/3729160656 conn(0x560355ce7050 msgr2=0x560355dc6400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.481+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3729160656 shutdown_connections 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.481+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3729160656 wait complete. 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.481+0000 7fc4728f5b80 1 -- start start 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.482+0000 7fc4728f5b80 1 Processor -- start 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.482+0000 7fc4728f5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db7d40 0x560355dcfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.482+0000 7fc4728f5b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db8660 0x560355dbe7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.482+0000 7fc4728f5b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 0x560355dbed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.482+0000 7fc4728f5b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc4601a99f0 con 0x560355db8660 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.482+0000 7fc4728f5b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc4601aae60 con 0x560355dc78e0 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.482+0000 7fc4728f5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc4601aa0c0 con 0x560355db7d40 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.482+0000 7fc4728f5b80 10 client.0 _ll_register_callbacks cb 0x560355ca3820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.482+0000 7fc464c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db8660 0x560355dbe7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.482+0000 7fc466000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 0x560355dbed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.483+0000 7fc466000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 0x560355dbed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34860/0 (socket says 172.21.15.6:34860) 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.483+0000 7fc466000700 1 -- 172.21.15.6:0/3564482787 learned_addr learned my addr 172.21.15.6:0/3564482787 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.483+0000 7fc465600700 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db7d40 0x560355dcfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.483+0000 7fc466000700 1 -- 172.21.15.6:0/3564482787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db7d40 msgr2=0x560355dcfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.483+0000 7fc466000700 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db7d40 0x560355dcfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.483+0000 7fc466000700 1 -- 172.21.15.6:0/3564482787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db8660 msgr2=0x560355dbe7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.483+0000 7fc466000700 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db8660 0x560355dbe7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.483+0000 7fc466000700 1 -- 172.21.15.6:0/3564482787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560355dbcf70 con 0x560355dc78e0 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.483+0000 7fc465600700 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db7d40 0x560355dcfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.483+0000 7fc464c00700 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db8660 0x560355dbe7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.484+0000 7fc466000700 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 0x560355dbed00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc4580093b0 tx=0x7fc458016bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45cd12641db2d1c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.484+0000 7fc447400700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:03:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.484+0000 7fc447400700 1 -- 172.21.15.6:0/3564482787 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45802d5e0 con 0x560355dc78e0 2024-04-26T02:03:25.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.484+0000 7fc447400700 1 -- 172.21.15.6:0/3564482787 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7fc45802d780 con 0x560355dc78e0 2024-04-26T02:03:25.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.484+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3564482787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc454003680 con 0x560355dc78e0 2024-04-26T02:03:25.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.484+0000 7fc447400700 1 -- 172.21.15.6:0/3564482787 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fc458016650 con 0x560355dc78e0 2024-04-26T02:03:25.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.484+0000 7fc4728f5b80 10 client.27212 fetch_fsmap learned FSMap version 53 2024-04-26T02:03:25.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.484+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3564482787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x560355dceba0 con 0x560355dc78e0 2024-04-26T02:03:25.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.485+0000 7fc447400700 1 -- 172.21.15.6:0/3564482787 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc454003680 con 0x560355dc78e0 2024-04-26T02:03:25.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.485+0000 7fc447400700 1 -- 172.21.15.6:0/3564482787 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7fc458036030 con 0x560355dc78e0 2024-04-26T02:03:25.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.485+0000 7fc4728f5b80 10 client.27212 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T02:03:25.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.485+0000 7fc4728f5b80 10 client.27212 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:03:25.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.485+0000 7fc4728f5b80 20 client.27212 populate_metadata read hostname 'smithi006' 2024-04-26T02:03:25.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.485+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560355dcf2b0 0x560355e3d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.485+0000 7fc4728f5b80 4 client.27212 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "33ab2393", "release": true, "await": true}] 2024-04-26T02:03:25.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.485+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3564482787 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "33ab2393", "release": true, "await": true}) -- 0x560355d12820 con 0x560355dcf2b0 2024-04-26T02:03:25.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.485+0000 7fc465600700 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560355dcf2b0 0x560355e3d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.486+0000 7fc465600700 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560355dcf2b0 0x560355e3d5b0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41a0735993696e49 server_cookie=f06406862c443d73 in_seq=0 out_seq=0 2024-04-26T02:03:25.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.486+0000 7fc447400700 10 client.27212 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:25.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.528+0000 7fc447400700 1 -- 172.21.15.6:0/3564482787 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x560355d12820 con 0x560355dcf2b0 2024-04-26T02:03:25.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.528+0000 7fc447400700 10 client.27212 handle_command_reply: tid=0 2024-04-26T02:03:25.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.529+0000 7fc4728f5b80 1 client.27212 shutdown 2024-04-26T02:03:25.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.529+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3564482787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 msgr2=0x560355dbed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.529+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 0x560355dbed00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc4580093b0 tx=0x7fc458016bb0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.529+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3564482787 shutdown_connections 2024-04-26T02:03:25.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.529+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560355dc78e0 0x560355dbed00 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.529+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560355dcf2b0 0x560355e3d5b0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.529+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560355db7d40 0x560355dcfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.529+0000 7fc4728f5b80 1 --2- 172.21.15.6:0/3564482787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560355db8660 0x560355dbe7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.529+0000 7fc447400700 0 client.27212 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:25.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.529+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3564482787 >> 172.21.15.6:0/3564482787 conn(0x560355ce7050 msgr2=0x560355dc5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:25.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.530+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3564482787 shutdown_connections 2024-04-26T02:03:25.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.530+0000 7fc4728f5b80 1 -- 172.21.15.6:0/3564482787 wait complete. 2024-04-26T02:03:25.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.530+0000 7fc4728f5b80 20 client.27212 trim_cache size 0 max 16384 2024-04-26T02:03:25.530 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:03:25.530 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 53, 2024-04-26T02:03:25.530 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T02:03:25.530 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 52, 2024-04-26T02:03:25.530 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T02:03:25.530 INFO:teuthology.orchestra.run.smithi006.stdout: "33ab2393": { 2024-04-26T02:03:25.530 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 52, 2024-04-26T02:03:25.530 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T02:03:25.530 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:03:25.530 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T02:03:25.530 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 82.5, 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RELEASED", 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:03:25.531 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:03:25.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.531+0000 7fc468000700 1 -- 172.21.15.6:0/1101242332 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc4480a3e30 msgr2=0x7fc4480a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.531+0000 7fc468000700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc4480a3e30 0x7fc4480a62f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x560355db9710 tx=0x7fc45800b040 comp rx=0 tx=0).stop 2024-04-26T02:03:25.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.531+0000 7fc468000700 1 -- 172.21.15.6:0/1101242332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc460101d10 msgr2=0x7fc460199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.531+0000 7fc468000700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc460101d10 0x7fc460199720 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00e970 tx=0x7fc44c00ec80 comp rx=0 tx=0).stop 2024-04-26T02:03:25.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.532+0000 7fc468000700 1 -- 172.21.15.6:0/1101242332 shutdown_connections 2024-04-26T02:03:25.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.532+0000 7fc468000700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc4480a3e30 0x7fc4480a62f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.532+0000 7fc468000700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4601063f0 0x7fc46019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.532+0000 7fc468000700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc460101360 0x7fc4601991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.532+0000 7fc468000700 1 --2- 172.21.15.6:0/1101242332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc460101d10 0x7fc460199720 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.532+0000 7fc468000700 1 -- 172.21.15.6:0/1101242332 >> 172.21.15.6:0/1101242332 conn(0x7fc460077fb0 msgr2=0x7fc4600ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:25.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.532+0000 7fc468000700 1 -- 172.21.15.6:0/1101242332 shutdown_connections 2024-04-26T02:03:25.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:25.532+0000 7fc468000700 1 -- 172.21.15.6:0/1101242332 wait complete. 2024-04-26T02:03:26.117 INFO:tasks.workunit.client.0.smithi006.stdout: 10 78717 22.92 MB/sec execute 109 sec latency 30926.734 ms 2024-04-26T02:03:26.391 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '33ab2393', seconds elapsed: 39.5 2024-04-26T02:03:26.391 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.55 2024-04-26T02:03:26.391 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:03:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.817+0000 7f1a37400640 1 -- 172.21.15.6:0/1482197303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a301085a0 msgr2=0x7f1a30108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.817+0000 7f1a37400640 1 --2- 172.21.15.6:0/1482197303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a301085a0 0x7f1a30108a20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c009930 tx=0x7f1a1c02a420 comp rx=0 tx=0).stop 2024-04-26T02:03:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.818+0000 7f1a37400640 1 -- 172.21.15.6:0/1482197303 shutdown_connections 2024-04-26T02:03:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.818+0000 7f1a37400640 1 --2- 172.21.15.6:0/1482197303 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a30108f60 0x7f1a30115420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.818+0000 7f1a37400640 1 --2- 172.21.15.6:0/1482197303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a301085a0 0x7f1a30108a20 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.818+0000 7f1a37400640 1 --2- 172.21.15.6:0/1482197303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a3010e660 0x7f1a3010ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.818+0000 7f1a37400640 1 -- 172.21.15.6:0/1482197303 >> 172.21.15.6:0/1482197303 conn(0x7f1a30076460 msgr2=0x7f1a30078880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.818+0000 7f1a37400640 1 -- 172.21.15.6:0/1482197303 shutdown_connections 2024-04-26T02:03:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.818+0000 7f1a37400640 1 -- 172.21.15.6:0/1482197303 wait complete. 2024-04-26T02:03:26.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:26 smithi006 ceph-mon[28419]: pgmap v557: 97 pgs: 97 active+clean; 791 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 3.2 MiB/s wr, 163 op/s 2024-04-26T02:03:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.820+0000 7f1a37400640 1 Processor -- start 2024-04-26T02:03:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.820+0000 7f1a37400640 1 -- start start 2024-04-26T02:03:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.821+0000 7f1a37400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a301085a0 0x7f1a30073820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.821+0000 7f1a37400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a30108f60 0x7f1a3006c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.821+0000 7f1a37400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a3010e660 0x7f1a3006cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.821+0000 7f1a37400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1a301188a0 con 0x7f1a30108f60 2024-04-26T02:03:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.821+0000 7f1a37400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1a30118720 con 0x7f1a301085a0 2024-04-26T02:03:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.821+0000 7f1a37400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1a30118a20 con 0x7f1a3010e660 2024-04-26T02:03:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.822+0000 7f1a35400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a3010e660 0x7f1a3006cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.822+0000 7f1a35400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a3010e660 0x7f1a3006cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59758/0 (socket says 172.21.15.6:59758) 2024-04-26T02:03:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.822+0000 7f1a35400640 1 -- 172.21.15.6:0/2959803155 learned_addr learned my addr 172.21.15.6:0/2959803155 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.822+0000 7f1a34a00640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a301085a0 0x7f1a30073820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.822+0000 7f1a2fe00640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a30108f60 0x7f1a3006c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.822+0000 7f1a35400640 1 -- 172.21.15.6:0/2959803155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a301085a0 msgr2=0x7f1a30073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.822+0000 7f1a35400640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a301085a0 0x7f1a30073820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.822+0000 7f1a35400640 1 -- 172.21.15.6:0/2959803155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a30108f60 msgr2=0x7f1a3006c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.822+0000 7f1a35400640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a30108f60 0x7f1a3006c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.822+0000 7f1a35400640 1 -- 172.21.15.6:0/2959803155 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3006d480 con 0x7f1a3010e660 2024-04-26T02:03:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.822+0000 7f1a34a00640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a301085a0 0x7f1a30073820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.823+0000 7f1a2fe00640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a30108f60 0x7f1a3006c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:03:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.824+0000 7f1a35400640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a3010e660 0x7f1a3006cde0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800eb10 tx=0x7f1a2800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.825+0000 7f1a2d600640 1 -- 172.21.15.6:0/2959803155 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a28004b20 con 0x7f1a3010e660 2024-04-26T02:03:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.825+0000 7f1a2d600640 1 -- 172.21.15.6:0/2959803155 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a28004cc0 con 0x7f1a3010e660 2024-04-26T02:03:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.825+0000 7f1a37400640 1 -- 172.21.15.6:0/2959803155 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a30071340 con 0x7f1a3010e660 2024-04-26T02:03:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.825+0000 7f1a2d600640 1 -- 172.21.15.6:0/2959803155 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a28010670 con 0x7f1a3010e660 2024-04-26T02:03:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.825+0000 7f1a37400640 1 -- 172.21.15.6:0/2959803155 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a300718e0 con 0x7f1a3010e660 2024-04-26T02:03:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.832+0000 7f1a2d600640 1 -- 172.21.15.6:0/2959803155 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f1a28005020 con 0x7f1a3010e660 2024-04-26T02:03:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.832+0000 7f1a3a173740 1 Processor -- start 2024-04-26T02:03:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.833+0000 7f1a3a173740 1 -- start start 2024-04-26T02:03:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.833+0000 7f1a3a173740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001d720 0x55576001db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.833+0000 7f1a3a173740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55576001e040 0x5557600394d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.833+0000 7f1a3a173740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555760039a10 0x55576003be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.833+0000 7f1a3a173740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55575ff83900 con 0x555760039a10 2024-04-26T02:03:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.833+0000 7f1a3a173740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55575ff29ef0 con 0x55576001e040 2024-04-26T02:03:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.833+0000 7f1a3a173740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55575ff7f8c0 con 0x55576001d720 2024-04-26T02:03:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.834+0000 7f1a34a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55576001e040 0x5557600394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.834+0000 7f1a34a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55576001e040 0x5557600394d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34896/0 (socket says 172.21.15.6:34896) 2024-04-26T02:03:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.834+0000 7f1a34a00640 1 -- 172.21.15.6:0/1003594018 learned_addr learned my addr 172.21.15.6:0/1003594018 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.834+0000 7f1a34a00640 1 -- 172.21.15.6:0/1003594018 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001d720 msgr2=0x55576001db00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T02:03:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.834+0000 7f1a34a00640 1 --2- 172.21.15.6:0/1003594018 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001d720 0x55576001db00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.834+0000 7f1a34a00640 1 -- 172.21.15.6:0/1003594018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555760039a10 msgr2=0x55576003be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.834+0000 7f1a2fe00640 1 --2- 172.21.15.6:0/1003594018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555760039a10 0x55576003be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.835+0000 7f1a2d600640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1a180a3d90 0x7f1a180a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.835+0000 7f1a2d600640 1 -- 172.21.15.6:0/2959803155 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7f1a280d0e80 con 0x7f1a3010e660 2024-04-26T02:03:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.835+0000 7f1a34a00640 1 --2- 172.21.15.6:0/1003594018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555760039a10 0x55576003be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.835+0000 7f1a34a00640 1 -- 172.21.15.6:0/1003594018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55575fc96210 con 0x55576001e040 2024-04-26T02:03:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.835+0000 7f1a2fe00640 1 --2- 172.21.15.6:0/1003594018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555760039a10 0x55576003be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:03:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.836+0000 7f1a2fe00640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1a180a3d90 0x7f1a180a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.836+0000 7f1a34a00640 1 --2- 172.21.15.6:0/1003594018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55576001e040 0x5557600394d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009cd0 tx=0x7f1a240117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b01ec8d8a9b893a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.836+0000 7f1a21c00640 1 -- 172.21.15.6:0/1003594018 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24012450 con 0x55576001e040 2024-04-26T02:03:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.836+0000 7f1a21c00640 1 -- 172.21.15.6:0/1003594018 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a24012aa0 con 0x55576001e040 2024-04-26T02:03:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.836+0000 7f1a21c00640 1 -- 172.21.15.6:0/1003594018 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2400fbf0 con 0x55576001e040 2024-04-26T02:03:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.837+0000 7f1a3a173740 1 -- 172.21.15.6:0/1003594018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55576001e040 msgr2=0x5557600394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.837+0000 7f1a3a173740 1 --2- 172.21.15.6:0/1003594018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55576001e040 0x5557600394d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009cd0 tx=0x7f1a240117b0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.838+0000 7f1a3a173740 1 -- 172.21.15.6:0/1003594018 shutdown_connections 2024-04-26T02:03:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.838+0000 7f1a3a173740 1 --2- 172.21.15.6:0/1003594018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555760039a10 0x55576003be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.838+0000 7f1a3a173740 1 --2- 172.21.15.6:0/1003594018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55576001e040 0x5557600394d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.838+0000 7f1a3a173740 1 --2- 172.21.15.6:0/1003594018 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001d720 0x55576001db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.838+0000 7f1a3a173740 1 -- 172.21.15.6:0/1003594018 >> 172.21.15.6:0/1003594018 conn(0x555760019d30 msgr2=0x55576001a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.838+0000 7f1a2fe00640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1a180a3d90 0x7f1a180a6250 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55576001f2e0 tx=0x7f1a1c009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.838+0000 7f1a3a173740 1 -- 172.21.15.6:0/1003594018 shutdown_connections 2024-04-26T02:03:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.838+0000 7f1a3a173740 1 -- 172.21.15.6:0/1003594018 wait complete. 2024-04-26T02:03:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.838+0000 7f1a3a173740 1 -- start start 2024-04-26T02:03:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.839+0000 7f1a3a173740 1 Processor -- start 2024-04-26T02:03:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.839+0000 7f1a3a173740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001d720 0x555760036970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.839+0000 7f1a3a173740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55576001e040 0x555760036eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.839+0000 7f1a3a173740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555760039a10 0x5557600373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a3a173740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55576003c640 con 0x555760039a10 2024-04-26T02:03:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a3a173740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55575ff3d310 con 0x55576001e040 2024-04-26T02:03:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a3a173740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55576003c7c0 con 0x55576001d720 2024-04-26T02:03:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a34a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001d720 0x555760036970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a34a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001d720 0x555760036970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:59784/0 (socket says 172.21.15.6:59784) 2024-04-26T02:03:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a34a00640 1 -- 172.21.15.6:0/3892716405 learned_addr learned my addr 172.21.15.6:0/3892716405 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a3a173740 10 client.0 _ll_register_callbacks cb 0x55575ff0bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a2fe00640 1 --2- 172.21.15.6:0/3892716405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55576001e040 0x555760036eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a34a00640 1 -- 172.21.15.6:0/3892716405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55576001e040 msgr2=0x555760036eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a34a00640 1 --2- 172.21.15.6:0/3892716405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55576001e040 0x555760036eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a34a00640 1 -- 172.21.15.6:0/3892716405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555760039a10 msgr2=0x5557600373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a34a00640 1 --2- 172.21.15.6:0/3892716405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555760039a10 0x5557600373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.840+0000 7f1a34a00640 1 -- 172.21.15.6:0/3892716405 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55575fd62620 con 0x55576001d720 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.841+0000 7f1a34a00640 1 --2- 172.21.15.6:0/3892716405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001d720 0x555760036970 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400f810 tx=0x7f1a240056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b34bd0279f8a06f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.841+0000 7f1a16000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.841+0000 7f1a16000640 1 -- 172.21.15.6:0/3892716405 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24028020 con 0x55576001d720 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.841+0000 7f1a16000640 1 -- 172.21.15.6:0/3892716405 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7f1a24012650 con 0x55576001d720 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.841+0000 7f1a3a173740 1 -- 172.21.15.6:0/3892716405 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a24012aa0 con 0x55576001d720 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.842+0000 7f1a16000640 1 -- 172.21.15.6:0/3892716405 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f1a24033050 con 0x55576001d720 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.842+0000 7f1a3a173740 10 client.27277 fetch_fsmap learned FSMap version 53 2024-04-26T02:03:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.842+0000 7f1a3a173740 1 -- 172.21.15.6:0/3892716405 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x555760048850 con 0x55576001d720 2024-04-26T02:03:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.844+0000 7f1a16000640 1 -- 172.21.15.6:0/3892716405 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a24012aa0 con 0x55576001d720 2024-04-26T02:03:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.844+0000 7f1a16000640 1 -- 172.21.15.6:0/3892716405 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f1a240038e0 con 0x55576001d720 2024-04-26T02:03:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.844+0000 7f1a3a173740 10 client.27277 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T02:03:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.844+0000 7f1a3a173740 10 client.27277 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:03:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.844+0000 7f1a3a173740 20 client.27277 populate_metadata read hostname 'smithi006' 2024-04-26T02:03:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.844+0000 7f1a3a173740 1 --2- 172.21.15.6:0/3892716405 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555760049cb0 0x55576004c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.844+0000 7f1a3a173740 4 client.27277 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:03:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.844+0000 7f1a3a173740 1 -- 172.21.15.6:0/3892716405 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55575ff978b0 con 0x555760049cb0 2024-04-26T02:03:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.845+0000 7f1a2fe00640 1 --2- 172.21.15.6:0/3892716405 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555760049cb0 0x55576004c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.846+0000 7f1a16000640 10 client.27277 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.846+0000 7f1a2fe00640 1 --2- 172.21.15.6:0/3892716405 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555760049cb0 0x55576004c0a0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14af4f24b02d2691 server_cookie=2c8bbec76305ed29 in_seq=0 out_seq=0 2024-04-26T02:03:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.849+0000 7f1a16000640 1 -- 172.21.15.6:0/3892716405 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55575ff978b0 con 0x555760049cb0 2024-04-26T02:03:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.849+0000 7f1a16000640 10 client.27277 handle_command_reply: tid=0 2024-04-26T02:03:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.849+0000 7f1a3a173740 1 client.27277 shutdown 2024-04-26T02:03:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.850+0000 7f1a3a173740 1 -- 172.21.15.6:0/3892716405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001d720 msgr2=0x555760036970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.850+0000 7f1a3a173740 1 --2- 172.21.15.6:0/3892716405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001d720 0x555760036970 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400f810 tx=0x7f1a240056d0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.850+0000 7f1a3a173740 1 -- 172.21.15.6:0/3892716405 shutdown_connections 2024-04-26T02:03:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.850+0000 7f1a3a173740 1 --2- 172.21.15.6:0/3892716405 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x555760049cb0 0x55576004c0a0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.850+0000 7f1a3a173740 1 --2- 172.21.15.6:0/3892716405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555760039a10 0x5557600373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.850+0000 7f1a3a173740 1 --2- 172.21.15.6:0/3892716405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55576001e040 0x555760036eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.850+0000 7f1a3a173740 1 --2- 172.21.15.6:0/3892716405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001d720 0x555760036970 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.850+0000 7f1a3a173740 1 -- 172.21.15.6:0/3892716405 >> 172.21.15.6:0/3892716405 conn(0x555760019d30 msgr2=0x55576001fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.850+0000 7f1a16000640 0 client.27277 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.850+0000 7f1a3a173740 1 -- 172.21.15.6:0/3892716405 shutdown_connections 2024-04-26T02:03:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.850+0000 7f1a3a173740 1 -- 172.21.15.6:0/3892716405 wait complete. 2024-04-26T02:03:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.851+0000 7f1a3a173740 20 client.27277 trim_cache size 0 max 16384 2024-04-26T02:03:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.875+0000 7f1a3a173740 1 Processor -- start 2024-04-26T02:03:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.875+0000 7f1a3a173740 1 -- start start 2024-04-26T02:03:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.876+0000 7f1a3a173740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55576001d720 0x555760045500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.876+0000 7f1a3a173740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001e040 0x55576001fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.876+0000 7f1a3a173740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 0x5557600516a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.876+0000 7f1a3a173740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1a301b6e00 con 0x55576001d720 2024-04-26T02:03:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.876+0000 7f1a3a173740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1a30082930 con 0x555760039a10 2024-04-26T02:03:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.876+0000 7f1a3a173740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1a300807a0 con 0x55576001e040 2024-04-26T02:03:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.876+0000 7f1a34a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55576001d720 0x555760045500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.876+0000 7f1a34a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55576001d720 0x555760045500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45142/0 (socket says 172.21.15.6:45142) 2024-04-26T02:03:26.875 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.876+0000 7f1a34a00640 1 -- 172.21.15.6:0/3269713852 learned_addr learned my addr 172.21.15.6:0/3269713852 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.877+0000 7f1a35400640 1 --2- 172.21.15.6:0/3269713852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 0x5557600516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.877+0000 7f1a2fe00640 1 --2- 172.21.15.6:0/3269713852 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001e040 0x55576001fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.877+0000 7f1a35400640 1 -- 172.21.15.6:0/3269713852 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001e040 msgr2=0x55576001fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.877+0000 7f1a35400640 1 --2- 172.21.15.6:0/3269713852 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001e040 0x55576001fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.877+0000 7f1a35400640 1 -- 172.21.15.6:0/3269713852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55576001d720 msgr2=0x555760045500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.877+0000 7f1a35400640 1 --2- 172.21.15.6:0/3269713852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55576001d720 0x555760045500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.877+0000 7f1a35400640 1 -- 172.21.15.6:0/3269713852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55576001de70 con 0x555760039a10 2024-04-26T02:03:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.878+0000 7f1a35400640 1 --2- 172.21.15.6:0/3269713852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 0x5557600516a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1a280ce580 tx=0x7f1a2809cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2df8b2db0d6a76c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:26.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.879+0000 7f1a21200640 1 -- 172.21.15.6:0/3269713852 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2809cdd0 con 0x555760039a10 2024-04-26T02:03:26.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.879+0000 7f1a21200640 1 -- 172.21.15.6:0/3269713852 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a280a1180 con 0x555760039a10 2024-04-26T02:03:26.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.880+0000 7f1a3a173740 1 -- 172.21.15.6:0/3269713852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 msgr2=0x5557600516a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:26.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.880+0000 7f1a3a173740 1 --2- 172.21.15.6:0/3269713852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 0x5557600516a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1a280ce580 tx=0x7f1a2809cc60 comp rx=0 tx=0).stop 2024-04-26T02:03:26.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.881+0000 7f1a3a173740 1 -- 172.21.15.6:0/3269713852 shutdown_connections 2024-04-26T02:03:26.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.881+0000 7f1a3a173740 1 --2- 172.21.15.6:0/3269713852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 0x5557600516a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.881+0000 7f1a3a173740 1 --2- 172.21.15.6:0/3269713852 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001e040 0x55576001fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.881+0000 7f1a3a173740 1 --2- 172.21.15.6:0/3269713852 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55576001d720 0x555760045500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.879 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.881+0000 7f1a3a173740 1 -- 172.21.15.6:0/3269713852 >> 172.21.15.6:0/3269713852 conn(0x555760019d30 msgr2=0x55576001f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:26.880 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.882+0000 7f1a3a173740 1 -- 172.21.15.6:0/3269713852 shutdown_connections 2024-04-26T02:03:26.881 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.882+0000 7f1a3a173740 1 -- 172.21.15.6:0/3269713852 wait complete. 2024-04-26T02:03:26.881 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.882+0000 7f1a3a173740 1 -- start start 2024-04-26T02:03:26.881 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.883+0000 7f1a3a173740 1 Processor -- start 2024-04-26T02:03:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:26 smithi188 ceph-mon[32751]: pgmap v557: 97 pgs: 97 active+clean; 791 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 3.2 MiB/s wr, 163 op/s 2024-04-26T02:03:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.892+0000 7f1a3a173740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55576001d720 0x55576003e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.892+0000 7f1a3a173740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001e040 0x55576003ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.892+0000 7f1a3a173740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 0x5557600adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.892+0000 7f1a3a173740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1a30080df0 con 0x55576001d720 2024-04-26T02:03:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.892+0000 7f1a3a173740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1a30082260 con 0x555760039a10 2024-04-26T02:03:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.892+0000 7f1a3a173740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1a300814c0 con 0x55576001e040 2024-04-26T02:03:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.892+0000 7f1a3a173740 10 client.0 _ll_register_callbacks cb 0x7f1a24004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.892+0000 7f1a34a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55576001d720 0x55576003e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.893+0000 7f1a35400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 0x5557600adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.893+0000 7f1a35400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 0x5557600adf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34912/0 (socket says 172.21.15.6:34912) 2024-04-26T02:03:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.893+0000 7f1a35400640 1 -- 172.21.15.6:0/198408711 learned_addr learned my addr 172.21.15.6:0/198408711 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.894+0000 7f1a2fe00640 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001e040 0x55576003ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.896+0000 7f1a35400640 1 -- 172.21.15.6:0/198408711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001e040 msgr2=0x55576003ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.896+0000 7f1a35400640 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001e040 0x55576003ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.896+0000 7f1a35400640 1 -- 172.21.15.6:0/198408711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55576001d720 msgr2=0x55576003e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.896+0000 7f1a35400640 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55576001d720 0x55576003e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.896+0000 7f1a35400640 1 -- 172.21.15.6:0/198408711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55576003f7f0 con 0x555760039a10 2024-04-26T02:03:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.896+0000 7f1a2fe00640 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001e040 0x55576003ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.898+0000 7f1a35400640 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 0x5557600adf10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1a280997a0 tx=0x7f1a2809ccd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdd3f13653b56d29 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.898+0000 7f1a16000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:03:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.898+0000 7f1a16000640 1 -- 172.21.15.6:0/198408711 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2809cdd0 con 0x555760039a10 2024-04-26T02:03:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.898+0000 7f1a3a173740 1 -- 172.21.15.6:0/198408711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55575ff91320 con 0x555760039a10 2024-04-26T02:03:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.898+0000 7f1a16000640 1 -- 172.21.15.6:0/198408711 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7f1a2809c220 con 0x555760039a10 2024-04-26T02:03:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.898+0000 7f1a16000640 1 -- 172.21.15.6:0/198408711 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f1a280a1180 con 0x555760039a10 2024-04-26T02:03:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.898+0000 7f1a3a173740 10 client.27230 fetch_fsmap learned FSMap version 53 2024-04-26T02:03:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.898+0000 7f1a3a173740 1 -- 172.21.15.6:0/198408711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5557600af5f0 con 0x555760039a10 2024-04-26T02:03:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.899+0000 7f1a16000640 1 -- 172.21.15.6:0/198408711 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55575ff91320 con 0x555760039a10 2024-04-26T02:03:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.899+0000 7f1a16000640 1 -- 172.21.15.6:0/198408711 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f1a280bd1a0 con 0x555760039a10 2024-04-26T02:03:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.900+0000 7f1a3a173740 10 client.27230 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T02:03:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.900+0000 7f1a3a173740 10 client.27230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:03:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.900+0000 7f1a3a173740 20 client.27230 populate_metadata read hostname 'smithi006' 2024-04-26T02:03:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.900+0000 7f1a3a173740 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5557600afc40 0x5557600b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.900+0000 7f1a3a173740 4 client.27230 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:03:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.900+0000 7f1a3a173740 1 -- 172.21.15.6:0/198408711 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55575ff8b190 con 0x5557600afc40 2024-04-26T02:03:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.902+0000 7f1a34a00640 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5557600afc40 0x5557600b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.909+0000 7f1a16000640 10 client.27230 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:26.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:26.909+0000 7f1a34a00640 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5557600afc40 0x5557600b2060 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=538fbae2b43114e server_cookie=999fa8f81ba06a75 in_seq=0 out_seq=0 2024-04-26T02:03:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:26 smithi029 ceph-mon[32775]: pgmap v557: 97 pgs: 97 active+clean; 791 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 3.2 MiB/s wr, 163 op/s 2024-04-26T02:03:27.038 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:03:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.104+0000 7f1a16000640 1 -- 172.21.15.6:0/198408711 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55575ff8b190 con 0x5557600afc40 2024-04-26T02:03:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.104+0000 7f1a16000640 10 client.27230 handle_command_reply: tid=0 2024-04-26T02:03:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.104+0000 7f1a3a173740 1 client.27230 shutdown 2024-04-26T02:03:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.104+0000 7f1a3a173740 1 -- 172.21.15.6:0/198408711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 msgr2=0x5557600adf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.104+0000 7f1a3a173740 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 0x5557600adf10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1a280997a0 tx=0x7f1a2809ccd0 comp rx=0 tx=0).stop 2024-04-26T02:03:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.104+0000 7f1a3a173740 1 -- 172.21.15.6:0/198408711 shutdown_connections 2024-04-26T02:03:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.104+0000 7f1a3a173740 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5557600afc40 0x5557600b2060 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.104+0000 7f1a3a173740 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555760039a10 0x5557600adf10 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.105+0000 7f1a3a173740 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55576001e040 0x55576003ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.105+0000 7f1a3a173740 1 --2- 172.21.15.6:0/198408711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55576001d720 0x55576003e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.105+0000 7f1a3a173740 1 -- 172.21.15.6:0/198408711 >> 172.21.15.6:0/198408711 conn(0x555760019d30 msgr2=0x555760018e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.105+0000 7f1a16000640 0 client.27230 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.105+0000 7f1a3a173740 1 -- 172.21.15.6:0/198408711 shutdown_connections 2024-04-26T02:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.105+0000 7f1a3a173740 1 -- 172.21.15.6:0/198408711 wait complete. 2024-04-26T02:03:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.105+0000 7f1a3a173740 20 client.27230 trim_cache size 0 max 16384 2024-04-26T02:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:03:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.107+0000 7f1a37400640 1 -- 172.21.15.6:0/2959803155 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1a180a3d90 msgr2=0x7f1a180a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.107+0000 7f1a37400640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1a180a3d90 0x7f1a180a6250 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55576001f2e0 tx=0x7f1a1c009260 comp rx=0 tx=0).stop 2024-04-26T02:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.107+0000 7f1a37400640 1 -- 172.21.15.6:0/2959803155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a3010e660 msgr2=0x7f1a3006cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.107+0000 7f1a37400640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a3010e660 0x7f1a3006cde0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800eb10 tx=0x7f1a2800c2d0 comp rx=0 tx=0).stop 2024-04-26T02:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.107+0000 7f1a37400640 1 -- 172.21.15.6:0/2959803155 shutdown_connections 2024-04-26T02:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.107+0000 7f1a37400640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f1a180a3d90 0x7f1a180a6250 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.107+0000 7f1a37400640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a30108f60 0x7f1a3006c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.107+0000 7f1a37400640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a3010e660 0x7f1a3006cde0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.107+0000 7f1a37400640 1 --2- 172.21.15.6:0/2959803155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a301085a0 0x7f1a30073820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.107+0000 7f1a37400640 1 -- 172.21.15.6:0/2959803155 >> 172.21.15.6:0/2959803155 conn(0x7f1a30076460 msgr2=0x7f1a30077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.107+0000 7f1a37400640 1 -- 172.21.15.6:0/2959803155 shutdown_connections 2024-04-26T02:03:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:27.107+0000 7f1a37400640 1 -- 172.21.15.6:0/2959803155 wait complete. 2024-04-26T02:03:27.117 INFO:tasks.workunit.client.0.smithi006.stdout: 10 79303 23.08 MB/sec execute 110 sec latency 132.415 ms 2024-04-26T02:03:28.117 INFO:tasks.workunit.client.0.smithi006.stdout: 10 79829 23.16 MB/sec execute 111 sec latency 78.446 ms 2024-04-26T02:03:28.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:28 smithi006 ceph-mon[28419]: pgmap v558: 97 pgs: 97 active+clean; 793 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 282 op/s 2024-04-26T02:03:28.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:28 smithi188 ceph-mon[32751]: pgmap v558: 97 pgs: 97 active+clean; 793 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 282 op/s 2024-04-26T02:03:28.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:28 smithi029 ceph-mon[32775]: pgmap v558: 97 pgs: 97 active+clean; 793 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 282 op/s 2024-04-26T02:03:29.118 INFO:tasks.workunit.client.0.smithi006.stdout: 10 80284 23.17 MB/sec execute 112 sec latency 57.663 ms 2024-04-26T02:03:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.368+0000 7f5771000700 1 -- 172.21.15.6:0/1285910686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f576c108a30 msgr2=0x7f576c108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.368+0000 7f5771000700 1 --2- 172.21.15.6:0/1285910686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f576c108a30 0x7f576c108e10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f575c009990 tx=0x7f575c009ca0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.368+0000 7f5771000700 1 -- 172.21.15.6:0/1285910686 shutdown_connections 2024-04-26T02:03:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.368+0000 7f5771000700 1 --2- 172.21.15.6:0/1285910686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f576c06b0f0 0x7f576c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.368+0000 7f5771000700 1 --2- 172.21.15.6:0/1285910686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f576c108a30 0x7f576c108e10 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.368+0000 7f5771000700 1 --2- 172.21.15.6:0/1285910686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f576c06a750 0x7f576c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.368+0000 7f5771000700 1 -- 172.21.15.6:0/1285910686 >> 172.21.15.6:0/1285910686 conn(0x7f576c0788f0 msgr2=0x7f576c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.369+0000 7f5771000700 1 -- 172.21.15.6:0/1285910686 shutdown_connections 2024-04-26T02:03:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.369+0000 7f5771000700 1 -- 172.21.15.6:0/1285910686 wait complete. 2024-04-26T02:03:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.370+0000 7f5771000700 1 Processor -- start 2024-04-26T02:03:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.370+0000 7f5771000700 1 -- start start 2024-04-26T02:03:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.371+0000 7f5771000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f576c06a750 0x7f576c199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.371+0000 7f5771000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f576c06b0f0 0x7f576c199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.371+0000 7f5771000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f576c108a30 0x7f576c19d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.371+0000 7f5771000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f576c112c00 con 0x7f576c108a30 2024-04-26T02:03:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.371+0000 7f5771000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f576c112a80 con 0x7f576c06b0f0 2024-04-26T02:03:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.371+0000 7f5771000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f576c112d80 con 0x7f576c06a750 2024-04-26T02:03:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.371+0000 7f5769400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f576c06b0f0 0x7f576c199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.371+0000 7f5769e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f576c06a750 0x7f576c199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.372+0000 7f5769400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f576c06b0f0 0x7f576c199650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:50594/0 (socket says 172.21.15.6:50594) 2024-04-26T02:03:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.372+0000 7f5769400700 1 -- 172.21.15.6:0/2749350441 learned_addr learned my addr 172.21.15.6:0/2749350441 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.372+0000 7f576a800700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f576c108a30 0x7f576c19d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.372+0000 7f5769400700 1 -- 172.21.15.6:0/2749350441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f576c06a750 msgr2=0x7f576c199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.372+0000 7f5769400700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f576c06a750 0x7f576c199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.372+0000 7f5769400700 1 -- 172.21.15.6:0/2749350441 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f576c108a30 msgr2=0x7f576c19d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.372+0000 7f5769400700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f576c108a30 0x7f576c19d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.372+0000 7f5769400700 1 -- 172.21.15.6:0/2749350441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576c19e120 con 0x7f576c06b0f0 2024-04-26T02:03:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.372+0000 7f5769e00700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f576c06a750 0x7f576c199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.372+0000 7f5769400700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f576c06b0f0 0x7f576c199650 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f575800d950 tx=0x7f575800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.373+0000 7f5762a00700 1 -- 172.21.15.6:0/2749350441 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758016070 con 0x7f576c06b0f0 2024-04-26T02:03:29.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.373+0000 7f5771000700 1 -- 172.21.15.6:0/2749350441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f576c19e410 con 0x7f576c06b0f0 2024-04-26T02:03:29.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.373+0000 7f5762a00700 1 -- 172.21.15.6:0/2749350441 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f575800fd40 con 0x7f576c06b0f0 2024-04-26T02:03:29.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.373+0000 7f5762a00700 1 -- 172.21.15.6:0/2749350441 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575800e3b0 con 0x7f576c06b0f0 2024-04-26T02:03:29.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.373+0000 7f5771000700 1 -- 172.21.15.6:0/2749350441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f576c1a5c90 con 0x7f576c06b0f0 2024-04-26T02:03:29.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.374+0000 7f5771000700 1 -- 172.21.15.6:0/2749350441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f576c069730 con 0x7f576c06b0f0 2024-04-26T02:03:29.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.379+0000 7f5762a00700 1 -- 172.21.15.6:0/2749350441 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f5758019070 con 0x7f576c06b0f0 2024-04-26T02:03:29.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.380+0000 7f5762a00700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f57500a3db0 0x7f57500a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.381+0000 7f5762a00700 1 -- 172.21.15.6:0/2749350441 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7f57580d06c0 con 0x7f576c06b0f0 2024-04-26T02:03:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.381+0000 7f5769e00700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f57500a3db0 0x7f57500a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.382+0000 7f5769e00700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f57500a3db0 0x7f57500a6270 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f575c006010 tx=0x7f575c00b540 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.382+0000 7f5762a00700 1 -- 172.21.15.6:0/2749350441 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f575809ccb0 con 0x7f576c06b0f0 2024-04-26T02:03:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.785+0000 7f5771000700 1 -- 172.21.15.6:0/2749350441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f576c02df80 con 0x7f576c06b0f0 2024-04-26T02:03:29.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.786+0000 7f5762a00700 1 -- 172.21.15.6:0/2749350441 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11850 (secure 0 0 0) 0x7f575809c5b0 con 0x7f576c06b0f0 2024-04-26T02:03:29.785 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:03:29.785 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T02:03:04.019513+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26935":{"gid":26935,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1750728695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1750728695},{"type":"v1","addr":"172.21.15.188:6835","nonce":1750728695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T02:03:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.789+0000 7f5771000700 1 -- 172.21.15.6:0/2749350441 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f57500a3db0 msgr2=0x7f57500a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.789+0000 7f5771000700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f57500a3db0 0x7f57500a6270 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f575c006010 tx=0x7f575c00b540 comp rx=0 tx=0).stop 2024-04-26T02:03:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.790+0000 7f5771000700 1 -- 172.21.15.6:0/2749350441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f576c06b0f0 msgr2=0x7f576c199650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.790+0000 7f5771000700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f576c06b0f0 0x7f576c199650 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f575800d950 tx=0x7f575800dc60 comp rx=0 tx=0).stop 2024-04-26T02:03:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.790+0000 7f5771000700 1 -- 172.21.15.6:0/2749350441 shutdown_connections 2024-04-26T02:03:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.790+0000 7f5771000700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f57500a3db0 0x7f57500a6270 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.790+0000 7f5771000700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f576c108a30 0x7f576c19d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.790+0000 7f5771000700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f576c06a750 0x7f576c199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.790+0000 7f5771000700 1 --2- 172.21.15.6:0/2749350441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f576c06b0f0 0x7f576c199650 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.790+0000 7f5771000700 1 -- 172.21.15.6:0/2749350441 >> 172.21.15.6:0/2749350441 conn(0x7f576c0788f0 msgr2=0x7f576c0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:29.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.790+0000 7f5771000700 1 -- 172.21.15.6:0/2749350441 shutdown_connections 2024-04-26T02:03:29.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:29.790+0000 7f5771000700 1 -- 172.21.15.6:0/2749350441 wait complete. 2024-04-26T02:03:29.789 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 53 2024-04-26T02:03:30.118 INFO:tasks.workunit.client.0.smithi006.stdout: 10 80857 23.14 MB/sec execute 113 sec latency 78.616 ms 2024-04-26T02:03:30.657 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db / --timeout 60 --expiration 82.55 --await-for 42.0' 2024-04-26T02:03:30.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db / --timeout 60 --expiration 82.55 --await-for 42.0 2024-04-26T02:03:30.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:30 smithi006 ceph-mon[28419]: pgmap v559: 97 pgs: 97 active+clean; 793 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 200 op/s 2024-04-26T02:03:30.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:30 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2749350441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:03:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:30 smithi188 ceph-mon[32751]: pgmap v559: 97 pgs: 97 active+clean; 793 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 200 op/s 2024-04-26T02:03:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:30 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2749350441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:03:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:30 smithi029 ceph-mon[32775]: pgmap v559: 97 pgs: 97 active+clean; 793 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 200 op/s 2024-04-26T02:03:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:30 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2749350441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:03:31.118 INFO:tasks.workunit.client.0.smithi006.stdout: 10 81581 23.24 MB/sec execute 114 sec latency 361.387 ms 2024-04-26T02:03:31.177 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:03:32.118 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82203 23.48 MB/sec execute 115 sec latency 1361.463 ms 2024-04-26T02:03:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:32 smithi006 ceph-mon[28419]: pgmap v560: 97 pgs: 97 active+clean; 827 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 9.1 MiB/s wr, 476 op/s 2024-04-26T02:03:32.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:32 smithi188 ceph-mon[32751]: pgmap v560: 97 pgs: 97 active+clean; 827 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 9.1 MiB/s wr, 476 op/s 2024-04-26T02:03:32.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:32 smithi029 ceph-mon[32775]: pgmap v560: 97 pgs: 97 active+clean; 827 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 9.1 MiB/s wr, 476 op/s 2024-04-26T02:03:33.118 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82568 23.44 MB/sec execute 116 sec latency 2361.535 ms 2024-04-26T02:03:33.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.390+0000 7f13e0200700 1 -- 172.21.15.6:0/2503574659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d81034b0 msgr2=0x7f13d810f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.390+0000 7f13e0200700 1 --2- 172.21.15.6:0/2503574659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d81034b0 0x7f13d810f920 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009fc0 tx=0x7f13d000b330 comp rx=0 tx=0).stop 2024-04-26T02:03:33.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.390+0000 7f13e0200700 1 -- 172.21.15.6:0/2503574659 shutdown_connections 2024-04-26T02:03:33.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.390+0000 7f13e0200700 1 --2- 172.21.15.6:0/2503574659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d81034b0 0x7f13d810f920 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.390+0000 7f13e0200700 1 --2- 172.21.15.6:0/2503574659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13d8107940 0x7f13d8107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.390+0000 7f13e0200700 1 --2- 172.21.15.6:0/2503574659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13d8102af0 0x7f13d8102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.390+0000 7f13e0200700 1 -- 172.21.15.6:0/2503574659 >> 172.21.15.6:0/2503574659 conn(0x7f13d8079ba0 msgr2=0x7f13d8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:33.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.390+0000 7f13e0200700 1 -- 172.21.15.6:0/2503574659 shutdown_connections 2024-04-26T02:03:33.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.390+0000 7f13e0200700 1 -- 172.21.15.6:0/2503574659 wait complete. 2024-04-26T02:03:33.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.391+0000 7f13e0200700 1 Processor -- start 2024-04-26T02:03:33.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.392+0000 7f13e0200700 1 -- start start 2024-04-26T02:03:33.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.392+0000 7f13e0200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13d8102af0 0x7f13d8109910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.392+0000 7f13e0200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d81034b0 0x7f13d8109e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.392+0000 7f13e0200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13d8107940 0x7f13d810a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.392+0000 7f13e0200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8112c90 con 0x7f13d8102af0 2024-04-26T02:03:33.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.392+0000 7f13e0200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8112b10 con 0x7f13d81034b0 2024-04-26T02:03:33.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.392+0000 7f13e0200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8112e10 con 0x7f13d8107940 2024-04-26T02:03:33.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.393+0000 7f13dce00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d81034b0 0x7f13d8109e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.393+0000 7f13de200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13d8107940 0x7f13d810a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.393+0000 7f13dce00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d81034b0 0x7f13d8109e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:50610/0 (socket says 172.21.15.6:50610) 2024-04-26T02:03:33.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.393+0000 7f13dce00700 1 -- 172.21.15.6:0/2731174598 learned_addr learned my addr 172.21.15.6:0/2731174598 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:33.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.393+0000 7f13dd800700 1 --2- 172.21.15.6:0/2731174598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13d8102af0 0x7f13d8109910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.393+0000 7f13de200700 1 -- 172.21.15.6:0/2731174598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d81034b0 msgr2=0x7f13d8109e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.393+0000 7f13de200700 1 --2- 172.21.15.6:0/2731174598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d81034b0 0x7f13d8109e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.393+0000 7f13de200700 1 -- 172.21.15.6:0/2731174598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13d8102af0 msgr2=0x7f13d8109910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.393+0000 7f13de200700 1 --2- 172.21.15.6:0/2731174598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13d8102af0 0x7f13d8109910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.393+0000 7f13de200700 1 -- 172.21.15.6:0/2731174598 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d81ab940 con 0x7f13d8107940 2024-04-26T02:03:33.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.393+0000 7f13dd800700 1 --2- 172.21.15.6:0/2731174598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13d8102af0 0x7f13d8109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:33.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.393+0000 7f13dce00700 1 --2- 172.21.15.6:0/2731174598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d81034b0 0x7f13d8109e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:33.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.394+0000 7f13de200700 1 --2- 172.21.15.6:0/2731174598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13d8107940 0x7f13d810a390 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f13d000cc30 tx=0x7f13d0004f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:33.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.394+0000 7f13d6000700 1 -- 172.21.15.6:0/2731174598 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0021070 con 0x7f13d8107940 2024-04-26T02:03:33.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.394+0000 7f13d6000700 1 -- 172.21.15.6:0/2731174598 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13d0007ac0 con 0x7f13d8107940 2024-04-26T02:03:33.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.394+0000 7f13e0200700 1 -- 172.21.15.6:0/2731174598 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d81abbd0 con 0x7f13d8107940 2024-04-26T02:03:33.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.394+0000 7f13d6000700 1 -- 172.21.15.6:0/2731174598 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0005040 con 0x7f13d8107940 2024-04-26T02:03:33.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.394+0000 7f13e0200700 1 -- 172.21.15.6:0/2731174598 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d81ac080 con 0x7f13d8107940 2024-04-26T02:03:33.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.399+0000 7f13d6000700 1 -- 172.21.15.6:0/2731174598 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f13d0024070 con 0x7f13d8107940 2024-04-26T02:03:33.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.401+0000 7f13eab87b80 1 Processor -- start 2024-04-26T02:03:33.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.402+0000 7f13eab87b80 1 -- start start 2024-04-26T02:03:33.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.402+0000 7f13eab87b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 0x55d9c533bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.402+0000 7f13d6000700 1 --2- 172.21.15.6:0/2731174598 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f13c00a3ed0 0x7f13c00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.402+0000 7f13eab87b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c533c2d0 0x55d9c5346c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.402+0000 7f13eab87b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c5347180 0x55d9c5349570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.402+0000 7f13eab87b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d9c528f020 con 0x55d9c533b9b0 2024-04-26T02:03:33.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.402+0000 7f13eab87b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d9c52358c0 con 0x55d9c533c2d0 2024-04-26T02:03:33.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.402+0000 7f13eab87b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d9c528afe0 con 0x55d9c5347180 2024-04-26T02:03:33.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.403+0000 7f13d6000700 1 -- 172.21.15.6:0/2731174598 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7f13d000f020 con 0x7f13d8107940 2024-04-26T02:03:33.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.403+0000 7f13de200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c5347180 0x55d9c5349570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.403+0000 7f13de200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c5347180 0x55d9c5349570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38068/0 (socket says 172.21.15.6:38068) 2024-04-26T02:03:33.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.403+0000 7f13de200700 1 -- 172.21.15.6:0/2113699770 learned_addr learned my addr 172.21.15.6:0/2113699770 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:33.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.403+0000 7f13de200700 1 -- 172.21.15.6:0/2113699770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c533c2d0 msgr2=0x55d9c5346c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T02:03:33.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.403+0000 7f13de200700 1 --2- 172.21.15.6:0/2113699770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c533c2d0 0x55d9c5346c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.403+0000 7f13de200700 1 -- 172.21.15.6:0/2113699770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 msgr2=0x55d9c533bd90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:03:33.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.403+0000 7f13de200700 1 --2- 172.21.15.6:0/2113699770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 0x55d9c533bd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.403+0000 7f13de200700 1 -- 172.21.15.6:0/2113699770 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9c522b810 con 0x55d9c5347180 2024-04-26T02:03:33.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.404+0000 7f13de200700 1 --2- 172.21.15.6:0/2113699770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c5347180 0x55d9c5349570 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f13d009a4a0 tx=0x7f13d009d4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98485f610dbc9cc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:33.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.404+0000 7f13caa00700 1 -- 172.21.15.6:0/2113699770 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d009dad0 con 0x55d9c5347180 2024-04-26T02:03:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.404+0000 7f13dce00700 1 --2- 172.21.15.6:0/2731174598 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f13c00a3ed0 0x7f13c00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.405+0000 7f13caa00700 1 -- 172.21.15.6:0/2113699770 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13d00a1e80 con 0x55d9c5347180 2024-04-26T02:03:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.405+0000 7f13caa00700 1 -- 172.21.15.6:0/2113699770 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d00cf470 con 0x55d9c5347180 2024-04-26T02:03:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.406+0000 7f13eab87b80 1 -- 172.21.15.6:0/2113699770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c5347180 msgr2=0x55d9c5349570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.406+0000 7f13eab87b80 1 --2- 172.21.15.6:0/2113699770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c5347180 0x55d9c5349570 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f13d009a4a0 tx=0x7f13d009d4b0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.406+0000 7f13eab87b80 1 -- 172.21.15.6:0/2113699770 shutdown_connections 2024-04-26T02:03:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.406+0000 7f13eab87b80 1 --2- 172.21.15.6:0/2113699770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c5347180 0x55d9c5349570 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.406+0000 7f13eab87b80 1 --2- 172.21.15.6:0/2113699770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 0x55d9c533bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.406+0000 7f13eab87b80 1 --2- 172.21.15.6:0/2113699770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c533c2d0 0x55d9c5346c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.406+0000 7f13eab87b80 1 -- 172.21.15.6:0/2113699770 >> 172.21.15.6:0/2113699770 conn(0x55d9c526a730 msgr2=0x55d9c5337e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.407+0000 7f13eab87b80 1 -- 172.21.15.6:0/2113699770 shutdown_connections 2024-04-26T02:03:33.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.407+0000 7f13eab87b80 1 -- 172.21.15.6:0/2113699770 wait complete. 2024-04-26T02:03:33.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.408+0000 7f13eab87b80 1 -- start start 2024-04-26T02:03:33.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.408+0000 7f13dce00700 1 --2- 172.21.15.6:0/2731174598 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f13c00a3ed0 0x7f13c00a6390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009fd0 tx=0x7f13c4009310 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:33.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.408+0000 7f13eab87b80 1 Processor -- start 2024-04-26T02:03:33.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.409+0000 7f13eab87b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 0x55d9c53468a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.409+0000 7f13eab87b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533c2d0 0x55d9c5343ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.409+0000 7f13eab87b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c53440e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.409+0000 7f13eab87b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55d9c535aac0 con 0x55d9c533b9b0 2024-04-26T02:03:33.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.409+0000 7f13eab87b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d9c52489f0 con 0x55d9c5347180 2024-04-26T02:03:33.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.409+0000 7f13eab87b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d9c535ac40 con 0x55d9c533c2d0 2024-04-26T02:03:33.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.409+0000 7f13eab87b80 10 client.0 _ll_register_callbacks cb 0x55d9c52278b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:33.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.410+0000 7f13dce00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533c2d0 0x55d9c5343ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.410+0000 7f13dce00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533c2d0 0x55d9c5343ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38074/0 (socket says 172.21.15.6:38074) 2024-04-26T02:03:33.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.410+0000 7f13dce00700 1 -- 172.21.15.6:0/3400187669 learned_addr learned my addr 172.21.15.6:0/3400187669 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:33.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.410+0000 7f13dce00700 1 -- 172.21.15.6:0/3400187669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 msgr2=0x55d9c53440e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:03:33.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.410+0000 7f13dce00700 1 --2- 172.21.15.6:0/3400187669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c53440e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.410+0000 7f13dce00700 1 -- 172.21.15.6:0/3400187669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 msgr2=0x55d9c53468a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.410+0000 7f13dce00700 1 --2- 172.21.15.6:0/3400187669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 0x55d9c53468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.410+0000 7f13dce00700 1 -- 172.21.15.6:0/3400187669 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9c51dca60 con 0x55d9c533c2d0 2024-04-26T02:03:33.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.410+0000 7f13dce00700 1 --2- 172.21.15.6:0/3400187669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533c2d0 0x55d9c5343ba0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f13c400ece0 tx=0x7f13c40047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb14c051b26d6163 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:33.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.410+0000 7f13bf400700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:03:33.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.412+0000 7f13bf400700 1 -- 172.21.15.6:0/3400187669 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c401cdc0 con 0x55d9c533c2d0 2024-04-26T02:03:33.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.412+0000 7f13bf400700 1 -- 172.21.15.6:0/3400187669 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7f13c4004830 con 0x55d9c533c2d0 2024-04-26T02:03:33.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.412+0000 7f13eab87b80 1 -- 172.21.15.6:0/3400187669 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13d00a1e80 con 0x55d9c533c2d0 2024-04-26T02:03:33.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.412+0000 7f13bf400700 1 -- 172.21.15.6:0/3400187669 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f13c4004a30 con 0x55d9c533c2d0 2024-04-26T02:03:33.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.413+0000 7f13eab87b80 10 client.27301 fetch_fsmap learned FSMap version 53 2024-04-26T02:03:33.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.413+0000 7f13eab87b80 1 -- 172.21.15.6:0/3400187669 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55d9c5225c80 con 0x55d9c533c2d0 2024-04-26T02:03:33.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.414+0000 7f13bf400700 1 -- 172.21.15.6:0/3400187669 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13d00a1e80 con 0x55d9c533c2d0 2024-04-26T02:03:33.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.414+0000 7f13bf400700 1 -- 172.21.15.6:0/3400187669 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f13c4031650 con 0x55d9c533c2d0 2024-04-26T02:03:33.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.414+0000 7f13eab87b80 10 client.27301 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T02:03:33.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.414+0000 7f13eab87b80 10 client.27301 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:03:33.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.414+0000 7f13eab87b80 20 client.27301 populate_metadata read hostname 'smithi006' 2024-04-26T02:03:33.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.414+0000 7f13eab87b80 1 --2- 172.21.15.6:0/3400187669 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d9c5367ab0 0x55d9c5369f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.414+0000 7f13eab87b80 4 client.27301 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:03:33.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.414+0000 7f13eab87b80 1 -- 172.21.15.6:0/3400187669 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9c529c810 con 0x55d9c5367ab0 2024-04-26T02:03:33.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.415+0000 7f13dd800700 1 --2- 172.21.15.6:0/3400187669 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d9c5367ab0 0x55d9c5369f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.416+0000 7f13dd800700 1 --2- 172.21.15.6:0/3400187669 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d9c5367ab0 0x55d9c5369f70 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d25c52d40d15d58c server_cookie=be17be0a991f605c in_seq=0 out_seq=0 2024-04-26T02:03:33.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.416+0000 7f13bf400700 10 client.27301 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.417+0000 7f13bf400700 1 -- 172.21.15.6:0/3400187669 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9c529c810 con 0x55d9c5367ab0 2024-04-26T02:03:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.417+0000 7f13bf400700 10 client.27301 handle_command_reply: tid=0 2024-04-26T02:03:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.418+0000 7f13eab87b80 1 client.27301 shutdown 2024-04-26T02:03:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.418+0000 7f13eab87b80 1 -- 172.21.15.6:0/3400187669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533c2d0 msgr2=0x55d9c5343ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.418+0000 7f13eab87b80 1 --2- 172.21.15.6:0/3400187669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533c2d0 0x55d9c5343ba0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f13c400ece0 tx=0x7f13c40047c0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.418+0000 7f13eab87b80 1 -- 172.21.15.6:0/3400187669 shutdown_connections 2024-04-26T02:03:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.419+0000 7f13eab87b80 1 --2- 172.21.15.6:0/3400187669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c53440e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.419+0000 7f13eab87b80 1 --2- 172.21.15.6:0/3400187669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 0x55d9c53468a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.419+0000 7f13eab87b80 1 --2- 172.21.15.6:0/3400187669 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d9c5367ab0 0x55d9c5369f70 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.419+0000 7f13eab87b80 1 --2- 172.21.15.6:0/3400187669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533c2d0 0x55d9c5343ba0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.419+0000 7f13eab87b80 1 -- 172.21.15.6:0/3400187669 >> 172.21.15.6:0/3400187669 conn(0x55d9c526a730 msgr2=0x55d9c5337a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:33.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.419+0000 7f13bf400700 0 client.27301 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:33.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.419+0000 7f13eab87b80 1 -- 172.21.15.6:0/3400187669 shutdown_connections 2024-04-26T02:03:33.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.420+0000 7f13eab87b80 1 -- 172.21.15.6:0/3400187669 wait complete. 2024-04-26T02:03:33.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.420+0000 7f13eab87b80 20 client.27301 trim_cache size 0 max 16384 2024-04-26T02:03:33.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.452+0000 7f13eab87b80 1 Processor -- start 2024-04-26T02:03:33.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.452+0000 7f13eab87b80 1 -- start start 2024-04-26T02:03:33.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.452+0000 7f13eab87b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 0x55d9c536f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.452+0000 7f13eab87b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c536fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.452+0000 7f13eab87b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c5342ba0 0x55d9c535faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.452+0000 7f13eab87b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f13d81ad970 con 0x55d9c533b9b0 2024-04-26T02:03:33.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.452+0000 7f13eab87b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8073c20 con 0x55d9c5347180 2024-04-26T02:03:33.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.452+0000 7f13eab87b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8071a90 con 0x55d9c5342ba0 2024-04-26T02:03:33.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.453+0000 7f13dce00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c536fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.453+0000 7f13dd800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 0x55d9c536f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.453+0000 7f13de200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c5342ba0 0x55d9c535faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.453+0000 7f13dce00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c536fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:50634/0 (socket says 172.21.15.6:50634) 2024-04-26T02:03:33.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.453+0000 7f13dce00700 1 -- 172.21.15.6:0/4013833217 learned_addr learned my addr 172.21.15.6:0/4013833217 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:33.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.453+0000 7f13dce00700 1 -- 172.21.15.6:0/4013833217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c5342ba0 msgr2=0x55d9c535faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.453+0000 7f13dce00700 1 --2- 172.21.15.6:0/4013833217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c5342ba0 0x55d9c535faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.453+0000 7f13dce00700 1 -- 172.21.15.6:0/4013833217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 msgr2=0x55d9c536f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.453+0000 7f13dce00700 1 --2- 172.21.15.6:0/4013833217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 0x55d9c536f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.453+0000 7f13dce00700 1 -- 172.21.15.6:0/4013833217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9c5365150 con 0x55d9c5347180 2024-04-26T02:03:33.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.453+0000 7f13dce00700 1 --2- 172.21.15.6:0/4013833217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c536fdc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f13c4005950 tx=0x7f13c4004710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e77c3e237db9e058 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:33.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.454+0000 7f13ca000700 1 -- 172.21.15.6:0/4013833217 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4031420 con 0x55d9c5347180 2024-04-26T02:03:33.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.454+0000 7f13ca000700 1 -- 172.21.15.6:0/4013833217 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13c4036070 con 0x55d9c5347180 2024-04-26T02:03:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.454+0000 7f13ca000700 1 -- 172.21.15.6:0/4013833217 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c401d900 con 0x55d9c5347180 2024-04-26T02:03:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.455+0000 7f13eab87b80 1 -- 172.21.15.6:0/4013833217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 msgr2=0x55d9c536fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.455+0000 7f13eab87b80 1 --2- 172.21.15.6:0/4013833217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c536fdc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f13c4005950 tx=0x7f13c4004710 comp rx=0 tx=0).stop 2024-04-26T02:03:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.455+0000 7f13eab87b80 1 -- 172.21.15.6:0/4013833217 shutdown_connections 2024-04-26T02:03:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.455+0000 7f13eab87b80 1 --2- 172.21.15.6:0/4013833217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c5342ba0 0x55d9c535faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.455+0000 7f13eab87b80 1 --2- 172.21.15.6:0/4013833217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c533b9b0 0x55d9c536f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.455+0000 7f13eab87b80 1 --2- 172.21.15.6:0/4013833217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c536fdc0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.455+0000 7f13eab87b80 1 -- 172.21.15.6:0/4013833217 >> 172.21.15.6:0/4013833217 conn(0x55d9c526a730 msgr2=0x55d9c5342490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.455+0000 7f13eab87b80 1 -- 172.21.15.6:0/4013833217 shutdown_connections 2024-04-26T02:03:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.455+0000 7f13eab87b80 1 -- 172.21.15.6:0/4013833217 wait complete. 2024-04-26T02:03:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.455+0000 7f13eab87b80 1 -- start start 2024-04-26T02:03:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.455+0000 7f13eab87b80 1 Processor -- start 2024-04-26T02:03:33.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.456+0000 7f13eab87b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533b9b0 0x55d9c535fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.456+0000 7f13eab87b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c5342ba0 0x55d9c535c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.456+0000 7f13eab87b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c535c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.456+0000 7f13eab87b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f13d80720e0 con 0x55d9c5342ba0 2024-04-26T02:03:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.456+0000 7f13eab87b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8073550 con 0x55d9c5347180 2024-04-26T02:03:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.456+0000 7f13eab87b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f13d80727b0 con 0x55d9c533b9b0 2024-04-26T02:03:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.456+0000 7f13eab87b80 10 client.0 _ll_register_callbacks cb 0x55d9c52278b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.456+0000 7f13dd800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533b9b0 0x55d9c535fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.456+0000 7f13dd800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533b9b0 0x55d9c535fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:38096/0 (socket says 172.21.15.6:38096) 2024-04-26T02:03:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.456+0000 7f13dd800700 1 -- 172.21.15.6:0/3939197746 learned_addr learned my addr 172.21.15.6:0/3939197746 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.456+0000 7f13de200700 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c535c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.457+0000 7f13dce00700 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c5342ba0 0x55d9c535c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.457+0000 7f13de200700 1 -- 172.21.15.6:0/3939197746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533b9b0 msgr2=0x55d9c535fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.457+0000 7f13de200700 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533b9b0 0x55d9c535fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.457+0000 7f13de200700 1 -- 172.21.15.6:0/3939197746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c5342ba0 msgr2=0x55d9c535c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.457+0000 7f13de200700 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c5342ba0 0x55d9c535c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.457+0000 7f13de200700 1 -- 172.21.15.6:0/3939197746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9c535cf80 con 0x55d9c5347180 2024-04-26T02:03:33.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.457+0000 7f13de200700 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c535c740 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f13d009a4a0 tx=0x7f13d009bfe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b4bdc8035c5e97c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:33.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.459+0000 7f13bf400700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:03:33.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.459+0000 7f13bf400700 1 -- 172.21.15.6:0/3939197746 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d00b6070 con 0x55d9c5347180 2024-04-26T02:03:33.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.459+0000 7f13bf400700 1 -- 172.21.15.6:0/3939197746 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7f13d0007400 con 0x55d9c5347180 2024-04-26T02:03:33.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.459+0000 7f13eab87b80 1 -- 172.21.15.6:0/3939197746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13c4036070 con 0x55d9c5347180 2024-04-26T02:03:33.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.459+0000 7f13bf400700 1 -- 172.21.15.6:0/3939197746 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f13d009dad0 con 0x55d9c5347180 2024-04-26T02:03:33.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.459+0000 7f13eab87b80 10 client.27248 fetch_fsmap learned FSMap version 53 2024-04-26T02:03:33.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.459+0000 7f13eab87b80 1 -- 172.21.15.6:0/3939197746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55d9c53cf090 con 0x55d9c5347180 2024-04-26T02:03:33.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.460+0000 7f13bf400700 1 -- 172.21.15.6:0/3939197746 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13c4036070 con 0x55d9c5347180 2024-04-26T02:03:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.461+0000 7f13bf400700 1 -- 172.21.15.6:0/3939197746 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f13d0011070 con 0x55d9c5347180 2024-04-26T02:03:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.461+0000 7f13eab87b80 10 client.27248 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T02:03:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.461+0000 7f13eab87b80 10 client.27248 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:03:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.461+0000 7f13eab87b80 20 client.27248 populate_metadata read hostname 'smithi006' 2024-04-26T02:03:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.461+0000 7f13eab87b80 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d9c53cf510 0x55d9c53d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.461+0000 7f13eab87b80 4 client.27248 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.55, "await_for": 42.0}] 2024-04-26T02:03:33.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.461+0000 7f13eab87b80 1 -- 172.21.15.6:0/3939197746 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.55, "await_for": 42.0}) -- 0x55d9c52968b0 con 0x55d9c53cf510 2024-04-26T02:03:33.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.461+0000 7f13dd800700 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d9c53cf510 0x55d9c53d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.462+0000 7f13dd800700 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d9c53cf510 0x55d9c53d1900 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a403666726689b5 server_cookie=c310af2e6c7a85df in_seq=0 out_seq=0 2024-04-26T02:03:33.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:03:33.462+0000 7f13bf400700 10 client.27248 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:34.119 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 23.28 MB/sec execute 117 sec latency 3361.670 ms 2024-04-26T02:03:34.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:34 smithi006 ceph-mon[28419]: pgmap v561: 97 pgs: 97 active+clean; 827 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 6.8 MiB/s wr, 395 op/s 2024-04-26T02:03:34.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:34 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:03:34.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:34 smithi188 ceph-mon[32751]: pgmap v561: 97 pgs: 97 active+clean; 827 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 6.8 MiB/s wr, 395 op/s 2024-04-26T02:03:34.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:34 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:03:34.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:34 smithi029 ceph-mon[32775]: pgmap v561: 97 pgs: 97 active+clean; 827 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 6.8 MiB/s wr, 395 op/s 2024-04-26T02:03:34.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:34 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:03:35.118 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 23.09 MB/sec execute 118 sec latency 4361.799 ms 2024-04-26T02:03:36.118 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 22.89 MB/sec execute 119 sec latency 5361.939 ms 2024-04-26T02:03:36.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:36 smithi006 ceph-mon[28419]: pgmap v562: 97 pgs: 97 active+clean; 827 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 6.8 MiB/s wr, 395 op/s 2024-04-26T02:03:36.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:36 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:03:36.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:36 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:03:36.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:36 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:03:36.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:36 smithi188 ceph-mon[32751]: pgmap v562: 97 pgs: 97 active+clean; 827 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 6.8 MiB/s wr, 395 op/s 2024-04-26T02:03:36.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:36 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:03:36.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:36 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:03:36.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:36 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:03:36.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:36 smithi029 ceph-mon[32775]: pgmap v562: 97 pgs: 97 active+clean; 827 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 6.8 MiB/s wr, 395 op/s 2024-04-26T02:03:36.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:36 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:03:36.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:36 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:03:36.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:36 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:03:37.119 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 22.70 MB/sec execute 120 sec latency 6362.094 ms 2024-04-26T02:03:38.119 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 22.51 MB/sec execute 121 sec latency 7362.247 ms 2024-04-26T02:03:38.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:38 smithi006 ceph-mon[28419]: pgmap v563: 97 pgs: 97 active+clean; 839 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.0 MiB/s wr, 509 op/s 2024-04-26T02:03:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:38 smithi188 ceph-mon[32751]: pgmap v563: 97 pgs: 97 active+clean; 839 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.0 MiB/s wr, 509 op/s 2024-04-26T02:03:38.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:38 smithi029 ceph-mon[32775]: pgmap v563: 97 pgs: 97 active+clean; 839 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.0 MiB/s wr, 509 op/s 2024-04-26T02:03:39.119 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 22.33 MB/sec execute 122 sec latency 8362.414 ms 2024-04-26T02:03:40.119 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 22.15 MB/sec execute 123 sec latency 9362.549 ms 2024-04-26T02:03:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.258+0000 7f43fa200640 1 -- 172.21.15.6:0/977904858 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f410e9a0 msgr2=0x7f43f4110d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.258+0000 7f43fa200640 1 --2- 172.21.15.6:0/977904858 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f410e9a0 0x7f43f4110d90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f43ec009ca0 tx=0x7f43ec02a8c0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.259+0000 7f43fa200640 1 -- 172.21.15.6:0/977904858 shutdown_connections 2024-04-26T02:03:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.259+0000 7f43fa200640 1 --2- 172.21.15.6:0/977904858 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f41017a0 0x7f43f410e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.259+0000 7f43fa200640 1 --2- 172.21.15.6:0/977904858 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f410e9a0 0x7f43f4110d90 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.259+0000 7f43fa200640 1 --2- 172.21.15.6:0/977904858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f4100df0 0x7f43f41011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.259+0000 7f43fa200640 1 -- 172.21.15.6:0/977904858 >> 172.21.15.6:0/977904858 conn(0x7f43f4078000 msgr2=0x7f43f4078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:40.259 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.259+0000 7f43fa200640 1 -- 172.21.15.6:0/977904858 shutdown_connections 2024-04-26T02:03:40.259 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.259+0000 7f43fa200640 1 -- 172.21.15.6:0/977904858 wait complete. 2024-04-26T02:03:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.260+0000 7f43fa200640 1 Processor -- start 2024-04-26T02:03:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.260+0000 7f43fa200640 1 -- start start 2024-04-26T02:03:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.260+0000 7f43fa200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f4100df0 0x7f43f41a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.260+0000 7f43fa200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f41017a0 0x7f43f41a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.260+0000 7f43fa200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f410e9a0 0x7f43f419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.260+0000 7f43fa200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f43f4114550 con 0x7f43f41017a0 2024-04-26T02:03:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.260+0000 7f43fa200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41143d0 con 0x7f43f4100df0 2024-04-26T02:03:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.260+0000 7f43fa200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41146d0 con 0x7f43f410e9a0 2024-04-26T02:03:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.261+0000 7f43ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f41017a0 0x7f43f41a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.261+0000 7f43ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f41017a0 0x7f43f41a0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60346/0 (socket says 172.21.15.6:60346) 2024-04-26T02:03:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.261+0000 7f43ebe00640 1 -- 172.21.15.6:0/1180198814 learned_addr learned my addr 172.21.15.6:0/1180198814 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.261+0000 7f43f3e00640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f410e9a0 0x7f43f419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.261+0000 7f43f3400640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f4100df0 0x7f43f41a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.261+0000 7f43ebe00640 1 -- 172.21.15.6:0/1180198814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f410e9a0 msgr2=0x7f43f419a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.261+0000 7f43ebe00640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f410e9a0 0x7f43f419a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.261+0000 7f43ebe00640 1 -- 172.21.15.6:0/1180198814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f4100df0 msgr2=0x7f43f41a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.261+0000 7f43ebe00640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f4100df0 0x7f43f41a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.261+0000 7f43ebe00640 1 -- 172.21.15.6:0/1180198814 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f419b030 con 0x7f43f41017a0 2024-04-26T02:03:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.262+0000 7f43ebe00640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f41017a0 0x7f43f41a0c40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00e9f0 tx=0x7f43dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.262+0000 7f43f0c00640 1 -- 172.21.15.6:0/1180198814 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc00cd80 con 0x7f43f41017a0 2024-04-26T02:03:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.262+0000 7f43fa200640 1 -- 172.21.15.6:0/1180198814 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f419b320 con 0x7f43f41017a0 2024-04-26T02:03:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.262+0000 7f43f0c00640 1 -- 172.21.15.6:0/1180198814 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43dc0108e0 con 0x7f43f41017a0 2024-04-26T02:03:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.262+0000 7f43f0c00640 1 -- 172.21.15.6:0/1180198814 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc018980 con 0x7f43f41017a0 2024-04-26T02:03:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.262+0000 7f43fa200640 1 -- 172.21.15.6:0/1180198814 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f41a75c0 con 0x7f43f41017a0 2024-04-26T02:03:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.266+0000 7f43fcf89740 1 Processor -- start 2024-04-26T02:03:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.266+0000 7f43f0c00640 1 -- 172.21.15.6:0/1180198814 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f43dc010430 con 0x7f43f41017a0 2024-04-26T02:03:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.266+0000 7f43fcf89740 1 -- start start 2024-04-26T02:03:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.266+0000 7f43fcf89740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e681ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.266+0000 7f43fcf89740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e68294c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.266+0000 7f43fcf89740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e682bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.266+0000 7f43fcf89740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55f2e6784900 con 0x55f2e6829a00 2024-04-26T02:03:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.266+0000 7f43fcf89740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f2e672aef0 con 0x55f2e681f140 2024-04-26T02:03:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.266+0000 7f43fcf89740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f2e67808c0 con 0x55f2e681e820 2024-04-26T02:03:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.267+0000 7f43f3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e681ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.267+0000 7f43f3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e681ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34478/0 (socket says 172.21.15.6:34478) 2024-04-26T02:03:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.267+0000 7f43f3e00640 1 -- 172.21.15.6:0/4285819312 learned_addr learned my addr 172.21.15.6:0/4285819312 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.267+0000 7f43f3400640 1 --2- 172.21.15.6:0/4285819312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e68294c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.267+0000 7f43ebe00640 1 --2- 172.21.15.6:0/4285819312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e682bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.267+0000 7f43f3e00640 1 -- 172.21.15.6:0/4285819312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 msgr2=0x55f2e68294c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.267+0000 7f43f3e00640 1 --2- 172.21.15.6:0/4285819312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e68294c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.267+0000 7f43f3e00640 1 -- 172.21.15.6:0/4285819312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 msgr2=0x55f2e682bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.267+0000 7f43f3e00640 1 --2- 172.21.15.6:0/4285819312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e682bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.267+0000 7f43f3e00640 1 -- 172.21.15.6:0/4285819312 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2e6497210 con 0x55f2e681e820 2024-04-26T02:03:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.267+0000 7f43f3e00640 1 --2- 172.21.15.6:0/4285819312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e681ec00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f43f4101d40 tx=0x7f43ec005fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1c6b81e93aeabb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.268+0000 7f43f0c00640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43d80a3d90 0x7f43d80a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.268+0000 7f43e3e00640 1 -- 172.21.15.6:0/4285819312 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec031cf0 con 0x55f2e681e820 2024-04-26T02:03:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.268+0000 7f43e3e00640 1 -- 172.21.15.6:0/4285819312 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43ec036030 con 0x55f2e681e820 2024-04-26T02:03:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.268+0000 7f43e3e00640 1 -- 172.21.15.6:0/4285819312 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec0042a0 con 0x55f2e681e820 2024-04-26T02:03:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.268+0000 7f43f0c00640 1 -- 172.21.15.6:0/1180198814 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7f43dc0d0a80 con 0x7f43f41017a0 2024-04-26T02:03:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.268+0000 7f43f3400640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43d80a3d90 0x7f43d80a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.268+0000 7f43fcf89740 1 -- 172.21.15.6:0/4285819312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 msgr2=0x55f2e681ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.268+0000 7f43fcf89740 1 --2- 172.21.15.6:0/4285819312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e681ec00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f43f4101d40 tx=0x7f43ec005fa0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.269+0000 7f43fcf89740 1 -- 172.21.15.6:0/4285819312 shutdown_connections 2024-04-26T02:03:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.269+0000 7f43fcf89740 1 --2- 172.21.15.6:0/4285819312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e682bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.269+0000 7f43fcf89740 1 --2- 172.21.15.6:0/4285819312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e68294c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.269+0000 7f43fcf89740 1 --2- 172.21.15.6:0/4285819312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e681ec00 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.269+0000 7f43fcf89740 1 -- 172.21.15.6:0/4285819312 >> 172.21.15.6:0/4285819312 conn(0x55f2e681ad30 msgr2=0x55f2e681b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.269+0000 7f43f3400640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43d80a3d90 0x7f43d80a6250 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55f2e681fdc0 tx=0x7f43e4005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.269+0000 7f43fcf89740 1 -- 172.21.15.6:0/4285819312 shutdown_connections 2024-04-26T02:03:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.269+0000 7f43fcf89740 1 -- 172.21.15.6:0/4285819312 wait complete. 2024-04-26T02:03:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.269+0000 7f43fcf89740 1 -- start start 2024-04-26T02:03:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.269+0000 7f43fcf89740 1 Processor -- start 2024-04-26T02:03:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43fcf89740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e6826990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43fcf89740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e6826ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43fcf89740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e6827410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43fcf89740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55f2e682c450 con 0x55f2e6829a00 2024-04-26T02:03:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43fcf89740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f2e673e310 con 0x55f2e681f140 2024-04-26T02:03:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43fcf89740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f2e682c5d0 con 0x55f2e681e820 2024-04-26T02:03:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43fcf89740 10 client.0 _ll_register_callbacks cb 0x55f2e670ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e6827410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43f3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e6826990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43f3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e6826990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34494/0 (socket says 172.21.15.6:34494) 2024-04-26T02:03:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e6827410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60374/0 (socket says 172.21.15.6:60374) 2024-04-26T02:03:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43f3400640 1 -- 172.21.15.6:0/3821180450 learned_addr learned my addr 172.21.15.6:0/3821180450 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.270+0000 7f43ebe00640 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e6826ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.271+0000 7f43ebe00640 1 -- 172.21.15.6:0/3821180450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 msgr2=0x55f2e6826990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.271+0000 7f43ebe00640 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e6826990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.271+0000 7f43ebe00640 1 -- 172.21.15.6:0/3821180450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 msgr2=0x55f2e6827410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.271+0000 7f43ebe00640 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e6827410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.271+0000 7f43ebe00640 1 -- 172.21.15.6:0/3821180450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2e6563620 con 0x55f2e681f140 2024-04-26T02:03:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.271+0000 7f43f3400640 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e6826990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.271+0000 7f43ebe00640 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e6826ed0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0ce120 tx=0x7f43dc09c620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=131a014e542fe170 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.271+0000 7f43e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:03:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.271+0000 7f43e0c00640 1 -- 172.21.15.6:0/3821180450 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc0ce5a0 con 0x55f2e681f140 2024-04-26T02:03:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.272+0000 7f43e0c00640 1 -- 172.21.15.6:0/3821180450 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7f43dc087110 con 0x55f2e681f140 2024-04-26T02:03:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.272+0000 7f43fcf89740 1 -- 172.21.15.6:0/3821180450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43ec036030 con 0x55f2e681f140 2024-04-26T02:03:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.272+0000 7f43e0c00640 1 -- 172.21.15.6:0/3821180450 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f43dc09ca20 con 0x55f2e681f140 2024-04-26T02:03:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.272+0000 7f43fcf89740 10 client.27260 fetch_fsmap learned FSMap version 53 2024-04-26T02:03:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.272+0000 7f43fcf89740 1 -- 172.21.15.6:0/3821180450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55f2e6838740 con 0x55f2e681f140 2024-04-26T02:03:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.272+0000 7f43e0c00640 1 -- 172.21.15.6:0/3821180450 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43ec036030 con 0x55f2e681f140 2024-04-26T02:03:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.272+0000 7f43e0c00640 1 -- 172.21.15.6:0/3821180450 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f43dc018b20 con 0x55f2e681f140 2024-04-26T02:03:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.272+0000 7f43fcf89740 10 client.27260 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T02:03:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.272+0000 7f43fcf89740 10 client.27260 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:03:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.272+0000 7f43fcf89740 20 client.27260 populate_metadata read hostname 'smithi006' 2024-04-26T02:03:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.272+0000 7f43fcf89740 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f2e6839bb0 0x55f2e683bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.273+0000 7f43fcf89740 4 client.27260 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:03:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.273+0000 7f43fcf89740 1 -- 172.21.15.6:0/3821180450 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2e67988b0 con 0x55f2e6839bb0 2024-04-26T02:03:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.273+0000 7f43f3400640 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f2e6839bb0 0x55f2e683bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.274+0000 7f43f3400640 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f2e6839bb0 0x55f2e683bfa0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86855ada200c268b server_cookie=8a6796b884145458 in_seq=0 out_seq=0 2024-04-26T02:03:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.274+0000 7f43e0c00640 10 client.27260 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.274+0000 7f43e0c00640 1 -- 172.21.15.6:0/3821180450 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f2e67988b0 con 0x55f2e6839bb0 2024-04-26T02:03:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.274+0000 7f43e0c00640 10 client.27260 handle_command_reply: tid=0 2024-04-26T02:03:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43fcf89740 1 client.27260 shutdown 2024-04-26T02:03:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43fcf89740 1 -- 172.21.15.6:0/3821180450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 msgr2=0x55f2e6826ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43fcf89740 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e6826ed0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0ce120 tx=0x7f43dc09c620 comp rx=0 tx=0).stop 2024-04-26T02:03:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43fcf89740 1 -- 172.21.15.6:0/3821180450 shutdown_connections 2024-04-26T02:03:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43fcf89740 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f2e6839bb0 0x55f2e683bfa0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43fcf89740 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e6827410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43fcf89740 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e6826ed0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43e0c00640 0 client.27260 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43fcf89740 1 --2- 172.21.15.6:0/3821180450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e6826990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43fcf89740 1 -- 172.21.15.6:0/3821180450 >> 172.21.15.6:0/3821180450 conn(0x55f2e681ad30 msgr2=0x55f2e6820a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43fcf89740 1 -- 172.21.15.6:0/3821180450 shutdown_connections 2024-04-26T02:03:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43fcf89740 1 -- 172.21.15.6:0/3821180450 wait complete. 2024-04-26T02:03:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.275+0000 7f43fcf89740 20 client.27260 trim_cache size 0 max 16384 2024-04-26T02:03:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.292+0000 7f43fcf89740 1 Processor -- start 2024-04-26T02:03:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.292+0000 7f43fcf89740 1 -- start start 2024-04-26T02:03:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.292+0000 7f43fcf89740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e681e820 0x55f2e6835310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.292+0000 7f43fcf89740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681f140 0x55f2e68211e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.292+0000 7f43fcf89740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e6829a00 0x55f2e681b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.292+0000 7f43fcf89740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41b27a0 con 0x55f2e681e820 2024-04-26T02:03:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.292+0000 7f43fcf89740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41b0070 con 0x55f2e6829a00 2024-04-26T02:03:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.292+0000 7f43fcf89740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41adee0 con 0x55f2e681f140 2024-04-26T02:03:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.292+0000 7f43f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e681e820 0x55f2e6835310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.292+0000 7f43f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e681e820 0x55f2e6835310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60388/0 (socket says 172.21.15.6:60388) 2024-04-26T02:03:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.292+0000 7f43f3400640 1 -- 172.21.15.6:0/43697407 learned_addr learned my addr 172.21.15.6:0/43697407 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.292+0000 7f43f3e00640 1 --2- 172.21.15.6:0/43697407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e6829a00 0x55f2e681b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.293+0000 7f43ebe00640 1 --2- 172.21.15.6:0/43697407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681f140 0x55f2e68211e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.293+0000 7f43f3400640 1 -- 172.21.15.6:0/43697407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681f140 msgr2=0x55f2e68211e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.293+0000 7f43f3400640 1 --2- 172.21.15.6:0/43697407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681f140 0x55f2e68211e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.293+0000 7f43f3400640 1 -- 172.21.15.6:0/43697407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e6829a00 msgr2=0x55f2e681b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.293+0000 7f43f3400640 1 --2- 172.21.15.6:0/43697407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e6829a00 0x55f2e681b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.293+0000 7f43f3400640 1 -- 172.21.15.6:0/43697407 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2e6825940 con 0x55f2e681e820 2024-04-26T02:03:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.293+0000 7f43ebe00640 1 --2- 172.21.15.6:0/43697407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681f140 0x55f2e68211e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.293+0000 7f43f3400640 1 --2- 172.21.15.6:0/43697407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e681e820 0x55f2e6835310 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x55f2e683c7f0 tx=0x7f43e401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68e4676f331fb6f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.293+0000 7f43e3400640 1 -- 172.21.15.6:0/43697407 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e402b4e0 con 0x55f2e681e820 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.293+0000 7f43e3400640 1 -- 172.21.15.6:0/43697407 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43e4030070 con 0x55f2e681e820 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.293+0000 7f43e3400640 1 -- 172.21.15.6:0/43697407 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e401e420 con 0x55f2e681e820 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.294+0000 7f43fcf89740 1 -- 172.21.15.6:0/43697407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e681e820 msgr2=0x55f2e6835310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.294+0000 7f43fcf89740 1 --2- 172.21.15.6:0/43697407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e681e820 0x55f2e6835310 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x55f2e683c7f0 tx=0x7f43e401b000 comp rx=0 tx=0).stop 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.294+0000 7f43fcf89740 1 -- 172.21.15.6:0/43697407 shutdown_connections 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.294+0000 7f43fcf89740 1 --2- 172.21.15.6:0/43697407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e6829a00 0x55f2e681b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.294+0000 7f43fcf89740 1 --2- 172.21.15.6:0/43697407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681f140 0x55f2e68211e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.294+0000 7f43fcf89740 1 --2- 172.21.15.6:0/43697407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e681e820 0x55f2e6835310 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.294+0000 7f43fcf89740 1 -- 172.21.15.6:0/43697407 >> 172.21.15.6:0/43697407 conn(0x55f2e681ad30 msgr2=0x55f2e6819e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.294+0000 7f43fcf89740 1 -- 172.21.15.6:0/43697407 shutdown_connections 2024-04-26T02:03:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.294+0000 7f43fcf89740 1 -- 172.21.15.6:0/43697407 wait complete. 2024-04-26T02:03:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.295+0000 7f43fcf89740 1 -- start start 2024-04-26T02:03:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.295+0000 7f43fcf89740 1 Processor -- start 2024-04-26T02:03:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.295+0000 7f43fcf89740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e682f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.295+0000 7f43fcf89740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e682f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.295+0000 7f43fcf89740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e682fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.295+0000 7f43fcf89740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41ae530 con 0x55f2e6829a00 2024-04-26T02:03:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.295+0000 7f43fcf89740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41af9a0 con 0x55f2e681f140 2024-04-26T02:03:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.295+0000 7f43fcf89740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41aec00 con 0x55f2e681e820 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.295+0000 7f43fcf89740 10 client.0 _ll_register_callbacks cb 0x7f43dc09c7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.295+0000 7f43f3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e682f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43f3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e682f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34514/0 (socket says 172.21.15.6:34514) 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43f3400640 1 -- 172.21.15.6:0/1110884473 learned_addr learned my addr 172.21.15.6:0/1110884473 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43ebe00640 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e682f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43f3e00640 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e682fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43f3400640 1 -- 172.21.15.6:0/1110884473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 msgr2=0x55f2e682f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43f3400640 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e682f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43f3400640 1 -- 172.21.15.6:0/1110884473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 msgr2=0x55f2e682fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43f3400640 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e682fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43f3400640 1 -- 172.21.15.6:0/1110884473 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2e683c310 con 0x55f2e681e820 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43f3e00640 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e682fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43f3400640 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e682f290 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f43e4005780 tx=0x7f43e402b4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=815ed0656a0e3c70 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.296+0000 7f43e0c00640 1 -- 172.21.15.6:0/1110884473 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e401f430 con 0x55f2e681e820 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43fcf89740 1 -- 172.21.15.6:0/1110884473 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43e4030070 con 0x55f2e681e820 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43e0c00640 1 -- 172.21.15.6:0/1110884473 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8600+0+0 (secure 0 0 0) 0x7f43e401f5d0 con 0x55f2e681e820 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43e0c00640 1 -- 172.21.15.6:0/1110884473 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f43e401f7d0 con 0x55f2e681e820 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43fcf89740 10 client.27322 fetch_fsmap learned FSMap version 53 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43fcf89740 1 -- 172.21.15.6:0/1110884473 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55f2e68a8a00 con 0x55f2e681e820 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43e0c00640 1 -- 172.21.15.6:0/1110884473 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43e4030070 con 0x55f2e681e820 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43e0c00640 1 -- 172.21.15.6:0/1110884473 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f43e403f9d0 con 0x55f2e681e820 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43fcf89740 10 client.27322 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43fcf89740 10 client.27322 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43fcf89740 20 client.27322 populate_metadata read hostname 'smithi006' 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43fcf89740 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f2e68a9130 0x55f2e68ab550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43fcf89740 4 client.27322 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.297+0000 7f43fcf89740 1 -- 172.21.15.6:0/1110884473 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f2e6792320 con 0x55f2e68a9130 2024-04-26T02:03:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.298+0000 7f43ebe00640 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f2e68a9130 0x55f2e68ab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.298+0000 7f43ebe00640 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f2e68a9130 0x55f2e68ab550 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=417701dd525beaa1 server_cookie=540d6166a4b6f7f1 in_seq=0 out_seq=0 2024-04-26T02:03:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:03:40.298+0000 7f43e0c00640 10 client.27322 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:03:40.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:40 smithi006 ceph-mon[28419]: pgmap v564: 97 pgs: 97 active+clean; 839 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.0 MiB/s wr, 391 op/s 2024-04-26T02:03:40.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:40 smithi188 ceph-mon[32751]: pgmap v564: 97 pgs: 97 active+clean; 839 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.0 MiB/s wr, 391 op/s 2024-04-26T02:03:40.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:40 smithi029 ceph-mon[32775]: pgmap v564: 97 pgs: 97 active+clean; 839 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.0 MiB/s wr, 391 op/s 2024-04-26T02:03:41.119 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 21.97 MB/sec execute 124 sec latency 10362.702 ms 2024-04-26T02:03:42.119 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 21.79 MB/sec execute 125 sec latency 11362.864 ms 2024-04-26T02:03:42.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:42 smithi006 ceph-mon[28419]: pgmap v565: 97 pgs: 97 active+clean; 847 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.7 MiB/s wr, 449 op/s 2024-04-26T02:03:42.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:42 smithi188 ceph-mon[32751]: pgmap v565: 97 pgs: 97 active+clean; 847 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.7 MiB/s wr, 449 op/s 2024-04-26T02:03:42.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:42 smithi029 ceph-mon[32775]: pgmap v565: 97 pgs: 97 active+clean; 847 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.7 MiB/s wr, 449 op/s 2024-04-26T02:03:43.119 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 21.62 MB/sec execute 126 sec latency 12363.017 ms 2024-04-26T02:03:44.120 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 21.45 MB/sec execute 127 sec latency 13363.175 ms 2024-04-26T02:03:44.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:44 smithi006 ceph-mon[28419]: pgmap v566: 97 pgs: 97 active+clean; 847 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-04-26T02:03:44.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:44 smithi188 ceph-mon[32751]: pgmap v566: 97 pgs: 97 active+clean; 847 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-04-26T02:03:44.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:44 smithi029 ceph-mon[32775]: pgmap v566: 97 pgs: 97 active+clean; 847 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-04-26T02:03:45.120 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 21.28 MB/sec execute 128 sec latency 14363.353 ms 2024-04-26T02:03:46.120 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 21.12 MB/sec execute 129 sec latency 15363.513 ms 2024-04-26T02:03:46.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:46 smithi006 ceph-mon[28419]: pgmap v567: 97 pgs: 97 active+clean; 847 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-04-26T02:03:46.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:46 smithi188 ceph-mon[32751]: pgmap v567: 97 pgs: 97 active+clean; 847 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-04-26T02:03:46.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:46 smithi029 ceph-mon[32775]: pgmap v567: 97 pgs: 97 active+clean; 847 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-04-26T02:03:47.120 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 20.95 MB/sec execute 130 sec latency 16363.674 ms 2024-04-26T02:03:48.121 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 20.79 MB/sec execute 131 sec latency 17363.830 ms 2024-04-26T02:03:48.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:48 smithi006 ceph-mon[28419]: pgmap v568: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.1 MiB/s wr, 181 op/s 2024-04-26T02:03:48.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:48 smithi188 ceph-mon[32751]: pgmap v568: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.1 MiB/s wr, 181 op/s 2024-04-26T02:03:48.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:48 smithi029 ceph-mon[32775]: pgmap v568: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.1 MiB/s wr, 181 op/s 2024-04-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 20.64 MB/sec execute 132 sec latency 18364.006 ms 2024-04-26T02:03:50.122 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 20.48 MB/sec execute 133 sec latency 19364.174 ms 2024-04-26T02:03:50.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:50 smithi006 ceph-mon[28419]: pgmap v569: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 929 KiB/s wr, 67 op/s 2024-04-26T02:03:50.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:50 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:50.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:50 smithi188 ceph-mon[32751]: pgmap v569: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 929 KiB/s wr, 67 op/s 2024-04-26T02:03:50.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:50 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:50.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:50 smithi029 ceph-mon[32775]: pgmap v569: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 929 KiB/s wr, 67 op/s 2024-04-26T02:03:50.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:50 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:51.121 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 20.33 MB/sec execute 134 sec latency 20364.273 ms 2024-04-26T02:03:52.121 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 20.18 MB/sec execute 135 sec latency 21364.395 ms 2024-04-26T02:03:52.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:52 smithi006 ceph-mon[28419]: pgmap v570: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 972 KiB/s wr, 72 op/s 2024-04-26T02:03:52.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:52 smithi188 ceph-mon[32751]: pgmap v570: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 972 KiB/s wr, 72 op/s 2024-04-26T02:03:52.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:52 smithi029 ceph-mon[32775]: pgmap v570: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 972 KiB/s wr, 72 op/s 2024-04-26T02:03:53.121 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 20.03 MB/sec execute 136 sec latency 22364.529 ms 2024-04-26T02:03:54.122 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 19.88 MB/sec execute 137 sec latency 23364.664 ms 2024-04-26T02:03:54.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:54 smithi188 ceph-mon[32751]: pgmap v571: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 307 KiB/s wr, 13 op/s 2024-04-26T02:03:54.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:54 smithi029 ceph-mon[32775]: pgmap v571: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 307 KiB/s wr, 13 op/s 2024-04-26T02:03:55.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:54 smithi006 ceph-mon[28419]: pgmap v571: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 307 KiB/s wr, 13 op/s 2024-04-26T02:03:55.121 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 19.74 MB/sec execute 138 sec latency 24364.812 ms 2024-04-26T02:03:56.121 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 19.60 MB/sec execute 139 sec latency 25364.946 ms 2024-04-26T02:03:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:56 smithi188 ceph-mon[32751]: pgmap v572: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 307 KiB/s wr, 13 op/s 2024-04-26T02:03:56.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:56 smithi029 ceph-mon[32775]: pgmap v572: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 307 KiB/s wr, 13 op/s 2024-04-26T02:03:57.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:56 smithi006 ceph-mon[28419]: pgmap v572: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 307 KiB/s wr, 13 op/s 2024-04-26T02:03:57.121 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 19.46 MB/sec execute 140 sec latency 26365.093 ms 2024-04-26T02:03:58.122 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 19.32 MB/sec execute 141 sec latency 27365.228 ms 2024-04-26T02:03:58.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:03:58 smithi188 ceph-mon[32751]: pgmap v573: 97 pgs: 97 active+clean; 851 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 328 KiB/s wr, 18 op/s 2024-04-26T02:03:58.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:03:58 smithi029 ceph-mon[32775]: pgmap v573: 97 pgs: 97 active+clean; 851 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 328 KiB/s wr, 18 op/s 2024-04-26T02:03:59.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:03:58 smithi006 ceph-mon[28419]: pgmap v573: 97 pgs: 97 active+clean; 851 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 328 KiB/s wr, 18 op/s 2024-04-26T02:03:59.122 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 19.18 MB/sec execute 142 sec latency 28365.367 ms 2024-04-26T02:04:00.122 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 19.05 MB/sec execute 143 sec latency 29365.520 ms 2024-04-26T02:04:00.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:00 smithi188 ceph-mon[32751]: pgmap v574: 97 pgs: 97 active+clean; 851 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 64 KiB/s wr, 9 op/s 2024-04-26T02:04:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:00 smithi029 ceph-mon[32775]: pgmap v574: 97 pgs: 97 active+clean; 851 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 64 KiB/s wr, 9 op/s 2024-04-26T02:04:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:00 smithi006 ceph-mon[28419]: pgmap v574: 97 pgs: 97 active+clean; 851 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 64 KiB/s wr, 9 op/s 2024-04-26T02:04:01.122 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 18.92 MB/sec execute 144 sec latency 30365.619 ms 2024-04-26T02:04:02.122 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 18.79 MB/sec execute 145 sec latency 31365.758 ms 2024-04-26T02:04:02.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:02 smithi006 ceph-mon[28419]: pgmap v575: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 1.5 MiB/s wr, 65 op/s 2024-04-26T02:04:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:02 smithi188 ceph-mon[32751]: pgmap v575: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 1.5 MiB/s wr, 65 op/s 2024-04-26T02:04:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:02 smithi029 ceph-mon[32775]: pgmap v575: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 1.5 MiB/s wr, 65 op/s 2024-04-26T02:04:03.122 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 18.66 MB/sec execute 146 sec latency 32365.927 ms 2024-04-26T02:04:04.123 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 18.53 MB/sec execute 147 sec latency 33366.073 ms 2024-04-26T02:04:04.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:04 smithi188 ceph-mon[32751]: pgmap v576: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 1.5 MiB/s wr, 60 op/s 2024-04-26T02:04:04.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:04 smithi029 ceph-mon[32775]: pgmap v576: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 1.5 MiB/s wr, 60 op/s 2024-04-26T02:04:05.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:04 smithi006 ceph-mon[28419]: pgmap v576: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 1.5 MiB/s wr, 60 op/s 2024-04-26T02:04:05.123 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 18.41 MB/sec execute 148 sec latency 34366.340 ms 2024-04-26T02:04:05.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:05 smithi188 ceph-mon[32751]: osdmap e126: 12 total, 12 up, 12 in 2024-04-26T02:04:05.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:05 smithi029 ceph-mon[32775]: osdmap e126: 12 total, 12 up, 12 in 2024-04-26T02:04:06.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:05 smithi006 ceph-mon[28419]: osdmap e126: 12 total, 12 up, 12 in 2024-04-26T02:04:06.123 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 18.28 MB/sec execute 149 sec latency 35366.473 ms 2024-04-26T02:04:06.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:06 smithi188 ceph-mon[32751]: pgmap v578: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 1.7 MiB/s wr, 72 op/s 2024-04-26T02:04:06.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:06 smithi029 ceph-mon[32775]: pgmap v578: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 1.7 MiB/s wr, 72 op/s 2024-04-26T02:04:07.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:06 smithi006 ceph-mon[28419]: pgmap v578: 97 pgs: 97 active+clean; 850 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 1.7 MiB/s wr, 72 op/s 2024-04-26T02:04:07.123 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 18.16 MB/sec execute 150 sec latency 36366.637 ms 2024-04-26T02:04:08.123 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 18.04 MB/sec execute 151 sec latency 37366.800 ms 2024-04-26T02:04:08.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:08 smithi188 ceph-mon[32751]: pgmap v579: 97 pgs: 97 active+clean; 850 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 4.0 MiB/s wr, 152 op/s 2024-04-26T02:04:08.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:08 smithi029 ceph-mon[32775]: pgmap v579: 97 pgs: 97 active+clean; 850 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 4.0 MiB/s wr, 152 op/s 2024-04-26T02:04:09.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:08 smithi006 ceph-mon[28419]: pgmap v579: 97 pgs: 97 active+clean; 850 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 4.0 MiB/s wr, 152 op/s 2024-04-26T02:04:09.123 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 17.92 MB/sec execute 152 sec latency 38366.904 ms 2024-04-26T02:04:10.123 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 17.80 MB/sec execute 153 sec latency 39366.999 ms 2024-04-26T02:04:10.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:10 smithi188 ceph-mon[32751]: pgmap v580: 97 pgs: 97 active+clean; 850 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 4.0 MiB/s wr, 152 op/s 2024-04-26T02:04:10.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:10 smithi188 ceph-mon[32751]: osdmap e127: 12 total, 12 up, 12 in 2024-04-26T02:04:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:10 smithi029 ceph-mon[32775]: pgmap v580: 97 pgs: 97 active+clean; 850 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 4.0 MiB/s wr, 152 op/s 2024-04-26T02:04:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:10 smithi029 ceph-mon[32775]: osdmap e127: 12 total, 12 up, 12 in 2024-04-26T02:04:11.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:10 smithi006 ceph-mon[28419]: pgmap v580: 97 pgs: 97 active+clean; 850 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 4.0 MiB/s wr, 152 op/s 2024-04-26T02:04:11.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:10 smithi006 ceph-mon[28419]: osdmap e127: 12 total, 12 up, 12 in 2024-04-26T02:04:11.124 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 17.69 MB/sec execute 154 sec latency 40367.161 ms 2024-04-26T02:04:12.124 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 17.57 MB/sec execute 155 sec latency 41367.295 ms 2024-04-26T02:04:12.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:12 smithi188 ceph-mon[32751]: pgmap v582: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 3.0 MiB/s wr, 120 op/s 2024-04-26T02:04:12.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:12 smithi029 ceph-mon[32775]: pgmap v582: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 3.0 MiB/s wr, 120 op/s 2024-04-26T02:04:13.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:12 smithi006 ceph-mon[28419]: pgmap v582: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 3.0 MiB/s wr, 120 op/s 2024-04-26T02:04:13.124 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 17.46 MB/sec execute 156 sec latency 42367.492 ms 2024-04-26T02:04:14.124 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 17.35 MB/sec execute 157 sec latency 43367.627 ms 2024-04-26T02:04:14.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:14 smithi188 ceph-mon[32751]: pgmap v583: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 2.7 MiB/s wr, 109 op/s 2024-04-26T02:04:14.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:14 smithi029 ceph-mon[32775]: pgmap v583: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 2.7 MiB/s wr, 109 op/s 2024-04-26T02:04:15.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:14 smithi006 ceph-mon[28419]: pgmap v583: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 2.7 MiB/s wr, 109 op/s 2024-04-26T02:04:15.124 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 17.24 MB/sec execute 158 sec latency 44367.780 ms 2024-04-26T02:04:15.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.463+0000 7f13bf400700 1 -- 172.21.15.6:0/3939197746 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: -115 ) ==== 8+0+594 (crc 0 0 0) 0x55d9c52968b0 con 0x55d9c53cf510 2024-04-26T02:04:15.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.463+0000 7f13bf400700 10 client.27248 handle_command_reply: tid=0 2024-04-26T02:04:15.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.463+0000 7f13eab87b80 1 client.27248 shutdown 2024-04-26T02:04:15.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.463+0000 7f13eab87b80 1 -- 172.21.15.6:0/3939197746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 msgr2=0x55d9c535c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:15.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.463+0000 7f13eab87b80 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c535c740 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f13d009a4a0 tx=0x7f13d009bfe0 comp rx=0 tx=0).stop 2024-04-26T02:04:15.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.464+0000 7f13eab87b80 1 -- 172.21.15.6:0/3939197746 shutdown_connections 2024-04-26T02:04:15.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.464+0000 7f13eab87b80 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9c5347180 0x55d9c535c740 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:15.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.464+0000 7f13eab87b80 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55d9c53cf510 0x55d9c53d1900 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:15.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.464+0000 7f13eab87b80 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d9c533b9b0 0x55d9c535fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:15.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.464+0000 7f13eab87b80 1 --2- 172.21.15.6:0/3939197746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55d9c5342ba0 0x55d9c535c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:15.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.464+0000 7f13eab87b80 1 -- 172.21.15.6:0/3939197746 >> 172.21.15.6:0/3939197746 conn(0x55d9c526a730 msgr2=0x55d9c53453d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:15.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.464+0000 7f13bf400700 0 client.27248 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:15.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.464+0000 7f13eab87b80 1 -- 172.21.15.6:0/3939197746 shutdown_connections 2024-04-26T02:04:15.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.464+0000 7f13eab87b80 1 -- 172.21.15.6:0/3939197746 wait complete. 2024-04-26T02:04:15.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:15.464+0000 7f13eab87b80 20 client.27248 trim_cache size 0 max 16384 2024-04-26T02:04:15.467 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:04:15.467 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 54, 2024-04-26T02:04:15.467 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T02:04:15.467 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 53, 2024-04-26T02:04:15.467 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T02:04:15.467 INFO:teuthology.orchestra.run.smithi006.stdout: "ef592526": { 2024-04-26T02:04:15.467 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 53, 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 0.0, 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCING", 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 42.0 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 82.5, 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCING", 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 42.0 2024-04-26T02:04:15.468 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:04:15.469 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:04:15.469 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:04:15.469 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:04:15.469 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:04:15.469 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:04:15.469 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINPROGRESS: 2024-04-26T02:04:16.125 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 17.13 MB/sec execute 159 sec latency 45367.930 ms 2024-04-26T02:04:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 115 2024-04-26T02:04:16.391 WARNING:tasks.quiescer.fs.[cephfs]:Set 'ef592526' hasn't quiesced after 50.0 seconds (timeout: 60). Dumping ops with locks from all ranks. 2024-04-26T02:04:16.392 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:04:16.858 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:04:16.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:16 smithi029 ceph-mon[32775]: pgmap v584: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 2.4 MiB/s wr, 96 op/s 2024-04-26T02:04:17.017 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:16 smithi006 ceph-mon[28419]: pgmap v584: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 2.4 MiB/s wr, 96 op/s 2024-04-26T02:04:17.125 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 17.03 MB/sec execute 160 sec latency 46368.083 ms 2024-04-26T02:04:17.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:16 smithi188 ceph-mon[32751]: pgmap v584: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 2.4 MiB/s wr, 96 op/s 2024-04-26T02:04:18.125 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 16.92 MB/sec execute 161 sec latency 47368.229 ms 2024-04-26T02:04:18.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:18 smithi029 ceph-mon[32775]: pgmap v585: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 160 KiB/s wr, 16 op/s 2024-04-26T02:04:18.946 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:18 smithi006 ceph-mon[28419]: pgmap v585: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 160 KiB/s wr, 16 op/s 2024-04-26T02:04:19.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.118+0000 7f8ddda00700 1 -- 172.21.15.6:0/2656548100 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dd8107940 msgr2=0x7f8dd8107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.118+0000 7f8ddda00700 1 --2- 172.21.15.6:0/2656548100 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dd8107940 0x7f8dd8107d20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80099c0 tx=0x7f8dc8009cd0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.119+0000 7f8ddda00700 1 -- 172.21.15.6:0/2656548100 shutdown_connections 2024-04-26T02:04:19.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.119+0000 7f8ddda00700 1 --2- 172.21.15.6:0/2656548100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dd81034b0 0x7f8dd810f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.119+0000 7f8ddda00700 1 --2- 172.21.15.6:0/2656548100 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dd8107940 0x7f8dd8107d20 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.119+0000 7f8ddda00700 1 --2- 172.21.15.6:0/2656548100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dd8102af0 0x7f8dd8102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.119+0000 7f8ddda00700 1 -- 172.21.15.6:0/2656548100 >> 172.21.15.6:0/2656548100 conn(0x7f8dd8079ba0 msgr2=0x7f8dd8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:19.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.119+0000 7f8ddda00700 1 -- 172.21.15.6:0/2656548100 shutdown_connections 2024-04-26T02:04:19.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.119+0000 7f8ddda00700 1 -- 172.21.15.6:0/2656548100 wait complete. 2024-04-26T02:04:19.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.120+0000 7f8ddda00700 1 Processor -- start 2024-04-26T02:04:19.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.120+0000 7f8ddda00700 1 -- start start 2024-04-26T02:04:19.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.120+0000 7f8ddda00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dd8102af0 0x7f8dd819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.120+0000 7f8ddda00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dd81034b0 0x7f8dd819b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.120+0000 7f8ddda00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dd8107940 0x7f8dd819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.120+0000 7f8ddda00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd8112ca0 con 0x7f8dd81034b0 2024-04-26T02:04:19.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.120+0000 7f8ddda00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd8112b20 con 0x7f8dd8102af0 2024-04-26T02:04:19.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.121+0000 7f8ddda00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd8112e20 con 0x7f8dd8107940 2024-04-26T02:04:19.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.121+0000 7f8dce000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dd81034b0 0x7f8dd819b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.121+0000 7f8dce000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dd81034b0 0x7f8dd819b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41252/0 (socket says 172.21.15.6:41252) 2024-04-26T02:04:19.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.121+0000 7f8dce000700 1 -- 172.21.15.6:0/3780188383 learned_addr learned my addr 172.21.15.6:0/3780188383 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:19.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.121+0000 7f8dd7400700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dd8107940 0x7f8dd819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.121+0000 7f8dd6a00700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dd8102af0 0x7f8dd819b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.121+0000 7f8dce000700 1 -- 172.21.15.6:0/3780188383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dd8107940 msgr2=0x7f8dd819fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.121+0000 7f8dce000700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dd8107940 0x7f8dd819fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.121+0000 7f8dce000700 1 -- 172.21.15.6:0/3780188383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dd8102af0 msgr2=0x7f8dd819b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.121+0000 7f8dce000700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dd8102af0 0x7f8dd819b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.121+0000 7f8dce000700 1 -- 172.21.15.6:0/3780188383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd81a03c0 con 0x7f8dd81034b0 2024-04-26T02:04:19.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.121+0000 7f8dd7400700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dd8107940 0x7f8dd819fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:19.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.122+0000 7f8dce000700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dd81034b0 0x7f8dd819b8f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000e9a0 tx=0x7f8dc000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:19.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.122+0000 7f8dcfe00700 1 -- 172.21.15.6:0/3780188383 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc000ca10 con 0x7f8dd81034b0 2024-04-26T02:04:19.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.122+0000 7f8dcfe00700 1 -- 172.21.15.6:0/3780188383 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dc000cbb0 con 0x7f8dd81034b0 2024-04-26T02:04:19.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.122+0000 7f8ddda00700 1 -- 172.21.15.6:0/3780188383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd81a06b0 con 0x7f8dd81034b0 2024-04-26T02:04:19.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.122+0000 7f8dcfe00700 1 -- 172.21.15.6:0/3780188383 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0018680 con 0x7f8dd81034b0 2024-04-26T02:04:19.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.122+0000 7f8ddda00700 1 -- 172.21.15.6:0/3780188383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd81a7f30 con 0x7f8dd81034b0 2024-04-26T02:04:19.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.123+0000 7f8ddda00700 1 -- 172.21.15.6:0/3780188383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd8069730 con 0x7f8dd81034b0 2024-04-26T02:04:19.124 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 16.82 MB/sec execute 162 sec latency 48368.309 ms 2024-04-26T02:04:19.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.127+0000 7f8dcfe00700 1 -- 172.21.15.6:0/3780188383 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f8dc0010430 con 0x7f8dd81034b0 2024-04-26T02:04:19.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.128+0000 7f8dcfe00700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8dbc0a3db0 0x7f8dbc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.129+0000 7f8dcfe00700 1 -- 172.21.15.6:0/3780188383 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f8dc00d0d60 con 0x7f8dd81034b0 2024-04-26T02:04:19.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.129+0000 7f8dd6a00700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8dbc0a3db0 0x7f8dbc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.129+0000 7f8dd6a00700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8dbc0a3db0 0x7f8dbc0a6270 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8000c00 tx=0x7f8dc8005fb0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:19.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.132+0000 7f8dcfe00700 1 -- 172.21.15.6:0/3780188383 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8dc009d350 con 0x7f8dd81034b0 2024-04-26T02:04:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:18 smithi188 ceph-mon[32751]: pgmap v585: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 160 KiB/s wr, 16 op/s 2024-04-26T02:04:19.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.413+0000 7f8ddda00700 1 -- 172.21.15.6:0/3780188383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dd819c710 con 0x7f8dd81034b0 2024-04-26T02:04:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.414+0000 7f8dcfe00700 1 -- 172.21.15.6:0/3780188383 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+11851 (secure 0 0 0) 0x7f8dc009cc50 con 0x7f8dd81034b0 2024-04-26T02:04:19.414 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:04:19.414 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":54,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T02:03:48.934257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26935":{"gid":26935,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1750728695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1750728695},{"type":"v1","addr":"172.21.15.188:6835","nonce":1750728695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24512,24505]},"id":1}]} 2024-04-26T02:04:19.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.417+0000 7f8ddda00700 1 -- 172.21.15.6:0/3780188383 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8dbc0a3db0 msgr2=0x7f8dbc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.417+0000 7f8ddda00700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8dbc0a3db0 0x7f8dbc0a6270 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8000c00 tx=0x7f8dc8005fb0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.417+0000 7f8ddda00700 1 -- 172.21.15.6:0/3780188383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dd81034b0 msgr2=0x7f8dd819b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.417+0000 7f8ddda00700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dd81034b0 0x7f8dd819b8f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000e9a0 tx=0x7f8dc000ecb0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.417+0000 7f8ddda00700 1 -- 172.21.15.6:0/3780188383 shutdown_connections 2024-04-26T02:04:19.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.417+0000 7f8ddda00700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f8dbc0a3db0 0x7f8dbc0a6270 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.417+0000 7f8ddda00700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dd8107940 0x7f8dd819fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.417+0000 7f8ddda00700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dd8102af0 0x7f8dd819b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.417+0000 7f8ddda00700 1 --2- 172.21.15.6:0/3780188383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dd81034b0 0x7f8dd819b8f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.417+0000 7f8ddda00700 1 -- 172.21.15.6:0/3780188383 >> 172.21.15.6:0/3780188383 conn(0x7f8dd8079ba0 msgr2=0x7f8dd8100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:19.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.417+0000 7f8ddda00700 1 -- 172.21.15.6:0/3780188383 shutdown_connections 2024-04-26T02:04:19.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:19.418+0000 7f8ddda00700 1 -- 172.21.15.6:0/3780188383 wait complete. 2024-04-26T02:04:19.418 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 54 2024-04-26T02:04:19.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:19 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3780188383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:04:19.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:19 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3780188383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:04:20.125 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 16.71 MB/sec execute 163 sec latency 49368.441 ms 2024-04-26T02:04:20.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:19 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3780188383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:04:20.202 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 2 (c) to a remote file /var/run/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ops-ef592526-1-mds.c.json 2024-04-26T02:04:20.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell mds.1:2 ops --flags=locks --path=/var/run/ceph/ops-ef592526-1-mds.c.json 2024-04-26T02:04:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:04:20.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:20 smithi029 ceph-mon[32775]: pgmap v586: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 160 KiB/s wr, 16 op/s 2024-04-26T02:04:21.014 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:20 smithi006 ceph-mon[28419]: pgmap v586: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 160 KiB/s wr, 16 op/s 2024-04-26T02:04:21.125 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 16.61 MB/sec execute 164 sec latency 50368.544 ms 2024-04-26T02:04:21.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:20 smithi188 ceph-mon[32751]: pgmap v586: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 160 KiB/s wr, 16 op/s 2024-04-26T02:04:22.125 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 16.51 MB/sec execute 165 sec latency 51368.642 ms 2024-04-26T02:04:22.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.845+0000 7f1962e00700 1 -- 172.21.15.6:0/1508037067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f195c10cf60 msgr2=0x7f195c10f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:22.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.845+0000 7f1962e00700 1 --2- 172.21.15.6:0/1508037067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f195c10cf60 0x7f195c10f420 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f195400b0a0 tx=0x7f195400b3b0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.845+0000 7f1962e00700 1 -- 172.21.15.6:0/1508037067 shutdown_connections 2024-04-26T02:04:22.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.845+0000 7f1962e00700 1 --2- 172.21.15.6:0/1508037067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f195c10cf60 0x7f195c10f420 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.845+0000 7f1962e00700 1 --2- 172.21.15.6:0/1508037067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f195c103b30 0x7f195c103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.845+0000 7f1962e00700 1 --2- 172.21.15.6:0/1508037067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f195c104450 0x7f195c10c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.845+0000 7f1962e00700 1 -- 172.21.15.6:0/1508037067 >> 172.21.15.6:0/1508037067 conn(0x7f195c0fd360 msgr2=0x7f195c0ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:22.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.845+0000 7f1962e00700 1 -- 172.21.15.6:0/1508037067 shutdown_connections 2024-04-26T02:04:22.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.845+0000 7f1962e00700 1 -- 172.21.15.6:0/1508037067 wait complete. 2024-04-26T02:04:22.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.846+0000 7f1962e00700 1 Processor -- start 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.846+0000 7f1962e00700 1 -- start start 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.846+0000 7f1962e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f195c103b30 0x7f195c19b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.846+0000 7f1962e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f195c104450 0x7f195c19b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1962e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f195c10cf60 0x7f195c19fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1962e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f195c112ba0 con 0x7f195c10cf60 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1962e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f195c112a20 con 0x7f195c103b30 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1962e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f195c112d20 con 0x7f195c104450 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1962400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f195c10cf60 0x7f195c19fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1961a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f195c103b30 0x7f195c19b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1961a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f195c103b30 0x7f195c19b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39854/0 (socket says 172.21.15.6:39854) 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1961a00700 1 -- 172.21.15.6:0/2966485095 learned_addr learned my addr 172.21.15.6:0/2966485095 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1961000700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f195c104450 0x7f195c19b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1961a00700 1 -- 172.21.15.6:0/2966485095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f195c104450 msgr2=0x7f195c19b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1961a00700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f195c104450 0x7f195c19b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1961a00700 1 -- 172.21.15.6:0/2966485095 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f195c10cf60 msgr2=0x7f195c19fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1961a00700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f195c10cf60 0x7f195c19fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.847+0000 7f1961a00700 1 -- 172.21.15.6:0/2966485095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195c1a0220 con 0x7f195c103b30 2024-04-26T02:04:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.848+0000 7f1962400700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f195c10cf60 0x7f195c19fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.848+0000 7f1961000700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f195c104450 0x7f195c19b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.848+0000 7f1961a00700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f195c103b30 0x7f195c19b210 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f195000e9a0 tx=0x7f195000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.848+0000 7f195a000700 1 -- 172.21.15.6:0/2966485095 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195000ca10 con 0x7f195c103b30 2024-04-26T02:04:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.848+0000 7f1962e00700 1 -- 172.21.15.6:0/2966485095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195c1a0510 con 0x7f195c103b30 2024-04-26T02:04:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.848+0000 7f195a000700 1 -- 172.21.15.6:0/2966485095 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f195000cbb0 con 0x7f195c103b30 2024-04-26T02:04:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.848+0000 7f195a000700 1 -- 172.21.15.6:0/2966485095 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950018680 con 0x7f195c103b30 2024-04-26T02:04:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.848+0000 7f1962e00700 1 -- 172.21.15.6:0/2966485095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f195c1a7d90 con 0x7f195c103b30 2024-04-26T02:04:22.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.851+0000 7f195a000700 1 -- 172.21.15.6:0/2966485095 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f1950010430 con 0x7f195c103b30 2024-04-26T02:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.853+0000 7f196ed26b80 1 Processor -- start 2024-04-26T02:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.853+0000 7f195a000700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f19440a3db0 0x7f19440a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.853+0000 7f196ed26b80 1 -- start start 2024-04-26T02:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.853+0000 7f195a000700 1 -- 172.21.15.6:0/2966485095 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f19500d0a20 con 0x7f195c103b30 2024-04-26T02:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.853+0000 7f1961000700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f19440a3db0 0x7f19440a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f196ed26b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3748e80 0x5623d3749260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f196ed26b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d3754110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f196ed26b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d3756a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f196ed26b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5623d369bf70 con 0x5623d3748e80 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f196ed26b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5623d3642810 con 0x5623d37497a0 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f196ed26b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5623d3697f30 con 0x5623d3754650 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1961a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3748e80 0x5623d3749260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1961000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d3754110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1961000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d3754110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39860/0 (socket says 172.21.15.6:39860) 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1961000700 1 -- 172.21.15.6:0/1708459178 learned_addr learned my addr 172.21.15.6:0/1708459178 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1962400700 1 --2- 172.21.15.6:0/1708459178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d3756a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1961000700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f19440a3db0 0x7f19440a6270 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f195c1037b0 tx=0x7f1948009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1961000700 1 -- 172.21.15.6:0/1708459178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 msgr2=0x5623d3756a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1961000700 1 --2- 172.21.15.6:0/1708459178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d3756a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1961000700 1 -- 172.21.15.6:0/1708459178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3748e80 msgr2=0x5623d3749260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:22.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1961000700 1 --2- 172.21.15.6:0/1708459178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3748e80 0x5623d3749260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1961000700 1 -- 172.21.15.6:0/1708459178 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623d3638760 con 0x5623d37497a0 2024-04-26T02:04:22.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1962400700 1 --2- 172.21.15.6:0/1708459178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d3756a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:22.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.854+0000 7f1961a00700 1 --2- 172.21.15.6:0/1708459178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3748e80 0x5623d3749260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:22.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.855+0000 7f1961000700 1 --2- 172.21.15.6:0/1708459178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d3754110 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f19480041d0 tx=0x7f194801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0d770abdcf19a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:22.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.855+0000 7f194ea00700 1 -- 172.21.15.6:0/1708459178 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1948029070 con 0x5623d37497a0 2024-04-26T02:04:22.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.855+0000 7f194ea00700 1 -- 172.21.15.6:0/1708459178 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1948003c90 con 0x5623d37497a0 2024-04-26T02:04:22.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.855+0000 7f194ea00700 1 -- 172.21.15.6:0/1708459178 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194801c660 con 0x5623d37497a0 2024-04-26T02:04:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.856+0000 7f196ed26b80 1 -- 172.21.15.6:0/1708459178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 msgr2=0x5623d3754110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.856+0000 7f196ed26b80 1 --2- 172.21.15.6:0/1708459178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d3754110 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f19480041d0 tx=0x7f194801c000 comp rx=0 tx=0).stop 2024-04-26T02:04:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.856+0000 7f196ed26b80 1 -- 172.21.15.6:0/1708459178 shutdown_connections 2024-04-26T02:04:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.856+0000 7f196ed26b80 1 --2- 172.21.15.6:0/1708459178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d3756a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.856+0000 7f196ed26b80 1 --2- 172.21.15.6:0/1708459178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3748e80 0x5623d3749260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.856+0000 7f196ed26b80 1 --2- 172.21.15.6:0/1708459178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d3754110 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.856+0000 7f196ed26b80 1 -- 172.21.15.6:0/1708459178 >> 172.21.15.6:0/1708459178 conn(0x5623d3678030 msgr2=0x5623d37453d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.856+0000 7f196ed26b80 1 -- 172.21.15.6:0/1708459178 shutdown_connections 2024-04-26T02:04:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.856+0000 7f196ed26b80 1 -- 172.21.15.6:0/1708459178 wait complete. 2024-04-26T02:04:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.857+0000 7f196ed26b80 1 -- start start 2024-04-26T02:04:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.857+0000 7f196ed26b80 1 Processor -- start 2024-04-26T02:04:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.857+0000 7f196ed26b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3748e80 0x5623d3753d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.857+0000 7f196ed26b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d3751090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.857+0000 7f196ed26b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d37515d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.857+0000 7f196ed26b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5623d37570e0 con 0x5623d3748e80 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.857+0000 7f196ed26b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5623d3655940 con 0x5623d37497a0 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.857+0000 7f196ed26b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5623d3757260 con 0x5623d3754650 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.858+0000 7f196ed26b80 10 client.0 _ll_register_callbacks cb 0x5623d3634800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.858+0000 7f1961000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d3751090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.858+0000 7f1961000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d3751090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39862/0 (socket says 172.21.15.6:39862) 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.858+0000 7f1961000700 1 -- 172.21.15.6:0/1775716521 learned_addr learned my addr 172.21.15.6:0/1775716521 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.858+0000 7f1961a00700 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3748e80 0x5623d3753d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.858+0000 7f1962400700 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d37515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.858+0000 7f1961000700 1 -- 172.21.15.6:0/1775716521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 msgr2=0x5623d37515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.858+0000 7f1961000700 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d37515d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.858+0000 7f1961000700 1 -- 172.21.15.6:0/1775716521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3748e80 msgr2=0x5623d3753d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.858+0000 7f1961000700 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3748e80 0x5623d3753d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.858+0000 7f1961000700 1 -- 172.21.15.6:0/1775716521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623d35e99b0 con 0x5623d37497a0 2024-04-26T02:04:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.858+0000 7f1961a00700 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3748e80 0x5623d3753d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.859+0000 7f1961000700 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d3751090 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1948005950 tx=0x7f1948024920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b721efc94d92c69 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.859+0000 7f1943400700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:04:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.859+0000 7f1943400700 1 -- 172.21.15.6:0/1775716521 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1948031020 con 0x5623d37497a0 2024-04-26T02:04:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.859+0000 7f1943400700 1 -- 172.21.15.6:0/1775716521 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f1948003e40 con 0x5623d37497a0 2024-04-26T02:04:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.859+0000 7f196ed26b80 1 -- 172.21.15.6:0/1775716521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1948003c90 con 0x5623d37497a0 2024-04-26T02:04:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.859+0000 7f1943400700 1 -- 172.21.15.6:0/1775716521 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f194803c050 con 0x5623d37497a0 2024-04-26T02:04:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.859+0000 7f196ed26b80 10 client.27284 fetch_fsmap learned FSMap version 54 2024-04-26T02:04:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.859+0000 7f196ed26b80 1 -- 172.21.15.6:0/1775716521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5623d3632bd0 con 0x5623d37497a0 2024-04-26T02:04:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.860+0000 7f1943400700 1 -- 172.21.15.6:0/1775716521 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1948003c90 con 0x5623d37497a0 2024-04-26T02:04:22.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.860+0000 7f1943400700 1 -- 172.21.15.6:0/1775716521 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 54) ==== 8342+0+0 (secure 0 0 0) 0x7f194801a050 con 0x5623d37497a0 2024-04-26T02:04:22.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.860+0000 7f196ed26b80 10 client.27284 fetch_fsmap finished waiting for FSMap version 54 2024-04-26T02:04:22.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.860+0000 7f196ed26b80 10 client.27284 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-26T02:04:22.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.860+0000 7f196ed26b80 20 client.27284 populate_metadata read hostname 'smithi006' 2024-04-26T02:04:22.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.860+0000 7f196ed26b80 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x5623d3763c70 0x5623d3766060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.860+0000 7f196ed26b80 4 client.27284 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:04:22.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.860+0000 7f196ed26b80 1 -- 172.21.15.6:0/1775716521 --> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623d36a9760 con 0x5623d3763c70 2024-04-26T02:04:22.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.861+0000 7f1961a00700 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x5623d3763c70 0x5623d3766060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.861+0000 7f1961a00700 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x5623d3763c70 0x5623d3766060 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=834f43b704928a22 server_cookie=767b0c59b6e561cb in_seq=0 out_seq=0 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.861+0000 7f1943400700 10 client.27284 ms_handle_connect on v2:172.21.15.188:6836/489767141 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f1943400700 1 -- 172.21.15.6:0/1775716521 <== mds.2 v2:172.21.15.188:6836/489767141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623d36a9760 con 0x5623d3763c70 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f1943400700 10 client.27284 handle_command_reply: tid=0 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f196ed26b80 1 client.27284 shutdown 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f196ed26b80 1 -- 172.21.15.6:0/1775716521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 msgr2=0x5623d3751090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f196ed26b80 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d3751090 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1948005950 tx=0x7f1948024920 comp rx=0 tx=0).stop 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f196ed26b80 1 -- 172.21.15.6:0/1775716521 shutdown_connections 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f196ed26b80 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d37515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f196ed26b80 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x5623d3763c70 0x5623d3766060 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f196ed26b80 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3748e80 0x5623d3753d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f196ed26b80 1 --2- 172.21.15.6:0/1775716521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d3751090 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f1943400700 0 client.27284 ms_handle_reset on v2:172.21.15.188:6836/489767141 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f196ed26b80 1 -- 172.21.15.6:0/1775716521 >> 172.21.15.6:0/1775716521 conn(0x5623d3678030 msgr2=0x5623d3744fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f196ed26b80 1 -- 172.21.15.6:0/1775716521 shutdown_connections 2024-04-26T02:04:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.862+0000 7f196ed26b80 1 -- 172.21.15.6:0/1775716521 wait complete. 2024-04-26T02:04:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.863+0000 7f196ed26b80 20 client.27284 trim_cache size 0 max 16384 2024-04-26T02:04:22.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.881+0000 7f196ed26b80 1 Processor -- start 2024-04-26T02:04:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.881+0000 7f196ed26b80 1 -- start start 2024-04-26T02:04:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f196ed26b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d3748e80 0x5623d3745550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f196ed26b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d37497a0 0x5623d376b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f196ed26b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d376bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f196ed26b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1b1da0 con 0x5623d37497a0 2024-04-26T02:04:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f196ed26b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1af600 con 0x5623d3748e80 2024-04-26T02:04:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f196ed26b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1ad470 con 0x5623d3754650 2024-04-26T02:04:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f1961000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d37497a0 0x5623d376b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f1961a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d3748e80 0x5623d3745550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f1961000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d37497a0 0x5623d376b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41306/0 (socket says 172.21.15.6:41306) 2024-04-26T02:04:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f1961000700 1 -- 172.21.15.6:0/706113267 learned_addr learned my addr 172.21.15.6:0/706113267 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f1962400700 1 --2- 172.21.15.6:0/706113267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d376bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f1961a00700 1 -- 172.21.15.6:0/706113267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 msgr2=0x5623d376bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:22.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.882+0000 7f1961a00700 1 --2- 172.21.15.6:0/706113267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d376bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.883+0000 7f1961a00700 1 -- 172.21.15.6:0/706113267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d37497a0 msgr2=0x5623d376b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.883+0000 7f1961a00700 1 --2- 172.21.15.6:0/706113267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d37497a0 0x5623d376b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.883+0000 7f1961a00700 1 -- 172.21.15.6:0/706113267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623d37665f0 con 0x5623d3748e80 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.883+0000 7f1961000700 1 --2- 172.21.15.6:0/706113267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d37497a0 0x5623d376b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.883+0000 7f1961a00700 1 --2- 172.21.15.6:0/706113267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d3748e80 0x5623d3745550 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f19500ce0c0 tx=0x7f1950009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeb919a739a49faf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.883+0000 7f194e000700 1 -- 172.21.15.6:0/706113267 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19500a5580 con 0x5623d3748e80 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.883+0000 7f194e000700 1 -- 172.21.15.6:0/706113267 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1950085e50 con 0x5623d3748e80 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.883+0000 7f194e000700 1 -- 172.21.15.6:0/706113267 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195009cdd0 con 0x5623d3748e80 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.884+0000 7f196ed26b80 1 -- 172.21.15.6:0/706113267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d3748e80 msgr2=0x5623d3745550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.884+0000 7f196ed26b80 1 --2- 172.21.15.6:0/706113267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d3748e80 0x5623d3745550 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f19500ce0c0 tx=0x7f1950009790 comp rx=0 tx=0).stop 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.884+0000 7f196ed26b80 1 -- 172.21.15.6:0/706113267 shutdown_connections 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.884+0000 7f196ed26b80 1 --2- 172.21.15.6:0/706113267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3754650 0x5623d376bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.884+0000 7f196ed26b80 1 --2- 172.21.15.6:0/706113267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d3748e80 0x5623d3745550 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.884+0000 7f196ed26b80 1 --2- 172.21.15.6:0/706113267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d37497a0 0x5623d376b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.884+0000 7f196ed26b80 1 -- 172.21.15.6:0/706113267 >> 172.21.15.6:0/706113267 conn(0x5623d3678030 msgr2=0x5623d3753150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.884+0000 7f196ed26b80 1 -- 172.21.15.6:0/706113267 shutdown_connections 2024-04-26T02:04:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.884+0000 7f196ed26b80 1 -- 172.21.15.6:0/706113267 wait complete. 2024-04-26T02:04:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.884+0000 7f196ed26b80 1 -- start start 2024-04-26T02:04:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f196ed26b80 1 Processor -- start 2024-04-26T02:04:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f196ed26b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3748e80 0x5623d375c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f196ed26b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d37594e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f196ed26b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3754650 0x5623d3759a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f196ed26b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1adac0 con 0x5623d3754650 2024-04-26T02:04:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f196ed26b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1aef30 con 0x5623d37497a0 2024-04-26T02:04:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f196ed26b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1ae190 con 0x5623d3748e80 2024-04-26T02:04:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f196ed26b80 10 client.0 _ll_register_callbacks cb 0x5623d3634800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f1961000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d37594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f1962400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3754650 0x5623d3759a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f1961a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3748e80 0x5623d375c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f1961000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d37594e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39888/0 (socket says 172.21.15.6:39888) 2024-04-26T02:04:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.885+0000 7f1961000700 1 -- 172.21.15.6:0/779682615 learned_addr learned my addr 172.21.15.6:0/779682615 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.886+0000 7f1962400700 1 -- 172.21.15.6:0/779682615 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3748e80 msgr2=0x5623d375c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.886+0000 7f1962400700 1 --2- 172.21.15.6:0/779682615 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3748e80 0x5623d375c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.886+0000 7f1962400700 1 -- 172.21.15.6:0/779682615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 msgr2=0x5623d37594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.886+0000 7f1962400700 1 --2- 172.21.15.6:0/779682615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d37594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.886+0000 7f1962400700 1 -- 172.21.15.6:0/779682615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623d375a060 con 0x5623d3754650 2024-04-26T02:04:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.886+0000 7f1961000700 1 --2- 172.21.15.6:0/779682615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d37594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.886+0000 7f1962400700 1 --2- 172.21.15.6:0/779682615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3754650 0x5623d3759a20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f19540062a0 tx=0x7f1954003e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b6187d972d2b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.886+0000 7f1943400700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:04:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.887+0000 7f1943400700 1 -- 172.21.15.6:0/779682615 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195402a020 con 0x5623d3754650 2024-04-26T02:04:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.887+0000 7f196ed26b80 1 -- 172.21.15.6:0/779682615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1950085e50 con 0x5623d3754650 2024-04-26T02:04:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.887+0000 7f1943400700 1 -- 172.21.15.6:0/779682615 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f19540078f0 con 0x5623d3754650 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.887+0000 7f1943400700 1 -- 172.21.15.6:0/779682615 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f1954032050 con 0x5623d3754650 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.887+0000 7f196ed26b80 10 client.17094 fetch_fsmap learned FSMap version 54 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.887+0000 7f196ed26b80 1 -- 172.21.15.6:0/779682615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5623d37caf60 con 0x5623d3754650 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.887+0000 7f1943400700 1 -- 172.21.15.6:0/779682615 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1950085e50 con 0x5623d3754650 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.887+0000 7f1943400700 1 -- 172.21.15.6:0/779682615 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 54) ==== 8342+0+0 (secure 0 0 0) 0x7f1954025070 con 0x5623d3754650 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.887+0000 7f196ed26b80 10 client.17094 fetch_fsmap finished waiting for FSMap version 54 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.887+0000 7f196ed26b80 10 client.17094 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.888+0000 7f196ed26b80 20 client.17094 populate_metadata read hostname 'smithi006' 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.888+0000 7f196ed26b80 1 --2- 172.21.15.6:0/779682615 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x5623d37cb6c0 0x5623d37cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.888+0000 7f196ed26b80 4 client.17094 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ef592526-1-mds.c.json"}] 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.888+0000 7f196ed26b80 1 -- 172.21.15.6:0/779682615 --> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ef592526-1-mds.c.json"}) -- 0x5623d36a3800 con 0x5623d37cb6c0 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.888+0000 7f1961a00700 1 --2- 172.21.15.6:0/779682615 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x5623d37cb6c0 0x5623d37cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.888+0000 7f1961a00700 1 --2- 172.21.15.6:0/779682615 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x5623d37cb6c0 0x5623d37cdae0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bc227cb007c8e90b server_cookie=7a416d94d1feb1b8 in_seq=0 out_seq=0 2024-04-26T02:04:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.888+0000 7f1943400700 10 client.17094 ms_handle_connect on v2:172.21.15.188:6836/489767141 2024-04-26T02:04:22.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.910+0000 7f1943400700 1 -- 172.21.15.6:0/779682615 <== mds.2 v2:172.21.15.188:6836/489767141 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x5623d36a3800 con 0x5623d37cb6c0 2024-04-26T02:04:22.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.910+0000 7f1943400700 10 client.17094 handle_command_reply: tid=0 2024-04-26T02:04:22.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.910+0000 7f196ed26b80 1 client.17094 shutdown 2024-04-26T02:04:22.910 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:22 smithi006 ceph-mon[28419]: pgmap v587: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 156 KiB/s wr, 19 op/s 2024-04-26T02:04:22.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.910+0000 7f196ed26b80 1 -- 172.21.15.6:0/779682615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3754650 msgr2=0x5623d3759a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:22.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.910+0000 7f196ed26b80 1 --2- 172.21.15.6:0/779682615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3754650 0x5623d3759a20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f19540062a0 tx=0x7f1954003e40 comp rx=0 tx=0).stop 2024-04-26T02:04:22.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.910+0000 7f196ed26b80 1 -- 172.21.15.6:0/779682615 shutdown_connections 2024-04-26T02:04:22.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.910+0000 7f196ed26b80 1 --2- 172.21.15.6:0/779682615 >> [v2:172.21.15.188:6836/489767141,v1:172.21.15.188:6837/489767141] conn(0x5623d37cb6c0 0x5623d37cdae0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.911+0000 7f196ed26b80 1 --2- 172.21.15.6:0/779682615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5623d3754650 0x5623d3759a20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.911+0000 7f196ed26b80 1 --2- 172.21.15.6:0/779682615 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d3748e80 0x5623d375c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.911+0000 7f196ed26b80 1 --2- 172.21.15.6:0/779682615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5623d37497a0 0x5623d37594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.911+0000 7f196ed26b80 1 -- 172.21.15.6:0/779682615 >> 172.21.15.6:0/779682615 conn(0x5623d3678030 msgr2=0x5623d3744540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:22.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.911+0000 7f1943400700 0 client.17094 ms_handle_reset on v2:172.21.15.188:6836/489767141 2024-04-26T02:04:22.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.911+0000 7f196ed26b80 1 -- 172.21.15.6:0/779682615 shutdown_connections 2024-04-26T02:04:22.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.911+0000 7f196ed26b80 1 -- 172.21.15.6:0/779682615 wait complete. 2024-04-26T02:04:22.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.911+0000 7f196ed26b80 20 client.17094 trim_cache size 0 max 16384 2024-04-26T02:04:22.911 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:04:22.912 INFO:teuthology.orchestra.run.smithi006.stdout: "path": "/var/run/ceph/ops-ef592526-1-mds.c.json" 2024-04-26T02:04:22.912 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:04:22.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.912+0000 7f1962e00700 1 -- 172.21.15.6:0/2966485095 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f19440a3db0 msgr2=0x7f19440a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:22.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.912+0000 7f1962e00700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f19440a3db0 0x7f19440a6270 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f195c1037b0 tx=0x7f1948009260 comp rx=0 tx=0).stop 2024-04-26T02:04:22.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.912+0000 7f1962e00700 1 -- 172.21.15.6:0/2966485095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f195c103b30 msgr2=0x7f195c19b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:22.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.912+0000 7f1962e00700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f195c103b30 0x7f195c19b210 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f195000e9a0 tx=0x7f195000ecb0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.912+0000 7f1962e00700 1 -- 172.21.15.6:0/2966485095 shutdown_connections 2024-04-26T02:04:22.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.912+0000 7f1962e00700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f19440a3db0 0x7f19440a6270 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.912+0000 7f1962e00700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f195c10cf60 0x7f195c19fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.912+0000 7f1962e00700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f195c103b30 0x7f195c19b210 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.912+0000 7f1962e00700 1 --2- 172.21.15.6:0/2966485095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f195c104450 0x7f195c19b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:22.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.912+0000 7f1962e00700 1 -- 172.21.15.6:0/2966485095 >> 172.21.15.6:0/2966485095 conn(0x7f195c0fd360 msgr2=0x7f195c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:22.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.912+0000 7f1962e00700 1 -- 172.21.15.6:0/2966485095 shutdown_connections 2024-04-26T02:04:22.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:22.912+0000 7f1962e00700 1 -- 172.21.15.6:0/2966485095 wait complete. 2024-04-26T02:04:22.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:22 smithi029 ceph-mon[32775]: pgmap v587: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 156 KiB/s wr, 19 op/s 2024-04-26T02:04:23.125 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 16.41 MB/sec execute 166 sec latency 52368.754 ms 2024-04-26T02:04:23.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:22 smithi188 ceph-mon[32751]: pgmap v587: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 156 KiB/s wr, 19 op/s 2024-04-26T02:04:23.659 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 1 (b) to a remote file /var/run/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ops-ef592526-1-mds.b.json 2024-04-26T02:04:23.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell mds.1:1 ops --flags=locks --path=/var/run/ceph/ops-ef592526-1-mds.b.json 2024-04-26T02:04:24.126 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 16.31 MB/sec execute 167 sec latency 53368.836 ms 2024-04-26T02:04:24.132 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:04:24.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:24 smithi029 ceph-mon[32775]: pgmap v588: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-04-26T02:04:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:24 smithi006 ceph-mon[28419]: pgmap v588: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-04-26T02:04:25.125 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 16.22 MB/sec execute 168 sec latency 54368.975 ms 2024-04-26T02:04:25.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:24 smithi188 ceph-mon[32751]: pgmap v588: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-04-26T02:04:26.126 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 16.12 MB/sec execute 169 sec latency 55369.080 ms 2024-04-26T02:04:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.366+0000 7f5db4c00700 1 -- 172.21.15.6:0/1963647796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5db0109c80 msgr2=0x7f5db010dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.366+0000 7f5db4c00700 1 --2- 172.21.15.6:0/1963647796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5db0109c80 0x7f5db010dcd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009990 tx=0x7f5d9c009ca0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.366+0000 7f5db4c00700 1 -- 172.21.15.6:0/1963647796 shutdown_connections 2024-04-26T02:04:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.366+0000 7f5db4c00700 1 --2- 172.21.15.6:0/1963647796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5db010e3b0 0x7f5db0115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.366+0000 7f5db4c00700 1 --2- 172.21.15.6:0/1963647796 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5db01092d0 0x7f5db01096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.366+0000 7f5db4c00700 1 --2- 172.21.15.6:0/1963647796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5db0109c80 0x7f5db010dcd0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.366+0000 7f5db4c00700 1 -- 172.21.15.6:0/1963647796 >> 172.21.15.6:0/1963647796 conn(0x7f5db0076210 msgr2=0x7f5db0107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.366+0000 7f5db4c00700 1 -- 172.21.15.6:0/1963647796 shutdown_connections 2024-04-26T02:04:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.367+0000 7f5db4c00700 1 -- 172.21.15.6:0/1963647796 wait complete. 2024-04-26T02:04:26.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.368+0000 7f5db4c00700 1 Processor -- start 2024-04-26T02:04:26.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.368+0000 7f5db4c00700 1 -- start start 2024-04-26T02:04:26.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.369+0000 7f5db4c00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5db01092d0 0x7f5db01a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.369+0000 7f5db4c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5db0109c80 0x7f5db01a5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.369+0000 7f5db4c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5db010e3b0 0x7f5db019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.369+0000 7f5db4c00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5db0119270 con 0x7f5db010e3b0 2024-04-26T02:04:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.369+0000 7f5db4c00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01190f0 con 0x7f5db01092d0 2024-04-26T02:04:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.369+0000 7f5db4c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01193f0 con 0x7f5db0109c80 2024-04-26T02:04:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.369+0000 7f5dade00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5db01092d0 0x7f5db01a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.369+0000 7f5dad400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5db0109c80 0x7f5db01a5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.369+0000 7f5dade00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5db01092d0 0x7f5db01a5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39902/0 (socket says 172.21.15.6:39902) 2024-04-26T02:04:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.369+0000 7f5dade00700 1 -- 172.21.15.6:0/3061776462 learned_addr learned my addr 172.21.15.6:0/3061776462 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.369+0000 7f5dae800700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5db010e3b0 0x7f5db019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.370+0000 7f5dade00700 1 -- 172.21.15.6:0/3061776462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5db0109c80 msgr2=0x7f5db01a5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.370+0000 7f5dade00700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5db0109c80 0x7f5db01a5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.370+0000 7f5dade00700 1 -- 172.21.15.6:0/3061776462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5db010e3b0 msgr2=0x7f5db019f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.370+0000 7f5dade00700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5db010e3b0 0x7f5db019f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.370+0000 7f5dade00700 1 -- 172.21.15.6:0/3061776462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db019fd70 con 0x7f5db01092d0 2024-04-26T02:04:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.370+0000 7f5dade00700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5db01092d0 0x7f5db01a5400 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5da000e9a0 tx=0x7f5da000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.370+0000 7f5da6a00700 1 -- 172.21.15.6:0/3061776462 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da000ca10 con 0x7f5db01092d0 2024-04-26T02:04:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.370+0000 7f5da6a00700 1 -- 172.21.15.6:0/3061776462 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da000cbb0 con 0x7f5db01092d0 2024-04-26T02:04:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.370+0000 7f5da6a00700 1 -- 172.21.15.6:0/3061776462 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0018680 con 0x7f5db01092d0 2024-04-26T02:04:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.370+0000 7f5db4c00700 1 -- 172.21.15.6:0/3061776462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db01a0060 con 0x7f5db01092d0 2024-04-26T02:04:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.371+0000 7f5db4c00700 1 -- 172.21.15.6:0/3061776462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db01ac2d0 con 0x7f5db01092d0 2024-04-26T02:04:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.374+0000 7f5da6a00700 1 -- 172.21.15.6:0/3061776462 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f5da0010430 con 0x7f5db01092d0 2024-04-26T02:04:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.376+0000 7f5dbf4b1b80 1 Processor -- start 2024-04-26T02:04:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.376+0000 7f5da6a00700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5d940a3db0 0x7f5d940a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.377+0000 7f5dbf4b1b80 1 -- start start 2024-04-26T02:04:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.377+0000 7f5da6a00700 1 -- 172.21.15.6:0/3061776462 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f5da00d0a20 con 0x7f5db01092d0 2024-04-26T02:04:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.377+0000 7f5dae800700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5d940a3db0 0x7f5d940a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.377+0000 7f5dbf4b1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e79d20 0x555f62e7a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.377+0000 7f5dbf4b1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e7a640 0x555f62e84f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.377+0000 7f5dbf4b1b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e854b0 0x555f62e878a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.377+0000 7f5dbf4b1b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555f62dccf70 con 0x555f62e854b0 2024-04-26T02:04:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.377+0000 7f5dbf4b1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555f62d73810 con 0x555f62e7a640 2024-04-26T02:04:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.377+0000 7f5dbf4b1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555f62dc8f30 con 0x555f62e79d20 2024-04-26T02:04:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dad400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e7a640 0x555f62e84f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dade00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e79d20 0x555f62e7a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dad400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e7a640 0x555f62e84f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39906/0 (socket says 172.21.15.6:39906) 2024-04-26T02:04:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dad400700 1 -- 172.21.15.6:0/2126659308 learned_addr learned my addr 172.21.15.6:0/2126659308 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dae800700 1 --2- 172.21.15.6:0/2126659308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e854b0 0x555f62e878a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dade00700 1 -- 172.21.15.6:0/2126659308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e7a640 msgr2=0x555f62e84f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dade00700 1 --2- 172.21.15.6:0/2126659308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e7a640 0x555f62e84f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dade00700 1 -- 172.21.15.6:0/2126659308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e854b0 msgr2=0x555f62e878a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dade00700 1 --2- 172.21.15.6:0/2126659308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e854b0 0x555f62e878a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dade00700 1 -- 172.21.15.6:0/2126659308 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f62d69760 con 0x555f62e79d20 2024-04-26T02:04:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dae800700 1 --2- 172.21.15.6:0/2126659308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e854b0 0x555f62e878a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dad400700 1 --2- 172.21.15.6:0/2126659308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e7a640 0x555f62e84f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dae800700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5d940a3db0 0x7f5d940a6270 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5da800f590 tx=0x7f5da8009920 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.378+0000 7f5dade00700 1 --2- 172.21.15.6:0/2126659308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e79d20 0x555f62e7a100 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5da00ce0c0 tx=0x7f5da009c580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1c3232e5d6c0c89 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.379+0000 7f5d9b400700 1 -- 172.21.15.6:0/2126659308 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da00a5580 con 0x555f62e79d20 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.379+0000 7f5d9b400700 1 -- 172.21.15.6:0/2126659308 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da0085e50 con 0x555f62e79d20 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.379+0000 7f5d9b400700 1 -- 172.21.15.6:0/2126659308 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da00a50f0 con 0x555f62e79d20 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.379+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2126659308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e79d20 msgr2=0x555f62e7a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.379+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2126659308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e79d20 0x555f62e7a100 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5da00ce0c0 tx=0x7f5da009c580 comp rx=0 tx=0).stop 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.380+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2126659308 shutdown_connections 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.380+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2126659308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e854b0 0x555f62e878a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.380+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2126659308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e79d20 0x555f62e7a100 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.380+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2126659308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e7a640 0x555f62e84f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.380+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2126659308 >> 172.21.15.6:0/2126659308 conn(0x555f62da9030 msgr2=0x555f62e76230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.380+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2126659308 shutdown_connections 2024-04-26T02:04:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.380+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2126659308 wait complete. 2024-04-26T02:04:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.380+0000 7f5dbf4b1b80 1 -- start start 2024-04-26T02:04:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.381+0000 7f5dbf4b1b80 1 Processor -- start 2024-04-26T02:04:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.381+0000 7f5dbf4b1b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e84cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.381+0000 7f5dbf4b1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e81fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.381+0000 7f5dbf4b1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e82510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.381+0000 7f5dbf4b1b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x555f62e87f40 con 0x555f62e79d20 2024-04-26T02:04:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.381+0000 7f5dbf4b1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x555f62d86940 con 0x555f62e854b0 2024-04-26T02:04:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.381+0000 7f5dbf4b1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555f62e880c0 con 0x555f62e7a640 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.381+0000 7f5dbf4b1b80 10 client.0 _ll_register_callbacks cb 0x555f62d65800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.382+0000 7f5dade00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e84cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.382+0000 7f5dade00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e84cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41348/0 (socket says 172.21.15.6:41348) 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.382+0000 7f5dade00700 1 -- 172.21.15.6:0/186346772 learned_addr learned my addr 172.21.15.6:0/186346772 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.382+0000 7f5dad400700 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e81fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.382+0000 7f5dae800700 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e82510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.382+0000 7f5dad400700 1 -- 172.21.15.6:0/186346772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 msgr2=0x555f62e82510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.382+0000 7f5dad400700 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e82510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.382+0000 7f5dad400700 1 -- 172.21.15.6:0/186346772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 msgr2=0x555f62e84cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.382+0000 7f5dad400700 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e84cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.382+0000 7f5dad400700 1 -- 172.21.15.6:0/186346772 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f62d1a9b0 con 0x555f62e7a640 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.382+0000 7f5dae800700 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e82510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.383+0000 7f5dad400700 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e81fd0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5db006c260 tx=0x7f5d9c005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e57c6741fed05c9c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.383+0000 7f5d8fe00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:04:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.383+0000 7f5d8fe00700 1 -- 172.21.15.6:0/186346772 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c029060 con 0x555f62e7a640 2024-04-26T02:04:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.383+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/186346772 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5da0085e50 con 0x555f62e7a640 2024-04-26T02:04:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.383+0000 7f5d8fe00700 1 -- 172.21.15.6:0/186346772 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f5d9c003680 con 0x555f62e7a640 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.383+0000 7f5d8fe00700 1 -- 172.21.15.6:0/186346772 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f5d9c0095d0 con 0x555f62e7a640 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.383+0000 7f5dbf4b1b80 10 client.27358 fetch_fsmap learned FSMap version 54 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.383+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/186346772 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x555f62d63bd0 con 0x555f62e7a640 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.384+0000 7f5d8fe00700 1 -- 172.21.15.6:0/186346772 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da0085e50 con 0x555f62e7a640 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.384+0000 7f5d8fe00700 1 -- 172.21.15.6:0/186346772 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 54) ==== 8342+0+0 (secure 0 0 0) 0x7f5d9c00fa60 con 0x555f62e7a640 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.384+0000 7f5dbf4b1b80 10 client.27358 fetch_fsmap finished waiting for FSMap version 54 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.384+0000 7f5dbf4b1b80 10 client.27358 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.384+0000 7f5dbf4b1b80 20 client.27358 populate_metadata read hostname 'smithi006' 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.384+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x555f62e94ca0 0x555f62e97090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.384+0000 7f5dbf4b1b80 4 client.27358 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.384+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/186346772 --> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f62dda760 con 0x555f62e94ca0 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.384+0000 7f5dade00700 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x555f62e94ca0 0x555f62e97090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.385+0000 7f5dade00700 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x555f62e94ca0 0x555f62e97090 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5e0bfddd52612198 server_cookie=733503656e11aa42 in_seq=0 out_seq=0 2024-04-26T02:04:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.385+0000 7f5d8fe00700 10 client.27358 ms_handle_connect on v2:172.21.15.29:6838/2325582359 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5d8fe00700 1 -- 172.21.15.6:0/186346772 <== mds.1 v2:172.21.15.29:6838/2325582359 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f62dda760 con 0x555f62e94ca0 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5d8fe00700 10 client.27358 handle_command_reply: tid=0 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5dbf4b1b80 1 client.27358 shutdown 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/186346772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 msgr2=0x555f62e81fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e81fd0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5db006c260 tx=0x7f5d9c005bc0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/186346772 shutdown_connections 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x555f62e94ca0 0x555f62e97090 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e82510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e84cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/186346772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e81fd0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5d8fe00700 0 client.27358 ms_handle_reset on v2:172.21.15.29:6838/2325582359 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/186346772 >> 172.21.15.6:0/186346772 conn(0x555f62da9030 msgr2=0x555f62e75e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/186346772 shutdown_connections 2024-04-26T02:04:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.386+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/186346772 wait complete. 2024-04-26T02:04:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.387+0000 7f5dbf4b1b80 20 client.27358 trim_cache size 0 max 16384 2024-04-26T02:04:26.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.406+0000 7f5dbf4b1b80 1 Processor -- start 2024-04-26T02:04:26.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.407+0000 7f5dbf4b1b80 1 -- start start 2024-04-26T02:04:26.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.407+0000 7f5dbf4b1b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e90e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.407+0000 7f5dbf4b1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e8d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.407+0000 7f5dbf4b1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e8d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.407+0000 7f5dbf4b1b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01b6450 con 0x555f62e79d20 2024-04-26T02:04:26.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.407+0000 7f5dbf4b1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01b3cb0 con 0x555f62e854b0 2024-04-26T02:04:26.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.407+0000 7f5dbf4b1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01b1b20 con 0x555f62e7a640 2024-04-26T02:04:26.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.407+0000 7f5dade00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e90e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.407+0000 7f5dade00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e90e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41362/0 (socket says 172.21.15.6:41362) 2024-04-26T02:04:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.407+0000 7f5dade00700 1 -- 172.21.15.6:0/2926987071 learned_addr learned my addr 172.21.15.6:0/2926987071 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.407+0000 7f5dad400700 1 --2- 172.21.15.6:0/2926987071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e8d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.407+0000 7f5dae800700 1 --2- 172.21.15.6:0/2926987071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e8d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.408+0000 7f5dade00700 1 -- 172.21.15.6:0/2926987071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 msgr2=0x555f62e8d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.408+0000 7f5dade00700 1 --2- 172.21.15.6:0/2926987071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e8d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.408+0000 7f5dade00700 1 -- 172.21.15.6:0/2926987071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 msgr2=0x555f62e8d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.408+0000 7f5dade00700 1 --2- 172.21.15.6:0/2926987071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e8d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.408+0000 7f5dade00700 1 -- 172.21.15.6:0/2926987071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f62e83e70 con 0x555f62e79d20 2024-04-26T02:04:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.408+0000 7f5dade00700 1 --2- 172.21.15.6:0/2926987071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e90e40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5da009abc0 tx=0x7f5da009ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992882c180a13914 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:26.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.408+0000 7f5d9aa00700 1 -- 172.21.15.6:0/2926987071 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da00870b0 con 0x555f62e79d20 2024-04-26T02:04:26.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.408+0000 7f5d9aa00700 1 -- 172.21.15.6:0/2926987071 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da0085e50 con 0x555f62e79d20 2024-04-26T02:04:26.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.408+0000 7f5d9aa00700 1 -- 172.21.15.6:0/2926987071 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da00a5350 con 0x555f62e79d20 2024-04-26T02:04:26.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.409+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2926987071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 msgr2=0x555f62e90e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.409+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2926987071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e90e40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5da009abc0 tx=0x7f5da009ca90 comp rx=0 tx=0).stop 2024-04-26T02:04:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.409+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2926987071 shutdown_connections 2024-04-26T02:04:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.409+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2926987071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e8d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.409+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2926987071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e90e40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.409+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2926987071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e8d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.409+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2926987071 >> 172.21.15.6:0/2926987071 conn(0x555f62da9030 msgr2=0x555f62e753e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.409+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2926987071 shutdown_connections 2024-04-26T02:04:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.409+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2926987071 wait complete. 2024-04-26T02:04:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.410+0000 7f5dbf4b1b80 1 -- start start 2024-04-26T02:04:26.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.410+0000 7f5dbf4b1b80 1 Processor -- start 2024-04-26T02:04:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.410+0000 7f5dbf4b1b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e8bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.410+0000 7f5dbf4b1b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e8c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.410+0000 7f5dbf4b1b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e8c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dbf4b1b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01b2170 con 0x555f62e79d20 2024-04-26T02:04:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dbf4b1b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01b35e0 con 0x555f62e854b0 2024-04-26T02:04:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dbf4b1b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01b2840 con 0x555f62e7a640 2024-04-26T02:04:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dbf4b1b80 10 client.0 _ll_register_callbacks cb 0x555f62d65800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dad400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e8c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dade00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e8bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dad400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e8c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:50226/0 (socket says 172.21.15.6:50226) 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dad400700 1 -- 172.21.15.6:0/2765096460 learned_addr learned my addr 172.21.15.6:0/2765096460 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dae800700 1 --2- 172.21.15.6:0/2765096460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e8c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dad400700 1 -- 172.21.15.6:0/2765096460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 msgr2=0x555f62e8c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dad400700 1 --2- 172.21.15.6:0/2765096460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e8c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dad400700 1 -- 172.21.15.6:0/2765096460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 msgr2=0x555f62e8bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dad400700 1 --2- 172.21.15.6:0/2765096460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e8bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.411+0000 7f5dad400700 1 -- 172.21.15.6:0/2765096460 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f62efb300 con 0x555f62e7a640 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.412+0000 7f5dad400700 1 --2- 172.21.15.6:0/2765096460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e8c460 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c0055c0 tx=0x7f5d9c017c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1df6e54580b136fa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.412+0000 7f5d8fe00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.412+0000 7f5d8fe00700 1 -- 172.21.15.6:0/2765096460 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c03b060 con 0x555f62e7a640 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.412+0000 7f5d8fe00700 1 -- 172.21.15.6:0/2765096460 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f5d9c003680 con 0x555f62e7a640 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.412+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2765096460 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5da0085e50 con 0x555f62e7a640 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.412+0000 7f5d8fe00700 1 -- 172.21.15.6:0/2765096460 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f5d9c0095d0 con 0x555f62e7a640 2024-04-26T02:04:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.412+0000 7f5dbf4b1b80 10 client.27364 fetch_fsmap learned FSMap version 54 2024-04-26T02:04:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.412+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2765096460 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x555f62efc5e0 con 0x555f62e7a640 2024-04-26T02:04:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.412+0000 7f5d8fe00700 1 -- 172.21.15.6:0/2765096460 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da0085e50 con 0x555f62e7a640 2024-04-26T02:04:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.413+0000 7f5d8fe00700 1 -- 172.21.15.6:0/2765096460 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 54) ==== 8342+0+0 (secure 0 0 0) 0x7f5d9c00f4f0 con 0x555f62e7a640 2024-04-26T02:04:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.413+0000 7f5dbf4b1b80 10 client.27364 fetch_fsmap finished waiting for FSMap version 54 2024-04-26T02:04:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.413+0000 7f5dbf4b1b80 10 client.27364 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-26T02:04:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.413+0000 7f5dbf4b1b80 20 client.27364 populate_metadata read hostname 'smithi006' 2024-04-26T02:04:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.413+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2765096460 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x555f62efcce0 0x555f62eff100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.413+0000 7f5dbf4b1b80 4 client.27364 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ef592526-1-mds.b.json"}] 2024-04-26T02:04:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.413+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2765096460 --> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ef592526-1-mds.b.json"}) -- 0x555f62e84940 con 0x555f62efcce0 2024-04-26T02:04:26.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.413+0000 7f5dade00700 1 --2- 172.21.15.6:0/2765096460 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x555f62efcce0 0x555f62eff100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:26.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.414+0000 7f5dade00700 1 --2- 172.21.15.6:0/2765096460 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x555f62efcce0 0x555f62eff100 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e528fc2ab6e660fb server_cookie=8c9ed70f6df974c4 in_seq=0 out_seq=0 2024-04-26T02:04:26.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.414+0000 7f5d8fe00700 10 client.27364 ms_handle_connect on v2:172.21.15.29:6838/2325582359 2024-04-26T02:04:26.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.436+0000 7f5d8fe00700 1 -- 172.21.15.6:0/2765096460 <== mds.1 v2:172.21.15.29:6838/2325582359 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x555f62e84940 con 0x555f62efcce0 2024-04-26T02:04:26.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.436+0000 7f5d8fe00700 10 client.27364 handle_command_reply: tid=0 2024-04-26T02:04:26.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.436+0000 7f5dbf4b1b80 1 client.27364 shutdown 2024-04-26T02:04:26.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.437+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2765096460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 msgr2=0x555f62e8c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:26.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.437+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2765096460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e8c460 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c0055c0 tx=0x7f5d9c017c80 comp rx=0 tx=0).stop 2024-04-26T02:04:26.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.437+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2765096460 shutdown_connections 2024-04-26T02:04:26.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.437+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2765096460 >> [v2:172.21.15.29:6838/2325582359,v1:172.21.15.29:6839/2325582359] conn(0x555f62efcce0 0x555f62eff100 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.437+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2765096460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555f62e854b0 0x555f62e8c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.437+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2765096460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x555f62e79d20 0x555f62e8bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.437+0000 7f5dbf4b1b80 1 --2- 172.21.15.6:0/2765096460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555f62e7a640 0x555f62e8c460 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.437+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2765096460 >> 172.21.15.6:0/2765096460 conn(0x555f62da9030 msgr2=0x555f62e75c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:26.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.437+0000 7f5d8fe00700 0 client.27364 ms_handle_reset on v2:172.21.15.29:6838/2325582359 2024-04-26T02:04:26.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.437+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2765096460 shutdown_connections 2024-04-26T02:04:26.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.437+0000 7f5dbf4b1b80 1 -- 172.21.15.6:0/2765096460 wait complete. 2024-04-26T02:04:26.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.437+0000 7f5dbf4b1b80 20 client.27364 trim_cache size 0 max 16384 2024-04-26T02:04:26.438 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:04:26.438 INFO:teuthology.orchestra.run.smithi006.stdout: "path": "/var/run/ceph/ops-ef592526-1-mds.b.json" 2024-04-26T02:04:26.438 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:04:26.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.438+0000 7f5db4c00700 1 -- 172.21.15.6:0/3061776462 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5d940a3db0 msgr2=0x7f5d940a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:26.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.438+0000 7f5db4c00700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5d940a3db0 0x7f5d940a6270 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5da800f590 tx=0x7f5da8009920 comp rx=0 tx=0).stop 2024-04-26T02:04:26.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.438+0000 7f5db4c00700 1 -- 172.21.15.6:0/3061776462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5db01092d0 msgr2=0x7f5db01a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:26.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.438+0000 7f5db4c00700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5db01092d0 0x7f5db01a5400 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5da000e9a0 tx=0x7f5da000ecb0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.439+0000 7f5db4c00700 1 -- 172.21.15.6:0/3061776462 shutdown_connections 2024-04-26T02:04:26.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.439+0000 7f5db4c00700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5d940a3db0 0x7f5d940a6270 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.439+0000 7f5db4c00700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5db010e3b0 0x7f5db019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.439+0000 7f5db4c00700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5db01092d0 0x7f5db01a5400 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.439+0000 7f5db4c00700 1 --2- 172.21.15.6:0/3061776462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5db0109c80 0x7f5db01a5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:26.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.439+0000 7f5db4c00700 1 -- 172.21.15.6:0/3061776462 >> 172.21.15.6:0/3061776462 conn(0x7f5db0076210 msgr2=0x7f5db010f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:26.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.439+0000 7f5db4c00700 1 -- 172.21.15.6:0/3061776462 shutdown_connections 2024-04-26T02:04:26.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:26.439+0000 7f5db4c00700 1 -- 172.21.15.6:0/3061776462 wait complete. 2024-04-26T02:04:26.801 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:26 smithi006 ceph-mon[28419]: pgmap v589: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-04-26T02:04:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:26 smithi029 ceph-mon[32775]: pgmap v589: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-04-26T02:04:27.126 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 16.02 MB/sec execute 170 sec latency 56369.724 ms 2024-04-26T02:04:27.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:26 smithi188 ceph-mon[32751]: pgmap v589: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-04-26T02:04:27.209 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 0 (i) to a remote file /var/run/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ops-ef592526-1-mds.i.json 2024-04-26T02:04:27.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph tell mds.1:0 ops --flags=locks --path=/var/run/ceph/ops-ef592526-1-mds.i.json 2024-04-26T02:04:27.694 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:04:28.126 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 15.93 MB/sec execute 171 sec latency 57369.845 ms 2024-04-26T02:04:28.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:28 smithi029 ceph-mon[32775]: pgmap v590: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 10 op/s 2024-04-26T02:04:29.030 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:28 smithi006 ceph-mon[28419]: pgmap v590: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 10 op/s 2024-04-26T02:04:29.126 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 15.84 MB/sec execute 172 sec latency 58369.973 ms 2024-04-26T02:04:29.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:28 smithi188 ceph-mon[32751]: pgmap v590: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 10 op/s 2024-04-26T02:04:30.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.123+0000 7fa7eb400700 1 -- 172.21.15.6:0/1745541927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e4102af0 msgr2=0x7fa7e4102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:30.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.123+0000 7fa7eb400700 1 --2- 172.21.15.6:0/1745541927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e4102af0 0x7fa7e4102f70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00099c0 tx=0x7fa7d0009cd0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.123+0000 7fa7eb400700 1 -- 172.21.15.6:0/1745541927 shutdown_connections 2024-04-26T02:04:30.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.123+0000 7fa7eb400700 1 --2- 172.21.15.6:0/1745541927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e41034b0 0x7fa7e410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.123+0000 7fa7eb400700 1 --2- 172.21.15.6:0/1745541927 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7e4107940 0x7fa7e4107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.123+0000 7fa7eb400700 1 --2- 172.21.15.6:0/1745541927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e4102af0 0x7fa7e4102f70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.123+0000 7fa7eb400700 1 -- 172.21.15.6:0/1745541927 >> 172.21.15.6:0/1745541927 conn(0x7fa7e4079ba0 msgr2=0x7fa7e4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:30.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.123+0000 7fa7eb400700 1 -- 172.21.15.6:0/1745541927 shutdown_connections 2024-04-26T02:04:30.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.123+0000 7fa7eb400700 1 -- 172.21.15.6:0/1745541927 wait complete. 2024-04-26T02:04:30.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.124+0000 7fa7eb400700 1 Processor -- start 2024-04-26T02:04:30.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.125+0000 7fa7eb400700 1 -- start start 2024-04-26T02:04:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.125+0000 7fa7eb400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e4102af0 0x7fa7e419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.125+0000 7fa7eb400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e41034b0 0x7fa7e419b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.125+0000 7fa7eb400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7e4107940 0x7fa7e419fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.125+0000 7fa7eb400700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4112cb0 con 0x7fa7e4102af0 2024-04-26T02:04:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.125+0000 7fa7eb400700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4112b30 con 0x7fa7e41034b0 2024-04-26T02:04:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.125+0000 7fa7eb400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4112e30 con 0x7fa7e4107940 2024-04-26T02:04:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.125+0000 7fa7e3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e41034b0 0x7fa7e419b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.125+0000 7fa7e8a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e4102af0 0x7fa7e419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.125+0000 7fa7e3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e41034b0 0x7fa7e419b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:38630/0 (socket says 172.21.15.6:38630) 2024-04-26T02:04:30.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.126+0000 7fa7e3e00700 1 -- 172.21.15.6:0/3531673283 learned_addr learned my addr 172.21.15.6:0/3531673283 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:30.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.126+0000 7fa7e9400700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7e4107940 0x7fa7e419fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:30.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.126+0000 7fa7e3e00700 1 -- 172.21.15.6:0/3531673283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7e4107940 msgr2=0x7fa7e419fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:30.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.126+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7e4107940 0x7fa7e419fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.126+0000 7fa7e3e00700 1 -- 172.21.15.6:0/3531673283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e4102af0 msgr2=0x7fa7e419b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:30.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.126+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e4102af0 0x7fa7e419b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.126+0000 7fa7e3e00700 1 -- 172.21.15.6:0/3531673283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e41a0410 con 0x7fa7e41034b0 2024-04-26T02:04:30.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.126+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e41034b0 0x7fa7e419b940 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0005910 tx=0x7fa7d0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:30.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.127+0000 7fa7e1600700 1 -- 172.21.15.6:0/3531673283 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0026070 con 0x7fa7e41034b0 2024-04-26T02:04:30.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.127+0000 7fa7eb400700 1 -- 172.21.15.6:0/3531673283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e41a06a0 con 0x7fa7e41034b0 2024-04-26T02:04:30.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.127+0000 7fa7eb400700 1 -- 172.21.15.6:0/3531673283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e41a7f80 con 0x7fa7e41034b0 2024-04-26T02:04:30.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.127+0000 7fa7e1600700 1 -- 172.21.15.6:0/3531673283 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7d0003c70 con 0x7fa7e41034b0 2024-04-26T02:04:30.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.127+0000 7fa7e1600700 1 -- 172.21.15.6:0/3531673283 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d000fa00 con 0x7fa7e41034b0 2024-04-26T02:04:30.127 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 15.75 MB/sec execute 173 sec latency 59370.071 ms 2024-04-26T02:04:30.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.130+0000 7fa7e1600700 1 -- 172.21.15.6:0/3531673283 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fa7d0029070 con 0x7fa7e41034b0 2024-04-26T02:04:30.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.132+0000 7fa7e1600700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa7cc0a3e30 0x7fa7cc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:30.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.132+0000 7fa7f5d81b80 1 Processor -- start 2024-04-26T02:04:30.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.132+0000 7fa7e1600700 1 -- 172.21.15.6:0/3531673283 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7fa7d00d0e80 con 0x7fa7e41034b0 2024-04-26T02:04:30.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.132+0000 7fa7f5d81b80 1 -- start start 2024-04-26T02:04:30.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.132+0000 7fa7e8a00700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa7cc0a3e30 0x7fa7cc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:30.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.132+0000 7fa7f5d81b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e9689900 0x5637e9689ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:30.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.132+0000 7fa7f5d81b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 0x5637e9694b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.132+0000 7fa7f5d81b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e96950d0 0x5637e96974c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7f5d81b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5637e95dcf70 con 0x5637e9689900 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7f5d81b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5637e9583810 con 0x5637e96950d0 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7f5d81b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5637e95d8f30 con 0x5637e968a220 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7e3e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 0x5637e9694b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7e3e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 0x5637e9694b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:40484/0 (socket says 172.21.15.6:40484) 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7e3e00700 1 -- 172.21.15.6:0/3556305181 learned_addr learned my addr 172.21.15.6:0/3556305181 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7e8a00700 1 --2- 172.21.15.6:0/3556305181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e9689900 0x5637e9689ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7e9400700 1 --2- 172.21.15.6:0/3556305181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e96950d0 0x5637e96974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7e3e00700 1 -- 172.21.15.6:0/3556305181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e96950d0 msgr2=0x5637e96974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/3556305181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e96950d0 0x5637e96974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7e3e00700 1 -- 172.21.15.6:0/3556305181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e9689900 msgr2=0x5637e9689ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/3556305181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e9689900 0x5637e9689ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7e8a00700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa7cc0a3e30 0x7fa7cc0a62f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8009c30 tx=0x7fa7d8011440 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:30.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7e3e00700 1 -- 172.21.15.6:0/3556305181 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637e9579760 con 0x5637e968a220 2024-04-26T02:04:30.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.133+0000 7fa7e9400700 1 --2- 172.21.15.6:0/3556305181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e96950d0 0x5637e96974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.134+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/3556305181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 0x5637e9694b90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00a54b0 tx=0x7fa7d009c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eebf95ed0d265e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.134+0000 7fa7d6000700 1 -- 172.21.15.6:0/3556305181 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0003e20 con 0x5637e968a220 2024-04-26T02:04:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.134+0000 7fa7d6000700 1 -- 172.21.15.6:0/3556305181 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7d00a0070 con 0x5637e968a220 2024-04-26T02:04:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.134+0000 7fa7d6000700 1 -- 172.21.15.6:0/3556305181 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d009b1e0 con 0x5637e968a220 2024-04-26T02:04:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.135+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/3556305181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 msgr2=0x5637e9694b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.135+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/3556305181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 0x5637e9694b90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00a54b0 tx=0x7fa7d009c9c0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.135+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/3556305181 shutdown_connections 2024-04-26T02:04:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.135+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/3556305181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e96950d0 0x5637e96974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.135+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/3556305181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e9689900 0x5637e9689ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.135+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/3556305181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 0x5637e9694b90 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.135+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/3556305181 >> 172.21.15.6:0/3556305181 conn(0x5637e95b8680 msgr2=0x5637e9685db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:30.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.135+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/3556305181 shutdown_connections 2024-04-26T02:04:30.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.135+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/3556305181 wait complete. 2024-04-26T02:04:30.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.135+0000 7fa7f5d81b80 1 -- start start 2024-04-26T02:04:30.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.136+0000 7fa7f5d81b80 1 Processor -- start 2024-04-26T02:04:30.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.136+0000 7fa7f5d81b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e9689900 0x5637e9694770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:30.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.136+0000 7fa7f5d81b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 0x5637e9691b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:30.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.136+0000 7fa7f5d81b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e96950d0 0x5637e9692050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.136+0000 7fa7f5d81b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5637e9697a00 con 0x5637e96950d0 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.136+0000 7fa7f5d81b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5637e9596940 con 0x5637e9689900 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.136+0000 7fa7f5d81b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5637e9697b80 con 0x5637e968a220 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.136+0000 7fa7f5d81b80 10 client.0 _ll_register_callbacks cb 0x5637e9575800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.137+0000 7fa7e9400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e96950d0 0x5637e9692050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.137+0000 7fa7e9400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e96950d0 0x5637e9692050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55936/0 (socket says 172.21.15.6:55936) 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.137+0000 7fa7e9400700 1 -- 172.21.15.6:0/147163591 learned_addr learned my addr 172.21.15.6:0/147163591 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.137+0000 7fa7e8a00700 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e9689900 0x5637e9694770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.137+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 0x5637e9691b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.137+0000 7fa7e3e00700 1 -- 172.21.15.6:0/147163591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e9689900 msgr2=0x5637e9694770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.137+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e9689900 0x5637e9694770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.137+0000 7fa7e3e00700 1 -- 172.21.15.6:0/147163591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e96950d0 msgr2=0x5637e9692050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.137+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e96950d0 0x5637e9692050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.137+0000 7fa7e3e00700 1 -- 172.21.15.6:0/147163591 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637e952a9b0 con 0x5637e968a220 2024-04-26T02:04:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.138+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 0x5637e9691b10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00a54b0 tx=0x7fa7d00ce4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d3cb4cc4412b152 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.138+0000 7fa7caa00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:04:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.138+0000 7fa7caa00700 1 -- 172.21.15.6:0/147163591 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d009ce00 con 0x5637e968a220 2024-04-26T02:04:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.138+0000 7fa7caa00700 1 -- 172.21.15.6:0/147163591 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7fa7d00a5550 con 0x5637e968a220 2024-04-26T02:04:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.138+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/147163591 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7d00a0070 con 0x5637e968a220 2024-04-26T02:04:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.138+0000 7fa7caa00700 1 -- 172.21.15.6:0/147163591 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fa7d00a11b0 con 0x5637e968a220 2024-04-26T02:04:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.138+0000 7fa7f5d81b80 10 client.27376 fetch_fsmap learned FSMap version 54 2024-04-26T02:04:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.138+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/147163591 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5637e9573bd0 con 0x5637e968a220 2024-04-26T02:04:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.138+0000 7fa7caa00700 1 -- 172.21.15.6:0/147163591 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7d00a0070 con 0x5637e968a220 2024-04-26T02:04:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.139+0000 7fa7caa00700 1 -- 172.21.15.6:0/147163591 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 54) ==== 8342+0+0 (secure 0 0 0) 0x7fa7d000fba0 con 0x5637e968a220 2024-04-26T02:04:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.139+0000 7fa7f5d81b80 10 client.27376 fetch_fsmap finished waiting for FSMap version 54 2024-04-26T02:04:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.139+0000 7fa7f5d81b80 10 client.27376 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-26T02:04:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.139+0000 7fa7f5d81b80 20 client.27376 populate_metadata read hostname 'smithi006' 2024-04-26T02:04:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.139+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5637e96a44a0 0x5637e96a6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.139+0000 7fa7f5d81b80 4 client.27376 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:04:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.139+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/147163591 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637e95ea760 con 0x5637e96a44a0 2024-04-26T02:04:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.139+0000 7fa7e8a00700 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5637e96a44a0 0x5637e96a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.140+0000 7fa7e8a00700 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5637e96a44a0 0x5637e96a6890 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c928eb2d962fa27b server_cookie=ffb432de2022683e in_seq=0 out_seq=0 2024-04-26T02:04:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:30.140+0000 7fa7caa00700 10 client.27376 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:30 smithi029 ceph-mon[32775]: pgmap v591: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-26T02:04:30.943 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:30 smithi188 ceph-mon[32751]: pgmap v591: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-26T02:04:31.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:30 smithi006 ceph-mon[28419]: pgmap v591: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-26T02:04:31.127 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 15.66 MB/sec execute 174 sec latency 60370.157 ms 2024-04-26T02:04:32.127 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 15.57 MB/sec execute 175 sec latency 61370.292 ms 2024-04-26T02:04:32.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:32 smithi029 ceph-mon[32775]: pgmap v592: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-04-26T02:04:33.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:32 smithi006 ceph-mon[28419]: pgmap v592: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-04-26T02:04:33.127 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82724 15.48 MB/sec execute 176 sec latency 62370.429 ms 2024-04-26T02:04:33.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:32 smithi188 ceph-mon[32751]: pgmap v592: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-04-26T02:04:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.638+0000 7f43e0c00640 1 -- 172.21.15.6:0/1110884473 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f2e6792320 con 0x55f2e68a9130 2024-04-26T02:04:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.638+0000 7f43e0c00640 10 client.27322 handle_command_reply: tid=0 2024-04-26T02:04:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.638+0000 7f43fcf89740 1 client.27322 shutdown 2024-04-26T02:04:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.638+0000 7f43fcf89740 1 -- 172.21.15.6:0/1110884473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 msgr2=0x55f2e682f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.638+0000 7f43fcf89740 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e682f290 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f43e4005780 tx=0x7f43e402b4e0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.639+0000 7f43fcf89740 1 -- 172.21.15.6:0/1110884473 shutdown_connections 2024-04-26T02:04:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.639+0000 7f43fcf89740 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55f2e68a9130 0x55f2e68ab550 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.639+0000 7f43fcf89740 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55f2e6829a00 0x55f2e682fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.639+0000 7f43fcf89740 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e681f140 0x55f2e682f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.639+0000 7f43fcf89740 1 --2- 172.21.15.6:0/1110884473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f2e681e820 0x55f2e682f290 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.639+0000 7f43e0c00640 0 client.27322 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.639+0000 7f43fcf89740 1 -- 172.21.15.6:0/1110884473 >> 172.21.15.6:0/1110884473 conn(0x55f2e681ad30 msgr2=0x55f2e6819e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.639+0000 7f43fcf89740 1 -- 172.21.15.6:0/1110884473 shutdown_connections 2024-04-26T02:04:33.639 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.640+0000 7f43fcf89740 1 -- 172.21.15.6:0/1110884473 wait complete. 2024-04-26T02:04:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.639+0000 7fa7caa00700 1 -- 172.21.15.6:0/147163591 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5637e95ea760 con 0x5637e96a44a0 2024-04-26T02:04:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.639+0000 7fa7caa00700 10 client.27376 handle_command_reply: tid=0 2024-04-26T02:04:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.640+0000 7fa7f5d81b80 1 client.27376 shutdown 2024-04-26T02:04:33.639 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.640+0000 7f43fcf89740 20 client.27322 trim_cache size 0 max 16384 2024-04-26T02:04:33.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:04:33.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:04:33.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:04:33.640 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:04:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.641+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/147163591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 msgr2=0x5637e9691b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.641+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 0x5637e9691b10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00a54b0 tx=0x7fa7d00ce4c0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.642+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/147163591 shutdown_connections 2024-04-26T02:04:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.642+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e96950d0 0x5637e9692050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.642+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e9689900 0x5637e9694770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.642+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5637e96a44a0 0x5637e96a6890 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.642+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/147163591 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e968a220 0x5637e9691b10 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.642+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/147163591 >> 172.21.15.6:0/147163591 conn(0x5637e95b8680 msgr2=0x5637e96859a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:33.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.642+0000 7f43fa200640 1 -- 172.21.15.6:0/1180198814 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43d80a3d90 msgr2=0x7f43d80a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:33.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.642+0000 7f43fa200640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43d80a3d90 0x7f43d80a6250 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55f2e681fdc0 tx=0x7f43e4005f90 comp rx=0 tx=0).stop 2024-04-26T02:04:33.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.643+0000 7f43fa200640 1 -- 172.21.15.6:0/1180198814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f41017a0 msgr2=0x7f43f41a0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:33.642 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.643+0000 7f43fa200640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f41017a0 0x7f43f41a0c40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00e9f0 tx=0x7f43dc00eeb0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.642+0000 7fa7caa00700 0 client.27376 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:33.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.642+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/147163591 shutdown_connections 2024-04-26T02:04:33.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.643+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/147163591 wait complete. 2024-04-26T02:04:33.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.643+0000 7fa7f5d81b80 20 client.27376 trim_cache size 0 max 16384 2024-04-26T02:04:33.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.643+0000 7f43fa200640 1 -- 172.21.15.6:0/1180198814 shutdown_connections 2024-04-26T02:04:33.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.643+0000 7f43fa200640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f43d80a3d90 0x7f43d80a6250 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.643+0000 7f43fa200640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f41017a0 0x7f43f41a0c40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.643+0000 7f43fa200640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43f410e9a0 0x7f43f419a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.643 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.643+0000 7f43fa200640 1 --2- 172.21.15.6:0/1180198814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f4100df0 0x7f43f41a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.643+0000 7f43fa200640 1 -- 172.21.15.6:0/1180198814 >> 172.21.15.6:0/1180198814 conn(0x7f43f4078000 msgr2=0x7f43f410fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:33.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.644+0000 7f43fa200640 1 -- 172.21.15.6:0/1180198814 shutdown_connections 2024-04-26T02:04:33.644 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:33.644+0000 7f43fa200640 1 -- 172.21.15.6:0/1180198814 wait complete. 2024-04-26T02:04:33.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.679+0000 7fa7f5d81b80 1 Processor -- start 2024-04-26T02:04:33.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.679+0000 7fa7f5d81b80 1 -- start start 2024-04-26T02:04:33.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.680+0000 7fa7f5d81b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e9689900 0x5637e96909e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:33.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.680+0000 7fa7f5d81b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e968a220 0x5637e96abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:33.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.680+0000 7fa7f5d81b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e96950d0 0x5637e96ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:33.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.680+0000 7fa7f5d81b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e41b1ee0 con 0x5637e96950d0 2024-04-26T02:04:33.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.680+0000 7fa7f5d81b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e41af740 con 0x5637e968a220 2024-04-26T02:04:33.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.680+0000 7fa7f5d81b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e41ad5b0 con 0x5637e9689900 2024-04-26T02:04:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.680+0000 7fa7e3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e968a220 0x5637e96abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.680+0000 7fa7e3e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e968a220 0x5637e96abfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:38670/0 (socket says 172.21.15.6:38670) 2024-04-26T02:04:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.680+0000 7fa7e3e00700 1 -- 172.21.15.6:0/31049576 learned_addr learned my addr 172.21.15.6:0/31049576 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.680+0000 7fa7e9400700 1 --2- 172.21.15.6:0/31049576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e96950d0 0x5637e96ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.680+0000 7fa7e8a00700 1 --2- 172.21.15.6:0/31049576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e9689900 0x5637e96909e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.681+0000 7fa7e3e00700 1 -- 172.21.15.6:0/31049576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e9689900 msgr2=0x5637e96909e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.681+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/31049576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e9689900 0x5637e96909e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.681+0000 7fa7e3e00700 1 -- 172.21.15.6:0/31049576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e96950d0 msgr2=0x5637e96ac4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.681+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/31049576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e96950d0 0x5637e96ac4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.681+0000 7fa7e3e00700 1 -- 172.21.15.6:0/31049576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637e9690080 con 0x5637e968a220 2024-04-26T02:04:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.681+0000 7fa7e9400700 1 --2- 172.21.15.6:0/31049576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e96950d0 0x5637e96ac4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:33.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.681+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/31049576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e968a220 0x5637e96abfb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0020000 tx=0x7fa7d00b7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45ab842ae83741d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:33.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.681+0000 7fa7d5600700 1 -- 172.21.15.6:0/31049576 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d009d500 con 0x5637e968a220 2024-04-26T02:04:33.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.681+0000 7fa7d5600700 1 -- 172.21.15.6:0/31049576 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7d00a0040 con 0x5637e968a220 2024-04-26T02:04:33.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.681+0000 7fa7d5600700 1 -- 172.21.15.6:0/31049576 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d00a5550 con 0x5637e968a220 2024-04-26T02:04:33.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.682+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/31049576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e968a220 msgr2=0x5637e96abfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:33.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.682+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/31049576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e968a220 0x5637e96abfb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0020000 tx=0x7fa7d00b7000 comp rx=0 tx=0).stop 2024-04-26T02:04:33.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.682+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/31049576 shutdown_connections 2024-04-26T02:04:33.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.682+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/31049576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e96950d0 0x5637e96ac4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.682+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/31049576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e9689900 0x5637e96909e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.682+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/31049576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e968a220 0x5637e96abfb0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.682+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/31049576 >> 172.21.15.6:0/31049576 conn(0x5637e95b8680 msgr2=0x5637e96902d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:33.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.683+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/31049576 shutdown_connections 2024-04-26T02:04:33.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.683+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/31049576 wait complete. 2024-04-26T02:04:33.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.683+0000 7fa7f5d81b80 1 -- start start 2024-04-26T02:04:33.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.684+0000 7fa7f5d81b80 1 Processor -- start 2024-04-26T02:04:33.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.684+0000 7fa7f5d81b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e9689900 0x5637e9698ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:33.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.684+0000 7fa7f5d81b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e968a220 0x5637e9699010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:33.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.684+0000 7fa7f5d81b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e96950d0 0x5637e9699590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:33.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.684+0000 7fa7f5d81b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e41adc00 con 0x5637e968a220 2024-04-26T02:04:33.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.684+0000 7fa7f5d81b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e41af070 con 0x5637e96950d0 2024-04-26T02:04:33.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.684+0000 7fa7f5d81b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e41ae2d0 con 0x5637e9689900 2024-04-26T02:04:33.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.685+0000 7fa7f5d81b80 10 client.0 _ll_register_callbacks cb 0x5637e9575800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:33.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.685+0000 7fa7e8a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e9689900 0x5637e9698ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:33.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.685+0000 7fa7e3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e968a220 0x5637e9699010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:33.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.685+0000 7fa7e3e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e968a220 0x5637e9699010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55960/0 (socket says 172.21.15.6:55960) 2024-04-26T02:04:33.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.685+0000 7fa7e8a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e9689900 0x5637e9698ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:40512/0 (socket says 172.21.15.6:40512) 2024-04-26T02:04:33.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.685+0000 7fa7e3e00700 1 -- 172.21.15.6:0/165466347 learned_addr learned my addr 172.21.15.6:0/165466347 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:33.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.685+0000 7fa7e9400700 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e96950d0 0x5637e9699590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:33.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.686+0000 7fa7e8a00700 1 -- 172.21.15.6:0/165466347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e96950d0 msgr2=0x5637e9699590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:33.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.686+0000 7fa7e8a00700 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e96950d0 0x5637e9699590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.686+0000 7fa7e8a00700 1 -- 172.21.15.6:0/165466347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e968a220 msgr2=0x5637e9699010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:33.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.686+0000 7fa7e8a00700 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e968a220 0x5637e9699010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.686+0000 7fa7e8a00700 1 -- 172.21.15.6:0/165466347 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637e9709b20 con 0x5637e9689900 2024-04-26T02:04:33.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.686+0000 7fa7e9400700 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e96950d0 0x5637e9699590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:04:33.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.686+0000 7fa7e8a00700 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e9689900 0x5637e9698ad0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8005d90 tx=0x7fa7d80047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e978ffabe7d2ad57 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:33.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.686+0000 7fa7caa00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:04:33.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.686+0000 7fa7caa00700 1 -- 172.21.15.6:0/165466347 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d802e060 con 0x5637e9689900 2024-04-26T02:04:33.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.686+0000 7fa7caa00700 1 -- 172.21.15.6:0/165466347 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7fa7d801dc70 con 0x5637e9689900 2024-04-26T02:04:33.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.686+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/165466347 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7d00a0040 con 0x5637e9689900 2024-04-26T02:04:33.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.687+0000 7fa7caa00700 1 -- 172.21.15.6:0/165466347 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fa7d8017040 con 0x5637e9689900 2024-04-26T02:04:33.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.687+0000 7fa7f5d81b80 10 client.27382 fetch_fsmap learned FSMap version 54 2024-04-26T02:04:33.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.687+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/165466347 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5637e970ae60 con 0x5637e9689900 2024-04-26T02:04:33.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.687+0000 7fa7caa00700 1 -- 172.21.15.6:0/165466347 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7d00a0040 con 0x5637e9689900 2024-04-26T02:04:33.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.687+0000 7fa7caa00700 1 -- 172.21.15.6:0/165466347 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 54) ==== 8342+0+0 (secure 0 0 0) 0x7fa7d800f810 con 0x5637e9689900 2024-04-26T02:04:33.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.688+0000 7fa7f5d81b80 10 client.27382 fetch_fsmap finished waiting for FSMap version 54 2024-04-26T02:04:33.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.688+0000 7fa7f5d81b80 10 client.27382 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-26T02:04:33.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.688+0000 7fa7f5d81b80 20 client.27382 populate_metadata read hostname 'smithi006' 2024-04-26T02:04:33.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.688+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5637e970b540 0x5637e970d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:33.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.688+0000 7fa7f5d81b80 4 client.27382 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ef592526-1-mds.i.json"}] 2024-04-26T02:04:33.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.688+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/165466347 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ef592526-1-mds.i.json"}) -- 0x5637e95e4800 con 0x5637e970b540 2024-04-26T02:04:33.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.688+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5637e970b540 0x5637e970d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:33.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.689+0000 7fa7e3e00700 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5637e970b540 0x5637e970d960 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d967cc92f5eb30 server_cookie=56c0f525513a6ced in_seq=0 out_seq=0 2024-04-26T02:04:33.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.689+0000 7fa7caa00700 10 client.27382 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.694+0000 7fa7caa00700 1 -- 172.21.15.6:0/165466347 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x5637e95e4800 con 0x5637e970b540 2024-04-26T02:04:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.694+0000 7fa7caa00700 10 client.27382 handle_command_reply: tid=0 2024-04-26T02:04:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.694+0000 7fa7f5d81b80 1 client.27382 shutdown 2024-04-26T02:04:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.694+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/165466347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e9689900 msgr2=0x5637e9698ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.694+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e9689900 0x5637e9698ad0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8005d90 tx=0x7fa7d80047c0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.694+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/165466347 shutdown_connections 2024-04-26T02:04:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.694+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637e96950d0 0x5637e9699590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.694+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5637e970b540 0x5637e970d960 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.694+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5637e9689900 0x5637e9698ad0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.694+0000 7fa7f5d81b80 1 --2- 172.21.15.6:0/165466347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5637e968a220 0x5637e9699010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.694+0000 7fa7caa00700 0 client.27382 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:33.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.694+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/165466347 >> 172.21.15.6:0/165466347 conn(0x5637e95b8680 msgr2=0x5637e9690b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:33.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.695+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/165466347 shutdown_connections 2024-04-26T02:04:33.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.695+0000 7fa7f5d81b80 1 -- 172.21.15.6:0/165466347 wait complete. 2024-04-26T02:04:33.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.695+0000 7fa7f5d81b80 20 client.27382 trim_cache size 0 max 16384 2024-04-26T02:04:33.695 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:04:33.695 INFO:teuthology.orchestra.run.smithi006.stdout: "path": "/var/run/ceph/ops-ef592526-1-mds.i.json" 2024-04-26T02:04:33.695 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:04:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.696+0000 7fa7eb400700 1 -- 172.21.15.6:0/3531673283 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa7cc0a3e30 msgr2=0x7fa7cc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.696+0000 7fa7eb400700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa7cc0a3e30 0x7fa7cc0a62f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8009c30 tx=0x7fa7d8011440 comp rx=0 tx=0).stop 2024-04-26T02:04:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.696+0000 7fa7eb400700 1 -- 172.21.15.6:0/3531673283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e41034b0 msgr2=0x7fa7e419b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.696+0000 7fa7eb400700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e41034b0 0x7fa7e419b940 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0005910 tx=0x7fa7d0004270 comp rx=0 tx=0).stop 2024-04-26T02:04:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.697+0000 7fa7eb400700 1 -- 172.21.15.6:0/3531673283 shutdown_connections 2024-04-26T02:04:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.697+0000 7fa7eb400700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa7cc0a3e30 0x7fa7cc0a62f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.697+0000 7fa7eb400700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7e4107940 0x7fa7e419fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.697+0000 7fa7eb400700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e4102af0 0x7fa7e419b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.697+0000 7fa7eb400700 1 --2- 172.21.15.6:0/3531673283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e41034b0 0x7fa7e419b940 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.697+0000 7fa7eb400700 1 -- 172.21.15.6:0/3531673283 >> 172.21.15.6:0/3531673283 conn(0x7fa7e4079ba0 msgr2=0x7fa7e4100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:33.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.697+0000 7fa7eb400700 1 -- 172.21.15.6:0/3531673283 shutdown_connections 2024-04-26T02:04:33.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:33.697+0000 7fa7eb400700 1 -- 172.21.15.6:0/3531673283 wait complete. 2024-04-26T02:04:34.127 INFO:tasks.workunit.client.0.smithi006.stdout: 10 82927 15.43 MB/sec execute 177 sec latency 62834.369 ms 2024-04-26T02:04:34.519 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-04-26T02:04:34.538 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.uW3cYYthIo 2024-04-26T02:04:34.539 DEBUG:teuthology.orchestra.run.smithi188:> sudo cp /var/run/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ops-ef592526-1-mds.c.json /tmp/tmp.uW3cYYthIo 2024-04-26T02:04:34.623 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.uW3cYYthIo 2024-04-26T02:04:34.771 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.uW3cYYthIo is 656KB 2024-04-26T02:04:34.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:34 smithi006 ceph-mon[28419]: pgmap v593: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.8 KiB/s wr, 6 op/s 2024-04-26T02:04:34.869 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.uW3cYYthIo 2024-04-26T02:04:34.887 DEBUG:tasks.quiescer.fs.[cephfs]:read 672110B of ops from '/var/run/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ops-ef592526-1-mds.c.json' on mds.2 (c) 2024-04-26T02:04:34.928 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_path:mds.2:1612 fp=#0x1' 2024-04-26T02:04:34.929 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1510' 2024-04-26T02:04:34.929 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1805 fp=#0x10000001c25' 2024-04-26T02:04:34.930 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1644' 2024-04-26T02:04:34.931 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:client.25058:326375' 2024-04-26T02:04:34.931 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1613 fp=#0x1' 2024-04-26T02:04:34.932 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1749 fp=#0x10000001bdf' 2024-04-26T02:04:34.932 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1513' 2024-04-26T02:04:34.932 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1806 fp=#0x10000001c11' 2024-04-26T02:04:34.933 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1645' 2024-04-26T02:04:34.934 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1750 fp=#0x10000001bed' 2024-04-26T02:04:34.935 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1646' 2024-04-26T02:04:34.935 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op fragmentdir:mds.2:1957' 2024-04-26T02:04:34.935 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1614 fp=#0x10000000000' 2024-04-26T02:04:34.936 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1751 fp=#0x10000001c1f' 2024-04-26T02:04:34.936 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1514' 2024-04-26T02:04:34.936 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1572' 2024-04-26T02:04:34.937 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1647' 2024-04-26T02:04:34.938 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1615 fp=#0x10000000001' 2024-04-26T02:04:34.938 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1635 fp=#0x10000001a55' 2024-04-26T02:04:34.938 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1752 fp=#0x10000001c56' 2024-04-26T02:04:34.939 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1573' 2024-04-26T02:04:34.940 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1648' 2024-04-26T02:04:34.940 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1753 fp=#0x10000001c0a' 2024-04-26T02:04:34.940 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1515' 2024-04-26T02:04:34.942 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1649' 2024-04-26T02:04:34.942 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1754 fp=#0x10000001c7c' 2024-04-26T02:04:34.943 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1516' 2024-04-26T02:04:34.943 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1662' 2024-04-26T02:04:34.944 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1650' 2024-04-26T02:04:34.944 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1755 fp=#0x10000001c87' 2024-04-26T02:04:34.945 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1663' 2024-04-26T02:04:34.946 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1651' 2024-04-26T02:04:34.946 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op fragmentdir:mds.2:1610' 2024-04-26T02:04:34.946 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1616 fp=#0x100000001fa' 2024-04-26T02:04:34.947 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1756 fp=#0x10000001c32' 2024-04-26T02:04:34.948 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1652' 2024-04-26T02:04:34.949 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1757 fp=#0x10000001c68' 2024-04-26T02:04:34.950 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1653' 2024-04-26T02:04:34.950 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1641 fp=#0x10000001a5e' 2024-04-26T02:04:34.951 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1758 fp=#0x10000001c92' 2024-04-26T02:04:34.952 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1654' 2024-04-26T02:04:34.953 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1617 fp=#0x100000001fb' 2024-04-26T02:04:34.954 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1759 fp=#0x10000001c3b' 2024-04-26T02:04:34.955 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1624' 2024-04-26T02:04:34.955 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1655' 2024-04-26T02:04:34.956 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1702 fp=#0x10000001bd3' 2024-04-26T02:04:34.956 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1760 fp=#0x10000001bd6' 2024-04-26T02:04:34.957 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1664' 2024-04-26T02:04:34.957 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1625' 2024-04-26T02:04:34.957 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1656' 2024-04-26T02:04:34.958 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1761 fp=#0x10000001bf6' 2024-04-26T02:04:34.959 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1665' 2024-04-26T02:04:34.960 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1626' 2024-04-26T02:04:34.960 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1657' 2024-04-26T02:04:34.961 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1762 fp=#0x10000001c5f' 2024-04-26T02:04:34.961 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1787 fp=#0x10000001bf3' 2024-04-26T02:04:34.961 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1666' 2024-04-26T02:04:34.962 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1627' 2024-04-26T02:04:34.962 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1658' 2024-04-26T02:04:34.962 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1618 fp=#0x10000000005' 2024-04-26T02:04:34.963 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1507' 2024-04-26T02:04:34.963 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1763 fp=#0x10000001c44' 2024-04-26T02:04:34.963 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1788 fp=#0x10000001bfd' 2024-04-26T02:04:34.963 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1574' 2024-04-26T02:04:34.964 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1667' 2024-04-26T02:04:34.964 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1628' 2024-04-26T02:04:34.964 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1659' 2024-04-26T02:04:34.965 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1764 fp=#0x10000001c71' 2024-04-26T02:04:34.965 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1789 fp=#0x10000001c2f' 2024-04-26T02:04:34.965 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1575' 2024-04-26T02:04:34.965 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1668' 2024-04-26T02:04:34.966 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1629' 2024-04-26T02:04:34.966 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1619 fp=#0x10000000006' 2024-04-26T02:04:34.967 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1765 fp=#0x10000001c4c' 2024-04-26T02:04:34.967 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1790 fp=#0x10000001c66' 2024-04-26T02:04:34.967 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1576' 2024-04-26T02:04:34.968 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1669' 2024-04-26T02:04:34.968 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1630' 2024-04-26T02:04:34.968 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1497' 2024-04-26T02:04:34.969 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1766 fp=#0x10000001c28' 2024-04-26T02:04:34.969 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1791 fp=#0x10000001c1c' 2024-04-26T02:04:34.969 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1577' 2024-04-26T02:04:34.970 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1670' 2024-04-26T02:04:34.970 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1631' 2024-04-26T02:04:34.970 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'client_request(client.25058:323817 lookup #0x10000001be5/COURSES 2024-04-26T02:03:31.570006+0000 FWD=1 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:04:34.971 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1620 fp=#0x10000000007' 2024-04-26T02:04:34.971 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1767 fp=#0x10000001c14' 2024-04-26T02:04:34.972 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1792 fp=#0x10000001c90' 2024-04-26T02:04:34.972 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1671' 2024-04-26T02:04:34.972 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1632' 2024-04-26T02:04:34.973 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1768 fp=#0x10000001c00' 2024-04-26T02:04:34.973 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1793 fp=#0x10000001c9a' 2024-04-26T02:04:34.974 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1672' 2024-04-26T02:04:34.975 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1633' 2024-04-26T02:04:34.975 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1511' 2024-04-26T02:04:34.976 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1794 fp=#0x10000001c41' 2024-04-26T02:04:34.976 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1673' 2024-04-26T02:04:34.976 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1634' 2024-04-26T02:04:34.977 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'client_request(client.25058:323228 lookup #0x10000001bd3/COURSES 2024-04-26T02:03:31.163011+0000 FWD=1 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:04:34.977 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1711 fp=#0x10000001be5' 2024-04-26T02:04:34.978 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1795 fp=#0x10000001c79' 2024-04-26T02:04:34.978 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1578' 2024-04-26T02:04:34.979 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1674' 2024-04-26T02:04:34.979 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1635' 2024-04-26T02:04:34.979 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op fragmentdir:mds.2:1611' 2024-04-26T02:04:34.980 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1796 fp=#0x10000001ca4' 2024-04-26T02:04:34.981 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1675' 2024-04-26T02:04:34.981 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1636' 2024-04-26T02:04:34.983 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1797 fp=#0x10000001c4a' 2024-04-26T02:04:34.983 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1579' 2024-04-26T02:04:34.984 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1487' 2024-04-26T02:04:34.984 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1637' 2024-04-26T02:04:34.985 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1512' 2024-04-26T02:04:34.985 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1798 fp=#0x10000001be9' 2024-04-26T02:04:34.986 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1676' 2024-04-26T02:04:34.986 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1638' 2024-04-26T02:04:34.986 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1625 fp=#0x10000001a54' 2024-04-26T02:04:34.987 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1506' 2024-04-26T02:04:34.988 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1799 fp=#0x10000001c07' 2024-04-26T02:04:34.988 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1677' 2024-04-26T02:04:34.989 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1639' 2024-04-26T02:04:34.989 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1626 fp=#0x10000001a5d' 2024-04-26T02:04:34.990 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1508' 2024-04-26T02:04:34.990 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1800 fp=#0x10000001c6f' 2024-04-26T02:04:34.991 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1678' 2024-04-26T02:04:34.991 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1640' 2024-04-26T02:04:34.992 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1801 fp=#0x10000001c54' 2024-04-26T02:04:34.993 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1580' 2024-04-26T02:04:34.993 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1490' 2024-04-26T02:04:34.993 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1641' 2024-04-26T02:04:34.995 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1802 fp=#0x10000001c84' 2024-04-26T02:04:34.996 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1679' 2024-04-26T02:04:34.996 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1642' 2024-04-26T02:04:34.996 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1498' 2024-04-26T02:04:34.997 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1803 fp=#0x10000001c5d' 2024-04-26T02:04:34.998 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1530' 2024-04-26T02:04:34.998 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'client_request(client.25058:326368 create owner_uid=1000, owner_gid=1316 #0x10000001aab/HANDOUT.DOC 2024-04-26T02:03:33.724982+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:04:34.999 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.1:1509' 2024-04-26T02:04:34.999 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'internal op quiesce_inode:mds.2:1804 fp=#0x10000001c38' 2024-04-26T02:04:35.000 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1534' 2024-04-26T02:04:35.000 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (c) for ef592526-1: 'peer_request:mds.0:1643' 2024-04-26T02:04:35.001 INFO:tasks.quiescer.fs.[cephfs]:Pulled 431 ops from rank 2 (c) into home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/ops-ef592526-1-mds.c.json 2024-04-26T02:04:35.001 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -- /var/run/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ops-ef592526-1-mds.c.json 2024-04-26T02:04:35.002 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:34 smithi188 ceph-mon[32751]: pgmap v593: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.8 KiB/s wr, 6 op/s 2024-04-26T02:04:35.028 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-04-26T02:04:35.045 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.trG2gImZeP 2024-04-26T02:04:35.045 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /var/run/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ops-ef592526-1-mds.b.json /tmp/tmp.trG2gImZeP 2024-04-26T02:04:35.110 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:34 smithi029 ceph-mon[32775]: pgmap v593: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.8 KiB/s wr, 6 op/s 2024-04-26T02:04:35.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.trG2gImZeP 2024-04-26T02:04:35.127 INFO:tasks.workunit.client.0.smithi006.stdout: 10 83427 15.46 MB/sec execute 178 sec latency 102.300 ms 2024-04-26T02:04:35.245 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.trG2gImZeP is 667KB 2024-04-26T02:04:35.319 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.trG2gImZeP 2024-04-26T02:04:35.337 DEBUG:tasks.quiescer.fs.[cephfs]:read 682888B of ops from '/var/run/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ops-ef592526-1-mds.b.json' on mds.1 (b) 2024-04-26T02:04:35.350 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1725 fp=#0x10000001c5a' 2024-04-26T02:04:35.350 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1475 fp=#0x10000001a59' 2024-04-26T02:04:35.352 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1726 fp=#0x10000001c35' 2024-04-26T02:04:35.355 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1727 fp=#0x10000001c0e' 2024-04-26T02:04:35.355 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1465' 2024-04-26T02:04:35.356 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1744' 2024-04-26T02:04:35.356 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1728 fp=#0x2000000304c' 2024-04-26T02:04:35.357 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1475' 2024-04-26T02:04:35.358 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1745' 2024-04-26T02:04:35.358 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1729 fp=#0x10000001bf9' 2024-04-26T02:04:35.359 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1834' 2024-04-26T02:04:35.359 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1602' 2024-04-26T02:04:35.360 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1730 fp=#0x10000001be2' 2024-04-26T02:04:35.361 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1603' 2024-04-26T02:04:35.362 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1731 fp=#0x20000003037' 2024-04-26T02:04:35.363 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1604' 2024-04-26T02:04:35.365 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1605' 2024-04-26T02:04:35.366 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1664' 2024-04-26T02:04:35.367 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1606' 2024-04-26T02:04:35.367 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1647' 2024-04-26T02:04:35.368 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1652' 2024-04-26T02:04:35.368 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1737' 2024-04-26T02:04:35.370 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1665' 2024-04-26T02:04:35.371 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1738' 2024-04-26T02:04:35.372 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op fragmentdir:mds.1:1462' 2024-04-26T02:04:35.372 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1655' 2024-04-26T02:04:35.373 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1739' 2024-04-26T02:04:35.375 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1740' 2024-04-26T02:04:35.376 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1485 fp=#0x10000001a5a' 2024-04-26T02:04:35.379 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1666' 2024-04-26T02:04:35.381 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1615' 2024-04-26T02:04:35.381 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1540 fp=#0x10000001bad' 2024-04-26T02:04:35.381 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1656' 2024-04-26T02:04:35.382 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1708 fp=#0x10000001bc0' 2024-04-26T02:04:35.383 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1616' 2024-04-26T02:04:35.383 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_path:mds.1:1464 fp=#0x1' 2024-04-26T02:04:35.385 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1741' 2024-04-26T02:04:35.385 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1709 fp=#0x20000003050' 2024-04-26T02:04:35.386 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1617' 2024-04-26T02:04:35.386 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1465 fp=#0x1' 2024-04-26T02:04:35.387 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1742' 2024-04-26T02:04:35.387 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1710 fp=#0x10000001bcc' 2024-04-26T02:04:35.388 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1618' 2024-04-26T02:04:35.389 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1743' 2024-04-26T02:04:35.390 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1711 fp=#0x10000001c04' 2024-04-26T02:04:35.390 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1562' 2024-04-26T02:04:35.390 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1619' 2024-04-26T02:04:35.391 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1648' 2024-04-26T02:04:35.391 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1653' 2024-04-26T02:04:35.392 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1712 fp=#0x10000001c3e' 2024-04-26T02:04:35.392 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1563' 2024-04-26T02:04:35.393 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1620' 2024-04-26T02:04:35.393 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1466 fp=#0x10000000000' 2024-04-26T02:04:35.393 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1649' 2024-04-26T02:04:35.394 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1667' 2024-04-26T02:04:35.394 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1713 fp=#0x2000000303c' 2024-04-26T02:04:35.395 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1496' 2024-04-26T02:04:35.395 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1564' 2024-04-26T02:04:35.395 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1621' 2024-04-26T02:04:35.395 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1650' 2024-04-26T02:04:35.396 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1668' 2024-04-26T02:04:35.397 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1714 fp=#0x10000001bf0' 2024-04-26T02:04:35.397 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1565' 2024-04-26T02:04:35.397 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1622' 2024-04-26T02:04:35.397 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'client_request(client.25058:325543 lookup #0x10000001bad/* 2024-04-26T02:03:32.865991+0000 FWD=1 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:04:35.398 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1669' 2024-04-26T02:04:35.399 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1715 fp=#0x10000001c63' 2024-04-26T02:04:35.399 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1566' 2024-04-26T02:04:35.399 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1623' 2024-04-26T02:04:35.400 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1467 fp=#0x10000000001' 2024-04-26T02:04:35.401 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1670' 2024-04-26T02:04:35.401 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1716 fp=#0x10000001c6c' 2024-04-26T02:04:35.402 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1567' 2024-04-26T02:04:35.402 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1468 fp=#0x100000001fa' 2024-04-26T02:04:35.403 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1671' 2024-04-26T02:04:35.403 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1717 fp=#0x10000001c19' 2024-04-26T02:04:35.404 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1568' 2024-04-26T02:04:35.405 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1672' 2024-04-26T02:04:35.405 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1718 fp=#0x10000001c50' 2024-04-26T02:04:35.406 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1569' 2024-04-26T02:04:35.406 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op fragmentdir:mds.1:1812' 2024-04-26T02:04:35.406 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1469 fp=#0x100000001fb' 2024-04-26T02:04:35.408 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1719 fp=#0x10000001c76' 2024-04-26T02:04:35.408 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1570' 2024-04-26T02:04:35.409 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1470 fp=#0x10000000005' 2024-04-26T02:04:35.409 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1633' 2024-04-26T02:04:35.410 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1720 fp=#0x10000001c22' 2024-04-26T02:04:35.411 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1598' 2024-04-26T02:04:35.411 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op exportdir:mds.1:1814' 2024-04-26T02:04:35.411 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1471 fp=#0x10000000006' 2024-04-26T02:04:35.413 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1721 fp=#0x10000001bb5' 2024-04-26T02:04:35.413 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1599' 2024-04-26T02:04:35.414 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op exportdir:mds.1:1815' 2024-04-26T02:04:35.414 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1472 fp=#0x10000000007' 2024-04-26T02:04:35.414 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1634' 2024-04-26T02:04:35.415 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1722 fp=#0x10000001bd7' 2024-04-26T02:04:35.416 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1600' 2024-04-26T02:04:35.417 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1723 fp=#0x10000001c47' 2024-04-26T02:04:35.418 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.0:1601' 2024-04-26T02:04:35.419 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1651' 2024-04-26T02:04:35.419 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'peer_request:mds.2:1654' 2024-04-26T02:04:35.420 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'internal op quiesce_inode:mds.1:1724 fp=#0x10000001c2c' 2024-04-26T02:04:35.420 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (b) for ef592526-1: 'client_request(client.25058:326357 create owner_uid=1000, owner_gid=1316 #0x10000001b8c/PMD394.TMP 2024-04-26T02:03:33.578983+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:04:35.421 INFO:tasks.quiescer.fs.[cephfs]:Pulled 415 ops from rank 1 (b) into home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/ops-ef592526-1-mds.b.json 2024-04-26T02:04:35.421 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -- /var/run/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ops-ef592526-1-mds.b.json 2024-04-26T02:04:35.450 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-04-26T02:04:35.467 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.z4BEGV0xlw 2024-04-26T02:04:35.467 DEBUG:teuthology.orchestra.run.smithi188:> sudo cp /var/run/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ops-ef592526-1-mds.i.json /tmp/tmp.z4BEGV0xlw 2024-04-26T02:04:35.540 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.z4BEGV0xlw 2024-04-26T02:04:35.666 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.z4BEGV0xlw is 94KB 2024-04-26T02:04:35.689 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.z4BEGV0xlw 2024-04-26T02:04:35.705 DEBUG:tasks.quiescer.fs.[cephfs]:read 96360B of ops from '/var/run/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ops-ef592526-1-mds.i.json' on mds.0 (i) 2024-04-26T02:04:35.709 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (i) for ef592526-1: 'client_request(client.25058:326479 unlink #0x10000001be7/ANSWER.DB 2024-04-26T02:04:33.653300+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:04:35.711 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (i) for ef592526-1: 'client_request(client.25058:326536 create owner_uid=1000, owner_gid=1316 #0x10000001bde/__3F2C4.DB 2024-04-26T02:04:33.683300+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:04:35.713 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (i) for ef592526-1: 'client_request(client.25058:326548 create owner_uid=1000, owner_gid=1316 #0x10000001bab/__QB4.MB 2024-04-26T02:04:33.689300+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:04:35.715 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (i) for ef592526-1: 'client_request(client.25058:326466 unlink #0x10000001be7/ANSWER.MB 2024-04-26T02:04:33.643301+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:04:35.716 INFO:tasks.quiescer.fs.[cephfs]:Pulled 4 ops from rank 0 (i) into home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/ops-ef592526-1-mds.i.json 2024-04-26T02:04:35.717 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -- /var/run/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ops-ef592526-1-mds.i.json 2024-04-26T02:04:35.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:04:35.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:35 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:04:36.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:35 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:04:36.127 INFO:tasks.workunit.client.0.smithi006.stdout: 10 84121 15.55 MB/sec execute 179 sec latency 104.780 ms 2024-04-26T02:04:36.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:35 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:04:36.399 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:04:36.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:36 smithi006 ceph-mon[28419]: pgmap v594: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.8 KiB/s wr, 6 op/s 2024-04-26T02:04:36.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:36 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:04:37.127 INFO:tasks.workunit.client.0.smithi006.stdout: 10 84867 15.76 MB/sec execute 180 sec latency 68.559 ms 2024-04-26T02:04:37.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:36 smithi188 ceph-mon[32751]: pgmap v594: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.8 KiB/s wr, 6 op/s 2024-04-26T02:04:37.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:36 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:04:37.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:36 smithi029 ceph-mon[32775]: pgmap v594: 97 pgs: 97 active+clean; 851 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.8 KiB/s wr, 6 op/s 2024-04-26T02:04:37.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:36 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:04:38.127 INFO:tasks.workunit.client.0.smithi006.stdout: 10 85376 15.87 MB/sec execute 181 sec latency 75.497 ms 2024-04-26T02:04:38.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.614+0000 7f38e7200700 1 -- 172.21.15.6:0/1560026027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38e006b0f0 msgr2=0x7f38e010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:38.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.614+0000 7f38e7200700 1 --2- 172.21.15.6:0/1560026027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38e006b0f0 0x7f38e010f900 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009f90 tx=0x7f38d800b330 comp rx=0 tx=0).stop 2024-04-26T02:04:38.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.615+0000 7f38e7200700 1 -- 172.21.15.6:0/1560026027 shutdown_connections 2024-04-26T02:04:38.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.615+0000 7f38e7200700 1 --2- 172.21.15.6:0/1560026027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38e006b0f0 0x7f38e010f900 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:38.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.615+0000 7f38e7200700 1 --2- 172.21.15.6:0/1560026027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38e0108a30 0x7f38e0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:38.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.615+0000 7f38e7200700 1 --2- 172.21.15.6:0/1560026027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e006a750 0x7f38e006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:38.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.615+0000 7f38e7200700 1 -- 172.21.15.6:0/1560026027 >> 172.21.15.6:0/1560026027 conn(0x7f38e00788f0 msgr2=0x7f38e0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:38.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.615+0000 7f38e7200700 1 -- 172.21.15.6:0/1560026027 shutdown_connections 2024-04-26T02:04:38.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.615+0000 7f38e7200700 1 -- 172.21.15.6:0/1560026027 wait complete. 2024-04-26T02:04:38.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.616+0000 7f38e7200700 1 Processor -- start 2024-04-26T02:04:38.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.617+0000 7f38e7200700 1 -- start start 2024-04-26T02:04:38.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.617+0000 7f38e7200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e006a750 0x7f38e019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:38.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.617+0000 7f38e7200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38e006b0f0 0x7f38e019b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:38.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.618+0000 7f38e7200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38e0108a30 0x7f38e019fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:38.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.618+0000 7f38e7200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f38e0112c20 con 0x7f38e0108a30 2024-04-26T02:04:38.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.618+0000 7f38e7200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f38e0112aa0 con 0x7f38e006b0f0 2024-04-26T02:04:38.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.618+0000 7f38e7200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f38e0112da0 con 0x7f38e006a750 2024-04-26T02:04:38.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.618+0000 7f38dfe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e006a750 0x7f38e019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:38.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.618+0000 7f38dfe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e006a750 0x7f38e019b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:33916/0 (socket says 172.21.15.6:33916) 2024-04-26T02:04:38.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.618+0000 7f38dfe00700 1 -- 172.21.15.6:0/4168487846 learned_addr learned my addr 172.21.15.6:0/4168487846 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:38.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.618+0000 7f38df400700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38e006b0f0 0x7f38e019b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:38.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.618+0000 7f38dfe00700 1 -- 172.21.15.6:0/4168487846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38e006b0f0 msgr2=0x7f38e019b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:38.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.618+0000 7f38dfe00700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38e006b0f0 0x7f38e019b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:38.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.618+0000 7f38dfe00700 1 -- 172.21.15.6:0/4168487846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38e0108a30 msgr2=0x7f38e019fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:38.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.618+0000 7f38e5200700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38e0108a30 0x7f38e019fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:38.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.619+0000 7f38dfe00700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38e0108a30 0x7f38e019fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:38.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.619+0000 7f38dfe00700 1 -- 172.21.15.6:0/4168487846 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e01a02c0 con 0x7f38e006a750 2024-04-26T02:04:38.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.619+0000 7f38e5200700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38e0108a30 0x7f38e019fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:04:38.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.619+0000 7f38dfe00700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e006a750 0x7f38e019b300 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f38d000d950 tx=0x7f38d000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:38.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.619+0000 7f38dcc00700 1 -- 172.21.15.6:0/4168487846 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d0016070 con 0x7f38e006a750 2024-04-26T02:04:38.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.619+0000 7f38e7200700 1 -- 172.21.15.6:0/4168487846 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e01a05b0 con 0x7f38e006a750 2024-04-26T02:04:38.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.619+0000 7f38dcc00700 1 -- 172.21.15.6:0/4168487846 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38d0009df0 con 0x7f38e006a750 2024-04-26T02:04:38.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.619+0000 7f38dcc00700 1 -- 172.21.15.6:0/4168487846 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d000f3c0 con 0x7f38e006a750 2024-04-26T02:04:38.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.620+0000 7f38e7200700 1 -- 172.21.15.6:0/4168487846 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e01a7e30 con 0x7f38e006a750 2024-04-26T02:04:38.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.621+0000 7f38e7200700 1 -- 172.21.15.6:0/4168487846 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38e0069730 con 0x7f38e006a750 2024-04-26T02:04:38.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.623+0000 7f38dcc00700 1 -- 172.21.15.6:0/4168487846 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f38d0019070 con 0x7f38e006a750 2024-04-26T02:04:38.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.625+0000 7f38dcc00700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38c40a3df0 0x7f38c40a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:38.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.625+0000 7f38dcc00700 1 -- 172.21.15.6:0/4168487846 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f38d00d0410 con 0x7f38e006a750 2024-04-26T02:04:38.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.625+0000 7f38df400700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38c40a3df0 0x7f38c40a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:38.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.626+0000 7f38df400700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38c40a3df0 0x7f38c40a62b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f38cc009750 tx=0x7f38cc009310 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:38.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:38.632+0000 7f38dcc00700 1 -- 172.21.15.6:0/4168487846 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f38d009ca00 con 0x7f38e006a750 2024-04-26T02:04:39.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:38 smithi006 ceph-mon[28419]: pgmap v595: 97 pgs: 97 active+clean; 858 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-04-26T02:04:39.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:38 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:04:39.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:38 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:04:39.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.096+0000 7f38e7200700 1 -- 172.21.15.6:0/4168487846 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38e002df80 con 0x7f38e006a750 2024-04-26T02:04:39.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.097+0000 7f38dcc00700 1 -- 172.21.15.6:0/4168487846 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11853 (secure 0 0 0) 0x7f38d009c300 con 0x7f38e006a750 2024-04-26T02:04:39.096 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:04:39.096 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T02:04:33.934720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26935":{"gid":26935,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1750728695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1750728695},{"type":"v1","addr":"172.21.15.188:6835","nonce":1750728695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T02:04:39.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.101+0000 7f38e7200700 1 -- 172.21.15.6:0/4168487846 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38c40a3df0 msgr2=0x7f38c40a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:39.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.101+0000 7f38e7200700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38c40a3df0 0x7f38c40a62b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f38cc009750 tx=0x7f38cc009310 comp rx=0 tx=0).stop 2024-04-26T02:04:39.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.101+0000 7f38e7200700 1 -- 172.21.15.6:0/4168487846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e006a750 msgr2=0x7f38e019b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.101+0000 7f38e7200700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e006a750 0x7f38e019b300 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f38d000d950 tx=0x7f38d000dc60 comp rx=0 tx=0).stop 2024-04-26T02:04:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.102+0000 7f38e7200700 1 -- 172.21.15.6:0/4168487846 shutdown_connections 2024-04-26T02:04:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.102+0000 7f38e7200700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f38c40a3df0 0x7f38c40a62b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.102+0000 7f38e7200700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38e0108a30 0x7f38e019fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.102+0000 7f38e7200700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e006a750 0x7f38e019b300 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.102+0000 7f38e7200700 1 --2- 172.21.15.6:0/4168487846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38e006b0f0 0x7f38e019b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.102+0000 7f38e7200700 1 -- 172.21.15.6:0/4168487846 >> 172.21.15.6:0/4168487846 conn(0x7f38e00788f0 msgr2=0x7f38e010dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:39.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.103+0000 7f38e7200700 1 -- 172.21.15.6:0/4168487846 shutdown_connections 2024-04-26T02:04:39.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:39.103+0000 7f38e7200700 1 -- 172.21.15.6:0/4168487846 wait complete. 2024-04-26T02:04:39.103 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 55 2024-04-26T02:04:39.127 INFO:tasks.workunit.client.0.smithi006.stdout: 10 85795 15.89 MB/sec execute 182 sec latency 80.976 ms 2024-04-26T02:04:39.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:38 smithi188 ceph-mon[32751]: pgmap v595: 97 pgs: 97 active+clean; 858 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-04-26T02:04:39.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:38 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:04:39.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:38 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:04:39.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:38 smithi029 ceph-mon[32775]: pgmap v595: 97 pgs: 97 active+clean; 858 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-04-26T02:04:39.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:38 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:04:39.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:38 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:04:39.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:39 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/4168487846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:04:39.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:39 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:04:39.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:39 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:04:39.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:39 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:04:39.914 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --set-id ef592526 --await-for 42.0' 2024-04-26T02:04:39.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --set-id ef592526 --await-for 42.0 2024-04-26T02:04:40.092 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:39 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/4168487846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:04:40.093 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:39 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:04:40.093 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:39 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:04:40.093 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:39 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:04:40.127 INFO:tasks.workunit.client.0.smithi006.stdout: 10 86304 15.92 MB/sec execute 183 sec latency 74.855 ms 2024-04-26T02:04:40.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:39 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/4168487846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:04:40.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:39 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:04:40.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:39 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:04:40.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:39 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:04:40.456 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:04:40.932 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:40 smithi006 ceph-mon[28419]: pgmap v596: 97 pgs: 97 active+clean; 858 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-04-26T02:04:41.128 INFO:tasks.workunit.client.0.smithi006.stdout: 10 86840 15.97 MB/sec execute 184 sec latency 84.824 ms 2024-04-26T02:04:41.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:40 smithi188 ceph-mon[32751]: pgmap v596: 97 pgs: 97 active+clean; 858 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-04-26T02:04:41.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:40 smithi029 ceph-mon[32775]: pgmap v596: 97 pgs: 97 active+clean; 858 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-04-26T02:04:42.128 INFO:tasks.workunit.client.0.smithi006.stdout: 10 87469 16.07 MB/sec execute 185 sec latency 91.746 ms 2024-04-26T02:04:42.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:42 smithi006 ceph-mon[28419]: pgmap v597: 97 pgs: 97 active+clean; 889 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4.2 MiB/s wr, 322 op/s 2024-04-26T02:04:42.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.868+0000 7f0660e00700 1 -- 172.21.15.6:0/2888790452 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f065c10f4a0 msgr2=0x7f065c111890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:42.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.868+0000 7f0660e00700 1 --2- 172.21.15.6:0/2888790452 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f065c10f4a0 0x7f065c111890 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f065400b0a0 tx=0x7f065400b3b0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.868+0000 7f0660e00700 1 -- 172.21.15.6:0/2888790452 shutdown_connections 2024-04-26T02:04:42.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.868+0000 7f0660e00700 1 --2- 172.21.15.6:0/2888790452 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f065c10f4a0 0x7f065c111890 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.868+0000 7f0660e00700 1 --2- 172.21.15.6:0/2888790452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c074c70 0x7f065c075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.868+0000 7f0660e00700 1 --2- 172.21.15.6:0/2888790452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f065c075590 0x7f065c10ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.868+0000 7f0660e00700 1 -- 172.21.15.6:0/2888790452 >> 172.21.15.6:0/2888790452 conn(0x7f065c079b90 msgr2=0x7f065c079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:42.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.868+0000 7f0660e00700 1 -- 172.21.15.6:0/2888790452 shutdown_connections 2024-04-26T02:04:42.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.868+0000 7f0660e00700 1 -- 172.21.15.6:0/2888790452 wait complete. 2024-04-26T02:04:42.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.870+0000 7f0660e00700 1 Processor -- start 2024-04-26T02:04:42.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.870+0000 7f0660e00700 1 -- start start 2024-04-26T02:04:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.870+0000 7f0660e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c074c70 0x7f065c19b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.870+0000 7f0660e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f065c075590 0x7f065c19b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.870+0000 7f0660e00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f065c10f4a0 0x7f065c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0660e00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f065c114e90 con 0x7f065c10f4a0 2024-04-26T02:04:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0660e00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f065c114d10 con 0x7f065c074c70 2024-04-26T02:04:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0660e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f065c115010 con 0x7f065c075590 2024-04-26T02:04:42.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0659400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f065c075590 0x7f065c19b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0659e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c074c70 0x7f065c19b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0659e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c074c70 0x7f065c19b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:35978/0 (socket says 172.21.15.6:35978) 2024-04-26T02:04:42.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0659e00700 1 -- 172.21.15.6:0/365355268 learned_addr learned my addr 172.21.15.6:0/365355268 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:42.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0659400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f065c075590 0x7f065c19b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:33942/0 (socket says 172.21.15.6:33942) 2024-04-26T02:04:42.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0659e00700 1 -- 172.21.15.6:0/365355268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f065c075590 msgr2=0x7f065c19b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:42.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f065a800700 1 --2- 172.21.15.6:0/365355268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f065c10f4a0 0x7f065c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0659e00700 1 --2- 172.21.15.6:0/365355268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f065c075590 0x7f065c19b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0659e00700 1 -- 172.21.15.6:0/365355268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f065c10f4a0 msgr2=0x7f065c19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:42.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0659e00700 1 --2- 172.21.15.6:0/365355268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f065c10f4a0 0x7f065c19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0659e00700 1 -- 172.21.15.6:0/365355268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f065c1a03a0 con 0x7f065c074c70 2024-04-26T02:04:42.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.871+0000 7f0659400700 1 --2- 172.21.15.6:0/365355268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f065c075590 0x7f065c19b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:42.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.872+0000 7f065a800700 1 --2- 172.21.15.6:0/365355268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f065c10f4a0 0x7f065c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:04:42.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.872+0000 7f0659e00700 1 --2- 172.21.15.6:0/365355268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f065c074c70 0x7f065c19b3e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f064c00e9a0 tx=0x7f064c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:42.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.872+0000 7f0652a00700 1 -- 172.21.15.6:0/365355268 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c00cbd0 con 0x7f065c074c70 2024-04-26T02:04:42.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.872+0000 7f0660e00700 1 -- 172.21.15.6:0/365355268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f065c1a0690 con 0x7f065c074c70 2024-04-26T02:04:42.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.872+0000 7f0660e00700 1 -- 172.21.15.6:0/365355268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f065c1a7f10 con 0x7f065c074c70 2024-04-26T02:04:42.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.873+0000 7f0652a00700 1 -- 172.21.15.6:0/365355268 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f064c00cd70 con 0x7f065c074c70 2024-04-26T02:04:42.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.873+0000 7f0652a00700 1 -- 172.21.15.6:0/365355268 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c018680 con 0x7f065c074c70 2024-04-26T02:04:42.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.876+0000 7f0652a00700 1 -- 172.21.15.6:0/365355268 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f064c010430 con 0x7f065c074c70 2024-04-26T02:04:42.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.878+0000 7f0652a00700 1 --2- 172.21.15.6:0/365355268 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f06400a3e30 0x7f06400a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.879+0000 7f0652a00700 1 -- 172.21.15.6:0/365355268 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f064c0d09a0 con 0x7f065c074c70 2024-04-26T02:04:42.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.879+0000 7f0659400700 1 --2- 172.21.15.6:0/365355268 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f06400a3e30 0x7f06400a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.879+0000 7f066b7fbb80 1 Processor -- start 2024-04-26T02:04:42.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.880+0000 7f066b7fbb80 1 -- start start 2024-04-26T02:04:42.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.880+0000 7f0659400700 1 --2- 172.21.15.6:0/365355268 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f06400a3e30 0x7f06400a62f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f065c1026b0 tx=0x7f0648009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:42.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.880+0000 7f066b7fbb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f81d40 0x5634b2f82120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.880+0000 7f066b7fbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 0x5634b2f8cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.880+0000 7f066b7fbb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f8d4d0 0x5634b2f8f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.880+0000 7f066b7fbb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5634b2ed4f90 con 0x5634b2f81d40 2024-04-26T02:04:42.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.880+0000 7f066b7fbb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5634b2e7b830 con 0x5634b2f8d4d0 2024-04-26T02:04:42.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.880+0000 7f066b7fbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5634b2ed0f50 con 0x5634b2f82660 2024-04-26T02:04:42.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.881+0000 7f065a800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f8d4d0 0x5634b2f8f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.881+0000 7f0659400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 0x5634b2f8cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.881+0000 7f065a800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f8d4d0 0x5634b2f8f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:35982/0 (socket says 172.21.15.6:35982) 2024-04-26T02:04:42.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.881+0000 7f065a800700 1 -- 172.21.15.6:0/207471710 learned_addr learned my addr 172.21.15.6:0/207471710 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:42.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.881+0000 7f0659e00700 1 --2- 172.21.15.6:0/207471710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f81d40 0x5634b2f82120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.881+0000 7f0659400700 1 -- 172.21.15.6:0/207471710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f8d4d0 msgr2=0x5634b2f8f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:42.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.881+0000 7f0659400700 1 --2- 172.21.15.6:0/207471710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f8d4d0 0x5634b2f8f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.881+0000 7f0659400700 1 -- 172.21.15.6:0/207471710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f81d40 msgr2=0x5634b2f82120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:42.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.881+0000 7f0659400700 1 --2- 172.21.15.6:0/207471710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f81d40 0x5634b2f82120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.881+0000 7f0659400700 1 -- 172.21.15.6:0/207471710 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634b2e71780 con 0x5634b2f82660 2024-04-26T02:04:42.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.882+0000 7f0659400700 1 --2- 172.21.15.6:0/207471710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 0x5634b2f8cf90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f064800e650 tx=0x7f0648014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=252842b4d7c64f16 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:42.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.882+0000 7f0647400700 1 -- 172.21.15.6:0/207471710 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648026600 con 0x5634b2f82660 2024-04-26T02:04:42.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.882+0000 7f0647400700 1 -- 172.21.15.6:0/207471710 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0648026c50 con 0x5634b2f82660 2024-04-26T02:04:42.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.883+0000 7f0647400700 1 -- 172.21.15.6:0/207471710 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064801ec80 con 0x5634b2f82660 2024-04-26T02:04:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.884+0000 7f066b7fbb80 1 -- 172.21.15.6:0/207471710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 msgr2=0x5634b2f8cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.884+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/207471710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 0x5634b2f8cf90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f064800e650 tx=0x7f0648014000 comp rx=0 tx=0).stop 2024-04-26T02:04:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.885+0000 7f066b7fbb80 1 -- 172.21.15.6:0/207471710 shutdown_connections 2024-04-26T02:04:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.885+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/207471710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f8d4d0 0x5634b2f8f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.885+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/207471710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f81d40 0x5634b2f82120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.885+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/207471710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 0x5634b2f8cf90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.885+0000 7f066b7fbb80 1 -- 172.21.15.6:0/207471710 >> 172.21.15.6:0/207471710 conn(0x5634b2eb1050 msgr2=0x5634b2f7e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.885+0000 7f066b7fbb80 1 -- 172.21.15.6:0/207471710 shutdown_connections 2024-04-26T02:04:42.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.885+0000 7f066b7fbb80 1 -- 172.21.15.6:0/207471710 wait complete. 2024-04-26T02:04:42.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.886+0000 7f066b7fbb80 1 -- start start 2024-04-26T02:04:42.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.886+0000 7f066b7fbb80 1 Processor -- start 2024-04-26T02:04:42.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.887+0000 7f066b7fbb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f81d40 0x5634b2f8cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.887+0000 7f066b7fbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 0x5634b2f89ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.887+0000 7f066b7fbb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f8d4d0 0x5634b2f8a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.887+0000 7f066b7fbb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5634b2f90010 con 0x5634b2f81d40 2024-04-26T02:04:42.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.887+0000 7f066b7fbb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5634b2e8e960 con 0x5634b2f8d4d0 2024-04-26T02:04:42.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.887+0000 7f066b7fbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5634b2f90190 con 0x5634b2f82660 2024-04-26T02:04:42.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.887+0000 7f066b7fbb80 10 client.0 _ll_register_callbacks cb 0x5634b2e6d820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:42.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.887+0000 7f0659400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 0x5634b2f89ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.887+0000 7f0659400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 0x5634b2f89ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:33966/0 (socket says 172.21.15.6:33966) 2024-04-26T02:04:42.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.887+0000 7f0659400700 1 -- 172.21.15.6:0/1248279200 learned_addr learned my addr 172.21.15.6:0/1248279200 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:42.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.887+0000 7f065a800700 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f8d4d0 0x5634b2f8a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.887+0000 7f0659e00700 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f81d40 0x5634b2f8cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.888+0000 7f0659400700 1 -- 172.21.15.6:0/1248279200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f8d4d0 msgr2=0x5634b2f8a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:42.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.888+0000 7f0659400700 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f8d4d0 0x5634b2f8a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.888+0000 7f0659400700 1 -- 172.21.15.6:0/1248279200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f81d40 msgr2=0x5634b2f8cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:42.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.888+0000 7f0659400700 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f81d40 0x5634b2f8cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.888+0000 7f0659400700 1 -- 172.21.15.6:0/1248279200 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634b2e229d0 con 0x5634b2f82660 2024-04-26T02:04:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.888+0000 7f065a800700 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f8d4d0 0x5634b2f8a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.888+0000 7f0659400700 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 0x5634b2f89ea0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f06480056c0 tx=0x7f06480056f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba7f1714f18eb45 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.888+0000 7f063be00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:04:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.888+0000 7f063be00700 1 -- 172.21.15.6:0/1248279200 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064801ed10 con 0x5634b2f82660 2024-04-26T02:04:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.888+0000 7f063be00700 1 -- 172.21.15.6:0/1248279200 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f0648026e00 con 0x5634b2f82660 2024-04-26T02:04:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.888+0000 7f066b7fbb80 1 -- 172.21.15.6:0/1248279200 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0648026c50 con 0x5634b2f82660 2024-04-26T02:04:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.889+0000 7f063be00700 1 -- 172.21.15.6:0/1248279200 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f06480050d0 con 0x5634b2f82660 2024-04-26T02:04:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.889+0000 7f066b7fbb80 10 client.27400 fetch_fsmap learned FSMap version 55 2024-04-26T02:04:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.889+0000 7f066b7fbb80 1 -- 172.21.15.6:0/1248279200 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5634b2e6bbf0 con 0x5634b2f82660 2024-04-26T02:04:42.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.889+0000 7f063be00700 1 -- 172.21.15.6:0/1248279200 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0648026c50 con 0x5634b2f82660 2024-04-26T02:04:42.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.889+0000 7f063be00700 1 -- 172.21.15.6:0/1248279200 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7f06480036a0 con 0x5634b2f82660 2024-04-26T02:04:42.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.890+0000 7f066b7fbb80 10 client.27400 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:04:42.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.890+0000 7f066b7fbb80 10 client.27400 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:04:42.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.890+0000 7f066b7fbb80 20 client.27400 populate_metadata read hostname 'smithi006' 2024-04-26T02:04:42.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.890+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634b2f9cbf0 0x5634b2f9efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.890+0000 7f066b7fbb80 4 client.27400 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:04:42.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.890+0000 7f066b7fbb80 1 -- 172.21.15.6:0/1248279200 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634b2ee2780 con 0x5634b2f9cbf0 2024-04-26T02:04:42.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.890+0000 7f0659e00700 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634b2f9cbf0 0x5634b2f9efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.891+0000 7f0659e00700 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634b2f9cbf0 0x5634b2f9efe0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=916f8f5d209e689b server_cookie=7f100d5af17b37e2 in_seq=0 out_seq=0 2024-04-26T02:04:42.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.891+0000 7f063be00700 10 client.27400 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:42.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.893+0000 7f063be00700 1 -- 172.21.15.6:0/1248279200 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5634b2ee2780 con 0x5634b2f9cbf0 2024-04-26T02:04:42.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.893+0000 7f063be00700 10 client.27400 handle_command_reply: tid=0 2024-04-26T02:04:42.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.893+0000 7f066b7fbb80 1 client.27400 shutdown 2024-04-26T02:04:42.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.893+0000 7f066b7fbb80 1 -- 172.21.15.6:0/1248279200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 msgr2=0x5634b2f89ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:42.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.893+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 0x5634b2f89ea0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f06480056c0 tx=0x7f06480056f0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.893+0000 7f066b7fbb80 1 -- 172.21.15.6:0/1248279200 shutdown_connections 2024-04-26T02:04:42.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.893+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f8d4d0 0x5634b2f8a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.893+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f81d40 0x5634b2f8cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.893+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634b2f9cbf0 0x5634b2f9efe0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.893+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/1248279200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f82660 0x5634b2f89ea0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.893+0000 7f063be00700 0 client.27400 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:42.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.893+0000 7f066b7fbb80 1 -- 172.21.15.6:0/1248279200 >> 172.21.15.6:0/1248279200 conn(0x5634b2eb1050 msgr2=0x5634b2f7de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:42.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.894+0000 7f066b7fbb80 1 -- 172.21.15.6:0/1248279200 shutdown_connections 2024-04-26T02:04:42.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.894+0000 7f066b7fbb80 1 -- 172.21.15.6:0/1248279200 wait complete. 2024-04-26T02:04:42.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.894+0000 7f066b7fbb80 20 client.27400 trim_cache size 0 max 16384 2024-04-26T02:04:42.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.925+0000 7f066b7fbb80 1 Processor -- start 2024-04-26T02:04:42.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.925+0000 7f066b7fbb80 1 -- start start 2024-04-26T02:04:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.925+0000 7f066b7fbb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f81d40 0x5634b2f7e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.925+0000 7f066b7fbb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f82660 0x5634b2f98bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.925+0000 7f066b7fbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f8d4d0 0x5634b2f99100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.925+0000 7f066b7fbb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f063c00fff0 con 0x5634b2f82660 2024-04-26T02:04:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.925+0000 7f066b7fbb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f063c00d820 con 0x5634b2f81d40 2024-04-26T02:04:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.925+0000 7f066b7fbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f063c00b690 con 0x5634b2f8d4d0 2024-04-26T02:04:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.925+0000 7f0659e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f81d40 0x5634b2f7e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f0659e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f81d40 0x5634b2f7e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36000/0 (socket says 172.21.15.6:36000) 2024-04-26T02:04:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f0659e00700 1 -- 172.21.15.6:0/3946083221 learned_addr learned my addr 172.21.15.6:0/3946083221 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f0659400700 1 --2- 172.21.15.6:0/3946083221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f82660 0x5634b2f98bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f065a800700 1 --2- 172.21.15.6:0/3946083221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f8d4d0 0x5634b2f99100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f0659e00700 1 -- 172.21.15.6:0/3946083221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f8d4d0 msgr2=0x5634b2f99100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:42.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f0659e00700 1 --2- 172.21.15.6:0/3946083221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f8d4d0 0x5634b2f99100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f0659e00700 1 -- 172.21.15.6:0/3946083221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f82660 msgr2=0x5634b2f98bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:42.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f0659e00700 1 --2- 172.21.15.6:0/3946083221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f82660 0x5634b2f98bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f0659e00700 1 -- 172.21.15.6:0/3946083221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634b2f82ce0 con 0x5634b2f81d40 2024-04-26T02:04:42.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f0659400700 1 --2- 172.21.15.6:0/3946083221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f82660 0x5634b2f98bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:42.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f0659e00700 1 --2- 172.21.15.6:0/3946083221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f81d40 0x5634b2f7e410 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f064c099260 tx=0x7f064c010cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7c8577f44b2a520 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:42.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f0646a00700 1 -- 172.21.15.6:0/3946083221 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c0a5500 con 0x5634b2f81d40 2024-04-26T02:04:42.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.926+0000 7f0646a00700 1 -- 172.21.15.6:0/3946083221 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f064c09ab90 con 0x5634b2f81d40 2024-04-26T02:04:42.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.927+0000 7f0646a00700 1 -- 172.21.15.6:0/3946083221 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c0a52d0 con 0x5634b2f81d40 2024-04-26T02:04:42.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.927+0000 7f066b7fbb80 1 -- 172.21.15.6:0/3946083221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f81d40 msgr2=0x5634b2f7e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:42.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.927+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/3946083221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f81d40 0x5634b2f7e410 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f064c099260 tx=0x7f064c010cb0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.928+0000 7f066b7fbb80 1 -- 172.21.15.6:0/3946083221 shutdown_connections 2024-04-26T02:04:42.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.928+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/3946083221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f8d4d0 0x5634b2f99100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.928+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/3946083221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f81d40 0x5634b2f7e410 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.928+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/3946083221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f82660 0x5634b2f98bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.928+0000 7f066b7fbb80 1 -- 172.21.15.6:0/3946083221 >> 172.21.15.6:0/3946083221 conn(0x5634b2eb1050 msgr2=0x5634b2f984b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:42.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.928+0000 7f066b7fbb80 1 -- 172.21.15.6:0/3946083221 shutdown_connections 2024-04-26T02:04:42.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.928+0000 7f066b7fbb80 1 -- 172.21.15.6:0/3946083221 wait complete. 2024-04-26T02:04:42.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.929+0000 7f066b7fbb80 1 -- start start 2024-04-26T02:04:42.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.929+0000 7f066b7fbb80 1 Processor -- start 2024-04-26T02:04:42.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.930+0000 7f066b7fbb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f81d40 0x5634b2fa1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.930+0000 7f066b7fbb80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f82660 0x5634b2f9e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.930+0000 7f066b7fbb80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f8d4d0 0x5634b2f9e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.930+0000 7f066b7fbb80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f063c00bce0 con 0x5634b2f8d4d0 2024-04-26T02:04:42.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.930+0000 7f066b7fbb80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f063c00d150 con 0x5634b2f82660 2024-04-26T02:04:42.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.930+0000 7f066b7fbb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f063c00c3b0 con 0x5634b2f81d40 2024-04-26T02:04:42.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.930+0000 7f066b7fbb80 10 client.0 _ll_register_callbacks cb 0x5634b2e6d820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:42.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.930+0000 7f0659e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f81d40 0x5634b2fa1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.930+0000 7f0659e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f81d40 0x5634b2fa1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:33992/0 (socket says 172.21.15.6:33992) 2024-04-26T02:04:42.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.930+0000 7f0659e00700 1 -- 172.21.15.6:0/693384369 learned_addr learned my addr 172.21.15.6:0/693384369 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:42.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.930+0000 7f065a800700 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f8d4d0 0x5634b2f9e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.930+0000 7f0659400700 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f82660 0x5634b2f9e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.931+0000 7f0659e00700 1 -- 172.21.15.6:0/693384369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f82660 msgr2=0x5634b2f9e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:42.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.931+0000 7f0659e00700 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f82660 0x5634b2f9e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.931+0000 7f0659e00700 1 -- 172.21.15.6:0/693384369 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f8d4d0 msgr2=0x5634b2f9e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:42.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.931+0000 7f0659e00700 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f8d4d0 0x5634b2f9e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.931+0000 7f0659e00700 1 -- 172.21.15.6:0/693384369 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634b2f9eec0 con 0x5634b2f81d40 2024-04-26T02:04:42.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.931+0000 7f0659e00700 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f81d40 0x5634b2fa1ee0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f064c09c590 tx=0x7f064c0a0d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c8e8fc954ca5dc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:42.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.931+0000 7f063be00700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:04:42.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.931+0000 7f063be00700 1 -- 172.21.15.6:0/693384369 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c09ccf0 con 0x5634b2f81d40 2024-04-26T02:04:42.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.931+0000 7f066b7fbb80 1 -- 172.21.15.6:0/693384369 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f064c09ab90 con 0x5634b2f81d40 2024-04-26T02:04:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.931+0000 7f063be00700 1 -- 172.21.15.6:0/693384369 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f064c0a5070 con 0x5634b2f81d40 2024-04-26T02:04:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.932+0000 7f063be00700 1 -- 172.21.15.6:0/693384369 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f064c080050 con 0x5634b2f81d40 2024-04-26T02:04:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.932+0000 7f066b7fbb80 10 client.27406 fetch_fsmap learned FSMap version 55 2024-04-26T02:04:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.932+0000 7f066b7fbb80 1 -- 172.21.15.6:0/693384369 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5634b300c580 con 0x5634b2f81d40 2024-04-26T02:04:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.932+0000 7f063be00700 1 -- 172.21.15.6:0/693384369 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f064c09ab90 con 0x5634b2f81d40 2024-04-26T02:04:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.932+0000 7f063be00700 1 -- 172.21.15.6:0/693384369 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7f064c0985e0 con 0x5634b2f81d40 2024-04-26T02:04:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.932+0000 7f066b7fbb80 10 client.27406 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:04:42.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.932+0000 7f066b7fbb80 10 client.27406 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:04:42.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.932+0000 7f066b7fbb80 20 client.27406 populate_metadata read hostname 'smithi006' 2024-04-26T02:04:42.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.933+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634b300cb90 0x5634b300efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:42.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.933+0000 7f066b7fbb80 4 client.27406 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ef592526", "await_for": 42.0}] 2024-04-26T02:04:42.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.933+0000 7f066b7fbb80 1 -- 172.21.15.6:0/693384369 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ef592526", "await_for": 42.0}) -- 0x5634b2edc820 con 0x5634b300cb90 2024-04-26T02:04:42.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.933+0000 7f0659400700 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634b300cb90 0x5634b300efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:42.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.934+0000 7f0659400700 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634b300cb90 0x5634b300efb0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b761ad58fdf37005 server_cookie=7b6f04ccfd8e111d in_seq=0 out_seq=0 2024-04-26T02:04:42.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.934+0000 7f063be00700 10 client.27406 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:42.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.935+0000 7f063be00700 1 -- 172.21.15.6:0/693384369 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: -110 ) ==== 8+0+592 (crc 0 0 0) 0x5634b2edc820 con 0x5634b300cb90 2024-04-26T02:04:42.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.935+0000 7f063be00700 10 client.27406 handle_command_reply: tid=0 2024-04-26T02:04:42.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.935+0000 7f066b7fbb80 1 client.27406 shutdown 2024-04-26T02:04:42.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.935+0000 7f066b7fbb80 1 -- 172.21.15.6:0/693384369 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f81d40 msgr2=0x5634b2fa1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:42.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.935+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f81d40 0x5634b2fa1ee0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f064c09c590 tx=0x7f064c0a0d70 comp rx=0 tx=0).stop 2024-04-26T02:04:42.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.936+0000 7f066b7fbb80 1 -- 172.21.15.6:0/693384369 shutdown_connections 2024-04-26T02:04:42.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.936+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5634b2f8d4d0 0x5634b2f9e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.936+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5634b300cb90 0x5634b300efb0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.936+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5634b2f81d40 0x5634b2fa1ee0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.936+0000 7f066b7fbb80 1 --2- 172.21.15.6:0/693384369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5634b2f82660 0x5634b2f9e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:42.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.936+0000 7f066b7fbb80 1 -- 172.21.15.6:0/693384369 >> 172.21.15.6:0/693384369 conn(0x5634b2eb1050 msgr2=0x5634b2f98d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:42.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.936+0000 7f063be00700 0 client.27406 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:42.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.937+0000 7f066b7fbb80 1 -- 172.21.15.6:0/693384369 shutdown_connections 2024-04-26T02:04:42.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.937+0000 7f066b7fbb80 1 -- 172.21.15.6:0/693384369 wait complete. 2024-04-26T02:04:42.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:04:42.937+0000 7f066b7fbb80 20 client.27406 trim_cache size 0 max 16384 2024-04-26T02:04:42.936 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:04:42.936 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 55, 2024-04-26T02:04:42.936 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T02:04:42.936 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 54, 2024-04-26T02:04:42.936 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": { 2024-04-26T02:04:42.936 INFO:teuthology.orchestra.run.smithi006.stdout: "ef592526": { 2024-04-26T02:04:42.936 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 54, 2024-04-26T02:04:42.936 INFO:teuthology.orchestra.run.smithi006.stdout: "age_ref": 9.5, 2024-04-26T02:04:42.936 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:04:42.936 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "TIMEDOUT", 2024-04-26T02:04:42.936 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: "timeout": 60.0, 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: "expiration": 82.5, 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: "members": { 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: "file:/": { 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: "excluded": false, 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: "state": { 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "QUIESCING", 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 60.0 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:04:42.937 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:04:42.938 INFO:teuthology.orchestra.run.smithi006.stderr:Error ETIMEDOUT: 2024-04-26T02:04:43.128 INFO:tasks.workunit.client.0.smithi006.stdout: 10 88143 16.18 MB/sec execute 186 sec latency 126.372 ms 2024-04-26T02:04:43.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:42 smithi188 ceph-mon[32751]: pgmap v597: 97 pgs: 97 active+clean; 889 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4.2 MiB/s wr, 322 op/s 2024-04-26T02:04:43.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:42 smithi029 ceph-mon[32775]: pgmap v597: 97 pgs: 97 active+clean; 889 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4.2 MiB/s wr, 322 op/s 2024-04-26T02:04:43.785 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-26T02:04:43.785 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 110 (ETIMEDOUT), stdout: { "epoch": 55, "leader": 24523, "set_version": 54, "sets": { "ef592526": { "version": 54, "age_ref": 9.5, "state": { "name": "TIMEDOUT", "age": 0.0 }, "timeout": 60.0, "expiration": 82.5, "members": { "file:/": { "excluded": false, "state": { "name": "QUIESCING", "age": 60.0 } } } } } } 2024-04-26T02:04:43.786 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/quiescer.py", line 354, in _run self.do_quiesce(d) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/quiescer.py", line 310, in do_quiesce raise RuntimeError(f"Error quiescing set '{set_id}': {rcinfo(rc)}") RuntimeError: Error quiescing set 'ef592526': 110 (ETIMEDOUT) 2024-04-26T02:04:44.128 INFO:tasks.workunit.client.0.smithi006.stdout: 10 88776 16.29 MB/sec execute 187 sec latency 74.534 ms 2024-04-26T02:04:45.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:44 smithi006 ceph-mon[28419]: pgmap v598: 97 pgs: 97 active+clean; 889 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4.2 MiB/s wr, 319 op/s 2024-04-26T02:04:45.128 INFO:tasks.workunit.client.0.smithi006.stdout: 10 89303 16.38 MB/sec execute 188 sec latency 61.239 ms 2024-04-26T02:04:45.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:44 smithi188 ceph-mon[32751]: pgmap v598: 97 pgs: 97 active+clean; 889 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4.2 MiB/s wr, 319 op/s 2024-04-26T02:04:45.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:44 smithi029 ceph-mon[32775]: pgmap v598: 97 pgs: 97 active+clean; 889 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4.2 MiB/s wr, 319 op/s 2024-04-26T02:04:46.128 INFO:tasks.workunit.client.0.smithi006.stdout: 10 89782 16.41 MB/sec execute 189 sec latency 138.861 ms 2024-04-26T02:04:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.842+0000 7f60ae600640 1 -- 172.21.15.6:0/919115510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60a8109490 msgr2=0x7f60a8111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.842+0000 7f60ae600640 1 --2- 172.21.15.6:0/919115510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60a8109490 0x7f60a8111060 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009ca0 tx=0x7f60a002a8c0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.842+0000 7f60ae600640 1 -- 172.21.15.6:0/919115510 shutdown_connections 2024-04-26T02:04:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.842+0000 7f60ae600640 1 --2- 172.21.15.6:0/919115510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60a8104d70 0x7f60a8108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.842+0000 7f60ae600640 1 --2- 172.21.15.6:0/919115510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60a8109490 0x7f60a8111060 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.842+0000 7f60ae600640 1 --2- 172.21.15.6:0/919115510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a81043c0 0x7f60a81047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.842+0000 7f60ae600640 1 -- 172.21.15.6:0/919115510 >> 172.21.15.6:0/919115510 conn(0x7f60a80ffc20 msgr2=0x7f60a8102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.843+0000 7f60ae600640 1 -- 172.21.15.6:0/919115510 shutdown_connections 2024-04-26T02:04:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.843+0000 7f60ae600640 1 -- 172.21.15.6:0/919115510 wait complete. 2024-04-26T02:04:46.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.845+0000 7f60ae600640 1 Processor -- start 2024-04-26T02:04:46.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.845+0000 7f60ae600640 1 -- start start 2024-04-26T02:04:46.843 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.845+0000 7f60ae600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60a81043c0 0x7f60a8110eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.845+0000 7f60ae600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60a8109490 0x7f60a8073a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.846+0000 7f60ae600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a810dfa0 0x7f60a810e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.846+0000 7f60ae600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f60a806c270 con 0x7f60a81043c0 2024-04-26T02:04:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.846+0000 7f60ae600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f60a806c0f0 con 0x7f60a810dfa0 2024-04-26T02:04:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.846+0000 7f60ae600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f60a806c3f0 con 0x7f60a8109490 2024-04-26T02:04:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.846+0000 7f60a6a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60a8109490 0x7f60a8073a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.846+0000 7f60a6a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60a8109490 0x7f60a8073a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34002/0 (socket says 172.21.15.6:34002) 2024-04-26T02:04:46.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.846+0000 7f60a6a00640 1 -- 172.21.15.6:0/622718026 learned_addr learned my addr 172.21.15.6:0/622718026 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:46.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.846+0000 7f60a7400640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60a81043c0 0x7f60a8110eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.845 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.846+0000 7f60a7e00640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a810dfa0 0x7f60a810e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.847+0000 7f60a6a00640 1 -- 172.21.15.6:0/622718026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a810dfa0 msgr2=0x7f60a810e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.847+0000 7f60a6a00640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a810dfa0 0x7f60a810e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.847+0000 7f60a6a00640 1 -- 172.21.15.6:0/622718026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60a81043c0 msgr2=0x7f60a8110eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.847+0000 7f60a6a00640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60a81043c0 0x7f60a8110eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.847+0000 7f60a6a00640 1 -- 172.21.15.6:0/622718026 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60a810eb80 con 0x7f60a8109490 2024-04-26T02:04:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.847+0000 7f60a7400640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60a81043c0 0x7f60a8110eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.847+0000 7f60a6a00640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60a8109490 0x7f60a8073a80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f609800c970 tx=0x7f609800ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.848+0000 7f609fe00640 1 -- 172.21.15.6:0/622718026 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6098014070 con 0x7f60a8109490 2024-04-26T02:04:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.848+0000 7f609fe00640 1 -- 172.21.15.6:0/622718026 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6098004440 con 0x7f60a8109490 2024-04-26T02:04:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.848+0000 7f609fe00640 1 -- 172.21.15.6:0/622718026 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6098003e20 con 0x7f60a8109490 2024-04-26T02:04:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.848+0000 7f60ae600640 1 -- 172.21.15.6:0/622718026 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60a81b4070 con 0x7f60a8109490 2024-04-26T02:04:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.848+0000 7f60ae600640 1 -- 172.21.15.6:0/622718026 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60a81b4420 con 0x7f60a8109490 2024-04-26T02:04:46.850 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.852+0000 7f60b1409740 1 Processor -- start 2024-04-26T02:04:46.850 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.852+0000 7f60b1409740 1 -- start start 2024-04-26T02:04:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.852+0000 7f60b1409740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ff9dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.852+0000 7f60b1409740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9e140 0x55640ffa84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.853+0000 7f60b1409740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffaadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.853+0000 7f60b1409740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55640ff03900 con 0x55640ff9e140 2024-04-26T02:04:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.853+0000 7f60b1409740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55640fea9ef0 con 0x55640ff9d820 2024-04-26T02:04:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.853+0000 7f60b1409740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55640feff8c0 con 0x55640ffa8a00 2024-04-26T02:04:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.853+0000 7f60a7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ff9dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.853+0000 7f60a7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ff9dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36040/0 (socket says 172.21.15.6:36040) 2024-04-26T02:04:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.853+0000 7f60a7e00640 1 -- 172.21.15.6:0/4161639921 learned_addr learned my addr 172.21.15.6:0/4161639921 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.853+0000 7f60a7400640 1 --2- 172.21.15.6:0/4161639921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9e140 0x55640ffa84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.853+0000 7f609fe00640 1 -- 172.21.15.6:0/622718026 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f6098017070 con 0x7f60a8109490 2024-04-26T02:04:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.853+0000 7f60a6a00640 1 --2- 172.21.15.6:0/4161639921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffaadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.854+0000 7f60a7e00640 1 -- 172.21.15.6:0/4161639921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 msgr2=0x55640ffaadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.854+0000 7f60a7e00640 1 --2- 172.21.15.6:0/4161639921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffaadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.854+0000 7f60a7e00640 1 -- 172.21.15.6:0/4161639921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9e140 msgr2=0x55640ffa84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.854+0000 7f60a7e00640 1 --2- 172.21.15.6:0/4161639921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9e140 0x55640ffa84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.853 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.854+0000 7f60a7e00640 1 -- 172.21.15.6:0/4161639921 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55640fc16210 con 0x55640ff9d820 2024-04-26T02:04:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.854+0000 7f60a6a00640 1 --2- 172.21.15.6:0/4161639921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffaadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.854+0000 7f60a7e00640 1 --2- 172.21.15.6:0/4161639921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ff9dc00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009430 tx=0x7f60a0009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df3143d939f95ecd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.854+0000 7f609cc00640 1 -- 172.21.15.6:0/4161639921 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0031cf0 con 0x55640ff9d820 2024-04-26T02:04:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.854+0000 7f609cc00640 1 -- 172.21.15.6:0/4161639921 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60a0036040 con 0x55640ff9d820 2024-04-26T02:04:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.854+0000 7f609cc00640 1 -- 172.21.15.6:0/4161639921 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0004330 con 0x55640ff9d820 2024-04-26T02:04:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.855+0000 7f609fe00640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f608c0a3d90 0x7f608c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.854 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.855+0000 7f609fe00640 1 -- 172.21.15.6:0/622718026 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f60980d03b0 con 0x7f60a8109490 2024-04-26T02:04:46.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.855+0000 7f60a7400640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f608c0a3d90 0x7f608c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.855+0000 7f60b1409740 1 -- 172.21.15.6:0/4161639921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 msgr2=0x55640ff9dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.855+0000 7f60b1409740 1 --2- 172.21.15.6:0/4161639921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ff9dc00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009430 tx=0x7f60a0009a20 comp rx=0 tx=0).stop 2024-04-26T02:04:46.855 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.855+0000 7f60b1409740 1 -- 172.21.15.6:0/4161639921 shutdown_connections 2024-04-26T02:04:46.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.855+0000 7f60b1409740 1 --2- 172.21.15.6:0/4161639921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9e140 0x55640ffa84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.855+0000 7f60b1409740 1 --2- 172.21.15.6:0/4161639921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffaadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.855+0000 7f60b1409740 1 --2- 172.21.15.6:0/4161639921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ff9dc00 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.855+0000 7f60b1409740 1 -- 172.21.15.6:0/4161639921 >> 172.21.15.6:0/4161639921 conn(0x55640ff99d30 msgr2=0x55640ff9a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:46.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.856+0000 7f60b1409740 1 -- 172.21.15.6:0/4161639921 shutdown_connections 2024-04-26T02:04:46.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.856+0000 7f60b1409740 1 -- 172.21.15.6:0/4161639921 wait complete. 2024-04-26T02:04:46.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.856+0000 7f60a7400640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f608c0a3d90 0x7f608c0a6250 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x55640ff9edc0 tx=0x7f6094014040 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:46.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.856+0000 7f60b1409740 1 -- start start 2024-04-26T02:04:46.856 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.857+0000 7f60b1409740 1 Processor -- start 2024-04-26T02:04:46.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.858+0000 7f60b1409740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ffa5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.858+0000 7f60b1409740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9e140 0x55640ffa5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.858+0000 7f60b1409740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffa63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.858+0000 7f60b1409740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55640ffab5d0 con 0x55640ff9e140 2024-04-26T02:04:46.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.858+0000 7f60b1409740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55640febd310 con 0x55640ff9d820 2024-04-26T02:04:46.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.858+0000 7f60b1409740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55640ffab750 con 0x55640ffa8a00 2024-04-26T02:04:46.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.858+0000 7f60b1409740 10 client.0 _ll_register_callbacks cb 0x55640fe8bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:46.857 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.858+0000 7f60a7400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ffa5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.858+0000 7f60a7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffa63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.858+0000 7f60a7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffa63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34032/0 (socket says 172.21.15.6:34032) 2024-04-26T02:04:46.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.858+0000 7f60a7e00640 1 -- 172.21.15.6:0/70518633 learned_addr learned my addr 172.21.15.6:0/70518633 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:46.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.858+0000 7f60a6a00640 1 --2- 172.21.15.6:0/70518633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9e140 0x55640ffa5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.859+0000 7f60a7400640 1 -- 172.21.15.6:0/70518633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 msgr2=0x55640ffa63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:46.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.859+0000 7f60a7400640 1 --2- 172.21.15.6:0/70518633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffa63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.859+0000 7f60a7400640 1 -- 172.21.15.6:0/70518633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9e140 msgr2=0x55640ffa5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:46.858 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.859+0000 7f60a7400640 1 --2- 172.21.15.6:0/70518633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9e140 0x55640ffa5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.859 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.859+0000 7f60a7400640 1 -- 172.21.15.6:0/70518633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55640fce2620 con 0x55640ff9d820 2024-04-26T02:04:46.859 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.860+0000 7f60a7400640 1 --2- 172.21.15.6:0/70518633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ffa5960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6094012cb0 tx=0x7f6094004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=167d044079542c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:46.859 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.860+0000 7f6091600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:04:46.859 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.860+0000 7f6091600640 1 -- 172.21.15.6:0/70518633 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609401ca80 con 0x55640ff9d820 2024-04-26T02:04:46.859 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.860+0000 7f6091600640 1 -- 172.21.15.6:0/70518633 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f609401cc20 con 0x55640ff9d820 2024-04-26T02:04:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.860+0000 7f60b1409740 1 -- 172.21.15.6:0/70518633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60a0036040 con 0x55640ff9d820 2024-04-26T02:04:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.861+0000 7f6091600640 1 -- 172.21.15.6:0/70518633 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f609401ce60 con 0x55640ff9d820 2024-04-26T02:04:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.861+0000 7f60b1409740 10 client.27359 fetch_fsmap learned FSMap version 55 2024-04-26T02:04:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.861+0000 7f60b1409740 1 -- 172.21.15.6:0/70518633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55640ffb7850 con 0x55640ff9d820 2024-04-26T02:04:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.861+0000 7f6091600640 1 -- 172.21.15.6:0/70518633 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60a0036040 con 0x55640ff9d820 2024-04-26T02:04:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.861+0000 7f6091600640 1 -- 172.21.15.6:0/70518633 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7f6094003c40 con 0x55640ff9d820 2024-04-26T02:04:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.861+0000 7f60b1409740 10 client.27359 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:04:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.861+0000 7f60b1409740 10 client.27359 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:04:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.862+0000 7f60b1409740 20 client.27359 populate_metadata read hostname 'smithi006' 2024-04-26T02:04:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.862+0000 7f60b1409740 1 --2- 172.21.15.6:0/70518633 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55640ffb8d10 0x55640ffbb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.862+0000 7f60b1409740 4 client.27359 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:04:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.862+0000 7f60b1409740 1 -- 172.21.15.6:0/70518633 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55640ff178b0 con 0x55640ffb8d10 2024-04-26T02:04:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.863+0000 7f60a6a00640 1 --2- 172.21.15.6:0/70518633 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55640ffb8d10 0x55640ffbb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.864+0000 7f6091600640 10 client.27359 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.865+0000 7f60a6a00640 1 --2- 172.21.15.6:0/70518633 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55640ffb8d10 0x55640ffbb100 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50be12181a34dc5b server_cookie=d5ef60e055c84e54 in_seq=0 out_seq=0 2024-04-26T02:04:46.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.866+0000 7f6091600640 1 -- 172.21.15.6:0/70518633 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55640ff178b0 con 0x55640ffb8d10 2024-04-26T02:04:46.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.866+0000 7f6091600640 10 client.27359 handle_command_reply: tid=0 2024-04-26T02:04:46.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.867+0000 7f60b1409740 1 client.27359 shutdown 2024-04-26T02:04:46.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.867+0000 7f60b1409740 1 -- 172.21.15.6:0/70518633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 msgr2=0x55640ffa5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.865 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.867+0000 7f60b1409740 1 --2- 172.21.15.6:0/70518633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ffa5960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6094012cb0 tx=0x7f6094004730 comp rx=0 tx=0).stop 2024-04-26T02:04:46.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.867+0000 7f60b1409740 1 -- 172.21.15.6:0/70518633 shutdown_connections 2024-04-26T02:04:46.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.867+0000 7f60b1409740 1 --2- 172.21.15.6:0/70518633 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55640ffb8d10 0x55640ffbb100 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.867+0000 7f60b1409740 1 --2- 172.21.15.6:0/70518633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9e140 0x55640ffa5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.867+0000 7f60b1409740 1 --2- 172.21.15.6:0/70518633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffa63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.867+0000 7f60b1409740 1 --2- 172.21.15.6:0/70518633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ffa5960 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.867+0000 7f60b1409740 1 -- 172.21.15.6:0/70518633 >> 172.21.15.6:0/70518633 conn(0x55640ff99d30 msgr2=0x55640ff9fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:46.866 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.868+0000 7f6091600640 0 client.27359 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:46.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.868+0000 7f60b1409740 1 -- 172.21.15.6:0/70518633 shutdown_connections 2024-04-26T02:04:46.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.868+0000 7f60b1409740 1 -- 172.21.15.6:0/70518633 wait complete. 2024-04-26T02:04:46.867 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.869+0000 7f60b1409740 20 client.27359 trim_cache size 0 max 16384 2024-04-26T02:04:46.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.904+0000 7f60b1409740 1 Processor -- start 2024-04-26T02:04:46.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.905+0000 7f60b1409740 1 -- start start 2024-04-26T02:04:46.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.905+0000 7f60b1409740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ffb4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.905+0000 7f60b1409740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ff9e140 0x55640ffb4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.905+0000 7f60b1409740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ffa8a00 0x55640ffa0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.905+0000 7f60b1409740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f60a81bf520 con 0x55640ffa8a00 2024-04-26T02:04:46.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.905+0000 7f60b1409740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f60a81bcd90 con 0x55640ff9d820 2024-04-26T02:04:46.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.905+0000 7f60b1409740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f60a81bac00 con 0x55640ff9e140 2024-04-26T02:04:46.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.905+0000 7f60a6a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ff9e140 0x55640ffb4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.905+0000 7f60a7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ffa8a00 0x55640ffa0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.905+0000 7f60a6a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ff9e140 0x55640ffb4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34038/0 (socket says 172.21.15.6:34038) 2024-04-26T02:04:46.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.905+0000 7f60a6a00640 1 -- 172.21.15.6:0/1306250749 learned_addr learned my addr 172.21.15.6:0/1306250749 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:46.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.905+0000 7f60a7400640 1 --2- 172.21.15.6:0/1306250749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ffb4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.906+0000 7f60a7400640 1 -- 172.21.15.6:0/1306250749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ff9e140 msgr2=0x55640ffb4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:46.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.906+0000 7f60a7400640 1 --2- 172.21.15.6:0/1306250749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ff9e140 0x55640ffb4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.906+0000 7f60a7400640 1 -- 172.21.15.6:0/1306250749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ffa8a00 msgr2=0x55640ffa0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:46.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.906+0000 7f60a7400640 1 --2- 172.21.15.6:0/1306250749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ffa8a00 0x55640ffa0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.906+0000 7f60a7400640 1 -- 172.21.15.6:0/1306250749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55640ffa4910 con 0x55640ff9d820 2024-04-26T02:04:46.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.906+0000 7f60a7400640 1 --2- 172.21.15.6:0/1306250749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ffb4f10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6094009b30 tx=0x7f6094025890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc8dd034d28cc10b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.906+0000 7f6093e00640 1 -- 172.21.15.6:0/1306250749 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6094033040 con 0x55640ff9d820 2024-04-26T02:04:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.907+0000 7f6093e00640 1 -- 172.21.15.6:0/1306250749 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f609401ca80 con 0x55640ff9d820 2024-04-26T02:04:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.907+0000 7f6093e00640 1 -- 172.21.15.6:0/1306250749 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6094004350 con 0x55640ff9d820 2024-04-26T02:04:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.908+0000 7f60b1409740 1 -- 172.21.15.6:0/1306250749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 msgr2=0x55640ffb4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.908+0000 7f60b1409740 1 --2- 172.21.15.6:0/1306250749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ffb4f10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6094009b30 tx=0x7f6094025890 comp rx=0 tx=0).stop 2024-04-26T02:04:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.908+0000 7f60b1409740 1 -- 172.21.15.6:0/1306250749 shutdown_connections 2024-04-26T02:04:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.908+0000 7f60b1409740 1 --2- 172.21.15.6:0/1306250749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ffa8a00 0x55640ffa0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.908+0000 7f60b1409740 1 --2- 172.21.15.6:0/1306250749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ff9e140 0x55640ffb4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.908+0000 7f60b1409740 1 --2- 172.21.15.6:0/1306250749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9d820 0x55640ffb4f10 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.908+0000 7f60b1409740 1 -- 172.21.15.6:0/1306250749 >> 172.21.15.6:0/1306250749 conn(0x55640ff99d30 msgr2=0x55640ff9a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.908+0000 7f60b1409740 1 -- 172.21.15.6:0/1306250749 shutdown_connections 2024-04-26T02:04:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.908+0000 7f60b1409740 1 -- 172.21.15.6:0/1306250749 wait complete. 2024-04-26T02:04:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.909+0000 7f60b1409740 1 -- start start 2024-04-26T02:04:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.909+0000 7f60b1409740 1 Processor -- start 2024-04-26T02:04:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.912+0000 7f60b1409740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9d820 0x55640ffb0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.912+0000 7f60b1409740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9e140 0x55640ffae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.912+0000 7f60b1409740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffae740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.912+0000 7f60b1409740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f60a81bb250 con 0x55640ff9d820 2024-04-26T02:04:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.912+0000 7f60b1409740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f60a81bc6c0 con 0x55640ff9e140 2024-04-26T02:04:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.912+0000 7f60b1409740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f60a81bb920 con 0x55640ffa8a00 2024-04-26T02:04:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.912+0000 7f60b1409740 10 client.0 _ll_register_callbacks cb 0x7f60940118d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.912+0000 7f60a7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.912+0000 7f60a7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffae740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:34042/0 (socket says 172.21.15.6:34042) 2024-04-26T02:04:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.912+0000 7f60a7e00640 1 -- 172.21.15.6:0/3289980482 learned_addr learned my addr 172.21.15.6:0/3289980482 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:04:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.913+0000 7f60a7e00640 1 -- 172.21.15.6:0/3289980482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9e140 msgr2=0x55640ffae1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T02:04:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.913+0000 7f60a7e00640 1 --2- 172.21.15.6:0/3289980482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9e140 0x55640ffae1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.913+0000 7f60a7e00640 1 -- 172.21.15.6:0/3289980482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9d820 msgr2=0x55640ffb0e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:04:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.914+0000 7f60a7400640 1 --2- 172.21.15.6:0/3289980482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9d820 0x55640ffb0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.914+0000 7f60a7e00640 1 --2- 172.21.15.6:0/3289980482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9d820 0x55640ffb0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.914+0000 7f60a7e00640 1 -- 172.21.15.6:0/3289980482 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55640ffaef70 con 0x55640ffa8a00 2024-04-26T02:04:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.915+0000 7f60a7e00640 1 --2- 172.21.15.6:0/3289980482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffae740 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009ca0 tx=0x7f60a0009540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12229d04f45b46b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.915+0000 7f6091600640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:04:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.915+0000 7f6091600640 1 -- 172.21.15.6:0/3289980482 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0054070 con 0x55640ffa8a00 2024-04-26T02:04:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.915+0000 7f6091600640 1 -- 172.21.15.6:0/3289980482 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f60a0031af0 con 0x55640ffa8a00 2024-04-26T02:04:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.915+0000 7f60b1409740 1 -- 172.21.15.6:0/3289980482 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f609401ca80 con 0x55640ffa8a00 2024-04-26T02:04:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.916+0000 7f6091600640 1 -- 172.21.15.6:0/3289980482 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f60a0060050 con 0x55640ffa8a00 2024-04-26T02:04:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.916+0000 7f60b1409740 10 client.27424 fetch_fsmap learned FSMap version 55 2024-04-26T02:04:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.916+0000 7f60b1409740 1 -- 172.21.15.6:0/3289980482 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55641001f370 con 0x55640ffa8a00 2024-04-26T02:04:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.916+0000 7f6091600640 1 -- 172.21.15.6:0/3289980482 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f609401ca80 con 0x55640ffa8a00 2024-04-26T02:04:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.916+0000 7f6091600640 1 -- 172.21.15.6:0/3289980482 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7f60a002f890 con 0x55640ffa8a00 2024-04-26T02:04:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.917+0000 7f60b1409740 10 client.27424 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:04:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.917+0000 7f60b1409740 10 client.27424 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:04:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.917+0000 7f60b1409740 20 client.27424 populate_metadata read hostname 'smithi006' 2024-04-26T02:04:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.917+0000 7f60b1409740 1 --2- 172.21.15.6:0/3289980482 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55641001f650 0x556410021a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.917+0000 7f60b1409740 4 client.27424 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:04:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.917+0000 7f60b1409740 1 -- 172.21.15.6:0/3289980482 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55640ff11320 con 0x55641001f650 2024-04-26T02:04:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.917+0000 7f60a7400640 1 --2- 172.21.15.6:0/3289980482 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55641001f650 0x556410021a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.919+0000 7f60a7400640 1 --2- 172.21.15.6:0/3289980482 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55641001f650 0x556410021a40 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c3951e73d7ceb16 server_cookie=62b85a54df85076e in_seq=0 out_seq=0 2024-04-26T02:04:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:46.919+0000 7f6091600640 10 client.27424 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:47.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:46 smithi006 ceph-mon[28419]: pgmap v599: 97 pgs: 97 active+clean; 889 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4.2 MiB/s wr, 319 op/s 2024-04-26T02:04:47.128 INFO:tasks.workunit.client.0.smithi006.stdout: 10 90296 16.47 MB/sec execute 190 sec latency 210.657 ms 2024-04-26T02:04:47.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:46 smithi188 ceph-mon[32751]: pgmap v599: 97 pgs: 97 active+clean; 889 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4.2 MiB/s wr, 319 op/s 2024-04-26T02:04:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.167+0000 7f6091600640 1 -- 172.21.15.6:0/3289980482 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55640ff11320 con 0x55641001f650 2024-04-26T02:04:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.167+0000 7f6091600640 10 client.27424 handle_command_reply: tid=0 2024-04-26T02:04:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.167+0000 7f60b1409740 1 client.27424 shutdown 2024-04-26T02:04:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.167+0000 7f60b1409740 1 -- 172.21.15.6:0/3289980482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 msgr2=0x55640ffae740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.167+0000 7f60b1409740 1 --2- 172.21.15.6:0/3289980482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffae740 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009ca0 tx=0x7f60a0009540 comp rx=0 tx=0).stop 2024-04-26T02:04:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.167+0000 7f60b1409740 1 -- 172.21.15.6:0/3289980482 shutdown_connections 2024-04-26T02:04:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.167+0000 7f60b1409740 1 --2- 172.21.15.6:0/3289980482 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55641001f650 0x556410021a40 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.167+0000 7f60b1409740 1 --2- 172.21.15.6:0/3289980482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55640ffa8a00 0x55640ffae740 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.167+0000 7f60b1409740 1 --2- 172.21.15.6:0/3289980482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55640ff9e140 0x55640ffae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.167+0000 7f60b1409740 1 --2- 172.21.15.6:0/3289980482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55640ff9d820 0x55640ffb0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.167+0000 7f60b1409740 1 -- 172.21.15.6:0/3289980482 >> 172.21.15.6:0/3289980482 conn(0x55640ff99d30 msgr2=0x55640ff98e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:47.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.167+0000 7f6091600640 0 client.27424 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:04:47.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.168+0000 7f60b1409740 1 -- 172.21.15.6:0/3289980482 shutdown_connections 2024-04-26T02:04:47.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.169+0000 7f60b1409740 1 -- 172.21.15.6:0/3289980482 wait complete. 2024-04-26T02:04:47.167 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.169+0000 7f60b1409740 20 client.27424 trim_cache size 0 max 16384 2024-04-26T02:04:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:04:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:04:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:04:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:04:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.172+0000 7f60ae600640 1 -- 172.21.15.6:0/622718026 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f608c0a3d90 msgr2=0x7f608c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.172+0000 7f60ae600640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f608c0a3d90 0x7f608c0a6250 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x55640ff9edc0 tx=0x7f6094014040 comp rx=0 tx=0).stop 2024-04-26T02:04:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.172+0000 7f60ae600640 1 -- 172.21.15.6:0/622718026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60a8109490 msgr2=0x7f60a8073a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.172+0000 7f60ae600640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60a8109490 0x7f60a8073a80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f609800c970 tx=0x7f609800ce30 comp rx=0 tx=0).stop 2024-04-26T02:04:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.172+0000 7f60ae600640 1 -- 172.21.15.6:0/622718026 shutdown_connections 2024-04-26T02:04:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.172+0000 7f60ae600640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f608c0a3d90 0x7f608c0a6250 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.172+0000 7f60ae600640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a810dfa0 0x7f60a810e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.172+0000 7f60ae600640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60a8109490 0x7f60a8073a80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.172+0000 7f60ae600640 1 --2- 172.21.15.6:0/622718026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60a81043c0 0x7f60a8110eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.172+0000 7f60ae600640 1 -- 172.21.15.6:0/622718026 >> 172.21.15.6:0/622718026 conn(0x7f60a80ffc20 msgr2=0x7f60a81006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.172+0000 7f60ae600640 1 -- 172.21.15.6:0/622718026 shutdown_connections 2024-04-26T02:04:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:04:47.172+0000 7f60ae600640 1 -- 172.21.15.6:0/622718026 wait complete. 2024-04-26T02:04:47.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:46 smithi029 ceph-mon[32775]: pgmap v599: 97 pgs: 97 active+clean; 889 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4.2 MiB/s wr, 319 op/s 2024-04-26T02:04:48.128 INFO:tasks.workunit.client.0.smithi006.stdout: 10 90793 16.52 MB/sec execute 191 sec latency 147.012 ms 2024-04-26T02:04:49.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:48 smithi006 ceph-mon[28419]: pgmap v600: 97 pgs: 97 active+clean; 919 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 6.9 MiB/s wr, 486 op/s 2024-04-26T02:04:49.128 INFO:tasks.workunit.client.0.smithi006.stdout: 10 91261 16.54 MB/sec execute 192 sec latency 121.599 ms 2024-04-26T02:04:49.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:48 smithi188 ceph-mon[32751]: pgmap v600: 97 pgs: 97 active+clean; 919 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 6.9 MiB/s wr, 486 op/s 2024-04-26T02:04:49.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:48 smithi029 ceph-mon[32775]: pgmap v600: 97 pgs: 97 active+clean; 919 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 6.9 MiB/s wr, 486 op/s 2024-04-26T02:04:50.128 INFO:tasks.workunit.client.0.smithi006.stdout: 10 91835 16.63 MB/sec execute 193 sec latency 201.364 ms 2024-04-26T02:04:51.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:50 smithi006 ceph-mon[28419]: pgmap v601: 97 pgs: 97 active+clean; 919 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 5.5 MiB/s wr, 356 op/s 2024-04-26T02:04:51.128 INFO:tasks.workunit.client.0.smithi006.stdout: 10 92420 16.70 MB/sec execute 194 sec latency 139.771 ms 2024-04-26T02:04:51.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:50 smithi188 ceph-mon[32751]: pgmap v601: 97 pgs: 97 active+clean; 919 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 5.5 MiB/s wr, 356 op/s 2024-04-26T02:04:51.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:50 smithi029 ceph-mon[32775]: pgmap v601: 97 pgs: 97 active+clean; 919 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 5.5 MiB/s wr, 356 op/s 2024-04-26T02:04:52.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:51 smithi006 ceph-mon[28419]: pgmap v602: 97 pgs: 97 active+clean; 935 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.5 MiB/s wr, 623 op/s 2024-04-26T02:04:52.128 INFO:tasks.workunit.client.0.smithi006.stdout: 10 93085 16.82 MB/sec execute 195 sec latency 58.162 ms 2024-04-26T02:04:52.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:51 smithi188 ceph-mon[32751]: pgmap v602: 97 pgs: 97 active+clean; 935 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.5 MiB/s wr, 623 op/s 2024-04-26T02:04:52.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:51 smithi029 ceph-mon[32775]: pgmap v602: 97 pgs: 97 active+clean; 935 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 8.5 MiB/s wr, 623 op/s 2024-04-26T02:04:53.129 INFO:tasks.workunit.client.0.smithi006.stdout: 10 93631 16.91 MB/sec execute 196 sec latency 77.935 ms 2024-04-26T02:04:54.129 INFO:tasks.workunit.client.0.smithi006.stdout: 10 94116 16.92 MB/sec execute 197 sec latency 85.445 ms 2024-04-26T02:04:54.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:54 smithi006 ceph-mon[28419]: pgmap v603: 97 pgs: 97 active+clean; 935 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 5.7 MiB/s wr, 434 op/s 2024-04-26T02:04:54.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:54 smithi188 ceph-mon[32751]: pgmap v603: 97 pgs: 97 active+clean; 935 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 5.7 MiB/s wr, 434 op/s 2024-04-26T02:04:54.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:54 smithi029 ceph-mon[32775]: pgmap v603: 97 pgs: 97 active+clean; 935 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 5.7 MiB/s wr, 434 op/s 2024-04-26T02:04:55.129 INFO:tasks.workunit.client.0.smithi006.stdout: 10 94700 17.01 MB/sec execute 198 sec latency 74.283 ms 2024-04-26T02:04:56.129 INFO:tasks.workunit.client.0.smithi006.stdout: 10 95217 17.05 MB/sec execute 199 sec latency 193.260 ms 2024-04-26T02:04:56.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:56 smithi006 ceph-mon[28419]: pgmap v604: 97 pgs: 97 active+clean; 935 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 5.7 MiB/s wr, 434 op/s 2024-04-26T02:04:56.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:56 smithi188 ceph-mon[32751]: pgmap v604: 97 pgs: 97 active+clean; 935 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 5.7 MiB/s wr, 434 op/s 2024-04-26T02:04:56.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:56 smithi029 ceph-mon[32775]: pgmap v604: 97 pgs: 97 active+clean; 935 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 5.7 MiB/s wr, 434 op/s 2024-04-26T02:04:57.129 INFO:tasks.workunit.client.0.smithi006.stdout: 10 95774 17.09 MB/sec execute 200 sec latency 96.976 ms 2024-04-26T02:04:58.130 INFO:tasks.workunit.client.0.smithi006.stdout: 10 96414 17.20 MB/sec execute 201 sec latency 123.797 ms 2024-04-26T02:04:58.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:04:58 smithi006 ceph-mon[28419]: pgmap v605: 97 pgs: 97 active+clean; 963 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.3 MiB/s wr, 596 op/s 2024-04-26T02:04:58.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:04:58 smithi188 ceph-mon[32751]: pgmap v605: 97 pgs: 97 active+clean; 963 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.3 MiB/s wr, 596 op/s 2024-04-26T02:04:58.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:04:58 smithi029 ceph-mon[32775]: pgmap v605: 97 pgs: 97 active+clean; 963 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.3 MiB/s wr, 596 op/s 2024-04-26T02:04:59.131 INFO:tasks.workunit.client.0.smithi006.stdout: 10 96970 17.27 MB/sec execute 202 sec latency 76.482 ms 2024-04-26T02:05:00.131 INFO:tasks.workunit.client.0.smithi006.stdout: 10 97528 17.32 MB/sec execute 203 sec latency 77.993 ms 2024-04-26T02:05:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.363+0000 7fbbe3200640 1 -- 172.21.15.6:0/4015516561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbe41092c0 msgr2=0x7fbbe4110e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.363+0000 7fbbe3200640 1 --2- 172.21.15.6:0/4015516561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbe41092c0 0x7fbbe4110e90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009be0 tx=0x7fbbdc02a6c0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.365+0000 7fbbe3200640 1 -- 172.21.15.6:0/4015516561 shutdown_connections 2024-04-26T02:05:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.365+0000 7fbbe3200640 1 --2- 172.21.15.6:0/4015516561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbe41092c0 0x7fbbe4110e90 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.365+0000 7fbbe3200640 1 --2- 172.21.15.6:0/4015516561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbe4104ba0 0x7fbbe4108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.365+0000 7fbbe3200640 1 --2- 172.21.15.6:0/4015516561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbe41041f0 0x7fbbe41045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.365+0000 7fbbe3200640 1 -- 172.21.15.6:0/4015516561 >> 172.21.15.6:0/4015516561 conn(0x7fbbe40ffa50 msgr2=0x7fbbe4101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.366+0000 7fbbe3200640 1 -- 172.21.15.6:0/4015516561 shutdown_connections 2024-04-26T02:05:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.366+0000 7fbbe3200640 1 -- 172.21.15.6:0/4015516561 wait complete. 2024-04-26T02:05:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.367+0000 7fbbe3200640 1 Processor -- start 2024-04-26T02:05:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.367+0000 7fbbe3200640 1 -- start start 2024-04-26T02:05:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.368+0000 7fbbe3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbe41041f0 0x7fbbe41a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.368+0000 7fbbe3200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbe41092c0 0x7fbbe41a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.368+0000 7fbbe3200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbe419a590 0x7fbbe419aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.368+0000 7fbbe3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe4114210 con 0x7fbbe41041f0 2024-04-26T02:05:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.368+0000 7fbbe3200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe4114090 con 0x7fbbe41092c0 2024-04-26T02:05:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.368+0000 7fbbe3200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe4114390 con 0x7fbbe419a590 2024-04-26T02:05:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.368+0000 7fbbe1400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbe41092c0 0x7fbbe41a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.368+0000 7fbbe1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbe41041f0 0x7fbbe41a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.368+0000 7fbbe1400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbe41092c0 0x7fbbe41a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39730/0 (socket says 172.21.15.6:39730) 2024-04-26T02:05:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.368+0000 7fbbe1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbe41041f0 0x7fbbe41a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45418/0 (socket says 172.21.15.6:45418) 2024-04-26T02:05:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.368+0000 7fbbe1400640 1 -- 172.21.15.6:0/1212597135 learned_addr learned my addr 172.21.15.6:0/1212597135 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.368+0000 7fbbe2800640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbe419a590 0x7fbbe419aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.369+0000 7fbbe1400640 1 -- 172.21.15.6:0/1212597135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbe419a590 msgr2=0x7fbbe419aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:00.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.369+0000 7fbbe1400640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbe419a590 0x7fbbe419aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.369+0000 7fbbe1400640 1 -- 172.21.15.6:0/1212597135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbe41041f0 msgr2=0x7fbbe41a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:00.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.369+0000 7fbbe1400640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbe41041f0 0x7fbbe41a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.369+0000 7fbbe1400640 1 -- 172.21.15.6:0/1212597135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe419b0b0 con 0x7fbbe41092c0 2024-04-26T02:05:00.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.369+0000 7fbbe1e00640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbe41041f0 0x7fbbe41a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:05:00.369 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.370+0000 7fbbe1400640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbe41092c0 0x7fbbe41a0920 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000ed60 tx=0x7fbbd000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:00.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.370+0000 7fbbdaa00640 1 -- 172.21.15.6:0/1212597135 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0004220 con 0x7fbbe41092c0 2024-04-26T02:05:00.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.370+0000 7fbbdaa00640 1 -- 172.21.15.6:0/1212597135 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbd000ee50 con 0x7fbbe41092c0 2024-04-26T02:05:00.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.370+0000 7fbbe3200640 1 -- 172.21.15.6:0/1212597135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbe419b3a0 con 0x7fbbe41092c0 2024-04-26T02:05:00.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.371+0000 7fbbdaa00640 1 -- 172.21.15.6:0/1212597135 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd00107b0 con 0x7fbbe41092c0 2024-04-26T02:05:00.370 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.371+0000 7fbbe3200640 1 -- 172.21.15.6:0/1212597135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbe41a75f0 con 0x7fbbe41092c0 2024-04-26T02:05:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.376+0000 7fbbebe80740 1 Processor -- start 2024-04-26T02:05:00.377 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.378+0000 7fbbdaa00640 1 -- 172.21.15.6:0/1212597135 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fbbd0010950 con 0x7fbbe41092c0 2024-04-26T02:05:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.380+0000 7fbbebe80740 1 -- start start 2024-04-26T02:05:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.380+0000 7fbbebe80740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 0x560862368d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.380+0000 7fbbebe80740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x560862373650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.380+0000 7fbbebe80740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862375f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.380+0000 7fbbebe80740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5608622ce900 con 0x5608623692d0 2024-04-26T02:05:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.380+0000 7fbbebe80740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560862274ef0 con 0x560862373b90 2024-04-26T02:05:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.380+0000 7fbbebe80740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5608622ca8c0 con 0x5608623689b0 2024-04-26T02:05:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.381+0000 7fbbe2800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 0x560862368d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.381+0000 7fbbe2800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 0x560862368d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44988/0 (socket says 172.21.15.6:44988) 2024-04-26T02:05:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.381+0000 7fbbe2800640 1 -- 172.21.15.6:0/3981255474 learned_addr learned my addr 172.21.15.6:0/3981255474 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.381+0000 7fbbe1e00640 1 --2- 172.21.15.6:0/3981255474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x560862373650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.381 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.382+0000 7fbbdaa00640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbbc80a3e50 0x7fbbc80a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.382+0000 7fbbe1400640 1 --2- 172.21.15.6:0/3981255474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862375f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.382+0000 7fbbdaa00640 1 -- 172.21.15.6:0/1212597135 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7fbbd00d0bd0 con 0x7fbbe41092c0 2024-04-26T02:05:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.383+0000 7fbbe2800640 1 -- 172.21.15.6:0/3981255474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 msgr2=0x560862375f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.383+0000 7fbbe2800640 1 --2- 172.21.15.6:0/3981255474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862375f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.383+0000 7fbbe2800640 1 -- 172.21.15.6:0/3981255474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 msgr2=0x560862373650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.383+0000 7fbbe2800640 1 --2- 172.21.15.6:0/3981255474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x560862373650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.383+0000 7fbbe2800640 1 -- 172.21.15.6:0/3981255474 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560861fe1210 con 0x5608623689b0 2024-04-26T02:05:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.383+0000 7fbbe1400640 1 --2- 172.21.15.6:0/3981255474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862375f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:05:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.383+0000 7fbbe1e00640 1 --2- 172.21.15.6:0/3981255474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x560862373650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:05:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.384+0000 7fbbe2800640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbbc80a3e50 0x7fbbc80a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.384+0000 7fbbe2800640 1 --2- 172.21.15.6:0/3981255474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 0x560862368d90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009be0 tx=0x7fbbdc0315d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e55ebbdf478928df server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:00.384 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.385+0000 7fbbcf400640 1 -- 172.21.15.6:0/3981255474 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc043040 con 0x5608623689b0 2024-04-26T02:05:00.384 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.385+0000 7fbbcf400640 1 -- 172.21.15.6:0/3981255474 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbdc030720 con 0x5608623689b0 2024-04-26T02:05:00.384 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.385+0000 7fbbcf400640 1 -- 172.21.15.6:0/3981255474 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc036040 con 0x5608623689b0 2024-04-26T02:05:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.386+0000 7fbbebe80740 1 -- 172.21.15.6:0/3981255474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 msgr2=0x560862368d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.386+0000 7fbbebe80740 1 --2- 172.21.15.6:0/3981255474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 0x560862368d90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009be0 tx=0x7fbbdc0315d0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.386+0000 7fbbe2800640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbbc80a3e50 0x7fbbc80a6310 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc049000 tx=0x7fbbdc031e00 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.386+0000 7fbbebe80740 1 -- 172.21.15.6:0/3981255474 shutdown_connections 2024-04-26T02:05:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.386+0000 7fbbebe80740 1 --2- 172.21.15.6:0/3981255474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x560862373650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.386+0000 7fbbebe80740 1 --2- 172.21.15.6:0/3981255474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862375f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.386+0000 7fbbebe80740 1 --2- 172.21.15.6:0/3981255474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 0x560862368d90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.386+0000 7fbbebe80740 1 -- 172.21.15.6:0/3981255474 >> 172.21.15.6:0/3981255474 conn(0x560862364ec0 msgr2=0x5608623652a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.386+0000 7fbbebe80740 1 -- 172.21.15.6:0/3981255474 shutdown_connections 2024-04-26T02:05:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.387+0000 7fbbebe80740 1 -- 172.21.15.6:0/3981255474 wait complete. 2024-04-26T02:05:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.387+0000 7fbbebe80740 1 -- start start 2024-04-26T02:05:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.387+0000 7fbbebe80740 1 Processor -- start 2024-04-26T02:05:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.391+0000 7fbbebe80740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 0x560862370b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.392+0000 7fbbebe80740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x5608623710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.392+0000 7fbbebe80740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862371600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.392+0000 7fbbe1e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 0x560862370b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.392+0000 7fbbe1e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 0x560862370b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45000/0 (socket says 172.21.15.6:45000) 2024-04-26T02:05:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.392+0000 7fbbe1e00640 1 -- 172.21.15.6:0/322312485 learned_addr learned my addr 172.21.15.6:0/322312485 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.392+0000 7fbbebe80740 1 -- 172.21.15.6:0/322312485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56086237efd0 con 0x5608623692d0 2024-04-26T02:05:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.392+0000 7fbbebe80740 1 -- 172.21.15.6:0/322312485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560862288310 con 0x560862373b90 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.392+0000 7fbbebe80740 1 -- 172.21.15.6:0/322312485 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56086237f150 con 0x5608623689b0 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.392+0000 7fbbe2800640 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862371600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.392+0000 7fbbe1400640 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x5608623710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.392+0000 7fbbebe80740 10 client.0 _ll_register_callbacks cb 0x560862256cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.393+0000 7fbbe2800640 1 -- 172.21.15.6:0/322312485 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 msgr2=0x560862370b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.393+0000 7fbbe2800640 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 0x560862370b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.393+0000 7fbbe2800640 1 -- 172.21.15.6:0/322312485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 msgr2=0x5608623710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.393+0000 7fbbe2800640 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x5608623710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.393+0000 7fbbe2800640 1 -- 172.21.15.6:0/322312485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608620ad620 con 0x560862373b90 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.393+0000 7fbbe1400640 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x5608623710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.393+0000 7fbbe1e00640 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 0x560862370b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.393+0000 7fbbe2800640 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862371600 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc02abe0 tx=0x7fbbdc042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=759761177472d828 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.393+0000 7fbbc3e00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:05:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.394+0000 7fbbc3e00640 1 -- 172.21.15.6:0/322312485 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc059070 con 0x560862373b90 2024-04-26T02:05:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.394+0000 7fbbc3e00640 1 -- 172.21.15.6:0/322312485 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7fbbdc0319c0 con 0x560862373b90 2024-04-26T02:05:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.394+0000 7fbbebe80740 1 -- 172.21.15.6:0/322312485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbdc030720 con 0x560862373b90 2024-04-26T02:05:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.394+0000 7fbbc3e00640 1 -- 172.21.15.6:0/322312485 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fbbdc065050 con 0x560862373b90 2024-04-26T02:05:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.394+0000 7fbbebe80740 10 client.27371 fetch_fsmap learned FSMap version 55 2024-04-26T02:05:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.394+0000 7fbbebe80740 1 -- 172.21.15.6:0/322312485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x56086238b200 con 0x560862373b90 2024-04-26T02:05:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.395+0000 7fbbc3e00640 1 -- 172.21.15.6:0/322312485 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbdc030720 con 0x560862373b90 2024-04-26T02:05:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.396+0000 7fbbc3e00640 1 -- 172.21.15.6:0/322312485 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7fbbdc050070 con 0x560862373b90 2024-04-26T02:05:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.396+0000 7fbbebe80740 10 client.27371 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:05:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.396+0000 7fbbebe80740 10 client.27371 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:05:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.396+0000 7fbbebe80740 20 client.27371 populate_metadata read hostname 'smithi006' 2024-04-26T02:05:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.396+0000 7fbbebe80740 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56086238c550 0x56086238e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.396+0000 7fbbebe80740 4 client.27371 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:05:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.396+0000 7fbbebe80740 1 -- 172.21.15.6:0/322312485 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608622e28b0 con 0x56086238c550 2024-04-26T02:05:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.397+0000 7fbbe1e00640 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56086238c550 0x56086238e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.398+0000 7fbbe1e00640 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56086238c550 0x56086238e940 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78743ea04da7ffb2 server_cookie=4571594493732921 in_seq=0 out_seq=0 2024-04-26T02:05:00.397 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.398+0000 7fbbc3e00640 10 client.27371 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:00.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.404+0000 7fbbc3e00640 1 -- 172.21.15.6:0/322312485 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5608622e28b0 con 0x56086238c550 2024-04-26T02:05:00.403 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.404+0000 7fbbc3e00640 10 client.27371 handle_command_reply: tid=0 2024-04-26T02:05:00.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.404+0000 7fbbebe80740 1 client.27371 shutdown 2024-04-26T02:05:00.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.405+0000 7fbbebe80740 1 -- 172.21.15.6:0/322312485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 msgr2=0x560862371600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:00.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.405+0000 7fbbebe80740 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862371600 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc02abe0 tx=0x7fbbdc042000 comp rx=0 tx=0).stop 2024-04-26T02:05:00.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.405+0000 7fbbebe80740 1 -- 172.21.15.6:0/322312485 shutdown_connections 2024-04-26T02:05:00.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.405+0000 7fbbebe80740 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56086238c550 0x56086238e940 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.405+0000 7fbbebe80740 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x5608623710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.405+0000 7fbbebe80740 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862371600 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.405+0000 7fbbebe80740 1 --2- 172.21.15.6:0/322312485 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623689b0 0x560862370b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.405+0000 7fbbc3e00640 0 client.27371 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:00.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.405+0000 7fbbebe80740 1 -- 172.21.15.6:0/322312485 >> 172.21.15.6:0/322312485 conn(0x560862364ec0 msgr2=0x56086236ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:00.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.405+0000 7fbbebe80740 1 -- 172.21.15.6:0/322312485 shutdown_connections 2024-04-26T02:05:00.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.405+0000 7fbbebe80740 1 -- 172.21.15.6:0/322312485 wait complete. 2024-04-26T02:05:00.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.406+0000 7fbbebe80740 20 client.27371 trim_cache size 0 max 16384 2024-04-26T02:05:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.435+0000 7fbbebe80740 1 Processor -- start 2024-04-26T02:05:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.436+0000 7fbbebe80740 1 -- start start 2024-04-26T02:05:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.436+0000 7fbbebe80740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623689b0 0x560862387e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.436+0000 7fbbebe80740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623692d0 0x560862393f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.436+0000 7fbbebe80740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862394480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.436+0000 7fbbebe80740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe41ae2e0 con 0x5608623689b0 2024-04-26T02:05:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.436+0000 7fbbebe80740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe41abb50 con 0x560862373b90 2024-04-26T02:05:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.436+0000 7fbbebe80740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe41a99c0 con 0x5608623692d0 2024-04-26T02:05:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.437+0000 7fbbe2800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862394480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.437+0000 7fbbe1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623689b0 0x560862387e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.437+0000 7fbbe2800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862394480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39772/0 (socket says 172.21.15.6:39772) 2024-04-26T02:05:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.437+0000 7fbbe2800640 1 -- 172.21.15.6:0/591543523 learned_addr learned my addr 172.21.15.6:0/591543523 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.437+0000 7fbbe1400640 1 --2- 172.21.15.6:0/591543523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623692d0 0x560862393f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.437+0000 7fbbe2800640 1 -- 172.21.15.6:0/591543523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623692d0 msgr2=0x560862393f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.437+0000 7fbbe2800640 1 --2- 172.21.15.6:0/591543523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623692d0 0x560862393f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.437+0000 7fbbe2800640 1 -- 172.21.15.6:0/591543523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623689b0 msgr2=0x560862387e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.437+0000 7fbbe2800640 1 --2- 172.21.15.6:0/591543523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623689b0 0x560862387e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.437+0000 7fbbe2800640 1 -- 172.21.15.6:0/591543523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560862369100 con 0x560862373b90 2024-04-26T02:05:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.437+0000 7fbbe1e00640 1 --2- 172.21.15.6:0/591543523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623689b0 0x560862387e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:05:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.438+0000 7fbbe2800640 1 --2- 172.21.15.6:0/591543523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862394480 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0046c0 tx=0x7fbbdc005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff4f699c700374c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.438+0000 7fbbcea00640 1 -- 172.21.15.6:0/591543523 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc043070 con 0x560862373b90 2024-04-26T02:05:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.438+0000 7fbbcea00640 1 -- 172.21.15.6:0/591543523 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbdc009800 con 0x560862373b90 2024-04-26T02:05:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.438+0000 7fbbcea00640 1 -- 172.21.15.6:0/591543523 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc059040 con 0x560862373b90 2024-04-26T02:05:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.439+0000 7fbbebe80740 1 -- 172.21.15.6:0/591543523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 msgr2=0x560862394480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.439+0000 7fbbebe80740 1 --2- 172.21.15.6:0/591543523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862394480 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0046c0 tx=0x7fbbdc005f70 comp rx=0 tx=0).stop 2024-04-26T02:05:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.439+0000 7fbbebe80740 1 -- 172.21.15.6:0/591543523 shutdown_connections 2024-04-26T02:05:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.439+0000 7fbbebe80740 1 --2- 172.21.15.6:0/591543523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560862373b90 0x560862394480 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.439+0000 7fbbebe80740 1 --2- 172.21.15.6:0/591543523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608623692d0 0x560862393f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.439+0000 7fbbebe80740 1 --2- 172.21.15.6:0/591543523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623689b0 0x560862387e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.439+0000 7fbbebe80740 1 -- 172.21.15.6:0/591543523 >> 172.21.15.6:0/591543523 conn(0x560862364ec0 msgr2=0x56086236be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.439+0000 7fbbebe80740 1 -- 172.21.15.6:0/591543523 shutdown_connections 2024-04-26T02:05:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.439+0000 7fbbebe80740 1 -- 172.21.15.6:0/591543523 wait complete. 2024-04-26T02:05:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.440+0000 7fbbebe80740 1 -- start start 2024-04-26T02:05:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.440+0000 7fbbebe80740 1 Processor -- start 2024-04-26T02:05:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.441+0000 7fbbebe80740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608623689b0 0x5608623844c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.441+0000 7fbbebe80740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x560862381950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.441+0000 7fbbebe80740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560862373b90 0x560862381e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.441+0000 7fbbebe80740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe41aa010 con 0x5608623692d0 2024-04-26T02:05:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.441+0000 7fbbebe80740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe41ab480 con 0x5608623689b0 2024-04-26T02:05:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.441+0000 7fbbebe80740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe41aa6e0 con 0x560862373b90 2024-04-26T02:05:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.441+0000 7fbbebe80740 10 client.0 _ll_register_callbacks cb 0x7fbbdc031d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.441+0000 7fbbe1e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608623689b0 0x5608623844c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.441+0000 7fbbe1e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608623689b0 0x5608623844c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39774/0 (socket says 172.21.15.6:39774) 2024-04-26T02:05:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.441+0000 7fbbe1e00640 1 -- 172.21.15.6:0/1048728263 learned_addr learned my addr 172.21.15.6:0/1048728263 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.442+0000 7fbbe2800640 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560862373b90 0x560862381e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.442+0000 7fbbe1400640 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x560862381950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.442+0000 7fbbe1e00640 1 -- 172.21.15.6:0/1048728263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560862373b90 msgr2=0x560862381e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.442+0000 7fbbe1e00640 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560862373b90 0x560862381e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.442+0000 7fbbe1e00640 1 -- 172.21.15.6:0/1048728263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 msgr2=0x560862381950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.442+0000 7fbbe1e00640 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x560862381950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.442+0000 7fbbe1e00640 1 -- 172.21.15.6:0/1048728263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560862382400 con 0x5608623689b0 2024-04-26T02:05:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.443+0000 7fbbe1e00640 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608623689b0 0x5608623844c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x56086238eee0 tx=0x7fbbd40108d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d69bc51fbf0c24d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.443+0000 7fbbc3e00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:05:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.443+0000 7fbbc3e00640 1 -- 172.21.15.6:0/1048728263 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd4004220 con 0x5608623689b0 2024-04-26T02:05:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.443+0000 7fbbc3e00640 1 -- 172.21.15.6:0/1048728263 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7fbbd40043c0 con 0x5608623689b0 2024-04-26T02:05:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.443+0000 7fbbebe80740 1 -- 172.21.15.6:0/1048728263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbdc009800 con 0x5608623689b0 2024-04-26T02:05:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.445+0000 7fbbc3e00640 1 -- 172.21.15.6:0/1048728263 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fbbd4005020 con 0x5608623689b0 2024-04-26T02:05:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.445+0000 7fbbebe80740 10 client.27377 fetch_fsmap learned FSMap version 55 2024-04-26T02:05:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.445+0000 7fbbebe80740 1 -- 172.21.15.6:0/1048728263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5608623f25d0 con 0x5608623689b0 2024-04-26T02:05:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.445+0000 7fbbc3e00640 1 -- 172.21.15.6:0/1048728263 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbdc009800 con 0x5608623689b0 2024-04-26T02:05:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.447+0000 7fbbc3e00640 1 -- 172.21.15.6:0/1048728263 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7fbbd4004700 con 0x5608623689b0 2024-04-26T02:05:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.447+0000 7fbbebe80740 10 client.27377 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:05:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.447+0000 7fbbebe80740 10 client.27377 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:05:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.447+0000 7fbbebe80740 20 client.27377 populate_metadata read hostname 'smithi006' 2024-04-26T02:05:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.447+0000 7fbbebe80740 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5608623f2d20 0x5608623f5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.447+0000 7fbbebe80740 4 client.27377 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:05:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.447+0000 7fbbebe80740 1 -- 172.21.15.6:0/1048728263 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5608622dc320 con 0x5608623f2d20 2024-04-26T02:05:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.447+0000 7fbbe1400640 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5608623f2d20 0x5608623f5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.448+0000 7fbbe1400640 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5608623f2d20 0x5608623f5140 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fc4a256cdeadafb server_cookie=baee3385ed3b43ab in_seq=0 out_seq=0 2024-04-26T02:05:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.448+0000 7fbbc3e00640 10 client.27377 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.721+0000 7fbbc3e00640 1 -- 172.21.15.6:0/1048728263 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5608622dc320 con 0x5608623f2d20 2024-04-26T02:05:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.721+0000 7fbbc3e00640 10 client.27377 handle_command_reply: tid=0 2024-04-26T02:05:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.721+0000 7fbbebe80740 1 client.27377 shutdown 2024-04-26T02:05:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.722+0000 7fbbebe80740 1 -- 172.21.15.6:0/1048728263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608623689b0 msgr2=0x5608623844c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.722+0000 7fbbebe80740 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608623689b0 0x5608623844c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x56086238eee0 tx=0x7fbbd40108d0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.722+0000 7fbbebe80740 1 -- 172.21.15.6:0/1048728263 shutdown_connections 2024-04-26T02:05:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.722+0000 7fbbebe80740 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5608623f2d20 0x5608623f5140 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.722+0000 7fbbebe80740 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5608623692d0 0x560862381950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.722+0000 7fbbebe80740 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560862373b90 0x560862381e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.722+0000 7fbbebe80740 1 --2- 172.21.15.6:0/1048728263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5608623689b0 0x5608623844c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.722+0000 7fbbebe80740 1 -- 172.21.15.6:0/1048728263 >> 172.21.15.6:0/1048728263 conn(0x560862364ec0 msgr2=0x56086236a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.722+0000 7fbbc3e00640 0 client.27377 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.723+0000 7fbbebe80740 1 -- 172.21.15.6:0/1048728263 shutdown_connections 2024-04-26T02:05:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.724+0000 7fbbebe80740 1 -- 172.21.15.6:0/1048728263 wait complete. 2024-04-26T02:05:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.724+0000 7fbbebe80740 20 client.27377 trim_cache size 0 max 16384 2024-04-26T02:05:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:05:00.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:05:00.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:05:00.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:05:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.726+0000 7fbbe3200640 1 -- 172.21.15.6:0/1212597135 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbbc80a3e50 msgr2=0x7fbbc80a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.726+0000 7fbbe3200640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbbc80a3e50 0x7fbbc80a6310 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc049000 tx=0x7fbbdc031e00 comp rx=0 tx=0).stop 2024-04-26T02:05:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.726+0000 7fbbe3200640 1 -- 172.21.15.6:0/1212597135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbe41092c0 msgr2=0x7fbbe41a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.726+0000 7fbbe3200640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbe41092c0 0x7fbbe41a0920 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000ed60 tx=0x7fbbd000c6a0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.726+0000 7fbbe3200640 1 -- 172.21.15.6:0/1212597135 shutdown_connections 2024-04-26T02:05:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.726+0000 7fbbe3200640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fbbc80a3e50 0x7fbbc80a6310 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.726+0000 7fbbe3200640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbe419a590 0x7fbbe419aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.726+0000 7fbbe3200640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbe41092c0 0x7fbbe41a0920 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.726+0000 7fbbe3200640 1 --2- 172.21.15.6:0/1212597135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbe41041f0 0x7fbbe41a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.727+0000 7fbbe3200640 1 -- 172.21.15.6:0/1212597135 >> 172.21.15.6:0/1212597135 conn(0x7fbbe40ffa50 msgr2=0x7fbbe4100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.728+0000 7fbbe3200640 1 -- 172.21.15.6:0/1212597135 shutdown_connections 2024-04-26T02:05:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:00.728+0000 7fbbe3200640 1 -- 172.21.15.6:0/1212597135 wait complete. 2024-04-26T02:05:00.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:00 smithi006 ceph-mon[28419]: pgmap v606: 97 pgs: 97 active+clean; 963 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.6 MiB/s wr, 428 op/s 2024-04-26T02:05:00.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:00 smithi188 ceph-mon[32751]: pgmap v606: 97 pgs: 97 active+clean; 963 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.6 MiB/s wr, 428 op/s 2024-04-26T02:05:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:00 smithi029 ceph-mon[32775]: pgmap v606: 97 pgs: 97 active+clean; 963 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.6 MiB/s wr, 428 op/s 2024-04-26T02:05:01.131 INFO:tasks.workunit.client.0.smithi006.stdout: 10 97883 17.33 MB/sec execute 204 sec latency 275.258 ms 2024-04-26T02:05:02.131 INFO:tasks.workunit.client.0.smithi006.stdout: 10 98270 17.34 MB/sec execute 205 sec latency 103.826 ms 2024-04-26T02:05:02.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:02 smithi006 ceph-mon[28419]: pgmap v607: 97 pgs: 97 active+clean; 974 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.4 MiB/s wr, 770 op/s 2024-04-26T02:05:02.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:02 smithi188 ceph-mon[32751]: pgmap v607: 97 pgs: 97 active+clean; 974 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.4 MiB/s wr, 770 op/s 2024-04-26T02:05:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:02 smithi029 ceph-mon[32775]: pgmap v607: 97 pgs: 97 active+clean; 974 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.4 MiB/s wr, 770 op/s 2024-04-26T02:05:03.131 INFO:tasks.workunit.client.0.smithi006.stdout: 10 98747 17.42 MB/sec execute 206 sec latency 80.671 ms 2024-04-26T02:05:04.131 INFO:tasks.workunit.client.0.smithi006.stdout: 10 99256 17.46 MB/sec execute 207 sec latency 110.809 ms 2024-04-26T02:05:04.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:04 smithi006 ceph-mon[28419]: pgmap v608: 97 pgs: 97 active+clean; 974 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.4 MiB/s wr, 502 op/s 2024-04-26T02:05:04.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:04 smithi188 ceph-mon[32751]: pgmap v608: 97 pgs: 97 active+clean; 974 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.4 MiB/s wr, 502 op/s 2024-04-26T02:05:04.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:04 smithi029 ceph-mon[32775]: pgmap v608: 97 pgs: 97 active+clean; 974 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.4 MiB/s wr, 502 op/s 2024-04-26T02:05:05.131 INFO:tasks.workunit.client.0.smithi006.stdout: 10 99730 17.47 MB/sec execute 208 sec latency 120.704 ms 2024-04-26T02:05:06.131 INFO:tasks.workunit.client.0.smithi006.stdout: 10 100259 17.52 MB/sec execute 209 sec latency 226.380 ms 2024-04-26T02:05:06.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:06 smithi006 ceph-mon[28419]: pgmap v609: 97 pgs: 97 active+clean; 974 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.4 MiB/s wr, 502 op/s 2024-04-26T02:05:06.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:06 smithi188 ceph-mon[32751]: pgmap v609: 97 pgs: 97 active+clean; 974 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.4 MiB/s wr, 502 op/s 2024-04-26T02:05:06.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:06 smithi029 ceph-mon[32775]: pgmap v609: 97 pgs: 97 active+clean; 974 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.4 MiB/s wr, 502 op/s 2024-04-26T02:05:07.131 INFO:tasks.workunit.client.0.smithi006.stdout: 10 100893 17.60 MB/sec execute 210 sec latency 87.776 ms 2024-04-26T02:05:08.132 INFO:tasks.workunit.client.0.smithi006.stdout: 10 101594 17.68 MB/sec execute 211 sec latency 71.741 ms 2024-04-26T02:05:08.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:08 smithi006 ceph-mon[28419]: pgmap v610: 97 pgs: 97 active+clean; 1005 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.6 MiB/s wr, 794 op/s 2024-04-26T02:05:08.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:08 smithi188 ceph-mon[32751]: pgmap v610: 97 pgs: 97 active+clean; 1005 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.6 MiB/s wr, 794 op/s 2024-04-26T02:05:08.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:08 smithi029 ceph-mon[32775]: pgmap v610: 97 pgs: 97 active+clean; 1005 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.6 MiB/s wr, 794 op/s 2024-04-26T02:05:09.132 INFO:tasks.workunit.client.0.smithi006.stdout: 10 102179 17.81 MB/sec execute 212 sec latency 176.959 ms 2024-04-26T02:05:10.132 INFO:tasks.workunit.client.0.smithi006.stdout: 10 102430 17.81 MB/sec execute 213 sec latency 364.701 ms 2024-04-26T02:05:10.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:10 smithi006 ceph-mon[28419]: pgmap v611: 97 pgs: 97 active+clean; 1005 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 6.0 MiB/s wr, 633 op/s 2024-04-26T02:05:10.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:10 smithi188 ceph-mon[32751]: pgmap v611: 97 pgs: 97 active+clean; 1005 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 6.0 MiB/s wr, 633 op/s 2024-04-26T02:05:10.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:10 smithi029 ceph-mon[32775]: pgmap v611: 97 pgs: 97 active+clean; 1005 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 6.0 MiB/s wr, 633 op/s 2024-04-26T02:05:11.132 INFO:tasks.workunit.client.0.smithi006.stdout: 10 102732 17.76 MB/sec execute 214 sec latency 249.819 ms 2024-04-26T02:05:12.132 INFO:tasks.workunit.client.0.smithi006.stdout: 10 103215 17.77 MB/sec execute 215 sec latency 83.653 ms 2024-04-26T02:05:12.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:12 smithi188 ceph-mon[32751]: pgmap v612: 97 pgs: 97 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-26T02:05:12.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:12 smithi029 ceph-mon[32775]: pgmap v612: 97 pgs: 97 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-26T02:05:13.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:12 smithi006 ceph-mon[28419]: pgmap v612: 97 pgs: 97 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-26T02:05:13.132 INFO:tasks.workunit.client.0.smithi006.stdout: 10 103781 17.82 MB/sec execute 216 sec latency 85.631 ms 2024-04-26T02:05:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.914+0000 7f91bb000640 1 -- 172.21.15.6:0/1803048916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91b4102b50 msgr2=0x7f91b4106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.914+0000 7f91bb000640 1 --2- 172.21.15.6:0/1803048916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91b4102b50 0x7f91b4106b90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009860 tx=0x7f91a002a340 comp rx=0 tx=0).stop 2024-04-26T02:05:13.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.916+0000 7f91bb000640 1 -- 172.21.15.6:0/1803048916 shutdown_connections 2024-04-26T02:05:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.916+0000 7f91bb000640 1 --2- 172.21.15.6:0/1803048916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b4107270 0x7f91b410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.916+0000 7f91bb000640 1 --2- 172.21.15.6:0/1803048916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91b4102b50 0x7f91b4106b90 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.916+0000 7f91bb000640 1 --2- 172.21.15.6:0/1803048916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91b41021a0 0x7f91b4102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.916+0000 7f91bb000640 1 -- 172.21.15.6:0/1803048916 >> 172.21.15.6:0/1803048916 conn(0x7f91b40fd9e0 msgr2=0x7f91b40ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.916+0000 7f91bb000640 1 -- 172.21.15.6:0/1803048916 shutdown_connections 2024-04-26T02:05:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.916+0000 7f91bb000640 1 -- 172.21.15.6:0/1803048916 wait complete. 2024-04-26T02:05:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.918+0000 7f91bb000640 1 Processor -- start 2024-04-26T02:05:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.919+0000 7f91bb000640 1 -- start start 2024-04-26T02:05:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.919+0000 7f91bb000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b41021a0 0x7f91b410e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.919+0000 7f91bb000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91b4107270 0x7f91b410e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.919+0000 7f91bb000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91b41a3650 0x7f91b41a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.919+0000 7f91bb000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f91b4112190 con 0x7f91b41021a0 2024-04-26T02:05:13.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.919+0000 7f91bb000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f91b4112010 con 0x7f91b41a3650 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.919+0000 7f91bb000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f91b4112310 con 0x7f91b4107270 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.920+0000 7f91b3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91b4107270 0x7f91b410e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.920+0000 7f91b3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b41021a0 0x7f91b410e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.920+0000 7f91b3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b41021a0 0x7f91b410e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53638/0 (socket says 172.21.15.6:53638) 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.920+0000 7f91b3e00640 1 -- 172.21.15.6:0/1709239915 learned_addr learned my addr 172.21.15.6:0/1709239915 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.920+0000 7f91b9000640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91b41a3650 0x7f91b41a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.920+0000 7f91b3e00640 1 -- 172.21.15.6:0/1709239915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91b4107270 msgr2=0x7f91b410e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.920+0000 7f91b3e00640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91b4107270 0x7f91b410e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.921+0000 7f91b3e00640 1 -- 172.21.15.6:0/1709239915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91b41a3650 msgr2=0x7f91b41a3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.921+0000 7f91b3e00640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91b41a3650 0x7f91b41a3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.921+0000 7f91b3e00640 1 -- 172.21.15.6:0/1709239915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b41a4140 con 0x7f91b41021a0 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.921+0000 7f91b9000640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91b41a3650 0x7f91b41a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:05:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.921+0000 7f91b3e00640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b41021a0 0x7f91b410e280 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f91a400e9c0 tx=0x7f91a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.922+0000 7f91b0c00640 1 -- 172.21.15.6:0/1709239915 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a400ccc0 con 0x7f91b41021a0 2024-04-26T02:05:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.922+0000 7f91bb000640 1 -- 172.21.15.6:0/1709239915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91b41a4430 con 0x7f91b41021a0 2024-04-26T02:05:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.922+0000 7f91bb000640 1 -- 172.21.15.6:0/1709239915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91b41abbf0 con 0x7f91b41021a0 2024-04-26T02:05:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.922+0000 7f91b0c00640 1 -- 172.21.15.6:0/1709239915 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91a40054d0 con 0x7f91b41021a0 2024-04-26T02:05:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.922+0000 7f91b0c00640 1 -- 172.21.15.6:0/1709239915 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a4010430 con 0x7f91b41021a0 2024-04-26T02:05:13.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.927+0000 7f91bdfa9740 1 Processor -- start 2024-04-26T02:05:13.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.927+0000 7f91bdfa9740 1 -- start start 2024-04-26T02:05:13.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.927+0000 7f91b0c00640 1 -- 172.21.15.6:0/1709239915 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f91a40046c0 con 0x7f91b41021a0 2024-04-26T02:05:13.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91bdfa9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833767820 0x55a833767c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91bdfa9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 0x55a8337724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91bdfa9740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833772a00 0x55a833774df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91bdfa9740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a8336cd900 con 0x55a833772a00 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91bdfa9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a833673ef0 con 0x55a833767820 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91bdfa9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a8336c98c0 con 0x55a833768140 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91b3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 0x55a8337724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91b3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 0x55a8337724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51154/0 (socket says 172.21.15.6:51154) 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91b3e00640 1 -- 172.21.15.6:0/178202164 learned_addr learned my addr 172.21.15.6:0/178202164 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91b3400640 1 --2- 172.21.15.6:0/178202164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833772a00 0x55a833774df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91b9000640 1 --2- 172.21.15.6:0/178202164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833767820 0x55a833767c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91b3e00640 1 -- 172.21.15.6:0/178202164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833767820 msgr2=0x55a833767c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.928+0000 7f91b3e00640 1 --2- 172.21.15.6:0/178202164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833767820 0x55a833767c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.929+0000 7f91b3e00640 1 -- 172.21.15.6:0/178202164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833772a00 msgr2=0x55a833774df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.929+0000 7f91b3e00640 1 --2- 172.21.15.6:0/178202164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833772a00 0x55a833774df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.929+0000 7f91b3e00640 1 -- 172.21.15.6:0/178202164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8333e0210 con 0x55a833768140 2024-04-26T02:05:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.929+0000 7f91b9000640 1 --2- 172.21.15.6:0/178202164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833767820 0x55a833767c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:05:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.929+0000 7f91b3e00640 1 --2- 172.21.15.6:0/178202164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 0x55a8337724c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f91a40cdeb0 tx=0x7f91a40d6b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90f0af570f4e945d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.929+0000 7f91a9600640 1 -- 172.21.15.6:0/178202164 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a40df070 con 0x55a833768140 2024-04-26T02:05:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.930+0000 7f91a9600640 1 -- 172.21.15.6:0/178202164 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91a40db440 con 0x55a833768140 2024-04-26T02:05:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.930+0000 7f91a9600640 1 -- 172.21.15.6:0/178202164 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a40da5e0 con 0x55a833768140 2024-04-26T02:05:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.930+0000 7f91b0c00640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f91980a3e60 0x7f91980a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.930+0000 7f91b0c00640 1 -- 172.21.15.6:0/1709239915 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f91a40d0840 con 0x7f91b41021a0 2024-04-26T02:05:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.930+0000 7f91b9000640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f91980a3e60 0x7f91980a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.931+0000 7f91bdfa9740 1 -- 172.21.15.6:0/178202164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 msgr2=0x55a8337724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.931+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/178202164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 0x55a8337724c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f91a40cdeb0 tx=0x7f91a40d6b40 comp rx=0 tx=0).stop 2024-04-26T02:05:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.931+0000 7f91b9000640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f91980a3e60 0x7f91980a6320 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f91ac005fd0 tx=0x7f91ac009500 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.932+0000 7f91bdfa9740 1 -- 172.21.15.6:0/178202164 shutdown_connections 2024-04-26T02:05:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.932+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/178202164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833772a00 0x55a833774df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.932+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/178202164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 0x55a8337724c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.932+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/178202164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833767820 0x55a833767c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.932+0000 7f91bdfa9740 1 -- 172.21.15.6:0/178202164 >> 172.21.15.6:0/178202164 conn(0x55a833763d30 msgr2=0x55a833764110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.932+0000 7f91bdfa9740 1 -- 172.21.15.6:0/178202164 shutdown_connections 2024-04-26T02:05:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.932+0000 7f91bdfa9740 1 -- 172.21.15.6:0/178202164 wait complete. 2024-04-26T02:05:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.932+0000 7f91bdfa9740 1 -- start start 2024-04-26T02:05:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.933+0000 7f91bdfa9740 1 Processor -- start 2024-04-26T02:05:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.933+0000 7f91bdfa9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833767820 0x55a83376f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.933+0000 7f91bdfa9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833768140 0x55a83376fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.933+0000 7f91bdfa9740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833772a00 0x55a8337703b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.933+0000 7f91bdfa9740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a833775630 con 0x55a833772a00 2024-04-26T02:05:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.934+0000 7f91bdfa9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a833687310 con 0x55a833768140 2024-04-26T02:05:13.933 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.934+0000 7f91bdfa9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a8337757b0 con 0x55a833767820 2024-04-26T02:05:13.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.934+0000 7f91b3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833768140 0x55a83376fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.934+0000 7f91b3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833768140 0x55a83376fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39970/0 (socket says 172.21.15.6:39970) 2024-04-26T02:05:13.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.934+0000 7f91b3400640 1 -- 172.21.15.6:0/2123901987 learned_addr learned my addr 172.21.15.6:0/2123901987 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:13.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.934+0000 7f91bdfa9740 10 client.0 _ll_register_callbacks cb 0x55a833655cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:13.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.935+0000 7f91b3400640 1 -- 172.21.15.6:0/2123901987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833767820 msgr2=0x55a83376f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.935+0000 7f91b9000640 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833772a00 0x55a8337703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.935+0000 7f91b3e00640 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833767820 0x55a83376f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.935+0000 7f91b3400640 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833767820 0x55a83376f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.936+0000 7f91b3400640 1 -- 172.21.15.6:0/2123901987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833772a00 msgr2=0x55a8337703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.936+0000 7f91b3400640 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833772a00 0x55a8337703b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.936+0000 7f91b3400640 1 -- 172.21.15.6:0/2123901987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8334ac620 con 0x55a833768140 2024-04-26T02:05:13.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.936+0000 7f91b9000640 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833772a00 0x55a8337703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:05:13.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.937+0000 7f91b3400640 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833768140 0x55a83376fe70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f91a002a930 tx=0x7f91a0030d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d202d9542a1c1c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:13.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.937+0000 7f919e000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:05:13.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.937+0000 7f919e000640 1 -- 172.21.15.6:0/2123901987 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0037040 con 0x55a833768140 2024-04-26T02:05:13.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.938+0000 7f919e000640 1 -- 172.21.15.6:0/2123901987 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f91a0004250 con 0x55a833768140 2024-04-26T02:05:13.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.938+0000 7f91bdfa9740 1 -- 172.21.15.6:0/2123901987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91a40db440 con 0x55a833768140 2024-04-26T02:05:13.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.938+0000 7f919e000640 1 -- 172.21.15.6:0/2123901987 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f91a0004450 con 0x55a833768140 2024-04-26T02:05:13.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.940+0000 7f91bdfa9740 10 client.27389 fetch_fsmap learned FSMap version 55 2024-04-26T02:05:13.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.940+0000 7f91bdfa9740 1 -- 172.21.15.6:0/2123901987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55a833781880 con 0x55a833768140 2024-04-26T02:05:13.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.940+0000 7f919e000640 1 -- 172.21.15.6:0/2123901987 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91a40db440 con 0x55a833768140 2024-04-26T02:05:13.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.941+0000 7f919e000640 1 -- 172.21.15.6:0/2123901987 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7f91a0003c60 con 0x55a833768140 2024-04-26T02:05:13.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.941+0000 7f91bdfa9740 10 client.27389 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:05:13.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.941+0000 7f91bdfa9740 10 client.27389 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:05:13.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.941+0000 7f91bdfa9740 20 client.27389 populate_metadata read hostname 'smithi006' 2024-04-26T02:05:13.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.941+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a833782db0 0x55a833785220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.941 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.941+0000 7f91bdfa9740 4 client.27389 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:05:13.941 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.941+0000 7f91bdfa9740 1 -- 172.21.15.6:0/2123901987 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8336e18b0 con 0x55a833782db0 2024-04-26T02:05:13.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.945+0000 7f91b3e00640 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a833782db0 0x55a833785220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.948+0000 7f91b3e00640 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a833782db0 0x55a833785220 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90b710e7829649f server_cookie=9ccef89f3d0fe8ff in_seq=0 out_seq=0 2024-04-26T02:05:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.948+0000 7f919e000640 10 client.27389 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:13.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.953+0000 7f919e000640 1 -- 172.21.15.6:0/2123901987 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8336e18b0 con 0x55a833782db0 2024-04-26T02:05:13.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.953+0000 7f919e000640 10 client.27389 handle_command_reply: tid=0 2024-04-26T02:05:13.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.953+0000 7f91bdfa9740 1 client.27389 shutdown 2024-04-26T02:05:13.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.954+0000 7f91bdfa9740 1 -- 172.21.15.6:0/2123901987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833768140 msgr2=0x55a83376fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.954+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833768140 0x55a83376fe70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f91a002a930 tx=0x7f91a0030d80 comp rx=0 tx=0).stop 2024-04-26T02:05:13.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.954+0000 7f91bdfa9740 1 -- 172.21.15.6:0/2123901987 shutdown_connections 2024-04-26T02:05:13.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.954+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a833782db0 0x55a833785220 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.954+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833772a00 0x55a8337703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.954+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833768140 0x55a83376fe70 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.954+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/2123901987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833767820 0x55a83376f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.954 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.954+0000 7f91bdfa9740 1 -- 172.21.15.6:0/2123901987 >> 172.21.15.6:0/2123901987 conn(0x55a833763d30 msgr2=0x55a833769a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:13.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.954+0000 7f919e000640 0 client.27389 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:13.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.954+0000 7f91bdfa9740 1 -- 172.21.15.6:0/2123901987 shutdown_connections 2024-04-26T02:05:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.954+0000 7f91bdfa9740 1 -- 172.21.15.6:0/2123901987 wait complete. 2024-04-26T02:05:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.956+0000 7f91bdfa9740 20 client.27389 trim_cache size 0 max 16384 2024-04-26T02:05:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.977+0000 7f91bdfa9740 1 Processor -- start 2024-04-26T02:05:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.977+0000 7f91bdfa9740 1 -- start start 2024-04-26T02:05:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.978+0000 7f91bdfa9740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833767820 0x55a83377e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.978+0000 7f91bdfa9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 0x55a833763610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.978+0000 7f91bdfa9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833772a00 0x55a83376ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.978+0000 7f91bdfa9740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f91940111f0 con 0x55a833767820 2024-04-26T02:05:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.978+0000 7f91bdfa9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f919400ea60 con 0x55a833772a00 2024-04-26T02:05:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.978+0000 7f91bdfa9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f919400c8d0 con 0x55a833768140 2024-04-26T02:05:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.978+0000 7f91b9000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833772a00 0x55a83376ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.978+0000 7f91b3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 0x55a833763610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.978+0000 7f91b3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 0x55a833763610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51172/0 (socket says 172.21.15.6:51172) 2024-04-26T02:05:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.978+0000 7f91b3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833767820 0x55a83377e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.978+0000 7f91b3400640 1 -- 172.21.15.6:0/3909568233 learned_addr learned my addr 172.21.15.6:0/3909568233 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.979+0000 7f91b3400640 1 -- 172.21.15.6:0/3909568233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833772a00 msgr2=0x55a83376ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.979+0000 7f91b3400640 1 --2- 172.21.15.6:0/3909568233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833772a00 0x55a83376ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.979+0000 7f91b3400640 1 -- 172.21.15.6:0/3909568233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833767820 msgr2=0x55a83377e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.979+0000 7f91b3400640 1 --2- 172.21.15.6:0/3909568233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833767820 0x55a83377e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.979+0000 7f91b3400640 1 -- 172.21.15.6:0/3909568233 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a833767f70 con 0x55a833768140 2024-04-26T02:05:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.979+0000 7f91b9000640 1 --2- 172.21.15.6:0/3909568233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833772a00 0x55a83376ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:05:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.979+0000 7f91b3400640 1 --2- 172.21.15.6:0/3909568233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 0x55a833763610 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f91a002a860 tx=0x7f91a002a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2136d09ea69a0eaf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.981+0000 7f91a8c00640 1 -- 172.21.15.6:0/3909568233 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0037040 con 0x55a833768140 2024-04-26T02:05:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.981+0000 7f91a8c00640 1 -- 172.21.15.6:0/3909568233 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91a0030900 con 0x55a833768140 2024-04-26T02:05:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.981+0000 7f91a8c00640 1 -- 172.21.15.6:0/3909568233 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0050b90 con 0x55a833768140 2024-04-26T02:05:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.982+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3909568233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 msgr2=0x55a833763610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.982+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/3909568233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 0x55a833763610 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f91a002a860 tx=0x7f91a002a340 comp rx=0 tx=0).stop 2024-04-26T02:05:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.982+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3909568233 shutdown_connections 2024-04-26T02:05:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.982+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/3909568233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833772a00 0x55a83376ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.982+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/3909568233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833768140 0x55a833763610 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.982+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/3909568233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833767820 0x55a83377e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.982+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3909568233 >> 172.21.15.6:0/3909568233 conn(0x55a833763d30 msgr2=0x55a833764110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:13.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.985+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3909568233 shutdown_connections 2024-04-26T02:05:13.984 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.985+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3909568233 wait complete. 2024-04-26T02:05:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.986+0000 7f91bdfa9740 1 -- start start 2024-04-26T02:05:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.986+0000 7f91bdfa9740 1 Processor -- start 2024-04-26T02:05:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.987+0000 7f91bdfa9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833767820 0x55a833777bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.987+0000 7f91bdfa9740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833768140 0x55a833778100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.987+0000 7f91bdfa9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833772a00 0x55a8337e73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.987+0000 7f91bdfa9740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f919400cf20 con 0x55a833768140 2024-04-26T02:05:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.987+0000 7f91bdfa9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f919400e390 con 0x55a833772a00 2024-04-26T02:05:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.987+0000 7f91bdfa9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f919400d5f0 con 0x55a833767820 2024-04-26T02:05:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.987+0000 7f91bdfa9740 10 client.0 _ll_register_callbacks cb 0x7f91a00096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.988+0000 7f91b9000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833772a00 0x55a8337e73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.988+0000 7f91b3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833767820 0x55a833777bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.988+0000 7f91b9000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833772a00 0x55a8337e73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:39998/0 (socket says 172.21.15.6:39998) 2024-04-26T02:05:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.988+0000 7f91b9000640 1 -- 172.21.15.6:0/3385677768 learned_addr learned my addr 172.21.15.6:0/3385677768 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.988+0000 7f91b3400640 1 --2- 172.21.15.6:0/3385677768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833768140 0x55a833778100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.990 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.990+0000 7f91b3400640 1 -- 172.21.15.6:0/3385677768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833767820 msgr2=0x55a833777bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.990+0000 7f91b3400640 1 --2- 172.21.15.6:0/3385677768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833767820 0x55a833777bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.990+0000 7f91b3400640 1 -- 172.21.15.6:0/3385677768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833772a00 msgr2=0x55a8337e73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.990+0000 7f91b3400640 1 --2- 172.21.15.6:0/3385677768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833772a00 0x55a8337e73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.990+0000 7f91b3400640 1 -- 172.21.15.6:0/3385677768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8337f8940 con 0x55a833768140 2024-04-26T02:05:13.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.991+0000 7f91b3400640 1 --2- 172.21.15.6:0/3385677768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833768140 0x55a833778100 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x55a8337856b0 tx=0x7f91a0050690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd7d948e8955b2d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:13.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.991+0000 7f919e000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:05:13.991 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.992+0000 7f919e000640 1 -- 172.21.15.6:0/3385677768 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a005e060 con 0x55a833768140 2024-04-26T02:05:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.992+0000 7f919e000640 1 -- 172.21.15.6:0/3385677768 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f91a0050890 con 0x55a833768140 2024-04-26T02:05:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.992+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3385677768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91a0030900 con 0x55a833768140 2024-04-26T02:05:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.993+0000 7f919e000640 1 -- 172.21.15.6:0/3385677768 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f91a0037040 con 0x55a833768140 2024-04-26T02:05:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.993+0000 7f91bdfa9740 10 client.17193 fetch_fsmap learned FSMap version 55 2024-04-26T02:05:13.995 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.995+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3385677768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55a8337f9c20 con 0x55a833768140 2024-04-26T02:05:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.998+0000 7f919e000640 1 -- 172.21.15.6:0/3385677768 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91a0030900 con 0x55a833768140 2024-04-26T02:05:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.998+0000 7f919e000640 1 -- 172.21.15.6:0/3385677768 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7f91a0059950 con 0x55a833768140 2024-04-26T02:05:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.998+0000 7f91bdfa9740 10 client.17193 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:05:13.997 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.998+0000 7f91bdfa9740 10 client.17193 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:05:13.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.998+0000 7f91bdfa9740 20 client.17193 populate_metadata read hostname 'smithi006' 2024-04-26T02:05:13.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.998+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/3385677768 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a8337fa300 0x55a8337fc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.998+0000 7f91bdfa9740 4 client.17193 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:05:13.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.998+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3385677768 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a8336db320 con 0x55a8337fa300 2024-04-26T02:05:13.998 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:13.999+0000 7f91b3e00640 1 --2- 172.21.15.6:0/3385677768 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a8337fa300 0x55a8337fc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:14.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.000+0000 7f91b3e00640 1 --2- 172.21.15.6:0/3385677768 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a8337fa300 0x55a8337fc720 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db8c166b25faef44 server_cookie=8d7f8449278ff4c9 in_seq=0 out_seq=0 2024-04-26T02:05:14.000 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.000+0000 7f919e000640 10 client.17193 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:14.132 INFO:tasks.workunit.client.0.smithi006.stdout: 10 104397 17.88 MB/sec execute 217 sec latency 126.728 ms 2024-04-26T02:05:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.264+0000 7f919e000640 1 -- 172.21.15.6:0/3385677768 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a8336db320 con 0x55a8337fa300 2024-04-26T02:05:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.264+0000 7f919e000640 10 client.17193 handle_command_reply: tid=0 2024-04-26T02:05:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.264+0000 7f91bdfa9740 1 client.17193 shutdown 2024-04-26T02:05:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.264+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3385677768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833768140 msgr2=0x55a833778100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.264+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/3385677768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833768140 0x55a833778100 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x55a8337856b0 tx=0x7f91a0050690 comp rx=0 tx=0).stop 2024-04-26T02:05:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.265+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3385677768 shutdown_connections 2024-04-26T02:05:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.265+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/3385677768 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55a8337fa300 0x55a8337fc720 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.265+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/3385677768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a833768140 0x55a833778100 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:14.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.265+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/3385677768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a833772a00 0x55a8337e73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:14.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.265+0000 7f91bdfa9740 1 --2- 172.21.15.6:0/3385677768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a833767820 0x55a833777bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:14.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.265+0000 7f919e000640 0 client.17193 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:14.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.265+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3385677768 >> 172.21.15.6:0/3385677768 conn(0x55a833763d30 msgr2=0x55a83376ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:14.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.265+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3385677768 shutdown_connections 2024-04-26T02:05:14.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.265+0000 7f91bdfa9740 1 -- 172.21.15.6:0/3385677768 wait complete. 2024-04-26T02:05:14.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.265+0000 7f91bdfa9740 20 client.17193 trim_cache size 0 max 16384 2024-04-26T02:05:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:05:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:05:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:05:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:05:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.267+0000 7f91bb000640 1 -- 172.21.15.6:0/1709239915 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f91980a3e60 msgr2=0x7f91980a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.267+0000 7f91bb000640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f91980a3e60 0x7f91980a6320 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f91ac005fd0 tx=0x7f91ac009500 comp rx=0 tx=0).stop 2024-04-26T02:05:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.267+0000 7f91bb000640 1 -- 172.21.15.6:0/1709239915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b41021a0 msgr2=0x7f91b410e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.267+0000 7f91bb000640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b41021a0 0x7f91b410e280 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f91a400e9c0 tx=0x7f91a400ee80 comp rx=0 tx=0).stop 2024-04-26T02:05:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.268+0000 7f91bb000640 1 -- 172.21.15.6:0/1709239915 shutdown_connections 2024-04-26T02:05:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.268+0000 7f91bb000640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f91980a3e60 0x7f91980a6320 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.268+0000 7f91bb000640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91b41a3650 0x7f91b41a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.268+0000 7f91bb000640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91b4107270 0x7f91b410e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.268+0000 7f91bb000640 1 --2- 172.21.15.6:0/1709239915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91b41021a0 0x7f91b410e280 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.268+0000 7f91bb000640 1 -- 172.21.15.6:0/1709239915 >> 172.21.15.6:0/1709239915 conn(0x7f91b40fd9e0 msgr2=0x7f91b40fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:14.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.268+0000 7f91bb000640 1 -- 172.21.15.6:0/1709239915 shutdown_connections 2024-04-26T02:05:14.268 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:14.268+0000 7f91bb000640 1 -- 172.21.15.6:0/1709239915 wait complete. 2024-04-26T02:05:14.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:14 smithi188 ceph-mon[32751]: pgmap v613: 97 pgs: 97 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.8 MiB/s wr, 527 op/s 2024-04-26T02:05:14.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:14 smithi029 ceph-mon[32775]: pgmap v613: 97 pgs: 97 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.8 MiB/s wr, 527 op/s 2024-04-26T02:05:15.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:14 smithi006 ceph-mon[28419]: pgmap v613: 97 pgs: 97 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.8 MiB/s wr, 527 op/s 2024-04-26T02:05:15.132 INFO:tasks.workunit.client.0.smithi006.stdout: 10 104871 17.96 MB/sec execute 218 sec latency 197.041 ms 2024-04-26T02:05:16.133 INFO:tasks.workunit.client.0.smithi006.stdout: 10 105295 18.00 MB/sec execute 219 sec latency 334.456 ms 2024-04-26T02:05:16.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:16 smithi188 ceph-mon[32751]: pgmap v614: 97 pgs: 97 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.8 MiB/s wr, 527 op/s 2024-04-26T02:05:16.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:16 smithi029 ceph-mon[32775]: pgmap v614: 97 pgs: 97 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.8 MiB/s wr, 527 op/s 2024-04-26T02:05:17.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:16 smithi006 ceph-mon[28419]: pgmap v614: 97 pgs: 97 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.8 MiB/s wr, 527 op/s 2024-04-26T02:05:17.132 INFO:tasks.workunit.client.0.smithi006.stdout: 10 105763 18.02 MB/sec execute 220 sec latency 59.330 ms 2024-04-26T02:05:18.133 INFO:tasks.workunit.client.0.smithi006.stdout: 10 106252 18.06 MB/sec execute 221 sec latency 91.401 ms 2024-04-26T02:05:18.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:18 smithi188 ceph-mon[32751]: pgmap v615: 97 pgs: 97 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 14 MiB/s wr, 851 op/s 2024-04-26T02:05:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:18 smithi029 ceph-mon[32775]: pgmap v615: 97 pgs: 97 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 14 MiB/s wr, 851 op/s 2024-04-26T02:05:19.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:18 smithi006 ceph-mon[28419]: pgmap v615: 97 pgs: 97 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 14 MiB/s wr, 851 op/s 2024-04-26T02:05:19.133 INFO:tasks.workunit.client.0.smithi006.stdout: 10 106694 18.07 MB/sec execute 222 sec latency 136.102 ms 2024-04-26T02:05:20.133 INFO:tasks.workunit.client.0.smithi006.stdout: 10 107207 18.07 MB/sec execute 223 sec latency 117.271 ms 2024-04-26T02:05:20.871 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:20 smithi006 ceph-mon[28419]: pgmap v616: 97 pgs: 97 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 559 op/s 2024-04-26T02:05:20.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:20 smithi188 ceph-mon[32751]: pgmap v616: 97 pgs: 97 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 559 op/s 2024-04-26T02:05:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:20 smithi029 ceph-mon[32775]: pgmap v616: 97 pgs: 97 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 559 op/s 2024-04-26T02:05:21.133 INFO:tasks.workunit.client.0.smithi006.stdout: 10 107719 18.15 MB/sec execute 224 sec latency 247.406 ms 2024-04-26T02:05:22.134 INFO:tasks.workunit.client.0.smithi006.stdout: 10 108386 18.26 MB/sec execute 225 sec latency 114.039 ms 2024-04-26T02:05:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:22 smithi188 ceph-mon[32751]: pgmap v617: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 774 op/s 2024-04-26T02:05:22.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:22 smithi029 ceph-mon[32775]: pgmap v617: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 774 op/s 2024-04-26T02:05:23.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:22 smithi006 ceph-mon[28419]: pgmap v617: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 774 op/s 2024-04-26T02:05:23.134 INFO:tasks.workunit.client.0.smithi006.stdout: 10 108817 18.29 MB/sec execute 226 sec latency 172.747 ms 2024-04-26T02:05:24.134 INFO:tasks.workunit.client.0.smithi006.stdout: 10 109309 18.30 MB/sec execute 227 sec latency 75.200 ms 2024-04-26T02:05:24.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:24 smithi188 ceph-mon[32751]: pgmap v618: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 9.8 MiB/s wr, 537 op/s 2024-04-26T02:05:24.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:24 smithi029 ceph-mon[32775]: pgmap v618: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 9.8 MiB/s wr, 537 op/s 2024-04-26T02:05:25.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:24 smithi006 ceph-mon[28419]: pgmap v618: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 9.8 MiB/s wr, 537 op/s 2024-04-26T02:05:25.134 INFO:tasks.workunit.client.0.smithi006.stdout: 10 109906 18.36 MB/sec execute 228 sec latency 96.290 ms 2024-04-26T02:05:26.134 INFO:tasks.workunit.client.0.smithi006.stdout: 10 110230 18.35 MB/sec execute 229 sec latency 257.364 ms 2024-04-26T02:05:26.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:26 smithi188 ceph-mon[32751]: pgmap v619: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 9.8 MiB/s wr, 537 op/s 2024-04-26T02:05:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:26 smithi029 ceph-mon[32775]: pgmap v619: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 9.8 MiB/s wr, 537 op/s 2024-04-26T02:05:27.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:26 smithi006 ceph-mon[28419]: pgmap v619: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 9.8 MiB/s wr, 537 op/s 2024-04-26T02:05:27.134 INFO:tasks.workunit.client.0.smithi006.stdout: 10 110844 18.40 MB/sec execute 230 sec latency 130.228 ms 2024-04-26T02:05:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.434+0000 7f27da000640 1 -- 172.21.15.6:0/2483737706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27d4102970 msgr2=0x7f27d41069b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.434+0000 7f27da000640 1 --2- 172.21.15.6:0/2483737706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27d4102970 0x7f27d41069b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f27c0009900 tx=0x7f27c002a410 comp rx=0 tx=0).stop 2024-04-26T02:05:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.434+0000 7f27da000640 1 -- 172.21.15.6:0/2483737706 shutdown_connections 2024-04-26T02:05:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.434+0000 7f27da000640 1 --2- 172.21.15.6:0/2483737706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27d4102970 0x7f27d41069b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.434+0000 7f27da000640 1 --2- 172.21.15.6:0/2483737706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27d4107090 0x7f27d410ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.434+0000 7f27da000640 1 --2- 172.21.15.6:0/2483737706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27d4101fc0 0x7f27d41023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.434+0000 7f27da000640 1 -- 172.21.15.6:0/2483737706 >> 172.21.15.6:0/2483737706 conn(0x7f27d40fd880 msgr2=0x7f27d40ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.434+0000 7f27da000640 1 -- 172.21.15.6:0/2483737706 shutdown_connections 2024-04-26T02:05:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.434+0000 7f27da000640 1 -- 172.21.15.6:0/2483737706 wait complete. 2024-04-26T02:05:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.435+0000 7f27da000640 1 Processor -- start 2024-04-26T02:05:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.436+0000 7f27da000640 1 -- start start 2024-04-26T02:05:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.436+0000 7f27da000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27d4101fc0 0x7f27d41a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.436+0000 7f27da000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27d4107090 0x7f27d41a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.436+0000 7f27da000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27d419a5a0 0x7f27d419aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.436+0000 7f27da000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f27d4112020 con 0x7f27d4107090 2024-04-26T02:05:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.436+0000 7f27da000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f27d4111ea0 con 0x7f27d419a5a0 2024-04-26T02:05:27.436 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.436+0000 7f27da000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f27d41121a0 con 0x7f27d4101fc0 2024-04-26T02:05:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.437+0000 7f27d8c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27d4101fc0 0x7f27d41a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.437+0000 7f27d9600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27d419a5a0 0x7f27d419aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.437+0000 7f27d8c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27d4101fc0 0x7f27d41a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44362/0 (socket says 172.21.15.6:44362) 2024-04-26T02:05:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.437+0000 7f27d8c00640 1 -- 172.21.15.6:0/646060708 learned_addr learned my addr 172.21.15.6:0/646060708 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.437+0000 7f27d3e00640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27d4107090 0x7f27d41a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.437+0000 7f27d8c00640 1 -- 172.21.15.6:0/646060708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27d419a5a0 msgr2=0x7f27d419aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.437+0000 7f27d8c00640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27d419a5a0 0x7f27d419aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.437+0000 7f27d8c00640 1 -- 172.21.15.6:0/646060708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27d4107090 msgr2=0x7f27d41a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.437+0000 7f27d8c00640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27d4107090 0x7f27d41a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.437+0000 7f27d8c00640 1 -- 172.21.15.6:0/646060708 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d419b0c0 con 0x7f27d4101fc0 2024-04-26T02:05:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.438+0000 7f27d8c00640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27d4101fc0 0x7f27d41a03f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f27c800ea60 tx=0x7f27c800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.438+0000 7f27d1600640 1 -- 172.21.15.6:0/646060708 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c800cde0 con 0x7f27d4101fc0 2024-04-26T02:05:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.438+0000 7f27d1600640 1 -- 172.21.15.6:0/646060708 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c80035f0 con 0x7f27d4101fc0 2024-04-26T02:05:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.438+0000 7f27d1600640 1 -- 172.21.15.6:0/646060708 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8010690 con 0x7f27d4101fc0 2024-04-26T02:05:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.438+0000 7f27da000640 1 -- 172.21.15.6:0/646060708 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27d419b3b0 con 0x7f27d4101fc0 2024-04-26T02:05:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.438+0000 7f27da000640 1 -- 172.21.15.6:0/646060708 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27d41a7650 con 0x7f27d4101fc0 2024-04-26T02:05:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.442+0000 7f27de386740 1 Processor -- start 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.442+0000 7f27de386740 1 -- start start 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.443+0000 7f27d1600640 1 -- 172.21.15.6:0/646060708 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f27c80037a0 con 0x7f27d4101fc0 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.443+0000 7f27de386740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997c9b0 0x55af2997cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.443+0000 7f27de386740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997d2d0 0x55af29987650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.443+0000 7f27de386740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af29987b90 0x55af29989f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.443+0000 7f27de386740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55af298e2900 con 0x55af2997c9b0 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.443+0000 7f27de386740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55af29888ef0 con 0x55af2997d2d0 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.443+0000 7f27de386740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55af298de8c0 con 0x55af29987b90 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.443+0000 7f27d8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997d2d0 0x55af29987650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.443+0000 7f27d8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997d2d0 0x55af29987650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47274/0 (socket says 172.21.15.6:47274) 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.443+0000 7f27d8c00640 1 -- 172.21.15.6:0/2725499086 learned_addr learned my addr 172.21.15.6:0/2725499086 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.443+0000 7f27d9600640 1 --2- 172.21.15.6:0/2725499086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997c9b0 0x55af2997cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.443+0000 7f27d3e00640 1 --2- 172.21.15.6:0/2725499086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af29987b90 0x55af29989f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.444+0000 7f27d8c00640 1 -- 172.21.15.6:0/2725499086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af29987b90 msgr2=0x55af29989f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.444+0000 7f27d8c00640 1 --2- 172.21.15.6:0/2725499086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af29987b90 0x55af29989f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.444+0000 7f27d8c00640 1 -- 172.21.15.6:0/2725499086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997c9b0 msgr2=0x55af2997cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.444+0000 7f27d8c00640 1 --2- 172.21.15.6:0/2725499086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997c9b0 0x55af2997cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.444+0000 7f27d8c00640 1 -- 172.21.15.6:0/2725499086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af295f5210 con 0x55af2997d2d0 2024-04-26T02:05:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.445+0000 7f27d8c00640 1 --2- 172.21.15.6:0/2725499086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997d2d0 0x55af29987650 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f27c80ce1f0 tx=0x7f27c80d96c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=433f54f0d8c6ea98 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.445+0000 7f27d1600640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27bc0a3d80 0x7f27bc0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.445+0000 7f27c5c00640 1 -- 172.21.15.6:0/2725499086 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c80e0070 con 0x55af2997d2d0 2024-04-26T02:05:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.445+0000 7f27d1600640 1 -- 172.21.15.6:0/646060708 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f27c80d0b80 con 0x7f27d4101fc0 2024-04-26T02:05:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.445+0000 7f27c5c00640 1 -- 172.21.15.6:0/2725499086 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c809d1d0 con 0x55af2997d2d0 2024-04-26T02:05:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.445+0000 7f27c5c00640 1 -- 172.21.15.6:0/2725499086 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8098d70 con 0x55af2997d2d0 2024-04-26T02:05:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.445+0000 7f27d9600640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27bc0a3d80 0x7f27bc0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.447+0000 7f27de386740 1 -- 172.21.15.6:0/2725499086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997d2d0 msgr2=0x55af29987650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.447+0000 7f27d9600640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27bc0a3d80 0x7f27bc0a6240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f27cc005fd0 tx=0x7f27cc009500 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.447+0000 7f27de386740 1 --2- 172.21.15.6:0/2725499086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997d2d0 0x55af29987650 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f27c80ce1f0 tx=0x7f27c80d96c0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.447+0000 7f27de386740 1 -- 172.21.15.6:0/2725499086 shutdown_connections 2024-04-26T02:05:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.447+0000 7f27de386740 1 --2- 172.21.15.6:0/2725499086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af29987b90 0x55af29989f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.447+0000 7f27de386740 1 --2- 172.21.15.6:0/2725499086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997d2d0 0x55af29987650 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.447+0000 7f27de386740 1 --2- 172.21.15.6:0/2725499086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997c9b0 0x55af2997cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.447+0000 7f27de386740 1 -- 172.21.15.6:0/2725499086 >> 172.21.15.6:0/2725499086 conn(0x55af29978ec0 msgr2=0x55af299792a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.447+0000 7f27de386740 1 -- 172.21.15.6:0/2725499086 shutdown_connections 2024-04-26T02:05:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.447+0000 7f27de386740 1 -- 172.21.15.6:0/2725499086 wait complete. 2024-04-26T02:05:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.448+0000 7f27de386740 1 -- start start 2024-04-26T02:05:27.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.448+0000 7f27de386740 1 Processor -- start 2024-04-26T02:05:27.448 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27de386740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 0x55af29984b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27de386740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997d2d0 0x55af29985090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27de386740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af29987b90 0x55af299855d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27de386740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55af2999b7d0 con 0x55af2997d2d0 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27de386740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55af2989c310 con 0x55af2997c9b0 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27de386740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55af2999b950 con 0x55af29987b90 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27de386740 10 client.0 _ll_register_callbacks cb 0x55af2986acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27d8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 0x55af29984b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27d9600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af29987b90 0x55af299855d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27d8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 0x55af29984b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47290/0 (socket says 172.21.15.6:47290) 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27d8c00640 1 -- 172.21.15.6:0/3094146118 learned_addr learned my addr 172.21.15.6:0/3094146118 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27d3e00640 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997d2d0 0x55af29985090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27d8c00640 1 -- 172.21.15.6:0/3094146118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af29987b90 msgr2=0x55af299855d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27d8c00640 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af29987b90 0x55af299855d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27d8c00640 1 -- 172.21.15.6:0/3094146118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997d2d0 msgr2=0x55af29985090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27d8c00640 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997d2d0 0x55af29985090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27d8c00640 1 -- 172.21.15.6:0/3094146118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af296c1620 con 0x55af2997c9b0 2024-04-26T02:05:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.449+0000 7f27d9600640 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af29987b90 0x55af299855d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:05:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.450+0000 7f27d8c00640 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 0x55af29984b50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f27c801f000 tx=0x7f27c8005ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b156350c60185d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.450+0000 7f27ba000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.450+0000 7f27ba000640 1 -- 172.21.15.6:0/3094146118 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c80e0070 con 0x55af2997c9b0 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.450+0000 7f27ba000640 1 -- 172.21.15.6:0/3094146118 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f27c801da90 con 0x55af2997c9b0 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.450+0000 7f27de386740 1 -- 172.21.15.6:0/3094146118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c809d1d0 con 0x55af2997c9b0 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.450+0000 7f27ba000640 1 -- 172.21.15.6:0/3094146118 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f27c805d280 con 0x55af2997c9b0 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.450+0000 7f27de386740 10 client.27407 fetch_fsmap learned FSMap version 55 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.450+0000 7f27de386740 1 -- 172.21.15.6:0/3094146118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55af299a7a60 con 0x55af2997c9b0 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.451+0000 7f27ba000640 1 -- 172.21.15.6:0/3094146118 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c809d1d0 con 0x55af2997c9b0 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.451+0000 7f27ba000640 1 -- 172.21.15.6:0/3094146118 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7f27c80da040 con 0x55af2997c9b0 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.451+0000 7f27de386740 10 client.27407 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.451+0000 7f27de386740 10 client.27407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.451+0000 7f27de386740 20 client.27407 populate_metadata read hostname 'smithi006' 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.451+0000 7f27de386740 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af299a8db0 0x55af299ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.451+0000 7f27de386740 4 client.27407 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.451+0000 7f27de386740 1 -- 172.21.15.6:0/3094146118 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af298f68b0 con 0x55af299a8db0 2024-04-26T02:05:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.451+0000 7f27d3e00640 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af299a8db0 0x55af299ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.452+0000 7f27d3e00640 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af299a8db0 0x55af299ab1a0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82f36d4feb4f9958 server_cookie=b94b97473e486b5 in_seq=0 out_seq=0 2024-04-26T02:05:27.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.452+0000 7f27ba000640 10 client.27407 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:27.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.455+0000 7f27ba000640 1 -- 172.21.15.6:0/3094146118 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55af298f68b0 con 0x55af299a8db0 2024-04-26T02:05:27.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.455+0000 7f27ba000640 10 client.27407 handle_command_reply: tid=0 2024-04-26T02:05:27.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.455+0000 7f27de386740 1 client.27407 shutdown 2024-04-26T02:05:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.455+0000 7f27de386740 1 -- 172.21.15.6:0/3094146118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 msgr2=0x55af29984b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.455+0000 7f27de386740 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 0x55af29984b50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f27c801f000 tx=0x7f27c8005ab0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.455+0000 7f27de386740 1 -- 172.21.15.6:0/3094146118 shutdown_connections 2024-04-26T02:05:27.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.456+0000 7f27de386740 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af299a8db0 0x55af299ab1a0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.456+0000 7f27de386740 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997d2d0 0x55af29985090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.456+0000 7f27de386740 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af29987b90 0x55af299855d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.456+0000 7f27de386740 1 --2- 172.21.15.6:0/3094146118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 0x55af29984b50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.456+0000 7f27de386740 1 -- 172.21.15.6:0/3094146118 >> 172.21.15.6:0/3094146118 conn(0x55af29978ec0 msgr2=0x55af2997ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:27.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.456+0000 7f27ba000640 0 client.27407 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:27.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.456+0000 7f27de386740 1 -- 172.21.15.6:0/3094146118 shutdown_connections 2024-04-26T02:05:27.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.456+0000 7f27de386740 1 -- 172.21.15.6:0/3094146118 wait complete. 2024-04-26T02:05:27.456 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.456+0000 7f27de386740 20 client.27407 trim_cache size 0 max 16384 2024-04-26T02:05:27.477 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.477+0000 7f27de386740 1 Processor -- start 2024-04-26T02:05:27.477 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.477+0000 7f27de386740 1 -- start start 2024-04-26T02:05:27.477 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.477+0000 7f27de386740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 0x55af2997f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.477 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.477+0000 7f27de386740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997d2d0 0x55af299853f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.477 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.477+0000 7f27de386740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af29987b90 0x55af29977790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.477 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.477+0000 7f27de386740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f27d41aa000 con 0x55af29987b90 2024-04-26T02:05:27.477 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.477+0000 7f27de386740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f27d41a78e0 con 0x55af2997c9b0 2024-04-26T02:05:27.477 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.477+0000 7f27de386740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f27d419bad0 con 0x55af2997d2d0 2024-04-26T02:05:27.477 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997d2d0 0x55af299853f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.477 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 0x55af2997f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.478 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997d2d0 0x55af299853f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44398/0 (socket says 172.21.15.6:44398) 2024-04-26T02:05:27.478 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 0x55af2997f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:47300/0 (socket says 172.21.15.6:47300) 2024-04-26T02:05:27.478 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d3e00640 1 -- 172.21.15.6:0/307273412 learned_addr learned my addr 172.21.15.6:0/307273412 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:27.478 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d9600640 1 --2- 172.21.15.6:0/307273412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af29987b90 0x55af29977790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d3e00640 1 -- 172.21.15.6:0/307273412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 msgr2=0x55af2997f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d3e00640 1 --2- 172.21.15.6:0/307273412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 0x55af2997f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d3e00640 1 -- 172.21.15.6:0/307273412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af29987b90 msgr2=0x55af29977790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d3e00640 1 --2- 172.21.15.6:0/307273412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af29987b90 0x55af29977790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d3e00640 1 -- 172.21.15.6:0/307273412 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af2999b830 con 0x55af2997d2d0 2024-04-26T02:05:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d8c00640 1 --2- 172.21.15.6:0/307273412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 0x55af2997f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:05:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.478+0000 7f27d3e00640 1 --2- 172.21.15.6:0/307273412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997d2d0 0x55af299853f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f27c002a9d0 tx=0x7f27c0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19e459fdd60f59a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.479+0000 7f27c5200640 1 -- 172.21.15.6:0/307273412 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0035040 con 0x55af2997d2d0 2024-04-26T02:05:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.479+0000 7f27c5200640 1 -- 172.21.15.6:0/307273412 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c004a070 con 0x55af2997d2d0 2024-04-26T02:05:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.479+0000 7f27c5200640 1 -- 172.21.15.6:0/307273412 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c00046c0 con 0x55af2997d2d0 2024-04-26T02:05:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.479+0000 7f27de386740 1 -- 172.21.15.6:0/307273412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997d2d0 msgr2=0x55af299853f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.479+0000 7f27de386740 1 --2- 172.21.15.6:0/307273412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997d2d0 0x55af299853f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f27c002a9d0 tx=0x7f27c0000bc0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.480+0000 7f27de386740 1 -- 172.21.15.6:0/307273412 shutdown_connections 2024-04-26T02:05:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.480+0000 7f27de386740 1 --2- 172.21.15.6:0/307273412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af29987b90 0x55af29977790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.480+0000 7f27de386740 1 --2- 172.21.15.6:0/307273412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997d2d0 0x55af299853f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.480+0000 7f27de386740 1 --2- 172.21.15.6:0/307273412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af2997c9b0 0x55af2997f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.480+0000 7f27de386740 1 -- 172.21.15.6:0/307273412 >> 172.21.15.6:0/307273412 conn(0x55af29978ec0 msgr2=0x55af29986c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.480+0000 7f27de386740 1 -- 172.21.15.6:0/307273412 shutdown_connections 2024-04-26T02:05:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.480+0000 7f27de386740 1 -- 172.21.15.6:0/307273412 wait complete. 2024-04-26T02:05:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.480+0000 7f27de386740 1 -- start start 2024-04-26T02:05:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.480+0000 7f27de386740 1 Processor -- start 2024-04-26T02:05:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.481+0000 7f27de386740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997c9b0 0x55af299ad490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.481+0000 7f27de386740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997d2d0 0x55af299ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.481+0000 7f27de386740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af29987b90 0x55af29a06f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.481+0000 7f27de386740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f27d4072650 con 0x55af2997d2d0 2024-04-26T02:05:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.481+0000 7f27de386740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f27d4073aa0 con 0x55af29987b90 2024-04-26T02:05:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.481+0000 7f27de386740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f27d4072d00 con 0x55af2997c9b0 2024-04-26T02:05:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.481+0000 7f27de386740 10 client.0 _ll_register_callbacks cb 0x7f27c80d4a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.481+0000 7f27d8c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997c9b0 0x55af299ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.481+0000 7f27d8c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997c9b0 0x55af299ad490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44408/0 (socket says 172.21.15.6:44408) 2024-04-26T02:05:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.481+0000 7f27d8c00640 1 -- 172.21.15.6:0/3695007334 learned_addr learned my addr 172.21.15.6:0/3695007334 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.481+0000 7f27d9600640 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af29987b90 0x55af29a06f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.481+0000 7f27d3e00640 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997d2d0 0x55af299ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.482+0000 7f27d8c00640 1 -- 172.21.15.6:0/3695007334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af29987b90 msgr2=0x55af29a06f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.482+0000 7f27d8c00640 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af29987b90 0x55af29a06f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.482+0000 7f27d8c00640 1 -- 172.21.15.6:0/3695007334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997d2d0 msgr2=0x55af299ad9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.482+0000 7f27d8c00640 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997d2d0 0x55af299ad9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.482+0000 7f27d8c00640 1 -- 172.21.15.6:0/3695007334 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af29a075a0 con 0x55af2997c9b0 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.482+0000 7f27d8c00640 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997c9b0 0x55af299ad490 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f27c80a0ea0 tx=0x7f27c80d96c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=748f545c899b780b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.482+0000 7f27ba000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.482+0000 7f27ba000640 1 -- 172.21.15.6:0/3695007334 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c80e0030 con 0x55af2997c9b0 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.482+0000 7f27de386740 1 -- 172.21.15.6:0/3695007334 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c004a070 con 0x55af2997c9b0 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.482+0000 7f27ba000640 1 -- 172.21.15.6:0/3695007334 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7f27c8099140 con 0x55af2997c9b0 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.482+0000 7f27ba000640 1 -- 172.21.15.6:0/3695007334 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f27c805bb10 con 0x55af2997c9b0 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.483+0000 7f27de386740 10 client.27472 fetch_fsmap learned FSMap version 55 2024-04-26T02:05:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.483+0000 7f27de386740 1 -- 172.21.15.6:0/3695007334 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55af29a0e920 con 0x55af2997c9b0 2024-04-26T02:05:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.483+0000 7f27ba000640 1 -- 172.21.15.6:0/3695007334 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c004a070 con 0x55af2997c9b0 2024-04-26T02:05:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.483+0000 7f27ba000640 1 -- 172.21.15.6:0/3695007334 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7f27c80d4cc0 con 0x55af2997c9b0 2024-04-26T02:05:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.483+0000 7f27de386740 10 client.27472 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:05:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.483+0000 7f27de386740 10 client.27472 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:05:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.483+0000 7f27de386740 20 client.27472 populate_metadata read hostname 'smithi006' 2024-04-26T02:05:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.483+0000 7f27de386740 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af29a0eea0 0x55af29a11290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.483+0000 7f27de386740 4 client.27472 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:05:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.483+0000 7f27de386740 1 -- 172.21.15.6:0/3695007334 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55af298f0320 con 0x55af29a0eea0 2024-04-26T02:05:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.484+0000 7f27d3e00640 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af29a0eea0 0x55af29a11290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.484+0000 7f27d3e00640 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af29a0eea0 0x55af29a11290 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=378531a734e9cd2b server_cookie=f90ef981a8057730 in_seq=0 out_seq=0 2024-04-26T02:05:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.484+0000 7f27ba000640 10 client.27472 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:27.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.752+0000 7f27ba000640 1 -- 172.21.15.6:0/3695007334 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55af298f0320 con 0x55af29a0eea0 2024-04-26T02:05:27.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.752+0000 7f27ba000640 10 client.27472 handle_command_reply: tid=0 2024-04-26T02:05:27.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.752+0000 7f27de386740 1 client.27472 shutdown 2024-04-26T02:05:27.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.752+0000 7f27de386740 1 -- 172.21.15.6:0/3695007334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997c9b0 msgr2=0x55af299ad490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:27.752 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.752+0000 7f27de386740 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997c9b0 0x55af299ad490 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f27c80a0ea0 tx=0x7f27c80d96c0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.753+0000 7f27de386740 1 -- 172.21.15.6:0/3695007334 shutdown_connections 2024-04-26T02:05:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.753+0000 7f27de386740 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55af29a0eea0 0x55af29a11290 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.753+0000 7f27de386740 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2997d2d0 0x55af299ad9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.753+0000 7f27de386740 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55af29987b90 0x55af29a06f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.753+0000 7f27de386740 1 --2- 172.21.15.6:0/3695007334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af2997c9b0 0x55af299ad490 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.753+0000 7f27de386740 1 -- 172.21.15.6:0/3695007334 >> 172.21.15.6:0/3695007334 conn(0x55af29978ec0 msgr2=0x55af29986380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.753+0000 7f27ba000640 0 client.27472 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.753+0000 7f27de386740 1 -- 172.21.15.6:0/3695007334 shutdown_connections 2024-04-26T02:05:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.753+0000 7f27de386740 1 -- 172.21.15.6:0/3695007334 wait complete. 2024-04-26T02:05:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.753+0000 7f27de386740 20 client.27472 trim_cache size 0 max 16384 2024-04-26T02:05:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:05:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:05:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:05:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:05:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.754+0000 7f27da000640 1 -- 172.21.15.6:0/646060708 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27bc0a3d80 msgr2=0x7f27bc0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.754+0000 7f27da000640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27bc0a3d80 0x7f27bc0a6240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f27cc005fd0 tx=0x7f27cc009500 comp rx=0 tx=0).stop 2024-04-26T02:05:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.754+0000 7f27da000640 1 -- 172.21.15.6:0/646060708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27d4101fc0 msgr2=0x7f27d41a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.754+0000 7f27da000640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27d4101fc0 0x7f27d41a03f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f27c800ea60 tx=0x7f27c800ef20 comp rx=0 tx=0).stop 2024-04-26T02:05:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.754+0000 7f27da000640 1 -- 172.21.15.6:0/646060708 shutdown_connections 2024-04-26T02:05:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.754+0000 7f27da000640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f27bc0a3d80 0x7f27bc0a6240 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.754+0000 7f27da000640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27d4107090 0x7f27d41a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.754+0000 7f27da000640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27d419a5a0 0x7f27d419aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.754+0000 7f27da000640 1 --2- 172.21.15.6:0/646060708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27d4101fc0 0x7f27d41a03f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.754+0000 7f27da000640 1 -- 172.21.15.6:0/646060708 >> 172.21.15.6:0/646060708 conn(0x7f27d40fd880 msgr2=0x7f27d40fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.756+0000 7f27da000640 1 -- 172.21.15.6:0/646060708 shutdown_connections 2024-04-26T02:05:27.756 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:27.756+0000 7f27da000640 1 -- 172.21.15.6:0/646060708 wait complete. 2024-04-26T02:05:28.134 INFO:tasks.workunit.client.0.smithi006.stdout: 10 111388 18.47 MB/sec execute 231 sec latency 189.222 ms 2024-04-26T02:05:28.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:28 smithi188 ceph-mon[32751]: pgmap v620: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 758 op/s 2024-04-26T02:05:28.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:28 smithi029 ceph-mon[32775]: pgmap v620: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 758 op/s 2024-04-26T02:05:29.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:28 smithi006 ceph-mon[28419]: pgmap v620: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 758 op/s 2024-04-26T02:05:29.135 INFO:tasks.workunit.client.0.smithi006.stdout: 10 111933 18.53 MB/sec execute 232 sec latency 100.312 ms 2024-04-26T02:05:30.135 INFO:tasks.workunit.client.0.smithi006.stdout: 10 112458 18.55 MB/sec execute 233 sec latency 89.121 ms 2024-04-26T02:05:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:30 smithi188 ceph-mon[32751]: pgmap v621: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.8 MiB/s wr, 435 op/s 2024-04-26T02:05:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:30 smithi029 ceph-mon[32775]: pgmap v621: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.8 MiB/s wr, 435 op/s 2024-04-26T02:05:31.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:30 smithi006 ceph-mon[28419]: pgmap v621: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.8 MiB/s wr, 435 op/s 2024-04-26T02:05:31.135 INFO:tasks.workunit.client.0.smithi006.stdout: 10 112857 18.54 MB/sec execute 234 sec latency 337.952 ms 2024-04-26T02:05:32.135 INFO:tasks.workunit.client.0.smithi006.stdout: 10 113426 18.61 MB/sec execute 235 sec latency 114.749 ms 2024-04-26T02:05:32.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:32 smithi188 ceph-mon[32751]: pgmap v622: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 739 op/s 2024-04-26T02:05:32.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:32 smithi029 ceph-mon[32775]: pgmap v622: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 739 op/s 2024-04-26T02:05:33.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:32 smithi006 ceph-mon[28419]: pgmap v622: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 739 op/s 2024-04-26T02:05:33.135 INFO:tasks.workunit.client.0.smithi006.stdout: 10 114023 18.68 MB/sec execute 236 sec latency 79.015 ms 2024-04-26T02:05:34.135 INFO:tasks.workunit.client.0.smithi006.stdout: 10 114623 18.73 MB/sec execute 237 sec latency 84.126 ms 2024-04-26T02:05:34.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:34 smithi188 ceph-mon[32751]: pgmap v623: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 8.5 MiB/s wr, 524 op/s 2024-04-26T02:05:34.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:34 smithi029 ceph-mon[32775]: pgmap v623: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 8.5 MiB/s wr, 524 op/s 2024-04-26T02:05:35.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:34 smithi006 ceph-mon[28419]: pgmap v623: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 8.5 MiB/s wr, 524 op/s 2024-04-26T02:05:35.135 INFO:tasks.workunit.client.0.smithi006.stdout: 10 115196 18.79 MB/sec execute 238 sec latency 71.549 ms 2024-04-26T02:05:36.135 INFO:tasks.workunit.client.0.smithi006.stdout: 10 115597 18.82 MB/sec execute 239 sec latency 205.683 ms 2024-04-26T02:05:36.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:36 smithi188 ceph-mon[32751]: pgmap v624: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.5 MiB/s wr, 524 op/s 2024-04-26T02:05:36.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:36 smithi029 ceph-mon[32775]: pgmap v624: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.5 MiB/s wr, 524 op/s 2024-04-26T02:05:37.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:36 smithi006 ceph-mon[28419]: pgmap v624: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.5 MiB/s wr, 524 op/s 2024-04-26T02:05:37.135 INFO:tasks.workunit.client.0.smithi006.stdout: 10 115987 18.80 MB/sec execute 240 sec latency 277.369 ms 2024-04-26T02:05:38.135 INFO:tasks.workunit.client.0.smithi006.stdout: 10 116583 18.84 MB/sec execute 241 sec latency 122.138 ms 2024-04-26T02:05:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:38 smithi188 ceph-mon[32751]: pgmap v625: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 729 op/s 2024-04-26T02:05:38.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:38 smithi029 ceph-mon[32775]: pgmap v625: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 729 op/s 2024-04-26T02:05:39.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:38 smithi006 ceph-mon[28419]: pgmap v625: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 729 op/s 2024-04-26T02:05:39.135 INFO:tasks.workunit.client.0.smithi006.stdout: 10 117084 18.88 MB/sec execute 242 sec latency 102.784 ms 2024-04-26T02:05:40.135 INFO:tasks.workunit.client.0.smithi006.stdout: 10 117686 18.95 MB/sec execute 243 sec latency 174.309 ms 2024-04-26T02:05:40.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.914+0000 7fda4fa00640 1 -- 172.21.15.6:0/719845504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda48100d00 msgr2=0x7fda4810e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.914+0000 7fda4fa00640 1 --2- 172.21.15.6:0/719845504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda48100d00 0x7fda4810e3e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fda34009900 tx=0x7fda3402a480 comp rx=0 tx=0).stop 2024-04-26T02:05:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.914+0000 7fda4fa00640 1 -- 172.21.15.6:0/719845504 shutdown_connections 2024-04-26T02:05:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.914+0000 7fda4fa00640 1 --2- 172.21.15.6:0/719845504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda4810e920 0x7fda48110d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.914+0000 7fda4fa00640 1 --2- 172.21.15.6:0/719845504 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda48100d00 0x7fda4810e3e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.914+0000 7fda4fa00640 1 --2- 172.21.15.6:0/719845504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda481003e0 0x7fda481007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.914+0000 7fda4fa00640 1 -- 172.21.15.6:0/719845504 >> 172.21.15.6:0/719845504 conn(0x7fda48075d60 msgr2=0x7fda48076170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.914+0000 7fda4fa00640 1 -- 172.21.15.6:0/719845504 shutdown_connections 2024-04-26T02:05:40.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.914+0000 7fda4fa00640 1 -- 172.21.15.6:0/719845504 wait complete. 2024-04-26T02:05:40.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.915+0000 7fda4fa00640 1 Processor -- start 2024-04-26T02:05:40.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.915+0000 7fda4fa00640 1 -- start start 2024-04-26T02:05:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.916+0000 7fda4fa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda481003e0 0x7fda481a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.916+0000 7fda4fa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda48100d00 0x7fda481a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.916+0000 7fda4fa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda4810e920 0x7fda4819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.916+0000 7fda4fa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fda481143d0 con 0x7fda4810e920 2024-04-26T02:05:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.916+0000 7fda4fa00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fda48114250 con 0x7fda48100d00 2024-04-26T02:05:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.916+0000 7fda4fa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fda48114550 con 0x7fda481003e0 2024-04-26T02:05:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.916+0000 7fda4d000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda481003e0 0x7fda481a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.916+0000 7fda4d000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda481003e0 0x7fda481a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44500/0 (socket says 172.21.15.6:44500) 2024-04-26T02:05:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.916+0000 7fda4d000640 1 -- 172.21.15.6:0/3888617676 learned_addr learned my addr 172.21.15.6:0/3888617676 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.916+0000 7fda4da00640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda4810e920 0x7fda4819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.917+0000 7fda4d000640 1 -- 172.21.15.6:0/3888617676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda48100d00 msgr2=0x7fda481a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.917+0000 7fda4d000640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda48100d00 0x7fda481a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.917+0000 7fda4d000640 1 -- 172.21.15.6:0/3888617676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda4810e920 msgr2=0x7fda4819a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.917+0000 7fda4d000640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda4810e920 0x7fda4819a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.917+0000 7fda4d000640 1 -- 172.21.15.6:0/3888617676 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4819afa0 con 0x7fda481003e0 2024-04-26T02:05:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.917+0000 7fda4d000640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda481003e0 0x7fda481a0660 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00ea60 tx=0x7fda3c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.918+0000 7fda45600640 1 -- 172.21.15.6:0/3888617676 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c00cde0 con 0x7fda481003e0 2024-04-26T02:05:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.918+0000 7fda45600640 1 -- 172.21.15.6:0/3888617676 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda3c0035f0 con 0x7fda481003e0 2024-04-26T02:05:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.918+0000 7fda45600640 1 -- 172.21.15.6:0/3888617676 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c010690 con 0x7fda481003e0 2024-04-26T02:05:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.918+0000 7fda4fa00640 1 -- 172.21.15.6:0/3888617676 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda4819b290 con 0x7fda481003e0 2024-04-26T02:05:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.918+0000 7fda4fa00640 1 -- 172.21.15.6:0/3888617676 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda481a74b0 con 0x7fda481003e0 2024-04-26T02:05:40.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.923+0000 7fda52879740 1 Processor -- start 2024-04-26T02:05:40.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.923+0000 7fda52879740 1 -- start start 2024-04-26T02:05:40.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.923+0000 7fda45600640 1 -- 172.21.15.6:0/3888617676 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fda3c0037a0 con 0x7fda481003e0 2024-04-26T02:05:40.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.923+0000 7fda52879740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b030 0x561bb1c3b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.923+0000 7fda52879740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c3b950 0x561bb1c45cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.923+0000 7fda52879740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c46210 0x561bb1c48600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.923+0000 7fda52879740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x561bb1b98900 con 0x561bb1c3b950 2024-04-26T02:05:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.923+0000 7fda52879740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561bb1b3eef0 con 0x561bb1c46210 2024-04-26T02:05:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.923+0000 7fda52879740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561bb1b948c0 con 0x561bb1c3b030 2024-04-26T02:05:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.924+0000 7fda4da00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b030 0x561bb1c3b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.924+0000 7fda4da00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b030 0x561bb1c3b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44510/0 (socket says 172.21.15.6:44510) 2024-04-26T02:05:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.924+0000 7fda4da00640 1 -- 172.21.15.6:0/2514121677 learned_addr learned my addr 172.21.15.6:0/2514121677 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.924+0000 7fda4da00640 1 -- 172.21.15.6:0/2514121677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c46210 msgr2=0x561bb1c48600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T02:05:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.924+0000 7fda47e00640 1 --2- 172.21.15.6:0/2514121677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c46210 0x561bb1c48600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.924 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.924+0000 7fda4da00640 1 --2- 172.21.15.6:0/2514121677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c46210 0x561bb1c48600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.925 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.924+0000 7fda4da00640 1 -- 172.21.15.6:0/2514121677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c3b950 msgr2=0x561bb1c45cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.925 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.924+0000 7fda4da00640 1 --2- 172.21.15.6:0/2514121677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c3b950 0x561bb1c45cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.925 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.924+0000 7fda4da00640 1 -- 172.21.15.6:0/2514121677 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bb18ab210 con 0x561bb1c3b030 2024-04-26T02:05:40.925 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.924+0000 7fda47e00640 1 --2- 172.21.15.6:0/2514121677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c46210 0x561bb1c48600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:05:40.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.925+0000 7fda4da00640 1 --2- 172.21.15.6:0/2514121677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b030 0x561bb1c3b410 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fda40009f50 tx=0x7fda400118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6bcbcbc6725cc9f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:40.926 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.925+0000 7fda45600640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fda300a3d90 0x7fda300a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.925+0000 7fda3a000640 1 -- 172.21.15.6:0/2514121677 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40004d20 con 0x561bb1c3b030 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.925+0000 7fda3a000640 1 -- 172.21.15.6:0/2514121677 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda40004050 con 0x561bb1c3b030 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.925+0000 7fda3a000640 1 -- 172.21.15.6:0/2514121677 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40005280 con 0x561bb1c3b030 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.925+0000 7fda45600640 1 -- 172.21.15.6:0/3888617676 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7fda3c0d0be0 con 0x7fda481003e0 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.926+0000 7fda4d000640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fda300a3d90 0x7fda300a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.926+0000 7fda52879740 1 -- 172.21.15.6:0/2514121677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b030 msgr2=0x561bb1c3b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.926+0000 7fda52879740 1 --2- 172.21.15.6:0/2514121677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b030 0x561bb1c3b410 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fda40009f50 tx=0x7fda400118d0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.927+0000 7fda52879740 1 -- 172.21.15.6:0/2514121677 shutdown_connections 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.927+0000 7fda52879740 1 --2- 172.21.15.6:0/2514121677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c3b950 0x561bb1c45cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.927+0000 7fda52879740 1 --2- 172.21.15.6:0/2514121677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c46210 0x561bb1c48600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.927+0000 7fda52879740 1 --2- 172.21.15.6:0/2514121677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b030 0x561bb1c3b410 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.927+0000 7fda52879740 1 -- 172.21.15.6:0/2514121677 >> 172.21.15.6:0/2514121677 conn(0x561bb1c37540 msgr2=0x561bb1c37920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.927+0000 7fda4d000640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fda300a3d90 0x7fda300a6250 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x561bb1c3c5d0 tx=0x7fda3c0094d0 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:40.927 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.927+0000 7fda52879740 1 -- 172.21.15.6:0/2514121677 shutdown_connections 2024-04-26T02:05:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.927+0000 7fda52879740 1 -- 172.21.15.6:0/2514121677 wait complete. 2024-04-26T02:05:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.928+0000 7fda52879740 1 -- start start 2024-04-26T02:05:40.928 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.928+0000 7fda52879740 1 Processor -- start 2024-04-26T02:05:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.929+0000 7fda52879740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b030 0x561bb1c431b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.929+0000 7fda52879740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 0x561bb1c436f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.929+0000 7fda52879740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c46210 0x561bb1c43c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.929+0000 7fda52879740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x561bb1c48e40 con 0x561bb1c46210 2024-04-26T02:05:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.929+0000 7fda52879740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561bb1b52310 con 0x561bb1c3b950 2024-04-26T02:05:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.929+0000 7fda52879740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561bb1c48fc0 con 0x561bb1c3b030 2024-04-26T02:05:40.929 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.930+0000 7fda52879740 10 client.0 _ll_register_callbacks cb 0x561bb1b20cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.931+0000 7fda47e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 0x561bb1c436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.931+0000 7fda47e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 0x561bb1c436f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59024/0 (socket says 172.21.15.6:59024) 2024-04-26T02:05:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.931+0000 7fda47e00640 1 -- 172.21.15.6:0/544323380 learned_addr learned my addr 172.21.15.6:0/544323380 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.931+0000 7fda4da00640 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c46210 0x561bb1c43c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.931+0000 7fda4da00640 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b030 0x561bb1c431b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.932+0000 7fda47e00640 1 -- 172.21.15.6:0/544323380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b030 msgr2=0x561bb1c431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.932+0000 7fda47e00640 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b030 0x561bb1c431b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.932+0000 7fda47e00640 1 -- 172.21.15.6:0/544323380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c46210 msgr2=0x561bb1c43c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.932+0000 7fda47e00640 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c46210 0x561bb1c43c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.932+0000 7fda47e00640 1 -- 172.21.15.6:0/544323380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bb1977620 con 0x561bb1c3b950 2024-04-26T02:05:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.933+0000 7fda47e00640 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 0x561bb1c436f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fda34005ab0 tx=0x7fda34004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=290e7099c1cbc648 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.933+0000 7fda2ea00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:05:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.934+0000 7fda2ea00640 1 -- 172.21.15.6:0/544323380 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3403f070 con 0x561bb1c3b950 2024-04-26T02:05:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.934+0000 7fda2ea00640 1 -- 172.21.15.6:0/544323380 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7fda34005240 con 0x561bb1c3b950 2024-04-26T02:05:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.935+0000 7fda52879740 1 -- 172.21.15.6:0/544323380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda40004050 con 0x561bb1c3b950 2024-04-26T02:05:40.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.935+0000 7fda2ea00640 1 -- 172.21.15.6:0/544323380 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fda34049050 con 0x561bb1c3b950 2024-04-26T02:05:40.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.935+0000 7fda52879740 10 client.27419 fetch_fsmap learned FSMap version 55 2024-04-26T02:05:40.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.935+0000 7fda52879740 1 -- 172.21.15.6:0/544323380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x561bb1c2ff70 con 0x561bb1c3b950 2024-04-26T02:05:40.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.935+0000 7fda2ea00640 1 -- 172.21.15.6:0/544323380 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda40004050 con 0x561bb1c3b950 2024-04-26T02:05:40.935 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.936+0000 7fda2ea00640 1 -- 172.21.15.6:0/544323380 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7fda34003680 con 0x561bb1c3b950 2024-04-26T02:05:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.936+0000 7fda52879740 10 client.27419 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:05:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.936+0000 7fda52879740 10 client.27419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:05:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.936+0000 7fda52879740 20 client.27419 populate_metadata read hostname 'smithi006' 2024-04-26T02:05:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.936+0000 7fda52879740 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x561bb1c31480 0x561bb1c338c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.936+0000 7fda52879740 4 client.27419 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:05:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.936+0000 7fda52879740 1 -- 172.21.15.6:0/544323380 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bb1bac8b0 con 0x561bb1c31480 2024-04-26T02:05:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.937+0000 7fda4da00640 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x561bb1c31480 0x561bb1c338c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.937+0000 7fda4da00640 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x561bb1c31480 0x561bb1c338c0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53446e2f962b843d server_cookie=1965acbcdec90156 in_seq=0 out_seq=0 2024-04-26T02:05:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.937+0000 7fda2ea00640 10 client.27419 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.939+0000 7fda2ea00640 1 -- 172.21.15.6:0/544323380 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561bb1bac8b0 con 0x561bb1c31480 2024-04-26T02:05:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.939+0000 7fda2ea00640 10 client.27419 handle_command_reply: tid=0 2024-04-26T02:05:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.939+0000 7fda52879740 1 client.27419 shutdown 2024-04-26T02:05:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.939+0000 7fda52879740 1 -- 172.21.15.6:0/544323380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 msgr2=0x561bb1c436f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.939+0000 7fda52879740 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 0x561bb1c436f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fda34005ab0 tx=0x7fda34004270 comp rx=0 tx=0).stop 2024-04-26T02:05:40.939 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.939+0000 7fda52879740 1 -- 172.21.15.6:0/544323380 shutdown_connections 2024-04-26T02:05:40.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.939+0000 7fda52879740 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x561bb1c31480 0x561bb1c338c0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.939+0000 7fda52879740 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c46210 0x561bb1c43c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.939+0000 7fda52879740 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 0x561bb1c436f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.939+0000 7fda52879740 1 --2- 172.21.15.6:0/544323380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b030 0x561bb1c431b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.939+0000 7fda52879740 1 -- 172.21.15.6:0/544323380 >> 172.21.15.6:0/544323380 conn(0x561bb1c37540 msgr2=0x561bb1c3d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:40.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.939+0000 7fda2ea00640 0 client.27419 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:40.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.940+0000 7fda52879740 1 -- 172.21.15.6:0/544323380 shutdown_connections 2024-04-26T02:05:40.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.940+0000 7fda52879740 1 -- 172.21.15.6:0/544323380 wait complete. 2024-04-26T02:05:40.940 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.940+0000 7fda52879740 20 client.27419 trim_cache size 0 max 16384 2024-04-26T02:05:40.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:40 smithi029 ceph-mon[32775]: pgmap v626: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.9 MiB/s wr, 508 op/s 2024-04-26T02:05:40.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:40 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:05:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.973+0000 7fda52879740 1 Processor -- start 2024-04-26T02:05:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.973+0000 7fda52879740 1 -- start start 2024-04-26T02:05:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.974+0000 7fda52879740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c3b030 0x561bb1c51d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.974+0000 7fda52879740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 0x561bb1c43a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.974+0000 7fda52879740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c46210 0x561bb1c36880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.974+0000 7fda52879740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fda481ae610 con 0x561bb1c3b030 2024-04-26T02:05:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.974+0000 7fda52879740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fda481abe80 con 0x561bb1c3b950 2024-04-26T02:05:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.974+0000 7fda52879740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fda481a9d10 con 0x561bb1c46210 2024-04-26T02:05:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.974+0000 7fda47e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 0x561bb1c43a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.974+0000 7fda4da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c3b030 0x561bb1c51d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.974+0000 7fda47e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 0x561bb1c43a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59036/0 (socket says 172.21.15.6:59036) 2024-04-26T02:05:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.974+0000 7fda47e00640 1 -- 172.21.15.6:0/884717881 learned_addr learned my addr 172.21.15.6:0/884717881 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.974+0000 7fda4da00640 1 --2- 172.21.15.6:0/884717881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c46210 0x561bb1c36880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.975+0000 7fda47e00640 1 -- 172.21.15.6:0/884717881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c46210 msgr2=0x561bb1c36880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.975+0000 7fda47e00640 1 --2- 172.21.15.6:0/884717881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c46210 0x561bb1c36880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.975+0000 7fda47e00640 1 -- 172.21.15.6:0/884717881 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c3b030 msgr2=0x561bb1c51d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.975 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.975+0000 7fda47e00640 1 --2- 172.21.15.6:0/884717881 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c3b030 0x561bb1c51d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.975+0000 7fda47e00640 1 -- 172.21.15.6:0/884717881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bb1c292a0 con 0x561bb1c3b950 2024-04-26T02:05:40.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.975+0000 7fda4da00640 1 --2- 172.21.15.6:0/884717881 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c3b030 0x561bb1c51d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:05:40.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.975+0000 7fda47e00640 1 --2- 172.21.15.6:0/884717881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 0x561bb1c43a50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fda3402aa70 tx=0x7fda34003da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2df451084e177865 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:40.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.975+0000 7fda39600640 1 -- 172.21.15.6:0/884717881 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3403f040 con 0x561bb1c3b950 2024-04-26T02:05:40.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.975+0000 7fda39600640 1 -- 172.21.15.6:0/884717881 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda340037f0 con 0x561bb1c3b950 2024-04-26T02:05:40.976 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.975+0000 7fda39600640 1 -- 172.21.15.6:0/884717881 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34035030 con 0x561bb1c3b950 2024-04-26T02:05:40.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.976+0000 7fda52879740 1 -- 172.21.15.6:0/884717881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 msgr2=0x561bb1c43a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.976+0000 7fda52879740 1 --2- 172.21.15.6:0/884717881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 0x561bb1c43a50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fda3402aa70 tx=0x7fda34003da0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.977+0000 7fda52879740 1 -- 172.21.15.6:0/884717881 shutdown_connections 2024-04-26T02:05:40.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.977+0000 7fda52879740 1 --2- 172.21.15.6:0/884717881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c46210 0x561bb1c36880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.977+0000 7fda52879740 1 --2- 172.21.15.6:0/884717881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b950 0x561bb1c43a50 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.977+0000 7fda52879740 1 --2- 172.21.15.6:0/884717881 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c3b030 0x561bb1c51d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.977+0000 7fda52879740 1 -- 172.21.15.6:0/884717881 >> 172.21.15.6:0/884717881 conn(0x561bb1c37540 msgr2=0x561bb1c36170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:40.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.977+0000 7fda52879740 1 -- 172.21.15.6:0/884717881 shutdown_connections 2024-04-26T02:05:40.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.977+0000 7fda52879740 1 -- 172.21.15.6:0/884717881 wait complete. 2024-04-26T02:05:40.977 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.978+0000 7fda52879740 1 -- start start 2024-04-26T02:05:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.978+0000 7fda52879740 1 Processor -- start 2024-04-26T02:05:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.978+0000 7fda52879740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b030 0x561bb1c4e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.978+0000 7fda52879740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b950 0x561bb1c4b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda52879740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c46210 0x561bb1c4b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda52879740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fda481aa360 con 0x561bb1c46210 2024-04-26T02:05:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda52879740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fda481ab7b0 con 0x561bb1c3b030 2024-04-26T02:05:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda52879740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fda481aaa10 con 0x561bb1c3b950 2024-04-26T02:05:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda52879740 10 client.0 _ll_register_callbacks cb 0x7fda34009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda47e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b950 0x561bb1c4b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda47e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b950 0x561bb1c4b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:44526/0 (socket says 172.21.15.6:44526) 2024-04-26T02:05:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda47e00640 1 -- 172.21.15.6:0/4060412593 learned_addr learned my addr 172.21.15.6:0/4060412593 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:40.980 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda4da00640 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c46210 0x561bb1c4b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda4da00640 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b030 0x561bb1c4e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda47e00640 1 -- 172.21.15.6:0/4060412593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b030 msgr2=0x561bb1c4e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda47e00640 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b030 0x561bb1c4e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda47e00640 1 -- 172.21.15.6:0/4060412593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c46210 msgr2=0x561bb1c4b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda47e00640 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c46210 0x561bb1c4b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.979+0000 7fda47e00640 1 -- 172.21.15.6:0/4060412593 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bb1c4bc60 con 0x561bb1c3b950 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.980+0000 7fda47e00640 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b950 0x561bb1c4b1e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fda34006010 tx=0x7fda34032bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcf39e09484d5a41 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.980+0000 7fda2ea00640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.980+0000 7fda2ea00640 1 -- 172.21.15.6:0/4060412593 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3403f070 con 0x561bb1c3b950 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.980+0000 7fda52879740 1 -- 172.21.15.6:0/4060412593 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda340037f0 con 0x561bb1c3b950 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.980+0000 7fda2ea00640 1 -- 172.21.15.6:0/4060412593 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7fda3403b460 con 0x561bb1c3b950 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.981+0000 7fda2ea00640 1 -- 172.21.15.6:0/4060412593 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fda34035040 con 0x561bb1c3b950 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.981+0000 7fda52879740 10 client.27490 fetch_fsmap learned FSMap version 55 2024-04-26T02:05:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.981+0000 7fda52879740 1 -- 172.21.15.6:0/4060412593 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x561bb1cc4dc0 con 0x561bb1c3b950 2024-04-26T02:05:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.981+0000 7fda2ea00640 1 -- 172.21.15.6:0/4060412593 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda340037f0 con 0x561bb1c3b950 2024-04-26T02:05:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.981+0000 7fda2ea00640 1 -- 172.21.15.6:0/4060412593 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7fda3403bb10 con 0x561bb1c3b950 2024-04-26T02:05:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.981+0000 7fda52879740 10 client.27490 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:05:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.981+0000 7fda52879740 10 client.27490 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:05:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.981+0000 7fda52879740 20 client.27490 populate_metadata read hostname 'smithi006' 2024-04-26T02:05:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.981+0000 7fda52879740 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x561bb1cc5460 0x561bb1cc7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.981+0000 7fda52879740 4 client.27490 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:05:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.981+0000 7fda52879740 1 -- 172.21.15.6:0/4060412593 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561bb1ba6320 con 0x561bb1cc5460 2024-04-26T02:05:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.982+0000 7fda4da00640 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x561bb1cc5460 0x561bb1cc7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.982+0000 7fda4da00640 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x561bb1cc5460 0x561bb1cc7880 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdb09a1f53ee0f15 server_cookie=7ea8fd4d1679ab2 in_seq=0 out_seq=0 2024-04-26T02:05:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:40.982+0000 7fda2ea00640 10 client.27490 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:41.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:40 smithi006 ceph-mon[28419]: pgmap v626: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.9 MiB/s wr, 508 op/s 2024-04-26T02:05:41.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:40 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:05:41.136 INFO:tasks.workunit.client.0.smithi006.stdout: 10 118013 18.95 MB/sec execute 244 sec latency 274.603 ms 2024-04-26T02:05:41.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:40 smithi188 ceph-mon[32751]: pgmap v626: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.9 MiB/s wr, 508 op/s 2024-04-26T02:05:41.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:40 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:05:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.231+0000 7fda2ea00640 1 -- 172.21.15.6:0/4060412593 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561bb1ba6320 con 0x561bb1cc5460 2024-04-26T02:05:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.231+0000 7fda2ea00640 10 client.27490 handle_command_reply: tid=0 2024-04-26T02:05:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.231+0000 7fda52879740 1 client.27490 shutdown 2024-04-26T02:05:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.231+0000 7fda52879740 1 -- 172.21.15.6:0/4060412593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b950 msgr2=0x561bb1c4b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.231+0000 7fda52879740 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b950 0x561bb1c4b1e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fda34006010 tx=0x7fda34032bb0 comp rx=0 tx=0).stop 2024-04-26T02:05:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.231+0000 7fda52879740 1 -- 172.21.15.6:0/4060412593 shutdown_connections 2024-04-26T02:05:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.231+0000 7fda52879740 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x561bb1cc5460 0x561bb1cc7880 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.231+0000 7fda52879740 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561bb1c46210 0x561bb1c4b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.231+0000 7fda52879740 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561bb1c3b950 0x561bb1c4b1e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.231+0000 7fda2ea00640 0 client.27490 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.231+0000 7fda52879740 1 --2- 172.21.15.6:0/4060412593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561bb1c3b030 0x561bb1c4e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.231+0000 7fda52879740 1 -- 172.21.15.6:0/4060412593 >> 172.21.15.6:0/4060412593 conn(0x561bb1c37540 msgr2=0x561bb1c41680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.232+0000 7fda52879740 1 -- 172.21.15.6:0/4060412593 shutdown_connections 2024-04-26T02:05:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.232+0000 7fda52879740 1 -- 172.21.15.6:0/4060412593 wait complete. 2024-04-26T02:05:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.232+0000 7fda52879740 20 client.27490 trim_cache size 0 max 16384 2024-04-26T02:05:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:05:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:05:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:05:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:05:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.234+0000 7fda4fa00640 1 -- 172.21.15.6:0/3888617676 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fda300a3d90 msgr2=0x7fda300a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.234+0000 7fda4fa00640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fda300a3d90 0x7fda300a6250 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x561bb1c3c5d0 tx=0x7fda3c0094d0 comp rx=0 tx=0).stop 2024-04-26T02:05:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.234+0000 7fda4fa00640 1 -- 172.21.15.6:0/3888617676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda481003e0 msgr2=0x7fda481a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.234+0000 7fda4fa00640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda481003e0 0x7fda481a0660 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00ea60 tx=0x7fda3c00ef20 comp rx=0 tx=0).stop 2024-04-26T02:05:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.234+0000 7fda4fa00640 1 -- 172.21.15.6:0/3888617676 shutdown_connections 2024-04-26T02:05:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.234+0000 7fda4fa00640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fda300a3d90 0x7fda300a6250 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x561bb1c3c5d0 tx=0x7fda3c0094d0 comp rx=0 tx=0).stop 2024-04-26T02:05:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.234+0000 7fda4fa00640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda4810e920 0x7fda4819a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.234+0000 7fda4fa00640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda48100d00 0x7fda481a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.234+0000 7fda4fa00640 1 --2- 172.21.15.6:0/3888617676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda481003e0 0x7fda481a0660 secure :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00ea60 tx=0x7fda3c00ef20 comp rx=0 tx=0).stop 2024-04-26T02:05:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.234+0000 7fda4fa00640 1 -- 172.21.15.6:0/3888617676 >> 172.21.15.6:0/3888617676 conn(0x7fda48075d60 msgr2=0x7fda4810ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.235+0000 7fda4fa00640 1 -- 172.21.15.6:0/3888617676 shutdown_connections 2024-04-26T02:05:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:41.235+0000 7fda4fa00640 1 -- 172.21.15.6:0/3888617676 wait complete. 2024-04-26T02:05:41.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:41 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:05:41.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:41 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:05:41.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:41 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:05:42.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:41 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:05:42.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:41 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:05:42.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:41 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:05:42.136 INFO:tasks.workunit.client.0.smithi006.stdout: 10 118517 18.96 MB/sec execute 245 sec latency 183.933 ms 2024-04-26T02:05:42.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:41 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:05:42.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:41 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:05:42.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:41 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:05:42.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:42 smithi029 ceph-mon[32775]: pgmap v627: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 13 MiB/s wr, 776 op/s 2024-04-26T02:05:43.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:42 smithi006 ceph-mon[28419]: pgmap v627: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 13 MiB/s wr, 776 op/s 2024-04-26T02:05:43.136 INFO:tasks.workunit.client.0.smithi006.stdout: 10 119175 19.07 MB/sec execute 246 sec latency 66.566 ms 2024-04-26T02:05:43.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:42 smithi188 ceph-mon[32751]: pgmap v627: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 13 MiB/s wr, 776 op/s 2024-04-26T02:05:44.136 INFO:tasks.workunit.client.0.smithi006.stdout: 10 119754 19.11 MB/sec execute 247 sec latency 82.187 ms 2024-04-26T02:05:44.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:44 smithi029 ceph-mon[32775]: pgmap v628: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 472 op/s 2024-04-26T02:05:45.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:44 smithi006 ceph-mon[28419]: pgmap v628: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 472 op/s 2024-04-26T02:05:45.136 INFO:tasks.workunit.client.0.smithi006.stdout: 10 120194 19.11 MB/sec execute 248 sec latency 131.185 ms 2024-04-26T02:05:45.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:44 smithi188 ceph-mon[32751]: pgmap v628: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 472 op/s 2024-04-26T02:05:46.136 INFO:tasks.workunit.client.0.smithi006.stdout: 10 120612 19.10 MB/sec execute 249 sec latency 281.066 ms 2024-04-26T02:05:46.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:46 smithi029 ceph-mon[32775]: pgmap v629: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 472 op/s 2024-04-26T02:05:47.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:46 smithi006 ceph-mon[28419]: pgmap v629: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 472 op/s 2024-04-26T02:05:47.136 INFO:tasks.workunit.client.0.smithi006.stdout: 10 121198 19.17 MB/sec execute 250 sec latency 105.777 ms 2024-04-26T02:05:47.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:46 smithi188 ceph-mon[32751]: pgmap v629: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 472 op/s 2024-04-26T02:05:48.136 INFO:tasks.workunit.client.0.smithi006.stdout: 10 121902 19.26 MB/sec execute 251 sec latency 93.925 ms 2024-04-26T02:05:48.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:48 smithi029 ceph-mon[32775]: pgmap v630: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 14 MiB/s wr, 745 op/s 2024-04-26T02:05:49.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:48 smithi006 ceph-mon[28419]: pgmap v630: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 14 MiB/s wr, 745 op/s 2024-04-26T02:05:49.136 INFO:tasks.workunit.client.0.smithi006.stdout: 10 122473 19.31 MB/sec execute 252 sec latency 94.514 ms 2024-04-26T02:05:49.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:48 smithi188 ceph-mon[32751]: pgmap v630: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 14 MiB/s wr, 745 op/s 2024-04-26T02:05:50.137 INFO:tasks.workunit.client.0.smithi006.stdout: 10 123094 19.38 MB/sec execute 253 sec latency 80.059 ms 2024-04-26T02:05:50.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:50 smithi029 ceph-mon[32775]: pgmap v631: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 540 op/s 2024-04-26T02:05:51.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:50 smithi006 ceph-mon[28419]: pgmap v631: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 540 op/s 2024-04-26T02:05:51.137 INFO:tasks.workunit.client.0.smithi006.stdout: 10 123513 19.40 MB/sec execute 254 sec latency 265.429 ms 2024-04-26T02:05:51.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:50 smithi188 ceph-mon[32751]: pgmap v631: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 540 op/s 2024-04-26T02:05:52.137 INFO:tasks.workunit.client.0.smithi006.stdout: 10 123885 19.38 MB/sec execute 255 sec latency 127.865 ms 2024-04-26T02:05:52.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:52 smithi029 ceph-mon[32775]: pgmap v632: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 763 op/s 2024-04-26T02:05:53.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:52 smithi006 ceph-mon[28419]: pgmap v632: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 763 op/s 2024-04-26T02:05:53.138 INFO:tasks.workunit.client.0.smithi006.stdout: 10 124418 19.36 MB/sec execute 256 sec latency 110.631 ms 2024-04-26T02:05:53.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:52 smithi188 ceph-mon[32751]: pgmap v632: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 763 op/s 2024-04-26T02:05:54.137 INFO:tasks.workunit.client.0.smithi006.stdout: 10 125103 19.49 MB/sec execute 257 sec latency 83.848 ms 2024-04-26T02:05:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.404+0000 7fc5da200640 1 -- 172.21.15.6:0/1400148009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d410e470 msgr2=0x7fc5d410e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.404+0000 7fc5da200640 1 --2- 172.21.15.6:0/1400148009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d410e470 0x7fc5d410e850 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4009900 tx=0x7fc5c402a480 comp rx=0 tx=0).stop 2024-04-26T02:05:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.405+0000 7fc5da200640 1 -- 172.21.15.6:0/1400148009 shutdown_connections 2024-04-26T02:05:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.405+0000 7fc5da200640 1 --2- 172.21.15.6:0/1400148009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5d4108d50 0x7fc5d4115210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.405+0000 7fc5da200640 1 --2- 172.21.15.6:0/1400148009 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5d41083b0 0x7fc5d4108810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.405+0000 7fc5da200640 1 --2- 172.21.15.6:0/1400148009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d410e470 0x7fc5d410e850 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.405+0000 7fc5da200640 1 -- 172.21.15.6:0/1400148009 >> 172.21.15.6:0/1400148009 conn(0x7fc5d4077b00 msgr2=0x7fc5d4079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.405+0000 7fc5da200640 1 -- 172.21.15.6:0/1400148009 shutdown_connections 2024-04-26T02:05:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.405+0000 7fc5da200640 1 -- 172.21.15.6:0/1400148009 wait complete. 2024-04-26T02:05:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.406+0000 7fc5da200640 1 Processor -- start 2024-04-26T02:05:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.406+0000 7fc5da200640 1 -- start start 2024-04-26T02:05:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5da200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5d41083b0 0x7fc5d41a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5da200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5d4108d50 0x7fc5d41a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5da200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d410e470 0x7fc5d419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5da200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d4118630 con 0x7fc5d4108d50 2024-04-26T02:05:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5da200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d41184b0 con 0x7fc5d410e470 2024-04-26T02:05:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5da200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d41187b0 con 0x7fc5d41083b0 2024-04-26T02:05:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5d8e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5d41083b0 0x7fc5d41a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5d8e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5d41083b0 0x7fc5d41a4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:48134/0 (socket says 172.21.15.6:48134) 2024-04-26T02:05:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5d8e00640 1 -- 172.21.15.6:0/1418909851 learned_addr learned my addr 172.21.15.6:0/1418909851 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5cbe00640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5d4108d50 0x7fc5d41a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5d9800640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d410e470 0x7fc5d419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5d8e00640 1 -- 172.21.15.6:0/1418909851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d410e470 msgr2=0x7fc5d419e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5d8e00640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d410e470 0x7fc5d419e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5d8e00640 1 -- 172.21.15.6:0/1418909851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5d4108d50 msgr2=0x7fc5d41a4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5d8e00640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5d4108d50 0x7fc5d41a4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.407+0000 7fc5d8e00640 1 -- 172.21.15.6:0/1418909851 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d419f090 con 0x7fc5d41083b0 2024-04-26T02:05:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.408+0000 7fc5d8e00640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5d41083b0 0x7fc5d41a4760 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4005ab0 tx=0x7fc5c4003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.408+0000 7fc5d2000640 1 -- 172.21.15.6:0/1418909851 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c403f070 con 0x7fc5d41083b0 2024-04-26T02:05:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.408+0000 7fc5d2000640 1 -- 172.21.15.6:0/1418909851 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5c4031b50 con 0x7fc5d41083b0 2024-04-26T02:05:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.408+0000 7fc5da200640 1 -- 172.21.15.6:0/1418909851 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d419f320 con 0x7fc5d41083b0 2024-04-26T02:05:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.408+0000 7fc5d2000640 1 -- 172.21.15.6:0/1418909851 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4035030 con 0x7fc5d41083b0 2024-04-26T02:05:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.408+0000 7fc5da200640 1 -- 172.21.15.6:0/1418909851 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d41ab620 con 0x7fc5d41083b0 2024-04-26T02:05:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.412+0000 7fc5d2000640 1 -- 172.21.15.6:0/1418909851 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fc5c4042070 con 0x7fc5d41083b0 2024-04-26T02:05:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.412+0000 7fc5de58e740 1 Processor -- start 2024-04-26T02:05:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.413+0000 7fc5de58e740 1 -- start start 2024-04-26T02:05:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.413+0000 7fc5de58e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 0x560eca912d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.413+0000 7fc5de58e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9132d0 0x560eca91d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.413+0000 7fc5de58e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91db90 0x560eca91ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.413+0000 7fc5de58e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560eca878900 con 0x560eca91db90 2024-04-26T02:05:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.413+0000 7fc5de58e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560eca81eef0 con 0x560eca9129b0 2024-04-26T02:05:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.413+0000 7fc5de58e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560eca8748c0 con 0x560eca9132d0 2024-04-26T02:05:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.414+0000 7fc5d9800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 0x560eca912d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.414+0000 7fc5d9800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 0x560eca912d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:58530/0 (socket says 172.21.15.6:58530) 2024-04-26T02:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.414+0000 7fc5d9800640 1 -- 172.21.15.6:0/1978668717 learned_addr learned my addr 172.21.15.6:0/1978668717 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.414+0000 7fc5cbe00640 1 --2- 172.21.15.6:0/1978668717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91db90 0x560eca91ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.414+0000 7fc5d9800640 1 -- 172.21.15.6:0/1978668717 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9132d0 msgr2=0x560eca91d650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.414+0000 7fc5d9800640 1 --2- 172.21.15.6:0/1978668717 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9132d0 0x560eca91d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.414+0000 7fc5d9800640 1 -- 172.21.15.6:0/1978668717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91db90 msgr2=0x560eca91ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.414+0000 7fc5d9800640 1 --2- 172.21.15.6:0/1978668717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91db90 0x560eca91ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.414+0000 7fc5d9800640 1 -- 172.21.15.6:0/1978668717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eca58b210 con 0x560eca9129b0 2024-04-26T02:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.414+0000 7fc5d2000640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc5b80a3d60 0x7fc5b80a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.414+0000 7fc5d2000640 1 -- 172.21.15.6:0/1418909851 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7fc5c40f0f70 con 0x7fc5d41083b0 2024-04-26T02:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.414+0000 7fc5d9800640 1 --2- 172.21.15.6:0/1978668717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 0x560eca912d90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc010370 tx=0x7fc5cc010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9ed21e016e436a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.415+0000 7fc5ca000640 1 -- 172.21.15.6:0/1978668717 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc016070 con 0x560eca9129b0 2024-04-26T02:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.415+0000 7fc5ca000640 1 -- 172.21.15.6:0/1978668717 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5cc004d20 con 0x560eca9129b0 2024-04-26T02:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.415+0000 7fc5ca000640 1 -- 172.21.15.6:0/1978668717 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc005380 con 0x560eca9129b0 2024-04-26T02:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.415+0000 7fc5d8e00640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc5b80a3d60 0x7fc5b80a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.416+0000 7fc5de58e740 1 -- 172.21.15.6:0/1978668717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 msgr2=0x560eca912d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.416+0000 7fc5de58e740 1 --2- 172.21.15.6:0/1978668717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 0x560eca912d90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc010370 tx=0x7fc5cc010830 comp rx=0 tx=0).stop 2024-04-26T02:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.416+0000 7fc5d8e00640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc5b80a3d60 0x7fc5b80a6220 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x560eca913f50 tx=0x7fc5c4009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.416+0000 7fc5de58e740 1 -- 172.21.15.6:0/1978668717 shutdown_connections 2024-04-26T02:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.416+0000 7fc5de58e740 1 --2- 172.21.15.6:0/1978668717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91db90 0x560eca91ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.416+0000 7fc5de58e740 1 --2- 172.21.15.6:0/1978668717 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9132d0 0x560eca91d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.416+0000 7fc5de58e740 1 --2- 172.21.15.6:0/1978668717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 0x560eca912d90 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.416+0000 7fc5de58e740 1 -- 172.21.15.6:0/1978668717 >> 172.21.15.6:0/1978668717 conn(0x560eca90eec0 msgr2=0x560eca90f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.416+0000 7fc5de58e740 1 -- 172.21.15.6:0/1978668717 shutdown_connections 2024-04-26T02:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.416+0000 7fc5de58e740 1 -- 172.21.15.6:0/1978668717 wait complete. 2024-04-26T02:05:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.417+0000 7fc5de58e740 1 -- start start 2024-04-26T02:05:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.417+0000 7fc5de58e740 1 Processor -- start 2024-04-26T02:05:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.418+0000 7fc5de58e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 0x560eca91ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.418+0000 7fc5de58e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca9132d0 0x560eca91b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.418+0000 7fc5de58e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca91db90 0x560eca91b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.418+0000 7fc5de58e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560eca9207c0 con 0x560eca9132d0 2024-04-26T02:05:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.418+0000 7fc5de58e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560eca832310 con 0x560eca91db90 2024-04-26T02:05:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.418+0000 7fc5de58e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560eca920940 con 0x560eca9129b0 2024-04-26T02:05:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.418+0000 7fc5de58e740 10 client.0 _ll_register_callbacks cb 0x560eca800cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.418+0000 7fc5cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca9132d0 0x560eca91b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.418+0000 7fc5d9800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 0x560eca91ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.418+0000 7fc5d9800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 0x560eca91ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:48144/0 (socket says 172.21.15.6:48144) 2024-04-26T02:05:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.418+0000 7fc5d9800640 1 -- 172.21.15.6:0/4063868185 learned_addr learned my addr 172.21.15.6:0/4063868185 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.418+0000 7fc5d9800640 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca91db90 0x560eca91b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.419+0000 7fc5cbe00640 1 -- 172.21.15.6:0/4063868185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 msgr2=0x560eca91ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.419+0000 7fc5cbe00640 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 0x560eca91ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.419+0000 7fc5cbe00640 1 -- 172.21.15.6:0/4063868185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca91db90 msgr2=0x560eca91b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.419+0000 7fc5cbe00640 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca91db90 0x560eca91b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.419+0000 7fc5cbe00640 1 -- 172.21.15.6:0/4063868185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eca657620 con 0x560eca9132d0 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.419+0000 7fc5d9800640 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca91db90 0x560eca91b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.419+0000 7fc5cbe00640 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca9132d0 0x560eca91b060 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc5d41078b0 tx=0x7fc5bc012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100e4646f216f7a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.419+0000 7fc5c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.419+0000 7fc5c2a00640 1 -- 172.21.15.6:0/4063868185 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc00f970 con 0x560eca9132d0 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.419+0000 7fc5c2a00640 1 -- 172.21.15.6:0/4063868185 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7fc5bc00fb10 con 0x560eca9132d0 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.420+0000 7fc5de58e740 1 -- 172.21.15.6:0/4063868185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5cc004d20 con 0x560eca9132d0 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.420+0000 7fc5c2a00640 1 -- 172.21.15.6:0/4063868185 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fc5bc00fd10 con 0x560eca9132d0 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.420+0000 7fc5de58e740 10 client.17238 fetch_fsmap learned FSMap version 55 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.420+0000 7fc5de58e740 1 -- 172.21.15.6:0/4063868185 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x560eca92cac0 con 0x560eca9132d0 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.421+0000 7fc5c2a00640 1 -- 172.21.15.6:0/4063868185 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5cc004d20 con 0x560eca9132d0 2024-04-26T02:05:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.421+0000 7fc5c2a00640 1 -- 172.21.15.6:0/4063868185 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7fc5bc010430 con 0x560eca9132d0 2024-04-26T02:05:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.421+0000 7fc5de58e740 10 client.17238 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:05:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.421+0000 7fc5de58e740 10 client.17238 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:05:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.421+0000 7fc5de58e740 20 client.17238 populate_metadata read hostname 'smithi006' 2024-04-26T02:05:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.421+0000 7fc5de58e740 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560eca92dfd0 0x560eca9303c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.421+0000 7fc5de58e740 4 client.17238 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:05:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.421+0000 7fc5de58e740 1 -- 172.21.15.6:0/4063868185 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560eca88c8b0 con 0x560eca92dfd0 2024-04-26T02:05:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.421+0000 7fc5d9800640 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560eca92dfd0 0x560eca9303c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.422+0000 7fc5d9800640 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560eca92dfd0 0x560eca9303c0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=942c766d9576a7af server_cookie=85b54f726e12e34b in_seq=0 out_seq=0 2024-04-26T02:05:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.422+0000 7fc5c2a00640 10 client.17238 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.423+0000 7fc5c2a00640 1 -- 172.21.15.6:0/4063868185 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560eca88c8b0 con 0x560eca92dfd0 2024-04-26T02:05:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.423+0000 7fc5c2a00640 10 client.17238 handle_command_reply: tid=0 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.423+0000 7fc5de58e740 1 client.17238 shutdown 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.423+0000 7fc5de58e740 1 -- 172.21.15.6:0/4063868185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca9132d0 msgr2=0x560eca91b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.423+0000 7fc5de58e740 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca9132d0 0x560eca91b060 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc5d41078b0 tx=0x7fc5bc012b60 comp rx=0 tx=0).stop 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.424+0000 7fc5de58e740 1 -- 172.21.15.6:0/4063868185 shutdown_connections 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.424+0000 7fc5de58e740 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560eca92dfd0 0x560eca9303c0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.424+0000 7fc5de58e740 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca9132d0 0x560eca91b060 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.424+0000 7fc5de58e740 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca91db90 0x560eca91b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.424+0000 7fc5de58e740 1 --2- 172.21.15.6:0/4063868185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 0x560eca91ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.424+0000 7fc5de58e740 1 -- 172.21.15.6:0/4063868185 >> 172.21.15.6:0/4063868185 conn(0x560eca90eec0 msgr2=0x560eca914bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.424+0000 7fc5c2a00640 0 client.17238 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.424+0000 7fc5de58e740 1 -- 172.21.15.6:0/4063868185 shutdown_connections 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.424+0000 7fc5de58e740 1 -- 172.21.15.6:0/4063868185 wait complete. 2024-04-26T02:05:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.424+0000 7fc5de58e740 20 client.17238 trim_cache size 0 max 16384 2024-04-26T02:05:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.444+0000 7fc5de58e740 1 Processor -- start 2024-04-26T02:05:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.444+0000 7fc5de58e740 1 -- start start 2024-04-26T02:05:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.445+0000 7fc5de58e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 0x560eca919a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.445+0000 7fc5de58e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91db90 0x560eca915370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.445+0000 7fc5de58e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca91d360 0x560eca91b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.445+0000 7fc5de58e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d41b6ae0 con 0x560eca91db90 2024-04-26T02:05:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.445+0000 7fc5de58e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d41b4350 con 0x560eca91d360 2024-04-26T02:05:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.445+0000 7fc5de58e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d41b21c0 con 0x560eca9129b0 2024-04-26T02:05:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.445+0000 7fc5d9800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 0x560eca919a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.445+0000 7fc5d9800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 0x560eca919a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:48148/0 (socket says 172.21.15.6:48148) 2024-04-26T02:05:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.445+0000 7fc5d9800640 1 -- 172.21.15.6:0/4237868616 learned_addr learned my addr 172.21.15.6:0/4237868616 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.445+0000 7fc5d9800640 1 --2- 172.21.15.6:0/4237868616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca91d360 0x560eca91b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.446+0000 7fc5d9800640 1 -- 172.21.15.6:0/4237868616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca91d360 msgr2=0x560eca91b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.446+0000 7fc5cbe00640 1 --2- 172.21.15.6:0/4237868616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91db90 0x560eca915370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.446+0000 7fc5d9800640 1 --2- 172.21.15.6:0/4237868616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca91d360 0x560eca91b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.446+0000 7fc5d9800640 1 -- 172.21.15.6:0/4237868616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91db90 msgr2=0x560eca915370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.446+0000 7fc5d9800640 1 --2- 172.21.15.6:0/4237868616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91db90 0x560eca915370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.446+0000 7fc5d9800640 1 -- 172.21.15.6:0/4237868616 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eca909490 con 0x560eca9129b0 2024-04-26T02:05:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.446+0000 7fc5cbe00640 1 --2- 172.21.15.6:0/4237868616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91db90 0x560eca915370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:05:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.447+0000 7fc5d9800640 1 --2- 172.21.15.6:0/4237868616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 0x560eca919a80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc015040 tx=0x7fc5cc000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4d247a15f742672 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.447+0000 7fc5c9600640 1 -- 172.21.15.6:0/4237868616 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc02de60 con 0x560eca9129b0 2024-04-26T02:05:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.447+0000 7fc5c9600640 1 -- 172.21.15.6:0/4237868616 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5cc012bc0 con 0x560eca9129b0 2024-04-26T02:05:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.447+0000 7fc5c9600640 1 -- 172.21.15.6:0/4237868616 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc016040 con 0x560eca9129b0 2024-04-26T02:05:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.448+0000 7fc5de58e740 1 -- 172.21.15.6:0/4237868616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 msgr2=0x560eca919a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.448+0000 7fc5de58e740 1 --2- 172.21.15.6:0/4237868616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 0x560eca919a80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc015040 tx=0x7fc5cc000f40 comp rx=0 tx=0).stop 2024-04-26T02:05:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.448+0000 7fc5de58e740 1 -- 172.21.15.6:0/4237868616 shutdown_connections 2024-04-26T02:05:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.448+0000 7fc5de58e740 1 --2- 172.21.15.6:0/4237868616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91db90 0x560eca915370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.448+0000 7fc5de58e740 1 --2- 172.21.15.6:0/4237868616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca91d360 0x560eca91b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.448+0000 7fc5de58e740 1 --2- 172.21.15.6:0/4237868616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca9129b0 0x560eca919a80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.448+0000 7fc5de58e740 1 -- 172.21.15.6:0/4237868616 >> 172.21.15.6:0/4237868616 conn(0x560eca90eec0 msgr2=0x560eca90e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.448+0000 7fc5de58e740 1 -- 172.21.15.6:0/4237868616 shutdown_connections 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.448+0000 7fc5de58e740 1 -- 172.21.15.6:0/4237868616 wait complete. 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.449+0000 7fc5de58e740 1 -- start start 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.449+0000 7fc5de58e740 1 Processor -- start 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.449+0000 7fc5de58e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 0x560eca9260f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.449+0000 7fc5de58e740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91d360 0x560eca923480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.449+0000 7fc5de58e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca91db90 0x560eca923b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.449+0000 7fc5de58e740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d41b2810 con 0x560eca91d360 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.449+0000 7fc5de58e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d41b3c80 con 0x560eca9129b0 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.449+0000 7fc5de58e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d41b2ee0 con 0x560eca91db90 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.449+0000 7fc5d9800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 0x560eca9260f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.450+0000 7fc5d9800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 0x560eca9260f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:58572/0 (socket says 172.21.15.6:58572) 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.450+0000 7fc5d9800640 1 -- 172.21.15.6:0/1554052941 learned_addr learned my addr 172.21.15.6:0/1554052941 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.450+0000 7fc5de58e740 10 client.0 _ll_register_callbacks cb 0x7fc5bc009e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.450+0000 7fc5cbe00640 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91d360 0x560eca923480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.450+0000 7fc5d9800640 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca91db90 0x560eca923b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.450+0000 7fc5d9800640 1 -- 172.21.15.6:0/1554052941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca91db90 msgr2=0x560eca923b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.450+0000 7fc5d9800640 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca91db90 0x560eca923b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.450+0000 7fc5d9800640 1 -- 172.21.15.6:0/1554052941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91d360 msgr2=0x560eca923480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.450+0000 7fc5d9800640 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91d360 0x560eca923480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.450+0000 7fc5d9800640 1 -- 172.21.15.6:0/1554052941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eca993550 con 0x560eca9129b0 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.450+0000 7fc5d9800640 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 0x560eca9260f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc009810 tx=0x7fc5cc004fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74a36a93d6c84aab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.450+0000 7fc5c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.451+0000 7fc5c2a00640 1 -- 172.21.15.6:0/1554052941 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc016030 con 0x560eca9129b0 2024-04-26T02:05:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.451+0000 7fc5c2a00640 1 -- 172.21.15.6:0/1554052941 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8600+0+0 (secure 0 0 0) 0x7fc5cc012710 con 0x560eca9129b0 2024-04-26T02:05:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.451+0000 7fc5de58e740 1 -- 172.21.15.6:0/1554052941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5cc012bc0 con 0x560eca9129b0 2024-04-26T02:05:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.451+0000 7fc5c2a00640 1 -- 172.21.15.6:0/1554052941 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fc5cc012910 con 0x560eca9129b0 2024-04-26T02:05:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.451+0000 7fc5de58e740 10 client.27440 fetch_fsmap learned FSMap version 55 2024-04-26T02:05:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.451+0000 7fc5de58e740 1 -- 172.21.15.6:0/1554052941 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x560eca994670 con 0x560eca9129b0 2024-04-26T02:05:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.451+0000 7fc5c2a00640 1 -- 172.21.15.6:0/1554052941 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5cc012bc0 con 0x560eca9129b0 2024-04-26T02:05:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.451+0000 7fc5c2a00640 1 -- 172.21.15.6:0/1554052941 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7fc5cc00f040 con 0x560eca9129b0 2024-04-26T02:05:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.452+0000 7fc5de58e740 10 client.27440 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:05:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.452+0000 7fc5de58e740 10 client.27440 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:05:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.452+0000 7fc5de58e740 20 client.27440 populate_metadata read hostname 'smithi006' 2024-04-26T02:05:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.452+0000 7fc5de58e740 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560eca994a90 0x560eca996e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.452+0000 7fc5de58e740 4 client.27440 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:05:54.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.452+0000 7fc5de58e740 1 -- 172.21.15.6:0/1554052941 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560eca886320 con 0x560eca994a90 2024-04-26T02:05:54.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.452+0000 7fc5cbe00640 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560eca994a90 0x560eca996e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:54.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.452+0000 7fc5cbe00640 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560eca994a90 0x560eca996e80 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e30c654cf988cd8 server_cookie=5a757c62449dbf5c in_seq=0 out_seq=0 2024-04-26T02:05:54.455 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.452+0000 7fc5c2a00640 10 client.27440 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.737+0000 7fc5c2a00640 1 -- 172.21.15.6:0/1554052941 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560eca886320 con 0x560eca994a90 2024-04-26T02:05:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.737+0000 7fc5c2a00640 10 client.27440 handle_command_reply: tid=0 2024-04-26T02:05:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.737+0000 7fc5de58e740 1 client.27440 shutdown 2024-04-26T02:05:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.738+0000 7fc5de58e740 1 -- 172.21.15.6:0/1554052941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 msgr2=0x560eca9260f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.738+0000 7fc5de58e740 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 0x560eca9260f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc009810 tx=0x7fc5cc004fb0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.738+0000 7fc5de58e740 1 -- 172.21.15.6:0/1554052941 shutdown_connections 2024-04-26T02:05:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.738+0000 7fc5de58e740 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x560eca994a90 0x560eca996e80 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.738+0000 7fc5de58e740 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560eca91d360 0x560eca923480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.738+0000 7fc5de58e740 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560eca91db90 0x560eca923b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.738+0000 7fc5de58e740 1 --2- 172.21.15.6:0/1554052941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eca9129b0 0x560eca9260f0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.738+0000 7fc5de58e740 1 -- 172.21.15.6:0/1554052941 >> 172.21.15.6:0/1554052941 conn(0x560eca90eec0 msgr2=0x560eca91c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.738+0000 7fc5c2a00640 0 client.27440 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:05:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.739+0000 7fc5de58e740 1 -- 172.21.15.6:0/1554052941 shutdown_connections 2024-04-26T02:05:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.739+0000 7fc5de58e740 1 -- 172.21.15.6:0/1554052941 wait complete. 2024-04-26T02:05:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.739+0000 7fc5de58e740 20 client.27440 trim_cache size 0 max 16384 2024-04-26T02:05:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:05:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:05:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:05:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:05:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.740+0000 7fc5da200640 1 -- 172.21.15.6:0/1418909851 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc5b80a3d60 msgr2=0x7fc5b80a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.740+0000 7fc5da200640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc5b80a3d60 0x7fc5b80a6220 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x560eca913f50 tx=0x7fc5c4009260 comp rx=0 tx=0).stop 2024-04-26T02:05:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.740+0000 7fc5da200640 1 -- 172.21.15.6:0/1418909851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5d41083b0 msgr2=0x7fc5d41a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.740+0000 7fc5da200640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5d41083b0 0x7fc5d41a4760 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4005ab0 tx=0x7fc5c4003c20 comp rx=0 tx=0).stop 2024-04-26T02:05:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.740+0000 7fc5da200640 1 -- 172.21.15.6:0/1418909851 shutdown_connections 2024-04-26T02:05:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.740+0000 7fc5da200640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc5b80a3d60 0x7fc5b80a6220 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.740+0000 7fc5da200640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5d4108d50 0x7fc5d41a4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.740+0000 7fc5da200640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d410e470 0x7fc5d419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.740+0000 7fc5da200640 1 --2- 172.21.15.6:0/1418909851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5d41083b0 0x7fc5d41a4760 secure :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4005ab0 tx=0x7fc5c4003c20 comp rx=0 tx=0).stop 2024-04-26T02:05:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.740+0000 7fc5da200640 1 -- 172.21.15.6:0/1418909851 >> 172.21.15.6:0/1418909851 conn(0x7fc5d4077b00 msgr2=0x7fc5d4079450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.741+0000 7fc5da200640 1 -- 172.21.15.6:0/1418909851 shutdown_connections 2024-04-26T02:05:54.742 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:05:54.741+0000 7fc5da200640 1 -- 172.21.15.6:0/1418909851 wait complete. 2024-04-26T02:05:54.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:54 smithi029 ceph-mon[32775]: pgmap v633: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 495 op/s 2024-04-26T02:05:55.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:54 smithi006 ceph-mon[28419]: pgmap v633: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 495 op/s 2024-04-26T02:05:55.137 INFO:tasks.workunit.client.0.smithi006.stdout: 10 125479 19.49 MB/sec execute 258 sec latency 264.235 ms 2024-04-26T02:05:55.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:54 smithi188 ceph-mon[32751]: pgmap v633: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 495 op/s 2024-04-26T02:05:56.137 INFO:tasks.workunit.client.0.smithi006.stdout: 10 125871 19.48 MB/sec execute 259 sec latency 194.854 ms 2024-04-26T02:05:56.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:56 smithi029 ceph-mon[32775]: pgmap v634: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 496 op/s 2024-04-26T02:05:57.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:56 smithi006 ceph-mon[28419]: pgmap v634: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 496 op/s 2024-04-26T02:05:57.137 INFO:tasks.workunit.client.0.smithi006.stdout: 10 126366 19.51 MB/sec execute 260 sec latency 114.451 ms 2024-04-26T02:05:57.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:56 smithi188 ceph-mon[32751]: pgmap v634: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.4 MiB/s wr, 496 op/s 2024-04-26T02:05:58.138 INFO:tasks.workunit.client.0.smithi006.stdout: 10 126925 19.56 MB/sec execute 261 sec latency 88.243 ms 2024-04-26T02:05:59.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:05:58 smithi006 ceph-mon[28419]: pgmap v635: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 792 op/s 2024-04-26T02:05:59.139 INFO:tasks.workunit.client.0.smithi006.stdout: 10 127484 19.59 MB/sec execute 262 sec latency 89.531 ms 2024-04-26T02:05:59.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:05:58 smithi188 ceph-mon[32751]: pgmap v635: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 792 op/s 2024-04-26T02:05:59.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:05:58 smithi029 ceph-mon[32775]: pgmap v635: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 792 op/s 2024-04-26T02:06:00.140 INFO:tasks.workunit.client.0.smithi006.stdout: 10 127911 19.61 MB/sec execute 263 sec latency 307.854 ms 2024-04-26T02:06:01.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:00 smithi006 ceph-mon[28419]: pgmap v636: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.3 MiB/s wr, 519 op/s 2024-04-26T02:06:01.140 INFO:tasks.workunit.client.0.smithi006.stdout: 10 128185 19.59 MB/sec execute 264 sec latency 469.829 ms 2024-04-26T02:06:01.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:00 smithi188 ceph-mon[32751]: pgmap v636: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.3 MiB/s wr, 519 op/s 2024-04-26T02:06:01.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:00 smithi029 ceph-mon[32775]: pgmap v636: 97 pgs: 97 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 8.3 MiB/s wr, 519 op/s 2024-04-26T02:06:02.140 INFO:tasks.workunit.client.0.smithi006.stdout: 10 128185 19.52 MB/sec execute 265 sec latency 1469.940 ms 2024-04-26T02:06:03.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:02 smithi006 ceph-mon[28419]: pgmap v637: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 723 op/s 2024-04-26T02:06:03.140 INFO:tasks.workunit.client.0.smithi006.stdout: 10 128185 19.44 MB/sec execute 266 sec latency 2470.109 ms 2024-04-26T02:06:03.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:02 smithi188 ceph-mon[32751]: pgmap v637: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 723 op/s 2024-04-26T02:06:03.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:02 smithi029 ceph-mon[32775]: pgmap v637: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 723 op/s 2024-04-26T02:06:04.140 INFO:tasks.workunit.client.0.smithi006.stdout: 10 128245 19.38 MB/sec execute 267 sec latency 3391.358 ms 2024-04-26T02:06:05.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:04 smithi006 ceph-mon[28419]: pgmap v638: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.1 MiB/s wr, 500 op/s 2024-04-26T02:06:05.140 INFO:tasks.workunit.client.0.smithi006.stdout: 10 128591 19.34 MB/sec execute 268 sec latency 440.977 ms 2024-04-26T02:06:05.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:04 smithi188 ceph-mon[32751]: pgmap v638: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.1 MiB/s wr, 500 op/s 2024-04-26T02:06:05.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:04 smithi029 ceph-mon[32775]: pgmap v638: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.1 MiB/s wr, 500 op/s 2024-04-26T02:06:06.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:05 smithi006 ceph-mon[28419]: osdmap e128: 12 total, 12 up, 12 in 2024-04-26T02:06:06.140 INFO:tasks.workunit.client.0.smithi006.stdout: 10 128685 19.28 MB/sec execute 269 sec latency 475.086 ms 2024-04-26T02:06:06.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:05 smithi188 ceph-mon[32751]: osdmap e128: 12 total, 12 up, 12 in 2024-04-26T02:06:06.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:05 smithi029 ceph-mon[32775]: osdmap e128: 12 total, 12 up, 12 in 2024-04-26T02:06:07.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:06 smithi006 ceph-mon[28419]: pgmap v640: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.7 MiB/s wr, 600 op/s 2024-04-26T02:06:07.140 INFO:tasks.workunit.client.0.smithi006.stdout: 10 128774 19.24 MB/sec execute 270 sec latency 378.719 ms 2024-04-26T02:06:07.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:06 smithi188 ceph-mon[32751]: pgmap v640: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.7 MiB/s wr, 600 op/s 2024-04-26T02:06:07.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:06 smithi029 ceph-mon[32775]: pgmap v640: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.7 MiB/s wr, 600 op/s 2024-04-26T02:06:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.899+0000 7fd886600640 1 -- 172.21.15.6:0/2397875334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8801043c0 msgr2=0x7fd8801047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.899+0000 7fd886600640 1 --2- 172.21.15.6:0/2397875334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8801043c0 0x7fd8801047a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd870009900 tx=0x7fd87002a410 comp rx=0 tx=0).stop 2024-04-26T02:06:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.899+0000 7fd886600640 1 -- 172.21.15.6:0/2397875334 shutdown_connections 2024-04-26T02:06:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.899+0000 7fd886600640 1 --2- 172.21.15.6:0/2397875334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd880109490 0x7fd880111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.899+0000 7fd886600640 1 --2- 172.21.15.6:0/2397875334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd880104d70 0x7fd880108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.899+0000 7fd886600640 1 --2- 172.21.15.6:0/2397875334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8801043c0 0x7fd8801047a0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.899+0000 7fd886600640 1 -- 172.21.15.6:0/2397875334 >> 172.21.15.6:0/2397875334 conn(0x7fd8800ffc40 msgr2=0x7fd880102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.899+0000 7fd886600640 1 -- 172.21.15.6:0/2397875334 shutdown_connections 2024-04-26T02:06:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.899+0000 7fd886600640 1 -- 172.21.15.6:0/2397875334 wait complete. 2024-04-26T02:06:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.900+0000 7fd886600640 1 Processor -- start 2024-04-26T02:06:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.901+0000 7fd886600640 1 -- start start 2024-04-26T02:06:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.901+0000 7fd886600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8801043c0 0x7fd880076460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.901+0000 7fd886600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd880109490 0x7fd8800769a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.901+0000 7fd886600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd880076ee0 0x7fd8801aaff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.901+0000 7fd886600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd880114440 con 0x7fd8801043c0 2024-04-26T02:06:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.901+0000 7fd886600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801142c0 con 0x7fd880109490 2024-04-26T02:06:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.901+0000 7fd886600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801145c0 con 0x7fd880076ee0 2024-04-26T02:06:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.901+0000 7fd87f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8801043c0 0x7fd880076460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.901+0000 7fd87f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8801043c0 0x7fd880076460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55630/0 (socket says 172.21.15.6:55630) 2024-04-26T02:06:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.901+0000 7fd877e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd880109490 0x7fd8800769a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.902+0000 7fd877e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd880109490 0x7fd8800769a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52662/0 (socket says 172.21.15.6:52662) 2024-04-26T02:06:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.902+0000 7fd877e00640 1 -- 172.21.15.6:0/2188144901 learned_addr learned my addr 172.21.15.6:0/2188144901 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.902+0000 7fd87fe00640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd880076ee0 0x7fd8801aaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.902+0000 7fd877e00640 1 -- 172.21.15.6:0/2188144901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd880076ee0 msgr2=0x7fd8801aaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.902+0000 7fd877e00640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd880076ee0 0x7fd8801aaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.902+0000 7fd877e00640 1 -- 172.21.15.6:0/2188144901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8801043c0 msgr2=0x7fd880076460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.902+0000 7fd877e00640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8801043c0 0x7fd880076460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.902+0000 7fd877e00640 1 -- 172.21.15.6:0/2188144901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88019a790 con 0x7fd880109490 2024-04-26T02:06:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.903+0000 7fd877e00640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd880109490 0x7fd8800769a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd86800ea20 tx=0x7fd86800eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.903+0000 7fd87cc00640 1 -- 172.21.15.6:0/2188144901 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86800cd20 con 0x7fd880109490 2024-04-26T02:06:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.903+0000 7fd87cc00640 1 -- 172.21.15.6:0/2188144901 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8680054d0 con 0x7fd880109490 2024-04-26T02:06:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.903+0000 7fd87cc00640 1 -- 172.21.15.6:0/2188144901 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd868010430 con 0x7fd880109490 2024-04-26T02:06:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.903+0000 7fd886600640 1 -- 172.21.15.6:0/2188144901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88019aa80 con 0x7fd880109490 2024-04-26T02:06:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.903+0000 7fd886600640 1 -- 172.21.15.6:0/2188144901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd88019af80 con 0x7fd880109490 2024-04-26T02:06:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.907+0000 7fd87cc00640 1 -- 172.21.15.6:0/2188144901 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fd8680046c0 con 0x7fd880109490 2024-04-26T02:06:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.908+0000 7fd889416740 1 Processor -- start 2024-04-26T02:06:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.908+0000 7fd889416740 1 -- start start 2024-04-26T02:06:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.909+0000 7fd889416740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8a820 0x55c8daa8ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.909+0000 7fd889416740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8b140 0x55c8daa954c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.909+0000 7fd889416740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa95a00 0x55c8daa97df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.909+0000 7fd889416740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55c8da9f0900 con 0x55c8daa95a00 2024-04-26T02:06:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.909+0000 7fd889416740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c8da996ef0 con 0x55c8daa8b140 2024-04-26T02:06:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.909+0000 7fd889416740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c8da9ec8c0 con 0x55c8daa8a820 2024-04-26T02:06:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.910+0000 7fd87cc00640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd8640a3d10 0x7fd8640a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.910+0000 7fd87cc00640 1 -- 172.21.15.6:0/2188144901 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8644+0+0 (secure 0 0 0) 0x7fd8680d0cd0 con 0x7fd880109490 2024-04-26T02:06:07.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.911+0000 7fd877e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa95a00 0x55c8daa97df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.911+0000 7fd877e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa95a00 0x55c8daa97df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55636/0 (socket says 172.21.15.6:55636) 2024-04-26T02:06:07.910 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.911+0000 7fd877e00640 1 -- 172.21.15.6:0/1228205510 learned_addr learned my addr 172.21.15.6:0/1228205510 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.911+0000 7fd87fe00640 1 --2- 172.21.15.6:0/1228205510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8a820 0x55c8daa8ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.911+0000 7fd87f400640 1 --2- 172.21.15.6:0/1228205510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8b140 0x55c8daa954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.911+0000 7fd87fe00640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd8640a3d10 0x7fd8640a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.912+0000 7fd87fe00640 1 -- 172.21.15.6:0/1228205510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8b140 msgr2=0x55c8daa954c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.912+0000 7fd87fe00640 1 --2- 172.21.15.6:0/1228205510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8b140 0x55c8daa954c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.912+0000 7fd87fe00640 1 -- 172.21.15.6:0/1228205510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa95a00 msgr2=0x55c8daa97df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.912+0000 7fd87fe00640 1 --2- 172.21.15.6:0/1228205510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa95a00 0x55c8daa97df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.912+0000 7fd87fe00640 1 -- 172.21.15.6:0/1228205510 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8da703210 con 0x55c8daa8a820 2024-04-26T02:06:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.912+0000 7fd87fe00640 1 --2- 172.21.15.6:0/1228205510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8a820 0x55c8daa8ac00 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd878005fd0 tx=0x7fd8780107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84b1ecc6679a16e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.912+0000 7fd87fe00640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd8640a3d10 0x7fd8640a61d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd878010ca0 tx=0x7fd878008000 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.913+0000 7fd874c00640 1 -- 172.21.15.6:0/1228205510 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd878012770 con 0x55c8daa8a820 2024-04-26T02:06:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.913+0000 7fd874c00640 1 -- 172.21.15.6:0/1228205510 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd878012d20 con 0x55c8daa8a820 2024-04-26T02:06:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.913+0000 7fd874c00640 1 -- 172.21.15.6:0/1228205510 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd878026790 con 0x55c8daa8a820 2024-04-26T02:06:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.914+0000 7fd889416740 1 -- 172.21.15.6:0/1228205510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8a820 msgr2=0x55c8daa8ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.914+0000 7fd889416740 1 --2- 172.21.15.6:0/1228205510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8a820 0x55c8daa8ac00 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd878005fd0 tx=0x7fd8780107a0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.914+0000 7fd889416740 1 -- 172.21.15.6:0/1228205510 shutdown_connections 2024-04-26T02:06:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.914+0000 7fd889416740 1 --2- 172.21.15.6:0/1228205510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa95a00 0x55c8daa97df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.914+0000 7fd889416740 1 --2- 172.21.15.6:0/1228205510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8b140 0x55c8daa954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.914+0000 7fd889416740 1 --2- 172.21.15.6:0/1228205510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8a820 0x55c8daa8ac00 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.914+0000 7fd889416740 1 -- 172.21.15.6:0/1228205510 >> 172.21.15.6:0/1228205510 conn(0x55c8daa86d30 msgr2=0x55c8daa87110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.915+0000 7fd889416740 1 -- 172.21.15.6:0/1228205510 shutdown_connections 2024-04-26T02:06:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.915+0000 7fd889416740 1 -- 172.21.15.6:0/1228205510 wait complete. 2024-04-26T02:06:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.915+0000 7fd889416740 1 -- start start 2024-04-26T02:06:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.916+0000 7fd889416740 1 Processor -- start 2024-04-26T02:06:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.916+0000 7fd889416740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa8a820 0x55c8daa929d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.917+0000 7fd889416740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8b140 0x55c8daa92f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.917+0000 7fd889416740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa95a00 0x55c8daa93450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.917+0000 7fd889416740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55c8daa985d0 con 0x55c8daa8a820 2024-04-26T02:06:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.917+0000 7fd87fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa95a00 0x55c8daa93450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.917+0000 7fd87fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa95a00 0x55c8daa93450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51676/0 (socket says 172.21.15.6:51676) 2024-04-26T02:06:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.917+0000 7fd87fe00640 1 -- 172.21.15.6:0/654589968 learned_addr learned my addr 172.21.15.6:0/654589968 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.917+0000 7fd877e00640 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8b140 0x55c8daa92f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.917+0000 7fd889416740 1 -- 172.21.15.6:0/654589968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c8da9aa310 con 0x55c8daa8b140 2024-04-26T02:06:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.917+0000 7fd889416740 1 -- 172.21.15.6:0/654589968 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c8daa98750 con 0x55c8daa95a00 2024-04-26T02:06:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.918+0000 7fd87f400640 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa8a820 0x55c8daa929d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.918+0000 7fd877e00640 1 -- 172.21.15.6:0/654589968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa95a00 msgr2=0x55c8daa93450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.918+0000 7fd877e00640 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa95a00 0x55c8daa93450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.918+0000 7fd877e00640 1 -- 172.21.15.6:0/654589968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa8a820 msgr2=0x55c8daa929d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.918+0000 7fd877e00640 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa8a820 0x55c8daa929d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.918+0000 7fd877e00640 1 -- 172.21.15.6:0/654589968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8da7cf620 con 0x55c8daa8b140 2024-04-26T02:06:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.918+0000 7fd87fe00640 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa95a00 0x55c8daa93450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:06:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.918+0000 7fd877e00640 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8b140 0x55c8daa92f10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd86809c890 tx=0x7fd86809c5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6f7d86b7766fe91 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.918+0000 7fd86d600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:06:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.919+0000 7fd86d600640 1 -- 172.21.15.6:0/654589968 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8680a0f40 con 0x55c8daa8b140 2024-04-26T02:06:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.919+0000 7fd86d600640 1 -- 172.21.15.6:0/654589968 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 8644+0+0 (secure 0 0 0) 0x7fd868099200 con 0x55c8daa8b140 2024-04-26T02:06:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.919+0000 7fd889416740 10 client.0 _ll_register_callbacks cb 0x55c8da978cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.919+0000 7fd889416740 1 -- 172.21.15.6:0/654589968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd878012d20 con 0x55c8daa8b140 2024-04-26T02:06:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.919+0000 7fd86d600640 1 -- 172.21.15.6:0/654589968 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fd8680a5800 con 0x55c8daa8b140 2024-04-26T02:06:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.919+0000 7fd889416740 10 client.27452 fetch_fsmap learned FSMap version 55 2024-04-26T02:06:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.919+0000 7fd889416740 1 -- 172.21.15.6:0/654589968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=55}) -- 0x55c8daaa46a0 con 0x55c8daa8b140 2024-04-26T02:06:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.920+0000 7fd86d600640 1 -- 172.21.15.6:0/654589968 <== mon.2 v2:172.21.15.29:3300/0 4 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7fd8680a5370 con 0x55c8daa8b140 2024-04-26T02:06:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.920+0000 7fd889416740 10 client.27452 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:06:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.920+0000 7fd889416740 10 client.27452 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:06:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.920+0000 7fd889416740 20 client.27452 populate_metadata read hostname 'smithi006' 2024-04-26T02:06:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.920+0000 7fd889416740 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c8daaa5a50 0x55c8daaa7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.920+0000 7fd889416740 4 client.27452 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:06:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.920+0000 7fd889416740 1 -- 172.21.15.6:0/654589968 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8daa048b0 con 0x55c8daaa5a50 2024-04-26T02:06:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.920+0000 7fd87f400640 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c8daaa5a50 0x55c8daaa7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.921+0000 7fd87f400640 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c8daaa5a50 0x55c8daaa7ea0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e618ee56452ea504 server_cookie=7d5b6297ffe010c4 in_seq=0 out_seq=0 2024-04-26T02:06:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.921+0000 7fd86d600640 10 client.27452 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.922+0000 7fd86d600640 1 -- 172.21.15.6:0/654589968 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8daa048b0 con 0x55c8daaa5a50 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.922+0000 7fd86d600640 10 client.27452 handle_command_reply: tid=0 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.922+0000 7fd889416740 1 client.27452 shutdown 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.922+0000 7fd889416740 1 -- 172.21.15.6:0/654589968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8b140 msgr2=0x55c8daa92f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.922+0000 7fd889416740 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8b140 0x55c8daa92f10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd86809c890 tx=0x7fd86809c5b0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.923+0000 7fd889416740 1 -- 172.21.15.6:0/654589968 shutdown_connections 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.923+0000 7fd889416740 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c8daaa5a50 0x55c8daaa7ea0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.923+0000 7fd889416740 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa95a00 0x55c8daa93450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.923+0000 7fd889416740 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8b140 0x55c8daa92f10 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.923+0000 7fd889416740 1 --2- 172.21.15.6:0/654589968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa8a820 0x55c8daa929d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.923+0000 7fd889416740 1 -- 172.21.15.6:0/654589968 >> 172.21.15.6:0/654589968 conn(0x55c8daa86d30 msgr2=0x55c8daa8cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.923+0000 7fd86d600640 0 client.27452 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.923+0000 7fd889416740 1 -- 172.21.15.6:0/654589968 shutdown_connections 2024-04-26T02:06:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.923+0000 7fd889416740 1 -- 172.21.15.6:0/654589968 wait complete. 2024-04-26T02:06:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.923+0000 7fd889416740 20 client.27452 trim_cache size 0 max 16384 2024-04-26T02:06:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.950+0000 7fd889416740 1 Processor -- start 2024-04-26T02:06:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.950+0000 7fd889416740 1 -- start start 2024-04-26T02:06:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.950+0000 7fd889416740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa8a820 0x55c8daaa1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.950+0000 7fd889416740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 0x55c8daaad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.950+0000 7fd889416740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa95a00 0x55c8daaadd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.950+0000 7fd889416740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801bf5d0 con 0x55c8daa8a820 2024-04-26T02:06:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.950+0000 7fd889416740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801bce40 con 0x55c8daa95a00 2024-04-26T02:06:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.950+0000 7fd889416740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801a1710 con 0x55c8daa8b140 2024-04-26T02:06:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.951+0000 7fd87f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa8a820 0x55c8daaa1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.951+0000 7fd877e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 0x55c8daaad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.951+0000 7fd877e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 0x55c8daaad800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:51692/0 (socket says 172.21.15.6:51692) 2024-04-26T02:06:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.951+0000 7fd877e00640 1 -- 172.21.15.6:0/2453810506 learned_addr learned my addr 172.21.15.6:0/2453810506 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.951+0000 7fd87fe00640 1 --2- 172.21.15.6:0/2453810506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa95a00 0x55c8daaadd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.951+0000 7fd877e00640 1 -- 172.21.15.6:0/2453810506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa95a00 msgr2=0x55c8daaadd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.951+0000 7fd877e00640 1 --2- 172.21.15.6:0/2453810506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa95a00 0x55c8daaadd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.951+0000 7fd877e00640 1 -- 172.21.15.6:0/2453810506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa8a820 msgr2=0x55c8daaa1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.951+0000 7fd877e00640 1 --2- 172.21.15.6:0/2453810506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa8a820 0x55c8daaa1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.951+0000 7fd877e00640 1 -- 172.21.15.6:0/2453810506 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8daaa2970 con 0x55c8daa8b140 2024-04-26T02:06:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.951+0000 7fd87f400640 1 --2- 172.21.15.6:0/2453810506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa8a820 0x55c8daaa1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:06:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.952+0000 7fd877e00640 1 --2- 172.21.15.6:0/2453810506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 0x55c8daaad800 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd868099560 tx=0x7fd8680d0000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fa59f464662762d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.952+0000 7fd86fe00640 1 -- 172.21.15.6:0/2453810506 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8680a0f40 con 0x55c8daa8b140 2024-04-26T02:06:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.952+0000 7fd86fe00640 1 -- 172.21.15.6:0/2453810506 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd878012d20 con 0x55c8daa8b140 2024-04-26T02:06:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.952+0000 7fd86fe00640 1 -- 172.21.15.6:0/2453810506 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86809c200 con 0x55c8daa8b140 2024-04-26T02:06:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.953+0000 7fd889416740 1 -- 172.21.15.6:0/2453810506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 msgr2=0x55c8daaad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.953+0000 7fd889416740 1 --2- 172.21.15.6:0/2453810506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 0x55c8daaad800 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd868099560 tx=0x7fd8680d0000 comp rx=0 tx=0).stop 2024-04-26T02:06:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.953+0000 7fd889416740 1 -- 172.21.15.6:0/2453810506 shutdown_connections 2024-04-26T02:06:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.953+0000 7fd889416740 1 --2- 172.21.15.6:0/2453810506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa95a00 0x55c8daaadd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.953+0000 7fd889416740 1 --2- 172.21.15.6:0/2453810506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 0x55c8daaad800 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.953+0000 7fd889416740 1 --2- 172.21.15.6:0/2453810506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa8a820 0x55c8daaa1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.953+0000 7fd889416740 1 -- 172.21.15.6:0/2453810506 >> 172.21.15.6:0/2453810506 conn(0x55c8daa86d30 msgr2=0x55c8daa87110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.953+0000 7fd889416740 1 -- 172.21.15.6:0/2453810506 shutdown_connections 2024-04-26T02:06:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.953+0000 7fd889416740 1 -- 172.21.15.6:0/2453810506 wait complete. 2024-04-26T02:06:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.954+0000 7fd889416740 1 -- start start 2024-04-26T02:06:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.954+0000 7fd889416740 1 Processor -- start 2024-04-26T02:06:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.954+0000 7fd889416740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8a820 0x55c8daa9af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.954+0000 7fd889416740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 0x55c8daa9b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.954+0000 7fd889416740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa95a00 0x55c8dab03030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.954+0000 7fd889416740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801a1d60 con 0x55c8daa95a00 2024-04-26T02:06:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.954+0000 7fd889416740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801bc770 con 0x55c8daa8a820 2024-04-26T02:06:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.954+0000 7fd889416740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801a2430 con 0x55c8daa8b140 2024-04-26T02:06:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.954+0000 7fd889416740 10 client.0 _ll_register_callbacks cb 0x7fd86809c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.955+0000 7fd87f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8a820 0x55c8daa9af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.955+0000 7fd87f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8a820 0x55c8daa9af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:52712/0 (socket says 172.21.15.6:52712) 2024-04-26T02:06:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.955+0000 7fd87f400640 1 -- 172.21.15.6:0/983666875 learned_addr learned my addr 172.21.15.6:0/983666875 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.955+0000 7fd87fe00640 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa95a00 0x55c8dab03030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.955+0000 7fd877e00640 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 0x55c8daa9b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.955+0000 7fd87f400640 1 -- 172.21.15.6:0/983666875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 msgr2=0x55c8daa9b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.955+0000 7fd87f400640 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 0x55c8daa9b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.955+0000 7fd87f400640 1 -- 172.21.15.6:0/983666875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa95a00 msgr2=0x55c8dab03030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.956+0000 7fd87f400640 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa95a00 0x55c8dab03030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.956+0000 7fd87f400640 1 -- 172.21.15.6:0/983666875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8dab03670 con 0x55c8daa8a820 2024-04-26T02:06:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.956+0000 7fd877e00640 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 0x55c8daa9b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:06:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.956+0000 7fd87fe00640 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa95a00 0x55c8dab03030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:06:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.956+0000 7fd87f400640 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8a820 0x55c8daa9af70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd870005ab0 tx=0x7fd8700044e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3ef0424f2397b2a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.956+0000 7fd86d600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:06:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.957+0000 7fd86d600640 1 -- 172.21.15.6:0/983666875 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd870035040 con 0x55c8daa8a820 2024-04-26T02:06:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.957+0000 7fd86d600640 1 -- 172.21.15.6:0/983666875 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 8644+0+0 (secure 0 0 0) 0x7fd870005240 con 0x55c8daa8a820 2024-04-26T02:06:07.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.957+0000 7fd889416740 1 -- 172.21.15.6:0/983666875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd878012d20 con 0x55c8daa8a820 2024-04-26T02:06:07.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.958+0000 7fd86d600640 1 -- 172.21.15.6:0/983666875 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fd870005440 con 0x55c8daa8a820 2024-04-26T02:06:07.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.958+0000 7fd889416740 10 client.27458 fetch_fsmap learned FSMap version 55 2024-04-26T02:06:07.966 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.958+0000 7fd889416740 1 -- 172.21.15.6:0/983666875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55c8dab0bc90 con 0x55c8daa8a820 2024-04-26T02:06:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.959+0000 7fd86d600640 1 -- 172.21.15.6:0/983666875 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd878012d20 con 0x55c8daa8a820 2024-04-26T02:06:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.959+0000 7fd86d600640 1 -- 172.21.15.6:0/983666875 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7fd8700049f0 con 0x55c8daa8a820 2024-04-26T02:06:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.959+0000 7fd889416740 10 client.27458 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:06:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.959+0000 7fd889416740 10 client.27458 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:06:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.959+0000 7fd889416740 20 client.27458 populate_metadata read hostname 'smithi006' 2024-04-26T02:06:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.959+0000 7fd889416740 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c8dab0c390 0x55c8dab0e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.959+0000 7fd889416740 4 client.27458 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:06:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.959+0000 7fd889416740 1 -- 172.21.15.6:0/983666875 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c8da9fe320 con 0x55c8dab0c390 2024-04-26T02:06:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.960+0000 7fd877e00640 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c8dab0c390 0x55c8dab0e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.961+0000 7fd877e00640 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c8dab0c390 0x55c8dab0e7b0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4722867b3aceab server_cookie=b3e04ed3c7300afb in_seq=0 out_seq=0 2024-04-26T02:06:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:07.961+0000 7fd86d600640 10 client.27458 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:08.140 INFO:tasks.workunit.client.0.smithi006.stdout: 10 129131 19.24 MB/sec execute 271 sec latency 167.274 ms 2024-04-26T02:06:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.181+0000 7fd86d600640 1 -- 172.21.15.6:0/983666875 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c8da9fe320 con 0x55c8dab0c390 2024-04-26T02:06:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.181+0000 7fd86d600640 10 client.27458 handle_command_reply: tid=0 2024-04-26T02:06:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.181+0000 7fd889416740 1 client.27458 shutdown 2024-04-26T02:06:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.181+0000 7fd889416740 1 -- 172.21.15.6:0/983666875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8a820 msgr2=0x55c8daa9af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.181+0000 7fd889416740 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8a820 0x55c8daa9af70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd870005ab0 tx=0x7fd8700044e0 comp rx=0 tx=0).stop 2024-04-26T02:06:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.182+0000 7fd889416740 1 -- 172.21.15.6:0/983666875 shutdown_connections 2024-04-26T02:06:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.182+0000 7fd889416740 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55c8dab0c390 0x55c8dab0e7b0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.182+0000 7fd889416740 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55c8daa95a00 0x55c8dab03030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.182+0000 7fd889416740 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c8daa8b140 0x55c8daa9b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.182+0000 7fd889416740 1 --2- 172.21.15.6:0/983666875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8daa8a820 0x55c8daa9af70 secure :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd870005ab0 tx=0x7fd8700044e0 comp rx=0 tx=0).stop 2024-04-26T02:06:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.182+0000 7fd889416740 1 -- 172.21.15.6:0/983666875 >> 172.21.15.6:0/983666875 conn(0x55c8daa86d30 msgr2=0x55c8daa8c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.182+0000 7fd86d600640 0 client.27458 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.182+0000 7fd889416740 1 -- 172.21.15.6:0/983666875 shutdown_connections 2024-04-26T02:06:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.182+0000 7fd889416740 1 -- 172.21.15.6:0/983666875 wait complete. 2024-04-26T02:06:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.182+0000 7fd889416740 20 client.27458 trim_cache size 0 max 16384 2024-04-26T02:06:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:06:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:06:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:06:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:06:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.184+0000 7fd886600640 1 -- 172.21.15.6:0/2188144901 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd8640a3d10 msgr2=0x7fd8640a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.184+0000 7fd886600640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd8640a3d10 0x7fd8640a61d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd878010ca0 tx=0x7fd878008000 comp rx=0 tx=0).stop 2024-04-26T02:06:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.185+0000 7fd886600640 1 -- 172.21.15.6:0/2188144901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd880109490 msgr2=0x7fd8800769a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.185+0000 7fd886600640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd880109490 0x7fd8800769a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd86800ea20 tx=0x7fd86800eee0 comp rx=0 tx=0).stop 2024-04-26T02:06:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.185+0000 7fd886600640 1 -- 172.21.15.6:0/2188144901 shutdown_connections 2024-04-26T02:06:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.185+0000 7fd886600640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd8640a3d10 0x7fd8640a61d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.185+0000 7fd886600640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd880076ee0 0x7fd8801aaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.185+0000 7fd886600640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd880109490 0x7fd8800769a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.185+0000 7fd886600640 1 --2- 172.21.15.6:0/2188144901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8801043c0 0x7fd880076460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.185+0000 7fd886600640 1 -- 172.21.15.6:0/2188144901 >> 172.21.15.6:0/2188144901 conn(0x7fd8800ffc40 msgr2=0x7fd880101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.186+0000 7fd886600640 1 -- 172.21.15.6:0/2188144901 shutdown_connections 2024-04-26T02:06:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:08.186+0000 7fd886600640 1 -- 172.21.15.6:0/2188144901 wait complete. 2024-04-26T02:06:09.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:08 smithi006 ceph-mon[28419]: pgmap v641: 97 pgs: 12 active+clean+snaptrim, 55 active+clean, 30 active+clean+snaptrim_wait; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.5 MiB/s wr, 448 op/s 2024-04-26T02:06:09.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:08 smithi188 ceph-mon[32751]: pgmap v641: 97 pgs: 12 active+clean+snaptrim, 55 active+clean, 30 active+clean+snaptrim_wait; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.5 MiB/s wr, 448 op/s 2024-04-26T02:06:09.141 INFO:tasks.workunit.client.0.smithi006.stdout: 10 129805 19.34 MB/sec execute 272 sec latency 170.997 ms 2024-04-26T02:06:09.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:08 smithi029 ceph-mon[32775]: pgmap v641: 97 pgs: 12 active+clean+snaptrim, 55 active+clean, 30 active+clean+snaptrim_wait; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.5 MiB/s wr, 448 op/s 2024-04-26T02:06:10.141 INFO:tasks.workunit.client.0.smithi006.stdout: 10 130295 19.37 MB/sec execute 273 sec latency 125.128 ms 2024-04-26T02:06:11.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:10 smithi006 ceph-mon[28419]: pgmap v642: 97 pgs: 12 active+clean+snaptrim, 55 active+clean, 30 active+clean+snaptrim_wait; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.5 MiB/s wr, 448 op/s 2024-04-26T02:06:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:10 smithi188 ceph-mon[32751]: pgmap v642: 97 pgs: 12 active+clean+snaptrim, 55 active+clean, 30 active+clean+snaptrim_wait; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.5 MiB/s wr, 448 op/s 2024-04-26T02:06:11.141 INFO:tasks.workunit.client.0.smithi006.stdout: 10 130783 19.39 MB/sec execute 274 sec latency 63.314 ms 2024-04-26T02:06:11.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:10 smithi029 ceph-mon[32775]: pgmap v642: 97 pgs: 12 active+clean+snaptrim, 55 active+clean, 30 active+clean+snaptrim_wait; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.5 MiB/s wr, 448 op/s 2024-04-26T02:06:12.141 INFO:tasks.workunit.client.0.smithi006.stdout: 10 131142 19.39 MB/sec execute 275 sec latency 158.022 ms 2024-04-26T02:06:13.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:12 smithi006 ceph-mon[28419]: pgmap v643: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.5 MiB/s wr, 651 op/s 2024-04-26T02:06:13.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:12 smithi188 ceph-mon[32751]: pgmap v643: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.5 MiB/s wr, 651 op/s 2024-04-26T02:06:13.141 INFO:tasks.workunit.client.0.smithi006.stdout: 10 131607 19.42 MB/sec execute 276 sec latency 139.855 ms 2024-04-26T02:06:13.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:12 smithi029 ceph-mon[32775]: pgmap v643: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.5 MiB/s wr, 651 op/s 2024-04-26T02:06:14.141 INFO:tasks.workunit.client.0.smithi006.stdout: 10 132107 19.43 MB/sec execute 277 sec latency 141.847 ms 2024-04-26T02:06:15.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:14 smithi006 ceph-mon[28419]: pgmap v644: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.5 MiB/s wr, 651 op/s 2024-04-26T02:06:15.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:14 smithi188 ceph-mon[32751]: pgmap v644: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.5 MiB/s wr, 651 op/s 2024-04-26T02:06:15.141 INFO:tasks.workunit.client.0.smithi006.stdout: 10 132559 19.43 MB/sec execute 278 sec latency 129.745 ms 2024-04-26T02:06:15.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:14 smithi029 ceph-mon[32775]: pgmap v644: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.5 MiB/s wr, 651 op/s 2024-04-26T02:06:16.141 INFO:tasks.workunit.client.0.smithi006.stdout: 10 133216 19.51 MB/sec execute 279 sec latency 142.195 ms 2024-04-26T02:06:16.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:16 smithi188 ceph-mon[32751]: pgmap v645: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.0 MiB/s wr, 604 op/s 2024-04-26T02:06:16.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:16 smithi188 ceph-mon[32751]: osdmap e129: 12 total, 12 up, 12 in 2024-04-26T02:06:16.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:16 smithi029 ceph-mon[32775]: pgmap v645: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.0 MiB/s wr, 604 op/s 2024-04-26T02:06:16.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:16 smithi029 ceph-mon[32775]: osdmap e129: 12 total, 12 up, 12 in 2024-04-26T02:06:17.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:16 smithi006 ceph-mon[28419]: pgmap v645: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.0 MiB/s wr, 604 op/s 2024-04-26T02:06:17.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:16 smithi006 ceph-mon[28419]: osdmap e129: 12 total, 12 up, 12 in 2024-04-26T02:06:17.141 INFO:tasks.workunit.client.0.smithi006.stdout: 10 133614 19.52 MB/sec execute 280 sec latency 274.217 ms 2024-04-26T02:06:18.141 INFO:tasks.workunit.client.0.smithi006.stdout: 10 134108 19.55 MB/sec execute 281 sec latency 289.251 ms 2024-04-26T02:06:18.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:18 smithi188 ceph-mon[32751]: pgmap v647: 97 pgs: 97 active+clean; 1.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 12 MiB/s wr, 702 op/s 2024-04-26T02:06:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:18 smithi029 ceph-mon[32775]: pgmap v647: 97 pgs: 97 active+clean; 1.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 12 MiB/s wr, 702 op/s 2024-04-26T02:06:19.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:18 smithi006 ceph-mon[28419]: pgmap v647: 97 pgs: 97 active+clean; 1.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 12 MiB/s wr, 702 op/s 2024-04-26T02:06:19.141 INFO:tasks.workunit.client.0.smithi006.stdout: 10 134557 19.56 MB/sec execute 282 sec latency 104.943 ms 2024-04-26T02:06:20.143 INFO:tasks.workunit.client.0.smithi006.stdout: 10 134995 19.56 MB/sec execute 283 sec latency 210.381 ms 2024-04-26T02:06:20.873 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:20 smithi006 ceph-mon[28419]: pgmap v648: 97 pgs: 97 active+clean; 1.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 12 MiB/s wr, 702 op/s 2024-04-26T02:06:20.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:20 smithi188 ceph-mon[32751]: pgmap v648: 97 pgs: 97 active+clean; 1.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 12 MiB/s wr, 702 op/s 2024-04-26T02:06:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:20 smithi029 ceph-mon[32775]: pgmap v648: 97 pgs: 97 active+clean; 1.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 12 MiB/s wr, 702 op/s 2024-04-26T02:06:21.142 INFO:tasks.workunit.client.0.smithi006.stdout: 10 135528 19.60 MB/sec execute 284 sec latency 318.140 ms 2024-04-26T02:06:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.349+0000 7fa2a5600640 1 -- 172.21.15.6:0/3490440253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a006a880 msgr2=0x7fa2a006ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.349+0000 7fa2a5600640 1 --2- 172.21.15.6:0/3490440253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a006a880 0x7fa2a006ac60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa290009900 tx=0x7fa29002a400 comp rx=0 tx=0).stop 2024-04-26T02:06:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.349+0000 7fa2a5600640 1 -- 172.21.15.6:0/3490440253 shutdown_connections 2024-04-26T02:06:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.349+0000 7fa2a5600640 1 --2- 172.21.15.6:0/3490440253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2a0107320 0x7fa2a010eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.349+0000 7fa2a5600640 1 --2- 172.21.15.6:0/3490440253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2a006b230 0x7fa2a0106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.349+0000 7fa2a5600640 1 --2- 172.21.15.6:0/3490440253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a006a880 0x7fa2a006ac60 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.349+0000 7fa2a5600640 1 -- 172.21.15.6:0/3490440253 >> 172.21.15.6:0/3490440253 conn(0x7fa2a0077cd0 msgr2=0x7fa2a00780e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.349+0000 7fa2a5600640 1 -- 172.21.15.6:0/3490440253 shutdown_connections 2024-04-26T02:06:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.349+0000 7fa2a5600640 1 -- 172.21.15.6:0/3490440253 wait complete. 2024-04-26T02:06:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.350+0000 7fa2a5600640 1 Processor -- start 2024-04-26T02:06:21.349 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.350+0000 7fa2a5600640 1 -- start start 2024-04-26T02:06:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.351+0000 7fa2a5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2a006a880 0x7fa2a0101170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.351+0000 7fa2a5600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2a0107320 0x7fa2a01016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.351+0000 7fa2a5600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0101bf0 0x7fa2a01a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.351+0000 7fa2a5600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a0112260 con 0x7fa2a006a880 2024-04-26T02:06:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.351+0000 7fa2a5600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a01120e0 con 0x7fa2a0101bf0 2024-04-26T02:06:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.351+0000 7fa2a5600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a01123e0 con 0x7fa2a0107320 2024-04-26T02:06:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.351+0000 7fa29ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2a006a880 0x7fa2a0101170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.351+0000 7fa29ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2a006a880 0x7fa2a0101170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57472/0 (socket says 172.21.15.6:57472) 2024-04-26T02:06:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.351+0000 7fa29ea00640 1 -- 172.21.15.6:0/1891703618 learned_addr learned my addr 172.21.15.6:0/1891703618 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.351+0000 7fa29f400640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0101bf0 0x7fa2a01a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.351+0000 7fa29e000640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2a0107320 0x7fa2a01016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.352+0000 7fa29f400640 1 -- 172.21.15.6:0/1891703618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2a0107320 msgr2=0x7fa2a01016b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.352+0000 7fa29f400640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2a0107320 0x7fa2a01016b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.352+0000 7fa29f400640 1 -- 172.21.15.6:0/1891703618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2a006a880 msgr2=0x7fa2a0101170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.352+0000 7fa29f400640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2a006a880 0x7fa2a0101170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.352+0000 7fa29f400640 1 -- 172.21.15.6:0/1891703618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2a01a6f50 con 0x7fa2a0101bf0 2024-04-26T02:06:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.352+0000 7fa29ea00640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2a006a880 0x7fa2a0101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:06:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.352+0000 7fa29f400640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0101bf0 0x7fa2a01a6850 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa29800ebc0 tx=0x7fa298009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.353+0000 7fa297400640 1 -- 172.21.15.6:0/1891703618 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa298003970 con 0x7fa2a0101bf0 2024-04-26T02:06:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.353+0000 7fa297400640 1 -- 172.21.15.6:0/1891703618 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa298004020 con 0x7fa2a0101bf0 2024-04-26T02:06:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.353+0000 7fa297400640 1 -- 172.21.15.6:0/1891703618 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa298010690 con 0x7fa2a0101bf0 2024-04-26T02:06:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.353+0000 7fa2a5600640 1 -- 172.21.15.6:0/1891703618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2a01a7240 con 0x7fa2a0101bf0 2024-04-26T02:06:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.353+0000 7fa2a5600640 1 -- 172.21.15.6:0/1891703618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2a01a7740 con 0x7fa2a0101bf0 2024-04-26T02:06:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.356+0000 7fa297400640 1 -- 172.21.15.6:0/1891703618 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fa298003b10 con 0x7fa2a0101bf0 2024-04-26T02:06:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.357+0000 7fa2a83ac740 1 Processor -- start 2024-04-26T02:06:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.357+0000 7fa2a83ac740 1 -- start start 2024-04-26T02:06:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa2a83ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 0x55b8ae68e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa2a83ac740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68e950 0x55b8ae698cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa2a83ac740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae699210 0x55b8ae69b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa2a83ac740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55b8ae5eb900 con 0x55b8ae699210 2024-04-26T02:06:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa2a83ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b8ae591ef0 con 0x55b8ae68e030 2024-04-26T02:06:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa2a83ac740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b8ae5e78c0 con 0x55b8ae68e950 2024-04-26T02:06:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa297400640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa2840a3dc0 0x7fa2840a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa29ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68e950 0x55b8ae698cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa29e000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 0x55b8ae68e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa29ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68e950 0x55b8ae698cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:45040/0 (socket says 172.21.15.6:45040) 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa29ea00640 1 -- 172.21.15.6:0/538989702 learned_addr learned my addr 172.21.15.6:0/538989702 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa297400640 1 -- 172.21.15.6:0/1891703618 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8600+0+0 (secure 0 0 0) 0x7fa2980d0e00 con 0x7fa2a0101bf0 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa29e000640 1 --2- 172.21.15.6:0/538989702 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae699210 0x55b8ae69b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.358+0000 7fa29f400640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa2840a3dc0 0x7fa2840a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.359+0000 7fa29e000640 1 -- 172.21.15.6:0/538989702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68e950 msgr2=0x55b8ae698cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.359+0000 7fa29e000640 1 --2- 172.21.15.6:0/538989702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68e950 0x55b8ae698cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.359+0000 7fa29e000640 1 -- 172.21.15.6:0/538989702 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae699210 msgr2=0x55b8ae69b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.359+0000 7fa29e000640 1 --2- 172.21.15.6:0/538989702 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae699210 0x55b8ae69b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.359+0000 7fa29e000640 1 -- 172.21.15.6:0/538989702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8ae2fe210 con 0x55b8ae68e030 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.359+0000 7fa29ea00640 1 --2- 172.21.15.6:0/538989702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68e950 0x55b8ae698cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.359+0000 7fa29e000640 1 --2- 172.21.15.6:0/538989702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 0x55b8ae68e410 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0102c00 tx=0x7fa28c00a770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a8297a4bd5b0a9b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.359+0000 7fa28b400640 1 -- 172.21.15.6:0/538989702 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c005220 con 0x55b8ae68e030 2024-04-26T02:06:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.359+0000 7fa28b400640 1 -- 172.21.15.6:0/538989702 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa28c004c50 con 0x55b8ae68e030 2024-04-26T02:06:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.359+0000 7fa28b400640 1 -- 172.21.15.6:0/538989702 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c00fa70 con 0x55b8ae68e030 2024-04-26T02:06:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.360+0000 7fa2a83ac740 1 -- 172.21.15.6:0/538989702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 msgr2=0x55b8ae68e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.360+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/538989702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 0x55b8ae68e410 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0102c00 tx=0x7fa28c00a770 comp rx=0 tx=0).stop 2024-04-26T02:06:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.360+0000 7fa29f400640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa2840a3dc0 0x7fa2840a6280 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa29800cfd0 tx=0x7fa29800cf40 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.360+0000 7fa2a83ac740 1 -- 172.21.15.6:0/538989702 shutdown_connections 2024-04-26T02:06:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.360+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/538989702 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae699210 0x55b8ae69b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.360+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/538989702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68e950 0x55b8ae698cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.360+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/538989702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 0x55b8ae68e410 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.360+0000 7fa2a83ac740 1 -- 172.21.15.6:0/538989702 >> 172.21.15.6:0/538989702 conn(0x55b8ae68a540 msgr2=0x55b8ae68a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.360+0000 7fa2a83ac740 1 -- 172.21.15.6:0/538989702 shutdown_connections 2024-04-26T02:06:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.360+0000 7fa2a83ac740 1 -- 172.21.15.6:0/538989702 wait complete. 2024-04-26T02:06:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.361+0000 7fa2a83ac740 1 -- start start 2024-04-26T02:06:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.361+0000 7fa2a83ac740 1 Processor -- start 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa2a83ac740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e030 0x55b8ae6961a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa2a83ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e950 0x55b8ae6966e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa2a83ac740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae699210 0x55b8ae696c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa2a83ac740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55b8ae69be40 con 0x55b8ae68e030 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa2a83ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b8ae5a5310 con 0x55b8ae68e950 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa2a83ac740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b8ae69bfc0 con 0x55b8ae699210 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa2a83ac740 10 client.0 _ll_register_callbacks cb 0x55b8ae573cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa29ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e950 0x55b8ae6966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa29ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e950 0x55b8ae6966e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:51010/0 (socket says 172.21.15.6:51010) 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa29ea00640 1 -- 172.21.15.6:0/3996553915 learned_addr learned my addr 172.21.15.6:0/3996553915 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa29e000640 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae699210 0x55b8ae696c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa29e000640 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e030 0x55b8ae6961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa29ea00640 1 -- 172.21.15.6:0/3996553915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae699210 msgr2=0x55b8ae696c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa29ea00640 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae699210 0x55b8ae696c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa29ea00640 1 -- 172.21.15.6:0/3996553915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e030 msgr2=0x55b8ae6961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa29ea00640 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e030 0x55b8ae6961a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa29ea00640 1 -- 172.21.15.6:0/3996553915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8ae3ca620 con 0x55b8ae68e950 2024-04-26T02:06:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.362+0000 7fa29e000640 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e030 0x55b8ae6961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:06:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.363+0000 7fa29ea00640 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e950 0x55b8ae6966e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa290000c00 tx=0x7fa290003890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8872b083a17ca804 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.363+0000 7fa27fe00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:06:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.363+0000 7fa27fe00640 1 -- 172.21.15.6:0/3996553915 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa290035040 con 0x55b8ae68e950 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.363+0000 7fa27fe00640 1 -- 172.21.15.6:0/3996553915 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 8600+0+0 (secure 0 0 0) 0x7fa2900314a0 con 0x55b8ae68e950 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.363+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3996553915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa28c004c50 con 0x55b8ae68e950 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.364+0000 7fa27fe00640 1 -- 172.21.15.6:0/3996553915 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fa2900316a0 con 0x55b8ae68e950 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.364+0000 7fa2a83ac740 10 client.27470 fetch_fsmap learned FSMap version 55 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.364+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3996553915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55b8ae6a80a0 con 0x55b8ae68e950 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.364+0000 7fa27fe00640 1 -- 172.21.15.6:0/3996553915 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa28c004c50 con 0x55b8ae68e950 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.364+0000 7fa27fe00640 1 -- 172.21.15.6:0/3996553915 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7fa290046970 con 0x55b8ae68e950 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.364+0000 7fa2a83ac740 10 client.27470 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.364+0000 7fa2a83ac740 10 client.27470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.364+0000 7fa2a83ac740 20 client.27470 populate_metadata read hostname 'smithi006' 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.364+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8ae6a9560 0x55b8ae6ab970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.364+0000 7fa2a83ac740 4 client.27470 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.364+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3996553915 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8ae5ff8b0 con 0x55b8ae6a9560 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.365+0000 7fa29e000640 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8ae6a9560 0x55b8ae6ab970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.365+0000 7fa29e000640 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8ae6a9560 0x55b8ae6ab970 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=970e35bcef9da61f server_cookie=d26e4dcb45b507aa in_seq=0 out_seq=0 2024-04-26T02:06:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.365+0000 7fa27fe00640 10 client.27470 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa27fe00640 1 -- 172.21.15.6:0/3996553915 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b8ae5ff8b0 con 0x55b8ae6a9560 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa27fe00640 10 client.27470 handle_command_reply: tid=0 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa2a83ac740 1 client.27470 shutdown 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3996553915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e950 msgr2=0x55b8ae6966e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e950 0x55b8ae6966e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa290000c00 tx=0x7fa290003890 comp rx=0 tx=0).stop 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3996553915 shutdown_connections 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8ae6a9560 0x55b8ae6ab970 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae699210 0x55b8ae696c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e950 0x55b8ae6966e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/3996553915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e030 0x55b8ae6961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3996553915 >> 172.21.15.6:0/3996553915 conn(0x55b8ae68a540 msgr2=0x55b8ae680e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa27fe00640 0 client.27470 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3996553915 shutdown_connections 2024-04-26T02:06:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.367+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3996553915 wait complete. 2024-04-26T02:06:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.368+0000 7fa2a83ac740 20 client.27470 trim_cache size 0 max 16384 2024-04-26T02:06:21.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.387+0000 7fa2a83ac740 1 Processor -- start 2024-04-26T02:06:21.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.387+0000 7fa2a83ac740 1 -- start start 2024-04-26T02:06:21.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.387+0000 7fa2a83ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68a540 0x55b8ae6ab950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.387+0000 7fa2a83ac740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e030 0x55b8ae6a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.387+0000 7fa2a83ac740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68e950 0x55b8ae6900f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.387+0000 7fa2a83ac740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a01ae450 con 0x55b8ae68e030 2024-04-26T02:06:21.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.387+0000 7fa2a83ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a01abcc0 con 0x55b8ae68a540 2024-04-26T02:06:21.386 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.387+0000 7fa2a83ac740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a01a9b30 con 0x55b8ae68e950 2024-04-26T02:06:21.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.388+0000 7fa29ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e030 0x55b8ae6a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.387 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.388+0000 7fa29e000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68a540 0x55b8ae6ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.388+0000 7fa29e000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68a540 0x55b8ae6ab950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:51020/0 (socket says 172.21.15.6:51020) 2024-04-26T02:06:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.388+0000 7fa29e000640 1 -- 172.21.15.6:0/1124661429 learned_addr learned my addr 172.21.15.6:0/1124661429 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.388+0000 7fa29e000640 1 --2- 172.21.15.6:0/1124661429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68e950 0x55b8ae6900f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.388+0000 7fa29ea00640 1 -- 172.21.15.6:0/1124661429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68e950 msgr2=0x55b8ae6900f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.388+0000 7fa29ea00640 1 --2- 172.21.15.6:0/1124661429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68e950 0x55b8ae6900f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.388+0000 7fa29ea00640 1 -- 172.21.15.6:0/1124661429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68a540 msgr2=0x55b8ae6ab950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.388+0000 7fa29ea00640 1 --2- 172.21.15.6:0/1124661429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68a540 0x55b8ae6ab950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.388+0000 7fa29ea00640 1 -- 172.21.15.6:0/1124661429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8ae6abff0 con 0x55b8ae68e030 2024-04-26T02:06:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.388+0000 7fa29ea00640 1 --2- 172.21.15.6:0/1124661429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e030 0x55b8ae6a4d00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa290001eb0 tx=0x7fa29002a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89641b5f70f42094 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.389+0000 7fa28aa00640 1 -- 172.21.15.6:0/1124661429 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa290000e60 con 0x55b8ae68e030 2024-04-26T02:06:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.389+0000 7fa28aa00640 1 -- 172.21.15.6:0/1124661429 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa29003c440 con 0x55b8ae68e030 2024-04-26T02:06:21.388 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.389+0000 7fa28aa00640 1 -- 172.21.15.6:0/1124661429 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa290035070 con 0x55b8ae68e030 2024-04-26T02:06:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.390+0000 7fa2a83ac740 1 -- 172.21.15.6:0/1124661429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e030 msgr2=0x55b8ae6a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.390+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/1124661429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e030 0x55b8ae6a4d00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa290001eb0 tx=0x7fa29002a400 comp rx=0 tx=0).stop 2024-04-26T02:06:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.390+0000 7fa2a83ac740 1 -- 172.21.15.6:0/1124661429 shutdown_connections 2024-04-26T02:06:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.390+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/1124661429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e030 0x55b8ae6a4d00 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.390+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/1124661429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68e950 0x55b8ae6900f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.390+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/1124661429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68a540 0x55b8ae6ab950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.390+0000 7fa2a83ac740 1 -- 172.21.15.6:0/1124661429 >> 172.21.15.6:0/1124661429 conn(0x55b8ae691b70 msgr2=0x55b8ae68f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.390+0000 7fa2a83ac740 1 -- 172.21.15.6:0/1124661429 shutdown_connections 2024-04-26T02:06:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.390+0000 7fa2a83ac740 1 -- 172.21.15.6:0/1124661429 wait complete. 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.390+0000 7fa2a83ac740 1 -- start start 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa2a83ac740 1 Processor -- start 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa2a83ac740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68a540 0x55b8ae6a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa2a83ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 0x55b8ae69ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa2a83ac740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e950 0x55b8ae69f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa2a83ac740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a01aa180 con 0x55b8ae68e950 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa2a83ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a01ab5f0 con 0x55b8ae68e030 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa2a83ac740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a01aa850 con 0x55b8ae68a540 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa2a83ac740 10 client.0 _ll_register_callbacks cb 0x7fa2900045b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa29ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 0x55b8ae69ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa29ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 0x55b8ae69ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:51022/0 (socket says 172.21.15.6:51022) 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa29ea00640 1 -- 172.21.15.6:0/3954369907 learned_addr learned my addr 172.21.15.6:0/3954369907 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa29e000640 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68a540 0x55b8ae6a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.391+0000 7fa29e000640 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e950 0x55b8ae69f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.392+0000 7fa29ea00640 1 -- 172.21.15.6:0/3954369907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68a540 msgr2=0x55b8ae6a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.392+0000 7fa29ea00640 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68a540 0x55b8ae6a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.392+0000 7fa29ea00640 1 -- 172.21.15.6:0/3954369907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e950 msgr2=0x55b8ae69f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.392+0000 7fa29ea00640 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e950 0x55b8ae69f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.392+0000 7fa29ea00640 1 -- 172.21.15.6:0/3954369907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8ae69f870 con 0x55b8ae68e030 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.392+0000 7fa29e000640 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e950 0x55b8ae69f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.392+0000 7fa29ea00640 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 0x55b8ae69ea60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa290046d40 tx=0x7fa290046e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebeef4d4fa5f88b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.392+0000 7fa27fe00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.392+0000 7fa27fe00640 1 -- 172.21.15.6:0/3954369907 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa290035040 con 0x55b8ae68e030 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.392+0000 7fa27fe00640 1 -- 172.21.15.6:0/3954369907 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 8600+0+0 (secure 0 0 0) 0x7fa290046790 con 0x55b8ae68e030 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.392+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3954369907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa29003c440 con 0x55b8ae68e030 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.393+0000 7fa27fe00640 1 -- 172.21.15.6:0/3954369907 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fa290063050 con 0x55b8ae68e030 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.393+0000 7fa2a83ac740 10 client.27476 fetch_fsmap learned FSMap version 55 2024-04-26T02:06:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.393+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3954369907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55b8ae70fc10 con 0x55b8ae68e030 2024-04-26T02:06:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.393+0000 7fa27fe00640 1 -- 172.21.15.6:0/3954369907 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa29003c440 con 0x55b8ae68e030 2024-04-26T02:06:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.393+0000 7fa27fe00640 1 -- 172.21.15.6:0/3954369907 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 55) ==== 8346+0+0 (secure 0 0 0) 0x7fa29005e030 con 0x55b8ae68e030 2024-04-26T02:06:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.393+0000 7fa2a83ac740 10 client.27476 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T02:06:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.393+0000 7fa2a83ac740 10 client.27476 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:06:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.393+0000 7fa2a83ac740 20 client.27476 populate_metadata read hostname 'smithi006' 2024-04-26T02:06:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.393+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8ae699210 0x55b8ae712310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.393+0000 7fa2a83ac740 4 client.27476 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:06:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.393+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3954369907 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b8ae5f9320 con 0x55b8ae699210 2024-04-26T02:06:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.394+0000 7fa29e000640 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8ae699210 0x55b8ae712310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.394+0000 7fa29e000640 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8ae699210 0x55b8ae712310 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f60a95ca8b93a1 server_cookie=2ad43bf4fc08630d in_seq=0 out_seq=0 2024-04-26T02:06:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.394+0000 7fa27fe00640 10 client.27476 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.723+0000 7fa27fe00640 1 -- 172.21.15.6:0/3954369907 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b8ae5f9320 con 0x55b8ae699210 2024-04-26T02:06:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.723+0000 7fa27fe00640 10 client.27476 handle_command_reply: tid=0 2024-04-26T02:06:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.723+0000 7fa2a83ac740 1 client.27476 shutdown 2024-04-26T02:06:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.724+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3954369907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 msgr2=0x55b8ae69ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.724+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 0x55b8ae69ea60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa290046d40 tx=0x7fa290046e60 comp rx=0 tx=0).stop 2024-04-26T02:06:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.724+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3954369907 shutdown_connections 2024-04-26T02:06:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.724+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55b8ae699210 0x55b8ae712310 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.724+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b8ae68e950 0x55b8ae69f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.724+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8ae68e030 0x55b8ae69ea60 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.724+0000 7fa2a83ac740 1 --2- 172.21.15.6:0/3954369907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b8ae68a540 0x55b8ae6a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.724+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3954369907 >> 172.21.15.6:0/3954369907 conn(0x55b8ae691b70 msgr2=0x55b8ae6969f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.725+0000 7fa27fe00640 0 client.27476 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.725+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3954369907 shutdown_connections 2024-04-26T02:06:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.725+0000 7fa2a83ac740 1 -- 172.21.15.6:0/3954369907 wait complete. 2024-04-26T02:06:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.726+0000 7fa2a83ac740 20 client.27476 trim_cache size 0 max 16384 2024-04-26T02:06:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:06:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:06:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:06:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:06:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.727+0000 7fa2a5600640 1 -- 172.21.15.6:0/1891703618 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa2840a3dc0 msgr2=0x7fa2840a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.727+0000 7fa2a5600640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa2840a3dc0 0x7fa2840a6280 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa29800cfd0 tx=0x7fa29800cf40 comp rx=0 tx=0).stop 2024-04-26T02:06:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.727+0000 7fa2a5600640 1 -- 172.21.15.6:0/1891703618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0101bf0 msgr2=0x7fa2a01a6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.727+0000 7fa2a5600640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0101bf0 0x7fa2a01a6850 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa29800ebc0 tx=0x7fa298009f70 comp rx=0 tx=0).stop 2024-04-26T02:06:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.728+0000 7fa2a5600640 1 -- 172.21.15.6:0/1891703618 shutdown_connections 2024-04-26T02:06:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.728+0000 7fa2a5600640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fa2840a3dc0 0x7fa2840a6280 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.728+0000 7fa2a5600640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a0101bf0 0x7fa2a01a6850 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.728+0000 7fa2a5600640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2a0107320 0x7fa2a01016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.728+0000 7fa2a5600640 1 --2- 172.21.15.6:0/1891703618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2a006a880 0x7fa2a0101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.728+0000 7fa2a5600640 1 -- 172.21.15.6:0/1891703618 >> 172.21.15.6:0/1891703618 conn(0x7fa2a0077cd0 msgr2=0x7fa2a00ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.729+0000 7fa2a5600640 1 -- 172.21.15.6:0/1891703618 shutdown_connections 2024-04-26T02:06:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:21.729+0000 7fa2a5600640 1 -- 172.21.15.6:0/1891703618 wait complete. 2024-04-26T02:06:22.142 INFO:tasks.workunit.client.0.smithi006.stdout: 10 135858 19.60 MB/sec execute 285 sec latency 257.978 ms 2024-04-26T02:06:22.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:22 smithi188 ceph-mon[32751]: pgmap v649: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 12 MiB/s wr, 491 op/s 2024-04-26T02:06:22.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:22 smithi029 ceph-mon[32775]: pgmap v649: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 12 MiB/s wr, 491 op/s 2024-04-26T02:06:23.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:22 smithi006 ceph-mon[28419]: pgmap v649: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 12 MiB/s wr, 491 op/s 2024-04-26T02:06:23.142 INFO:tasks.workunit.client.0.smithi006.stdout: 10 136310 19.63 MB/sec execute 286 sec latency 349.594 ms 2024-04-26T02:06:24.142 INFO:tasks.workunit.client.0.smithi006.stdout: 10 136755 19.62 MB/sec execute 287 sec latency 118.052 ms 2024-04-26T02:06:24.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:24 smithi188 ceph-mon[32751]: pgmap v650: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 528 op/s 2024-04-26T02:06:24.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:24 smithi029 ceph-mon[32775]: pgmap v650: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 528 op/s 2024-04-26T02:06:25.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:24 smithi006 ceph-mon[28419]: pgmap v650: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 528 op/s 2024-04-26T02:06:25.142 INFO:tasks.workunit.client.0.smithi006.stdout: 10 137291 19.64 MB/sec execute 288 sec latency 97.276 ms 2024-04-26T02:06:26.142 INFO:tasks.workunit.client.0.smithi006.stdout: 10 137986 19.73 MB/sec execute 289 sec latency 92.499 ms 2024-04-26T02:06:26.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:26 smithi188 ceph-mon[32751]: pgmap v651: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 528 op/s 2024-04-26T02:06:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:26 smithi029 ceph-mon[32775]: pgmap v651: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 528 op/s 2024-04-26T02:06:27.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:26 smithi006 ceph-mon[28419]: pgmap v651: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 528 op/s 2024-04-26T02:06:27.147 INFO:tasks.workunit.client.0.smithi006.stdout: 10 138535 19.75 MB/sec execute 290 sec latency 72.951 ms 2024-04-26T02:06:28.147 INFO:tasks.workunit.client.0.smithi006.stdout: 10 139011 19.78 MB/sec execute 291 sec latency 182.713 ms 2024-04-26T02:06:28.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:28 smithi188 ceph-mon[32751]: pgmap v652: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 16 MiB/s wr, 763 op/s 2024-04-26T02:06:28.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:28 smithi029 ceph-mon[32775]: pgmap v652: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 16 MiB/s wr, 763 op/s 2024-04-26T02:06:29.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:28 smithi006 ceph-mon[28419]: pgmap v652: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 16 MiB/s wr, 763 op/s 2024-04-26T02:06:29.147 INFO:tasks.workunit.client.0.smithi006.stdout: 10 139492 19.80 MB/sec execute 292 sec latency 113.300 ms 2024-04-26T02:06:30.147 INFO:tasks.workunit.client.0.smithi006.stdout: 10 139954 19.83 MB/sec execute 293 sec latency 121.848 ms 2024-04-26T02:06:30.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:30 smithi188 ceph-mon[32751]: pgmap v653: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 11 MiB/s wr, 544 op/s 2024-04-26T02:06:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:30 smithi029 ceph-mon[32775]: pgmap v653: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 11 MiB/s wr, 544 op/s 2024-04-26T02:06:31.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:30 smithi006 ceph-mon[28419]: pgmap v653: 97 pgs: 97 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 11 MiB/s wr, 544 op/s 2024-04-26T02:06:31.148 INFO:tasks.workunit.client.0.smithi006.stdout: 10 140467 19.83 MB/sec execute 294 sec latency 151.856 ms 2024-04-26T02:06:32.147 INFO:tasks.workunit.client.0.smithi006.stdout: 10 141047 19.87 MB/sec execute 295 sec latency 73.051 ms 2024-04-26T02:06:32.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:32 smithi188 ceph-mon[32751]: pgmap v654: 97 pgs: 97 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 747 op/s 2024-04-26T02:06:32.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:32 smithi029 ceph-mon[32775]: pgmap v654: 97 pgs: 97 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 747 op/s 2024-04-26T02:06:33.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:32 smithi006 ceph-mon[28419]: pgmap v654: 97 pgs: 97 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 747 op/s 2024-04-26T02:06:33.147 INFO:tasks.workunit.client.0.smithi006.stdout: 10 141632 19.90 MB/sec execute 296 sec latency 178.732 ms 2024-04-26T02:06:34.147 INFO:tasks.workunit.client.0.smithi006.stdout: 10 142205 19.99 MB/sec execute 297 sec latency 94.309 ms 2024-04-26T02:06:34.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:34 smithi188 ceph-mon[32751]: pgmap v655: 97 pgs: 97 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 571 op/s 2024-04-26T02:06:34.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.901+0000 7f3e20800640 1 -- 172.21.15.6:0/1530479784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1806a880 msgr2=0x7f3e1806ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.901+0000 7f3e20800640 1 --2- 172.21.15.6:0/1530479784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1806a880 0x7f3e1806ac60 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009900 tx=0x7f3e0c02a410 comp rx=0 tx=0).stop 2024-04-26T02:06:34.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.902+0000 7f3e20800640 1 -- 172.21.15.6:0/1530479784 shutdown_connections 2024-04-26T02:06:34.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.902+0000 7f3e20800640 1 --2- 172.21.15.6:0/1530479784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e18107320 0x7f3e1810eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.901 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.902+0000 7f3e20800640 1 --2- 172.21.15.6:0/1530479784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e1806b230 0x7f3e18106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.902+0000 7f3e20800640 1 --2- 172.21.15.6:0/1530479784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1806a880 0x7f3e1806ac60 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.902+0000 7f3e20800640 1 -- 172.21.15.6:0/1530479784 >> 172.21.15.6:0/1530479784 conn(0x7f3e18077cd0 msgr2=0x7f3e180780e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.902+0000 7f3e20800640 1 -- 172.21.15.6:0/1530479784 shutdown_connections 2024-04-26T02:06:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.902+0000 7f3e20800640 1 -- 172.21.15.6:0/1530479784 wait complete. 2024-04-26T02:06:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.903+0000 7f3e20800640 1 Processor -- start 2024-04-26T02:06:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.903+0000 7f3e20800640 1 -- start start 2024-04-26T02:06:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.903+0000 7f3e20800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e1806a880 0x7f3e181a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.903+0000 7f3e20800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e18107320 0x7f3e181a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.903+0000 7f3e20800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1819a740 0x7f3e1819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.903+0000 7f3e20800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3e18112230 con 0x7f3e18107320 2024-04-26T02:06:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.903+0000 7f3e20800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181120b0 con 0x7f3e1819a740 2024-04-26T02:06:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.903+0000 7f3e20800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181123b0 con 0x7f3e1806a880 2024-04-26T02:06:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.904+0000 7f3e1de00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e1806a880 0x7f3e181a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.904+0000 7f3e1de00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e1806a880 0x7f3e181a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:43178/0 (socket says 172.21.15.6:43178) 2024-04-26T02:06:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.904+0000 7f3e1de00640 1 -- 172.21.15.6:0/28062479 learned_addr learned my addr 172.21.15.6:0/28062479 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.904+0000 7f3e1e800640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1819a740 0x7f3e1819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.904+0000 7f3e1d400640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e18107320 0x7f3e181a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.904+0000 7f3e1de00640 1 -- 172.21.15.6:0/28062479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1819a740 msgr2=0x7f3e1819abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.904+0000 7f3e1de00640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1819a740 0x7f3e1819abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.904+0000 7f3e1de00640 1 -- 172.21.15.6:0/28062479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e18107320 msgr2=0x7f3e181a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.904+0000 7f3e1de00640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e18107320 0x7f3e181a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.904+0000 7f3e1de00640 1 -- 172.21.15.6:0/28062479 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1819b340 con 0x7f3e1806a880 2024-04-26T02:06:34.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.905+0000 7f3e1de00640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e1806a880 0x7f3e181a0680 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c005940 tx=0x7f3e0c031750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:34.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.905+0000 7f3e12a00640 1 -- 172.21.15.6:0/28062479 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c04d070 con 0x7f3e1806a880 2024-04-26T02:06:34.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.905+0000 7f3e20800640 1 -- 172.21.15.6:0/28062479 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1819b5d0 con 0x7f3e1806a880 2024-04-26T02:06:34.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.905+0000 7f3e12a00640 1 -- 172.21.15.6:0/28062479 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e0c004450 con 0x7f3e1806a880 2024-04-26T02:06:34.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.905+0000 7f3e12a00640 1 -- 172.21.15.6:0/28062479 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c035070 con 0x7f3e1806a880 2024-04-26T02:06:34.906 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.905+0000 7f3e20800640 1 -- 172.21.15.6:0/28062479 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e181090d0 con 0x7f3e1806a880 2024-04-26T02:06:34.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.909+0000 7f3e235fe740 1 Processor -- start 2024-04-26T02:06:34.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.909+0000 7f3e12a00640 1 -- 172.21.15.6:0/28062479 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f3e0c050070 con 0x7f3e1806a880 2024-04-26T02:06:34.909 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.910+0000 7f3e235fe740 1 -- start start 2024-04-26T02:06:34.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.910+0000 7f3e235fe740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e2820 0x56441a6e2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.910+0000 7f3e235fe740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6ed4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.910+0000 7f3e235fe740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6eda00 0x56441a6efdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.910+0000 7f3e235fe740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56441a648900 con 0x56441a6eda00 2024-04-26T02:06:34.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.910+0000 7f3e235fe740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56441a5eeef0 con 0x56441a6e3140 2024-04-26T02:06:34.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.910+0000 7f3e235fe740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56441a6448c0 con 0x56441a6e2820 2024-04-26T02:06:34.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.910+0000 7f3e1e800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e2820 0x56441a6e2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.910+0000 7f3e1e800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e2820 0x56441a6e2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:43188/0 (socket says 172.21.15.6:43188) 2024-04-26T02:06:34.911 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.910+0000 7f3e1e800640 1 -- 172.21.15.6:0/796739269 learned_addr learned my addr 172.21.15.6:0/796739269 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.910+0000 7f3e1de00640 1 --2- 172.21.15.6:0/796739269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6ed4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e1d400640 1 --2- 172.21.15.6:0/796739269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6eda00 0x56441a6efdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e1e800640 1 -- 172.21.15.6:0/796739269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 msgr2=0x56441a6ed4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e1e800640 1 --2- 172.21.15.6:0/796739269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6ed4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e1e800640 1 -- 172.21.15.6:0/796739269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6eda00 msgr2=0x56441a6efdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e1e800640 1 --2- 172.21.15.6:0/796739269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6eda00 0x56441a6efdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e1e800640 1 -- 172.21.15.6:0/796739269 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56441a35b210 con 0x56441a6e2820 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e1de00640 1 --2- 172.21.15.6:0/796739269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6ed4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e1d400640 1 --2- 172.21.15.6:0/796739269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6eda00 0x56441a6efdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e1e800640 1 --2- 172.21.15.6:0/796739269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e2820 0x56441a6e2c00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009fc0 tx=0x7f3e140118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65a688911a075d6d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e07000640 1 -- 172.21.15.6:0/796739269 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e14004d20 con 0x56441a6e2820 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e07000640 1 -- 172.21.15.6:0/796739269 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e14004050 con 0x56441a6e2820 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e07000640 1 -- 172.21.15.6:0/796739269 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e14005260 con 0x56441a6e2820 2024-04-26T02:06:34.912 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.911+0000 7f3e12a00640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3e000a3d10 0x7f3e000a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.912+0000 7f3e12a00640 1 -- 172.21.15.6:0/28062479 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8600+0+0 (secure 0 0 0) 0x7f3e0c0f8e40 con 0x7f3e1806a880 2024-04-26T02:06:34.913 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.912+0000 7f3e1de00640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3e000a3d10 0x7f3e000a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.912+0000 7f3e235fe740 1 -- 172.21.15.6:0/796739269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e2820 msgr2=0x56441a6e2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.912+0000 7f3e235fe740 1 --2- 172.21.15.6:0/796739269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e2820 0x56441a6e2c00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009fc0 tx=0x7f3e140118d0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.912+0000 7f3e235fe740 1 -- 172.21.15.6:0/796739269 shutdown_connections 2024-04-26T02:06:34.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.912+0000 7f3e235fe740 1 --2- 172.21.15.6:0/796739269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6eda00 0x56441a6efdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.916 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.912+0000 7f3e235fe740 1 --2- 172.21.15.6:0/796739269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6ed4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.912+0000 7f3e235fe740 1 --2- 172.21.15.6:0/796739269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e2820 0x56441a6e2c00 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.912+0000 7f3e235fe740 1 -- 172.21.15.6:0/796739269 >> 172.21.15.6:0/796739269 conn(0x56441a6ded30 msgr2=0x56441a6df110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.912+0000 7f3e235fe740 1 -- 172.21.15.6:0/796739269 shutdown_connections 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.913+0000 7f3e235fe740 1 -- 172.21.15.6:0/796739269 wait complete. 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.913+0000 7f3e1de00640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3e000a3d10 0x7f3e000a61d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x56441a6e3dc0 tx=0x7f3e0c02a410 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.913+0000 7f3e235fe740 1 -- start start 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.913+0000 7f3e235fe740 1 Processor -- start 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e235fe740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6e2820 0x56441a6ea9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e235fe740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6eaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e235fe740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6eda00 0x56441a6eb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e235fe740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x56441a6f0630 con 0x56441a6e2820 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e235fe740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56441a602310 con 0x56441a6e3140 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e235fe740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56441a6f07b0 con 0x56441a6eda00 2024-04-26T02:06:34.917 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e235fe740 10 client.0 _ll_register_callbacks cb 0x56441a5d0cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e1e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6e2820 0x56441a6ea9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e1d400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6eaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e1e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6e2820 0x56441a6ea9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44910/0 (socket says 172.21.15.6:44910) 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e1d400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6eaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34596/0 (socket says 172.21.15.6:34596) 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e1e800640 1 -- 172.21.15.6:0/505326664 learned_addr learned my addr 172.21.15.6:0/505326664 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.914+0000 7f3e1e800640 1 --2- 172.21.15.6:0/505326664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6eda00 0x56441a6eb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.915+0000 7f3e1d400640 1 -- 172.21.15.6:0/505326664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6eda00 msgr2=0x56441a6eb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.915+0000 7f3e1d400640 1 --2- 172.21.15.6:0/505326664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6eda00 0x56441a6eb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.915+0000 7f3e1d400640 1 -- 172.21.15.6:0/505326664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6e2820 msgr2=0x56441a6ea9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.915+0000 7f3e1d400640 1 --2- 172.21.15.6:0/505326664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6e2820 0x56441a6ea9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.915+0000 7f3e1d400640 1 -- 172.21.15.6:0/505326664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56441a427620 con 0x56441a6e3140 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.915+0000 7f3e1d400640 1 --2- 172.21.15.6:0/505326664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6eaef0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x56441a6dfec0 tx=0x7f3e08013b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7364879e4c770ded server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:34.918 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.915+0000 7f3dfb400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.915+0000 7f3dfb400640 1 -- 172.21.15.6:0/505326664 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08011990 con 0x56441a6e3140 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.915+0000 7f3dfb400640 1 -- 172.21.15.6:0/505326664 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 8600+0+0 (secure 0 0 0) 0x7f3e08011b30 con 0x56441a6e3140 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.915+0000 7f3e235fe740 1 -- 172.21.15.6:0/505326664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e14004050 con 0x56441a6e3140 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.916+0000 7f3dfb400640 1 -- 172.21.15.6:0/505326664 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f3e08011d30 con 0x56441a6e3140 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.916+0000 7f3e235fe740 10 client.27488 fetch_fsmap learned FSMap version 56 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.916+0000 7f3e235fe740 1 -- 172.21.15.6:0/505326664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x56441a6fc8c0 con 0x56441a6e3140 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.916+0000 7f3dfb400640 1 -- 172.21.15.6:0/505326664 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e14004050 con 0x56441a6e3140 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.916+0000 7f3dfb400640 1 -- 172.21.15.6:0/505326664 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 56) ==== 8342+0+0 (secure 0 0 0) 0x7f3e0801a070 con 0x56441a6e3140 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.916+0000 7f3e235fe740 10 client.27488 fetch_fsmap finished waiting for FSMap version 56 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.916+0000 7f3e235fe740 10 client.27488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.916+0000 7f3e235fe740 20 client.27488 populate_metadata read hostname 'smithi006' 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.916+0000 7f3e235fe740 1 --2- 172.21.15.6:0/505326664 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56441a6fdd30 0x56441a700140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.916+0000 7f3e235fe740 4 client.27488 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:06:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.916+0000 7f3e235fe740 1 -- 172.21.15.6:0/505326664 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56441a65c8b0 con 0x56441a6fdd30 2024-04-26T02:06:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.917+0000 7f3e1e800640 1 --2- 172.21.15.6:0/505326664 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56441a6fdd30 0x56441a700140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.917+0000 7f3e1e800640 1 --2- 172.21.15.6:0/505326664 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56441a6fdd30 0x56441a700140 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e61f2da1f960d503 server_cookie=e366bdbb4ecd4472 in_seq=0 out_seq=0 2024-04-26T02:06:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.917+0000 7f3dfb400640 10 client.27488 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.921+0000 7f3dfb400640 1 -- 172.21.15.6:0/505326664 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56441a65c8b0 con 0x56441a6fdd30 2024-04-26T02:06:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.921+0000 7f3dfb400640 10 client.27488 handle_command_reply: tid=0 2024-04-26T02:06:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.921+0000 7f3e235fe740 1 client.27488 shutdown 2024-04-26T02:06:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.921+0000 7f3e235fe740 1 -- 172.21.15.6:0/505326664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 msgr2=0x56441a6eaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.921+0000 7f3e235fe740 1 --2- 172.21.15.6:0/505326664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6eaef0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x56441a6dfec0 tx=0x7f3e08013b70 comp rx=0 tx=0).stop 2024-04-26T02:06:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.921+0000 7f3e235fe740 1 -- 172.21.15.6:0/505326664 shutdown_connections 2024-04-26T02:06:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.921+0000 7f3e235fe740 1 --2- 172.21.15.6:0/505326664 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56441a6fdd30 0x56441a700140 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.921+0000 7f3e235fe740 1 --2- 172.21.15.6:0/505326664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6eda00 0x56441a6eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.921+0000 7f3e235fe740 1 --2- 172.21.15.6:0/505326664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6eaef0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.921+0000 7f3e235fe740 1 --2- 172.21.15.6:0/505326664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6e2820 0x56441a6ea9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.921+0000 7f3e235fe740 1 -- 172.21.15.6:0/505326664 >> 172.21.15.6:0/505326664 conn(0x56441a6ded30 msgr2=0x56441a6e4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:34.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.921+0000 7f3dfb400640 0 client.27488 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:34.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.922+0000 7f3e235fe740 1 -- 172.21.15.6:0/505326664 shutdown_connections 2024-04-26T02:06:34.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.922+0000 7f3e235fe740 1 -- 172.21.15.6:0/505326664 wait complete. 2024-04-26T02:06:34.922 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.922+0000 7f3e235fe740 20 client.27488 trim_cache size 0 max 16384 2024-04-26T02:06:34.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:34 smithi029 ceph-mon[32775]: pgmap v655: 97 pgs: 97 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 571 op/s 2024-04-26T02:06:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.943+0000 7f3e235fe740 1 Processor -- start 2024-04-26T02:06:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.943+0000 7f3e235fe740 1 -- start start 2024-04-26T02:06:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.944+0000 7f3e235fe740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e2820 0x56441a6f94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.944+0000 7f3e235fe740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e3140 0x56441a705a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.944+0000 7f3e235fe740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6eda00 0x56441a705f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.944+0000 7f3e235fe740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181ae350 con 0x56441a6eda00 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.944+0000 7f3e235fe740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181abbf0 con 0x56441a6e2820 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.944+0000 7f3e235fe740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181a9ac0 con 0x56441a6e3140 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.944+0000 7f3e1d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e3140 0x56441a705a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.944+0000 7f3e1d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e3140 0x56441a705a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:43202/0 (socket says 172.21.15.6:43202) 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.944+0000 7f3e1d400640 1 -- 172.21.15.6:0/449700428 learned_addr learned my addr 172.21.15.6:0/449700428 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.945+0000 7f3e1e800640 1 --2- 172.21.15.6:0/449700428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e2820 0x56441a6f94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.945+0000 7f3e1d400640 1 -- 172.21.15.6:0/449700428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e2820 msgr2=0x56441a6f94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.945+0000 7f3e1d400640 1 --2- 172.21.15.6:0/449700428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e2820 0x56441a6f94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.945+0000 7f3e1d400640 1 -- 172.21.15.6:0/449700428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6eda00 msgr2=0x56441a705f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.945+0000 7f3e1d400640 1 --2- 172.21.15.6:0/449700428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6eda00 0x56441a705f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.945+0000 7f3e1d400640 1 -- 172.21.15.6:0/449700428 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56441a6e8f00 con 0x56441a6e3140 2024-04-26T02:06:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.945+0000 7f3e1e800640 1 --2- 172.21.15.6:0/449700428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e2820 0x56441a6f94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:06:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.945+0000 7f3e1d400640 1 --2- 172.21.15.6:0/449700428 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e3140 0x56441a705a50 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3e08004ea0 tx=0x7f3e08011f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5133b5e0ef40048 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.945+0000 7f3e06600640 1 -- 172.21.15.6:0/449700428 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08028040 con 0x56441a6e3140 2024-04-26T02:06:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.945+0000 7f3e06600640 1 -- 172.21.15.6:0/449700428 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e08005c50 con 0x56441a6e3140 2024-04-26T02:06:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.945+0000 7f3e06600640 1 -- 172.21.15.6:0/449700428 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08005420 con 0x56441a6e3140 2024-04-26T02:06:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.946+0000 7f3e235fe740 1 -- 172.21.15.6:0/449700428 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e3140 msgr2=0x56441a705a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.946+0000 7f3e235fe740 1 --2- 172.21.15.6:0/449700428 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e3140 0x56441a705a50 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3e08004ea0 tx=0x7f3e08011f50 comp rx=0 tx=0).stop 2024-04-26T02:06:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.946+0000 7f3e235fe740 1 -- 172.21.15.6:0/449700428 shutdown_connections 2024-04-26T02:06:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.946+0000 7f3e235fe740 1 --2- 172.21.15.6:0/449700428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6eda00 0x56441a705f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.946+0000 7f3e235fe740 1 --2- 172.21.15.6:0/449700428 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6e3140 0x56441a705a50 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.946+0000 7f3e235fe740 1 --2- 172.21.15.6:0/449700428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e2820 0x56441a6f94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.946+0000 7f3e235fe740 1 -- 172.21.15.6:0/449700428 >> 172.21.15.6:0/449700428 conn(0x56441a6ded30 msgr2=0x56441a6e5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.947+0000 7f3e235fe740 1 -- 172.21.15.6:0/449700428 shutdown_connections 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.947+0000 7f3e235fe740 1 -- 172.21.15.6:0/449700428 wait complete. 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.947+0000 7f3e235fe740 1 -- start start 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.947+0000 7f3e235fe740 1 Processor -- start 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.948+0000 7f3e235fe740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6e2820 0x56441a6f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.948+0000 7f3e235fe740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6f3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.948+0000 7f3e235fe740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6eda00 0x56441a6f36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.948+0000 7f3e235fe740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181aa110 con 0x56441a6e2820 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.948+0000 7f3e235fe740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181ab520 con 0x56441a6e3140 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.948+0000 7f3e235fe740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181aa780 con 0x56441a6eda00 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.948+0000 7f3e235fe740 10 client.0 _ll_register_callbacks cb 0x7f3e08009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.948+0000 7f3e1d400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.948+0000 7f3e1d400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6f3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:34614/0 (socket says 172.21.15.6:34614) 2024-04-26T02:06:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.948+0000 7f3e1e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6e2820 0x56441a6f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.948+0000 7f3e1d400640 1 -- 172.21.15.6:0/2125442827 learned_addr learned my addr 172.21.15.6:0/2125442827 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.948+0000 7f3e1e800640 1 --2- 172.21.15.6:0/2125442827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6eda00 0x56441a6f36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.949+0000 7f3e1d400640 1 -- 172.21.15.6:0/2125442827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6eda00 msgr2=0x56441a6f36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.949+0000 7f3e1d400640 1 --2- 172.21.15.6:0/2125442827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6eda00 0x56441a6f36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.949+0000 7f3e1d400640 1 -- 172.21.15.6:0/2125442827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6e2820 msgr2=0x56441a6f5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.949+0000 7f3e1d400640 1 --2- 172.21.15.6:0/2125442827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6e2820 0x56441a6f5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.950 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.949+0000 7f3e1d400640 1 -- 172.21.15.6:0/2125442827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56441a6f3f00 con 0x56441a6e3140 2024-04-26T02:06:34.951 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.949+0000 7f3e1d400640 1 --2- 172.21.15.6:0/2125442827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6f3190 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009360 tx=0x7f3e080052c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9611d47780e2f685 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.949+0000 7f3dfb400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:06:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.949+0000 7f3dfb400640 1 -- 172.21.15.6:0/2125442827 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08028050 con 0x56441a6e3140 2024-04-26T02:06:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.949+0000 7f3e235fe740 1 -- 172.21.15.6:0/2125442827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e08005c50 con 0x56441a6e3140 2024-04-26T02:06:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.949+0000 7f3dfb400640 1 -- 172.21.15.6:0/2125442827 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 8600+0+0 (secure 0 0 0) 0x7f3e080043e0 con 0x56441a6e3140 2024-04-26T02:06:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.950+0000 7f3dfb400640 1 -- 172.21.15.6:0/2125442827 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f3e080045e0 con 0x56441a6e3140 2024-04-26T02:06:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.950+0000 7f3e235fe740 10 client.27491 fetch_fsmap learned FSMap version 56 2024-04-26T02:06:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.950+0000 7f3e235fe740 1 -- 172.21.15.6:0/2125442827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x56441a7642b0 con 0x56441a6e3140 2024-04-26T02:06:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.950+0000 7f3dfb400640 1 -- 172.21.15.6:0/2125442827 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e08005c50 con 0x56441a6e3140 2024-04-26T02:06:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.950+0000 7f3dfb400640 1 -- 172.21.15.6:0/2125442827 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 56) ==== 8342+0+0 (secure 0 0 0) 0x7f3e0801a050 con 0x56441a6e3140 2024-04-26T02:06:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.950+0000 7f3e235fe740 10 client.27491 fetch_fsmap finished waiting for FSMap version 56 2024-04-26T02:06:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.950+0000 7f3e235fe740 10 client.27491 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:06:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.950+0000 7f3e235fe740 20 client.27491 populate_metadata read hostname 'smithi006' 2024-04-26T02:06:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.950+0000 7f3e235fe740 1 --2- 172.21.15.6:0/2125442827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56441a764920 0x56441a766d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.950+0000 7f3e235fe740 4 client.27491 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:06:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.950+0000 7f3e235fe740 1 -- 172.21.15.6:0/2125442827 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56441a656320 con 0x56441a764920 2024-04-26T02:06:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.951+0000 7f3e1e800640 1 --2- 172.21.15.6:0/2125442827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56441a764920 0x56441a766d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.951+0000 7f3e1e800640 1 --2- 172.21.15.6:0/2125442827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56441a764920 0x56441a766d10 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6398681a475513eb server_cookie=8a030c7074418321 in_seq=0 out_seq=0 2024-04-26T02:06:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:34.951+0000 7f3dfb400640 10 client.27491 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:35.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:34 smithi006 ceph-mon[28419]: pgmap v655: 97 pgs: 97 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 571 op/s 2024-04-26T02:06:35.147 INFO:tasks.workunit.client.0.smithi006.stdout: 10 142618 19.99 MB/sec execute 298 sec latency 190.026 ms 2024-04-26T02:06:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.486+0000 7f3dfb400640 1 -- 172.21.15.6:0/2125442827 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56441a656320 con 0x56441a764920 2024-04-26T02:06:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.486+0000 7f3dfb400640 10 client.27491 handle_command_reply: tid=0 2024-04-26T02:06:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.487+0000 7f3e235fe740 1 client.27491 shutdown 2024-04-26T02:06:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.487+0000 7f3e235fe740 1 -- 172.21.15.6:0/2125442827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 msgr2=0x56441a6f3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.487+0000 7f3e235fe740 1 --2- 172.21.15.6:0/2125442827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6f3190 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009360 tx=0x7f3e080052c0 comp rx=0 tx=0).stop 2024-04-26T02:06:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.487+0000 7f3e235fe740 1 -- 172.21.15.6:0/2125442827 shutdown_connections 2024-04-26T02:06:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.487+0000 7f3e235fe740 1 --2- 172.21.15.6:0/2125442827 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x56441a764920 0x56441a766d10 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:35.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.487+0000 7f3e235fe740 1 --2- 172.21.15.6:0/2125442827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56441a6eda00 0x56441a6f36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:35.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.487+0000 7f3e235fe740 1 --2- 172.21.15.6:0/2125442827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56441a6e3140 0x56441a6f3190 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:35.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.487+0000 7f3e235fe740 1 --2- 172.21.15.6:0/2125442827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56441a6e2820 0x56441a6f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:35.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.487+0000 7f3dfb400640 0 client.27491 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:35.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.487+0000 7f3e235fe740 1 -- 172.21.15.6:0/2125442827 >> 172.21.15.6:0/2125442827 conn(0x56441a6ded30 msgr2=0x56441a6e41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:35.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.487+0000 7f3e235fe740 1 -- 172.21.15.6:0/2125442827 shutdown_connections 2024-04-26T02:06:35.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.487+0000 7f3e235fe740 1 -- 172.21.15.6:0/2125442827 wait complete. 2024-04-26T02:06:35.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.488+0000 7f3e235fe740 20 client.27491 trim_cache size 0 max 16384 2024-04-26T02:06:35.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:06:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:06:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:06:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:06:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.488+0000 7f3e20800640 1 -- 172.21.15.6:0/28062479 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3e000a3d10 msgr2=0x7f3e000a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.488+0000 7f3e20800640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3e000a3d10 0x7f3e000a61d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x56441a6e3dc0 tx=0x7f3e0c02a410 comp rx=0 tx=0).stop 2024-04-26T02:06:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.488+0000 7f3e20800640 1 -- 172.21.15.6:0/28062479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e1806a880 msgr2=0x7f3e181a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.488+0000 7f3e20800640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e1806a880 0x7f3e181a0680 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c005940 tx=0x7f3e0c031750 comp rx=0 tx=0).stop 2024-04-26T02:06:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.489+0000 7f3e20800640 1 -- 172.21.15.6:0/28062479 shutdown_connections 2024-04-26T02:06:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.489+0000 7f3e20800640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f3e000a3d10 0x7f3e000a61d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.489+0000 7f3e20800640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e18107320 0x7f3e181a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.489+0000 7f3e20800640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1819a740 0x7f3e1819abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.489+0000 7f3e20800640 1 --2- 172.21.15.6:0/28062479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e1806a880 0x7f3e181a0680 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.489+0000 7f3e20800640 1 -- 172.21.15.6:0/28062479 >> 172.21.15.6:0/28062479 conn(0x7f3e18077cd0 msgr2=0x7f3e1810d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.489+0000 7f3e20800640 1 -- 172.21.15.6:0/28062479 shutdown_connections 2024-04-26T02:06:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:35.489+0000 7f3e20800640 1 -- 172.21.15.6:0/28062479 wait complete. 2024-04-26T02:06:35.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:35 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:35.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:35 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:36.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:35 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:36.148 INFO:tasks.workunit.client.0.smithi006.stdout: 10 142866 19.95 MB/sec execute 299 sec latency 485.370 ms 2024-04-26T02:06:36.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:06:35 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: ignoring --setuser ceph since I am not root 2024-04-26T02:06:36.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:06:35 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: ignoring --setgroup ceph since I am not root 2024-04-26T02:06:36.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:06:35 smithi188 ceph-mds[68153]: -- 172.21.15.188:0/539980812 <== mon.2 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557c3831ba40 con 0x557c382ee800 2024-04-26T02:06:36.391 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:06:35 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: starting mds.f at 2024-04-26T02:06:36.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:36 smithi029 ceph-mon[32775]: pgmap v656: 97 pgs: 97 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 540 op/s 2024-04-26T02:06:36.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:36 smithi029 ceph-mon[32775]: daemon mds.f restarted 2024-04-26T02:06:37.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:36 smithi006 ceph-mon[28419]: pgmap v656: 97 pgs: 97 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 540 op/s 2024-04-26T02:06:37.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:36 smithi006 ceph-mon[28419]: daemon mds.f restarted 2024-04-26T02:06:37.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:36 smithi188 ceph-mon[32751]: pgmap v656: 97 pgs: 97 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 540 op/s 2024-04-26T02:06:37.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:36 smithi188 ceph-mon[32751]: daemon mds.f restarted 2024-04-26T02:06:37.148 INFO:tasks.workunit.client.0.smithi006.stdout: 10 143354 19.96 MB/sec execute 300 sec latency 135.882 ms 2024-04-26T02:06:37.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:37 smithi029 ceph-mon[32775]: osdmap e130: 12 total, 12 up, 12 in 2024-04-26T02:06:37.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:37 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T02:06:37.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:37 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6834/1973934809,v1:172.21.15.188:6835/1973934809] up:boot 2024-04-26T02:06:37.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:37 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T02:06:37.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:37 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:38.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:37 smithi006 ceph-mon[28419]: osdmap e130: 12 total, 12 up, 12 in 2024-04-26T02:06:38.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:37 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T02:06:38.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:37 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6834/1973934809,v1:172.21.15.188:6835/1973934809] up:boot 2024-04-26T02:06:38.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:37 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T02:06:38.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:37 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:38.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:37 smithi188 ceph-mon[32751]: osdmap e130: 12 total, 12 up, 12 in 2024-04-26T02:06:38.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:37 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T02:06:38.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:37 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6834/1973934809,v1:172.21.15.188:6835/1973934809] up:boot 2024-04-26T02:06:38.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:37 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T02:06:38.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:37 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:38.148 INFO:tasks.workunit.client.0.smithi006.stdout: 10 143848 19.98 MB/sec execute 301 sec latency 84.405 ms 2024-04-26T02:06:39.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:38 smithi006 ceph-mon[28419]: pgmap v658: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.9 MiB/s wr, 599 op/s 2024-04-26T02:06:39.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:38 smithi188 ceph-mon[32751]: pgmap v658: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.9 MiB/s wr, 599 op/s 2024-04-26T02:06:39.148 INFO:tasks.workunit.client.0.smithi006.stdout: 10 144434 20.00 MB/sec execute 302 sec latency 130.333 ms 2024-04-26T02:06:39.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:38 smithi029 ceph-mon[32775]: pgmap v658: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.9 MiB/s wr, 599 op/s 2024-04-26T02:06:40.148 INFO:tasks.workunit.client.0.smithi006.stdout: 10 145055 20.07 MB/sec execute 303 sec latency 77.365 ms 2024-04-26T02:06:41.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:40 smithi006 ceph-mon[28419]: pgmap v659: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.9 MiB/s wr, 599 op/s 2024-04-26T02:06:41.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:40 smithi188 ceph-mon[32751]: pgmap v659: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.9 MiB/s wr, 599 op/s 2024-04-26T02:06:41.148 INFO:tasks.workunit.client.0.smithi006.stdout: 10 145472 20.09 MB/sec execute 304 sec latency 284.887 ms 2024-04-26T02:06:41.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:40 smithi029 ceph-mon[32775]: pgmap v659: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.9 MiB/s wr, 599 op/s 2024-04-26T02:06:42.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:41 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:06:42.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:41 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:06:42.148 INFO:tasks.workunit.client.0.smithi006.stdout: 10 145680 20.05 MB/sec execute 305 sec latency 529.528 ms 2024-04-26T02:06:42.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:41 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:06:43.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:42 smithi006 ceph-mon[28419]: pgmap v660: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.6 MiB/s wr, 550 op/s 2024-04-26T02:06:43.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:42 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:43.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:42 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:43.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:42 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:43.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:42 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:43.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:42 smithi188 ceph-mon[32751]: pgmap v660: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.6 MiB/s wr, 550 op/s 2024-04-26T02:06:43.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:42 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:43.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:42 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:43.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:42 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:43.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:42 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:43.148 INFO:tasks.workunit.client.0.smithi006.stdout: 10 146008 20.05 MB/sec execute 306 sec latency 316.043 ms 2024-04-26T02:06:43.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:42 smithi029 ceph-mon[32775]: pgmap v660: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.6 MiB/s wr, 550 op/s 2024-04-26T02:06:43.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:42 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:43.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:42 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:43.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:42 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:43.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:42 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:44.148 INFO:tasks.workunit.client.0.smithi006.stdout: 10 146269 20.05 MB/sec execute 307 sec latency 275.865 ms 2024-04-26T02:06:44.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:44 smithi029 ceph-mon[32775]: pgmap v661: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.2 MiB/s wr, 541 op/s 2024-04-26T02:06:44.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:44 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:44.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:44 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:45.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:44 smithi006 ceph-mon[28419]: pgmap v661: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.2 MiB/s wr, 541 op/s 2024-04-26T02:06:45.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:44 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:45.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:44 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:45.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:44 smithi188 ceph-mon[32751]: pgmap v661: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.2 MiB/s wr, 541 op/s 2024-04-26T02:06:45.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:44 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:45.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:44 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:45.148 INFO:tasks.workunit.client.0.smithi006.stdout: 10 146570 20.02 MB/sec execute 308 sec latency 197.343 ms 2024-04-26T02:06:46.149 INFO:tasks.workunit.client.0.smithi006.stdout: 10 146853 20.00 MB/sec execute 309 sec latency 404.457 ms 2024-04-26T02:06:46.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:46.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:46.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: pgmap v662: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.2 MiB/s wr, 541 op/s 2024-04-26T02:06:46.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.821 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.821 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.821 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.821 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.821 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:46 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:46.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:46.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: pgmap v662: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.2 MiB/s wr, 541 op/s 2024-04-26T02:06:46.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.893 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.893 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.893 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.893 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.893 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.893 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.893 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:46 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: pgmap v662: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.2 MiB/s wr, 541 op/s 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.943 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.943 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.943 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.943 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:46.943 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:46 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:47.149 INFO:tasks.workunit.client.0.smithi006.stdout: 10 147278 19.98 MB/sec execute 310 sec latency 197.526 ms 2024-04-26T02:06:47.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:47 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi006 to 177.5M 2024-04-26T02:06:47.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:47 smithi006 ceph-mon[28419]: Unable to set osd_memory_target on smithi006 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T02:06:47.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:47 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-04-26T02:06:47.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:47 smithi006 ceph-mon[28419]: Unable to set osd_memory_target on smithi029 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-26T02:06:47.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:47 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi006 to 177.5M 2024-04-26T02:06:47.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:47 smithi188 ceph-mon[32751]: Unable to set osd_memory_target on smithi006 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T02:06:47.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:47 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-04-26T02:06:47.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:47 smithi188 ceph-mon[32751]: Unable to set osd_memory_target on smithi029 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-26T02:06:47.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:47 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi006 to 177.5M 2024-04-26T02:06:47.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:47 smithi029 ceph-mon[32775]: Unable to set osd_memory_target on smithi006 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T02:06:47.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:47 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-04-26T02:06:47.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:47 smithi029 ceph-mon[32775]: Unable to set osd_memory_target on smithi029 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-26T02:06:48.149 INFO:tasks.workunit.client.0.smithi006.stdout: 10 147603 19.99 MB/sec execute 311 sec latency 447.610 ms 2024-04-26T02:06:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.683+0000 7f9a9b200640 1 -- 172.21.15.6:0/339795539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a941092c0 msgr2=0x7f9a94110e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.683+0000 7f9a9b200640 1 --2- 172.21.15.6:0/339795539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a941092c0 0x7f9a94110e90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009ca0 tx=0x7f9a9002a8c0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.683+0000 7f9a9b200640 1 -- 172.21.15.6:0/339795539 shutdown_connections 2024-04-26T02:06:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.683+0000 7f9a9b200640 1 --2- 172.21.15.6:0/339795539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a941092c0 0x7f9a94110e90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.683+0000 7f9a9b200640 1 --2- 172.21.15.6:0/339795539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a94104ba0 0x7f9a94108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.683+0000 7f9a9b200640 1 --2- 172.21.15.6:0/339795539 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a941041f0 0x7f9a941045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.683+0000 7f9a9b200640 1 -- 172.21.15.6:0/339795539 >> 172.21.15.6:0/339795539 conn(0x7f9a940ffa50 msgr2=0x7f9a94101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.683+0000 7f9a9b200640 1 -- 172.21.15.6:0/339795539 shutdown_connections 2024-04-26T02:06:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.683+0000 7f9a9b200640 1 -- 172.21.15.6:0/339795539 wait complete. 2024-04-26T02:06:48.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.684+0000 7f9a9b200640 1 Processor -- start 2024-04-26T02:06:48.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.684+0000 7f9a9b200640 1 -- start start 2024-04-26T02:06:48.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.685+0000 7f9a9b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a941041f0 0x7f9a9419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.685+0000 7f9a9b200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a941092c0 0x7f9a9419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.685+0000 7f9a9b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a9419f020 0x7f9a9419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.685+0000 7f9a9b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9a94114260 con 0x7f9a9419f020 2024-04-26T02:06:48.685 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.685+0000 7f9a9b200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941140e0 con 0x7f9a941092c0 2024-04-26T02:06:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.685+0000 7f9a9b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941143e0 con 0x7f9a941041f0 2024-04-26T02:06:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.685+0000 7f9a99e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a941041f0 0x7f9a9419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.685+0000 7f9a99e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a941041f0 0x7f9a9419a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:48702/0 (socket says 172.21.15.6:48702) 2024-04-26T02:06:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.685+0000 7f9a99e00640 1 -- 172.21.15.6:0/1443184000 learned_addr learned my addr 172.21.15.6:0/1443184000 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.685+0000 7f9a99400640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a941092c0 0x7f9a9419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.685+0000 7f9a9a800640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a9419f020 0x7f9a9419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.686+0000 7f9a99e00640 1 -- 172.21.15.6:0/1443184000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a941092c0 msgr2=0x7f9a9419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.686+0000 7f9a99e00640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a941092c0 0x7f9a9419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.686+0000 7f9a99e00640 1 -- 172.21.15.6:0/1443184000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a9419f020 msgr2=0x7f9a9419f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.686+0000 7f9a99e00640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a9419f020 0x7f9a9419f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.686+0000 7f9a99e00640 1 -- 172.21.15.6:0/1443184000 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9419fb10 con 0x7f9a941041f0 2024-04-26T02:06:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.686+0000 7f9a99e00640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a941041f0 0x7f9a9419a760 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800e9f0 tx=0x7f9a8800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.686+0000 7f9a8ea00640 1 -- 172.21.15.6:0/1443184000 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8800cd80 con 0x7f9a941041f0 2024-04-26T02:06:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.686+0000 7f9a8ea00640 1 -- 172.21.15.6:0/1443184000 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a880108e0 con 0x7f9a941041f0 2024-04-26T02:06:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.686+0000 7f9a9b200640 1 -- 172.21.15.6:0/1443184000 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9419fe00 con 0x7f9a941041f0 2024-04-26T02:06:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.686+0000 7f9a8ea00640 1 -- 172.21.15.6:0/1443184000 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88018980 con 0x7f9a941041f0 2024-04-26T02:06:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.686+0000 7f9a9b200640 1 -- 172.21.15.6:0/1443184000 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a940766b0 con 0x7f9a941041f0 2024-04-26T02:06:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.690+0000 7f9a8ea00640 1 -- 172.21.15.6:0/1443184000 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f9a88010430 con 0x7f9a941041f0 2024-04-26T02:06:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.690+0000 7f9a9f1fd740 1 Processor -- start 2024-04-26T02:06:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.691+0000 7f9a9f1fd740 1 -- start start 2024-04-26T02:06:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.691+0000 7f9a9f1fd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59549b0 0x5613d5954d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.691+0000 7f9a9f1fd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59552d0 0x5613d595f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.691+0000 7f9a9f1fd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d595fb90 0x5613d5961f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.691+0000 7f9a9f1fd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5613d58ba900 con 0x5613d59549b0 2024-04-26T02:06:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.691+0000 7f9a9f1fd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5613d5860ef0 con 0x5613d595fb90 2024-04-26T02:06:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.691+0000 7f9a9f1fd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5613d58b68c0 con 0x5613d59552d0 2024-04-26T02:06:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.692+0000 7f9a99400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d595fb90 0x5613d5961f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.692+0000 7f9a99e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59552d0 0x5613d595f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.692+0000 7f9a9a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59549b0 0x5613d5954d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.692+0000 7f9a99400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d595fb90 0x5613d5961f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:45668/0 (socket says 172.21.15.6:45668) 2024-04-26T02:06:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.692+0000 7f9a99400640 1 -- 172.21.15.6:0/942524859 learned_addr learned my addr 172.21.15.6:0/942524859 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.692+0000 7f9a9a800640 1 -- 172.21.15.6:0/942524859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59552d0 msgr2=0x5613d595f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.692+0000 7f9a9a800640 1 --2- 172.21.15.6:0/942524859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59552d0 0x5613d595f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.692+0000 7f9a9a800640 1 -- 172.21.15.6:0/942524859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d595fb90 msgr2=0x5613d5961f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.692+0000 7f9a9a800640 1 --2- 172.21.15.6:0/942524859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d595fb90 0x5613d5961f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.692+0000 7f9a9a800640 1 -- 172.21.15.6:0/942524859 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613d55cd210 con 0x5613d59549b0 2024-04-26T02:06:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.692+0000 7f9a99400640 1 --2- 172.21.15.6:0/942524859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d595fb90 0x5613d5961f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:06:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.693+0000 7f9a8ea00640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9a7c0a3d00 0x7f9a7c0a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.693+0000 7f9a9a800640 1 --2- 172.21.15.6:0/942524859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59549b0 0x5613d5954d90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9a90005e30 tx=0x7f9a90005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43fad0c3ff55f0fc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.693+0000 7f9a8ea00640 1 -- 172.21.15.6:0/1443184000 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 8556+0+0 (secure 0 0 0) 0x7f9a880d0cd0 con 0x7f9a941041f0 2024-04-26T02:06:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.693+0000 7f9a83000640 1 -- 172.21.15.6:0/942524859 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90040d20 con 0x5613d59549b0 2024-04-26T02:06:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.693+0000 7f9a99e00640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9a7c0a3d00 0x7f9a7c0a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.693+0000 7f9a83000640 1 -- 172.21.15.6:0/942524859 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a90036070 con 0x5613d59549b0 2024-04-26T02:06:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.694+0000 7f9a83000640 1 -- 172.21.15.6:0/942524859 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90003cf0 con 0x5613d59549b0 2024-04-26T02:06:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.694+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/942524859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59549b0 msgr2=0x5613d5954d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.694+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/942524859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59549b0 0x5613d5954d90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9a90005e30 tx=0x7f9a90005d80 comp rx=0 tx=0).stop 2024-04-26T02:06:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.694+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/942524859 shutdown_connections 2024-04-26T02:06:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.694+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/942524859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d595fb90 0x5613d5961f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.694+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/942524859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59552d0 0x5613d595f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.694+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/942524859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59549b0 0x5613d5954d90 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.694+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/942524859 >> 172.21.15.6:0/942524859 conn(0x5613d5950ec0 msgr2=0x5613d59512a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.694+0000 7f9a99e00640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9a7c0a3d00 0x7f9a7c0a61c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x5613d5955f50 tx=0x7f9a880d4780 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.695+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/942524859 shutdown_connections 2024-04-26T02:06:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.695+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/942524859 wait complete. 2024-04-26T02:06:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.695+0000 7f9a9f1fd740 1 -- start start 2024-04-26T02:06:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.696+0000 7f9a9f1fd740 1 Processor -- start 2024-04-26T02:06:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.696+0000 7f9a9f1fd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 0x5613d595cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.696+0000 7f9a9f1fd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59552d0 0x5613d595d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.696+0000 7f9a9f1fd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d595fb90 0x5613d595d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.696+0000 7f9a9f1fd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5613d5962760 con 0x5613d595fb90 2024-04-26T02:06:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.696+0000 7f9a9f1fd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5613d5874310 con 0x5613d59552d0 2024-04-26T02:06:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.696+0000 7f9a9f1fd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5613d59628e0 con 0x5613d59549b0 2024-04-26T02:06:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.697+0000 7f9a9f1fd740 10 client.0 _ll_register_callbacks cb 0x5613d5842cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.697+0000 7f9a9a800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 0x5613d595cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.697+0000 7f9a9a800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 0x5613d595cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:48722/0 (socket says 172.21.15.6:48722) 2024-04-26T02:06:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.697+0000 7f9a9a800640 1 -- 172.21.15.6:0/3161710133 learned_addr learned my addr 172.21.15.6:0/3161710133 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:48.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.697+0000 7f9a99400640 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59552d0 0x5613d595d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.699 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.697+0000 7f9a9a800640 1 -- 172.21.15.6:0/3161710133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59552d0 msgr2=0x5613d595d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.697+0000 7f9a9a800640 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59552d0 0x5613d595d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.697+0000 7f9a9a800640 1 -- 172.21.15.6:0/3161710133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d595fb90 msgr2=0x5613d595d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.697+0000 7f9a9a800640 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d595fb90 0x5613d595d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.697+0000 7f9a9a800640 1 -- 172.21.15.6:0/3161710133 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613d5699620 con 0x5613d59549b0 2024-04-26T02:06:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.697+0000 7f9a99400640 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59552d0 0x5613d595d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:06:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.698+0000 7f9a9a800640 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 0x5613d595cb50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9a90006010 tx=0x7f9a90003e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d42384e37f47fcc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.698+0000 7f9a77400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-26T02:06:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.698+0000 7f9a77400640 1 -- 172.21.15.6:0/3161710133 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90031db0 con 0x5613d59549b0 2024-04-26T02:06:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.698+0000 7f9a77400640 1 -- 172.21.15.6:0/3161710133 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 8556+0+0 (secure 0 0 0) 0x7f9a90031450 con 0x5613d59549b0 2024-04-26T02:06:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.698+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3161710133 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a90036070 con 0x5613d59549b0 2024-04-26T02:06:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.699+0000 7f9a77400640 1 -- 172.21.15.6:0/3161710133 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f9a90031650 con 0x5613d59549b0 2024-04-26T02:06:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.699+0000 7f9a9f1fd740 10 client.27574 fetch_fsmap learned FSMap version 58 2024-04-26T02:06:48.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.699+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3161710133 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5613d5962760 con 0x5613d59549b0 2024-04-26T02:06:48.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.699+0000 7f9a77400640 1 -- 172.21.15.6:0/3161710133 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a90036070 con 0x5613d59549b0 2024-04-26T02:06:48.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.699+0000 7f9a77400640 1 -- 172.21.15.6:0/3161710133 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 58) ==== 8342+0+0 (secure 0 0 0) 0x7f9a9004a070 con 0x5613d59549b0 2024-04-26T02:06:48.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.700+0000 7f9a9f1fd740 10 client.27574 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:06:48.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.700+0000 7f9a9f1fd740 10 client.27574 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:06:48.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.700+0000 7f9a9f1fd740 20 client.27574 populate_metadata read hostname 'smithi006' 2024-04-26T02:06:48.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.700+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5613d596fb70 0x5613d5971f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.700+0000 7f9a9f1fd740 4 client.27574 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:06:48.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.700+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3161710133 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613d58ce8b0 con 0x5613d596fb70 2024-04-26T02:06:48.701 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.700+0000 7f9a99400640 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5613d596fb70 0x5613d5971f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.701+0000 7f9a99400640 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5613d596fb70 0x5613d5971f60 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d879ab16d5df8f0 server_cookie=5827d262dc2cdaa4 in_seq=0 out_seq=0 2024-04-26T02:06:48.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.701+0000 7f9a77400640 10 client.27574 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:48.702 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.702+0000 7f9a77400640 1 -- 172.21.15.6:0/3161710133 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613d58ce8b0 con 0x5613d596fb70 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.702+0000 7f9a77400640 10 client.27574 handle_command_reply: tid=0 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.702+0000 7f9a9f1fd740 1 client.27574 shutdown 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.702+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3161710133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 msgr2=0x5613d595cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.702+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 0x5613d595cb50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9a90006010 tx=0x7f9a90003e50 comp rx=0 tx=0).stop 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.702+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3161710133 shutdown_connections 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.703+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5613d596fb70 0x5613d5971f60 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.703+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d595fb90 0x5613d595d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.703+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59552d0 0x5613d595d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.703+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/3161710133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 0x5613d595cb50 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.703+0000 7f9a77400640 0 client.27574 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.703+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3161710133 >> 172.21.15.6:0/3161710133 conn(0x5613d5950ec0 msgr2=0x5613d5956bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.703+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3161710133 shutdown_connections 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.703+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3161710133 wait complete. 2024-04-26T02:06:48.703 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.703+0000 7f9a9f1fd740 20 client.27574 trim_cache size 0 max 16384 2024-04-26T02:06:48.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.726+0000 7f9a9f1fd740 1 Processor -- start 2024-04-26T02:06:48.726 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.726+0000 7f9a9f1fd740 1 -- start start 2024-04-26T02:06:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.727+0000 7f9a9f1fd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 0x5613d59507a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.727+0000 7f9a9f1fd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59552d0 0x5613d5957370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.727+0000 7f9a9f1fd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d595fb90 0x5613d5977780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.727+0000 7f9a9f1fd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941ae090 con 0x5613d59552d0 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.727+0000 7f9a9f1fd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941ab930 con 0x5613d595fb90 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.727+0000 7f9a9f1fd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941a9780 con 0x5613d59549b0 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.727+0000 7f9a9a800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 0x5613d59507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.727+0000 7f9a99400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59552d0 0x5613d5957370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.727+0000 7f9a9a800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 0x5613d59507a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:48738/0 (socket says 172.21.15.6:48738) 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.727+0000 7f9a9a800640 1 -- 172.21.15.6:0/1705819760 learned_addr learned my addr 172.21.15.6:0/1705819760 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.727+0000 7f9a9a800640 1 --2- 172.21.15.6:0/1705819760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d595fb90 0x5613d5977780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.728+0000 7f9a9a800640 1 -- 172.21.15.6:0/1705819760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d595fb90 msgr2=0x5613d5977780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.728+0000 7f9a9a800640 1 --2- 172.21.15.6:0/1705819760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d595fb90 0x5613d5977780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.728+0000 7f9a9a800640 1 -- 172.21.15.6:0/1705819760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59552d0 msgr2=0x5613d5957370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.728+0000 7f9a9a800640 1 --2- 172.21.15.6:0/1705819760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59552d0 0x5613d5957370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.728+0000 7f9a9a800640 1 -- 172.21.15.6:0/1705819760 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613d59724f0 con 0x5613d59549b0 2024-04-26T02:06:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.728+0000 7f9a99400640 1 --2- 172.21.15.6:0/1705819760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59552d0 0x5613d5957370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:06:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.728+0000 7f9a9a800640 1 --2- 172.21.15.6:0/1705819760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 0x5613d59507a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x5613d595dee0 tx=0x7f9a90043040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9314bb559fdd49a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.728+0000 7f9a82600640 1 -- 172.21.15.6:0/1705819760 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90065030 con 0x5613d59549b0 2024-04-26T02:06:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.728+0000 7f9a82600640 1 -- 172.21.15.6:0/1705819760 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a90036070 con 0x5613d59549b0 2024-04-26T02:06:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.728+0000 7f9a82600640 1 -- 172.21.15.6:0/1705819760 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90031e10 con 0x5613d59549b0 2024-04-26T02:06:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.729+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/1705819760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 msgr2=0x5613d59507a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.729+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/1705819760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 0x5613d59507a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x5613d595dee0 tx=0x7f9a90043040 comp rx=0 tx=0).stop 2024-04-26T02:06:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.729+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/1705819760 shutdown_connections 2024-04-26T02:06:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.729+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/1705819760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59552d0 0x5613d5957370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.729+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/1705819760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d595fb90 0x5613d5977780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.729+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/1705819760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d59549b0 0x5613d59507a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.729+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/1705819760 >> 172.21.15.6:0/1705819760 conn(0x5613d5950ec0 msgr2=0x5613d595d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.729+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/1705819760 shutdown_connections 2024-04-26T02:06:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.729+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/1705819760 wait complete. 2024-04-26T02:06:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.730+0000 7f9a9f1fd740 1 -- start start 2024-04-26T02:06:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.730+0000 7f9a9f1fd740 1 Processor -- start 2024-04-26T02:06:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.731+0000 7f9a9f1fd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59549b0 0x5613d5967e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.731+0000 7f9a9f1fd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59552d0 0x5613d5965220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.731+0000 7f9a9f1fd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d595fb90 0x5613d5965760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.731+0000 7f9a9f1fd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941a9df0 con 0x5613d59552d0 2024-04-26T02:06:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.731+0000 7f9a9f1fd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941ab260 con 0x5613d59549b0 2024-04-26T02:06:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.731+0000 7f9a9f1fd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941aa4c0 con 0x5613d595fb90 2024-04-26T02:06:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.731+0000 7f9a9f1fd740 10 client.0 _ll_register_callbacks cb 0x7f9a9002f310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.731+0000 7f9a9a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59549b0 0x5613d5967e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.732+0000 7f9a9a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59549b0 0x5613d5967e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:45694/0 (socket says 172.21.15.6:45694) 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.732+0000 7f9a9a800640 1 -- 172.21.15.6:0/3925218557 learned_addr learned my addr 172.21.15.6:0/3925218557 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.732+0000 7f9a9a800640 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d595fb90 0x5613d5965760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.732+0000 7f9a99400640 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59552d0 0x5613d5965220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.732+0000 7f9a9a800640 1 -- 172.21.15.6:0/3925218557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d595fb90 msgr2=0x5613d5965760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.732+0000 7f9a9a800640 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d595fb90 0x5613d5965760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.732+0000 7f9a9a800640 1 -- 172.21.15.6:0/3925218557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59552d0 msgr2=0x5613d5965220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.732+0000 7f9a9a800640 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59552d0 0x5613d5965220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.732+0000 7f9a9a800640 1 -- 172.21.15.6:0/3925218557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613d5965f30 con 0x5613d59549b0 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.733+0000 7f9a9a800640 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59549b0 0x5613d5967e90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9a9002ae80 tx=0x7f9a900319c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c36bcd4ca740393 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.733+0000 7f9a77400640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.733+0000 7f9a77400640 1 -- 172.21.15.6:0/3925218557 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90065040 con 0x5613d59549b0 2024-04-26T02:06:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.733+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3925218557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a90036070 con 0x5613d59549b0 2024-04-26T02:06:48.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.733+0000 7f9a77400640 1 -- 172.21.15.6:0/3925218557 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 8556+0+0 (secure 0 0 0) 0x7f9a9002fe70 con 0x5613d59549b0 2024-04-26T02:06:48.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.733+0000 7f9a77400640 1 -- 172.21.15.6:0/3925218557 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f9a90004040 con 0x5613d59549b0 2024-04-26T02:06:48.734 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.733+0000 7f9a9f1fd740 10 client.27512 fetch_fsmap learned FSMap version 58 2024-04-26T02:06:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.733+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3925218557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5613d59d5fb0 con 0x5613d59549b0 2024-04-26T02:06:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.734+0000 7f9a77400640 1 -- 172.21.15.6:0/3925218557 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a90036070 con 0x5613d59549b0 2024-04-26T02:06:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.734+0000 7f9a77400640 1 -- 172.21.15.6:0/3925218557 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 58) ==== 8342+0+0 (secure 0 0 0) 0x7f9a9004a040 con 0x5613d59549b0 2024-04-26T02:06:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.734+0000 7f9a9f1fd740 10 client.27512 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:06:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.734+0000 7f9a9f1fd740 10 client.27512 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:06:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.734+0000 7f9a9f1fd740 20 client.27512 populate_metadata read hostname 'smithi006' 2024-04-26T02:06:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.734+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5613d59d6510 0x5613d59d8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.734+0000 7f9a9f1fd740 4 client.27512 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:06:48.735 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.734+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3925218557 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5613d58c8320 con 0x5613d59d6510 2024-04-26T02:06:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.773+0000 7f9a99400640 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5613d59d6510 0x5613d59d8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.774+0000 7f9a99400640 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5613d59d6510 0x5613d59d8900 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79570bc89e3307d0 server_cookie=83eb6fac9c35e039 in_seq=0 out_seq=0 2024-04-26T02:06:48.773 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:48.774+0000 7f9a77400640 10 client.27512 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:48.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:48 smithi006 ceph-mon[28419]: pgmap v663: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 7.4 MiB/s wr, 376 op/s 2024-04-26T02:06:48.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:48 smithi188 ceph-mon[32751]: pgmap v663: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 7.4 MiB/s wr, 376 op/s 2024-04-26T02:06:48.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:48 smithi029 ceph-mon[32775]: pgmap v663: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 7.4 MiB/s wr, 376 op/s 2024-04-26T02:06:49.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a77400640 1 -- 172.21.15.6:0/3925218557 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5613d58c8320 con 0x5613d59d6510 2024-04-26T02:06:49.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a77400640 10 client.27512 handle_command_reply: tid=0 2024-04-26T02:06:49.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a9f1fd740 1 client.27512 shutdown 2024-04-26T02:06:49.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3925218557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59549b0 msgr2=0x5613d5967e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:49.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59549b0 0x5613d5967e90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9a9002ae80 tx=0x7f9a900319c0 comp rx=0 tx=0).stop 2024-04-26T02:06:49.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3925218557 shutdown_connections 2024-04-26T02:06:49.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x5613d59d6510 0x5613d59d8900 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:49.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5613d59552d0 0x5613d5965220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:49.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5613d595fb90 0x5613d5965760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:49.081 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a9f1fd740 1 --2- 172.21.15.6:0/3925218557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613d59549b0 0x5613d5967e90 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a77400640 0 client.27512 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:06:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3925218557 >> 172.21.15.6:0/3925218557 conn(0x5613d5950ec0 msgr2=0x5613d595e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3925218557 shutdown_connections 2024-04-26T02:06:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a9f1fd740 1 -- 172.21.15.6:0/3925218557 wait complete. 2024-04-26T02:06:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.081+0000 7f9a9f1fd740 20 client.27512 trim_cache size 0 max 16384 2024-04-26T02:06:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:06:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:06:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:06:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:06:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.082+0000 7f9a9b200640 1 -- 172.21.15.6:0/1443184000 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9a7c0a3d00 msgr2=0x7f9a7c0a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.082+0000 7f9a9b200640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9a7c0a3d00 0x7f9a7c0a61c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x5613d5955f50 tx=0x7f9a880d4780 comp rx=0 tx=0).stop 2024-04-26T02:06:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.082+0000 7f9a9b200640 1 -- 172.21.15.6:0/1443184000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a941041f0 msgr2=0x7f9a9419a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.082+0000 7f9a9b200640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a941041f0 0x7f9a9419a760 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800e9f0 tx=0x7f9a8800eeb0 comp rx=0 tx=0).stop 2024-04-26T02:06:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.082+0000 7f9a9b200640 1 -- 172.21.15.6:0/1443184000 shutdown_connections 2024-04-26T02:06:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.082+0000 7f9a9b200640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f9a7c0a3d00 0x7f9a7c0a61c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.082+0000 7f9a9b200640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a9419f020 0x7f9a9419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.082+0000 7f9a9b200640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a941092c0 0x7f9a9419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.082+0000 7f9a9b200640 1 --2- 172.21.15.6:0/1443184000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a941041f0 0x7f9a9419a760 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.082+0000 7f9a9b200640 1 -- 172.21.15.6:0/1443184000 >> 172.21.15.6:0/1443184000 conn(0x7f9a940ffa50 msgr2=0x7f9a94101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.083+0000 7f9a9b200640 1 -- 172.21.15.6:0/1443184000 shutdown_connections 2024-04-26T02:06:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:06:49.083+0000 7f9a9b200640 1 -- 172.21.15.6:0/1443184000 wait complete. 2024-04-26T02:06:49.149 INFO:tasks.workunit.client.0.smithi006.stdout: 10 147898 19.97 MB/sec execute 312 sec latency 399.650 ms 2024-04-26T02:06:50.149 INFO:tasks.workunit.client.0.smithi006.stdout: 10 148470 20.02 MB/sec execute 313 sec latency 176.679 ms 2024-04-26T02:06:50.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:50 smithi006 ceph-mon[28419]: pgmap v664: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 6.7 MiB/s wr, 340 op/s 2024-04-26T02:06:50.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:50 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:50.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:50 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:50.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:50 smithi188 ceph-mon[32751]: pgmap v664: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 6.7 MiB/s wr, 340 op/s 2024-04-26T02:06:50.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:50 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:50.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:50 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:50.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:50 smithi029 ceph-mon[32775]: pgmap v664: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 6.7 MiB/s wr, 340 op/s 2024-04-26T02:06:50.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:50 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:50.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:50 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:51.149 INFO:tasks.workunit.client.0.smithi006.stdout: 10 148978 20.04 MB/sec execute 314 sec latency 84.452 ms 2024-04-26T02:06:51.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:51 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:51 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:51 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:51 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:51 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:51 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:51 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:51 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:51 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:06:51.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:51 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:06:51.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:51 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:51.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:51 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:51 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:51 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:51 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:51 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:51 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:51 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:51 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:51 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:06:51.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:51 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:06:51.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:51 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:51.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:51 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:51 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:51 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:51 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:51 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:51 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:51 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:51 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:06:51.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:51 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:06:51.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:51 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:06:51.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:51 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:06:52.149 INFO:tasks.workunit.client.0.smithi006.stdout: 10 149602 20.09 MB/sec execute 315 sec latency 107.399 ms 2024-04-26T02:06:52.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:52 smithi006 ceph-mon[28419]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-26T02:06:52.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:52 smithi006 ceph-mon[28419]: Unable to set osd_memory_target on smithi188 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-26T02:06:52.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:52 smithi006 ceph-mon[28419]: pgmap v665: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 586 op/s 2024-04-26T02:06:52.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:52 smithi188 ceph-mon[32751]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-26T02:06:52.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:52 smithi188 ceph-mon[32751]: Unable to set osd_memory_target on smithi188 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-26T02:06:52.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:52 smithi188 ceph-mon[32751]: pgmap v665: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 586 op/s 2024-04-26T02:06:52.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:52 smithi029 ceph-mon[32775]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-26T02:06:52.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:52 smithi029 ceph-mon[32775]: Unable to set osd_memory_target on smithi188 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-26T02:06:52.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:52 smithi029 ceph-mon[32775]: pgmap v665: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 586 op/s 2024-04-26T02:06:53.149 INFO:tasks.workunit.client.0.smithi006.stdout: 10 150006 20.11 MB/sec execute 316 sec latency 221.445 ms 2024-04-26T02:06:54.151 INFO:tasks.workunit.client.0.smithi006.stdout: 10 150462 20.10 MB/sec execute 317 sec latency 91.935 ms 2024-04-26T02:06:54.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:54 smithi006 ceph-mon[28419]: pgmap v666: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.4 MiB/s wr, 443 op/s 2024-04-26T02:06:54.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:54 smithi188 ceph-mon[32751]: pgmap v666: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.4 MiB/s wr, 443 op/s 2024-04-26T02:06:54.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:54 smithi029 ceph-mon[32775]: pgmap v666: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.4 MiB/s wr, 443 op/s 2024-04-26T02:06:55.151 INFO:tasks.workunit.client.0.smithi006.stdout: 10 150957 20.13 MB/sec execute 318 sec latency 102.875 ms 2024-04-26T02:06:55.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:55 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:55.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:55 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:56.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:55 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:56.151 INFO:tasks.workunit.client.0.smithi006.stdout: 10 151468 20.16 MB/sec execute 319 sec latency 112.605 ms 2024-04-26T02:06:56.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:56 smithi188 ceph-mon[32751]: pgmap v667: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.1 MiB/s wr, 429 op/s 2024-04-26T02:06:56.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:56 smithi029 ceph-mon[32775]: pgmap v667: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.1 MiB/s wr, 429 op/s 2024-04-26T02:06:57.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:56 smithi006 ceph-mon[28419]: pgmap v667: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.1 MiB/s wr, 429 op/s 2024-04-26T02:06:57.151 INFO:tasks.workunit.client.0.smithi006.stdout: 10 151919 20.15 MB/sec execute 320 sec latency 150.472 ms 2024-04-26T02:06:58.151 INFO:tasks.workunit.client.0.smithi006.stdout: 10 152311 20.14 MB/sec execute 321 sec latency 325.821 ms 2024-04-26T02:06:58.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:06:58 smithi188 ceph-mon[32751]: pgmap v668: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 657 op/s 2024-04-26T02:06:58.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:06:58 smithi029 ceph-mon[32775]: pgmap v668: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 657 op/s 2024-04-26T02:06:59.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:06:58 smithi006 ceph-mon[28419]: pgmap v668: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 657 op/s 2024-04-26T02:06:59.151 INFO:tasks.workunit.client.0.smithi006.stdout: 10 152930 20.20 MB/sec execute 322 sec latency 334.233 ms 2024-04-26T02:07:00.152 INFO:tasks.workunit.client.0.smithi006.stdout: 10 153530 20.26 MB/sec execute 323 sec latency 82.030 ms 2024-04-26T02:07:00.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:00 smithi188 ceph-mon[32751]: pgmap v669: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.8 MiB/s wr, 493 op/s 2024-04-26T02:07:00.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:00 smithi029 ceph-mon[32775]: pgmap v669: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.8 MiB/s wr, 493 op/s 2024-04-26T02:07:01.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:00 smithi006 ceph-mon[28419]: pgmap v669: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.8 MiB/s wr, 493 op/s 2024-04-26T02:07:01.152 INFO:tasks.workunit.client.0.smithi006.stdout: 10 153836 20.24 MB/sec execute 324 sec latency 432.916 ms 2024-04-26T02:07:02.152 INFO:tasks.workunit.client.0.smithi006.stdout: 10 153836 20.18 MB/sec execute 325 sec latency 1433.104 ms 2024-04-26T02:07:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.232+0000 7fac63000640 1 -- 172.21.15.6:0/543112611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac5c100f70 msgr2=0x7fac5c10bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.232+0000 7fac63000640 1 --2- 172.21.15.6:0/543112611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac5c100f70 0x7fac5c10bff0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fac48009860 tx=0x7fac4802a3d0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.232+0000 7fac63000640 1 -- 172.21.15.6:0/543112611 shutdown_connections 2024-04-26T02:07:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.232+0000 7fac63000640 1 --2- 172.21.15.6:0/543112611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac5c100f70 0x7fac5c10bff0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.232+0000 7fac63000640 1 --2- 172.21.15.6:0/543112611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac5c10c530 0x7fac5c10e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.232+0000 7fac63000640 1 --2- 172.21.15.6:0/543112611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac5c100650 0x7fac5c100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.232+0000 7fac63000640 1 -- 172.21.15.6:0/543112611 >> 172.21.15.6:0/543112611 conn(0x7fac5c075d60 msgr2=0x7fac5c076170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.232+0000 7fac63000640 1 -- 172.21.15.6:0/543112611 shutdown_connections 2024-04-26T02:07:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.232+0000 7fac63000640 1 -- 172.21.15.6:0/543112611 wait complete. 2024-04-26T02:07:02.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.233+0000 7fac63000640 1 Processor -- start 2024-04-26T02:07:02.233 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.234+0000 7fac63000640 1 -- start start 2024-04-26T02:07:02.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.234+0000 7fac63000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac5c100650 0x7fac5c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.234+0000 7fac63000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac5c100f70 0x7fac5c1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.234+0000 7fac63000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac5c10c530 0x7fac5c19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.234 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.234+0000 7fac63000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1121c0 con 0x7fac5c100650 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.234+0000 7fac63000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c112040 con 0x7fac5c100f70 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.234+0000 7fac63000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c112340 con 0x7fac5c10c530 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.234+0000 7fac5be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac5c100650 0x7fac5c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.234+0000 7fac5be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac5c100650 0x7fac5c1a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44650/0 (socket says 172.21.15.6:44650) 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.234+0000 7fac5be00640 1 -- 172.21.15.6:0/2511646805 learned_addr learned my addr 172.21.15.6:0/2511646805 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.234+0000 7fac5b400640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac5c100f70 0x7fac5c1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.235+0000 7fac61000640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac5c10c530 0x7fac5c19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.235+0000 7fac5b400640 1 -- 172.21.15.6:0/2511646805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac5c10c530 msgr2=0x7fac5c19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.235+0000 7fac5b400640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac5c10c530 0x7fac5c19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.235+0000 7fac5b400640 1 -- 172.21.15.6:0/2511646805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac5c100650 msgr2=0x7fac5c1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.235+0000 7fac5b400640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac5c100650 0x7fac5c1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.235+0000 7fac5b400640 1 -- 172.21.15.6:0/2511646805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5c19af90 con 0x7fac5c100f70 2024-04-26T02:07:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.235+0000 7fac5b400640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac5c100f70 0x7fac5c1a0b90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fac4802a8f0 tx=0x7fac48030fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.235+0000 7fac58c00640 1 -- 172.21.15.6:0/2511646805 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48040070 con 0x7fac5c100f70 2024-04-26T02:07:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.235+0000 7fac58c00640 1 -- 172.21.15.6:0/2511646805 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac48005570 con 0x7fac5c100f70 2024-04-26T02:07:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.235+0000 7fac63000640 1 -- 172.21.15.6:0/2511646805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5c19b220 con 0x7fac5c100f70 2024-04-26T02:07:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.236+0000 7fac58c00640 1 -- 172.21.15.6:0/2511646805 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac480046c0 con 0x7fac5c100f70 2024-04-26T02:07:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.236+0000 7fac63000640 1 -- 172.21.15.6:0/2511646805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5c1a74a0 con 0x7fac5c100f70 2024-04-26T02:07:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.239+0000 7fac65de3740 1 Processor -- start 2024-04-26T02:07:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.239+0000 7fac65de3740 1 -- start start 2024-04-26T02:07:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.239+0000 7fac58c00640 1 -- 172.21.15.6:0/2511646805 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fac48043070 con 0x7fac5c100f70 2024-04-26T02:07:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.239+0000 7fac65de3740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef17c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.239+0000 7fac65de3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef224c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.239+0000 7fac65de3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef24df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.239+0000 7fac65de3740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55fccee7d900 con 0x55fccef17820 2024-04-26T02:07:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.239+0000 7fac65de3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55fccee23ef0 con 0x55fccef18140 2024-04-26T02:07:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.239+0000 7fac65de3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fccee798c0 con 0x55fccef22a00 2024-04-26T02:07:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.240+0000 7fac61000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef17c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.240+0000 7fac61000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef17c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44658/0 (socket says 172.21.15.6:44658) 2024-04-26T02:07:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.240+0000 7fac5b400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef24df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.240+0000 7fac61000640 1 -- 172.21.15.6:0/3013452278 learned_addr learned my addr 172.21.15.6:0/3013452278 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.240+0000 7fac5be00640 1 --2- 172.21.15.6:0/3013452278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef224c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.240+0000 7fac61000640 1 -- 172.21.15.6:0/3013452278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 msgr2=0x55fccef24df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.240+0000 7fac61000640 1 --2- 172.21.15.6:0/3013452278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef24df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.240+0000 7fac61000640 1 -- 172.21.15.6:0/3013452278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 msgr2=0x55fccef224c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.240+0000 7fac61000640 1 --2- 172.21.15.6:0/3013452278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef224c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.240+0000 7fac61000640 1 -- 172.21.15.6:0/3013452278 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcceb90210 con 0x55fccef17820 2024-04-26T02:07:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.240+0000 7fac5b400640 1 --2- 172.21.15.6:0/3013452278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef24df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.241+0000 7fac61000640 1 --2- 172.21.15.6:0/3013452278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef17c00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fac5400e480 tx=0x7fac540095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f5cc35748d24aa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.241+0000 7fac51600640 1 -- 172.21.15.6:0/3013452278 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54003970 con 0x55fccef17820 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.241+0000 7fac51600640 1 -- 172.21.15.6:0/3013452278 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac5400d7d0 con 0x55fccef17820 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.241+0000 7fac51600640 1 -- 172.21.15.6:0/3013452278 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54004350 con 0x55fccef17820 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.241+0000 7fac58c00640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fac400a3d10 0x7fac400a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.241+0000 7fac58c00640 1 -- 172.21.15.6:0/2511646805 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 8556+0+0 (secure 0 0 0) 0x7fac480f13c0 con 0x7fac5c100f70 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.242+0000 7fac5be00640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fac400a3d10 0x7fac400a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.242+0000 7fac65de3740 1 -- 172.21.15.6:0/3013452278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 msgr2=0x55fccef17c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.242+0000 7fac65de3740 1 --2- 172.21.15.6:0/3013452278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef17c00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fac5400e480 tx=0x7fac540095c0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.242+0000 7fac65de3740 1 -- 172.21.15.6:0/3013452278 shutdown_connections 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.242+0000 7fac65de3740 1 --2- 172.21.15.6:0/3013452278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef24df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.242+0000 7fac65de3740 1 --2- 172.21.15.6:0/3013452278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef224c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.242+0000 7fac65de3740 1 --2- 172.21.15.6:0/3013452278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef17c00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.242+0000 7fac65de3740 1 -- 172.21.15.6:0/3013452278 >> 172.21.15.6:0/3013452278 conn(0x55fccef13d30 msgr2=0x55fccef14110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.242+0000 7fac65de3740 1 -- 172.21.15.6:0/3013452278 shutdown_connections 2024-04-26T02:07:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.242+0000 7fac65de3740 1 -- 172.21.15.6:0/3013452278 wait complete. 2024-04-26T02:07:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.242+0000 7fac5be00640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fac400a3d10 0x7fac400a61d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x55fccef18dc0 tx=0x7fac4c005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.243+0000 7fac65de3740 1 -- start start 2024-04-26T02:07:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.243+0000 7fac65de3740 1 Processor -- start 2024-04-26T02:07:02.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.243+0000 7fac65de3740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef1f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.243+0000 7fac65de3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef1feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.247 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.243+0000 7fac65de3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef203f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.243+0000 7fac65de3740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55fccef25630 con 0x55fccef17820 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.243+0000 7fac65de3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55fccee37310 con 0x55fccef18140 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.243+0000 7fac65de3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fccef257b0 con 0x55fccef22a00 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.243+0000 7fac65de3740 10 client.0 _ll_register_callbacks cb 0x55fccee05cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac5be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef1f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac5b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef1feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac5be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef1f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44660/0 (socket says 172.21.15.6:44660) 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac5be00640 1 -- 172.21.15.6:0/1844843779 learned_addr learned my addr 172.21.15.6:0/1844843779 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac5b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef1feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:50542/0 (socket says 172.21.15.6:50542) 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac61000640 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef203f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac5b400640 1 -- 172.21.15.6:0/1844843779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 msgr2=0x55fccef203f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac5b400640 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef203f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.248 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac5b400640 1 -- 172.21.15.6:0/1844843779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 msgr2=0x55fccef1f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac5b400640 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef1f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac5b400640 1 -- 172.21.15.6:0/1844843779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fccec5c620 con 0x55fccef18140 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac5be00640 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef1f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac5b400640 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef1feb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fac480eea60 tx=0x7fac48005720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad54dac7867e1327 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.244+0000 7fac46000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac46000640 1 -- 172.21.15.6:0/1844843779 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4804a070 con 0x55fccef18140 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac46000640 1 -- 172.21.15.6:0/1844843779 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 8556+0+0 (secure 0 0 0) 0x7fac480c5ac0 con 0x55fccef18140 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac65de3740 1 -- 172.21.15.6:0/1844843779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac5400d7d0 con 0x55fccef18140 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac46000640 1 -- 172.21.15.6:0/1844843779 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fac480bd160 con 0x55fccef18140 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac65de3740 10 client.27524 fetch_fsmap learned FSMap version 59 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac65de3740 1 -- 172.21.15.6:0/1844843779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55fccef31870 con 0x55fccef18140 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac46000640 1 -- 172.21.15.6:0/1844843779 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac5400d7d0 con 0x55fccef18140 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac46000640 1 -- 172.21.15.6:0/1844843779 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 59) ==== 8338+0+0 (secure 0 0 0) 0x7fac480bc7e0 con 0x55fccef18140 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac65de3740 10 client.27524 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T02:07:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac65de3740 10 client.27524 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac65de3740 20 client.27524 populate_metadata read hostname 'smithi006' 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac65de3740 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fccef32d20 0x55fccef35170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac65de3740 4 client.27524 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.245+0000 7fac65de3740 1 -- 172.21.15.6:0/1844843779 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fccee918b0 con 0x55fccef32d20 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.246+0000 7fac5be00640 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fccef32d20 0x55fccef35170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.246+0000 7fac5be00640 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fccef32d20 0x55fccef35170 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72955103879ff34a server_cookie=315dd7f5caa4050c in_seq=0 out_seq=0 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.246+0000 7fac46000640 10 client.27524 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac46000640 1 -- 172.21.15.6:0/1844843779 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fccee918b0 con 0x55fccef32d20 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac46000640 10 client.27524 handle_command_reply: tid=0 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac65de3740 1 client.27524 shutdown 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac65de3740 1 -- 172.21.15.6:0/1844843779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 msgr2=0x55fccef1feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac65de3740 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef1feb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fac480eea60 tx=0x7fac48005720 comp rx=0 tx=0).stop 2024-04-26T02:07:02.250 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac65de3740 1 -- 172.21.15.6:0/1844843779 shutdown_connections 2024-04-26T02:07:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac65de3740 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fccef32d20 0x55fccef35170 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac65de3740 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef203f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac65de3740 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef1feb0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac65de3740 1 --2- 172.21.15.6:0/1844843779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef1f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac65de3740 1 -- 172.21.15.6:0/1844843779 >> 172.21.15.6:0/1844843779 conn(0x55fccef13d30 msgr2=0x55fccef19ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac46000640 0 client.27524 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:07:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac65de3740 1 -- 172.21.15.6:0/1844843779 shutdown_connections 2024-04-26T02:07:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac65de3740 1 -- 172.21.15.6:0/1844843779 wait complete. 2024-04-26T02:07:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.248+0000 7fac65de3740 20 client.27524 trim_cache size 0 max 16384 2024-04-26T02:07:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.265+0000 7fac65de3740 1 Processor -- start 2024-04-26T02:07:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.265+0000 7fac65de3740 1 -- start start 2024-04-26T02:07:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.265+0000 7fac65de3740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef1a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.265+0000 7fac65de3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef18140 0x55fccef20040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.265+0000 7fac65de3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef22a00 0x55fccef3aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.265+0000 7fac65de3740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1b26e0 con 0x55fccef17820 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.265+0000 7fac65de3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1affb0 con 0x55fccef22a00 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.265+0000 7fac65de3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1ade00 con 0x55fccef18140 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.266+0000 7fac5be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef1a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.266+0000 7fac5be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef1a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44666/0 (socket says 172.21.15.6:44666) 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.266+0000 7fac5be00640 1 -- 172.21.15.6:0/1640638229 learned_addr learned my addr 172.21.15.6:0/1640638229 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.266+0000 7fac61000640 1 --2- 172.21.15.6:0/1640638229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef22a00 0x55fccef3aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.266+0000 7fac5b400640 1 --2- 172.21.15.6:0/1640638229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef18140 0x55fccef20040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.266+0000 7fac5be00640 1 -- 172.21.15.6:0/1640638229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef18140 msgr2=0x55fccef20040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.266+0000 7fac5be00640 1 --2- 172.21.15.6:0/1640638229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef18140 0x55fccef20040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.266+0000 7fac5be00640 1 -- 172.21.15.6:0/1640638229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef22a00 msgr2=0x55fccef3aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.266+0000 7fac5be00640 1 --2- 172.21.15.6:0/1640638229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef22a00 0x55fccef3aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.266+0000 7fac5be00640 1 -- 172.21.15.6:0/1640638229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fccef357f0 con 0x55fccef17820 2024-04-26T02:07:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.266+0000 7fac5be00640 1 --2- 172.21.15.6:0/1640638229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef1a230 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fac4c009c70 tx=0x7fac4c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e13542335a8018 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:02.269 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.267+0000 7fac50c00640 1 -- 172.21.15.6:0/1640638229 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c02b470 con 0x55fccef17820 2024-04-26T02:07:02.269 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.267+0000 7fac50c00640 1 -- 172.21.15.6:0/1640638229 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac4c030070 con 0x55fccef17820 2024-04-26T02:07:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.267+0000 7fac50c00640 1 -- 172.21.15.6:0/1640638229 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c01e650 con 0x55fccef17820 2024-04-26T02:07:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.267+0000 7fac65de3740 1 -- 172.21.15.6:0/1640638229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 msgr2=0x55fccef1a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.267+0000 7fac65de3740 1 --2- 172.21.15.6:0/1640638229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef1a230 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fac4c009c70 tx=0x7fac4c01b000 comp rx=0 tx=0).stop 2024-04-26T02:07:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.267+0000 7fac65de3740 1 -- 172.21.15.6:0/1640638229 shutdown_connections 2024-04-26T02:07:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.267+0000 7fac65de3740 1 --2- 172.21.15.6:0/1640638229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef22a00 0x55fccef3aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.267+0000 7fac65de3740 1 --2- 172.21.15.6:0/1640638229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef18140 0x55fccef20040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.267+0000 7fac65de3740 1 --2- 172.21.15.6:0/1640638229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef1a230 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.267+0000 7fac65de3740 1 -- 172.21.15.6:0/1640638229 >> 172.21.15.6:0/1640638229 conn(0x55fccef13d30 msgr2=0x55fccef1f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.267+0000 7fac65de3740 1 -- 172.21.15.6:0/1640638229 shutdown_connections 2024-04-26T02:07:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.267+0000 7fac65de3740 1 -- 172.21.15.6:0/1640638229 wait complete. 2024-04-26T02:07:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.268+0000 7fac65de3740 1 -- start start 2024-04-26T02:07:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.268+0000 7fac65de3740 1 Processor -- start 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.268+0000 7fac65de3740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef28670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.268+0000 7fac65de3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef28bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac65de3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef290f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac65de3740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1ae470 con 0x55fccef17820 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac65de3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1af8e0 con 0x55fccef18140 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac65de3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1aeb40 con 0x55fccef22a00 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac65de3740 10 client.0 _ll_register_callbacks cb 0x7fac48005a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac5be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef28670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac5be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef28670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44676/0 (socket says 172.21.15.6:44676) 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac5be00640 1 -- 172.21.15.6:0/204873758 learned_addr learned my addr 172.21.15.6:0/204873758 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac61000640 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef290f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac5b400640 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef28bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac5be00640 1 -- 172.21.15.6:0/204873758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 msgr2=0x55fccef290f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac5be00640 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef290f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac5be00640 1 -- 172.21.15.6:0/204873758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 msgr2=0x55fccef28bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac5be00640 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef28bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac5be00640 1 -- 172.21.15.6:0/204873758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fccefa0d60 con 0x55fccef17820 2024-04-26T02:07:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.269+0000 7fac5b400640 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef28bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:07:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.270+0000 7fac5be00640 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef28670 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fac4c02d040 tx=0x7fac4c012e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846b79c1466313c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.270+0000 7fac46000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:07:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.270+0000 7fac46000640 1 -- 172.21.15.6:0/204873758 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c038070 con 0x55fccef17820 2024-04-26T02:07:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.270+0000 7fac46000640 1 -- 172.21.15.6:0/204873758 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 8556+0+0 (secure 0 0 0) 0x7fac4c01f850 con 0x55fccef17820 2024-04-26T02:07:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.270+0000 7fac65de3740 1 -- 172.21.15.6:0/204873758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac4c030070 con 0x55fccef17820 2024-04-26T02:07:02.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.270+0000 7fac46000640 1 -- 172.21.15.6:0/204873758 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fac4c01fa70 con 0x55fccef17820 2024-04-26T02:07:02.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.270+0000 7fac65de3740 10 client.17319 fetch_fsmap learned FSMap version 59 2024-04-26T02:07:02.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.270+0000 7fac65de3740 1 -- 172.21.15.6:0/204873758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55fccefa2050 con 0x55fccef17820 2024-04-26T02:07:02.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.271+0000 7fac46000640 1 -- 172.21.15.6:0/204873758 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac4c030070 con 0x55fccef17820 2024-04-26T02:07:02.273 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.271+0000 7fac46000640 1 -- 172.21.15.6:0/204873758 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 59) ==== 8338+0+0 (secure 0 0 0) 0x7fac4c005410 con 0x55fccef17820 2024-04-26T02:07:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.271+0000 7fac65de3740 10 client.17319 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T02:07:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.271+0000 7fac65de3740 10 client.17319 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-26T02:07:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.271+0000 7fac65de3740 20 client.17319 populate_metadata read hostname 'smithi006' 2024-04-26T02:07:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.271+0000 7fac65de3740 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fccefa2710 0x55fccefa4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.271+0000 7fac65de3740 4 client.17319 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:07:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.271+0000 7fac65de3740 1 -- 172.21.15.6:0/204873758 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fccee8b320 con 0x55fccefa2710 2024-04-26T02:07:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.272+0000 7fac5b400640 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fccefa2710 0x55fccefa4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.272+0000 7fac5b400640 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fccefa2710 0x55fccefa4b00 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8096e7bde43744b8 server_cookie=b91ec543c1d99cbc in_seq=0 out_seq=0 2024-04-26T02:07:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.272+0000 7fac46000640 10 client.17319 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:07:02.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.484+0000 7fac46000640 1 -- 172.21.15.6:0/204873758 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fccee8b320 con 0x55fccefa2710 2024-04-26T02:07:02.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.484+0000 7fac46000640 10 client.17319 handle_command_reply: tid=0 2024-04-26T02:07:02.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac65de3740 1 client.17319 shutdown 2024-04-26T02:07:02.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac65de3740 1 -- 172.21.15.6:0/204873758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 msgr2=0x55fccef28670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac65de3740 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef28670 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fac4c02d040 tx=0x7fac4c012e70 comp rx=0 tx=0).stop 2024-04-26T02:07:02.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac65de3740 1 -- 172.21.15.6:0/204873758 shutdown_connections 2024-04-26T02:07:02.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac65de3740 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x55fccefa2710 0x55fccefa4b00 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac65de3740 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fccef22a00 0x55fccef290f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac65de3740 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fccef18140 0x55fccef28bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac65de3740 1 --2- 172.21.15.6:0/204873758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55fccef17820 0x55fccef28670 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac46000640 0 client.17319 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:07:02.485 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac65de3740 1 -- 172.21.15.6:0/204873758 >> 172.21.15.6:0/204873758 conn(0x55fccef13d30 msgr2=0x55fccef1f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:02.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac65de3740 1 -- 172.21.15.6:0/204873758 shutdown_connections 2024-04-26T02:07:02.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac65de3740 1 -- 172.21.15.6:0/204873758 wait complete. 2024-04-26T02:07:02.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.485+0000 7fac65de3740 20 client.17319 trim_cache size 0 max 16384 2024-04-26T02:07:02.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:{ 2024-04-26T02:07:02.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "message": "", 2024-04-26T02:07:02.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout: "return_code": 0 2024-04-26T02:07:02.486 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stdout:} 2024-04-26T02:07:02.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.487+0000 7fac63000640 1 -- 172.21.15.6:0/2511646805 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fac400a3d10 msgr2=0x7fac400a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.487+0000 7fac63000640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fac400a3d10 0x7fac400a61d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x55fccef18dc0 tx=0x7fac4c005f90 comp rx=0 tx=0).stop 2024-04-26T02:07:02.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.487+0000 7fac63000640 1 -- 172.21.15.6:0/2511646805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac5c100f70 msgr2=0x7fac5c1a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.487+0000 7fac63000640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac5c100f70 0x7fac5c1a0b90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fac4802a8f0 tx=0x7fac48030fa0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.487+0000 7fac63000640 1 -- 172.21.15.6:0/2511646805 shutdown_connections 2024-04-26T02:07:02.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.487+0000 7fac63000640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fac400a3d10 0x7fac400a61d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.487+0000 7fac63000640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac5c10c530 0x7fac5c19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.487+0000 7fac63000640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac5c100f70 0x7fac5c1a0b90 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.487 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.487+0000 7fac63000640 1 --2- 172.21.15.6:0/2511646805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac5c100650 0x7fac5c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.487+0000 7fac63000640 1 -- 172.21.15.6:0/2511646805 >> 172.21.15.6:0/2511646805 conn(0x7fac5c075d60 msgr2=0x7fac5c101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:02.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.487+0000 7fac63000640 1 -- 172.21.15.6:0/2511646805 shutdown_connections 2024-04-26T02:07:02.488 INFO:teuthology.task.background_exec.ubuntu@smithi006.front.sepia.ceph.com.smithi006.stderr:2024-04-26T02:07:02.487+0000 7fac63000640 1 -- 172.21.15.6:0/2511646805 wait complete. 2024-04-26T02:07:02.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:02 smithi188 ceph-mon[32751]: pgmap v670: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 673 op/s 2024-04-26T02:07:02.924 INFO:tasks.workunit.client.0.smithi006.stdout:[147565] write failed on handle 11914 (Resource temporarily unavailable) 2024-04-26T02:07:02.925 INFO:tasks.workunit.client.0.smithi006.stdout:Child failed with status 1 2024-04-26T02:07:02.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:07:02.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:02 smithi029 ceph-mon[32775]: pgmap v670: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 673 op/s 2024-04-26T02:07:03.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:02 smithi006 ceph-mon[28419]: pgmap v670: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 673 op/s 2024-04-26T02:07:03.152 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-04-26T02:07:03.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-26T02:07:04.429 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi006 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afa1933b0fdbb6c99c947d1eda34d661d23cd327 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-26T02:07:04.637 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c1ec8d2ba0174cc1916ce0d0ce23f3fd Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi006 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afa1933b0fdbb6c99c947d1eda34d661d23cd327 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-26T02:07:04.640 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-26T02:07:04.651 INFO:tasks.quiescer:joining Quiescers 2024-04-26T02:07:04.651 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-04-26T02:07:04.652 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:07:04.744 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:04 smithi006 ceph-mon[28419]: pgmap v671: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.5 MiB/s wr, 452 op/s 2024-04-26T02:07:04.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:04 smithi188 ceph-mon[32751]: pgmap v671: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.5 MiB/s wr, 452 op/s 2024-04-26T02:07:04.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:04 smithi029 ceph-mon[32775]: pgmap v671: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 9.5 MiB/s wr, 452 op/s 2024-04-26T02:07:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:05.796 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:05 smithi006 ceph-mon[28419]: osdmap e131: 12 total, 12 up, 12 in 2024-04-26T02:07:05.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:05 smithi188 ceph-mon[32751]: osdmap e131: 12 total, 12 up, 12 in 2024-04-26T02:07:05.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:05 smithi029 ceph-mon[32775]: osdmap e131: 12 total, 12 up, 12 in 2024-04-26T02:07:06.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:06 smithi006 ceph-mon[28419]: pgmap v673: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 520 op/s 2024-04-26T02:07:06.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:06 smithi188 ceph-mon[32751]: pgmap v673: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 520 op/s 2024-04-26T02:07:06.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:06 smithi029 ceph-mon[32775]: pgmap v673: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 520 op/s 2024-04-26T02:07:07.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.760+0000 7f0fbcc00700 1 -- 172.21.15.6:0/798060836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb8102970 msgr2=0x7f0fb8102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:07.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.760+0000 7f0fbcc00700 1 --2- 172.21.15.6:0/798060836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb8102970 0x7f0fb8102df0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009990 tx=0x7f0fa4009ca0 comp rx=0 tx=0).stop 2024-04-26T02:07:07.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.761+0000 7f0fbcc00700 1 -- 172.21.15.6:0/798060836 shutdown_connections 2024-04-26T02:07:07.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.761+0000 7f0fbcc00700 1 --2- 172.21.15.6:0/798060836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb8103330 0x7f0fb810f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:07.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.761+0000 7f0fbcc00700 1 --2- 172.21.15.6:0/798060836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fb8108970 0x7f0fb8108d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:07.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.761+0000 7f0fbcc00700 1 --2- 172.21.15.6:0/798060836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb8102970 0x7f0fb8102df0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:07.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.761+0000 7f0fbcc00700 1 -- 172.21.15.6:0/798060836 >> 172.21.15.6:0/798060836 conn(0x7f0fb80fe4c0 msgr2=0x7f0fb81008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:07.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.761+0000 7f0fbcc00700 1 -- 172.21.15.6:0/798060836 shutdown_connections 2024-04-26T02:07:07.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.761+0000 7f0fbcc00700 1 -- 172.21.15.6:0/798060836 wait complete. 2024-04-26T02:07:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.762+0000 7f0fbcc00700 1 Processor -- start 2024-04-26T02:07:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.763+0000 7f0fbcc00700 1 -- start start 2024-04-26T02:07:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.763+0000 7f0fbcc00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb8102970 0x7f0fb819b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.763+0000 7f0fbcc00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb8103330 0x7f0fb819b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:07.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.763+0000 7f0fbcc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fb8108970 0x7f0fb819fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.763+0000 7f0fbcc00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb8112c00 con 0x7f0fb8102970 2024-04-26T02:07:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.763+0000 7f0fbcc00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb8112a80 con 0x7f0fb8103330 2024-04-26T02:07:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.763+0000 7f0fbcc00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb8112d80 con 0x7f0fb8108970 2024-04-26T02:07:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.764+0000 7f0fb7400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb8102970 0x7f0fb819b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.764+0000 7f0fb7400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb8102970 0x7f0fb819b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44688/0 (socket says 172.21.15.6:44688) 2024-04-26T02:07:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.764+0000 7f0fb7400700 1 -- 172.21.15.6:0/3925128347 learned_addr learned my addr 172.21.15.6:0/3925128347 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.764+0000 7f0fb6a00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb8103330 0x7f0fb819b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:07.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.764+0000 7f0fb7e00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fb8108970 0x7f0fb819fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:07.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.764+0000 7f0fb7400700 1 -- 172.21.15.6:0/3925128347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fb8108970 msgr2=0x7f0fb819fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:07.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.764+0000 7f0fb7400700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fb8108970 0x7f0fb819fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:07.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.764+0000 7f0fb7400700 1 -- 172.21.15.6:0/3925128347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb8103330 msgr2=0x7f0fb819b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:07.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.764+0000 7f0fb7400700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb8103330 0x7f0fb819b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:07.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.764+0000 7f0fb7400700 1 -- 172.21.15.6:0/3925128347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb81a0240 con 0x7f0fb8102970 2024-04-26T02:07:07.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.764+0000 7f0fb7e00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fb8108970 0x7f0fb819fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:07:07.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.764+0000 7f0fb6a00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb8103330 0x7f0fb819b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:07.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.765+0000 7f0fb7400700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb8102970 0x7f0fb819b230 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800eb30 tx=0x7f0fa800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:07.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.765+0000 7f0fafe00700 1 -- 172.21.15.6:0/3925128347 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa800cc90 con 0x7f0fb8102970 2024-04-26T02:07:07.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.765+0000 7f0fafe00700 1 -- 172.21.15.6:0/3925128347 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa80044d0 con 0x7f0fb8102970 2024-04-26T02:07:07.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.765+0000 7f0fafe00700 1 -- 172.21.15.6:0/3925128347 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8010430 con 0x7f0fb8102970 2024-04-26T02:07:07.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.765+0000 7f0fbcc00700 1 -- 172.21.15.6:0/3925128347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb81a0530 con 0x7f0fb8102970 2024-04-26T02:07:07.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.765+0000 7f0fbcc00700 1 -- 172.21.15.6:0/3925128347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb81a7db0 con 0x7f0fb8102970 2024-04-26T02:07:07.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.767+0000 7f0fbcc00700 1 -- 172.21.15.6:0/3925128347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fb8069730 con 0x7f0fb8102970 2024-04-26T02:07:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.769+0000 7f0fafe00700 1 -- 172.21.15.6:0/3925128347 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f0fa8003ea0 con 0x7f0fb8102970 2024-04-26T02:07:07.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.771+0000 7f0fafe00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0f9c0a3e30 0x7f0f9c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.771+0000 7f0fafe00700 1 -- 172.21.15.6:0/3925128347 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 8644+0+0 (secure 0 0 0) 0x7f0fa80d0a50 con 0x7f0fb8102970 2024-04-26T02:07:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.771+0000 7f0fb6a00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0f9c0a3e30 0x7f0f9c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.772+0000 7f0fb6a00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0f9c0a3e30 0x7f0f9c0a62f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8076d60 tx=0x7f0fa400b260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:07.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:07.776+0000 7f0fafe00700 1 -- 172.21.15.6:0/3925128347 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0fa809d010 con 0x7f0fb8102970 2024-04-26T02:07:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.067+0000 7f0fbcc00700 1 -- 172.21.15.6:0/3925128347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0fb81a7fe0 con 0x7f0fb8102970 2024-04-26T02:07:08.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.074+0000 7f0fafe00700 1 -- 172.21.15.6:0/3925128347 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+11850 (secure 0 0 0) 0x7f0fa809c910 con 0x7f0fb8102970 2024-04-26T02:07:08.080 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:07:08.081 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":59,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T02:06:53.954620+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27562":{"gid":27562,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1973934809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1973934809},{"type":"v1","addr":"172.21.15.188:6835","nonce":1973934809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T02:07:08.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.082+0000 7f0fbcc00700 1 -- 172.21.15.6:0/3925128347 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0f9c0a3e30 msgr2=0x7f0f9c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:08.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.082+0000 7f0fbcc00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0f9c0a3e30 0x7f0f9c0a62f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8076d60 tx=0x7f0fa400b260 comp rx=0 tx=0).stop 2024-04-26T02:07:08.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.082+0000 7f0fbcc00700 1 -- 172.21.15.6:0/3925128347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb8102970 msgr2=0x7f0fb819b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:08.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.082+0000 7f0fbcc00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb8102970 0x7f0fb819b230 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800eb30 tx=0x7f0fa800ee40 comp rx=0 tx=0).stop 2024-04-26T02:07:08.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.083+0000 7f0fbcc00700 1 -- 172.21.15.6:0/3925128347 shutdown_connections 2024-04-26T02:07:08.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.083+0000 7f0fbcc00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f0f9c0a3e30 0x7f0f9c0a62f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.083+0000 7f0fbcc00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fb8108970 0x7f0fb819fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.083+0000 7f0fbcc00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb8102970 0x7f0fb819b230 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.083+0000 7f0fbcc00700 1 --2- 172.21.15.6:0/3925128347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fb8103330 0x7f0fb819b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.083+0000 7f0fbcc00700 1 -- 172.21.15.6:0/3925128347 >> 172.21.15.6:0/3925128347 conn(0x7f0fb80fe4c0 msgr2=0x7f0fb80fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:08.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.087+0000 7f0fbcc00700 1 -- 172.21.15.6:0/3925128347 shutdown_connections 2024-04-26T02:07:08.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:08.090+0000 7f0fbcc00700 1 -- 172.21.15.6:0/3925128347 wait complete. 2024-04-26T02:07:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 59 2024-04-26T02:07:08.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:08 smithi006 ceph-mon[28419]: pgmap v674: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.1 MiB/s wr, 475 op/s 2024-04-26T02:07:08.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:08 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/3925128347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:07:08.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:08 smithi188 ceph-mon[32751]: pgmap v674: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.1 MiB/s wr, 475 op/s 2024-04-26T02:07:08.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:08 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/3925128347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:07:08.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:08 smithi029 ceph-mon[32775]: pgmap v674: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.1 MiB/s wr, 475 op/s 2024-04-26T02:07:08.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:08 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/3925128347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:07:09.000 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24523 quiesce db --cancel --all' 2024-04-26T02:07:09.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24523 quiesce db --cancel --all 2024-04-26T02:07:09.479 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:10.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:10 smithi188 ceph-mon[32751]: pgmap v675: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.1 MiB/s wr, 475 op/s 2024-04-26T02:07:10.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:10 smithi029 ceph-mon[32775]: pgmap v675: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.1 MiB/s wr, 475 op/s 2024-04-26T02:07:11.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:10 smithi006 ceph-mon[28419]: pgmap v675: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 8.1 MiB/s wr, 475 op/s 2024-04-26T02:07:12.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.145+0000 7f9646a00700 1 -- 172.21.15.6:0/2535180951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9640077890 msgr2=0x7f96401132b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.145+0000 7f9646a00700 1 --2- 172.21.15.6:0/2535180951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9640077890 0x7f96401132b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f962c009990 tx=0x7f962c009ca0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.145+0000 7f9646a00700 1 -- 172.21.15.6:0/2535180951 shutdown_connections 2024-04-26T02:07:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.145+0000 7f9646a00700 1 --2- 172.21.15.6:0/2535180951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96401137f0 0x7f9640115be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.145+0000 7f9646a00700 1 --2- 172.21.15.6:0/2535180951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9640076f70 0x7f9640077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.145+0000 7f9646a00700 1 --2- 172.21.15.6:0/2535180951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9640077890 0x7f96401132b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.145+0000 7f9646a00700 1 -- 172.21.15.6:0/2535180951 >> 172.21.15.6:0/2535180951 conn(0x7f9640100680 msgr2=0x7f9640102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.146+0000 7f9646a00700 1 -- 172.21.15.6:0/2535180951 shutdown_connections 2024-04-26T02:07:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.146+0000 7f9646a00700 1 -- 172.21.15.6:0/2535180951 wait complete. 2024-04-26T02:07:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.147+0000 7f9646a00700 1 Processor -- start 2024-04-26T02:07:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.147+0000 7f9646a00700 1 -- start start 2024-04-26T02:07:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.148+0000 7f9646a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9640076f70 0x7f964019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.148+0000 7f9646a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9640077890 0x7f964019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.148+0000 7f9646a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96401137f0 0x7f96401a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.148+0000 7f9646a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f96401191c0 con 0x7f96401137f0 2024-04-26T02:07:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.148+0000 7f9646a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9640119040 con 0x7f9640076f70 2024-04-26T02:07:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.148+0000 7f9646a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9640119340 con 0x7f9640077890 2024-04-26T02:07:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.148+0000 7f9646000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96401137f0 0x7f96401a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.148+0000 7f9645600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9640076f70 0x7f964019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.148+0000 7f9645600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9640076f70 0x7f964019f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:49294/0 (socket says 172.21.15.6:49294) 2024-04-26T02:07:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.148+0000 7f9645600700 1 -- 172.21.15.6:0/3751680827 learned_addr learned my addr 172.21.15.6:0/3751680827 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.148+0000 7f9644c00700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9640077890 0x7f964019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.149+0000 7f9645600700 1 -- 172.21.15.6:0/3751680827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9640077890 msgr2=0x7f964019fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.149+0000 7f9645600700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9640077890 0x7f964019fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.149+0000 7f9645600700 1 -- 172.21.15.6:0/3751680827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96401137f0 msgr2=0x7f96401a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.149+0000 7f9645600700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96401137f0 0x7f96401a3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.149+0000 7f9645600700 1 -- 172.21.15.6:0/3751680827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96401a46b0 con 0x7f9640076f70 2024-04-26T02:07:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.149+0000 7f9645600700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9640076f70 0x7f964019f6f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f963400c370 tx=0x7f963400c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.149+0000 7f963e000700 1 -- 172.21.15.6:0/3751680827 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963400cea0 con 0x7f9640076f70 2024-04-26T02:07:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.149+0000 7f963e000700 1 -- 172.21.15.6:0/3751680827 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96340090d0 con 0x7f9640076f70 2024-04-26T02:07:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.149+0000 7f963e000700 1 -- 172.21.15.6:0/3751680827 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9634004660 con 0x7f9640076f70 2024-04-26T02:07:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.149+0000 7f9646a00700 1 -- 172.21.15.6:0/3751680827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96401a4940 con 0x7f9640076f70 2024-04-26T02:07:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.150+0000 7f9646a00700 1 -- 172.21.15.6:0/3751680827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96401ac220 con 0x7f9640076f70 2024-04-26T02:07:12.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.154+0000 7f963e000700 1 -- 172.21.15.6:0/3751680827 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f963401a020 con 0x7f9640076f70 2024-04-26T02:07:12.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.155+0000 7f96529f4b80 1 Processor -- start 2024-04-26T02:07:12.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.156+0000 7f96529f4b80 1 -- start start 2024-04-26T02:07:12.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.156+0000 7f963e000700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f96280a3db0 0x7f96280a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.156+0000 7f96529f4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb7a60 0x564a6cdb7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.156+0000 7f96529f4b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdb8380 0x564a6cdc2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.156+0000 7f96529f4b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdc3230 0x564a6cdc5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.156+0000 7f96529f4b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564a6cd0af00 con 0x564a6cdc3230 2024-04-26T02:07:12.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.156+0000 7f96529f4b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564a6ccb17a0 con 0x564a6cdb8380 2024-04-26T02:07:12.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.156+0000 7f96529f4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564a6cd06ec0 con 0x564a6cdb7a60 2024-04-26T02:07:12.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.156+0000 7f963e000700 1 -- 172.21.15.6:0/3751680827 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 8644+0+0 (secure 0 0 0) 0x7f96340d0220 con 0x7f9640076f70 2024-04-26T02:07:12.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.157+0000 7f9645600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb7a60 0x564a6cdb7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.157+0000 7f9646000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdc3230 0x564a6cdc5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.157+0000 7f9646000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdc3230 0x564a6cdc5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47536/0 (socket says 172.21.15.6:47536) 2024-04-26T02:07:12.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.157+0000 7f9646000700 1 -- 172.21.15.6:0/4210162293 learned_addr learned my addr 172.21.15.6:0/4210162293 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:12.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.157+0000 7f9645600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb7a60 0x564a6cdb7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47110/0 (socket says 172.21.15.6:47110) 2024-04-26T02:07:12.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.157+0000 7f9646000700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f96280a3db0 0x7f96280a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.157+0000 7f9644c00700 1 --2- 172.21.15.6:0/4210162293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdb8380 0x564a6cdc2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.157+0000 7f9645600700 1 -- 172.21.15.6:0/4210162293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdb8380 msgr2=0x564a6cdc2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.157+0000 7f9645600700 1 --2- 172.21.15.6:0/4210162293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdb8380 0x564a6cdc2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.157+0000 7f9645600700 1 -- 172.21.15.6:0/4210162293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdc3230 msgr2=0x564a6cdc5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.157+0000 7f9645600700 1 --2- 172.21.15.6:0/4210162293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdc3230 0x564a6cdc5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.157+0000 7f9645600700 1 -- 172.21.15.6:0/4210162293 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a6cca76f0 con 0x564a6cdb7a60 2024-04-26T02:07:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.158+0000 7f9645600700 1 --2- 172.21.15.6:0/4210162293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb7a60 0x564a6cdb7e40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f96340cd890 tx=0x7f963409ba40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ef1e294b8df260f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.158+0000 7f9632a00700 1 -- 172.21.15.6:0/4210162293 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9634018e70 con 0x564a6cdb7a60 2024-04-26T02:07:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.158+0000 7f9646000700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f96280a3db0 0x7f96280a6270 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9638013a90 tx=0x7f9638009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:12.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.158+0000 7f9632a00700 1 -- 172.21.15.6:0/4210162293 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f963409c7e0 con 0x564a6cdb7a60 2024-04-26T02:07:12.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.158+0000 7f9632a00700 1 -- 172.21.15.6:0/4210162293 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963409a3e0 con 0x564a6cdb7a60 2024-04-26T02:07:12.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.159+0000 7f96529f4b80 1 -- 172.21.15.6:0/4210162293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb7a60 msgr2=0x564a6cdb7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:12.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.159+0000 7f96529f4b80 1 --2- 172.21.15.6:0/4210162293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb7a60 0x564a6cdb7e40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f96340cd890 tx=0x7f963409ba40 comp rx=0 tx=0).stop 2024-04-26T02:07:12.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.159+0000 7f96529f4b80 1 -- 172.21.15.6:0/4210162293 shutdown_connections 2024-04-26T02:07:12.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.159+0000 7f96529f4b80 1 --2- 172.21.15.6:0/4210162293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdc3230 0x564a6cdc5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.159+0000 7f96529f4b80 1 --2- 172.21.15.6:0/4210162293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb7a60 0x564a6cdb7e40 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.159+0000 7f96529f4b80 1 --2- 172.21.15.6:0/4210162293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdb8380 0x564a6cdc2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.159+0000 7f96529f4b80 1 -- 172.21.15.6:0/4210162293 >> 172.21.15.6:0/4210162293 conn(0x564a6cce6610 msgr2=0x564a6cdb3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.159+0000 7f96529f4b80 1 -- 172.21.15.6:0/4210162293 shutdown_connections 2024-04-26T02:07:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.159+0000 7f96529f4b80 1 -- 172.21.15.6:0/4210162293 wait complete. 2024-04-26T02:07:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.160+0000 7f96529f4b80 1 -- start start 2024-04-26T02:07:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.160+0000 7f96529f4b80 1 Processor -- start 2024-04-26T02:07:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.160+0000 7f96529f4b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdb7a60 0x564a6cdc2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.160+0000 7f96529f4b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb8380 0x564a6cdbfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.160+0000 7f96529f4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdc3230 0x564a6cdc02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f96529f4b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564a6cdc5d50 con 0x564a6cdb8380 2024-04-26T02:07:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f96529f4b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564a6ccc48d0 con 0x564a6cdb7a60 2024-04-26T02:07:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f96529f4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564a6cdc5ed0 con 0x564a6cdc3230 2024-04-26T02:07:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f96529f4b80 10 client.0 _ll_register_callbacks cb 0x564a6cca3790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:07:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f9644c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb8380 0x564a6cdbfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f9644c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb8380 0x564a6cdbfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47538/0 (socket says 172.21.15.6:47538) 2024-04-26T02:07:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f9644c00700 1 -- 172.21.15.6:0/3878268994 learned_addr learned my addr 172.21.15.6:0/3878268994 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f9646000700 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdc3230 0x564a6cdc02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f9645600700 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdb7a60 0x564a6cdc2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f9644c00700 1 -- 172.21.15.6:0/3878268994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdc3230 msgr2=0x564a6cdc02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f9644c00700 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdc3230 0x564a6cdc02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f9644c00700 1 -- 172.21.15.6:0/3878268994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdb7a60 msgr2=0x564a6cdc2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f9644c00700 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdb7a60 0x564a6cdc2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f9644c00700 1 -- 172.21.15.6:0/3878268994 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a6cc58940 con 0x564a6cdb8380 2024-04-26T02:07:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.161+0000 7f9645600700 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdb7a60 0x564a6cdc2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.162+0000 7f9644c00700 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb8380 0x564a6cdbfd70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f962c009990 tx=0x7f962c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5af4d1bd95ad39 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.162+0000 7f9627400700 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-26T02:07:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.162+0000 7f9627400700 1 -- 172.21.15.6:0/3878268994 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c029060 con 0x564a6cdb8380 2024-04-26T02:07:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.162+0000 7f96529f4b80 1 -- 172.21.15.6:0/3878268994 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f963409c7e0 con 0x564a6cdb8380 2024-04-26T02:07:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.162+0000 7f9627400700 1 -- 172.21.15.6:0/3878268994 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 8644+0+0 (secure 0 0 0) 0x7f962c00bd00 con 0x564a6cdb8380 2024-04-26T02:07:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.163+0000 7f9627400700 1 -- 172.21.15.6:0/3878268994 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f962c003680 con 0x564a6cdb8380 2024-04-26T02:07:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.163+0000 7f96529f4b80 10 client.17337 fetch_fsmap learned FSMap version 59 2024-04-26T02:07:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.163+0000 7f96529f4b80 1 -- 172.21.15.6:0/3878268994 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x564a6cca1b60 con 0x564a6cdb8380 2024-04-26T02:07:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.163+0000 7f9627400700 1 -- 172.21.15.6:0/3878268994 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f963409c7e0 con 0x564a6cdb8380 2024-04-26T02:07:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.163+0000 7f9627400700 1 -- 172.21.15.6:0/3878268994 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 59) ==== 8338+0+0 (secure 0 0 0) 0x7f962c0247b0 con 0x564a6cdb8380 2024-04-26T02:07:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.163+0000 7f96529f4b80 10 client.17337 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T02:07:12.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.163+0000 7f96529f4b80 10 client.17337 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:07:12.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.163+0000 7f96529f4b80 20 client.17337 populate_metadata read hostname 'smithi006' 2024-04-26T02:07:12.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.163+0000 7f96529f4b80 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a6cdd2ab0 0x564a6cdd4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.163+0000 7f96529f4b80 4 client.17337 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:07:12.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.163+0000 7f96529f4b80 1 -- 172.21.15.6:0/3878268994 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a6cd186f0 con 0x564a6cdd2ab0 2024-04-26T02:07:12.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.164+0000 7f9645600700 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a6cdd2ab0 0x564a6cdd4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.164+0000 7f9645600700 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a6cdd2ab0 0x564a6cdd4ea0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50419b25455c6243 server_cookie=29fa51f73a4dc4c1 in_seq=0 out_seq=0 2024-04-26T02:07:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.164+0000 7f9627400700 10 client.17337 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:07:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.165+0000 7f9627400700 1 -- 172.21.15.6:0/3878268994 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a6cd186f0 con 0x564a6cdd2ab0 2024-04-26T02:07:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.165+0000 7f9627400700 10 client.17337 handle_command_reply: tid=0 2024-04-26T02:07:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.165+0000 7f96529f4b80 1 client.17337 shutdown 2024-04-26T02:07:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.166+0000 7f96529f4b80 1 -- 172.21.15.6:0/3878268994 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb8380 msgr2=0x564a6cdbfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.166+0000 7f96529f4b80 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb8380 0x564a6cdbfd70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f962c009990 tx=0x7f962c005800 comp rx=0 tx=0).stop 2024-04-26T02:07:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.166+0000 7f96529f4b80 1 -- 172.21.15.6:0/3878268994 shutdown_connections 2024-04-26T02:07:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.166+0000 7f96529f4b80 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a6cdd2ab0 0x564a6cdd4ea0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.166+0000 7f96529f4b80 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdc3230 0x564a6cdc02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.166+0000 7f96529f4b80 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdb7a60 0x564a6cdc2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.166+0000 7f96529f4b80 1 --2- 172.21.15.6:0/3878268994 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb8380 0x564a6cdbfd70 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.166+0000 7f9627400700 0 client.17337 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:07:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.166+0000 7f96529f4b80 1 -- 172.21.15.6:0/3878268994 >> 172.21.15.6:0/3878268994 conn(0x564a6cce6610 msgr2=0x564a6cdb9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.166+0000 7f96529f4b80 1 -- 172.21.15.6:0/3878268994 shutdown_connections 2024-04-26T02:07:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.166+0000 7f96529f4b80 1 -- 172.21.15.6:0/3878268994 wait complete. 2024-04-26T02:07:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.166+0000 7f96529f4b80 20 client.17337 trim_cache size 0 max 16384 2024-04-26T02:07:12.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.188+0000 7f96529f4b80 1 Processor -- start 2024-04-26T02:07:12.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.188+0000 7f96529f4b80 1 -- start start 2024-04-26T02:07:12.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.188+0000 7f96529f4b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb7a60 0x564a6cdcec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.189+0000 7f96529f4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 0x564a6cdda7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.189+0000 7f96529f4b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdc3230 0x564a6cddacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.189+0000 7f96529f4b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f96401b6170 con 0x564a6cdb7a60 2024-04-26T02:07:12.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.189+0000 7f96529f4b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f96401b3a00 con 0x564a6cdc3230 2024-04-26T02:07:12.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.189+0000 7f96529f4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f96401b1870 con 0x564a6cdb8380 2024-04-26T02:07:12.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.189+0000 7f9644c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 0x564a6cdda7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.189+0000 7f9644c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 0x564a6cdda7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47130/0 (socket says 172.21.15.6:47130) 2024-04-26T02:07:12.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.189+0000 7f9644c00700 1 -- 172.21.15.6:0/3180724212 learned_addr learned my addr 172.21.15.6:0/3180724212 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:12.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.189+0000 7f9646000700 1 --2- 172.21.15.6:0/3180724212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdc3230 0x564a6cddacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.189+0000 7f9645600700 1 --2- 172.21.15.6:0/3180724212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb7a60 0x564a6cdcec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.190+0000 7f9644c00700 1 -- 172.21.15.6:0/3180724212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdc3230 msgr2=0x564a6cddacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.190+0000 7f9644c00700 1 --2- 172.21.15.6:0/3180724212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdc3230 0x564a6cddacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.190+0000 7f9644c00700 1 -- 172.21.15.6:0/3180724212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb7a60 msgr2=0x564a6cdcec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.190+0000 7f9644c00700 1 --2- 172.21.15.6:0/3180724212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb7a60 0x564a6cdcec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.190+0000 7f9644c00700 1 -- 172.21.15.6:0/3180724212 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a6cdd5520 con 0x564a6cdb8380 2024-04-26T02:07:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.190+0000 7f9645600700 1 --2- 172.21.15.6:0/3180724212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb7a60 0x564a6cdcec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:07:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.190+0000 7f9644c00700 1 --2- 172.21.15.6:0/3180724212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 0x564a6cdda7b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f962c0098e0 tx=0x7f962c0092f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cd81069c6dd1702 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.191+0000 7f9632000700 1 -- 172.21.15.6:0/3180724212 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c03a070 con 0x564a6cdb8380 2024-04-26T02:07:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.191+0000 7f9632000700 1 -- 172.21.15.6:0/3180724212 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f962c024480 con 0x564a6cdb8380 2024-04-26T02:07:12.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.191+0000 7f9632000700 1 -- 172.21.15.6:0/3180724212 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c029040 con 0x564a6cdb8380 2024-04-26T02:07:12.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.191+0000 7f96529f4b80 1 -- 172.21.15.6:0/3180724212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 msgr2=0x564a6cdda7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:12.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.191+0000 7f96529f4b80 1 --2- 172.21.15.6:0/3180724212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 0x564a6cdda7b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f962c0098e0 tx=0x7f962c0092f0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.191+0000 7f96529f4b80 1 -- 172.21.15.6:0/3180724212 shutdown_connections 2024-04-26T02:07:12.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.191+0000 7f96529f4b80 1 --2- 172.21.15.6:0/3180724212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdc3230 0x564a6cddacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.191+0000 7f96529f4b80 1 --2- 172.21.15.6:0/3180724212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb7a60 0x564a6cdcec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.191+0000 7f96529f4b80 1 --2- 172.21.15.6:0/3180724212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 0x564a6cdda7b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.191+0000 7f96529f4b80 1 -- 172.21.15.6:0/3180724212 >> 172.21.15.6:0/3180724212 conn(0x564a6cce6610 msgr2=0x564a6cdbb410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:12.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.192+0000 7f96529f4b80 1 -- 172.21.15.6:0/3180724212 shutdown_connections 2024-04-26T02:07:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.192+0000 7f96529f4b80 1 -- 172.21.15.6:0/3180724212 wait complete. 2024-04-26T02:07:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.192+0000 7f96529f4b80 1 -- start start 2024-04-26T02:07:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.192+0000 7f96529f4b80 1 Processor -- start 2024-04-26T02:07:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.193+0000 7f96529f4b80 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb7a60 0x564a6cdc7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.193+0000 7f96529f4b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 0x564a6cdc83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.193+0000 7f96529f4b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdc3230 0x564a6ce39030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.193+0000 7f96529f4b80 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f96401b1ec0 con 0x564a6cdb7a60 2024-04-26T02:07:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.193+0000 7f96529f4b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f96401b3330 con 0x564a6cdc3230 2024-04-26T02:07:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.193+0000 7f96529f4b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f96401b2590 con 0x564a6cdb8380 2024-04-26T02:07:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.193+0000 7f96529f4b80 10 client.0 _ll_register_callbacks cb 0x564a6cca3790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:07:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.193+0000 7f9646000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdc3230 0x564a6ce39030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.193+0000 7f9646000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdc3230 0x564a6ce39030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:49334/0 (socket says 172.21.15.6:49334) 2024-04-26T02:07:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.193+0000 7f9646000700 1 -- 172.21.15.6:0/1621322679 learned_addr learned my addr 172.21.15.6:0/1621322679 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.193+0000 7f9644c00700 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 0x564a6cdc83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.193+0000 7f9645600700 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb7a60 0x564a6cdc7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.194+0000 7f9646000700 1 -- 172.21.15.6:0/1621322679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 msgr2=0x564a6cdc83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:12.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.194+0000 7f9646000700 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 0x564a6cdc83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.194+0000 7f9646000700 1 -- 172.21.15.6:0/1621322679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb7a60 msgr2=0x564a6cdc7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:12.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.194+0000 7f9646000700 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb7a60 0x564a6cdc7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.194+0000 7f9646000700 1 -- 172.21.15.6:0/1621322679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a6ce39570 con 0x564a6cdc3230 2024-04-26T02:07:12.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.194+0000 7f9644c00700 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 0x564a6cdc83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:12.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.194+0000 7f9646000700 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdc3230 0x564a6ce39030 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f963800c5b0 tx=0x7f96380047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=681119e3d5329940 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:12.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.194+0000 7f9627400700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-26T02:07:12.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.195+0000 7f9627400700 1 -- 172.21.15.6:0/1621322679 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9638003d20 con 0x564a6cdc3230 2024-04-26T02:07:12.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.195+0000 7f9627400700 1 -- 172.21.15.6:0/1621322679 <== mon.2 v2:172.21.15.29:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 8644+0+0 (secure 0 0 0) 0x7f9638009040 con 0x564a6cdc3230 2024-04-26T02:07:12.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.195+0000 7f96529f4b80 1 -- 172.21.15.6:0/1621322679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f962c024480 con 0x564a6cdc3230 2024-04-26T02:07:12.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.195+0000 7f9627400700 1 -- 172.21.15.6:0/1621322679 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f9638004830 con 0x564a6cdc3230 2024-04-26T02:07:12.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.195+0000 7f96529f4b80 10 client.27554 fetch_fsmap learned FSMap version 59 2024-04-26T02:07:12.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.195+0000 7f96529f4b80 1 -- 172.21.15.6:0/1621322679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x564a6ce3a6d0 con 0x564a6cdc3230 2024-04-26T02:07:12.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.195+0000 7f9627400700 1 -- 172.21.15.6:0/1621322679 <== mon.2 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f962c024480 con 0x564a6cdc3230 2024-04-26T02:07:12.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.195+0000 7f9627400700 1 -- 172.21.15.6:0/1621322679 <== mon.2 v2:172.21.15.29:3300/0 5 ==== fsmap(e 59) ==== 8338+0+0 (secure 0 0 0) 0x7f963802c030 con 0x564a6cdc3230 2024-04-26T02:07:12.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.196+0000 7f96529f4b80 10 client.27554 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T02:07:12.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.196+0000 7f96529f4b80 10 client.27554 resolve_mds: validated gid 24523 aka daemon mds.i 2024-04-26T02:07:12.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.196+0000 7f96529f4b80 20 client.27554 populate_metadata read hostname 'smithi006' 2024-04-26T02:07:12.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.196+0000 7f96529f4b80 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a6ce3ade0 0x564a6ce3d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:12.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.196+0000 7f96529f4b80 4 client.27554 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-26T02:07:12.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.196+0000 7f96529f4b80 1 -- 172.21.15.6:0/1621322679 --> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x564a6cd12790 con 0x564a6ce3ade0 2024-04-26T02:07:12.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.196+0000 7f9645600700 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a6ce3ade0 0x564a6ce3d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:12.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.196+0000 7f9645600700 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a6ce3ade0 0x564a6ce3d200 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17e0800c486bc342 server_cookie=3c4215413bfb6a8b in_seq=0 out_seq=0 2024-04-26T02:07:12.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.196+0000 7f9627400700 10 client.27554 ms_handle_connect on v2:172.21.15.188:6838/674623421 2024-04-26T02:07:12.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f9627400700 1 -- 172.21.15.6:0/1621322679 <== mds.0 v2:172.21.15.188:6838/674623421 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x564a6cd12790 con 0x564a6ce3ade0 2024-04-26T02:07:12.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f9627400700 10 client.27554 handle_command_reply: tid=0 2024-04-26T02:07:12.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f96529f4b80 1 client.27554 shutdown 2024-04-26T02:07:12.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f96529f4b80 1 -- 172.21.15.6:0/1621322679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdc3230 msgr2=0x564a6ce39030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:12.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f96529f4b80 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdc3230 0x564a6ce39030 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f963800c5b0 tx=0x7f96380047c0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f96529f4b80 1 -- 172.21.15.6:0/1621322679 shutdown_connections 2024-04-26T02:07:12.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f96529f4b80 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a6cdc3230 0x564a6ce39030 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f96529f4b80 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564a6cdb7a60 0x564a6cdc7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f96529f4b80 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.188:6838/674623421,v1:172.21.15.188:6839/674623421] conn(0x564a6ce3ade0 0x564a6ce3d200 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f96529f4b80 1 --2- 172.21.15.6:0/1621322679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a6cdb8380 0x564a6cdc83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f96529f4b80 1 -- 172.21.15.6:0/1621322679 >> 172.21.15.6:0/1621322679 conn(0x564a6cce6610 msgr2=0x564a6cdb99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:12.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f9627400700 0 client.27554 ms_handle_reset on v2:172.21.15.188:6838/674623421 2024-04-26T02:07:12.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f96529f4b80 1 -- 172.21.15.6:0/1621322679 shutdown_connections 2024-04-26T02:07:12.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.197+0000 7f96529f4b80 1 -- 172.21.15.6:0/1621322679 wait complete. 2024-04-26T02:07:12.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.198+0000 7f96529f4b80 20 client.27554 trim_cache size 0 max 16384 2024-04-26T02:07:12.203 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:07:12.204 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 59, 2024-04-26T02:07:12.204 INFO:teuthology.orchestra.run.smithi006.stdout: "leader": 24523, 2024-04-26T02:07:12.204 INFO:teuthology.orchestra.run.smithi006.stdout: "set_version": 54, 2024-04-26T02:07:12.204 INFO:teuthology.orchestra.run.smithi006.stdout: "sets": {} 2024-04-26T02:07:12.204 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:07:12.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.198+0000 7f9646a00700 1 -- 172.21.15.6:0/3751680827 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f96280a3db0 msgr2=0x7f96280a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:12.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.198+0000 7f9646a00700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f96280a3db0 0x7f96280a6270 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9638013a90 tx=0x7f9638009260 comp rx=0 tx=0).stop 2024-04-26T02:07:12.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.198+0000 7f9646a00700 1 -- 172.21.15.6:0/3751680827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9640076f70 msgr2=0x7f964019f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:12.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.198+0000 7f9646a00700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9640076f70 0x7f964019f6f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f963400c370 tx=0x7f963400c680 comp rx=0 tx=0).stop 2024-04-26T02:07:12.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.198+0000 7f9646a00700 1 -- 172.21.15.6:0/3751680827 shutdown_connections 2024-04-26T02:07:12.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.198+0000 7f9646a00700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f96280a3db0 0x7f96280a6270 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.198+0000 7f9646a00700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96401137f0 0x7f96401a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.198+0000 7f9646a00700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9640076f70 0x7f964019f6f0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.198+0000 7f9646a00700 1 --2- 172.21.15.6:0/3751680827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9640077890 0x7f964019fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:12.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.198+0000 7f9646a00700 1 -- 172.21.15.6:0/3751680827 >> 172.21.15.6:0/3751680827 conn(0x7f9640100680 msgr2=0x7f9640102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:12.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.199+0000 7f9646a00700 1 -- 172.21.15.6:0/3751680827 shutdown_connections 2024-04-26T02:07:12.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:12.199+0000 7f9646a00700 1 -- 172.21.15.6:0/3751680827 wait complete. 2024-04-26T02:07:12.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:12 smithi188 ceph-mon[32751]: pgmap v676: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 MiB/s wr, 331 op/s 2024-04-26T02:07:12.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:12 smithi029 ceph-mon[32775]: pgmap v676: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 MiB/s wr, 331 op/s 2024-04-26T02:07:13.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:12 smithi006 ceph-mon[28419]: pgmap v676: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 MiB/s wr, 331 op/s 2024-04-26T02:07:13.192 ERROR:teuthology.run_tasks:Manager failed: quiescer Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/quiescer.py", line 419, in task stop_all_quiescers(ctx.ceph[cluster_name].thrashers) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/quiescer.py", line 378, in stop_all_quiescers raise RuntimeError(f"error during quiesce thrashing: {thrasher.exception}") RuntimeError: error during quiesce thrashing: Error quiescing set 'ef592526': 110 (ETIMEDOUT) 2024-04-26T02:07:13.197 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-26T02:07:13.207 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi006.front.sepia.ceph.com 2024-04-26T02:07:13.240 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-26T02:07:13.250 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-26T02:07:13.250 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T02:07:13.251 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi006.front.sepia.ceph.com 2024-04-26T02:07:13.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-26T02:07:13.400 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-26T02:07:13.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-26T02:07:13.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.647+0000 7fde03200640 1 -- 172.21.15.6:0/1648393122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde0410e470 msgr2=0x7fde0410e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:13.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.647+0000 7fde03200640 1 --2- 172.21.15.6:0/1648393122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde0410e470 0x7fde0410e850 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fddf4009900 tx=0x7fddf402a480 comp rx=0 tx=0).stop 2024-04-26T02:07:13.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.648+0000 7fde03200640 1 -- 172.21.15.6:0/1648393122 shutdown_connections 2024-04-26T02:07:13.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.648+0000 7fde03200640 1 --2- 172.21.15.6:0/1648393122 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde04108d50 0x7fde04115210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:13.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.648+0000 7fde03200640 1 --2- 172.21.15.6:0/1648393122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde041083b0 0x7fde04108810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:13.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.648+0000 7fde03200640 1 --2- 172.21.15.6:0/1648393122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde0410e470 0x7fde0410e850 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:13.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.648+0000 7fde03200640 1 -- 172.21.15.6:0/1648393122 >> 172.21.15.6:0/1648393122 conn(0x7fde04077b00 msgr2=0x7fde04079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:13.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.648+0000 7fde03200640 1 -- 172.21.15.6:0/1648393122 shutdown_connections 2024-04-26T02:07:13.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.648+0000 7fde03200640 1 -- 172.21.15.6:0/1648393122 wait complete. 2024-04-26T02:07:13.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.649+0000 7fde03200640 1 Processor -- start 2024-04-26T02:07:13.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.649+0000 7fde03200640 1 -- start start 2024-04-26T02:07:13.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.649+0000 7fde03200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde041083b0 0x7fde041a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:13.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.649+0000 7fde03200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde04108d50 0x7fde041a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:13.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.649+0000 7fde03200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde0410e470 0x7fde0419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:13.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.649+0000 7fde03200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fde04118630 con 0x7fde0410e470 2024-04-26T02:07:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.649+0000 7fde03200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fde041184b0 con 0x7fde041083b0 2024-04-26T02:07:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.649+0000 7fde03200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fde041187b0 con 0x7fde04108d50 2024-04-26T02:07:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.650+0000 7fddf9400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde04108d50 0x7fde041a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.650+0000 7fde01e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde041083b0 0x7fde041a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.650+0000 7fde02800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde0410e470 0x7fde0419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.650+0000 7fddf9400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde04108d50 0x7fde041a4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:47164/0 (socket says 172.21.15.6:47164) 2024-04-26T02:07:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.650+0000 7fde01e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde041083b0 0x7fde041a47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:49348/0 (socket says 172.21.15.6:49348) 2024-04-26T02:07:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.650+0000 7fddf9400640 1 -- 172.21.15.6:0/3310724156 learned_addr learned my addr 172.21.15.6:0/3310724156 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:13.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.650+0000 7fddf9400640 1 -- 172.21.15.6:0/3310724156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde041083b0 msgr2=0x7fde041a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:13.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.650+0000 7fddf9400640 1 --2- 172.21.15.6:0/3310724156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde041083b0 0x7fde041a47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:13.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.650+0000 7fddf9400640 1 -- 172.21.15.6:0/3310724156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde0410e470 msgr2=0x7fde0419e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:13.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.650+0000 7fddf9400640 1 --2- 172.21.15.6:0/3310724156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde0410e470 0x7fde0419e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:13.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.650+0000 7fddf9400640 1 -- 172.21.15.6:0/3310724156 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0419f150 con 0x7fde04108d50 2024-04-26T02:07:13.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.650+0000 7fddf9400640 1 --2- 172.21.15.6:0/3310724156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde04108d50 0x7fde041a4d10 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fddec00ebc0 tx=0x7fddec00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:13.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.651+0000 7fddfb400640 1 -- 172.21.15.6:0/3310724156 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddec010630 con 0x7fde04108d50 2024-04-26T02:07:13.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.651+0000 7fddfb400640 1 -- 172.21.15.6:0/3310724156 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddec010c80 con 0x7fde04108d50 2024-04-26T02:07:13.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.651+0000 7fde03200640 1 -- 172.21.15.6:0/3310724156 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde0419f440 con 0x7fde04108d50 2024-04-26T02:07:13.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.651+0000 7fddfb400640 1 -- 172.21.15.6:0/3310724156 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddec018b60 con 0x7fde04108d50 2024-04-26T02:07:13.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.651+0000 7fde03200640 1 -- 172.21.15.6:0/3310724156 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde041ab690 con 0x7fde04108d50 2024-04-26T02:07:13.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.652+0000 7fde03200640 1 -- 172.21.15.6:0/3310724156 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde040687f0 con 0x7fde04108d50 2024-04-26T02:07:13.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.654+0000 7fddfb400640 1 -- 172.21.15.6:0/3310724156 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fddec018d00 con 0x7fde04108d50 2024-04-26T02:07:13.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.655+0000 7fddfb400640 1 --2- 172.21.15.6:0/3310724156 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdde80a3d00 0x7fdde80a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:13.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.655+0000 7fddfb400640 1 -- 172.21.15.6:0/3310724156 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 8644+0+0 (secure 0 0 0) 0x7fddec0d0ec0 con 0x7fde04108d50 2024-04-26T02:07:13.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.656+0000 7fde01e00640 1 --2- 172.21.15.6:0/3310724156 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdde80a3d00 0x7fdde80a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:13.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.656+0000 7fde01e00640 1 --2- 172.21.15.6:0/3310724156 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdde80a3d00 0x7fdde80a61c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fddf4005ab0 tx=0x7fddf4005a20 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:13.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.660+0000 7fddfb400640 1 -- 172.21.15.6:0/3310724156 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fddec09ce30 con 0x7fde04108d50 2024-04-26T02:07:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.884+0000 7fde03200640 1 -- 172.21.15.6:0/3310724156 --> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fde04062390 con 0x7fdde80a3d00 2024-04-26T02:07:13.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.905+0000 7fddfb400640 1 -- 172.21.15.6:0/3310724156 <== mgr.25037 v2:172.21.15.6:6800/3353570381 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7fde04062390 con 0x7fdde80a3d00 2024-04-26T02:07:13.904 INFO:teuthology.orchestra.run.smithi006.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-26T00:00:00", "created": "2024-04-26T01:45:55", "first": "2024-04-26T01:46:00", "last": "2024-04-26T02:07:00", "last_pruned": "2024-04-26T02:07:02", "created_count": 22, "pruned_count": 15, "active": true} 2024-04-26T02:07:13.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.907+0000 7fde03200640 1 -- 172.21.15.6:0/3310724156 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdde80a3d00 msgr2=0x7fdde80a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.907+0000 7fde03200640 1 --2- 172.21.15.6:0/3310724156 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdde80a3d00 0x7fdde80a61c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fddf4005ab0 tx=0x7fddf4005a20 comp rx=0 tx=0).stop 2024-04-26T02:07:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.907+0000 7fde03200640 1 -- 172.21.15.6:0/3310724156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde04108d50 msgr2=0x7fde041a4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.907+0000 7fde03200640 1 --2- 172.21.15.6:0/3310724156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde04108d50 0x7fde041a4d10 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fddec00ebc0 tx=0x7fddec00c6a0 comp rx=0 tx=0).stop 2024-04-26T02:07:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.907+0000 7fde03200640 1 -- 172.21.15.6:0/3310724156 shutdown_connections 2024-04-26T02:07:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.907+0000 7fde03200640 1 --2- 172.21.15.6:0/3310724156 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fdde80a3d00 0x7fdde80a61c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.907+0000 7fde03200640 1 --2- 172.21.15.6:0/3310724156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde0410e470 0x7fde0419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.907+0000 7fde03200640 1 --2- 172.21.15.6:0/3310724156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde04108d50 0x7fde041a4d10 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.907+0000 7fde03200640 1 --2- 172.21.15.6:0/3310724156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde041083b0 0x7fde041a47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.907+0000 7fde03200640 1 -- 172.21.15.6:0/3310724156 >> 172.21.15.6:0/3310724156 conn(0x7fde04077b00 msgr2=0x7fde04079450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.907+0000 7fde03200640 1 -- 172.21.15.6:0/3310724156 shutdown_connections 2024-04-26T02:07:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:13.907+0000 7fde03200640 1 -- 172.21.15.6:0/3310724156 wait complete. 2024-04-26T02:07:13.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-26T02:07:14.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.136+0000 7fe8ac000640 1 -- 172.21.15.6:0/13783538 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8a4102b60 msgr2=0x7fe8a4106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:14.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.136+0000 7fe8ac000640 1 --2- 172.21.15.6:0/13783538 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8a4102b60 0x7fe8a4106ba0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe890009900 tx=0x7fe89002a410 comp rx=0 tx=0).stop 2024-04-26T02:07:14.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.136+0000 7fe8ac000640 1 -- 172.21.15.6:0/13783538 shutdown_connections 2024-04-26T02:07:14.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.136+0000 7fe8ac000640 1 --2- 172.21.15.6:0/13783538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a4107280 0x7fe8a410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.136+0000 7fe8ac000640 1 --2- 172.21.15.6:0/13783538 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8a4102b60 0x7fe8a4106ba0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.136+0000 7fe8ac000640 1 --2- 172.21.15.6:0/13783538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a41021b0 0x7fe8a4102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.136+0000 7fe8ac000640 1 -- 172.21.15.6:0/13783538 >> 172.21.15.6:0/13783538 conn(0x7fe8a40fda10 msgr2=0x7fe8a40ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:14.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.136+0000 7fe8ac000640 1 -- 172.21.15.6:0/13783538 shutdown_connections 2024-04-26T02:07:14.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.137+0000 7fe8ac000640 1 -- 172.21.15.6:0/13783538 wait complete. 2024-04-26T02:07:14.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.138+0000 7fe8ac000640 1 Processor -- start 2024-04-26T02:07:14.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.138+0000 7fe8ac000640 1 -- start start 2024-04-26T02:07:14.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8ac000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a41021b0 0x7fe8a419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:14.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8ac000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a4107280 0x7fe8a419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:14.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8ac000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8a4198590 0x7fe8a4198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:14.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8ac000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a41121f0 con 0x7fe8a41021b0 2024-04-26T02:07:14.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8ac000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a4112070 con 0x7fe8a4107280 2024-04-26T02:07:14.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8ac000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a4112370 con 0x7fe8a4198590 2024-04-26T02:07:14.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8a9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a41021b0 0x7fe8a419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8a9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a41021b0 0x7fe8a419e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47584/0 (socket says 172.21.15.6:47584) 2024-04-26T02:07:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8a9600640 1 -- 172.21.15.6:0/7053051 learned_addr learned my addr 172.21.15.6:0/7053051 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8aa000640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8a4198590 0x7fe8a4198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8a8c00640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a4107280 0x7fe8a419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8a9600640 1 -- 172.21.15.6:0/7053051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8a4198590 msgr2=0x7fe8a4198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8a9600640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8a4198590 0x7fe8a4198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8a9600640 1 -- 172.21.15.6:0/7053051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a4107280 msgr2=0x7fe8a419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8a9600640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a4107280 0x7fe8a419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.139+0000 7fe8a9600640 1 -- 172.21.15.6:0/7053051 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a41990b0 con 0x7fe8a41021b0 2024-04-26T02:07:14.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.140+0000 7fe8a9600640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a41021b0 0x7fe8a419e3e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe89800d950 tx=0x7fe89800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:14.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.140+0000 7fe8a2000640 1 -- 172.21.15.6:0/7053051 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe898017070 con 0x7fe8a41021b0 2024-04-26T02:07:14.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.140+0000 7fe8a2000640 1 -- 172.21.15.6:0/7053051 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe898009c30 con 0x7fe8a41021b0 2024-04-26T02:07:14.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.140+0000 7fe8ac000640 1 -- 172.21.15.6:0/7053051 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a41993a0 con 0x7fe8a41021b0 2024-04-26T02:07:14.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.140+0000 7fe8a2000640 1 -- 172.21.15.6:0/7053051 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89800f620 con 0x7fe8a41021b0 2024-04-26T02:07:14.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.140+0000 7fe8ac000640 1 -- 172.21.15.6:0/7053051 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a41a55f0 con 0x7fe8a41021b0 2024-04-26T02:07:14.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.141+0000 7fe8ac000640 1 -- 172.21.15.6:0/7053051 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a4050530 con 0x7fe8a41021b0 2024-04-26T02:07:14.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.143+0000 7fe8a2000640 1 -- 172.21.15.6:0/7053051 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fe898010460 con 0x7fe8a41021b0 2024-04-26T02:07:14.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.145+0000 7fe8a2000640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe88c0a3d10 0x7fe88c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:14.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.146+0000 7fe8a2000640 1 -- 172.21.15.6:0/7053051 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 8644+0+0 (secure 0 0 0) 0x7fe8980d0470 con 0x7fe8a41021b0 2024-04-26T02:07:14.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.146+0000 7fe8a8c00640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe88c0a3d10 0x7fe88c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:14.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.147+0000 7fe8a8c00640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe88c0a3d10 0x7fe88c0a61d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe890031440 tx=0x7fe890005a20 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.151+0000 7fe8a2000640 1 -- 172.21.15.6:0/7053051 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe898009de0 con 0x7fe8a41021b0 2024-04-26T02:07:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.375+0000 7fe8ac000640 1 -- 172.21.15.6:0/7053051 --> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fe8a4102b60 con 0x7fe88c0a3d10 2024-04-26T02:07:14.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.399+0000 7fe8a2000640 1 -- 172.21.15.6:0/7053051 <== mgr.25037 v2:172.21.15.6:6800/3353570381 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fe8a4102b60 con 0x7fe88c0a3d10 2024-04-26T02:07:14.399 INFO:teuthology.orchestra.run.smithi006.stdout:/ 1m 6m3h 2024-04-26T02:07:14.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.401+0000 7fe8ac000640 1 -- 172.21.15.6:0/7053051 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe88c0a3d10 msgr2=0x7fe88c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:14.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.402+0000 7fe8ac000640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe88c0a3d10 0x7fe88c0a61d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe890031440 tx=0x7fe890005a20 comp rx=0 tx=0).stop 2024-04-26T02:07:14.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.402+0000 7fe8ac000640 1 -- 172.21.15.6:0/7053051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a41021b0 msgr2=0x7fe8a419e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:14.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.402+0000 7fe8ac000640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a41021b0 0x7fe8a419e3e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe89800d950 tx=0x7fe89800de10 comp rx=0 tx=0).stop 2024-04-26T02:07:14.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.402+0000 7fe8ac000640 1 -- 172.21.15.6:0/7053051 shutdown_connections 2024-04-26T02:07:14.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.402+0000 7fe8ac000640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fe88c0a3d10 0x7fe88c0a61d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.402+0000 7fe8ac000640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8a4198590 0x7fe8a4198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.402+0000 7fe8ac000640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a4107280 0x7fe8a419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.402+0000 7fe8ac000640 1 --2- 172.21.15.6:0/7053051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a41021b0 0x7fe8a419e3e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.402+0000 7fe8ac000640 1 -- 172.21.15.6:0/7053051 >> 172.21.15.6:0/7053051 conn(0x7fe8a40fda10 msgr2=0x7fe8a40fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.402+0000 7fe8ac000640 1 -- 172.21.15.6:0/7053051 shutdown_connections 2024-04-26T02:07:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.402+0000 7fe8ac000640 1 -- 172.21.15.6:0/7053051 wait complete. 2024-04-26T02:07:14.423 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-26T02:07:14.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:07:14.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.630+0000 7f62dbe00640 1 -- 172.21.15.6:0/3831785516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc104290 msgr2=0x7f62dc104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:14.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.630+0000 7f62dbe00640 1 --2- 172.21.15.6:0/3831785516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc104290 0x7f62dc104670 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009900 tx=0x7f62cc02a3f0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.630+0000 7f62dbe00640 1 -- 172.21.15.6:0/3831785516 shutdown_connections 2024-04-26T02:07:14.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.630+0000 7f62dbe00640 1 --2- 172.21.15.6:0/3831785516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62dc104c40 0x7f62dc108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.630+0000 7f62dbe00640 1 --2- 172.21.15.6:0/3831785516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62dc109360 0x7f62dc110f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.630+0000 7f62dbe00640 1 --2- 172.21.15.6:0/3831785516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc104290 0x7f62dc104670 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.630+0000 7f62dbe00640 1 -- 172.21.15.6:0/3831785516 >> 172.21.15.6:0/3831785516 conn(0x7f62dc0ffad0 msgr2=0x7f62dc101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:14.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.631+0000 7f62dbe00640 1 -- 172.21.15.6:0/3831785516 shutdown_connections 2024-04-26T02:07:14.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.631+0000 7f62dbe00640 1 -- 172.21.15.6:0/3831785516 wait complete. 2024-04-26T02:07:14.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.631+0000 7f62dbe00640 1 Processor -- start 2024-04-26T02:07:14.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.632+0000 7f62dbe00640 1 -- start start 2024-04-26T02:07:14.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.632+0000 7f62dbe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc104290 0x7f62dc1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:14.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.632+0000 7f62dbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62dc109360 0x7f62dc1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:14.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.632+0000 7f62dbe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62dc19a760 0x7f62dc19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:14.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.632+0000 7f62dbe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc114320 con 0x7f62dc109360 2024-04-26T02:07:14.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.632+0000 7f62dbe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc1141a0 con 0x7f62dc104290 2024-04-26T02:07:14.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.632+0000 7f62dbe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc1144a0 con 0x7f62dc19a760 2024-04-26T02:07:14.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.632+0000 7f62daa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc104290 0x7f62dc1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:14.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.632+0000 7f62daa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc104290 0x7f62dc1a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:49378/0 (socket says 172.21.15.6:49378) 2024-04-26T02:07:14.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.632+0000 7f62daa00640 1 -- 172.21.15.6:0/140363436 learned_addr learned my addr 172.21.15.6:0/140363436 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:14.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.632+0000 7f62db400640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62dc19a760 0x7f62dc19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:14.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.632+0000 7f62da000640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62dc109360 0x7f62dc1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:14.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.633+0000 7f62daa00640 1 -- 172.21.15.6:0/140363436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62dc19a760 msgr2=0x7f62dc19abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:14.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.633+0000 7f62daa00640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62dc19a760 0x7f62dc19abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.633+0000 7f62daa00640 1 -- 172.21.15.6:0/140363436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62dc109360 msgr2=0x7f62dc1a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:14.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.633+0000 7f62daa00640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62dc109360 0x7f62dc1a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.633+0000 7f62daa00640 1 -- 172.21.15.6:0/140363436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62dc19b360 con 0x7f62dc104290 2024-04-26T02:07:14.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.633+0000 7f62db400640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62dc19a760 0x7f62dc19abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:14.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.633+0000 7f62daa00640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc104290 0x7f62dc1a0650 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f62cc005ab0 tx=0x7f62cc003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:14.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.633+0000 7f62d3400640 1 -- 172.21.15.6:0/140363436 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc03f070 con 0x7f62dc104290 2024-04-26T02:07:14.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.633+0000 7f62d3400640 1 -- 172.21.15.6:0/140363436 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62cc031b60 con 0x7f62dc104290 2024-04-26T02:07:14.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.633+0000 7f62d3400640 1 -- 172.21.15.6:0/140363436 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc035030 con 0x7f62dc104290 2024-04-26T02:07:14.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.634+0000 7f62dbe00640 1 -- 172.21.15.6:0/140363436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62dc19b5f0 con 0x7f62dc104290 2024-04-26T02:07:14.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.634+0000 7f62dbe00640 1 -- 172.21.15.6:0/140363436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62dc1a7690 con 0x7f62dc104290 2024-04-26T02:07:14.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.635+0000 7f62dbe00640 1 -- 172.21.15.6:0/140363436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62dc105760 con 0x7f62dc104290 2024-04-26T02:07:14.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.637+0000 7f62d3400640 1 -- 172.21.15.6:0/140363436 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f62cc042070 con 0x7f62dc104290 2024-04-26T02:07:14.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.638+0000 7f62d3400640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f62c00a3d90 0x7f62c00a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:14.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.638+0000 7f62d3400640 1 -- 172.21.15.6:0/140363436 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 8644+0+0 (secure 0 0 0) 0x7f62cc0f3120 con 0x7f62dc104290 2024-04-26T02:07:14.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.639+0000 7f62da000640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f62c00a3d90 0x7f62c00a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:14.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.639+0000 7f62da000640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f62c00a3d90 0x7f62c00a6250 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f62c80095f0 tx=0x7f62c8005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:14.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.643+0000 7f62d3400640 1 -- 172.21.15.6:0/140363436 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f62cc0bcdf0 con 0x7f62dc104290 2024-04-26T02:07:14.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:14 smithi006 ceph-mon[28419]: pgmap v677: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 307 op/s 2024-04-26T02:07:14.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.878+0000 7f62dbe00640 1 -- 172.21.15.6:0/140363436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62dc10c590 con 0x7f62dc104290 2024-04-26T02:07:14.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.881+0000 7f62d3400640 1 -- 172.21.15.6:0/140363436 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) ==== 74+0+23214 (secure 0 0 0) 0x7f62cc0c10f0 con 0x7f62dc104290 2024-04-26T02:07:14.881 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:07:14.881 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":131,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:32:29.071544+0000","modified":"2024-04-26T02:07:04.593663+0000","last_up_change":"2024-04-26T01:40:49.915924+0000","last_in_change":"2024-04-26T01:40:29.559647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:36:35.182726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:41:59.612167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:42:04.106467+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":38,"snap_epoch":131,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f164c51d-ce5d-4140-b9e3-a4b84088f5ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6803","nonce":76518486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6805","nonce":76518486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6809","nonce":76518486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6807","nonce":76518486}]},"public_addr":"172.21.15.6:6803/76518486","cluster_addr":"172.21.15.6:6805/76518486","heartbeat_back_addr":"172.21.15.6:6809/76518486","heartbeat_front_addr":"172.21.15.6:6807/76518486","state":["exists","up"]},{"osd":1,"uuid":"57e875dd-ddf1-4ee4-a2a7-9589d7f42818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6801","nonce":400769116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6803","nonce":400769116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6807","nonce":400769116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6805","nonce":400769116}]},"public_addr":"172.21.15.29:6801/400769116","cluster_addr":"172.21.15.29:6803/400769116","heartbeat_back_addr":"172.21.15.29:6807/400769116","heartbeat_front_addr":"172.21.15.29:6805/400769116","state":["exists","up"]},{"osd":2,"uuid":"77e0c294-c48a-4318-938b-4604a533a2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6801","nonce":3508067672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6803","nonce":3508067672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6807","nonce":3508067672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6805","nonce":3508067672}]},"public_addr":"172.21.15.188:6801/3508067672","cluster_addr":"172.21.15.188:6803/3508067672","heartbeat_back_addr":"172.21.15.188:6807/3508067672","heartbeat_front_addr":"172.21.15.188:6805/3508067672","state":["exists","up"]},{"osd":3,"uuid":"6393dac5-996c-47af-9687-0f1393ecc483","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6811","nonce":1987311118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6813","nonce":1987311118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6817","nonce":1987311118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6815","nonce":1987311118}]},"public_addr":"172.21.15.6:6811/1987311118","cluster_addr":"172.21.15.6:6813/1987311118","heartbeat_back_addr":"172.21.15.6:6817/1987311118","heartbeat_front_addr":"172.21.15.6:6815/1987311118","state":["exists","up"]},{"osd":4,"uuid":"f2c1a535-26a0-46c6-a6d0-c2529c786739","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6809","nonce":2125876609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6811","nonce":2125876609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6815","nonce":2125876609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6813","nonce":2125876609}]},"public_addr":"172.21.15.29:6809/2125876609","cluster_addr":"172.21.15.29:6811/2125876609","heartbeat_back_addr":"172.21.15.29:6815/2125876609","heartbeat_front_addr":"172.21.15.29:6813/2125876609","state":["exists","up"]},{"osd":5,"uuid":"9788bb0a-0d79-4743-bce6-80f5342e8f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6809","nonce":2092077007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6811","nonce":2092077007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6815","nonce":2092077007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6813","nonce":2092077007}]},"public_addr":"172.21.15.188:6809/2092077007","cluster_addr":"172.21.15.188:6811/2092077007","heartbeat_back_addr":"172.21.15.188:6815/2092077007","heartbeat_front_addr":"172.21.15.188:6813/2092077007","state":["exists","up"]},{"osd":6,"uuid":"fd8584e1-69ee-4c30-85e4-a637357d1b6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6819","nonce":3256016695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6821","nonce":3256016695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6825","nonce":3256016695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6823","nonce":3256016695}]},"public_addr":"172.21.15.6:6819/3256016695","cluster_addr":"172.21.15.6:6821/3256016695","heartbeat_back_addr":"172.21.15.6:6825/3256016695","heartbeat_front_addr":"172.21.15.6:6823/3256016695","state":["exists","up"]},{"osd":7,"uuid":"6b7e6784-d574-42da-8416-a74a3b6b3755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6817","nonce":3478594037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6819","nonce":3478594037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6823","nonce":3478594037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6821","nonce":3478594037}]},"public_addr":"172.21.15.29:6817/3478594037","cluster_addr":"172.21.15.29:6819/3478594037","heartbeat_back_addr":"172.21.15.29:6823/3478594037","heartbeat_front_addr":"172.21.15.29:6821/3478594037","state":["exists","up"]},{"osd":8,"uuid":"30cd2ea1-2767-4062-8bd9-47214e4b4b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6817","nonce":3290364376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6819","nonce":3290364376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6823","nonce":3290364376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6821","nonce":3290364376}]},"public_addr":"172.21.15.188:6817/3290364376","cluster_addr":"172.21.15.188:6819/3290364376","heartbeat_back_addr":"172.21.15.188:6823/3290364376","heartbeat_front_addr":"172.21.15.188:6821/3290364376","state":["exists","up"]},{"osd":9,"uuid":"f033cdc0-1178-4a7f-a338-69ff32416087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6827","nonce":1918893912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6829","nonce":1918893912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6833","nonce":1918893912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6831","nonce":1918893912}]},"public_addr":"172.21.15.6:6827/1918893912","cluster_addr":"172.21.15.6:6829/1918893912","heartbeat_back_addr":"172.21.15.6:6833/1918893912","heartbeat_front_addr":"172.21.15.6:6831/1918893912","state":["exists","up"]},{"osd":10,"uuid":"04e9b5ef-ac0c-482b-a69e-a7121b36e7da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6825","nonce":1395777729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6827","nonce":1395777729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6831","nonce":1395777729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6829","nonce":1395777729}]},"public_addr":"172.21.15.29:6825/1395777729","cluster_addr":"172.21.15.29:6827/1395777729","heartbeat_back_addr":"172.21.15.29:6831/1395777729","heartbeat_front_addr":"172.21.15.29:6829/1395777729","state":["exists","up"]},{"osd":11,"uuid":"cb1638ce-4189-4241-b63e-b63c607e90ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6825","nonce":3322197880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6827","nonce":3322197880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6831","nonce":3322197880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6829","nonce":3322197880}]},"public_addr":"172.21.15.188:6825/3322197880","cluster_addr":"172.21.15.188:6827/3322197880","heartbeat_back_addr":"172.21.15.188:6831/3322197880","heartbeat_front_addr":"172.21.15.188:6829/3322197880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:41.553455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:07.641467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:31.748891+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:03.743553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:30.065262+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:54.976527+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:29.348206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:56.285928+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:21.246409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:54.468561+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:22.093015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:47.342379+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.19","mappings":[{"from":3,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6801/809793211":"2024-04-27T01:45:54.919747+0000","172.21.15.6:6800/809793211":"2024-04-27T01:45:54.919747+0000","172.21.15.6:0/286280546":"2024-04-27T01:45:54.919747+0000","172.21.15.6:0/402969404":"2024-04-27T01:45:54.919747+0000","172.21.15.6:0/2707582744":"2024-04-27T01:45:54.919747+0000","172.21.15.6:0/3143272062":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1325707596":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6800/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6801/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1359518071":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/1823104535":"2024-04-27T01:45:54.919747+0000","172.21.15.6:6801/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/2813598711":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/3140150861":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6800/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/65701414":"2024-04-27T01:33:47.021882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":26,"length":1},{"begin":38,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":26,"length":1},{"begin":38,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:07:14.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.883+0000 7f62dbe00640 1 -- 172.21.15.6:0/140363436 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f62c00a3d90 msgr2=0x7f62c00a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:14.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.883+0000 7f62dbe00640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f62c00a3d90 0x7f62c00a6250 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f62c80095f0 tx=0x7f62c8005f90 comp rx=0 tx=0).stop 2024-04-26T02:07:14.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.883+0000 7f62dbe00640 1 -- 172.21.15.6:0/140363436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc104290 msgr2=0x7f62dc1a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:14.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.883+0000 7f62dbe00640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc104290 0x7f62dc1a0650 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f62cc005ab0 tx=0x7f62cc003c10 comp rx=0 tx=0).stop 2024-04-26T02:07:14.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.883+0000 7f62dbe00640 1 -- 172.21.15.6:0/140363436 shutdown_connections 2024-04-26T02:07:14.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.883+0000 7f62dbe00640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f62c00a3d90 0x7f62c00a6250 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.883+0000 7f62dbe00640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62dc109360 0x7f62dc1a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.883+0000 7f62dbe00640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62dc19a760 0x7f62dc19abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.883+0000 7f62dbe00640 1 --2- 172.21.15.6:0/140363436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc104290 0x7f62dc1a0650 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.883+0000 7f62dbe00640 1 -- 172.21.15.6:0/140363436 >> 172.21.15.6:0/140363436 conn(0x7f62dc0ffad0 msgr2=0x7f62dc100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.883+0000 7f62dbe00640 1 -- 172.21.15.6:0/140363436 shutdown_connections 2024-04-26T02:07:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:14.883+0000 7f62dbe00640 1 -- 172.21.15.6:0/140363436 wait complete. 2024-04-26T02:07:14.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:14 smithi188 ceph-mon[32751]: pgmap v677: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 307 op/s 2024-04-26T02:07:14.901 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T01:36:35.182726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T01:41:59.612167+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T01:42:04.106467+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 38, 'snap_epoch': 131, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T02:07:14.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T02:07:14.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:14 smithi029 ceph-mon[32775]: pgmap v677: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 307 op/s 2024-04-26T02:07:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.091+0000 7f91af200640 1 -- 172.21.15.6:0/1189591018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a8108e30 msgr2=0x7f91a8115200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.091+0000 7f91af200640 1 --2- 172.21.15.6:0/1189591018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a8108e30 0x7f91a8115200 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f91a4009860 tx=0x7f91a402a340 comp rx=0 tx=0).stop 2024-04-26T02:07:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.091+0000 7f91af200640 1 -- 172.21.15.6:0/1189591018 shutdown_connections 2024-04-26T02:07:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.091+0000 7f91af200640 1 --2- 172.21.15.6:0/1189591018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a8108e30 0x7f91a8115200 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.091+0000 7f91af200640 1 --2- 172.21.15.6:0/1189591018 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91a8108490 0x7f91a81088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.091+0000 7f91af200640 1 --2- 172.21.15.6:0/1189591018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91a810e3d0 0x7f91a810e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.091+0000 7f91af200640 1 -- 172.21.15.6:0/1189591018 >> 172.21.15.6:0/1189591018 conn(0x7f91a8077d60 msgr2=0x7f91a8106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.091+0000 7f91af200640 1 -- 172.21.15.6:0/1189591018 shutdown_connections 2024-04-26T02:07:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.091+0000 7f91af200640 1 -- 172.21.15.6:0/1189591018 wait complete. 2024-04-26T02:07:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.092+0000 7f91af200640 1 Processor -- start 2024-04-26T02:07:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.092+0000 7f91af200640 1 -- start start 2024-04-26T02:07:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.092+0000 7f91af200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91a8108490 0x7f91a81a4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.092+0000 7f91af200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a8108e30 0x7f91a81a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.092+0000 7f91af200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91a810e3d0 0x7f91a819e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.092+0000 7f91af200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f91a81185b0 con 0x7f91a8108490 2024-04-26T02:07:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.092+0000 7f91af200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f91a8118430 con 0x7f91a8108e30 2024-04-26T02:07:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.092+0000 7f91af200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f91a8118730 con 0x7f91a810e3d0 2024-04-26T02:07:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.093+0000 7f91ade00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91a8108490 0x7f91a81a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.093+0000 7f91ad400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a8108e30 0x7f91a81a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.093+0000 7f91ade00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91a8108490 0x7f91a81a4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47608/0 (socket says 172.21.15.6:47608) 2024-04-26T02:07:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.093+0000 7f91ade00640 1 -- 172.21.15.6:0/1841967731 learned_addr learned my addr 172.21.15.6:0/1841967731 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.093+0000 7f91ae800640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91a810e3d0 0x7f91a819e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.093+0000 7f91ad400640 1 -- 172.21.15.6:0/1841967731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91a810e3d0 msgr2=0x7f91a819e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.093+0000 7f91ad400640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91a810e3d0 0x7f91a819e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.093+0000 7f91ad400640 1 -- 172.21.15.6:0/1841967731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91a8108490 msgr2=0x7f91a81a4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.093+0000 7f91ad400640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91a8108490 0x7f91a81a4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.093+0000 7f91ad400640 1 -- 172.21.15.6:0/1841967731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a819f070 con 0x7f91a8108e30 2024-04-26T02:07:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.093+0000 7f91ade00640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91a8108490 0x7f91a81a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.093+0000 7f91ad400640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a8108e30 0x7f91a81a4c80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9198009fd0 tx=0x7f919800ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.094+0000 7f919ea00640 1 -- 172.21.15.6:0/1841967731 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919800ccc0 con 0x7f91a8108e30 2024-04-26T02:07:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.094+0000 7f919ea00640 1 -- 172.21.15.6:0/1841967731 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f919800ee70 con 0x7f91a8108e30 2024-04-26T02:07:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.094+0000 7f919ea00640 1 -- 172.21.15.6:0/1841967731 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9198004a00 con 0x7f91a8108e30 2024-04-26T02:07:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.094+0000 7f91af200640 1 -- 172.21.15.6:0/1841967731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a819f360 con 0x7f91a8108e30 2024-04-26T02:07:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.094+0000 7f91af200640 1 -- 172.21.15.6:0/1841967731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a81ab600 con 0x7f91a8108e30 2024-04-26T02:07:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.095+0000 7f91af200640 1 -- 172.21.15.6:0/1841967731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91a804fce0 con 0x7f91a8108e30 2024-04-26T02:07:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.097+0000 7f919ea00640 1 -- 172.21.15.6:0/1841967731 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f9198004ba0 con 0x7f91a8108e30 2024-04-26T02:07:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.099+0000 7f919ea00640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f91940a3da0 0x7f91940a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.099+0000 7f919ea00640 1 -- 172.21.15.6:0/1841967731 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 8644+0+0 (secure 0 0 0) 0x7f91980d0700 con 0x7f91a8108e30 2024-04-26T02:07:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.099+0000 7f91ade00640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f91940a3da0 0x7f91940a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.100+0000 7f91ade00640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f91940a3da0 0x7f91940a6260 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f91a819fa30 tx=0x7f91a0009500 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.103+0000 7f919ea00640 1 -- 172.21.15.6:0/1841967731 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f919809ccc0 con 0x7f91a8108e30 2024-04-26T02:07:15.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.309+0000 7f91af200640 1 -- 172.21.15.6:0/1841967731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f91a802d5c0 con 0x7f91a8108e30 2024-04-26T02:07:15.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.310+0000 7f919ea00640 1 -- 172.21.15.6:0/1841967731 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) ==== 93+0+10 (secure 0 0 0) 0x7f919809c5c0 con 0x7f91a8108e30 2024-04-26T02:07:15.310 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-26T02:07:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.313+0000 7f91af200640 1 -- 172.21.15.6:0/1841967731 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f91940a3da0 msgr2=0x7f91940a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.313+0000 7f91af200640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f91940a3da0 0x7f91940a6260 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f91a819fa30 tx=0x7f91a0009500 comp rx=0 tx=0).stop 2024-04-26T02:07:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.313+0000 7f91af200640 1 -- 172.21.15.6:0/1841967731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a8108e30 msgr2=0x7f91a81a4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.313+0000 7f91af200640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a8108e30 0x7f91a81a4c80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9198009fd0 tx=0x7f919800ec20 comp rx=0 tx=0).stop 2024-04-26T02:07:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.313+0000 7f91af200640 1 -- 172.21.15.6:0/1841967731 shutdown_connections 2024-04-26T02:07:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.313+0000 7f91af200640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f91940a3da0 0x7f91940a6260 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.313+0000 7f91af200640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91a810e3d0 0x7f91a819e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.313+0000 7f91af200640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a8108e30 0x7f91a81a4c80 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.313+0000 7f91af200640 1 --2- 172.21.15.6:0/1841967731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91a8108490 0x7f91a81a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.313+0000 7f91af200640 1 -- 172.21.15.6:0/1841967731 >> 172.21.15.6:0/1841967731 conn(0x7f91a8077d60 msgr2=0x7f91a8078da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.313+0000 7f91af200640 1 -- 172.21.15.6:0/1841967731 shutdown_connections 2024-04-26T02:07:15.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.313+0000 7f91af200640 1 -- 172.21.15.6:0/1841967731 wait complete. 2024-04-26T02:07:15.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T02:07:15.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.559+0000 7f5d34c00640 1 -- 172.21.15.6:0/201958740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d30109470 msgr2=0x7f5d30111040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.559+0000 7f5d34c00640 1 --2- 172.21.15.6:0/201958740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d30109470 0x7f5d30111040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009900 tx=0x7f5d2002a410 comp rx=0 tx=0).stop 2024-04-26T02:07:15.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.560+0000 7f5d34c00640 1 -- 172.21.15.6:0/201958740 shutdown_connections 2024-04-26T02:07:15.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.560+0000 7f5d34c00640 1 --2- 172.21.15.6:0/201958740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d30109470 0x7f5d30111040 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.560+0000 7f5d34c00640 1 --2- 172.21.15.6:0/201958740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d30104d50 0x7f5d30108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.560+0000 7f5d34c00640 1 --2- 172.21.15.6:0/201958740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d301043a0 0x7f5d30104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.560+0000 7f5d34c00640 1 -- 172.21.15.6:0/201958740 >> 172.21.15.6:0/201958740 conn(0x7f5d300ffbe0 msgr2=0x7f5d30102000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.560+0000 7f5d34c00640 1 -- 172.21.15.6:0/201958740 shutdown_connections 2024-04-26T02:07:15.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.560+0000 7f5d34c00640 1 -- 172.21.15.6:0/201958740 wait complete. 2024-04-26T02:07:15.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.561+0000 7f5d34c00640 1 Processor -- start 2024-04-26T02:07:15.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.561+0000 7f5d34c00640 1 -- start start 2024-04-26T02:07:15.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.561+0000 7f5d34c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d301043a0 0x7f5d301a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.561+0000 7f5d34c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d30109470 0x7f5d301a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.561+0000 7f5d34c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3019a730 0x7f5d3019abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.561+0000 7f5d34c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5d301143d0 con 0x7f5d30109470 2024-04-26T02:07:15.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.561+0000 7f5d34c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5d30114250 con 0x7f5d3019a730 2024-04-26T02:07:15.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.561+0000 7f5d34c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5d30114550 con 0x7f5d301043a0 2024-04-26T02:07:15.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.561+0000 7f5d2d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d30109470 0x7f5d301a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.561+0000 7f5d2d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d30109470 0x7f5d301a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47618/0 (socket says 172.21.15.6:47618) 2024-04-26T02:07:15.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.561+0000 7f5d2d400640 1 -- 172.21.15.6:0/1121633358 learned_addr learned my addr 172.21.15.6:0/1121633358 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:15.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.561+0000 7f5d2de00640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d301043a0 0x7f5d301a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.562+0000 7f5d2e800640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3019a730 0x7f5d3019abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.562+0000 7f5d2d400640 1 -- 172.21.15.6:0/1121633358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d301043a0 msgr2=0x7f5d301a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.562+0000 7f5d2d400640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d301043a0 0x7f5d301a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.562+0000 7f5d2d400640 1 -- 172.21.15.6:0/1121633358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3019a730 msgr2=0x7f5d3019abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.562+0000 7f5d2d400640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3019a730 0x7f5d3019abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.562+0000 7f5d2d400640 1 -- 172.21.15.6:0/1121633358 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3019b330 con 0x7f5d30109470 2024-04-26T02:07:15.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.562+0000 7f5d2d400640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d30109470 0x7f5d301a0bb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5d1800e9f0 tx=0x7f5d1800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.563+0000 7f5d26a00640 1 -- 172.21.15.6:0/1121633358 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1800ccc0 con 0x7f5d30109470 2024-04-26T02:07:15.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.563+0000 7f5d26a00640 1 -- 172.21.15.6:0/1121633358 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d180054d0 con 0x7f5d30109470 2024-04-26T02:07:15.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.563+0000 7f5d34c00640 1 -- 172.21.15.6:0/1121633358 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d301a7410 con 0x7f5d30109470 2024-04-26T02:07:15.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.563+0000 7f5d26a00640 1 -- 172.21.15.6:0/1121633358 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d18010430 con 0x7f5d30109470 2024-04-26T02:07:15.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.563+0000 7f5d34c00640 1 -- 172.21.15.6:0/1121633358 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d301a77c0 con 0x7f5d30109470 2024-04-26T02:07:15.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.564+0000 7f5d34c00640 1 -- 172.21.15.6:0/1121633358 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d30105870 con 0x7f5d30109470 2024-04-26T02:07:15.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.566+0000 7f5d26a00640 1 -- 172.21.15.6:0/1121633358 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f5d180046c0 con 0x7f5d30109470 2024-04-26T02:07:15.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.567+0000 7f5d26a00640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5d140a3d20 0x7f5d140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.567+0000 7f5d26a00640 1 -- 172.21.15.6:0/1121633358 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 8644+0+0 (secure 0 0 0) 0x7f5d180d0bd0 con 0x7f5d30109470 2024-04-26T02:07:15.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.567+0000 7f5d2de00640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5d140a3d20 0x7f5d140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.568+0000 7f5d2de00640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5d140a3d20 0x7f5d140a61e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5d28005fd0 tx=0x7f5d28009500 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.571+0000 7f5d26a00640 1 -- 172.21.15.6:0/1121633358 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d1809d190 con 0x7f5d30109470 2024-04-26T02:07:15.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.778+0000 7f5d34c00640 1 -- 172.21.15.6:0/1121633358 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5d30069250 con 0x7f5d30109470 2024-04-26T02:07:15.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.779+0000 7f5d26a00640 1 -- 172.21.15.6:0/1121633358 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v132) ==== 104+0+11 (secure 0 0 0) 0x7f5d1809ca90 con 0x7f5d30109470 2024-04-26T02:07:15.779 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-26T02:07:15.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.781+0000 7f5d34c00640 1 -- 172.21.15.6:0/1121633358 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5d140a3d20 msgr2=0x7f5d140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.781+0000 7f5d34c00640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5d140a3d20 0x7f5d140a61e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5d28005fd0 tx=0x7f5d28009500 comp rx=0 tx=0).stop 2024-04-26T02:07:15.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.781+0000 7f5d34c00640 1 -- 172.21.15.6:0/1121633358 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d30109470 msgr2=0x7f5d301a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.781+0000 7f5d34c00640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d30109470 0x7f5d301a0bb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5d1800e9f0 tx=0x7f5d1800eeb0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.781+0000 7f5d34c00640 1 -- 172.21.15.6:0/1121633358 shutdown_connections 2024-04-26T02:07:15.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.781+0000 7f5d34c00640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5d140a3d20 0x7f5d140a61e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.781+0000 7f5d34c00640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d30109470 0x7f5d301a0bb0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.781+0000 7f5d34c00640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3019a730 0x7f5d3019abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.781+0000 7f5d34c00640 1 --2- 172.21.15.6:0/1121633358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d301043a0 0x7f5d301a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.782+0000 7f5d34c00640 1 -- 172.21.15.6:0/1121633358 >> 172.21.15.6:0/1121633358 conn(0x7f5d300ffbe0 msgr2=0x7f5d30101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.782+0000 7f5d34c00640 1 -- 172.21.15.6:0/1121633358 shutdown_connections 2024-04-26T02:07:15.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:15.782+0000 7f5d34c00640 1 -- 172.21.15.6:0/1121633358 wait complete. 2024-04-26T02:07:15.800 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T02:07:15.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:15 smithi029 ceph-mon[32775]: from='client.27628 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T02:07:15.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:15 smithi029 ceph-mon[32775]: from='client.17355 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T02:07:15.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:15 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/140363436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T02:07:15.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:15 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1841967731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T02:07:15.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:15 smithi029 ceph-mon[32775]: osdmap e132: 12 total, 12 up, 12 in 2024-04-26T02:07:16.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.005+0000 7f4dcd600640 1 -- 172.21.15.6:0/2094842918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dc810e8b0 msgr2=0x7f4dc8110ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:16.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.005+0000 7f4dcd600640 1 --2- 172.21.15.6:0/2094842918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dc810e8b0 0x7f4dc8110ca0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0009ca0 tx=0x7f4dc002a8c0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.005+0000 7f4dcd600640 1 -- 172.21.15.6:0/2094842918 shutdown_connections 2024-04-26T02:07:16.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.005+0000 7f4dcd600640 1 --2- 172.21.15.6:0/2094842918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dc810e8b0 0x7f4dc8110ca0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.005+0000 7f4dcd600640 1 --2- 172.21.15.6:0/2094842918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc8100c90 0x7f4dc810e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.005+0000 7f4dcd600640 1 --2- 172.21.15.6:0/2094842918 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dc8100370 0x7f4dc8100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.005+0000 7f4dcd600640 1 -- 172.21.15.6:0/2094842918 >> 172.21.15.6:0/2094842918 conn(0x7f4dc8075cf0 msgr2=0x7f4dc8076100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:16.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.005+0000 7f4dcd600640 1 -- 172.21.15.6:0/2094842918 shutdown_connections 2024-04-26T02:07:16.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.005+0000 7f4dcd600640 1 -- 172.21.15.6:0/2094842918 wait complete. 2024-04-26T02:07:16.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.006+0000 7f4dcd600640 1 Processor -- start 2024-04-26T02:07:16.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.007+0000 7f4dcd600640 1 -- start start 2024-04-26T02:07:16.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.007+0000 7f4dcd600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc8100370 0x7f4dc81a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:16.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.007+0000 7f4dcd600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dc8100c90 0x7f4dc81a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:16.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.007+0000 7f4dcd600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dc810e8b0 0x7f4dc819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:16.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.007+0000 7f4dcd600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc8114330 con 0x7f4dc810e8b0 2024-04-26T02:07:16.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.007+0000 7f4dcd600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc81141b0 con 0x7f4dc8100370 2024-04-26T02:07:16.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.007+0000 7f4dcd600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc81144b0 con 0x7f4dc8100c90 2024-04-26T02:07:16.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.007+0000 7f4dc7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dc810e8b0 0x7f4dc819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:16.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.007+0000 7f4dc6a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc8100370 0x7f4dc81a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:16.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.007+0000 7f4dc7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dc810e8b0 0x7f4dc819a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47626/0 (socket says 172.21.15.6:47626) 2024-04-26T02:07:16.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.007+0000 7f4dc7400640 1 -- 172.21.15.6:0/523553894 learned_addr learned my addr 172.21.15.6:0/523553894 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:16.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.007+0000 7f4dc6000640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dc8100c90 0x7f4dc81a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:16.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.008+0000 7f4dc6000640 1 -- 172.21.15.6:0/523553894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc8100370 msgr2=0x7f4dc81a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:16.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.008+0000 7f4dc6000640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc8100370 0x7f4dc81a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.008+0000 7f4dc6000640 1 -- 172.21.15.6:0/523553894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dc810e8b0 msgr2=0x7f4dc819a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:16.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.008+0000 7f4dc6000640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dc810e8b0 0x7f4dc819a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.008+0000 7f4dc6000640 1 -- 172.21.15.6:0/523553894 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc819af70 con 0x7f4dc8100c90 2024-04-26T02:07:16.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.008+0000 7f4dc6000640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dc8100c90 0x7f4dc81a0ae0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4db000e9f0 tx=0x7f4db000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:16.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.008+0000 7f4dbf400640 1 -- 172.21.15.6:0/523553894 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db000cd80 con 0x7f4dc8100c90 2024-04-26T02:07:16.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.008+0000 7f4dbf400640 1 -- 172.21.15.6:0/523553894 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4db00035f0 con 0x7f4dc8100c90 2024-04-26T02:07:16.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.008+0000 7f4dcd600640 1 -- 172.21.15.6:0/523553894 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc819b260 con 0x7f4dc8100c90 2024-04-26T02:07:16.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.009+0000 7f4dbf400640 1 -- 172.21.15.6:0/523553894 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0010670 con 0x7f4dc8100c90 2024-04-26T02:07:16.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.009+0000 7f4dcd600640 1 -- 172.21.15.6:0/523553894 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc81a7520 con 0x7f4dc8100c90 2024-04-26T02:07:16.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.010+0000 7f4dcd600640 1 -- 172.21.15.6:0/523553894 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dc8101590 con 0x7f4dc8100c90 2024-04-26T02:07:16.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.012+0000 7f4dbf400640 1 -- 172.21.15.6:0/523553894 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f4db00037a0 con 0x7f4dc8100c90 2024-04-26T02:07:16.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.013+0000 7f4dbf400640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4dac0a3d20 0x7f4dac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:16.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.013+0000 7f4dbf400640 1 -- 172.21.15.6:0/523553894 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8600+0+0 (secure 0 0 0) 0x7f4db00d0b20 con 0x7f4dc8100c90 2024-04-26T02:07:16.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.013+0000 7f4dc6a00640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4dac0a3d20 0x7f4dac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:16.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.014+0000 7f4dc6a00640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4dac0a3d20 0x7f4dac0a61e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009bd0 tx=0x7f4db8005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:16.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.017+0000 7f4dbf400640 1 -- 172.21.15.6:0/523553894 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4db009d110 con 0x7f4dc8100c90 2024-04-26T02:07:16.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:15 smithi006 ceph-mon[28419]: from='client.27628 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T02:07:16.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:15 smithi006 ceph-mon[28419]: from='client.17355 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T02:07:16.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:15 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/140363436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T02:07:16.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:15 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1841967731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T02:07:16.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:15 smithi006 ceph-mon[28419]: osdmap e132: 12 total, 12 up, 12 in 2024-04-26T02:07:16.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:15 smithi188 ceph-mon[32751]: from='client.27628 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T02:07:16.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:15 smithi188 ceph-mon[32751]: from='client.17355 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T02:07:16.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:15 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/140363436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T02:07:16.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:15 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1841967731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T02:07:16.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:15 smithi188 ceph-mon[32751]: osdmap e132: 12 total, 12 up, 12 in 2024-04-26T02:07:16.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.229+0000 7f4dcd600640 1 -- 172.21.15.6:0/523553894 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4dc8109a40 con 0x7f4dc8100c90 2024-04-26T02:07:16.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.231+0000 7f4dbf400640 1 -- 172.21.15.6:0/523553894 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v132) ==== 100+0+11 (secure 0 0 0) 0x7f4db009ca10 con 0x7f4dc8100c90 2024-04-26T02:07:16.230 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-04-26T02:07:16.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.233+0000 7f4dcd600640 1 -- 172.21.15.6:0/523553894 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4dac0a3d20 msgr2=0x7f4dac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:16.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.233+0000 7f4dcd600640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4dac0a3d20 0x7f4dac0a61e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009bd0 tx=0x7f4db8005f90 comp rx=0 tx=0).stop 2024-04-26T02:07:16.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.233+0000 7f4dcd600640 1 -- 172.21.15.6:0/523553894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dc8100c90 msgr2=0x7f4dc81a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.233+0000 7f4dcd600640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dc8100c90 0x7f4dc81a0ae0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4db000e9f0 tx=0x7f4db000eeb0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.233+0000 7f4dcd600640 1 -- 172.21.15.6:0/523553894 shutdown_connections 2024-04-26T02:07:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.233+0000 7f4dcd600640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f4dac0a3d20 0x7f4dac0a61e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.233+0000 7f4dcd600640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dc810e8b0 0x7f4dc819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.233+0000 7f4dcd600640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4dc8100c90 0x7f4dc81a0ae0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.233+0000 7f4dcd600640 1 --2- 172.21.15.6:0/523553894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc8100370 0x7f4dc81a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.233+0000 7f4dcd600640 1 -- 172.21.15.6:0/523553894 >> 172.21.15.6:0/523553894 conn(0x7f4dc8075cf0 msgr2=0x7f4dc810eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.234+0000 7f4dcd600640 1 -- 172.21.15.6:0/523553894 shutdown_connections 2024-04-26T02:07:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.234+0000 7f4dcd600640 1 -- 172.21.15.6:0/523553894 wait complete. 2024-04-26T02:07:16.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-26T02:07:16.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.458+0000 7f94f3e00640 1 -- 172.21.15.6:0/1930793166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94f4104290 msgr2=0x7f94f4104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:16.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.458+0000 7f94f3e00640 1 --2- 172.21.15.6:0/1930793166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94f4104290 0x7f94f4104670 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f94e4009860 tx=0x7f94e402a340 comp rx=0 tx=0).stop 2024-04-26T02:07:16.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.458+0000 7f94f3e00640 1 -- 172.21.15.6:0/1930793166 shutdown_connections 2024-04-26T02:07:16.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.458+0000 7f94f3e00640 1 --2- 172.21.15.6:0/1930793166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94f4109360 0x7f94f4110f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.458+0000 7f94f3e00640 1 --2- 172.21.15.6:0/1930793166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f4104c40 0x7f94f4108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.458+0000 7f94f3e00640 1 --2- 172.21.15.6:0/1930793166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94f4104290 0x7f94f4104670 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.458+0000 7f94f3e00640 1 -- 172.21.15.6:0/1930793166 >> 172.21.15.6:0/1930793166 conn(0x7f94f40ffad0 msgr2=0x7f94f4101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:16.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.458+0000 7f94f3e00640 1 -- 172.21.15.6:0/1930793166 shutdown_connections 2024-04-26T02:07:16.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.458+0000 7f94f3e00640 1 -- 172.21.15.6:0/1930793166 wait complete. 2024-04-26T02:07:16.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.459+0000 7f94f3e00640 1 Processor -- start 2024-04-26T02:07:16.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.459+0000 7f94f3e00640 1 -- start start 2024-04-26T02:07:16.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.459+0000 7f94f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94f4104290 0x7f94f41a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:16.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.459+0000 7f94f3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f4109360 0x7f94f41a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:16.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.459+0000 7f94f3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94f419a6c0 0x7f94f419ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:16.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f94f41142c0 con 0x7f94f4104290 2024-04-26T02:07:16.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f3e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f94f4114140 con 0x7f94f4109360 2024-04-26T02:07:16.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f3e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f94f4114440 con 0x7f94f419a6c0 2024-04-26T02:07:16.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94f4104290 0x7f94f41a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:16.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94f4104290 0x7f94f41a0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47650/0 (socket says 172.21.15.6:47650) 2024-04-26T02:07:16.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f2a00640 1 -- 172.21.15.6:0/2191683823 learned_addr learned my addr 172.21.15.6:0/2191683823 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:16.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f2000640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f4109360 0x7f94f41a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:16.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f3400640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94f419a6c0 0x7f94f419ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:16.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f2a00640 1 -- 172.21.15.6:0/2191683823 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94f419a6c0 msgr2=0x7f94f419ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:16.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f2a00640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94f419a6c0 0x7f94f419ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f2a00640 1 -- 172.21.15.6:0/2191683823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f4109360 msgr2=0x7f94f41a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:16.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f2a00640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f4109360 0x7f94f41a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:16.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f2a00640 1 -- 172.21.15.6:0/2191683823 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f419b2c0 con 0x7f94f4104290 2024-04-26T02:07:16.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.460+0000 7f94f2000640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f4109360 0x7f94f41a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:16.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.461+0000 7f94f2a00640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94f4104290 0x7f94f41a0600 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f94e4035040 tx=0x7f94e4032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:16.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.461+0000 7f94eb400640 1 -- 172.21.15.6:0/2191683823 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e4037040 con 0x7f94f4104290 2024-04-26T02:07:16.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.461+0000 7f94eb400640 1 -- 172.21.15.6:0/2191683823 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94e4041070 con 0x7f94f4104290 2024-04-26T02:07:16.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.461+0000 7f94f3e00640 1 -- 172.21.15.6:0/2191683823 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94f419b550 con 0x7f94f4104290 2024-04-26T02:07:16.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.461+0000 7f94eb400640 1 -- 172.21.15.6:0/2191683823 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e4003850 con 0x7f94f4104290 2024-04-26T02:07:16.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.461+0000 7f94f3e00640 1 -- 172.21.15.6:0/2191683823 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94f41a7680 con 0x7f94f4104290 2024-04-26T02:07:16.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.463+0000 7f94f3e00640 1 -- 172.21.15.6:0/2191683823 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94f4105760 con 0x7f94f4104290 2024-04-26T02:07:16.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.465+0000 7f94eb400640 1 -- 172.21.15.6:0/2191683823 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f94e40039f0 con 0x7f94f4104290 2024-04-26T02:07:16.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.467+0000 7f94eb400640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f94d80a3d90 0x7f94d80a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:16.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.467+0000 7f94eb400640 1 -- 172.21.15.6:0/2191683823 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8600+0+0 (secure 0 0 0) 0x7f94e40f01d0 con 0x7f94f4104290 2024-04-26T02:07:16.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.467+0000 7f94f2000640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f94d80a3d90 0x7f94d80a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:16.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.468+0000 7f94f2000640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f94d80a3d90 0x7f94d80a6250 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f94f419bf10 tx=0x7f94e0005f90 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:16.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.472+0000 7f94eb400640 1 -- 172.21.15.6:0/2191683823 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94e404b020 con 0x7f94f4104290 2024-04-26T02:07:16.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.733+0000 7f94f3e00640 1 -- 172.21.15.6:0/2191683823 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f94f410c590 con 0x7f94f4104290 2024-04-26T02:07:16.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.768+0000 7f94eb400640 1 -- 172.21.15.6:0/2191683823 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) ==== 81+0+746053 (secure 0 0 0) 0x7f94e40bc930 con 0x7f94f4104290 2024-04-26T02:07:16.781 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:07:16.781 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:07:16.781 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 27, 2024-04-26T02:07:16.781 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.782 INFO:teuthology.orchestra.run.smithi006.stdout: "active_gid": 25037, 2024-04-26T02:07:16.782 INFO:teuthology.orchestra.run.smithi006.stdout: "active_name": "x", 2024-04-26T02:07:16.782 INFO:teuthology.orchestra.run.smithi006.stdout: "active_addrs": { 2024-04-26T02:07:16.782 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-26T02:07:16.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:16.782 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-04-26T02:07:16.782 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6800", 2024-04-26T02:07:16.783 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3353570381 2024-04-26T02:07:16.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.783 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:16.783 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-26T02:07:16.783 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6801", 2024-04-26T02:07:16.783 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3353570381 2024-04-26T02:07:16.783 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:16.783 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-26T02:07:16.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.784 INFO:teuthology.orchestra.run.smithi006.stdout: "active_addr": "172.21.15.6:6801/3353570381", 2024-04-26T02:07:16.784 INFO:teuthology.orchestra.run.smithi006.stdout: "active_change": "2024-04-26T01:45:54.920039+0000", 2024-04-26T02:07:16.784 INFO:teuthology.orchestra.run.smithi006.stdout: "active_mgr_features": 4540701547738038271, 2024-04-26T02:07:16.784 INFO:teuthology.orchestra.run.smithi006.stdout: "available": true, 2024-04-26T02:07:16.784 INFO:teuthology.orchestra.run.smithi006.stdout: "standbys": [ 2024-04-26T02:07:16.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:16.784 INFO:teuthology.orchestra.run.smithi006.stdout: "gid": 25034, 2024-04-26T02:07:16.785 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "y", 2024-04-26T02:07:16.785 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_features": 4540701547738038271, 2024-04-26T02:07:16.785 INFO:teuthology.orchestra.run.smithi006.stdout: "available_modules": [ 2024-04-26T02:07:16.785 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:16.785 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "alerts", 2024-04-26T02:07:16.785 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:16.785 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:16.785 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:16.785 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:16.786 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:16.786 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:16.786 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.786 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.786 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:16.786 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.786 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.786 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.787 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T02:07:16.787 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.787 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.787 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.787 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:16.787 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:16.787 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.787 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.788 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.788 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.788 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.788 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.788 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:16.788 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:16.788 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:16.788 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:16.789 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:16.789 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:16.789 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:16.789 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:16.789 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.789 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.789 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.789 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.790 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:16.790 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:16.790 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.790 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.790 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.790 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.790 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.791 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.791 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.791 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.791 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.791 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.791 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.791 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.791 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:16.792 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:16.792 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.792 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.792 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.792 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:16.792 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.792 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.792 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:16.792 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:16.793 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:16.793 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:16.793 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:16.793 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:16.793 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:16.793 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:16.793 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.793 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.793 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.794 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.794 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:16.794 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:16.794 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.794 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.794 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.794 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.795 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.795 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.795 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.795 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.795 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.795 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.795 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.795 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.796 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_destination": { 2024-04-26T02:07:16.796 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_destination", 2024-04-26T02:07:16.796 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.796 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.796 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.796 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.796 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.796 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.797 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.797 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Email address to send alerts to", 2024-04-26T02:07:16.797 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.797 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.797 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.797 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_from_name": { 2024-04-26T02:07:16.797 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_from_name", 2024-04-26T02:07:16.798 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.798 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.798 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.798 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "Ceph", 2024-04-26T02:07:16.798 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.798 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.798 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.798 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Email From: name", 2024-04-26T02:07:16.799 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.799 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.799 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.799 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_host": { 2024-04-26T02:07:16.799 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_host", 2024-04-26T02:07:16.799 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.799 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.800 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.800 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.800 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.800 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.800 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.800 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "SMTP server", 2024-04-26T02:07:16.800 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.800 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.800 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.801 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_password": { 2024-04-26T02:07:16.801 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_password", 2024-04-26T02:07:16.801 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.801 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.801 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.801 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.801 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.802 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.802 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.802 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Password to authenticate with", 2024-04-26T02:07:16.802 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.802 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.802 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.802 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.802 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_port": { 2024-04-26T02:07:16.802 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_port", 2024-04-26T02:07:16.803 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:16.803 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.803 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.803 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "465", 2024-04-26T02:07:16.803 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.803 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.803 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.803 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "SMTP port", 2024-04-26T02:07:16.803 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.804 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.804 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.804 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.804 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_sender": { 2024-04-26T02:07:16.804 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_sender", 2024-04-26T02:07:16.804 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.804 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.804 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.805 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.805 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.805 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.805 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.805 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "SMTP envelope sender", 2024-04-26T02:07:16.805 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.805 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.805 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.806 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_ssl": { 2024-04-26T02:07:16.806 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_ssl", 2024-04-26T02:07:16.806 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.806 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.806 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.806 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.806 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.807 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.807 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.807 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T02:07:16.807 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.807 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.807 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.807 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_user": { 2024-04-26T02:07:16.807 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_user", 2024-04-26T02:07:16.808 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.808 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.808 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.808 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.808 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.808 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.808 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.808 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "User to authenticate as", 2024-04-26T02:07:16.808 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.809 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.809 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.809 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:16.809 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:16.809 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.809 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:16.809 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "balancer", 2024-04-26T02:07:16.809 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:16.809 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:16.810 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:16.810 INFO:teuthology.orchestra.run.smithi006.stdout: "active": { 2024-04-26T02:07:16.810 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "active", 2024-04-26T02:07:16.810 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.810 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.810 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.810 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.810 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.810 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.811 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.811 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T02:07:16.811 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.811 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.811 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.811 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.811 INFO:teuthology.orchestra.run.smithi006.stdout: "begin_time": { 2024-04-26T02:07:16.811 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "begin_time", 2024-04-26T02:07:16.811 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.812 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.812 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.812 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0000", 2024-04-26T02:07:16.812 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.812 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.812 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.813 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T02:07:16.814 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T02:07:16.814 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.814 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.814 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.814 INFO:teuthology.orchestra.run.smithi006.stdout: "begin_weekday": { 2024-04-26T02:07:16.814 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "begin_weekday", 2024-04-26T02:07:16.814 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:16.814 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.815 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.815 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:16.815 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "0", 2024-04-26T02:07:16.815 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "6", 2024-04-26T02:07:16.815 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.815 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T02:07:16.815 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T02:07:16.815 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.815 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.816 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_compat_max_iterations": { 2024-04-26T02:07:16.816 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crush_compat_max_iterations", 2024-04-26T02:07:16.816 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:16.816 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.816 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.816 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "25", 2024-04-26T02:07:16.816 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1", 2024-04-26T02:07:16.817 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "250", 2024-04-26T02:07:16.817 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.817 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T02:07:16.817 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.817 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.817 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.817 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_compat_metrics": { 2024-04-26T02:07:16.817 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crush_compat_metrics", 2024-04-26T02:07:16.818 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.818 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.818 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.818 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T02:07:16.818 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.818 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.818 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.818 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T02:07:16.818 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T02:07:16.819 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.819 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.819 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.819 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_compat_step": { 2024-04-26T02:07:16.819 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crush_compat_step", 2024-04-26T02:07:16.819 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:16.819 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.819 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.819 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0.5", 2024-04-26T02:07:16.819 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "0.001", 2024-04-26T02:07:16.820 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "0.999", 2024-04-26T02:07:16.820 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.820 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "aggressiveness of optimization", 2024-04-26T02:07:16.820 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T02:07:16.820 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.820 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.820 INFO:teuthology.orchestra.run.smithi006.stdout: "end_time": { 2024-04-26T02:07:16.820 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "end_time", 2024-04-26T02:07:16.820 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.821 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.821 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.821 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "2359", 2024-04-26T02:07:16.821 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.821 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.821 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.821 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T02:07:16.821 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T02:07:16.821 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.822 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.822 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.822 INFO:teuthology.orchestra.run.smithi006.stdout: "end_weekday": { 2024-04-26T02:07:16.822 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "end_weekday", 2024-04-26T02:07:16.822 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:16.822 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.822 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.822 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:16.822 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "0", 2024-04-26T02:07:16.823 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "6", 2024-04-26T02:07:16.823 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.823 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T02:07:16.823 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T02:07:16.823 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.823 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.823 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.823 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:16.823 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:16.823 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.824 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.824 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.824 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.824 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.824 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.824 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:16.824 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:16.824 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:16.824 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:16.824 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:16.825 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:16.825 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:16.825 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:16.825 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.825 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.825 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.825 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.825 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.825 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:16.825 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:16.826 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.826 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.826 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.826 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.826 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.826 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.826 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.826 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.827 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.827 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.827 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.827 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.827 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:16.827 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:16.827 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.827 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.827 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.828 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:16.828 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.828 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.828 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:16.828 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:16.828 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:16.828 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:16.828 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:16.828 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:16.828 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:16.829 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:16.829 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.829 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.829 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.829 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.829 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:16.829 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:16.829 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.829 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.830 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.830 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.830 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.830 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.830 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.830 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.830 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.830 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.831 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.831 INFO:teuthology.orchestra.run.smithi006.stdout: "min_score": { 2024-04-26T02:07:16.831 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "min_score", 2024-04-26T02:07:16.831 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:16.831 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.831 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.831 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:16.831 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.832 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.832 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.832 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T02:07:16.832 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.832 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.832 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.832 INFO:teuthology.orchestra.run.smithi006.stdout: "mode": { 2024-04-26T02:07:16.833 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mode", 2024-04-26T02:07:16.833 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.833 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.833 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.833 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "upmap", 2024-04-26T02:07:16.833 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.833 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.833 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:16.833 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-compat", 2024-04-26T02:07:16.833 INFO:teuthology.orchestra.run.smithi006.stdout: "none", 2024-04-26T02:07:16.834 INFO:teuthology.orchestra.run.smithi006.stdout: "read", 2024-04-26T02:07:16.834 INFO:teuthology.orchestra.run.smithi006.stdout: "upmap", 2024-04-26T02:07:16.834 INFO:teuthology.orchestra.run.smithi006.stdout: "upmap-read" 2024-04-26T02:07:16.834 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:16.834 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Balancer mode", 2024-04-26T02:07:16.834 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.834 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.834 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.834 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.835 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_ids": { 2024-04-26T02:07:16.835 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "pool_ids", 2024-04-26T02:07:16.835 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.835 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.835 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.835 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.835 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.835 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.835 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.836 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T02:07:16.836 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.836 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.836 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.836 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.836 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:16.836 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:16.836 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:16.836 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.836 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.837 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:16.837 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.837 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.837 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.837 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T02:07:16.837 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.837 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.837 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.838 INFO:teuthology.orchestra.run.smithi006.stdout: "upmap_max_deviation": { 2024-04-26T02:07:16.838 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "upmap_max_deviation", 2024-04-26T02:07:16.838 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:16.838 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.838 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.838 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5", 2024-04-26T02:07:16.838 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1", 2024-04-26T02:07:16.838 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.838 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.839 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T02:07:16.839 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T02:07:16.839 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.839 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.839 INFO:teuthology.orchestra.run.smithi006.stdout: "upmap_max_optimizations": { 2024-04-26T02:07:16.839 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "upmap_max_optimizations", 2024-04-26T02:07:16.839 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:16.839 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.840 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.840 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:16.840 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.840 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.840 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.840 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T02:07:16.840 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.840 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.840 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.840 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:16.841 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:16.841 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:16.841 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephadm", 2024-04-26T02:07:16.841 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:16.841 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:16.841 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:16.841 INFO:teuthology.orchestra.run.smithi006.stdout: "agent_down_multiplier": { 2024-04-26T02:07:16.842 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "agent_down_multiplier", 2024-04-26T02:07:16.842 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:16.842 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.842 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.842 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "3.0", 2024-04-26T02:07:16.842 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.842 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.842 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.842 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T02:07:16.843 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.843 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.843 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.843 INFO:teuthology.orchestra.run.smithi006.stdout: "agent_refresh_rate": { 2024-04-26T02:07:16.843 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "agent_refresh_rate", 2024-04-26T02:07:16.843 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:16.843 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.843 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.844 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "20", 2024-04-26T02:07:16.844 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.844 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.844 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.844 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T02:07:16.844 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.844 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.844 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.844 INFO:teuthology.orchestra.run.smithi006.stdout: "agent_starting_port": { 2024-04-26T02:07:16.845 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "agent_starting_port", 2024-04-26T02:07:16.845 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:16.845 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.845 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.845 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "4721", 2024-04-26T02:07:16.845 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.845 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.845 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.845 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T02:07:16.846 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.846 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.846 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.846 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.846 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_ptrace": { 2024-04-26T02:07:16.846 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "allow_ptrace", 2024-04-26T02:07:16.846 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.846 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.847 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.847 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.847 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.847 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.847 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.847 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T02:07:16.847 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T02:07:16.847 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.847 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.848 INFO:teuthology.orchestra.run.smithi006.stdout: "autotune_interval": { 2024-04-26T02:07:16.848 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "autotune_interval", 2024-04-26T02:07:16.848 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:16.848 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.848 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.848 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:16.848 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.848 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.848 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.849 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T02:07:16.849 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.849 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.849 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.849 INFO:teuthology.orchestra.run.smithi006.stdout: "autotune_memory_target_ratio": { 2024-04-26T02:07:16.849 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T02:07:16.849 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:16.849 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.849 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.850 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0.7", 2024-04-26T02:07:16.850 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.850 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.850 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.850 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T02:07:16.850 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.850 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.850 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.851 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm_log_destination": { 2024-04-26T02:07:16.851 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephadm_log_destination", 2024-04-26T02:07:16.851 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.851 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.851 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.851 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.851 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.851 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.851 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:16.852 INFO:teuthology.orchestra.run.smithi006.stdout: "file", 2024-04-26T02:07:16.852 INFO:teuthology.orchestra.run.smithi006.stdout: "file,syslog", 2024-04-26T02:07:16.852 INFO:teuthology.orchestra.run.smithi006.stdout: "syslog" 2024-04-26T02:07:16.852 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:16.852 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T02:07:16.852 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.852 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.852 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.853 INFO:teuthology.orchestra.run.smithi006.stdout: "cgroups_split": { 2024-04-26T02:07:16.853 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cgroups_split", 2024-04-26T02:07:16.853 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.853 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.853 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.853 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.853 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.854 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.854 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.854 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T02:07:16.854 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.854 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.854 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.854 INFO:teuthology.orchestra.run.smithi006.stdout: "config_checks_enabled": { 2024-04-26T02:07:16.855 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "config_checks_enabled", 2024-04-26T02:07:16.855 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.855 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.855 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.855 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.855 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.855 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.855 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.856 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T02:07:16.856 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.856 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.856 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.856 INFO:teuthology.orchestra.run.smithi006.stdout: "config_dashboard": { 2024-04-26T02:07:16.856 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "config_dashboard", 2024-04-26T02:07:16.856 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.857 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.857 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.857 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.857 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.857 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.857 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.857 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T02:07:16.857 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.858 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.858 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.858 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_alertmanager": { 2024-04-26T02:07:16.858 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_alertmanager", 2024-04-26T02:07:16.858 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.858 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.858 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.859 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T02:07:16.859 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.859 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.859 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.859 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:16.859 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.859 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.859 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.860 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_base": { 2024-04-26T02:07:16.860 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_base", 2024-04-26T02:07:16.860 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.860 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.860 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.860 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T02:07:16.861 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.861 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.861 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.861 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Container image name, without the tag", 2024-04-26T02:07:16.861 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.861 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.861 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.862 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_elasticsearch": { 2024-04-26T02:07:16.862 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_elasticsearch", 2024-04-26T02:07:16.862 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.862 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.862 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.862 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T02:07:16.862 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.862 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.863 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.863 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "elasticsearch container image", 2024-04-26T02:07:16.863 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.863 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.863 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.863 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_grafana": { 2024-04-26T02:07:16.864 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_grafana", 2024-04-26T02:07:16.864 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.864 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.864 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.864 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T02:07:16.864 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.864 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.864 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_haproxy": { 2024-04-26T02:07:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_haproxy", 2024-04-26T02:07:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.866 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.866 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.866 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T02:07:16.866 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.866 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.866 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.867 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "HAproxy container image", 2024-04-26T02:07:16.867 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.867 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.867 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.867 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.867 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_jaeger_agent": { 2024-04-26T02:07:16.867 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_jaeger_agent", 2024-04-26T02:07:16.867 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.868 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.868 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.868 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T02:07:16.868 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.868 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.868 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.868 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Jaeger agent container image", 2024-04-26T02:07:16.868 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.869 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.869 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.869 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_jaeger_collector": { 2024-04-26T02:07:16.869 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_jaeger_collector", 2024-04-26T02:07:16.869 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.869 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.869 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.870 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T02:07:16.870 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.870 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.870 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.870 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Jaeger collector container image", 2024-04-26T02:07:16.870 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.870 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.870 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.871 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_jaeger_query": { 2024-04-26T02:07:16.871 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_jaeger_query", 2024-04-26T02:07:16.871 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.871 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.871 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.871 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T02:07:16.871 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.871 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.871 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.872 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Jaeger query container image", 2024-04-26T02:07:16.872 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.872 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.872 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.872 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.872 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_keepalived": { 2024-04-26T02:07:16.872 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_keepalived", 2024-04-26T02:07:16.872 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.872 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.873 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.873 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T02:07:16.873 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.873 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.873 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.873 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Keepalived container image", 2024-04-26T02:07:16.873 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.873 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.874 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.874 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_loki": { 2024-04-26T02:07:16.874 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_loki", 2024-04-26T02:07:16.874 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.874 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.874 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.875 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T02:07:16.875 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.875 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.875 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.875 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Loki container image", 2024-04-26T02:07:16.875 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.875 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.875 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.876 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_node_exporter": { 2024-04-26T02:07:16.876 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_node_exporter", 2024-04-26T02:07:16.876 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.876 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.876 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.876 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T02:07:16.876 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.877 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.877 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.877 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:16.877 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.877 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.877 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.877 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_nvmeof": { 2024-04-26T02:07:16.877 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_nvmeof", 2024-04-26T02:07:16.878 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.878 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.878 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.878 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T02:07:16.878 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.878 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.878 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.879 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Nvme-of container image", 2024-04-26T02:07:16.879 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.879 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.879 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.879 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_prometheus": { 2024-04-26T02:07:16.879 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_prometheus", 2024-04-26T02:07:16.879 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.879 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.880 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.880 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T02:07:16.880 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.880 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.880 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.880 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:16.880 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.880 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.881 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.881 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_promtail": { 2024-04-26T02:07:16.881 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_promtail", 2024-04-26T02:07:16.881 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.881 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.881 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.881 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T02:07:16.881 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.882 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.882 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.882 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Promtail container image", 2024-04-26T02:07:16.882 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.882 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.882 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.882 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_samba": { 2024-04-26T02:07:16.883 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_samba", 2024-04-26T02:07:16.883 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.883 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.883 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.883 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T02:07:16.883 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.883 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.883 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.883 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Samba/SMB container image", 2024-04-26T02:07:16.884 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.884 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.884 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.884 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_snmp_gateway": { 2024-04-26T02:07:16.884 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_snmp_gateway", 2024-04-26T02:07:16.884 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.884 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.885 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.885 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T02:07:16.885 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.885 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.885 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.885 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "SNMP Gateway container image", 2024-04-26T02:07:16.885 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.885 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.885 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.886 INFO:teuthology.orchestra.run.smithi006.stdout: "container_init": { 2024-04-26T02:07:16.886 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_init", 2024-04-26T02:07:16.886 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.886 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.886 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.886 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.886 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.887 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.887 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.887 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T02:07:16.887 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.887 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.887 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.887 INFO:teuthology.orchestra.run.smithi006.stdout: "daemon_cache_timeout": { 2024-04-26T02:07:16.887 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "daemon_cache_timeout", 2024-04-26T02:07:16.887 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:16.888 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.888 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.888 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:16.888 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.888 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.888 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.888 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T02:07:16.888 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.888 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.889 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.889 INFO:teuthology.orchestra.run.smithi006.stdout: "default_cephadm_command_timeout": { 2024-04-26T02:07:16.889 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T02:07:16.889 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:16.889 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.889 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.889 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "900", 2024-04-26T02:07:16.889 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.889 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.890 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.890 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T02:07:16.890 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.890 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.890 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.890 INFO:teuthology.orchestra.run.smithi006.stdout: "default_registry": { 2024-04-26T02:07:16.890 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "default_registry", 2024-04-26T02:07:16.891 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.891 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.891 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.891 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "docker.io", 2024-04-26T02:07:16.891 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.891 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.891 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.891 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T02:07:16.891 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.891 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.892 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.892 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.892 INFO:teuthology.orchestra.run.smithi006.stdout: "device_cache_timeout": { 2024-04-26T02:07:16.892 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "device_cache_timeout", 2024-04-26T02:07:16.892 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:16.892 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.892 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.892 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "1800", 2024-04-26T02:07:16.892 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.893 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.893 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.893 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "seconds to cache device inventory", 2024-04-26T02:07:16.893 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.893 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.893 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.893 INFO:teuthology.orchestra.run.smithi006.stdout: "device_enhanced_scan": { 2024-04-26T02:07:16.893 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "device_enhanced_scan", 2024-04-26T02:07:16.893 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.894 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.894 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.894 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.894 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.894 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.894 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.894 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T02:07:16.894 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.895 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.895 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.895 INFO:teuthology.orchestra.run.smithi006.stdout: "facts_cache_timeout": { 2024-04-26T02:07:16.895 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "facts_cache_timeout", 2024-04-26T02:07:16.895 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:16.895 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.895 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.896 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:16.896 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.896 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.896 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.896 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "seconds to cache host facts data", 2024-04-26T02:07:16.896 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.896 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.896 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.896 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.897 INFO:teuthology.orchestra.run.smithi006.stdout: "grafana_dashboards_path": { 2024-04-26T02:07:16.897 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "grafana_dashboards_path", 2024-04-26T02:07:16.897 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.897 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.897 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.897 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T02:07:16.897 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.897 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.898 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.898 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T02:07:16.898 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.898 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.898 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.898 INFO:teuthology.orchestra.run.smithi006.stdout: "host_check_interval": { 2024-04-26T02:07:16.898 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "host_check_interval", 2024-04-26T02:07:16.899 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:16.899 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.899 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.899 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:16.899 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.899 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.899 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.899 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to perform a host check", 2024-04-26T02:07:16.899 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.900 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.900 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.900 INFO:teuthology.orchestra.run.smithi006.stdout: "hw_monitoring": { 2024-04-26T02:07:16.900 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "hw_monitoring", 2024-04-26T02:07:16.900 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.900 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.900 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.900 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.901 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.901 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.901 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.901 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T02:07:16.901 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.901 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.901 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.902 INFO:teuthology.orchestra.run.smithi006.stdout: "inventory_list_all": { 2024-04-26T02:07:16.902 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "inventory_list_all", 2024-04-26T02:07:16.902 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.902 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.902 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.902 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.902 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.902 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.902 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.903 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T02:07:16.903 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.903 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.903 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.903 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.903 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:16.903 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:16.903 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.903 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.903 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.904 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.904 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.904 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.904 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:16.904 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:16.904 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:16.904 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:16.904 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:16.904 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:16.905 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:16.905 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:16.905 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.905 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.905 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.905 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.905 INFO:teuthology.orchestra.run.smithi006.stdout: "log_refresh_metadata": { 2024-04-26T02:07:16.905 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_refresh_metadata", 2024-04-26T02:07:16.905 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.906 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.906 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.906 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.906 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.906 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.906 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.906 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T02:07:16.906 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.907 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.907 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.907 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:16.907 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:16.907 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.907 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.907 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.907 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.907 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.908 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.908 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.908 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T02:07:16.908 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.908 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.908 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.908 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.908 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:16.908 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:16.909 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.909 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.909 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.909 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:16.909 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.909 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.909 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:16.909 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:16.910 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:16.910 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:16.910 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:16.910 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:16.910 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:16.910 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:16.910 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.910 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.910 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.911 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.911 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:16.911 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:16.911 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.911 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.911 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.911 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.911 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.912 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.912 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.912 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.912 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.912 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.912 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.912 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.912 INFO:teuthology.orchestra.run.smithi006.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T02:07:16.912 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T02:07:16.912 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.913 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.913 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.913 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.913 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.913 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.913 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.913 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T02:07:16.913 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.913 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.914 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.914 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.914 INFO:teuthology.orchestra.run.smithi006.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T02:07:16.914 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T02:07:16.914 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.914 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.914 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.914 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "*", 2024-04-26T02:07:16.915 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.915 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.915 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.915 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T02:07:16.915 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.915 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.915 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.915 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.916 INFO:teuthology.orchestra.run.smithi006.stdout: "max_count_per_host": { 2024-04-26T02:07:16.916 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_count_per_host", 2024-04-26T02:07:16.916 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:16.916 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.916 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.916 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:16.916 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.916 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.916 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.917 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "max number of daemons per service per host", 2024-04-26T02:07:16.917 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.917 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.917 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.917 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.917 INFO:teuthology.orchestra.run.smithi006.stdout: "max_osd_draining_count": { 2024-04-26T02:07:16.917 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_osd_draining_count", 2024-04-26T02:07:16.917 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:16.917 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.918 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.918 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:16.918 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.918 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.918 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.918 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T02:07:16.918 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.918 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.919 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.919 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.919 INFO:teuthology.orchestra.run.smithi006.stdout: "migration_current": { 2024-04-26T02:07:16.919 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "migration_current", 2024-04-26T02:07:16.919 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:16.919 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.919 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.919 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.919 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.920 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.920 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.920 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "internal - do not modify", 2024-04-26T02:07:16.920 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.920 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.920 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.920 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.920 INFO:teuthology.orchestra.run.smithi006.stdout: "mode": { 2024-04-26T02:07:16.921 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mode", 2024-04-26T02:07:16.921 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.921 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.921 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.921 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "root", 2024-04-26T02:07:16.921 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.921 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.921 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:16.921 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm-package", 2024-04-26T02:07:16.922 INFO:teuthology.orchestra.run.smithi006.stdout: "root" 2024-04-26T02:07:16.922 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:16.922 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T02:07:16.922 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.922 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.922 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.922 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.923 INFO:teuthology.orchestra.run.smithi006.stdout: "oob_default_addr": { 2024-04-26T02:07:16.923 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "oob_default_addr", 2024-04-26T02:07:16.923 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.923 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.923 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.923 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "169.254.1.1", 2024-04-26T02:07:16.923 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.923 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.923 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.924 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T02:07:16.924 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.924 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.924 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.924 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.924 INFO:teuthology.orchestra.run.smithi006.stdout: "prometheus_alerts_path": { 2024-04-26T02:07:16.924 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "prometheus_alerts_path", 2024-04-26T02:07:16.924 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.924 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.925 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.925 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T02:07:16.925 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.925 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.925 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.925 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T02:07:16.925 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.925 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.925 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.926 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.926 INFO:teuthology.orchestra.run.smithi006.stdout: "registry_insecure": { 2024-04-26T02:07:16.926 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "registry_insecure", 2024-04-26T02:07:16.926 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.926 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.926 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.926 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.926 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.927 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.927 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.927 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T02:07:16.927 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.927 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.927 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.927 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.927 INFO:teuthology.orchestra.run.smithi006.stdout: "registry_password": { 2024-04-26T02:07:16.928 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "registry_password", 2024-04-26T02:07:16.928 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.928 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.928 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.928 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.928 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.928 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.928 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.928 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T02:07:16.928 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.929 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.929 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.929 INFO:teuthology.orchestra.run.smithi006.stdout: "registry_url": { 2024-04-26T02:07:16.929 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "registry_url", 2024-04-26T02:07:16.929 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.929 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.929 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.929 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.930 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.930 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.930 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.930 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T02:07:16.930 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.930 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.930 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.930 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.931 INFO:teuthology.orchestra.run.smithi006.stdout: "registry_username": { 2024-04-26T02:07:16.931 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "registry_username", 2024-04-26T02:07:16.931 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.931 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.931 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.931 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.931 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.931 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.931 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.931 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T02:07:16.932 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.932 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.932 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.932 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.932 INFO:teuthology.orchestra.run.smithi006.stdout: "secure_monitoring_stack": { 2024-04-26T02:07:16.932 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "secure_monitoring_stack", 2024-04-26T02:07:16.932 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.932 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.932 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.933 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.933 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.933 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.933 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.933 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T02:07:16.933 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.933 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.933 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.933 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.934 INFO:teuthology.orchestra.run.smithi006.stdout: "service_discovery_port": { 2024-04-26T02:07:16.934 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "service_discovery_port", 2024-04-26T02:07:16.934 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:16.934 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.934 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.934 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8765", 2024-04-26T02:07:16.934 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.934 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.934 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.934 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "cephadm service discovery port", 2024-04-26T02:07:16.935 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.935 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.935 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.935 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.935 INFO:teuthology.orchestra.run.smithi006.stdout: "ssh_config_file": { 2024-04-26T02:07:16.935 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ssh_config_file", 2024-04-26T02:07:16.935 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.935 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.935 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.936 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.936 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.936 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.936 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.936 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T02:07:16.936 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.936 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.936 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.936 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.936 INFO:teuthology.orchestra.run.smithi006.stdout: "use_agent": { 2024-04-26T02:07:16.937 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "use_agent", 2024-04-26T02:07:16.937 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.937 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.937 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.937 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.937 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.937 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.937 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.937 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T02:07:16.937 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.938 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.938 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.938 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.938 INFO:teuthology.orchestra.run.smithi006.stdout: "use_repo_digest": { 2024-04-26T02:07:16.938 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "use_repo_digest", 2024-04-26T02:07:16.938 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.938 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.939 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.939 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.939 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.939 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.939 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.939 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T02:07:16.939 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.939 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.939 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.940 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.940 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_on_failed_host_check": { 2024-04-26T02:07:16.940 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_on_failed_host_check", 2024-04-26T02:07:16.940 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.940 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.940 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.940 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.940 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.941 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.941 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.941 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T02:07:16.941 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.941 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.941 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.941 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.941 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_on_stray_daemons": { 2024-04-26T02:07:16.941 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_on_stray_daemons", 2024-04-26T02:07:16.942 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.942 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.942 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.942 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.942 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.942 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.942 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.942 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T02:07:16.943 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.943 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.943 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.943 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.943 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_on_stray_hosts": { 2024-04-26T02:07:16.943 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_on_stray_hosts", 2024-04-26T02:07:16.943 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.943 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.944 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.944 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.944 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.944 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.944 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.944 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T02:07:16.944 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.944 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.944 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.945 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:16.945 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:16.945 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.945 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:16.945 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crash", 2024-04-26T02:07:16.945 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:16.945 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:16.945 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:16.946 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:16.946 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:16.946 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.946 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.946 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.946 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.946 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.946 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.947 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:16.947 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:16.947 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:16.947 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:16.947 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:16.947 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:16.947 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:16.947 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:16.948 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.948 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.948 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.948 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.948 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.948 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:16.948 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:16.948 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.948 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.949 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.949 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.949 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.949 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.949 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.949 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.949 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.949 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.949 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.950 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.950 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:16.950 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:16.950 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.950 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.950 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.950 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:16.950 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.950 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.951 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:16.951 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:16.951 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:16.951 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:16.951 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:16.951 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:16.951 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:16.951 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:16.952 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.952 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.952 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.952 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.952 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.952 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:16.952 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:16.952 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.952 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.952 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.953 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.953 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.953 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.953 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.953 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.953 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.953 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.953 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.954 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.954 INFO:teuthology.orchestra.run.smithi006.stdout: "retain_interval": { 2024-04-26T02:07:16.954 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "retain_interval", 2024-04-26T02:07:16.954 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:16.954 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.954 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.954 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "31536000", 2024-04-26T02:07:16.954 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.954 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.955 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.955 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T02:07:16.955 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.955 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.955 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.955 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_recent_interval": { 2024-04-26T02:07:16.955 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_recent_interval", 2024-04-26T02:07:16.956 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:16.956 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.956 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:16.956 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "1209600", 2024-04-26T02:07:16.956 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.956 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.956 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.956 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T02:07:16.956 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.957 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.957 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.957 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:16.957 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:16.957 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:16.957 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "dashboard", 2024-04-26T02:07:16.957 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:16.958 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:16.958 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:16.958 INFO:teuthology.orchestra.run.smithi006.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T02:07:16.958 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T02:07:16.958 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:16.958 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.958 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.959 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:16.959 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.959 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.959 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.959 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.959 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.959 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.959 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.960 INFO:teuthology.orchestra.run.smithi006.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T02:07:16.960 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T02:07:16.960 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.960 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.960 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.960 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.960 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.961 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.961 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.961 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.961 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.961 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.961 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.961 INFO:teuthology.orchestra.run.smithi006.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T02:07:16.962 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T02:07:16.962 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.962 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.962 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.962 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.962 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.962 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.962 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.963 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.963 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.963 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.963 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.963 INFO:teuthology.orchestra.run.smithi006.stdout: "AUDIT_API_ENABLED": { 2024-04-26T02:07:16.963 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T02:07:16.963 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.963 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.964 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.964 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.964 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.964 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.964 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.964 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.964 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.964 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.964 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.965 INFO:teuthology.orchestra.run.smithi006.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T02:07:16.965 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T02:07:16.965 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.965 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.965 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.965 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.965 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.965 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.965 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.966 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.966 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.966 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.966 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.966 INFO:teuthology.orchestra.run.smithi006.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T02:07:16.966 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T02:07:16.966 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.966 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.967 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.967 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.967 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.967 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.967 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.967 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.967 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.967 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.967 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.968 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T02:07:16.968 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T02:07:16.968 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.968 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.968 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.968 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.968 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.968 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.969 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.969 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.969 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.969 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.969 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.969 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T02:07:16.969 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T02:07:16.969 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.969 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.970 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.970 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.970 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.970 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.970 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.970 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.970 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.970 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.971 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.971 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.971 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T02:07:16.971 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T02:07:16.971 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.971 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.971 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.971 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.971 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.972 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.972 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.972 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.972 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.972 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.972 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.972 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T02:07:16.972 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T02:07:16.972 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.973 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.973 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.973 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.973 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.973 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.973 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.973 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.973 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.973 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.974 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.974 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T02:07:16.974 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T02:07:16.974 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.974 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.974 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.974 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.974 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.975 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.975 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.975 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.975 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.975 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.975 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.975 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.975 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T02:07:16.975 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T02:07:16.976 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.976 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.976 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.976 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.976 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.976 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.976 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.976 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.976 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.977 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.977 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.977 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.977 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T02:07:16.977 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T02:07:16.977 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.977 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.977 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.977 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.978 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.978 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.978 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.978 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.978 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.978 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.978 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.979 INFO:teuthology.orchestra.run.smithi006.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T02:07:16.979 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T02:07:16.979 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.979 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.979 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.979 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.979 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.979 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.980 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.980 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.980 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.980 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.980 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.980 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T02:07:16.980 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T02:07:16.980 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.981 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.981 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.981 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "admin", 2024-04-26T02:07:16.981 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.981 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.981 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.981 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.981 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.981 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.982 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.982 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.982 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T02:07:16.982 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T02:07:16.982 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.982 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.982 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.982 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.982 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.983 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.983 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.983 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.983 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.983 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.983 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.983 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.983 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_API_URL": { 2024-04-26T02:07:16.983 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_API_URL", 2024-04-26T02:07:16.983 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.984 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.984 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.984 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.984 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.984 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.984 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.984 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.984 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.984 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.984 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.985 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T02:07:16.985 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T02:07:16.985 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.985 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.985 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.985 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "admin", 2024-04-26T02:07:16.985 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.985 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.986 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.986 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.986 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.986 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.986 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.986 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.986 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T02:07:16.986 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T02:07:16.986 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.987 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.987 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.987 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.987 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.987 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.987 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.987 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.987 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.987 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.988 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.988 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T02:07:16.988 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T02:07:16.988 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.988 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.988 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.988 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:16.989 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.989 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.989 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.989 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.989 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.989 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.989 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.989 INFO:teuthology.orchestra.run.smithi006.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T02:07:16.990 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T02:07:16.990 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.990 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.990 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.990 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.990 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.990 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.990 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.990 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.991 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.991 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.991 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.991 INFO:teuthology.orchestra.run.smithi006.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T02:07:16.991 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T02:07:16.991 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.991 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.992 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.992 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.992 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.992 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.992 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.992 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.992 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.992 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.992 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.993 INFO:teuthology.orchestra.run.smithi006.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T02:07:16.993 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T02:07:16.993 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.993 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.993 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.993 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "[]", 2024-04-26T02:07:16.993 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.994 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.994 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.994 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.994 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.994 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.994 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.994 INFO:teuthology.orchestra.run.smithi006.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T02:07:16.995 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T02:07:16.995 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.995 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.995 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.995 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "{}", 2024-04-26T02:07:16.995 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.995 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.995 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.995 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.996 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.996 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.996 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.996 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.996 INFO:teuthology.orchestra.run.smithi006.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T02:07:16.996 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T02:07:16.996 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:16.996 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.996 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.997 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:16.997 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.997 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.997 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.997 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.997 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.997 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.997 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.998 INFO:teuthology.orchestra.run.smithi006.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T02:07:16.998 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T02:07:16.998 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:16.998 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:16.998 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:16.998 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:16.998 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:16.998 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:16.999 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:16.999 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:16.999 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:16.999 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:16.999 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:16.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:16.999 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T02:07:16.999 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T02:07:16.999 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.000 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.000 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.000 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.000 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.000 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.000 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.000 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.000 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.000 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.000 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.001 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T02:07:17.001 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T02:07:17.001 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.001 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.001 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.001 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.001 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.001 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.002 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.002 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.002 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.002 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.002 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.002 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T02:07:17.002 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T02:07:17.003 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.003 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.003 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.003 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.003 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.003 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.003 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.003 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.003 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.004 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.004 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.004 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T02:07:17.004 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T02:07:17.004 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.004 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.004 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.005 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.005 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.005 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.005 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.005 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.005 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.005 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.005 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.006 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T02:07:17.006 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T02:07:17.006 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.006 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.006 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.006 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.006 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.006 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.007 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.007 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.007 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.007 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.007 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.007 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T02:07:17.007 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T02:07:17.008 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.008 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.008 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.008 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.008 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.008 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.008 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.008 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.009 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.009 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.009 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.009 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T02:07:17.009 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T02:07:17.009 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.009 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.010 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.010 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.010 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.010 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.010 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.010 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.010 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.010 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.011 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.011 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T02:07:17.011 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T02:07:17.011 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.011 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.011 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.011 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.012 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.012 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.012 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.012 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.012 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.012 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.012 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.012 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T02:07:17.013 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T02:07:17.013 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.013 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.013 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.013 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T02:07:17.013 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.013 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.013 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.014 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.014 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.014 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.014 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.014 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T02:07:17.014 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T02:07:17.014 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.014 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.015 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.015 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:17.015 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.015 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.015 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.015 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.015 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.015 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.016 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.016 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T02:07:17.016 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T02:07:17.016 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.016 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.016 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.016 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8", 2024-04-26T02:07:17.016 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.017 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.017 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.017 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.017 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.017 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.017 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.017 INFO:teuthology.orchestra.run.smithi006.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T02:07:17.017 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T02:07:17.017 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.018 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.018 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.018 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "45", 2024-04-26T02:07:17.018 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.018 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.018 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.018 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.018 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.019 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.019 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.019 INFO:teuthology.orchestra.run.smithi006.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T02:07:17.019 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T02:07:17.019 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.019 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.019 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.019 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.020 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.020 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.020 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.020 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.020 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.020 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.020 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.021 INFO:teuthology.orchestra.run.smithi006.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T02:07:17.021 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T02:07:17.021 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.021 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.021 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.021 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "admin", 2024-04-26T02:07:17.021 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.021 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.022 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.022 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.022 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.022 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.022 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.022 INFO:teuthology.orchestra.run.smithi006.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T02:07:17.022 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T02:07:17.022 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.023 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.023 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.023 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.023 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.023 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.023 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.023 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.023 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.023 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.024 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.024 INFO:teuthology.orchestra.run.smithi006.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T02:07:17.024 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T02:07:17.024 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.024 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.024 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.024 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.024 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T02:07:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T02:07:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T02:07:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T02:07:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T02:07:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T02:07:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T02:07:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T02:07:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5", 2024-04-26T02:07:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "cross_origin_url": { 2024-04-26T02:07:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cross_origin_url", 2024-04-26T02:07:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "crt_file": { 2024-04-26T02:07:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crt_file", 2024-04-26T02:07:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "debug": { 2024-04-26T02:07:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "debug", 2024-04-26T02:07:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Enable/disable debug options", 2024-04-26T02:07:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "jwt_token_ttl": { 2024-04-26T02:07:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "jwt_token_ttl", 2024-04-26T02:07:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "28800", 2024-04-26T02:07:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "key_file": { 2024-04-26T02:07:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "key_file", 2024-04-26T02:07:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "motd": { 2024-04-26T02:07:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "motd", 2024-04-26T02:07:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "The message of the day", 2024-04-26T02:07:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "redirect_resolve_ip_addr": { 2024-04-26T02:07:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T02:07:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "server_addr": { 2024-04-26T02:07:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_addr", 2024-04-26T02:07:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "::", 2024-04-26T02:07:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "server_port": { 2024-04-26T02:07:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_port", 2024-04-26T02:07:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8080", 2024-04-26T02:07:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "ssl": { 2024-04-26T02:07:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ssl", 2024-04-26T02:07:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "ssl_server_port": { 2024-04-26T02:07:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ssl_server_port", 2024-04-26T02:07:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8443", 2024-04-26T02:07:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "standby_behaviour": { 2024-04-26T02:07:17.057 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "standby_behaviour", 2024-04-26T02:07:17.057 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.057 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.057 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.057 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "redirect", 2024-04-26T02:07:17.057 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.057 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.057 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.057 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.058 INFO:teuthology.orchestra.run.smithi006.stdout: "redirect" 2024-04-26T02:07:17.058 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.058 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.058 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.058 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.058 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.058 INFO:teuthology.orchestra.run.smithi006.stdout: "standby_error_status_code": { 2024-04-26T02:07:17.059 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "standby_error_status_code", 2024-04-26T02:07:17.059 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.059 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.059 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.059 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "500", 2024-04-26T02:07:17.059 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "400", 2024-04-26T02:07:17.059 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "599", 2024-04-26T02:07:17.059 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi006.stdout: "url_prefix": { 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "url_prefix", 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.061 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.061 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.061 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.061 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.061 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.061 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.061 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.061 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.062 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.062 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.062 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.062 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.062 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "devicehealth", 2024-04-26T02:07:17.062 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi006.stdout: "enable_monitoring": { 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "enable_monitoring", 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "monitor device health metrics", 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.066 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.066 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.066 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.066 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.066 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.066 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.066 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.066 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.067 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.067 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.067 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.067 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.067 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.067 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.067 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.068 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.068 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.068 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.068 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.068 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.068 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.068 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.068 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.069 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.069 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.069 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.069 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.069 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.069 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.069 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.070 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.070 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.070 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.070 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.070 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.070 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.070 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.070 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.071 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.071 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.071 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.071 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.071 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.071 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.071 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.071 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.072 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.072 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.072 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.072 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.072 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.072 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.072 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.072 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.073 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.073 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.073 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.073 INFO:teuthology.orchestra.run.smithi006.stdout: "mark_out_threshold": { 2024-04-26T02:07:17.073 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mark_out_threshold", 2024-04-26T02:07:17.073 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.073 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.073 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.074 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "2419200", 2024-04-26T02:07:17.074 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.074 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.074 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.074 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T02:07:17.074 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.074 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.075 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.075 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": { 2024-04-26T02:07:17.075 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "pool_name", 2024-04-26T02:07:17.075 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.075 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.075 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.075 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "device_health_metrics", 2024-04-26T02:07:17.076 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.076 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.076 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.076 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T02:07:17.076 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.076 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.076 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.077 INFO:teuthology.orchestra.run.smithi006.stdout: "retention_period": { 2024-04-26T02:07:17.077 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "retention_period", 2024-04-26T02:07:17.077 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.077 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.077 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.077 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "15552000", 2024-04-26T02:07:17.077 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.077 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.078 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.078 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how long to retain device health metrics", 2024-04-26T02:07:17.078 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.078 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.078 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.078 INFO:teuthology.orchestra.run.smithi006.stdout: "scrape_frequency": { 2024-04-26T02:07:17.079 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "scrape_frequency", 2024-04-26T02:07:17.079 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.079 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.079 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.079 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "86400", 2024-04-26T02:07:17.079 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.079 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.079 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.079 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T02:07:17.080 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.080 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.080 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.080 INFO:teuthology.orchestra.run.smithi006.stdout: "self_heal": { 2024-04-26T02:07:17.080 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "self_heal", 2024-04-26T02:07:17.080 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.080 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.080 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.080 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.081 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.081 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.081 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.081 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T02:07:17.081 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.081 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.081 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.081 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:17.082 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:17.082 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.082 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.082 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.082 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:17.082 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.082 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.082 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.082 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T02:07:17.082 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.083 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.083 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.083 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_threshold": { 2024-04-26T02:07:17.083 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_threshold", 2024-04-26T02:07:17.083 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.083 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.083 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.083 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "7257600", 2024-04-26T02:07:17.084 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.084 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.084 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.084 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T02:07:17.084 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.084 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.084 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.084 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.084 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.085 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "diskprediction_local", 2024-04-26T02:07:17.085 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.085 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.085 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.085 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.085 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.085 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.085 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.085 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.086 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.086 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.086 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.086 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.086 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.086 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.086 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.086 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.086 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.086 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.087 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.087 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.087 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.087 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.087 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.087 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.087 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.087 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.087 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.088 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.088 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.088 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.088 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.088 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.088 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.088 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.089 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.089 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.089 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.089 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.089 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.089 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.089 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.089 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.089 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.090 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.090 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.090 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.090 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.090 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.090 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.090 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.090 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.091 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.091 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.091 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.091 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.091 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.091 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.091 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.092 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.092 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.092 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.092 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.092 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.092 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.092 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.092 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.093 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.093 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.093 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.093 INFO:teuthology.orchestra.run.smithi006.stdout: "predict_interval": { 2024-04-26T02:07:17.093 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "predict_interval", 2024-04-26T02:07:17.093 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.093 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.093 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.094 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "86400", 2024-04-26T02:07:17.094 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.094 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.094 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.094 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.094 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.094 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.094 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.095 INFO:teuthology.orchestra.run.smithi006.stdout: "predictor_model": { 2024-04-26T02:07:17.095 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "predictor_model", 2024-04-26T02:07:17.095 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.095 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.095 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.095 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "prophetstor", 2024-04-26T02:07:17.095 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.095 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.095 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.096 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.096 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.096 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.096 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.096 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:17.096 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:17.096 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.096 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.096 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.097 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:17.097 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.097 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.097 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.097 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.097 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.097 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.097 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.097 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.098 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.098 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.098 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "influx", 2024-04-26T02:07:17.098 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": false, 2024-04-26T02:07:17.098 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "influxdb python module not found", 2024-04-26T02:07:17.098 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.098 INFO:teuthology.orchestra.run.smithi006.stdout: "batch_size": { 2024-04-26T02:07:17.098 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "batch_size", 2024-04-26T02:07:17.099 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.099 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.099 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.099 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5000", 2024-04-26T02:07:17.099 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.099 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.099 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.099 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T02:07:17.099 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.100 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.100 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.100 INFO:teuthology.orchestra.run.smithi006.stdout: "database": { 2024-04-26T02:07:17.100 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "database", 2024-04-26T02:07:17.100 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.100 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.100 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.101 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "ceph", 2024-04-26T02:07:17.101 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.101 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.101 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.101 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T02:07:17.101 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.101 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.101 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.102 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": { 2024-04-26T02:07:17.102 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "hostname", 2024-04-26T02:07:17.102 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.102 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.102 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.102 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.102 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.102 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.103 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.103 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "InfluxDB server hostname", 2024-04-26T02:07:17.103 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.103 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.103 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.103 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:17.103 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:17.104 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.104 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.104 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.104 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "30", 2024-04-26T02:07:17.104 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "5", 2024-04-26T02:07:17.104 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.104 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.104 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T02:07:17.104 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.104 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "password": { 2024-04-26T02:07:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "password", 2024-04-26T02:07:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "password of InfluxDB server user", 2024-04-26T02:07:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "port": { 2024-04-26T02:07:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "port", 2024-04-26T02:07:17.114 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8086", 2024-04-26T02:07:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "InfluxDB server port", 2024-04-26T02:07:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "ssl": { 2024-04-26T02:07:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ssl", 2024-04-26T02:07:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "false", 2024-04-26T02:07:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T02:07:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "threads": { 2024-04-26T02:07:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "threads", 2024-04-26T02:07:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5", 2024-04-26T02:07:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1", 2024-04-26T02:07:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "32", 2024-04-26T02:07:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T02:07:17.119 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.119 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.119 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.119 INFO:teuthology.orchestra.run.smithi006.stdout: "username": { 2024-04-26T02:07:17.119 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "username", 2024-04-26T02:07:17.119 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.119 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.119 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "username of InfluxDB server user", 2024-04-26T02:07:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "verify_ssl": { 2024-04-26T02:07:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "verify_ssl", 2024-04-26T02:07:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "true", 2024-04-26T02:07:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.121 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T02:07:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "insights", 2024-04-26T02:07:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.124 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.125 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.126 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.126 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.126 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.126 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.126 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.126 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.126 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.126 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.127 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.127 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.127 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.127 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.127 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.127 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.127 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.127 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.127 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.127 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.128 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.128 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.128 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.128 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.128 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.128 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.128 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.128 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.128 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.128 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.129 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.129 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.129 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.129 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.129 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.129 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.129 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.129 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.129 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.130 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.130 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.130 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.130 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.130 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.130 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.131 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "iostat", 2024-04-26T02:07:17.131 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.131 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.131 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.131 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.131 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.131 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.131 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.132 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.132 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.132 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.132 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.132 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.132 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.132 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.132 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.132 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.132 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.133 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.133 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.133 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.133 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.133 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.133 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.133 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.133 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.134 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.134 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.134 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.134 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.134 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.134 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.134 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.134 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.135 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.135 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.135 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.135 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.135 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.135 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.135 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.135 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.136 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.136 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.136 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.136 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.136 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.136 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.136 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.136 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.136 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.137 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.137 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.137 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.137 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.137 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.137 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.137 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.137 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.138 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.138 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.138 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.138 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.138 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.138 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.138 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.138 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.139 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.139 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.139 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.139 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.139 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.139 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "k8sevents", 2024-04-26T02:07:17.140 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.140 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.140 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.140 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_event_retention_days": { 2024-04-26T02:07:17.140 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ceph_event_retention_days", 2024-04-26T02:07:17.140 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.140 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.140 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.141 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "7", 2024-04-26T02:07:17.141 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.141 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.141 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.141 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T02:07:17.141 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.141 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.141 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.142 INFO:teuthology.orchestra.run.smithi006.stdout: "config_check_secs": { 2024-04-26T02:07:17.142 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "config_check_secs", 2024-04-26T02:07:17.142 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.142 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.142 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.142 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:17.142 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "10", 2024-04-26T02:07:17.143 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.143 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.143 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T02:07:17.143 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.143 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.143 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.143 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.144 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.144 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.144 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.144 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.144 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.144 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.144 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.144 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.144 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.144 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.145 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.145 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.145 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.145 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.145 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.145 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.145 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.145 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.145 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.146 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.146 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.146 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.146 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.146 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.146 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.146 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.147 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.147 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.147 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.147 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.147 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.147 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.147 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.147 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.148 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.148 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.148 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.148 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.148 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.148 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.148 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.148 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.148 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.148 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.149 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.149 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.149 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.149 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.149 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.149 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.149 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.149 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.149 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.150 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.150 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.150 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.150 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.150 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.150 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.150 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.150 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.151 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.151 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.151 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.151 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.151 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.151 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.152 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "localpool", 2024-04-26T02:07:17.152 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.152 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.152 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.152 INFO:teuthology.orchestra.run.smithi006.stdout: "failure_domain": { 2024-04-26T02:07:17.152 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "failure_domain", 2024-04-26T02:07:17.152 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.152 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.153 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.153 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "host", 2024-04-26T02:07:17.153 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.153 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.153 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.153 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "failure domain for any created local pool", 2024-04-26T02:07:17.153 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T02:07:17.153 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.153 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.154 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.154 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.154 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.154 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.154 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.154 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.154 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.154 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.155 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.155 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.155 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.155 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.155 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.155 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.155 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.155 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.156 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.156 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.156 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.156 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.156 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.156 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.156 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.157 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.157 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.157 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.157 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.157 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.157 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.157 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.157 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.157 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.158 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.158 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.158 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.158 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.158 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.158 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.158 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.158 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.159 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.159 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.159 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.159 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.159 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.159 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.159 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.159 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.159 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.160 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.160 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.160 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.160 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.160 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.160 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.160 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.161 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.161 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.161 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.161 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.161 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.161 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.161 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.161 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.161 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.162 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.162 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.162 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.162 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.162 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": { 2024-04-26T02:07:17.162 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "min_size", 2024-04-26T02:07:17.162 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.162 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.163 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.163 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.163 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.163 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.163 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.163 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "default min_size for any created local pool", 2024-04-26T02:07:17.163 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T02:07:17.163 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.163 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.164 INFO:teuthology.orchestra.run.smithi006.stdout: "num_rep": { 2024-04-26T02:07:17.164 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "num_rep", 2024-04-26T02:07:17.164 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.164 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.164 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.164 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "3", 2024-04-26T02:07:17.164 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.165 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.165 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.165 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "default replica count for any created local pool", 2024-04-26T02:07:17.165 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.165 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.165 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.165 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": { 2024-04-26T02:07:17.165 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "pg_num", 2024-04-26T02:07:17.165 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.166 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.166 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.166 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "128", 2024-04-26T02:07:17.166 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.166 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.166 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.166 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T02:07:17.166 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.166 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.167 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.167 INFO:teuthology.orchestra.run.smithi006.stdout: "prefix": { 2024-04-26T02:07:17.167 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "prefix", 2024-04-26T02:07:17.167 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.167 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.167 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.167 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.168 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.168 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.168 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.168 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "name prefix for any created local pool", 2024-04-26T02:07:17.168 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.168 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.168 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.168 INFO:teuthology.orchestra.run.smithi006.stdout: "subtree": { 2024-04-26T02:07:17.168 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "subtree", 2024-04-26T02:07:17.169 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.169 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.169 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.169 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "rack", 2024-04-26T02:07:17.169 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.169 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.169 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.169 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T02:07:17.169 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T02:07:17.169 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.170 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.170 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.170 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.170 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mds_autoscaler", 2024-04-26T02:07:17.170 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.170 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.171 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.171 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.171 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.171 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.171 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.171 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.171 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.171 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.171 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.172 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.172 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.172 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.172 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.172 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.172 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.172 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.172 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.172 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.173 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.173 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.173 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.173 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.173 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.173 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.173 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.173 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.174 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.174 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.174 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.174 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.174 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.174 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.174 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.174 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.175 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.175 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.175 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.175 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.175 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.175 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.175 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.175 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.176 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.176 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.176 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.176 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.176 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.176 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.176 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.176 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.176 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.177 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.177 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.177 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.177 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.177 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.177 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.177 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.177 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.178 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.178 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.178 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.178 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.178 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.178 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.178 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.178 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.179 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.179 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.179 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mirroring", 2024-04-26T02:07:17.179 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.179 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.179 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.179 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.179 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.180 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.180 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.180 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.180 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.180 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.180 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.180 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.180 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.181 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.181 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.181 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.181 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.181 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.181 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.181 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.181 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.181 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.182 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.182 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.182 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.182 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.182 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.182 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.182 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.183 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.183 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.183 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.183 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.183 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.183 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.183 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.183 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.184 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.184 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.184 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.184 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.184 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.184 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.184 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.184 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.185 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.185 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.185 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.185 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.185 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.185 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.185 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.185 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.186 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.186 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.186 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.186 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.186 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.186 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.186 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.186 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.187 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.187 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.187 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.187 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.187 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.187 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.187 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.187 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.187 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.188 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.188 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.188 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.188 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "nfs", 2024-04-26T02:07:17.188 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.188 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.188 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.188 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.188 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.189 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.189 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.189 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.189 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.189 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.189 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.189 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.189 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.189 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.190 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.190 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.190 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.190 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.190 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.190 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.190 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.190 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.191 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.191 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.191 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.191 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.191 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.191 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.191 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.191 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.191 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.192 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.192 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.192 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.193 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.193 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.193 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.193 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.194 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.194 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.194 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.194 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.194 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.194 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.194 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.194 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.195 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.195 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.195 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.195 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.195 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.195 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.195 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.195 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "orchestrator", 2024-04-26T02:07:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "fail_fs": { 2024-04-26T02:07:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "fail_fs", 2024-04-26T02:07:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T02:07:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.207 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.207 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.207 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.207 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.207 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.207 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.207 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.207 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.207 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.208 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.208 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.208 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.208 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.208 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.208 INFO:teuthology.orchestra.run.smithi006.stdout: "orchestrator": { 2024-04-26T02:07:17.208 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "orchestrator", 2024-04-26T02:07:17.208 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.208 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.208 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.209 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.209 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.209 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.209 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.209 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm", 2024-04-26T02:07:17.209 INFO:teuthology.orchestra.run.smithi006.stdout: "rook", 2024-04-26T02:07:17.209 INFO:teuthology.orchestra.run.smithi006.stdout: "test_orchestrator" 2024-04-26T02:07:17.209 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.209 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Orchestrator backend", 2024-04-26T02:07:17.210 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.210 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.210 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.210 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.210 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.210 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.210 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.211 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "osd_perf_query", 2024-04-26T02:07:17.211 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.211 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.211 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.211 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.211 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.211 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.211 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.212 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.212 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.212 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.212 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.212 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.212 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.212 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.212 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.212 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.213 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.213 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.213 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.213 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.213 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.213 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.213 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.213 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.213 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.214 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.214 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.214 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.214 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.214 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.214 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.214 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.215 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.215 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.215 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.215 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.215 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.215 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.215 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.215 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.216 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.216 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.216 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.216 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.216 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.216 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.216 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.216 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.216 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.217 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.217 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.217 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.217 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.217 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.217 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.217 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.217 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.218 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.218 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.218 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.218 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.218 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.218 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.218 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.218 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.219 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.219 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.219 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.219 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.219 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.219 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.219 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.219 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.219 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.220 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.220 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.220 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "osd_support", 2024-04-26T02:07:17.220 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.220 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.220 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.220 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.220 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.221 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.221 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.221 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.221 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.221 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.221 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.221 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.221 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.222 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.222 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.222 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.222 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.222 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.222 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.222 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.222 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.223 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.223 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.223 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.223 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.223 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.223 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.223 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.223 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.223 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.224 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.224 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.224 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.224 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.224 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.224 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.224 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.224 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.224 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.225 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.225 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.225 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.225 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.225 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.225 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.225 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.225 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.226 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.226 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.226 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.226 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.226 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.226 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.226 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.226 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.227 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.227 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.227 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.227 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.227 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.227 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.227 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.227 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.227 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.228 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.228 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.228 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.228 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.228 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.228 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.228 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.228 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.229 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.229 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.229 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.229 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.229 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "pg_autoscaler", 2024-04-26T02:07:17.229 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.229 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.229 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.229 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.230 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.230 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.230 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.230 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.230 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.230 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.230 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.230 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.231 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.231 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.231 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.231 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.231 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.231 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.231 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.232 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.232 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.232 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.232 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.232 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.232 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.232 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.232 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.232 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.233 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.233 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.233 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.233 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.233 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.233 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.233 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.233 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.233 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.234 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.234 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.234 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.234 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.234 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.234 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.234 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.234 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.234 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.235 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.235 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.235 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.235 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.235 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.235 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.235 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.235 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.235 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.236 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.236 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.236 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.236 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.236 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.236 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.236 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.236 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.236 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.237 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.237 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.237 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.237 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.237 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.237 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.237 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.237 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.237 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.238 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:17.238 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:17.238 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.238 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.238 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.238 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:17.238 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.238 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.239 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.239 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.239 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.239 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.239 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.239 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.239 INFO:teuthology.orchestra.run.smithi006.stdout: "threshold": { 2024-04-26T02:07:17.239 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "threshold", 2024-04-26T02:07:17.239 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:17.239 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.240 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.240 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "3.0", 2024-04-26T02:07:17.240 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1.0", 2024-04-26T02:07:17.240 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.240 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.240 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "scaling threshold", 2024-04-26T02:07:17.240 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T02:07:17.240 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.241 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.241 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.241 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.241 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.241 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.241 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "progress", 2024-04-26T02:07:17.241 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.241 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.241 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.241 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_pg_recovery_event": { 2024-04-26T02:07:17.242 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "allow_pg_recovery_event", 2024-04-26T02:07:17.242 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.242 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.242 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.242 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.242 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.242 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.242 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.243 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T02:07:17.243 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.243 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.243 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.243 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.243 INFO:teuthology.orchestra.run.smithi006.stdout: "enabled": { 2024-04-26T02:07:17.243 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "enabled", 2024-04-26T02:07:17.243 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.243 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.244 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.244 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.244 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.244 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.244 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.244 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.244 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.244 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.245 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.245 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.245 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.245 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.245 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.245 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.245 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.245 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.245 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.246 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.246 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.246 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.246 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.246 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.246 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.246 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.246 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.246 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.247 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.247 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.247 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.247 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.247 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.247 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.247 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.247 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.247 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.248 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.248 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.248 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.248 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.248 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.248 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.248 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.248 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.248 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.248 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.249 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.249 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.249 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.249 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.249 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.249 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.249 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.249 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.249 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.250 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.250 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.250 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.250 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.250 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.250 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.250 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.250 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.250 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.251 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.251 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.251 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.251 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.251 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.251 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.251 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.251 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.252 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.252 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.252 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.252 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.252 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.252 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.252 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.252 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.252 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.253 INFO:teuthology.orchestra.run.smithi006.stdout: "max_completed_events": { 2024-04-26T02:07:17.253 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_completed_events", 2024-04-26T02:07:17.253 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.253 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.253 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.253 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "50", 2024-04-26T02:07:17.253 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.253 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.253 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.253 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "number of past completed events to remember", 2024-04-26T02:07:17.254 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.254 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.254 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.254 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.254 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:17.254 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:17.254 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.254 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.255 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.255 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5", 2024-04-26T02:07:17.255 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.255 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.255 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.255 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how long the module is going to sleep", 2024-04-26T02:07:17.255 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.255 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.255 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.256 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.256 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.256 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.256 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.256 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "prometheus", 2024-04-26T02:07:17.256 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.256 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.256 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.256 INFO:teuthology.orchestra.run.smithi006.stdout: "cache": { 2024-04-26T02:07:17.257 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cache", 2024-04-26T02:07:17.257 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.257 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.257 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.257 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.257 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.257 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.257 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.258 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.258 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.258 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.258 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.258 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.258 INFO:teuthology.orchestra.run.smithi006.stdout: "exclude_perf_counters": { 2024-04-26T02:07:17.258 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "exclude_perf_counters", 2024-04-26T02:07:17.258 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.258 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.259 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.259 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.259 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.259 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.259 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.259 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T02:07:17.259 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T02:07:17.259 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.260 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.260 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.260 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.260 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.260 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.260 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.260 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.260 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.261 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.261 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.261 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.261 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.261 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.261 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.261 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.261 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.262 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.262 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.262 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.262 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.262 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.262 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.262 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.262 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.263 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.263 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.263 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.263 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.263 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.263 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.263 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.263 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.263 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.264 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.264 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.264 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.264 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.264 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.264 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.264 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.264 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.265 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.265 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.265 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.265 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.265 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.265 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.265 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.265 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.266 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.266 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.266 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.266 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.266 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.266 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.266 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.266 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.267 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.267 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.267 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.267 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.267 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.267 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.267 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.267 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.268 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.268 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.268 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.268 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.268 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.268 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.268 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_stats_pools": { 2024-04-26T02:07:17.268 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rbd_stats_pools", 2024-04-26T02:07:17.269 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.269 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.269 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.269 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.269 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.269 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.269 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.269 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.269 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.270 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.270 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.270 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.270 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T02:07:17.270 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T02:07:17.270 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.270 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.271 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.271 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "300", 2024-04-26T02:07:17.271 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.271 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.271 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.271 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.271 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.271 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.271 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.271 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.272 INFO:teuthology.orchestra.run.smithi006.stdout: "scrape_interval": { 2024-04-26T02:07:17.272 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "scrape_interval", 2024-04-26T02:07:17.272 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:17.272 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.272 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.272 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "15.0", 2024-04-26T02:07:17.272 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.272 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.272 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.273 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.273 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.273 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.273 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.273 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.273 INFO:teuthology.orchestra.run.smithi006.stdout: "server_addr": { 2024-04-26T02:07:17.273 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_addr", 2024-04-26T02:07:17.273 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.273 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.274 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.274 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "::", 2024-04-26T02:07:17.274 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.274 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.274 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.274 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T02:07:17.274 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.274 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.275 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.275 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.275 INFO:teuthology.orchestra.run.smithi006.stdout: "server_port": { 2024-04-26T02:07:17.275 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_port", 2024-04-26T02:07:17.275 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.275 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.275 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.275 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "9283", 2024-04-26T02:07:17.275 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.276 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.276 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.276 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T02:07:17.276 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.276 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.276 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.276 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.276 INFO:teuthology.orchestra.run.smithi006.stdout: "stale_cache_strategy": { 2024-04-26T02:07:17.276 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "stale_cache_strategy", 2024-04-26T02:07:17.276 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.277 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.277 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.277 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "log", 2024-04-26T02:07:17.277 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.277 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.277 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.277 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.277 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.277 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.278 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.278 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.278 INFO:teuthology.orchestra.run.smithi006.stdout: "standby_behaviour": { 2024-04-26T02:07:17.278 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "standby_behaviour", 2024-04-26T02:07:17.278 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.278 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.278 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.278 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "default", 2024-04-26T02:07:17.279 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.279 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.279 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.279 INFO:teuthology.orchestra.run.smithi006.stdout: "default", 2024-04-26T02:07:17.279 INFO:teuthology.orchestra.run.smithi006.stdout: "error" 2024-04-26T02:07:17.279 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.279 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.279 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.279 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.280 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.280 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.280 INFO:teuthology.orchestra.run.smithi006.stdout: "standby_error_status_code": { 2024-04-26T02:07:17.280 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "standby_error_status_code", 2024-04-26T02:07:17.280 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.280 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.280 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.280 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "500", 2024-04-26T02:07:17.280 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "400", 2024-04-26T02:07:17.281 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "599", 2024-04-26T02:07:17.281 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.281 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.281 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.281 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.281 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.281 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.281 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.282 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.282 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.282 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rbd_support", 2024-04-26T02:07:17.282 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.282 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.282 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.282 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.282 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.282 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.283 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.283 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.283 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.283 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.283 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.283 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.283 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.283 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.284 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.284 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.284 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.284 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.284 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.284 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.284 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.284 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.285 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.285 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.285 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.285 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.285 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.285 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.285 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.285 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.286 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.286 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.286 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.286 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.286 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.286 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.286 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.287 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.287 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.287 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.287 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.287 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.287 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.287 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.287 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.287 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.288 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.288 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.288 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.288 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.288 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.288 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.288 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.288 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.288 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.289 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.289 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.289 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.289 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.289 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.289 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.289 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.289 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.290 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.290 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.290 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.290 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.290 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.290 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.290 INFO:teuthology.orchestra.run.smithi006.stdout: "max_concurrent_snap_create": { 2024-04-26T02:07:17.290 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_concurrent_snap_create", 2024-04-26T02:07:17.290 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.291 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.291 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.291 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:17.291 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.291 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.291 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.291 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.291 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.291 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.292 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.292 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.292 INFO:teuthology.orchestra.run.smithi006.stdout: "mirror_snapshot_schedule": { 2024-04-26T02:07:17.292 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T02:07:17.292 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.292 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.292 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.292 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.292 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.293 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.293 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.293 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.293 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.293 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.293 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.293 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.293 INFO:teuthology.orchestra.run.smithi006.stdout: "trash_purge_schedule": { 2024-04-26T02:07:17.294 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "trash_purge_schedule", 2024-04-26T02:07:17.294 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.294 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.294 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.294 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.294 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.294 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.295 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.295 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.295 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.295 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.295 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.295 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.295 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.295 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.296 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "restful", 2024-04-26T02:07:17.296 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.296 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.296 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.296 INFO:teuthology.orchestra.run.smithi006.stdout: "enable_auth": { 2024-04-26T02:07:17.296 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "enable_auth", 2024-04-26T02:07:17.296 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.296 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.297 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.297 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.297 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.297 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.297 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.297 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.297 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.297 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.298 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.298 INFO:teuthology.orchestra.run.smithi006.stdout: "key_file": { 2024-04-26T02:07:17.298 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "key_file", 2024-04-26T02:07:17.298 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.298 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.298 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.298 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.298 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.299 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.299 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.299 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.299 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.299 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.299 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.299 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.299 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.299 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.300 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.300 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.300 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.300 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.300 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.300 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.300 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.300 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.300 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.301 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.301 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.301 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.301 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.301 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.301 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.301 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.301 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.302 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.302 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.302 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.302 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.302 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.302 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.302 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.302 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.302 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.303 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.303 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.303 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.303 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.303 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.303 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.303 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.303 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.304 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.304 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.304 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.304 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.304 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.304 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.304 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.304 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.304 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.305 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.305 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.305 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.305 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.305 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.305 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.305 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.305 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.305 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.306 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.306 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.306 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.306 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.306 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.306 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.306 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.306 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.307 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.307 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.307 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.307 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.307 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.307 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.307 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.307 INFO:teuthology.orchestra.run.smithi006.stdout: "server_addr": { 2024-04-26T02:07:17.307 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_addr", 2024-04-26T02:07:17.308 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.308 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.308 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.308 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.308 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.308 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.308 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.308 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.309 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.309 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.309 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.309 INFO:teuthology.orchestra.run.smithi006.stdout: "server_port": { 2024-04-26T02:07:17.309 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_port", 2024-04-26T02:07:17.309 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.309 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.309 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.310 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.310 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.310 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.310 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.310 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.310 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.310 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.310 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.310 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.311 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.311 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rgw", 2024-04-26T02:07:17.311 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.311 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.311 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.311 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.311 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.312 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.312 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.312 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.312 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.312 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.312 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.312 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.312 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.313 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.313 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.313 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.313 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.313 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.313 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.313 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.313 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.313 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.313 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.314 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.314 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.314 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.314 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.314 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.314 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.314 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.315 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.315 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.315 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.315 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.315 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.315 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.315 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.315 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.315 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.315 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.316 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.316 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.316 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.316 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.316 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.316 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.316 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.316 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.317 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.317 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.317 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.317 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.317 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.317 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.317 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.317 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.318 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.318 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.318 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.318 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.318 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.318 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.318 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.319 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.319 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.319 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.319 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.319 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.319 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.319 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.319 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.319 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.320 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.320 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.320 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rook", 2024-04-26T02:07:17.320 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.320 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.320 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.320 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.320 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.320 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.321 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.321 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.321 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.321 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.321 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.321 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.321 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.321 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.321 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.322 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.322 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.322 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.322 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.322 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.322 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.322 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.322 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.323 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.323 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.323 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.323 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.323 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.323 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.323 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.324 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.324 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.324 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.324 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.324 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.324 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.324 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.325 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.325 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.325 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.325 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.325 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.325 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.325 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.325 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.325 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.326 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.326 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.326 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.326 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.326 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.326 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.326 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.326 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.327 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.327 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.327 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.327 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.327 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.327 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.327 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.327 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.327 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.328 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.328 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.328 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.328 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.328 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.328 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.328 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.328 INFO:teuthology.orchestra.run.smithi006.stdout: "storage_class": { 2024-04-26T02:07:17.329 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "storage_class", 2024-04-26T02:07:17.329 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.329 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.329 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.329 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "local", 2024-04-26T02:07:17.329 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.329 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.329 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.329 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T02:07:17.330 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.330 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.330 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.330 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.330 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.330 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "selftest", 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.332 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.332 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.332 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.332 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.332 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.332 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.332 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.332 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.332 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.333 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.333 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.333 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.333 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.333 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.333 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.333 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.333 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.333 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.334 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.334 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.334 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.334 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.334 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.334 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.334 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.334 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.335 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.335 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.335 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.335 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.335 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.335 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.335 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.336 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.336 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.336 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.336 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.336 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.336 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.336 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.336 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.337 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.337 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.337 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.337 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.337 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.337 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.337 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.337 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.338 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.338 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.338 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.338 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.338 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.338 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.338 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.338 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.339 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.339 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.339 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.339 INFO:teuthology.orchestra.run.smithi006.stdout: "roption1": { 2024-04-26T02:07:17.339 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "roption1", 2024-04-26T02:07:17.339 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.339 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.339 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.340 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.340 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.340 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.340 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.340 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.340 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.340 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.340 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.341 INFO:teuthology.orchestra.run.smithi006.stdout: "roption2": { 2024-04-26T02:07:17.341 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "roption2", 2024-04-26T02:07:17.341 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.341 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.341 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.341 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "xyz", 2024-04-26T02:07:17.341 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.341 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.342 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.342 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.342 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.342 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.342 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.342 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption1": { 2024-04-26T02:07:17.342 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption1", 2024-04-26T02:07:17.342 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.343 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.343 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.343 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.343 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.343 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.343 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.343 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.343 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.343 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.343 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.344 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption2": { 2024-04-26T02:07:17.344 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption2", 2024-04-26T02:07:17.344 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.344 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.344 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.344 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.344 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.344 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.345 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.345 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.345 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.345 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.345 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.345 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption3": { 2024-04-26T02:07:17.346 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption3", 2024-04-26T02:07:17.346 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:17.346 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.346 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.346 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.346 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.346 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.346 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.347 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.347 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.347 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.347 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.347 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption4": { 2024-04-26T02:07:17.347 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption4", 2024-04-26T02:07:17.347 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.347 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.347 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption5": { 2024-04-26T02:07:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption5", 2024-04-26T02:07:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption6": { 2024-04-26T02:07:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption6", 2024-04-26T02:07:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption7": { 2024-04-26T02:07:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption7", 2024-04-26T02:07:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1", 2024-04-26T02:07:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "42", 2024-04-26T02:07:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "testkey": { 2024-04-26T02:07:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "testkey", 2024-04-26T02:07:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "testlkey": { 2024-04-26T02:07:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "testlkey", 2024-04-26T02:07:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "testnewline": { 2024-04-26T02:07:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "testnewline", 2024-04-26T02:07:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "snap_schedule", 2024-04-26T02:07:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_m_granularity": { 2024-04-26T02:07:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "allow_m_granularity", 2024-04-26T02:07:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T02:07:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "dump_on_update": { 2024-04-26T02:07:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "dump_on_update", 2024-04-26T02:07:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.361 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "dump database to debug log on update", 2024-04-26T02:07:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.362 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.363 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.364 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.365 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.366 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.367 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.368 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.369 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.370 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "stats", 2024-04-26T02:07:17.371 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.372 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.373 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.374 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.375 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.376 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.377 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.378 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.379 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "status", 2024-04-26T02:07:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.381 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.384 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.385 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.386 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.388 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "telegraf", 2024-04-26T02:07:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "address": { 2024-04-26T02:07:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "address", 2024-04-26T02:07:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T02:07:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.391 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.392 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "15", 2024-04-26T02:07:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.393 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.395 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.396 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.397 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.398 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.399 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.400 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.400 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.400 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.400 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.400 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.400 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.400 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.400 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.400 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.401 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "telemetry", 2024-04-26T02:07:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.402 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_basic": { 2024-04-26T02:07:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_basic", 2024-04-26T02:07:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.403 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T02:07:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_crash": { 2024-04-26T02:07:17.404 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_crash", 2024-04-26T02:07:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T02:07:17.405 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_device": { 2024-04-26T02:07:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_device", 2024-04-26T02:07:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.406 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T02:07:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_ident": { 2024-04-26T02:07:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_ident", 2024-04-26T02:07:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.408 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T02:07:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_perf": { 2024-04-26T02:07:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_perf", 2024-04-26T02:07:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.409 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T02:07:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.410 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "contact": { 2024-04-26T02:07:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "contact", 2024-04-26T02:07:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.411 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "description": { 2024-04-26T02:07:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "description", 2024-04-26T02:07:17.412 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.413 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "device_url": { 2024-04-26T02:07:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "device_url", 2024-04-26T02:07:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T02:07:17.414 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.415 INFO:teuthology.orchestra.run.smithi006.stdout: "enabled": { 2024-04-26T02:07:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "enabled", 2024-04-26T02:07:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.416 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:17.417 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "24", 2024-04-26T02:07:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "8", 2024-04-26T02:07:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.418 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "last_opt_revision": { 2024-04-26T02:07:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "last_opt_revision", 2024-04-26T02:07:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.419 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "1", 2024-04-26T02:07:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.420 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "leaderboard": { 2024-04-26T02:07:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "leaderboard", 2024-04-26T02:07:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.421 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "leaderboard_description": { 2024-04-26T02:07:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "leaderboard_description", 2024-04-26T02:07:17.422 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.423 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.424 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.425 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.426 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.427 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.428 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.429 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.431 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "organization": { 2024-04-26T02:07:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "organization", 2024-04-26T02:07:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.432 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.433 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "proxy": { 2024-04-26T02:07:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "proxy", 2024-04-26T02:07:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.434 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "url": { 2024-04-26T02:07:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "url", 2024-04-26T02:07:17.435 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T02:07:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.436 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "test_orchestrator", 2024-04-26T02:07:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.437 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.438 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.439 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.440 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.441 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.442 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.443 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "volumes", 2024-04-26T02:07:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.446 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.447 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.448 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.449 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.450 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.451 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.452 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.453 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "max_concurrent_clones": { 2024-04-26T02:07:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_concurrent_clones", 2024-04-26T02:07:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.454 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "4", 2024-04-26T02:07:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T02:07:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.455 INFO:teuthology.orchestra.run.smithi006.stdout: "periodic_async_work": { 2024-04-26T02:07:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "periodic_async_work", 2024-04-26T02:07:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Periodically check for async work", 2024-04-26T02:07:17.457 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.457 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.457 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.457 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.457 INFO:teuthology.orchestra.run.smithi006.stdout: "snapshot_clone_delay": { 2024-04-26T02:07:17.457 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "snapshot_clone_delay", 2024-04-26T02:07:17.457 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.457 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.458 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.458 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:17.458 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.458 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.458 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.458 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T02:07:17.458 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.458 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.459 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.459 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.459 INFO:teuthology.orchestra.run.smithi006.stdout: "snapshot_clone_no_wait": { 2024-04-26T02:07:17.459 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T02:07:17.459 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.459 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.459 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.459 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.459 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.460 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.460 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.460 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T02:07:17.460 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.460 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.460 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.460 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.460 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.461 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "zabbix", 2024-04-26T02:07:17.461 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.461 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.461 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.461 INFO:teuthology.orchestra.run.smithi006.stdout: "discovery_interval": { 2024-04-26T02:07:17.461 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "discovery_interval", 2024-04-26T02:07:17.461 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:17.462 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.462 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.462 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "100", 2024-04-26T02:07:17.462 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.462 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.462 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.462 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.462 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.462 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.463 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.463 INFO:teuthology.orchestra.run.smithi006.stdout: "identifier": { 2024-04-26T02:07:17.463 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "identifier", 2024-04-26T02:07:17.463 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.463 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.463 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.463 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.463 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.464 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.464 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.464 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.464 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.464 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.464 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.464 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.464 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:17.464 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.466 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.466 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.466 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.466 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.466 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.466 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.466 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.466 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.470 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.470 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.470 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.470 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.470 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.470 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.470 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.471 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.471 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.471 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.471 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.471 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.471 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.471 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.471 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.471 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.472 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.472 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.472 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.472 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.472 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.472 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.472 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.472 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.473 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.473 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.473 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.473 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.473 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.473 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.473 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.473 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.473 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.474 INFO:teuthology.orchestra.run.smithi006.stdout: "zabbix_host": { 2024-04-26T02:07:17.474 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "zabbix_host", 2024-04-26T02:07:17.474 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.474 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.474 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.474 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.474 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.475 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.475 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.475 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.475 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.475 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.475 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.475 INFO:teuthology.orchestra.run.smithi006.stdout: "zabbix_port": { 2024-04-26T02:07:17.475 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "zabbix_port", 2024-04-26T02:07:17.476 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.476 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.476 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.476 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10051", 2024-04-26T02:07:17.476 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.476 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.476 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.476 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.476 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.476 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.477 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.477 INFO:teuthology.orchestra.run.smithi006.stdout: "zabbix_sender": { 2024-04-26T02:07:17.477 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "zabbix_sender", 2024-04-26T02:07:17.477 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.477 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.477 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.477 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T02:07:17.477 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.478 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.478 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.478 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.478 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.478 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.478 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.478 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.478 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.478 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.479 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-26T02:07:17.479 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.479 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.479 INFO:teuthology.orchestra.run.smithi006.stdout: "gid": 25069, 2024-04-26T02:07:17.479 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "z", 2024-04-26T02:07:17.479 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_features": 4540701547738038271, 2024-04-26T02:07:17.479 INFO:teuthology.orchestra.run.smithi006.stdout: "available_modules": [ 2024-04-26T02:07:17.479 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.480 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "alerts", 2024-04-26T02:07:17.480 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.480 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.480 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.480 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:17.480 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:17.480 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.480 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.480 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.481 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:17.481 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.481 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.481 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.481 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T02:07:17.481 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.481 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.481 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.482 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.482 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.482 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.482 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.482 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.482 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.482 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.482 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.483 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.483 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.483 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.483 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.483 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.483 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.483 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.483 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.483 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.484 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.484 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.484 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.484 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.484 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.484 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.484 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.484 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.485 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.485 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.485 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.485 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.485 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.485 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.485 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.485 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.486 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.486 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.486 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.486 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.486 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.486 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.486 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.486 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.486 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.487 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.487 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.487 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.487 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.487 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.487 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.487 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.487 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.487 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.488 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.488 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.488 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.488 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.488 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.488 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.488 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.488 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.489 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.489 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.489 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.489 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.489 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.489 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.489 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.489 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.490 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_destination": { 2024-04-26T02:07:17.490 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_destination", 2024-04-26T02:07:17.490 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.490 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.490 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.490 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.490 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.490 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.491 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.491 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Email address to send alerts to", 2024-04-26T02:07:17.491 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.491 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.491 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.491 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_from_name": { 2024-04-26T02:07:17.491 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_from_name", 2024-04-26T02:07:17.491 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.492 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.492 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.492 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "Ceph", 2024-04-26T02:07:17.492 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.492 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.492 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.492 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Email From: name", 2024-04-26T02:07:17.492 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.492 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.492 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.493 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_host": { 2024-04-26T02:07:17.493 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_host", 2024-04-26T02:07:17.493 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.493 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.493 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.493 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.493 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.493 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.494 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.494 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "SMTP server", 2024-04-26T02:07:17.494 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.494 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.494 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.494 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_password": { 2024-04-26T02:07:17.494 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_password", 2024-04-26T02:07:17.495 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.495 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.495 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.495 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.495 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.495 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.495 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.495 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Password to authenticate with", 2024-04-26T02:07:17.496 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.496 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.496 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.496 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.496 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_port": { 2024-04-26T02:07:17.496 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_port", 2024-04-26T02:07:17.496 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.496 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.497 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.497 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "465", 2024-04-26T02:07:17.497 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.497 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.497 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.497 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "SMTP port", 2024-04-26T02:07:17.497 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.497 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.498 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.498 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_sender": { 2024-04-26T02:07:17.498 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_sender", 2024-04-26T02:07:17.498 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.498 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.498 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.498 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.499 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.499 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.499 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.499 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "SMTP envelope sender", 2024-04-26T02:07:17.499 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.499 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.499 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.500 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_ssl": { 2024-04-26T02:07:17.500 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_ssl", 2024-04-26T02:07:17.500 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.500 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.500 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.500 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.500 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.500 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.500 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.501 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T02:07:17.501 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.501 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.501 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.501 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_user": { 2024-04-26T02:07:17.501 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_user", 2024-04-26T02:07:17.501 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.502 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.502 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.502 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.502 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.502 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.502 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.502 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "User to authenticate as", 2024-04-26T02:07:17.502 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.503 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.503 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.503 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.503 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.503 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "balancer", 2024-04-26T02:07:17.503 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.503 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.504 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.504 INFO:teuthology.orchestra.run.smithi006.stdout: "active": { 2024-04-26T02:07:17.504 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "active", 2024-04-26T02:07:17.504 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.504 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.504 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.504 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.504 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.504 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T02:07:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: "begin_time": { 2024-04-26T02:07:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "begin_time", 2024-04-26T02:07:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.505 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.506 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.506 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0000", 2024-04-26T02:07:17.506 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.506 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.506 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.506 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T02:07:17.506 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T02:07:17.506 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.507 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.507 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.507 INFO:teuthology.orchestra.run.smithi006.stdout: "begin_weekday": { 2024-04-26T02:07:17.507 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "begin_weekday", 2024-04-26T02:07:17.507 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:17.507 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.507 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.507 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:17.507 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "0", 2024-04-26T02:07:17.507 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "6", 2024-04-26T02:07:17.508 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.508 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T02:07:17.508 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T02:07:17.508 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.508 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.508 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.508 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_compat_max_iterations": { 2024-04-26T02:07:17.508 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crush_compat_max_iterations", 2024-04-26T02:07:17.508 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:17.508 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.509 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.509 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "25", 2024-04-26T02:07:17.509 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1", 2024-04-26T02:07:17.509 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "250", 2024-04-26T02:07:17.509 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.509 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T02:07:17.509 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.509 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.509 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.510 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_compat_metrics": { 2024-04-26T02:07:17.510 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crush_compat_metrics", 2024-04-26T02:07:17.510 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.510 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.510 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.510 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T02:07:17.510 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.510 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.511 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.511 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T02:07:17.511 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T02:07:17.511 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.511 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.511 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.511 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_compat_step": { 2024-04-26T02:07:17.511 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crush_compat_step", 2024-04-26T02:07:17.511 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:17.512 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.512 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.512 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0.5", 2024-04-26T02:07:17.512 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "0.001", 2024-04-26T02:07:17.512 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "0.999", 2024-04-26T02:07:17.512 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.512 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "aggressiveness of optimization", 2024-04-26T02:07:17.512 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T02:07:17.512 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.513 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.513 INFO:teuthology.orchestra.run.smithi006.stdout: "end_time": { 2024-04-26T02:07:17.513 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "end_time", 2024-04-26T02:07:17.513 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.513 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.513 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.513 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "2359", 2024-04-26T02:07:17.513 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.513 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.514 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.514 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T02:07:17.514 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T02:07:17.514 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.514 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.514 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.514 INFO:teuthology.orchestra.run.smithi006.stdout: "end_weekday": { 2024-04-26T02:07:17.514 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "end_weekday", 2024-04-26T02:07:17.514 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:17.515 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.515 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.515 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:17.515 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "0", 2024-04-26T02:07:17.515 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "6", 2024-04-26T02:07:17.515 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.515 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T02:07:17.515 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T02:07:17.515 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.516 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.516 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.516 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.516 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.516 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.516 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.516 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.516 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.516 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.516 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.517 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.517 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.517 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.517 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.517 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.517 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.517 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.517 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.517 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.518 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.518 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.518 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.518 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.518 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.518 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.518 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.519 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.519 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.519 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.519 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.519 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.519 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.519 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.519 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.519 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.520 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.520 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.520 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.520 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.520 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.520 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.520 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.520 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.520 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.521 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.521 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.521 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.521 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.521 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.521 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.521 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.521 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.522 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.522 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.522 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.522 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.522 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.522 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.522 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.522 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.522 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.522 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.523 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.523 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.523 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.523 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.523 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.523 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.523 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.523 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.523 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.524 INFO:teuthology.orchestra.run.smithi006.stdout: "min_score": { 2024-04-26T02:07:17.524 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "min_score", 2024-04-26T02:07:17.524 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:17.524 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.524 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.524 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:17.524 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.524 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.524 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.524 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T02:07:17.525 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.525 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.525 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.525 INFO:teuthology.orchestra.run.smithi006.stdout: "mode": { 2024-04-26T02:07:17.525 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mode", 2024-04-26T02:07:17.525 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.525 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.525 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.525 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "upmap", 2024-04-26T02:07:17.526 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.526 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.526 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.526 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-compat", 2024-04-26T02:07:17.526 INFO:teuthology.orchestra.run.smithi006.stdout: "none", 2024-04-26T02:07:17.526 INFO:teuthology.orchestra.run.smithi006.stdout: "read", 2024-04-26T02:07:17.526 INFO:teuthology.orchestra.run.smithi006.stdout: "upmap", 2024-04-26T02:07:17.526 INFO:teuthology.orchestra.run.smithi006.stdout: "upmap-read" 2024-04-26T02:07:17.527 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.527 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Balancer mode", 2024-04-26T02:07:17.527 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.527 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.527 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.527 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.527 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_ids": { 2024-04-26T02:07:17.527 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "pool_ids", 2024-04-26T02:07:17.527 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.528 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.528 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.528 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.528 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.528 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.528 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.528 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T02:07:17.528 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.528 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.528 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.529 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.529 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:17.529 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:17.529 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.529 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.529 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.529 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:17.529 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.529 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.530 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.530 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T02:07:17.530 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.530 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.530 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.530 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.530 INFO:teuthology.orchestra.run.smithi006.stdout: "upmap_max_deviation": { 2024-04-26T02:07:17.530 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "upmap_max_deviation", 2024-04-26T02:07:17.531 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.531 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.531 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.531 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5", 2024-04-26T02:07:17.531 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1", 2024-04-26T02:07:17.531 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.531 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.531 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T02:07:17.532 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T02:07:17.532 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.532 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.532 INFO:teuthology.orchestra.run.smithi006.stdout: "upmap_max_optimizations": { 2024-04-26T02:07:17.532 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "upmap_max_optimizations", 2024-04-26T02:07:17.532 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:17.532 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.532 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.533 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:17.533 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.533 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.533 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.533 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T02:07:17.533 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.533 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.533 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.534 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.534 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.534 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephadm", 2024-04-26T02:07:17.534 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.534 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.534 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.535 INFO:teuthology.orchestra.run.smithi006.stdout: "agent_down_multiplier": { 2024-04-26T02:07:17.535 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "agent_down_multiplier", 2024-04-26T02:07:17.535 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:17.535 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.535 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.535 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "3.0", 2024-04-26T02:07:17.535 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.535 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.536 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.536 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T02:07:17.536 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.536 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.536 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.536 INFO:teuthology.orchestra.run.smithi006.stdout: "agent_refresh_rate": { 2024-04-26T02:07:17.536 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "agent_refresh_rate", 2024-04-26T02:07:17.537 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.537 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.537 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.537 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "20", 2024-04-26T02:07:17.537 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.537 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.537 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.537 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T02:07:17.538 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.538 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.538 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.538 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.538 INFO:teuthology.orchestra.run.smithi006.stdout: "agent_starting_port": { 2024-04-26T02:07:17.538 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "agent_starting_port", 2024-04-26T02:07:17.538 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.538 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.538 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.539 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "4721", 2024-04-26T02:07:17.539 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.539 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.539 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.539 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T02:07:17.539 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.539 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.539 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.539 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.539 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_ptrace": { 2024-04-26T02:07:17.540 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "allow_ptrace", 2024-04-26T02:07:17.540 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.540 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.540 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.540 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.540 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.540 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.540 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.541 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T02:07:17.541 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T02:07:17.541 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.541 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.541 INFO:teuthology.orchestra.run.smithi006.stdout: "autotune_interval": { 2024-04-26T02:07:17.541 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "autotune_interval", 2024-04-26T02:07:17.541 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.542 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.542 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.542 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:17.542 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.542 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.542 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.542 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T02:07:17.542 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.542 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.543 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.543 INFO:teuthology.orchestra.run.smithi006.stdout: "autotune_memory_target_ratio": { 2024-04-26T02:07:17.543 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T02:07:17.543 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:17.543 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.543 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.543 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0.7", 2024-04-26T02:07:17.544 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.544 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.544 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.544 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T02:07:17.544 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.544 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.544 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.544 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm_log_destination": { 2024-04-26T02:07:17.544 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephadm_log_destination", 2024-04-26T02:07:17.545 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.545 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.545 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.545 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.545 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.545 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.545 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.545 INFO:teuthology.orchestra.run.smithi006.stdout: "file", 2024-04-26T02:07:17.545 INFO:teuthology.orchestra.run.smithi006.stdout: "file,syslog", 2024-04-26T02:07:17.546 INFO:teuthology.orchestra.run.smithi006.stdout: "syslog" 2024-04-26T02:07:17.546 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.546 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T02:07:17.546 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.546 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.546 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.546 INFO:teuthology.orchestra.run.smithi006.stdout: "cgroups_split": { 2024-04-26T02:07:17.546 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cgroups_split", 2024-04-26T02:07:17.547 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.547 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.547 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.547 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.547 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.547 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.547 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.547 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T02:07:17.547 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.548 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.548 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.548 INFO:teuthology.orchestra.run.smithi006.stdout: "config_checks_enabled": { 2024-04-26T02:07:17.548 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "config_checks_enabled", 2024-04-26T02:07:17.548 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.548 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.548 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.548 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.548 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.549 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.549 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.549 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T02:07:17.549 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.549 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.549 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.549 INFO:teuthology.orchestra.run.smithi006.stdout: "config_dashboard": { 2024-04-26T02:07:17.549 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "config_dashboard", 2024-04-26T02:07:17.550 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.550 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.550 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.550 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.550 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.550 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.550 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.550 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T02:07:17.550 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.551 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.551 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.551 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_alertmanager": { 2024-04-26T02:07:17.551 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_alertmanager", 2024-04-26T02:07:17.551 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.551 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.552 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.552 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T02:07:17.552 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.552 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.552 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.552 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:17.552 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.552 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_base": { 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_base", 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.554 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.554 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.554 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Container image name, without the tag", 2024-04-26T02:07:17.554 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.554 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.554 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.554 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_elasticsearch": { 2024-04-26T02:07:17.555 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_elasticsearch", 2024-04-26T02:07:17.555 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.555 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.555 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.555 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T02:07:17.555 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.555 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.555 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "elasticsearch container image", 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_grafana": { 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_grafana", 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_haproxy": { 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_haproxy", 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T02:07:17.559 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.559 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.559 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.559 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "HAproxy container image", 2024-04-26T02:07:17.559 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.559 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.559 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.560 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_jaeger_agent": { 2024-04-26T02:07:17.560 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_jaeger_agent", 2024-04-26T02:07:17.560 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.560 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.560 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.560 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T02:07:17.560 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.560 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.561 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.561 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Jaeger agent container image", 2024-04-26T02:07:17.561 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.561 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.561 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.561 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_jaeger_collector": { 2024-04-26T02:07:17.561 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_jaeger_collector", 2024-04-26T02:07:17.562 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.562 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.562 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.562 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T02:07:17.562 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.562 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.562 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.562 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Jaeger collector container image", 2024-04-26T02:07:17.563 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.563 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.563 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.563 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_jaeger_query": { 2024-04-26T02:07:17.563 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_jaeger_query", 2024-04-26T02:07:17.563 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.563 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.563 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.563 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T02:07:17.564 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.564 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.564 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.564 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Jaeger query container image", 2024-04-26T02:07:17.564 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.564 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.564 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.565 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_keepalived": { 2024-04-26T02:07:17.565 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_keepalived", 2024-04-26T02:07:17.565 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.565 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.565 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.565 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T02:07:17.565 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.565 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.565 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.565 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Keepalived container image", 2024-04-26T02:07:17.566 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.566 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.566 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.566 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_loki": { 2024-04-26T02:07:17.566 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_loki", 2024-04-26T02:07:17.566 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.566 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.567 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.567 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T02:07:17.567 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.567 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.567 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.567 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Loki container image", 2024-04-26T02:07:17.567 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.567 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.567 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.568 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_node_exporter": { 2024-04-26T02:07:17.568 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_node_exporter", 2024-04-26T02:07:17.568 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.568 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.568 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.568 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T02:07:17.569 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.569 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.569 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.569 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:17.569 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.569 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.569 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.569 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_nvmeof": { 2024-04-26T02:07:17.570 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_nvmeof", 2024-04-26T02:07:17.570 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.570 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.570 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.570 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T02:07:17.570 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.570 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.570 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.571 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Nvme-of container image", 2024-04-26T02:07:17.571 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.571 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.571 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.571 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_prometheus": { 2024-04-26T02:07:17.571 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_prometheus", 2024-04-26T02:07:17.571 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.571 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.571 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.572 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T02:07:17.572 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.572 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.572 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.572 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:17.572 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.572 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.572 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.573 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_promtail": { 2024-04-26T02:07:17.573 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_promtail", 2024-04-26T02:07:17.573 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.573 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.573 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.573 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T02:07:17.573 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.573 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.574 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.574 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Promtail container image", 2024-04-26T02:07:17.574 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.574 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.574 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.574 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_samba": { 2024-04-26T02:07:17.574 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_samba", 2024-04-26T02:07:17.575 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.575 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.575 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.575 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T02:07:17.575 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.575 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.575 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.575 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Samba/SMB container image", 2024-04-26T02:07:17.575 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.576 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.576 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.576 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_snmp_gateway": { 2024-04-26T02:07:17.576 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_snmp_gateway", 2024-04-26T02:07:17.576 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.576 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.576 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.577 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T02:07:17.577 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.577 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.577 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.577 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "SNMP Gateway container image", 2024-04-26T02:07:17.577 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.577 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.577 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.578 INFO:teuthology.orchestra.run.smithi006.stdout: "container_init": { 2024-04-26T02:07:17.578 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_init", 2024-04-26T02:07:17.578 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.578 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.578 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.578 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.578 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.578 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.578 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T02:07:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: "daemon_cache_timeout": { 2024-04-26T02:07:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "daemon_cache_timeout", 2024-04-26T02:07:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.580 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.580 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.580 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:17.580 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.580 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.580 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.580 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T02:07:17.580 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: "default_cephadm_command_timeout": { 2024-04-26T02:07:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T02:07:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "900", 2024-04-26T02:07:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T02:07:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.583 INFO:teuthology.orchestra.run.smithi006.stdout: "default_registry": { 2024-04-26T02:07:17.583 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "default_registry", 2024-04-26T02:07:17.583 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.583 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.583 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.583 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "docker.io", 2024-04-26T02:07:17.583 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.584 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.584 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.584 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T02:07:17.584 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.584 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.584 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.584 INFO:teuthology.orchestra.run.smithi006.stdout: "device_cache_timeout": { 2024-04-26T02:07:17.584 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "device_cache_timeout", 2024-04-26T02:07:17.585 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.585 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.585 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.585 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "1800", 2024-04-26T02:07:17.585 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.585 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.585 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.585 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "seconds to cache device inventory", 2024-04-26T02:07:17.586 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.586 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.586 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.586 INFO:teuthology.orchestra.run.smithi006.stdout: "device_enhanced_scan": { 2024-04-26T02:07:17.586 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "device_enhanced_scan", 2024-04-26T02:07:17.586 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.586 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.587 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.587 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.587 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.587 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.587 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.587 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T02:07:17.587 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.587 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.587 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.588 INFO:teuthology.orchestra.run.smithi006.stdout: "facts_cache_timeout": { 2024-04-26T02:07:17.588 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "facts_cache_timeout", 2024-04-26T02:07:17.588 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.588 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.588 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.588 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:17.588 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.589 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.589 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.589 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "seconds to cache host facts data", 2024-04-26T02:07:17.589 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.589 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.589 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.589 INFO:teuthology.orchestra.run.smithi006.stdout: "grafana_dashboards_path": { 2024-04-26T02:07:17.589 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "grafana_dashboards_path", 2024-04-26T02:07:17.590 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.590 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.590 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.590 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T02:07:17.590 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.590 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.590 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.591 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T02:07:17.591 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.591 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.591 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.591 INFO:teuthology.orchestra.run.smithi006.stdout: "host_check_interval": { 2024-04-26T02:07:17.591 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "host_check_interval", 2024-04-26T02:07:17.591 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.592 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.592 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.592 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:17.592 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.592 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.592 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.592 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to perform a host check", 2024-04-26T02:07:17.592 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.593 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.593 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.593 INFO:teuthology.orchestra.run.smithi006.stdout: "hw_monitoring": { 2024-04-26T02:07:17.593 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "hw_monitoring", 2024-04-26T02:07:17.593 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.593 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.593 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.594 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.594 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.594 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.594 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.594 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T02:07:17.594 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.594 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.594 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.595 INFO:teuthology.orchestra.run.smithi006.stdout: "inventory_list_all": { 2024-04-26T02:07:17.595 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "inventory_list_all", 2024-04-26T02:07:17.595 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.595 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.595 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.595 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.595 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.595 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.596 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.596 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T02:07:17.596 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.596 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.596 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.596 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.596 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.597 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.597 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.597 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.597 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.597 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.597 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.597 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.597 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.597 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.598 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.598 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.598 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.598 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.598 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.598 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.598 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.598 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.599 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.599 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.599 INFO:teuthology.orchestra.run.smithi006.stdout: "log_refresh_metadata": { 2024-04-26T02:07:17.599 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_refresh_metadata", 2024-04-26T02:07:17.599 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.599 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.599 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.599 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.599 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.601 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.601 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.601 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T02:07:17.601 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.601 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.601 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.602 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.602 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.602 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.602 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.602 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.602 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.602 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.603 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.603 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.603 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T02:07:17.603 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.603 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.603 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.603 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.603 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.604 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.604 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.604 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.604 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.604 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.604 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.604 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.604 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.604 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.605 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.605 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.605 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.605 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.605 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.605 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.605 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.605 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.605 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.606 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.606 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.606 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.606 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.606 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.606 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.606 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.606 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.607 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.607 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.607 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.607 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.607 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.607 INFO:teuthology.orchestra.run.smithi006.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T02:07:17.607 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T02:07:17.607 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.608 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.608 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.608 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.608 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.608 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.608 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.608 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T02:07:17.608 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.608 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.609 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.609 INFO:teuthology.orchestra.run.smithi006.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T02:07:17.609 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T02:07:17.609 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.609 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.609 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.609 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "*", 2024-04-26T02:07:17.609 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.610 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.610 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.610 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T02:07:17.610 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.610 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.610 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.610 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.611 INFO:teuthology.orchestra.run.smithi006.stdout: "max_count_per_host": { 2024-04-26T02:07:17.611 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_count_per_host", 2024-04-26T02:07:17.611 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.611 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.611 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.611 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:17.611 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.611 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.612 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.612 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "max number of daemons per service per host", 2024-04-26T02:07:17.612 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.612 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.612 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.612 INFO:teuthology.orchestra.run.smithi006.stdout: "max_osd_draining_count": { 2024-04-26T02:07:17.612 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_osd_draining_count", 2024-04-26T02:07:17.612 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.613 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.613 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.613 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:17.613 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.613 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.613 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.613 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T02:07:17.613 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.613 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.613 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.614 INFO:teuthology.orchestra.run.smithi006.stdout: "migration_current": { 2024-04-26T02:07:17.614 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "migration_current", 2024-04-26T02:07:17.614 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.614 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.614 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.614 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.614 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.614 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.615 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.615 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "internal - do not modify", 2024-04-26T02:07:17.615 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.615 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.615 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.615 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.615 INFO:teuthology.orchestra.run.smithi006.stdout: "mode": { 2024-04-26T02:07:17.615 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mode", 2024-04-26T02:07:17.615 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.615 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.616 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.616 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "root", 2024-04-26T02:07:17.616 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.616 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.616 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.616 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm-package", 2024-04-26T02:07:17.616 INFO:teuthology.orchestra.run.smithi006.stdout: "root" 2024-04-26T02:07:17.616 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.616 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T02:07:17.617 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.617 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.617 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.617 INFO:teuthology.orchestra.run.smithi006.stdout: "oob_default_addr": { 2024-04-26T02:07:17.617 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "oob_default_addr", 2024-04-26T02:07:17.617 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.617 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.618 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.618 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "169.254.1.1", 2024-04-26T02:07:17.618 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.618 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.618 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.618 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T02:07:17.618 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.618 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.618 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.619 INFO:teuthology.orchestra.run.smithi006.stdout: "prometheus_alerts_path": { 2024-04-26T02:07:17.619 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "prometheus_alerts_path", 2024-04-26T02:07:17.619 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.619 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.619 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.619 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T02:07:17.619 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.619 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.620 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.620 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T02:07:17.620 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.620 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.620 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.620 INFO:teuthology.orchestra.run.smithi006.stdout: "registry_insecure": { 2024-04-26T02:07:17.621 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "registry_insecure", 2024-04-26T02:07:17.621 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.621 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.621 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.621 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.621 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.621 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.621 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.621 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T02:07:17.621 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.622 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.622 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.622 INFO:teuthology.orchestra.run.smithi006.stdout: "registry_password": { 2024-04-26T02:07:17.622 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "registry_password", 2024-04-26T02:07:17.622 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.622 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.622 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.622 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.623 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.623 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.623 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.623 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T02:07:17.623 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.623 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.623 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.623 INFO:teuthology.orchestra.run.smithi006.stdout: "registry_url": { 2024-04-26T02:07:17.624 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "registry_url", 2024-04-26T02:07:17.624 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.624 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.624 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.624 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.624 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.624 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.624 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.624 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T02:07:17.625 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.625 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.625 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.625 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.625 INFO:teuthology.orchestra.run.smithi006.stdout: "registry_username": { 2024-04-26T02:07:17.625 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "registry_username", 2024-04-26T02:07:17.625 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.625 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.625 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.625 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.626 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.626 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.626 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.626 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T02:07:17.626 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.626 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.626 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.626 INFO:teuthology.orchestra.run.smithi006.stdout: "secure_monitoring_stack": { 2024-04-26T02:07:17.627 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "secure_monitoring_stack", 2024-04-26T02:07:17.627 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.627 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.627 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.627 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.627 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.627 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.627 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.627 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T02:07:17.628 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.628 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.628 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.628 INFO:teuthology.orchestra.run.smithi006.stdout: "service_discovery_port": { 2024-04-26T02:07:17.628 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "service_discovery_port", 2024-04-26T02:07:17.628 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.628 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.628 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.628 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8765", 2024-04-26T02:07:17.629 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.629 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.629 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.629 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "cephadm service discovery port", 2024-04-26T02:07:17.629 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.629 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.629 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.629 INFO:teuthology.orchestra.run.smithi006.stdout: "ssh_config_file": { 2024-04-26T02:07:17.630 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ssh_config_file", 2024-04-26T02:07:17.630 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.630 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.630 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.630 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.630 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.630 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.630 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.630 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T02:07:17.631 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.631 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.631 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.631 INFO:teuthology.orchestra.run.smithi006.stdout: "use_agent": { 2024-04-26T02:07:17.631 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "use_agent", 2024-04-26T02:07:17.631 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.631 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.631 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.632 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.632 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.632 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.632 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.632 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T02:07:17.632 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.632 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.632 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.633 INFO:teuthology.orchestra.run.smithi006.stdout: "use_repo_digest": { 2024-04-26T02:07:17.633 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "use_repo_digest", 2024-04-26T02:07:17.633 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.633 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.633 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.633 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.633 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.633 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.634 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.634 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T02:07:17.634 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.634 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.634 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.634 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_on_failed_host_check": { 2024-04-26T02:07:17.634 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_on_failed_host_check", 2024-04-26T02:07:17.635 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.635 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.635 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.635 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.635 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.635 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.635 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.635 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T02:07:17.635 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.636 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.636 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.636 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_on_stray_daemons": { 2024-04-26T02:07:17.636 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_on_stray_daemons", 2024-04-26T02:07:17.636 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.636 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.636 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.637 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.637 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.637 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.637 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.637 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T02:07:17.637 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.637 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.637 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.638 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_on_stray_hosts": { 2024-04-26T02:07:17.638 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_on_stray_hosts", 2024-04-26T02:07:17.638 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.638 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.638 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.638 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.638 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.638 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.638 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.639 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T02:07:17.639 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.639 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.639 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.639 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.639 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.639 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crash", 2024-04-26T02:07:17.640 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.640 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.640 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.640 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.640 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.640 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.640 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.640 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.641 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.641 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.641 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.641 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.641 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.641 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.641 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.641 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.641 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.642 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.642 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.642 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.642 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.642 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.642 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.642 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.642 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.643 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.643 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.643 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.643 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.643 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.643 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.643 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.643 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.644 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.644 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.644 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.644 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.644 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.644 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.644 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.644 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.647 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.647 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.647 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.647 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.647 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.647 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.647 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.647 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.648 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.648 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.648 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.648 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.648 INFO:teuthology.orchestra.run.smithi006.stdout: "retain_interval": { 2024-04-26T02:07:17.648 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "retain_interval", 2024-04-26T02:07:17.648 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.648 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.649 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.649 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "31536000", 2024-04-26T02:07:17.649 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.649 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.649 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.649 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T02:07:17.649 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.649 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.650 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.650 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_recent_interval": { 2024-04-26T02:07:17.650 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_recent_interval", 2024-04-26T02:07:17.650 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.650 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.650 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.650 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "1209600", 2024-04-26T02:07:17.651 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.651 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.651 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.651 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T02:07:17.651 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.651 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.651 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.651 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.651 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.652 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "dashboard", 2024-04-26T02:07:17.652 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.652 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.652 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.652 INFO:teuthology.orchestra.run.smithi006.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T02:07:17.652 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T02:07:17.653 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.653 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.653 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.653 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:17.653 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.653 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.653 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.653 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.653 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.654 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.654 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.654 INFO:teuthology.orchestra.run.smithi006.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T02:07:17.654 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T02:07:17.654 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.654 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.654 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.655 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.655 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.655 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.655 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.655 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.655 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.655 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.655 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.656 INFO:teuthology.orchestra.run.smithi006.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T02:07:17.656 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T02:07:17.656 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.656 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.656 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.656 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.656 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.656 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.656 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.657 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.657 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.657 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.657 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.657 INFO:teuthology.orchestra.run.smithi006.stdout: "AUDIT_API_ENABLED": { 2024-04-26T02:07:17.657 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T02:07:17.657 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.658 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.658 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.658 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.658 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.658 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.658 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.658 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.658 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.658 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.659 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.659 INFO:teuthology.orchestra.run.smithi006.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T02:07:17.659 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T02:07:17.659 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.659 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.659 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.659 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.659 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.660 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.660 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.660 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.660 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.660 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.660 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.660 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.660 INFO:teuthology.orchestra.run.smithi006.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T02:07:17.661 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T02:07:17.661 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.661 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.661 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.661 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.661 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.661 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.661 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.662 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.662 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.662 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.662 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.662 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.662 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T02:07:17.662 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T02:07:17.662 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.663 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.663 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.663 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.663 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.663 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.663 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.663 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.663 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.663 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.664 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.664 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.664 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T02:07:17.664 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T02:07:17.664 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.664 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.664 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.664 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.664 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.665 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.665 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.665 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.665 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.665 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.665 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.665 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T02:07:17.665 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T02:07:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T02:07:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T02:07:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.669 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T02:07:17.669 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T02:07:17.669 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.669 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.669 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.669 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.669 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.669 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.669 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.670 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.670 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.670 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.670 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.670 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T02:07:17.670 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T02:07:17.670 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.670 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.671 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.671 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.671 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.671 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.671 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.671 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.671 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.671 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.672 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.672 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T02:07:17.672 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T02:07:17.672 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.672 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.672 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.672 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.672 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T02:07:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T02:07:17.674 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.674 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.674 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.674 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.674 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.674 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.674 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.674 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.675 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.675 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.675 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.675 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T02:07:17.675 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T02:07:17.675 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.675 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.675 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.675 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "admin", 2024-04-26T02:07:17.676 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.676 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.676 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.676 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.676 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.676 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.676 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.676 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T02:07:17.677 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T02:07:17.677 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.677 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.677 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.677 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.677 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.677 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.677 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.677 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.678 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.678 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.678 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.678 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.678 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_API_URL": { 2024-04-26T02:07:17.678 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_API_URL", 2024-04-26T02:07:17.678 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.678 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.678 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.678 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.679 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.679 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.679 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.679 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.679 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.679 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.679 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.679 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.679 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T02:07:17.680 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T02:07:17.680 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.680 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.680 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.680 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "admin", 2024-04-26T02:07:17.680 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.680 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.680 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.681 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.681 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.681 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.681 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.681 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.681 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T02:07:17.681 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T02:07:17.681 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.682 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.682 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.682 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.682 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.682 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.682 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.682 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.682 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.683 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.683 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.683 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.683 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T02:07:17.683 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T02:07:17.683 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.683 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.683 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.683 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.684 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.684 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.684 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.684 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.684 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.684 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.684 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.684 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.684 INFO:teuthology.orchestra.run.smithi006.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T02:07:17.684 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T02:07:17.685 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.685 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.685 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.685 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.685 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.685 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.685 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.685 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.685 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.686 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.686 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.686 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.686 INFO:teuthology.orchestra.run.smithi006.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T02:07:17.686 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T02:07:17.686 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.686 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.686 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.686 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.687 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.687 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.687 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.687 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.687 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.687 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.687 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.687 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.687 INFO:teuthology.orchestra.run.smithi006.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T02:07:17.688 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T02:07:17.688 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.688 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.688 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.688 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "[]", 2024-04-26T02:07:17.688 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.688 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.688 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.688 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.688 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.689 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.689 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.689 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.689 INFO:teuthology.orchestra.run.smithi006.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T02:07:17.689 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T02:07:17.689 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.689 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.689 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.689 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "{}", 2024-04-26T02:07:17.690 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.690 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.690 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.690 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.690 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.690 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.690 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.690 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.691 INFO:teuthology.orchestra.run.smithi006.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T02:07:17.691 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T02:07:17.691 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.691 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.691 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.691 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.691 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.691 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.692 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.692 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.692 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.692 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.692 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.692 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.692 INFO:teuthology.orchestra.run.smithi006.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T02:07:17.692 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T02:07:17.692 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.693 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.693 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.693 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.693 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.693 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.693 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.693 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.693 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.694 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.694 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.694 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.694 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T02:07:17.694 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T02:07:17.694 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.694 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.694 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.694 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.695 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.695 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.695 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.695 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.695 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.695 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.695 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.695 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.696 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T02:07:17.696 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T02:07:17.696 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.696 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.696 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.696 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.696 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.696 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.696 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.696 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.697 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.697 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.697 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.697 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.697 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T02:07:17.697 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T02:07:17.697 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.697 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.697 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.698 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.698 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.698 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.698 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.698 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.698 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.698 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.698 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.699 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.699 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T02:07:17.699 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T02:07:17.699 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.699 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.699 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.699 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.699 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.699 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.700 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.700 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.700 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.700 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.700 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.700 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.700 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T02:07:17.700 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T02:07:17.701 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.701 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.701 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.701 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.701 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.701 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.701 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.701 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.702 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.702 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.702 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.702 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T02:07:17.702 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T02:07:17.702 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.702 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.703 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.703 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.703 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.703 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.703 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.703 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.703 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.703 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.703 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.704 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.704 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T02:07:17.704 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T02:07:17.704 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.704 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.704 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.704 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.704 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.705 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.705 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.705 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.705 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.705 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.705 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.705 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.705 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T02:07:17.705 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T02:07:17.706 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.706 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.706 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.706 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.706 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.706 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.706 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.706 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.707 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.707 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.707 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.707 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.707 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T02:07:17.707 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T02:07:17.707 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.707 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.707 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.708 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T02:07:17.708 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.708 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.708 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.708 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.708 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.708 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.708 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.709 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T02:07:17.709 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T02:07:17.709 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.709 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.709 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.709 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:17.709 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.709 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.709 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.709 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.710 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.710 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.710 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.710 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.710 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T02:07:17.710 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T02:07:17.710 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.710 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.711 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.711 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8", 2024-04-26T02:07:17.711 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.711 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.711 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.711 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.711 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.711 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.711 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.712 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.712 INFO:teuthology.orchestra.run.smithi006.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T02:07:17.712 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T02:07:17.712 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.712 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.712 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.712 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "45", 2024-04-26T02:07:17.712 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.712 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.713 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.713 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.713 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.713 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.713 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.713 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.713 INFO:teuthology.orchestra.run.smithi006.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T02:07:17.713 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T02:07:17.713 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.714 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.714 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.714 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.714 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.714 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.714 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.714 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.714 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.715 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.715 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.715 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.715 INFO:teuthology.orchestra.run.smithi006.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T02:07:17.715 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T02:07:17.715 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.715 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.715 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.715 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "admin", 2024-04-26T02:07:17.716 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.716 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.716 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.716 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.716 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.716 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.716 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.716 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.716 INFO:teuthology.orchestra.run.smithi006.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T02:07:17.716 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T02:07:17.717 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.717 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.717 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.717 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.717 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.717 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.717 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.717 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.717 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.718 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.718 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.718 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.718 INFO:teuthology.orchestra.run.smithi006.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T02:07:17.718 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T02:07:17.718 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.718 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.718 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.718 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.719 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.719 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.719 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.719 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.719 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.719 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.719 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.719 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.719 INFO:teuthology.orchestra.run.smithi006.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T02:07:17.720 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T02:07:17.720 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.720 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.720 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.720 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.720 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.720 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.720 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.721 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.721 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.721 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.721 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.721 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.721 INFO:teuthology.orchestra.run.smithi006.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T02:07:17.721 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T02:07:17.721 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.721 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.722 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.722 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:17.722 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.722 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.722 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.722 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.722 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.722 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.722 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.723 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.723 INFO:teuthology.orchestra.run.smithi006.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T02:07:17.723 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T02:07:17.723 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.723 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.723 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.723 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:17.723 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.723 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.724 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.724 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.724 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.724 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.724 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.724 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.724 INFO:teuthology.orchestra.run.smithi006.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T02:07:17.724 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T02:07:17.724 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.725 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.725 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.725 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5", 2024-04-26T02:07:17.725 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.725 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.725 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.725 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.725 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.725 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.725 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.726 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.726 INFO:teuthology.orchestra.run.smithi006.stdout: "cross_origin_url": { 2024-04-26T02:07:17.726 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cross_origin_url", 2024-04-26T02:07:17.726 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.726 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.726 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.726 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.726 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.727 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.727 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.727 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.727 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.727 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.727 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.727 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.727 INFO:teuthology.orchestra.run.smithi006.stdout: "crt_file": { 2024-04-26T02:07:17.727 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crt_file", 2024-04-26T02:07:17.728 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.728 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.728 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.728 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.728 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.728 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.728 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.728 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.729 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.729 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.729 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.729 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.729 INFO:teuthology.orchestra.run.smithi006.stdout: "debug": { 2024-04-26T02:07:17.729 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "debug", 2024-04-26T02:07:17.729 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.729 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.729 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.730 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.730 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.730 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.730 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.730 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Enable/disable debug options", 2024-04-26T02:07:17.730 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.730 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.730 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.731 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.731 INFO:teuthology.orchestra.run.smithi006.stdout: "jwt_token_ttl": { 2024-04-26T02:07:17.731 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "jwt_token_ttl", 2024-04-26T02:07:17.731 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.731 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.731 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.731 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "28800", 2024-04-26T02:07:17.731 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.732 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.732 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.732 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.732 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.732 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.732 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.732 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.732 INFO:teuthology.orchestra.run.smithi006.stdout: "key_file": { 2024-04-26T02:07:17.732 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "key_file", 2024-04-26T02:07:17.733 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.733 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.733 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.733 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.733 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.733 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.733 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.733 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.733 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.734 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.734 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.734 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.734 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.734 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.734 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.734 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.734 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.734 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.734 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.735 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.735 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.735 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.735 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.735 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.735 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.735 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.735 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.735 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.736 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.736 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.736 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.736 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.736 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.736 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.736 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.736 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.737 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.737 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.737 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.737 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.737 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.737 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.737 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.737 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.737 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.738 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.738 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.738 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.738 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.738 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.738 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.738 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.738 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.738 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.739 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.739 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.739 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.739 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.739 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.739 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.739 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.739 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.739 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.740 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.740 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.740 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.740 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.740 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.740 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.740 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.740 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.740 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.741 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.741 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.741 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.741 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.741 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.741 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.741 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.741 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.742 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.742 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.742 INFO:teuthology.orchestra.run.smithi006.stdout: "motd": { 2024-04-26T02:07:17.742 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "motd", 2024-04-26T02:07:17.742 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.742 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.742 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.742 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.742 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.743 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.743 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.743 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "The message of the day", 2024-04-26T02:07:17.743 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.743 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.743 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.743 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.743 INFO:teuthology.orchestra.run.smithi006.stdout: "redirect_resolve_ip_addr": { 2024-04-26T02:07:17.744 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T02:07:17.744 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.744 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.744 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.744 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.744 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.744 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.744 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.744 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.745 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.745 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.745 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.745 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.745 INFO:teuthology.orchestra.run.smithi006.stdout: "server_addr": { 2024-04-26T02:07:17.745 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_addr", 2024-04-26T02:07:17.745 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.745 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.746 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.746 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "::", 2024-04-26T02:07:17.746 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.746 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.746 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.746 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.746 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.746 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.746 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.747 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.747 INFO:teuthology.orchestra.run.smithi006.stdout: "server_port": { 2024-04-26T02:07:17.747 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_port", 2024-04-26T02:07:17.747 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.747 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.747 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.747 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8080", 2024-04-26T02:07:17.747 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.748 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.748 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.748 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.748 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.748 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.748 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.748 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.748 INFO:teuthology.orchestra.run.smithi006.stdout: "ssl": { 2024-04-26T02:07:17.748 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ssl", 2024-04-26T02:07:17.749 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.749 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.749 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.749 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.749 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.749 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.749 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.749 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.750 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.750 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.750 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.750 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.750 INFO:teuthology.orchestra.run.smithi006.stdout: "ssl_server_port": { 2024-04-26T02:07:17.750 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ssl_server_port", 2024-04-26T02:07:17.750 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.750 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.750 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.751 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8443", 2024-04-26T02:07:17.751 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.751 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.751 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.751 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.751 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.751 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.751 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.751 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.752 INFO:teuthology.orchestra.run.smithi006.stdout: "standby_behaviour": { 2024-04-26T02:07:17.752 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "standby_behaviour", 2024-04-26T02:07:17.752 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.752 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.752 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.752 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "redirect", 2024-04-26T02:07:17.752 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.752 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.753 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.753 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.753 INFO:teuthology.orchestra.run.smithi006.stdout: "redirect" 2024-04-26T02:07:17.753 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.753 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.753 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.753 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.753 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.753 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.754 INFO:teuthology.orchestra.run.smithi006.stdout: "standby_error_status_code": { 2024-04-26T02:07:17.754 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "standby_error_status_code", 2024-04-26T02:07:17.754 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.754 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.754 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.754 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "500", 2024-04-26T02:07:17.754 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "400", 2024-04-26T02:07:17.754 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "599", 2024-04-26T02:07:17.754 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.755 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.755 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.755 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.755 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.755 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.755 INFO:teuthology.orchestra.run.smithi006.stdout: "url_prefix": { 2024-04-26T02:07:17.755 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "url_prefix", 2024-04-26T02:07:17.755 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.755 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.755 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.756 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.756 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.756 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.756 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.756 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.756 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.756 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.756 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.756 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.756 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.757 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.757 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.757 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "devicehealth", 2024-04-26T02:07:17.757 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.757 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.757 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.757 INFO:teuthology.orchestra.run.smithi006.stdout: "enable_monitoring": { 2024-04-26T02:07:17.757 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "enable_monitoring", 2024-04-26T02:07:17.757 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.758 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.758 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.758 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.758 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.758 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.758 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.758 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "monitor device health metrics", 2024-04-26T02:07:17.758 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.758 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.758 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.759 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.759 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.759 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.759 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.759 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.759 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.759 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.759 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.759 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.759 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.760 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.760 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.760 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.760 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.760 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.760 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.760 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.760 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.760 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.761 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.761 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.761 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.761 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.761 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.761 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.761 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.761 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.762 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.762 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.762 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.762 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.762 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.762 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.762 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.762 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.762 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.763 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.763 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.763 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.763 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.763 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.763 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.763 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.763 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.763 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.764 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.764 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.764 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.764 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.764 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.764 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.764 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.764 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.764 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.764 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.765 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.765 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.765 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.765 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.765 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.765 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.765 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.765 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.766 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.766 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.766 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.766 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.766 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.766 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.766 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.766 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.766 INFO:teuthology.orchestra.run.smithi006.stdout: "mark_out_threshold": { 2024-04-26T02:07:17.767 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mark_out_threshold", 2024-04-26T02:07:17.767 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.767 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.767 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.767 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "2419200", 2024-04-26T02:07:17.767 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.767 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.767 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.767 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T02:07:17.768 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.768 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.768 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.768 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.768 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": { 2024-04-26T02:07:17.768 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "pool_name", 2024-04-26T02:07:17.768 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.768 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.768 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.769 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "device_health_metrics", 2024-04-26T02:07:17.769 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.769 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.769 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.769 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T02:07:17.769 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.769 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.769 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.769 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.770 INFO:teuthology.orchestra.run.smithi006.stdout: "retention_period": { 2024-04-26T02:07:17.770 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "retention_period", 2024-04-26T02:07:17.770 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.770 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.770 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.770 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "15552000", 2024-04-26T02:07:17.770 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.770 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.771 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.771 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how long to retain device health metrics", 2024-04-26T02:07:17.771 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.771 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.771 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.771 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.771 INFO:teuthology.orchestra.run.smithi006.stdout: "scrape_frequency": { 2024-04-26T02:07:17.771 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "scrape_frequency", 2024-04-26T02:07:17.771 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.772 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.772 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.772 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "86400", 2024-04-26T02:07:17.772 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.772 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.772 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.772 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T02:07:17.772 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.773 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.773 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.773 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.773 INFO:teuthology.orchestra.run.smithi006.stdout: "self_heal": { 2024-04-26T02:07:17.773 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "self_heal", 2024-04-26T02:07:17.773 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.773 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.773 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.773 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.774 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.774 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.774 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.774 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T02:07:17.774 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.774 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.774 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.774 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.775 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:17.775 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:17.775 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.775 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.775 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.775 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:17.775 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.775 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.776 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.776 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T02:07:17.776 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.776 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.776 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.776 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_threshold": { 2024-04-26T02:07:17.776 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_threshold", 2024-04-26T02:07:17.776 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.776 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.777 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.777 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "7257600", 2024-04-26T02:07:17.777 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.777 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.777 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.777 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T02:07:17.777 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.777 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.777 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.778 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.778 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.778 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.778 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.778 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "diskprediction_local", 2024-04-26T02:07:17.778 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.778 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.778 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.779 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.779 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.779 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.779 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.779 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.779 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.779 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.779 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.780 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.780 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.780 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.780 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.780 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.780 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.780 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.780 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.780 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.781 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.781 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.781 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.781 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.781 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.781 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.781 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.781 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.782 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.782 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.782 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.782 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.782 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.782 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.782 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.782 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.783 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.783 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.783 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.783 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.783 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.783 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.783 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.783 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.784 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.784 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.784 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.784 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.784 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.784 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.784 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.784 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.784 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.785 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.785 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.785 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.785 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.785 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.785 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.785 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.785 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.786 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.786 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.786 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.786 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.786 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.786 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.786 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.787 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.787 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.787 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.787 INFO:teuthology.orchestra.run.smithi006.stdout: "predict_interval": { 2024-04-26T02:07:17.787 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "predict_interval", 2024-04-26T02:07:17.787 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.787 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.787 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.788 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "86400", 2024-04-26T02:07:17.788 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.788 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.788 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.788 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.788 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.788 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.788 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.789 INFO:teuthology.orchestra.run.smithi006.stdout: "predictor_model": { 2024-04-26T02:07:17.789 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "predictor_model", 2024-04-26T02:07:17.789 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.789 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.789 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.789 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "prophetstor", 2024-04-26T02:07:17.789 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.789 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.790 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.790 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.790 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.790 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.790 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.790 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:17.790 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:17.791 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.791 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.791 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.791 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:17.791 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.791 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.791 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.791 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.791 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.792 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.792 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.792 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.792 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.792 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.792 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.792 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "influx", 2024-04-26T02:07:17.792 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": false, 2024-04-26T02:07:17.793 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "influxdb python module not found", 2024-04-26T02:07:17.793 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.793 INFO:teuthology.orchestra.run.smithi006.stdout: "batch_size": { 2024-04-26T02:07:17.793 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "batch_size", 2024-04-26T02:07:17.793 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.793 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.793 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.793 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5000", 2024-04-26T02:07:17.793 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.793 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.794 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.794 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T02:07:17.794 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.794 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.794 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.794 INFO:teuthology.orchestra.run.smithi006.stdout: "database": { 2024-04-26T02:07:17.794 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "database", 2024-04-26T02:07:17.795 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.795 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.795 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.795 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "ceph", 2024-04-26T02:07:17.795 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.795 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.795 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.795 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T02:07:17.795 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.796 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.796 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.796 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.796 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": { 2024-04-26T02:07:17.796 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "hostname", 2024-04-26T02:07:17.796 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.796 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.796 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.796 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.797 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.797 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.797 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.797 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "InfluxDB server hostname", 2024-04-26T02:07:17.797 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.797 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.797 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.797 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:17.798 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:17.798 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.798 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.798 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.798 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "30", 2024-04-26T02:07:17.798 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "5", 2024-04-26T02:07:17.798 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.798 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.798 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T02:07:17.799 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.799 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.799 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.799 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.799 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.799 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.799 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.799 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.800 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.800 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.800 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.800 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.800 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.800 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.800 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.800 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.801 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.801 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.801 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.801 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.801 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.801 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.801 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.802 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.802 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.802 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.802 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.802 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.802 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.802 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.802 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.802 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.803 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.803 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.803 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.803 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.803 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.803 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.803 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.803 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.804 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.804 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.804 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.804 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.804 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.804 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.804 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.804 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.804 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.805 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.805 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.805 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.805 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.805 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.805 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.805 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.805 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.806 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.806 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.806 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.806 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.806 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.806 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.806 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.806 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.807 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.807 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.807 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.807 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.807 INFO:teuthology.orchestra.run.smithi006.stdout: "password": { 2024-04-26T02:07:17.807 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "password", 2024-04-26T02:07:17.807 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.808 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.808 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.808 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.808 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.808 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.808 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.808 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "password of InfluxDB server user", 2024-04-26T02:07:17.808 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.808 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.808 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.809 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.809 INFO:teuthology.orchestra.run.smithi006.stdout: "port": { 2024-04-26T02:07:17.809 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "port", 2024-04-26T02:07:17.809 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.809 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.809 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.809 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8086", 2024-04-26T02:07:17.809 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.809 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.810 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.810 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "InfluxDB server port", 2024-04-26T02:07:17.810 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.810 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.810 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.810 INFO:teuthology.orchestra.run.smithi006.stdout: "ssl": { 2024-04-26T02:07:17.810 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ssl", 2024-04-26T02:07:17.810 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.811 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.811 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.811 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "false", 2024-04-26T02:07:17.811 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.811 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.811 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.811 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T02:07:17.811 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.811 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.812 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.812 INFO:teuthology.orchestra.run.smithi006.stdout: "threads": { 2024-04-26T02:07:17.812 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "threads", 2024-04-26T02:07:17.812 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.812 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.812 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.812 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5", 2024-04-26T02:07:17.813 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1", 2024-04-26T02:07:17.813 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "32", 2024-04-26T02:07:17.813 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.813 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T02:07:17.813 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.813 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.813 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.813 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.813 INFO:teuthology.orchestra.run.smithi006.stdout: "username": { 2024-04-26T02:07:17.814 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "username", 2024-04-26T02:07:17.814 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.814 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.814 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.814 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.814 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.814 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.814 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.815 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "username of InfluxDB server user", 2024-04-26T02:07:17.815 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.815 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.815 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.815 INFO:teuthology.orchestra.run.smithi006.stdout: "verify_ssl": { 2024-04-26T02:07:17.815 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "verify_ssl", 2024-04-26T02:07:17.815 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.815 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.816 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.816 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "true", 2024-04-26T02:07:17.816 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.816 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.816 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.816 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T02:07:17.816 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.816 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.816 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.817 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.817 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.817 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.817 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "insights", 2024-04-26T02:07:17.817 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.817 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.817 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.817 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.818 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.818 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.818 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.818 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.818 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.818 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.818 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.818 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.819 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.819 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.819 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.819 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.819 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.819 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.819 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.819 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.819 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.820 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.820 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.820 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.820 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.820 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.820 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.820 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.820 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.821 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.821 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.821 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.821 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.821 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.821 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.821 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.821 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.822 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.822 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.822 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.822 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.822 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.822 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.822 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.822 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.822 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.825 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.825 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.825 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.825 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.825 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.825 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.825 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.825 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.825 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.826 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.826 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.826 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "iostat", 2024-04-26T02:07:17.826 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.826 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.826 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.826 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.826 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.827 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.828 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.829 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.830 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.830 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.830 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.830 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.830 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.830 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.830 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.830 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.831 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.832 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.833 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.834 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "k8sevents", 2024-04-26T02:07:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_event_retention_days": { 2024-04-26T02:07:17.835 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ceph_event_retention_days", 2024-04-26T02:07:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "7", 2024-04-26T02:07:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T02:07:17.836 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "config_check_secs": { 2024-04-26T02:07:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "config_check_secs", 2024-04-26T02:07:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.837 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "10", 2024-04-26T02:07:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T02:07:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.838 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.839 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.840 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.841 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.843 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.844 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.845 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.846 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "localpool", 2024-04-26T02:07:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "failure_domain": { 2024-04-26T02:07:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "failure_domain", 2024-04-26T02:07:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "host", 2024-04-26T02:07:17.848 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "failure domain for any created local pool", 2024-04-26T02:07:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T02:07:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.849 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.850 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.851 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.852 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.854 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.855 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": { 2024-04-26T02:07:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "min_size", 2024-04-26T02:07:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.858 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "default min_size for any created local pool", 2024-04-26T02:07:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T02:07:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "num_rep": { 2024-04-26T02:07:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "num_rep", 2024-04-26T02:07:17.859 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "3", 2024-04-26T02:07:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "default replica count for any created local pool", 2024-04-26T02:07:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": { 2024-04-26T02:07:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "pg_num", 2024-04-26T02:07:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.861 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "128", 2024-04-26T02:07:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T02:07:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.862 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "prefix": { 2024-04-26T02:07:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "prefix", 2024-04-26T02:07:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.863 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "name prefix for any created local pool", 2024-04-26T02:07:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "subtree": { 2024-04-26T02:07:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "subtree", 2024-04-26T02:07:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "rack", 2024-04-26T02:07:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T02:07:17.865 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T02:07:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mds_autoscaler", 2024-04-26T02:07:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.866 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.867 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.868 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.869 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.871 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.872 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.873 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.874 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mirroring", 2024-04-26T02:07:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.875 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.876 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.877 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.878 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.880 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.881 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.882 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "nfs", 2024-04-26T02:07:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.884 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.885 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.886 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.887 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.889 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.889 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.889 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.889 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.889 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.889 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.889 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.889 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.889 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.890 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.891 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "orchestrator", 2024-04-26T02:07:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "fail_fs": { 2024-04-26T02:07:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "fail_fs", 2024-04-26T02:07:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.893 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T02:07:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.894 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.895 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.896 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.897 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.898 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.899 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.900 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.901 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "orchestrator": { 2024-04-26T02:07:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "orchestrator", 2024-04-26T02:07:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm", 2024-04-26T02:07:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "rook", 2024-04-26T02:07:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "test_orchestrator" 2024-04-26T02:07:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Orchestrator backend", 2024-04-26T02:07:17.904 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "osd_perf_query", 2024-04-26T02:07:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.905 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.906 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.907 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.908 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.909 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.910 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.911 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "osd_support", 2024-04-26T02:07:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.913 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.914 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.915 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.916 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.917 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.917 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.917 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.917 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.917 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.917 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.917 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.917 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.917 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.918 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.919 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.920 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.921 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "pg_autoscaler", 2024-04-26T02:07:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.922 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.923 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.924 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.925 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.926 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.927 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.928 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.929 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.930 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.931 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.932 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "threshold": { 2024-04-26T02:07:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "threshold", 2024-04-26T02:07:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "3.0", 2024-04-26T02:07:17.933 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1.0", 2024-04-26T02:07:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "scaling threshold", 2024-04-26T02:07:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T02:07:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.934 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "progress", 2024-04-26T02:07:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_pg_recovery_event": { 2024-04-26T02:07:17.935 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "allow_pg_recovery_event", 2024-04-26T02:07:17.936 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.936 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.936 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.936 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.936 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.936 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.936 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.936 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T02:07:17.936 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.937 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.937 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.937 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.937 INFO:teuthology.orchestra.run.smithi006.stdout: "enabled": { 2024-04-26T02:07:17.937 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "enabled", 2024-04-26T02:07:17.937 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.937 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.937 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.937 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.938 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.938 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.938 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.938 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.938 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.938 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.938 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.938 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.938 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.938 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.939 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.939 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.939 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.939 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.939 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.939 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.939 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.939 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.939 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.940 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.940 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.940 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.940 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.940 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.940 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.940 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.940 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.940 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.941 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.941 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.941 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.941 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.941 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.941 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.941 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.941 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.942 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.942 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.942 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.942 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.942 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.942 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.942 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.942 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.943 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.943 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.943 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.943 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.943 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.943 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.943 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.943 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.944 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.944 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.944 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.944 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.944 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.944 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.944 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.944 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.944 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.945 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.945 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.945 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.945 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.945 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.945 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.945 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.945 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.945 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.946 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.946 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.946 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.946 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.946 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.946 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.946 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.946 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.946 INFO:teuthology.orchestra.run.smithi006.stdout: "max_completed_events": { 2024-04-26T02:07:17.947 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_completed_events", 2024-04-26T02:07:17.947 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.947 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.947 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.947 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "50", 2024-04-26T02:07:17.947 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.947 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.947 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.947 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "number of past completed events to remember", 2024-04-26T02:07:17.948 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.948 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.948 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.948 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.948 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:17.948 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:17.948 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:17.948 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.948 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.948 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5", 2024-04-26T02:07:17.949 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.949 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.949 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.949 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how long the module is going to sleep", 2024-04-26T02:07:17.949 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.949 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.949 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.949 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.949 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.950 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.950 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.950 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "prometheus", 2024-04-26T02:07:17.950 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.950 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.950 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.950 INFO:teuthology.orchestra.run.smithi006.stdout: "cache": { 2024-04-26T02:07:17.950 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cache", 2024-04-26T02:07:17.951 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.951 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.951 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.951 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.951 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.951 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.951 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.951 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.951 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.952 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.952 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.952 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.952 INFO:teuthology.orchestra.run.smithi006.stdout: "exclude_perf_counters": { 2024-04-26T02:07:17.952 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "exclude_perf_counters", 2024-04-26T02:07:17.952 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.952 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.952 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.952 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.952 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.953 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.953 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.953 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T02:07:17.953 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T02:07:17.953 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.953 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.953 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.953 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.953 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.954 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.954 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.954 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.954 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.954 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.954 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.954 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.954 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.954 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.955 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.955 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.955 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.955 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.955 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.955 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.955 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.955 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.955 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.956 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.956 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.956 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.956 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.956 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.956 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.956 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.956 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.956 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.957 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.957 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.957 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.957 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.957 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.957 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.957 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.957 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.957 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.958 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.958 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.958 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.958 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.958 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.958 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.958 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.958 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.958 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.959 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.959 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.959 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.959 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.959 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.959 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.959 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.959 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.959 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.960 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.960 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.960 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.960 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.960 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.960 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.960 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.960 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.960 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.961 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.961 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.961 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.961 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_stats_pools": { 2024-04-26T02:07:17.961 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rbd_stats_pools", 2024-04-26T02:07:17.961 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.962 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.962 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.962 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.962 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.962 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.962 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.962 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.962 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.962 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.963 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.963 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T02:07:17.963 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T02:07:17.963 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.963 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.963 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.963 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "300", 2024-04-26T02:07:17.963 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.964 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.964 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.964 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.964 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.964 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.964 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.964 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.964 INFO:teuthology.orchestra.run.smithi006.stdout: "scrape_interval": { 2024-04-26T02:07:17.964 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "scrape_interval", 2024-04-26T02:07:17.965 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:17.965 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.965 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.965 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "15.0", 2024-04-26T02:07:17.965 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.965 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.965 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.965 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.965 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.965 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.966 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.966 INFO:teuthology.orchestra.run.smithi006.stdout: "server_addr": { 2024-04-26T02:07:17.966 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_addr", 2024-04-26T02:07:17.966 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.966 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.966 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.966 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "::", 2024-04-26T02:07:17.966 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.967 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.967 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.967 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T02:07:17.967 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.967 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.967 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.967 INFO:teuthology.orchestra.run.smithi006.stdout: "server_port": { 2024-04-26T02:07:17.967 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_port", 2024-04-26T02:07:17.968 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.968 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.968 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.968 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "9283", 2024-04-26T02:07:17.968 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.968 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.968 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.968 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T02:07:17.968 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.968 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.969 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.969 INFO:teuthology.orchestra.run.smithi006.stdout: "stale_cache_strategy": { 2024-04-26T02:07:17.969 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "stale_cache_strategy", 2024-04-26T02:07:17.969 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.969 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.969 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.969 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "log", 2024-04-26T02:07:17.969 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.970 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.970 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.970 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.970 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.970 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.970 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.970 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.970 INFO:teuthology.orchestra.run.smithi006.stdout: "standby_behaviour": { 2024-04-26T02:07:17.971 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "standby_behaviour", 2024-04-26T02:07:17.971 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.971 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.971 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.971 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "default", 2024-04-26T02:07:17.971 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.971 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.971 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.972 INFO:teuthology.orchestra.run.smithi006.stdout: "default", 2024-04-26T02:07:17.972 INFO:teuthology.orchestra.run.smithi006.stdout: "error" 2024-04-26T02:07:17.972 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.972 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.972 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.972 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.972 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.972 INFO:teuthology.orchestra.run.smithi006.stdout: "standby_error_status_code": { 2024-04-26T02:07:17.972 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "standby_error_status_code", 2024-04-26T02:07:17.973 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.973 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.973 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.973 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "500", 2024-04-26T02:07:17.973 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "400", 2024-04-26T02:07:17.973 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "599", 2024-04-26T02:07:17.973 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.973 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.973 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.973 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.974 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.974 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.974 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.974 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.974 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rbd_support", 2024-04-26T02:07:17.974 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.974 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.975 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.975 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.975 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.975 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.975 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.975 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.975 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.975 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.975 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.976 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.976 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.976 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.976 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.976 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.976 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.976 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.976 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.976 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.977 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.977 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.977 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.977 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.977 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.977 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.977 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.977 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.977 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.978 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.978 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.978 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.978 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.978 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.978 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.978 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.978 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.978 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.979 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.979 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.979 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.979 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.979 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.979 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.979 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.979 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.979 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.980 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.980 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.980 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.980 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.980 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.980 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.980 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.980 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.980 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.981 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.981 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.981 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.981 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.981 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.981 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.981 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.981 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.981 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.981 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.982 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.982 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.982 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.982 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.982 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.982 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.982 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.982 INFO:teuthology.orchestra.run.smithi006.stdout: "max_concurrent_snap_create": { 2024-04-26T02:07:17.982 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_concurrent_snap_create", 2024-04-26T02:07:17.983 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:17.983 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.983 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.983 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:17.983 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.983 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.983 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.983 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.983 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.984 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.984 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.984 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.984 INFO:teuthology.orchestra.run.smithi006.stdout: "mirror_snapshot_schedule": { 2024-04-26T02:07:17.984 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T02:07:17.984 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.984 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.984 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.984 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: "trash_purge_schedule": { 2024-04-26T02:07:17.985 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "trash_purge_schedule", 2024-04-26T02:07:17.986 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.986 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.986 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.986 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.986 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.986 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.986 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.986 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.986 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.987 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.987 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.987 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.987 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:17.987 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.987 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:17.987 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "restful", 2024-04-26T02:07:17.987 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:17.987 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:17.987 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:17.988 INFO:teuthology.orchestra.run.smithi006.stdout: "enable_auth": { 2024-04-26T02:07:17.988 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "enable_auth", 2024-04-26T02:07:17.988 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.988 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.988 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.988 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:17.988 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.988 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.988 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.989 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.989 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.989 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.989 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.989 INFO:teuthology.orchestra.run.smithi006.stdout: "key_file": { 2024-04-26T02:07:17.989 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "key_file", 2024-04-26T02:07:17.989 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.989 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.990 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.990 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.990 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.990 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.990 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.990 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.990 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.990 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.991 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.991 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:17.991 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:17.991 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.991 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.991 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.991 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.991 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.992 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.992 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.992 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.992 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.992 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.992 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.992 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.992 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.993 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.993 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.993 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.993 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.993 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.993 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:17.993 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:17.993 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.993 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.994 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.994 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.994 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.994 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.994 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.994 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.994 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.994 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.994 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.995 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:17.995 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:17.995 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.995 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.995 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.995 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:17.995 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.995 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.996 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:17.996 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:17.996 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:17.996 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:17.996 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:17.996 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:17.996 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:17.996 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:17.997 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.997 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.997 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.997 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.997 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:17.997 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:17.997 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:17.997 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.997 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:17.998 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:17.998 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.998 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:17.998 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:17.998 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:17.998 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:17.998 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:17.998 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:17.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:17.999 INFO:teuthology.orchestra.run.smithi006.stdout: "server_addr": { 2024-04-26T02:07:17.999 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_addr", 2024-04-26T02:07:17.999 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:17.999 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:17.999 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:17.999 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:17.999 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:17.999 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.000 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.000 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.000 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.000 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.000 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.000 INFO:teuthology.orchestra.run.smithi006.stdout: "server_port": { 2024-04-26T02:07:18.000 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_port", 2024-04-26T02:07:18.000 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.001 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.001 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.001 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.001 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.001 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.001 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.001 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.001 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.001 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.002 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.002 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.002 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.002 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rgw", 2024-04-26T02:07:18.002 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.002 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.002 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.003 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.003 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.003 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.003 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.003 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.003 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.003 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.003 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.003 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.003 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.004 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.004 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.004 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.004 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.004 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.004 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.004 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.004 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.004 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.005 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.005 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.005 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.005 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.005 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.005 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.005 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.005 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.006 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.006 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.006 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.006 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.006 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.006 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.006 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.007 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.007 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.007 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.007 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.007 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.007 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.007 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.007 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.007 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.007 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.008 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.008 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.008 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.008 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.008 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.008 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.008 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.008 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.008 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.009 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.009 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.009 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.009 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.009 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.009 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.009 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.009 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.009 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.010 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.010 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.010 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.010 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.010 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.010 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.011 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rook", 2024-04-26T02:07:18.011 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.011 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.011 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.011 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.011 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.011 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.011 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.011 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.012 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.012 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.012 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.012 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.012 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.012 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.012 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.012 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.013 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.013 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.013 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.013 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.013 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.013 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.013 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.013 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.013 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.014 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.014 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.014 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.014 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.014 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.014 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.014 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.014 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.014 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.015 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.015 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.015 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.015 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.015 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.015 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.015 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.016 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.016 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.016 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.016 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.016 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.016 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.016 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.016 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.017 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.017 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.017 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.017 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.017 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.017 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.017 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.018 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.018 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.018 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.018 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.018 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.018 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.018 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.018 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.018 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.019 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.019 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.019 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.019 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.019 INFO:teuthology.orchestra.run.smithi006.stdout: "storage_class": { 2024-04-26T02:07:18.019 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "storage_class", 2024-04-26T02:07:18.019 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.020 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.020 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.020 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "local", 2024-04-26T02:07:18.020 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.020 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.020 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.020 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T02:07:18.020 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.020 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.021 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.021 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.021 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.021 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "selftest", 2024-04-26T02:07:18.021 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.021 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.022 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.022 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.022 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.022 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.022 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.022 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.022 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.022 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.023 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.023 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.023 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.023 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.023 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.023 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.023 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.023 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.023 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.024 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.024 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.024 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.024 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.024 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.024 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.024 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.024 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.024 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.025 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.025 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.025 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.025 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.025 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.025 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.025 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.025 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.026 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.026 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.026 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.026 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.026 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.026 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.026 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.027 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.027 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.027 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.027 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.027 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.027 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.027 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.027 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.027 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.028 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.028 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.028 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.028 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.028 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.028 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.028 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.028 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.029 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.029 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.029 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.029 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.029 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.029 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.029 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.029 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.030 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.030 INFO:teuthology.orchestra.run.smithi006.stdout: "roption1": { 2024-04-26T02:07:18.030 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "roption1", 2024-04-26T02:07:18.030 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.030 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.030 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.030 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.030 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.031 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.031 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.031 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.031 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.031 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.031 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.031 INFO:teuthology.orchestra.run.smithi006.stdout: "roption2": { 2024-04-26T02:07:18.032 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "roption2", 2024-04-26T02:07:18.032 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.032 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.032 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.032 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "xyz", 2024-04-26T02:07:18.032 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.032 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.032 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.032 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.032 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.033 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.033 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.033 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption1": { 2024-04-26T02:07:18.033 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption1", 2024-04-26T02:07:18.033 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.033 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.033 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.033 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.034 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.034 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.034 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.034 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.034 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.034 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.034 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.034 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption2": { 2024-04-26T02:07:18.035 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption2", 2024-04-26T02:07:18.035 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.035 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.035 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.035 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.035 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.035 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.035 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.035 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.035 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.036 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.036 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.036 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption3": { 2024-04-26T02:07:18.036 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption3", 2024-04-26T02:07:18.036 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:18.036 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.036 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.036 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.037 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.037 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.037 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.037 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.037 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.037 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.037 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.037 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption4": { 2024-04-26T02:07:18.038 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption4", 2024-04-26T02:07:18.038 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.038 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.038 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.038 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.038 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.038 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.038 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.038 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.039 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.039 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.039 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.039 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption5": { 2024-04-26T02:07:18.039 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption5", 2024-04-26T02:07:18.039 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.039 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.039 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.039 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.040 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.040 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.040 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.040 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.040 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.040 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.040 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.040 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption6": { 2024-04-26T02:07:18.041 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption6", 2024-04-26T02:07:18.041 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.041 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.041 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.041 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.041 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.041 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.041 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.041 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption7": { 2024-04-26T02:07:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption7", 2024-04-26T02:07:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1", 2024-04-26T02:07:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "42", 2024-04-26T02:07:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.044 INFO:teuthology.orchestra.run.smithi006.stdout: "testkey": { 2024-04-26T02:07:18.044 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "testkey", 2024-04-26T02:07:18.044 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.044 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.044 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.044 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.044 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.044 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.044 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.045 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.045 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.045 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.045 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.045 INFO:teuthology.orchestra.run.smithi006.stdout: "testlkey": { 2024-04-26T02:07:18.045 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "testlkey", 2024-04-26T02:07:18.045 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.045 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.045 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.046 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.046 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.046 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.046 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.046 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.046 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.046 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.046 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.047 INFO:teuthology.orchestra.run.smithi006.stdout: "testnewline": { 2024-04-26T02:07:18.047 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "testnewline", 2024-04-26T02:07:18.047 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.047 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.047 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.047 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.047 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.047 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.048 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.048 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.048 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.048 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.048 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.048 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.048 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.049 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "snap_schedule", 2024-04-26T02:07:18.049 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.049 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.049 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.049 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_m_granularity": { 2024-04-26T02:07:18.049 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "allow_m_granularity", 2024-04-26T02:07:18.049 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.049 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.050 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.050 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.050 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.050 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.050 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.050 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T02:07:18.050 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.050 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.051 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.051 INFO:teuthology.orchestra.run.smithi006.stdout: "dump_on_update": { 2024-04-26T02:07:18.051 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "dump_on_update", 2024-04-26T02:07:18.051 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.051 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.051 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.051 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.051 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.051 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.052 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.052 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "dump database to debug log on update", 2024-04-26T02:07:18.052 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.052 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.052 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.052 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.052 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.053 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.053 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.053 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.053 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.053 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.053 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.053 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.053 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.053 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.054 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.054 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.054 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.054 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.054 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.054 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.054 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.054 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.054 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.055 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.055 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.055 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.055 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.055 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.055 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.055 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.055 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.055 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.056 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.056 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.056 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.056 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.056 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.056 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.056 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.056 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.056 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.057 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.057 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.057 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.057 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.057 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.057 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.057 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.057 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.057 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.058 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.058 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.058 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.058 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.058 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.058 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.058 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.059 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.059 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.059 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.059 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.059 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.059 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.059 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.059 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.059 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.059 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.060 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.060 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.060 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.060 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.060 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "stats", 2024-04-26T02:07:18.060 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.061 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.061 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.061 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.061 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.061 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.061 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.061 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.061 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.061 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.062 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.062 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.062 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.062 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.062 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.062 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.062 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.062 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.063 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.063 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.063 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.063 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.063 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.063 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.063 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.063 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.064 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.064 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.064 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.064 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.064 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.064 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.064 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.064 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.064 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.065 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.065 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.065 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.065 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.065 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.065 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.065 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.065 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.066 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.066 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.066 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.066 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.066 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.066 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.066 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.066 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.067 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.067 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.067 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.067 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.067 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.067 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.067 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.068 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.068 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.068 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.068 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.068 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.068 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.068 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.068 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.068 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.069 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.069 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.069 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.069 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.069 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "status", 2024-04-26T02:07:18.069 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.069 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.070 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.070 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.070 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.070 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.070 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.070 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.070 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.070 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.070 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.071 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.071 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.071 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.071 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.071 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.071 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.071 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.071 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.071 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.072 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.072 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.072 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.072 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.072 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.072 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.072 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.072 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.072 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.072 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.073 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.073 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.073 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.073 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.073 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.073 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.073 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.073 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.074 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.074 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.074 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.074 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.074 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.074 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.074 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.074 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.075 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.075 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.075 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.075 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.075 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.075 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.075 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.075 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.076 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.076 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.076 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.076 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.076 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.076 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.076 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.076 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.077 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.077 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.077 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.077 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.077 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.077 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.077 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.077 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.077 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.078 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.078 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "telegraf", 2024-04-26T02:07:18.078 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.078 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.078 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.078 INFO:teuthology.orchestra.run.smithi006.stdout: "address": { 2024-04-26T02:07:18.079 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "address", 2024-04-26T02:07:18.079 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.079 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.079 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.079 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T02:07:18.079 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.079 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.079 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.079 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.080 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.080 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.080 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.080 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:18.080 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:18.080 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.080 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.081 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.081 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "15", 2024-04-26T02:07:18.081 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.081 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.081 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.081 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.081 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.081 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.081 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.082 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.082 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.082 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.082 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.082 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.082 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.082 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.082 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.083 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.083 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.083 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.083 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.083 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.083 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.083 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.083 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.083 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.084 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.084 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.084 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.084 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.084 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.084 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.084 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.085 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.085 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.085 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.085 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.085 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.085 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.085 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.085 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.086 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.086 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.086 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.086 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.086 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.086 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.086 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.087 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.087 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.087 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.087 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.087 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.087 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.087 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.087 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.087 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.088 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.088 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.088 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.088 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.088 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.088 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.088 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.088 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "telemetry", 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_basic": { 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_basic", 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.092 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.092 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.092 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.092 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.092 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T02:07:18.092 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.092 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.092 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.093 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_crash": { 2024-04-26T02:07:18.093 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_crash", 2024-04-26T02:07:18.093 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.093 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.093 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.093 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.094 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.094 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.094 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.094 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T02:07:18.094 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.094 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.094 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.095 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_device": { 2024-04-26T02:07:18.095 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_device", 2024-04-26T02:07:18.095 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.095 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.095 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.095 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.095 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.096 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.096 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.096 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T02:07:18.096 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.096 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.096 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.096 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_ident": { 2024-04-26T02:07:18.096 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_ident", 2024-04-26T02:07:18.097 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.097 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.097 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.097 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.097 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.097 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.097 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.097 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T02:07:18.097 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.098 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.098 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.098 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_perf": { 2024-04-26T02:07:18.098 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_perf", 2024-04-26T02:07:18.098 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.098 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.098 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.098 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.099 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.099 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.099 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.099 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T02:07:18.099 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.099 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.099 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.099 INFO:teuthology.orchestra.run.smithi006.stdout: "contact": { 2024-04-26T02:07:18.100 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "contact", 2024-04-26T02:07:18.100 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.100 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.100 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.100 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.100 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.100 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.100 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.100 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.101 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.101 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.101 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.101 INFO:teuthology.orchestra.run.smithi006.stdout: "description": { 2024-04-26T02:07:18.101 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "description", 2024-04-26T02:07:18.101 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.101 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.101 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.102 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.102 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.102 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.102 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.102 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.102 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.102 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.102 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.103 INFO:teuthology.orchestra.run.smithi006.stdout: "device_url": { 2024-04-26T02:07:18.103 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "device_url", 2024-04-26T02:07:18.103 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.103 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.103 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.103 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T02:07:18.103 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.103 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.103 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.104 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.104 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.104 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.104 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.104 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.104 INFO:teuthology.orchestra.run.smithi006.stdout: "enabled": { 2024-04-26T02:07:18.104 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "enabled", 2024-04-26T02:07:18.104 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.104 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.104 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.105 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.105 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.105 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.105 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.105 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.105 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.105 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.105 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.105 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.105 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:18.106 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:18.106 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.106 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.106 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.106 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "24", 2024-04-26T02:07:18.106 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "8", 2024-04-26T02:07:18.106 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.106 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.106 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "last_opt_revision": { 2024-04-26T02:07:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "last_opt_revision", 2024-04-26T02:07:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.107 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.108 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "1", 2024-04-26T02:07:18.108 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.108 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.108 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.108 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.108 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.108 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.108 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.109 INFO:teuthology.orchestra.run.smithi006.stdout: "leaderboard": { 2024-04-26T02:07:18.109 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "leaderboard", 2024-04-26T02:07:18.109 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.109 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.109 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.109 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.109 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.109 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.110 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.110 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.110 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.110 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.110 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.110 INFO:teuthology.orchestra.run.smithi006.stdout: "leaderboard_description": { 2024-04-26T02:07:18.110 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "leaderboard_description", 2024-04-26T02:07:18.110 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.111 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.111 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.111 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.111 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.111 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.111 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.111 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.111 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.111 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.112 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.112 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.112 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.112 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.112 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.112 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.112 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.112 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.113 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.113 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.113 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.113 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.113 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.113 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.113 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.113 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.113 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.113 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.114 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.114 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.114 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.114 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.114 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.114 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.114 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.114 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.115 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.115 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.115 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.115 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.115 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.115 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.115 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.115 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.115 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.116 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.116 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.116 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.116 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.116 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.116 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.116 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.116 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.116 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.116 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.117 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.117 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.117 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.117 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.117 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.117 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.117 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.117 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.117 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.118 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.118 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.118 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.118 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.118 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.118 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.118 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.118 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.118 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.119 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.119 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.119 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.119 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.119 INFO:teuthology.orchestra.run.smithi006.stdout: "organization": { 2024-04-26T02:07:18.119 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "organization", 2024-04-26T02:07:18.119 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.119 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.119 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.120 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.120 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.120 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.120 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.120 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.120 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.120 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.120 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.120 INFO:teuthology.orchestra.run.smithi006.stdout: "proxy": { 2024-04-26T02:07:18.121 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "proxy", 2024-04-26T02:07:18.121 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.121 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.121 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.121 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.121 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.121 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.121 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.121 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.122 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.122 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.122 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.122 INFO:teuthology.orchestra.run.smithi006.stdout: "url": { 2024-04-26T02:07:18.122 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "url", 2024-04-26T02:07:18.122 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.122 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.123 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.123 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T02:07:18.123 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.123 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.123 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.123 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.123 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.123 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.123 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.124 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.124 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.124 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "test_orchestrator", 2024-04-26T02:07:18.124 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.124 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.124 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.124 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.124 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.125 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.125 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.125 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.125 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.125 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.125 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.125 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.125 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.125 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.126 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.126 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.126 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.126 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.126 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.126 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.126 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.126 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.126 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.127 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.127 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.127 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.127 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.127 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.127 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.127 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.127 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.127 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.128 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.128 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.128 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.128 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.128 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.128 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.128 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.128 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.128 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.129 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.129 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.129 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.129 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.129 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.129 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.129 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.129 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.129 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.130 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.130 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.130 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.130 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.130 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.130 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.130 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.130 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.131 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.131 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.131 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.131 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.131 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.131 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.131 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.131 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.131 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.132 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.132 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.132 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.132 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.132 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.132 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "volumes", 2024-04-26T02:07:18.132 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.133 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.133 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.133 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.133 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.133 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.133 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.133 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.133 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.133 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.134 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.134 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.134 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.134 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.134 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.134 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.134 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.134 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.135 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.135 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.135 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.135 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.135 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.135 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.135 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.135 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.136 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.136 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.136 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.136 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.136 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.136 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.136 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.136 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.137 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.137 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.137 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.137 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.137 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.137 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.137 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.137 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.138 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.138 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.138 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.138 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.138 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.138 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.138 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.138 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.138 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.139 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.139 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.139 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.139 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.139 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.139 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.139 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.140 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.140 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.140 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.140 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.140 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.140 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.140 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.140 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.140 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.141 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.141 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.141 INFO:teuthology.orchestra.run.smithi006.stdout: "max_concurrent_clones": { 2024-04-26T02:07:18.141 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_concurrent_clones", 2024-04-26T02:07:18.141 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.141 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.141 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.142 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "4", 2024-04-26T02:07:18.142 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.142 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.142 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.142 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T02:07:18.142 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.142 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.142 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.143 INFO:teuthology.orchestra.run.smithi006.stdout: "periodic_async_work": { 2024-04-26T02:07:18.143 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "periodic_async_work", 2024-04-26T02:07:18.143 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.143 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.143 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.143 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.143 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.143 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.144 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.144 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Periodically check for async work", 2024-04-26T02:07:18.144 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.144 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.144 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.144 INFO:teuthology.orchestra.run.smithi006.stdout: "snapshot_clone_delay": { 2024-04-26T02:07:18.144 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "snapshot_clone_delay", 2024-04-26T02:07:18.144 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.144 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.145 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.145 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:18.145 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.145 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.145 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.145 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T02:07:18.145 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.145 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.145 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.145 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.146 INFO:teuthology.orchestra.run.smithi006.stdout: "snapshot_clone_no_wait": { 2024-04-26T02:07:18.146 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T02:07:18.146 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.146 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.146 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.146 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.146 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.146 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.146 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.147 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T02:07:18.147 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.147 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.147 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.147 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.147 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.147 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "zabbix", 2024-04-26T02:07:18.147 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.148 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.148 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.148 INFO:teuthology.orchestra.run.smithi006.stdout: "discovery_interval": { 2024-04-26T02:07:18.148 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "discovery_interval", 2024-04-26T02:07:18.148 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:18.148 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.148 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.148 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "100", 2024-04-26T02:07:18.149 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.149 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.149 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.149 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.149 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.149 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.149 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.150 INFO:teuthology.orchestra.run.smithi006.stdout: "identifier": { 2024-04-26T02:07:18.150 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "identifier", 2024-04-26T02:07:18.150 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.150 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.150 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.150 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.150 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.150 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.151 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.151 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.151 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.151 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.151 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.151 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:18.151 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:18.151 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.151 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.152 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.153 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.154 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.154 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.154 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.154 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.154 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.154 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.154 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.154 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.154 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.154 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.155 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.155 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.155 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.155 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.155 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.155 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.155 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.155 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.156 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.156 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.156 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.156 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.156 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.156 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.156 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.156 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.157 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.157 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.157 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.157 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.157 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.157 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.157 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.157 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.157 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.158 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.158 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.158 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.158 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.158 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.158 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.158 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.158 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.158 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.159 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.159 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.159 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.159 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.159 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.159 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.159 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.159 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.160 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.160 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.160 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.160 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.160 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.160 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.160 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.160 INFO:teuthology.orchestra.run.smithi006.stdout: "zabbix_host": { 2024-04-26T02:07:18.160 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "zabbix_host", 2024-04-26T02:07:18.161 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.161 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.161 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.161 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.161 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.161 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.161 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.161 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.161 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.162 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.162 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.162 INFO:teuthology.orchestra.run.smithi006.stdout: "zabbix_port": { 2024-04-26T02:07:18.162 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "zabbix_port", 2024-04-26T02:07:18.162 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.162 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.162 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.163 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10051", 2024-04-26T02:07:18.163 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.163 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.163 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.163 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.163 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.163 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.163 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.164 INFO:teuthology.orchestra.run.smithi006.stdout: "zabbix_sender": { 2024-04-26T02:07:18.164 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "zabbix_sender", 2024-04-26T02:07:18.164 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.164 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.164 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.164 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T02:07:18.164 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.165 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.165 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.165 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.165 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.165 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.165 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.165 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.165 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.165 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.166 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-26T02:07:18.166 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.166 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.166 INFO:teuthology.orchestra.run.smithi006.stdout: "modules": [ 2024-04-26T02:07:18.166 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm", 2024-04-26T02:07:18.166 INFO:teuthology.orchestra.run.smithi006.stdout: "dashboard", 2024-04-26T02:07:18.166 INFO:teuthology.orchestra.run.smithi006.stdout: "iostat", 2024-04-26T02:07:18.166 INFO:teuthology.orchestra.run.smithi006.stdout: "nfs", 2024-04-26T02:07:18.167 INFO:teuthology.orchestra.run.smithi006.stdout: "restful", 2024-04-26T02:07:18.167 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_schedule" 2024-04-26T02:07:18.167 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.167 INFO:teuthology.orchestra.run.smithi006.stdout: "available_modules": [ 2024-04-26T02:07:18.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.167 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "alerts", 2024-04-26T02:07:18.167 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.167 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.167 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.167 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:18.168 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:18.168 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.168 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.168 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.168 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:18.168 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.168 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.168 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.168 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T02:07:18.169 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.169 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.169 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.169 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.169 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.169 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.169 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.169 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.170 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.170 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.170 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.170 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.170 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.170 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.170 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.170 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.170 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.171 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.171 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.171 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.171 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.171 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.171 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.171 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.171 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.172 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.172 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.172 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.172 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.172 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.172 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.172 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.172 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.173 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.173 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.173 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.173 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.173 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.173 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.173 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.173 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.173 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.174 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.174 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.174 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.174 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.174 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.174 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.174 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.174 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.174 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.175 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.175 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.175 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.175 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.175 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.175 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.175 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.175 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.176 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.176 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.176 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.176 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.176 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.176 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.176 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.176 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.177 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.177 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.177 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_destination": { 2024-04-26T02:07:18.177 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_destination", 2024-04-26T02:07:18.177 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.177 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.177 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.178 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.178 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.178 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.178 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.178 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Email address to send alerts to", 2024-04-26T02:07:18.178 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.178 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.178 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.179 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_from_name": { 2024-04-26T02:07:18.179 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_from_name", 2024-04-26T02:07:18.179 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.179 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.179 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.179 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "Ceph", 2024-04-26T02:07:18.179 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.179 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.180 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.180 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Email From: name", 2024-04-26T02:07:18.180 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.180 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.180 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.180 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_host": { 2024-04-26T02:07:18.180 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_host", 2024-04-26T02:07:18.180 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.181 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.181 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.181 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.181 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.181 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.181 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.181 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "SMTP server", 2024-04-26T02:07:18.181 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.181 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.182 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.182 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_password": { 2024-04-26T02:07:18.182 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_password", 2024-04-26T02:07:18.182 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.182 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.182 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.182 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.183 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.183 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.183 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.183 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Password to authenticate with", 2024-04-26T02:07:18.183 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.183 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.183 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.183 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_port": { 2024-04-26T02:07:18.184 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_port", 2024-04-26T02:07:18.184 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.184 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.184 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.184 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "465", 2024-04-26T02:07:18.184 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.184 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.184 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.184 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "SMTP port", 2024-04-26T02:07:18.185 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.185 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.185 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.185 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.185 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_sender": { 2024-04-26T02:07:18.185 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_sender", 2024-04-26T02:07:18.185 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.185 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.185 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.185 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.186 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.186 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.186 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.186 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "SMTP envelope sender", 2024-04-26T02:07:18.186 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.186 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.186 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.186 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.186 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_ssl": { 2024-04-26T02:07:18.187 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_ssl", 2024-04-26T02:07:18.187 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.187 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.187 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.187 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.187 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.187 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.187 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.187 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T02:07:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: "smtp_user": { 2024-04-26T02:07:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "smtp_user", 2024-04-26T02:07:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.189 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.189 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.189 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.189 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "User to authenticate as", 2024-04-26T02:07:18.189 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.189 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.189 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.189 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.189 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.190 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.190 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.190 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "balancer", 2024-04-26T02:07:18.190 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.190 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.190 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.190 INFO:teuthology.orchestra.run.smithi006.stdout: "active": { 2024-04-26T02:07:18.190 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "active", 2024-04-26T02:07:18.190 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.190 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.191 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.191 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.191 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.191 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.191 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.191 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T02:07:18.191 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.191 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.191 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.192 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.192 INFO:teuthology.orchestra.run.smithi006.stdout: "begin_time": { 2024-04-26T02:07:18.192 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "begin_time", 2024-04-26T02:07:18.192 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.192 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.192 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.192 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0000", 2024-04-26T02:07:18.192 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.192 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.192 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.193 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T02:07:18.193 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T02:07:18.193 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.193 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.193 INFO:teuthology.orchestra.run.smithi006.stdout: "begin_weekday": { 2024-04-26T02:07:18.193 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "begin_weekday", 2024-04-26T02:07:18.193 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:18.194 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.194 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.194 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:18.194 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "0", 2024-04-26T02:07:18.194 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "6", 2024-04-26T02:07:18.194 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.194 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T02:07:18.194 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T02:07:18.194 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.195 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.195 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.195 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_compat_max_iterations": { 2024-04-26T02:07:18.195 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crush_compat_max_iterations", 2024-04-26T02:07:18.195 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:18.195 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.195 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.195 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "25", 2024-04-26T02:07:18.196 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1", 2024-04-26T02:07:18.196 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "250", 2024-04-26T02:07:18.196 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.196 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T02:07:18.196 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.196 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.196 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.196 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.197 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_compat_metrics": { 2024-04-26T02:07:18.197 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crush_compat_metrics", 2024-04-26T02:07:18.197 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.197 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.197 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.197 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T02:07:18.197 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.197 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.197 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.198 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T02:07:18.198 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T02:07:18.198 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.198 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.198 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.198 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_compat_step": { 2024-04-26T02:07:18.198 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crush_compat_step", 2024-04-26T02:07:18.198 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:18.199 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.199 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.199 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0.5", 2024-04-26T02:07:18.199 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "0.001", 2024-04-26T02:07:18.199 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "0.999", 2024-04-26T02:07:18.199 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.199 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "aggressiveness of optimization", 2024-04-26T02:07:18.199 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T02:07:18.200 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.200 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.200 INFO:teuthology.orchestra.run.smithi006.stdout: "end_time": { 2024-04-26T02:07:18.200 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "end_time", 2024-04-26T02:07:18.200 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.200 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.200 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.201 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "2359", 2024-04-26T02:07:18.201 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.201 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.201 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.201 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T02:07:18.201 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T02:07:18.201 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.201 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.201 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.202 INFO:teuthology.orchestra.run.smithi006.stdout: "end_weekday": { 2024-04-26T02:07:18.202 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "end_weekday", 2024-04-26T02:07:18.202 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:18.202 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.202 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.202 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:18.202 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "0", 2024-04-26T02:07:18.202 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "6", 2024-04-26T02:07:18.203 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.203 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T02:07:18.203 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T02:07:18.203 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.203 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.203 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.203 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.203 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.203 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.203 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.204 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.204 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.204 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.204 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.204 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.204 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.204 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.204 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.205 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.205 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.205 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.205 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.205 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.205 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.205 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.205 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.205 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.206 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.206 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.206 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.206 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.206 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.206 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.206 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.206 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.206 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.207 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.207 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.207 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.207 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.207 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.207 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.207 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.207 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.208 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.208 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.208 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.208 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.208 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.208 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.208 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.208 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.209 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.209 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.209 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.209 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.209 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.209 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.209 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.209 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.209 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.210 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.210 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.210 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.210 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.210 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.210 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.210 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.210 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.211 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.211 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.211 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.211 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.211 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.211 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.211 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.211 INFO:teuthology.orchestra.run.smithi006.stdout: "min_score": { 2024-04-26T02:07:18.211 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "min_score", 2024-04-26T02:07:18.212 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:18.212 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.212 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.212 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:18.212 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.212 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.212 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.212 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T02:07:18.212 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.212 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.213 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.213 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.213 INFO:teuthology.orchestra.run.smithi006.stdout: "mode": { 2024-04-26T02:07:18.213 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mode", 2024-04-26T02:07:18.213 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.213 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.213 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.213 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "upmap", 2024-04-26T02:07:18.213 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.214 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.214 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.214 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-compat", 2024-04-26T02:07:18.214 INFO:teuthology.orchestra.run.smithi006.stdout: "none", 2024-04-26T02:07:18.214 INFO:teuthology.orchestra.run.smithi006.stdout: "read", 2024-04-26T02:07:18.214 INFO:teuthology.orchestra.run.smithi006.stdout: "upmap", 2024-04-26T02:07:18.214 INFO:teuthology.orchestra.run.smithi006.stdout: "upmap-read" 2024-04-26T02:07:18.214 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.214 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Balancer mode", 2024-04-26T02:07:18.215 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.215 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.215 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.215 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.215 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_ids": { 2024-04-26T02:07:18.215 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "pool_ids", 2024-04-26T02:07:18.215 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.215 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.215 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.216 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.216 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.216 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.216 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.216 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T02:07:18.216 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.216 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.216 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.216 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.216 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:18.217 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:18.217 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.217 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.217 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.217 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:18.217 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.217 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.217 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.217 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T02:07:18.217 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.218 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.218 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.218 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.218 INFO:teuthology.orchestra.run.smithi006.stdout: "upmap_max_deviation": { 2024-04-26T02:07:18.218 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "upmap_max_deviation", 2024-04-26T02:07:18.218 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.218 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.218 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.218 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5", 2024-04-26T02:07:18.219 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1", 2024-04-26T02:07:18.219 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.219 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.219 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T02:07:18.219 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T02:07:18.219 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.219 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.219 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.219 INFO:teuthology.orchestra.run.smithi006.stdout: "upmap_max_optimizations": { 2024-04-26T02:07:18.220 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "upmap_max_optimizations", 2024-04-26T02:07:18.220 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:18.220 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.220 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.220 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:18.220 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.220 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.220 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.220 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T02:07:18.221 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.221 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.221 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.221 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.221 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.221 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.221 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.221 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephadm", 2024-04-26T02:07:18.221 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.222 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.222 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.222 INFO:teuthology.orchestra.run.smithi006.stdout: "agent_down_multiplier": { 2024-04-26T02:07:18.222 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "agent_down_multiplier", 2024-04-26T02:07:18.222 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:18.222 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.222 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.222 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "3.0", 2024-04-26T02:07:18.222 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.223 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.223 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.223 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T02:07:18.223 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.223 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.223 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.223 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.223 INFO:teuthology.orchestra.run.smithi006.stdout: "agent_refresh_rate": { 2024-04-26T02:07:18.223 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "agent_refresh_rate", 2024-04-26T02:07:18.224 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.224 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.224 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.224 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "20", 2024-04-26T02:07:18.224 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.224 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.224 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.224 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T02:07:18.224 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.225 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.225 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.225 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.225 INFO:teuthology.orchestra.run.smithi006.stdout: "agent_starting_port": { 2024-04-26T02:07:18.225 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "agent_starting_port", 2024-04-26T02:07:18.225 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.225 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.225 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.225 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "4721", 2024-04-26T02:07:18.225 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.226 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.226 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.226 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T02:07:18.226 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.226 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.226 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.226 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.226 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_ptrace": { 2024-04-26T02:07:18.227 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "allow_ptrace", 2024-04-26T02:07:18.227 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.227 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.227 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.227 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.227 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.227 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.227 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.227 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T02:07:18.228 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T02:07:18.228 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.228 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.228 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.228 INFO:teuthology.orchestra.run.smithi006.stdout: "autotune_interval": { 2024-04-26T02:07:18.228 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "autotune_interval", 2024-04-26T02:07:18.228 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.228 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T02:07:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: "autotune_memory_target_ratio": { 2024-04-26T02:07:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T02:07:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0.7", 2024-04-26T02:07:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.231 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.231 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.231 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T02:07:18.231 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.231 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.231 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.231 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.231 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm_log_destination": { 2024-04-26T02:07:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephadm_log_destination", 2024-04-26T02:07:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: "file", 2024-04-26T02:07:18.233 INFO:teuthology.orchestra.run.smithi006.stdout: "file,syslog", 2024-04-26T02:07:18.233 INFO:teuthology.orchestra.run.smithi006.stdout: "syslog" 2024-04-26T02:07:18.233 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.233 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T02:07:18.233 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.233 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.233 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.234 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.234 INFO:teuthology.orchestra.run.smithi006.stdout: "cgroups_split": { 2024-04-26T02:07:18.234 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cgroups_split", 2024-04-26T02:07:18.234 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.234 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.234 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.234 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.234 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.234 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.235 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.235 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T02:07:18.235 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.235 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.235 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.235 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.235 INFO:teuthology.orchestra.run.smithi006.stdout: "config_checks_enabled": { 2024-04-26T02:07:18.235 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "config_checks_enabled", 2024-04-26T02:07:18.235 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.236 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.236 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.236 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.236 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.236 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.236 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.236 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T02:07:18.236 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.237 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.237 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.237 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.237 INFO:teuthology.orchestra.run.smithi006.stdout: "config_dashboard": { 2024-04-26T02:07:18.237 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "config_dashboard", 2024-04-26T02:07:18.237 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.237 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.237 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.237 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.238 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.238 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.238 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.238 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T02:07:18.238 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.238 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.238 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.238 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.239 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_alertmanager": { 2024-04-26T02:07:18.239 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_alertmanager", 2024-04-26T02:07:18.239 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.239 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.239 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.239 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T02:07:18.239 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.239 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.240 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.240 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:18.240 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.240 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.240 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.240 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.240 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_base": { 2024-04-26T02:07:18.240 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_base", 2024-04-26T02:07:18.240 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.241 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.241 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.241 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T02:07:18.241 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.241 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.241 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.241 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Container image name, without the tag", 2024-04-26T02:07:18.241 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.241 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.242 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.242 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.242 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_elasticsearch": { 2024-04-26T02:07:18.242 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_elasticsearch", 2024-04-26T02:07:18.242 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.242 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.242 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.242 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T02:07:18.243 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.243 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.243 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.243 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "elasticsearch container image", 2024-04-26T02:07:18.243 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.243 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.243 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.243 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.244 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_grafana": { 2024-04-26T02:07:18.244 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_grafana", 2024-04-26T02:07:18.244 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.244 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.244 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.244 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T02:07:18.244 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.244 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.244 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.245 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:18.245 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.245 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.245 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.245 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.245 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_haproxy": { 2024-04-26T02:07:18.245 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_haproxy", 2024-04-26T02:07:18.245 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.246 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.246 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.246 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T02:07:18.246 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.246 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.246 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.246 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "HAproxy container image", 2024-04-26T02:07:18.246 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.246 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.247 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.247 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.247 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_jaeger_agent": { 2024-04-26T02:07:18.247 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_jaeger_agent", 2024-04-26T02:07:18.247 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.247 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.247 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.247 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T02:07:18.247 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.248 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.248 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.248 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Jaeger agent container image", 2024-04-26T02:07:18.248 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.248 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.248 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.248 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.248 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_jaeger_collector": { 2024-04-26T02:07:18.249 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_jaeger_collector", 2024-04-26T02:07:18.249 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.249 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.249 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.249 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T02:07:18.249 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.249 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.249 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.250 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Jaeger collector container image", 2024-04-26T02:07:18.250 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.250 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.250 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.250 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.250 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_jaeger_query": { 2024-04-26T02:07:18.250 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_jaeger_query", 2024-04-26T02:07:18.250 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.251 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.251 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.251 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T02:07:18.251 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.251 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.251 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.251 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Jaeger query container image", 2024-04-26T02:07:18.251 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.251 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.252 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.252 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.252 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_keepalived": { 2024-04-26T02:07:18.252 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_keepalived", 2024-04-26T02:07:18.252 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.252 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.252 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.252 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T02:07:18.252 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.253 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.253 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.253 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Keepalived container image", 2024-04-26T02:07:18.253 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.253 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.253 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.253 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.253 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_loki": { 2024-04-26T02:07:18.253 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_loki", 2024-04-26T02:07:18.254 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.254 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.254 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.254 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T02:07:18.254 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.254 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.254 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.254 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Loki container image", 2024-04-26T02:07:18.255 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.255 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.255 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.255 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.255 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_node_exporter": { 2024-04-26T02:07:18.255 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_node_exporter", 2024-04-26T02:07:18.255 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.255 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.255 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.256 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T02:07:18.256 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.256 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.256 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.256 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:18.256 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.256 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.256 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.257 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.257 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_nvmeof": { 2024-04-26T02:07:18.257 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_nvmeof", 2024-04-26T02:07:18.257 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.257 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.257 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.257 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T02:07:18.257 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.257 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.258 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.258 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Nvme-of container image", 2024-04-26T02:07:18.258 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.258 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.258 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.258 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.258 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_prometheus": { 2024-04-26T02:07:18.258 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_prometheus", 2024-04-26T02:07:18.259 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.259 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.259 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.259 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T02:07:18.259 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.259 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.259 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.259 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:18.260 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.260 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.260 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.260 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.260 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_promtail": { 2024-04-26T02:07:18.260 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_promtail", 2024-04-26T02:07:18.260 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.260 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.260 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.261 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T02:07:18.261 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.261 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.261 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.261 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Promtail container image", 2024-04-26T02:07:18.261 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.261 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.261 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.261 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.262 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_samba": { 2024-04-26T02:07:18.262 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_samba", 2024-04-26T02:07:18.262 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.262 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.262 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.262 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T02:07:18.262 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.262 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.263 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.263 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Samba/SMB container image", 2024-04-26T02:07:18.263 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.263 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.263 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.263 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.263 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image_snmp_gateway": { 2024-04-26T02:07:18.263 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_image_snmp_gateway", 2024-04-26T02:07:18.264 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.264 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.264 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.264 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T02:07:18.264 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.264 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.264 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.264 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "SNMP Gateway container image", 2024-04-26T02:07:18.265 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.265 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.265 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.265 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.265 INFO:teuthology.orchestra.run.smithi006.stdout: "container_init": { 2024-04-26T02:07:18.265 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "container_init", 2024-04-26T02:07:18.265 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.265 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.266 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.266 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.266 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.266 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.266 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.266 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T02:07:18.266 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.266 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.266 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.267 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.267 INFO:teuthology.orchestra.run.smithi006.stdout: "daemon_cache_timeout": { 2024-04-26T02:07:18.267 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "daemon_cache_timeout", 2024-04-26T02:07:18.267 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.267 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.267 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.267 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:18.267 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.267 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.267 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.268 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T02:07:18.268 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.268 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.268 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.268 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.268 INFO:teuthology.orchestra.run.smithi006.stdout: "default_cephadm_command_timeout": { 2024-04-26T02:07:18.268 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T02:07:18.268 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.268 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.269 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.269 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "900", 2024-04-26T02:07:18.269 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.269 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.269 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.269 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T02:07:18.269 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.269 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.270 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.270 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.270 INFO:teuthology.orchestra.run.smithi006.stdout: "default_registry": { 2024-04-26T02:07:18.270 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "default_registry", 2024-04-26T02:07:18.270 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.270 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.270 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.270 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "docker.io", 2024-04-26T02:07:18.271 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.271 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.271 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.271 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T02:07:18.271 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.271 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.271 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.271 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.272 INFO:teuthology.orchestra.run.smithi006.stdout: "device_cache_timeout": { 2024-04-26T02:07:18.272 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "device_cache_timeout", 2024-04-26T02:07:18.272 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.272 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.272 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.272 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "1800", 2024-04-26T02:07:18.272 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.272 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.272 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.272 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "seconds to cache device inventory", 2024-04-26T02:07:18.273 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.273 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.273 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.273 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.273 INFO:teuthology.orchestra.run.smithi006.stdout: "device_enhanced_scan": { 2024-04-26T02:07:18.273 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "device_enhanced_scan", 2024-04-26T02:07:18.273 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.273 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.274 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.274 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.274 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.274 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.274 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.274 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T02:07:18.274 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.274 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.275 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.275 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.275 INFO:teuthology.orchestra.run.smithi006.stdout: "facts_cache_timeout": { 2024-04-26T02:07:18.275 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "facts_cache_timeout", 2024-04-26T02:07:18.275 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.275 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.275 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.275 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:18.275 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.275 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.276 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.276 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "seconds to cache host facts data", 2024-04-26T02:07:18.276 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.276 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.276 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.276 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.276 INFO:teuthology.orchestra.run.smithi006.stdout: "grafana_dashboards_path": { 2024-04-26T02:07:18.276 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "grafana_dashboards_path", 2024-04-26T02:07:18.276 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.277 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.277 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.277 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T02:07:18.277 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.277 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.277 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.277 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T02:07:18.277 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.278 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.278 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.278 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.278 INFO:teuthology.orchestra.run.smithi006.stdout: "host_check_interval": { 2024-04-26T02:07:18.278 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "host_check_interval", 2024-04-26T02:07:18.278 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.278 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.278 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.279 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:18.279 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.279 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.279 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.279 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to perform a host check", 2024-04-26T02:07:18.279 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.279 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.279 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.279 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.280 INFO:teuthology.orchestra.run.smithi006.stdout: "hw_monitoring": { 2024-04-26T02:07:18.280 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "hw_monitoring", 2024-04-26T02:07:18.280 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.280 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.280 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.280 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.280 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.281 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.281 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.281 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T02:07:18.281 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.281 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.281 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.281 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.281 INFO:teuthology.orchestra.run.smithi006.stdout: "inventory_list_all": { 2024-04-26T02:07:18.282 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "inventory_list_all", 2024-04-26T02:07:18.282 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.282 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.282 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.282 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.282 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.282 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.282 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.283 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T02:07:18.283 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.283 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.283 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.283 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.283 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.283 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.284 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.284 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.284 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.284 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.284 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.284 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.284 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.284 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.285 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.285 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.285 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.285 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.285 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.285 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.285 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.285 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.285 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.286 INFO:teuthology.orchestra.run.smithi006.stdout: "log_refresh_metadata": { 2024-04-26T02:07:18.286 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_refresh_metadata", 2024-04-26T02:07:18.286 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.286 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.286 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.286 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.286 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.287 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.287 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.287 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T02:07:18.287 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.287 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.287 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.287 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.288 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.288 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.288 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.288 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.288 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.288 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.288 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.288 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.288 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T02:07:18.288 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.289 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.289 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.289 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.289 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.289 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.289 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.289 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.289 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.290 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.290 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.290 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.290 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.290 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.290 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.290 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.290 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.290 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.291 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.291 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.291 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.291 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.291 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.291 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.291 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.292 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.292 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.292 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.292 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.292 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.292 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.292 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.292 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.292 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.293 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.293 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.293 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.293 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.293 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.293 INFO:teuthology.orchestra.run.smithi006.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T02:07:18.293 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T02:07:18.293 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.293 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.294 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.294 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.294 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.294 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.294 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.294 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T02:07:18.294 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.295 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.295 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.295 INFO:teuthology.orchestra.run.smithi006.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T02:07:18.295 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T02:07:18.295 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.295 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.295 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.295 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "*", 2024-04-26T02:07:18.296 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.296 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.296 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.296 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T02:07:18.296 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.296 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.296 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.296 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.297 INFO:teuthology.orchestra.run.smithi006.stdout: "max_count_per_host": { 2024-04-26T02:07:18.297 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_count_per_host", 2024-04-26T02:07:18.297 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.297 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.297 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.297 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:18.297 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.297 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.297 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.298 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "max number of daemons per service per host", 2024-04-26T02:07:18.298 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.298 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.298 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.298 INFO:teuthology.orchestra.run.smithi006.stdout: "max_osd_draining_count": { 2024-04-26T02:07:18.298 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_osd_draining_count", 2024-04-26T02:07:18.298 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.299 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.299 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.299 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:18.299 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.299 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.299 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.299 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T02:07:18.299 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.300 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.300 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.300 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.300 INFO:teuthology.orchestra.run.smithi006.stdout: "migration_current": { 2024-04-26T02:07:18.300 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "migration_current", 2024-04-26T02:07:18.300 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.300 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.300 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.300 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.301 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.301 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.301 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.301 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "internal - do not modify", 2024-04-26T02:07:18.301 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.301 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.301 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.301 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.301 INFO:teuthology.orchestra.run.smithi006.stdout: "mode": { 2024-04-26T02:07:18.302 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mode", 2024-04-26T02:07:18.302 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.302 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.302 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.302 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "root", 2024-04-26T02:07:18.302 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.302 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.303 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.303 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm-package", 2024-04-26T02:07:18.303 INFO:teuthology.orchestra.run.smithi006.stdout: "root" 2024-04-26T02:07:18.303 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.303 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T02:07:18.303 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.303 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.303 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.303 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.304 INFO:teuthology.orchestra.run.smithi006.stdout: "oob_default_addr": { 2024-04-26T02:07:18.304 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "oob_default_addr", 2024-04-26T02:07:18.304 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.304 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.304 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.304 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "169.254.1.1", 2024-04-26T02:07:18.304 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.304 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.304 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.305 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T02:07:18.305 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.305 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.305 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.305 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.305 INFO:teuthology.orchestra.run.smithi006.stdout: "prometheus_alerts_path": { 2024-04-26T02:07:18.305 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "prometheus_alerts_path", 2024-04-26T02:07:18.305 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.305 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.306 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.306 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T02:07:18.306 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.306 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.306 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.306 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T02:07:18.306 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.306 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.306 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: "registry_insecure": { 2024-04-26T02:07:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "registry_insecure", 2024-04-26T02:07:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.307 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T02:07:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "registry_password": { 2024-04-26T02:07:18.308 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "registry_password", 2024-04-26T02:07:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T02:07:18.309 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "registry_url": { 2024-04-26T02:07:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "registry_url", 2024-04-26T02:07:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T02:07:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.311 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "registry_username": { 2024-04-26T02:07:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "registry_username", 2024-04-26T02:07:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.312 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T02:07:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "secure_monitoring_stack": { 2024-04-26T02:07:18.313 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "secure_monitoring_stack", 2024-04-26T02:07:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T02:07:18.314 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "service_discovery_port": { 2024-04-26T02:07:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "service_discovery_port", 2024-04-26T02:07:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.315 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8765", 2024-04-26T02:07:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "cephadm service discovery port", 2024-04-26T02:07:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "ssh_config_file": { 2024-04-26T02:07:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ssh_config_file", 2024-04-26T02:07:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.317 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T02:07:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "use_agent": { 2024-04-26T02:07:18.318 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "use_agent", 2024-04-26T02:07:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T02:07:18.319 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: "use_repo_digest": { 2024-04-26T02:07:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "use_repo_digest", 2024-04-26T02:07:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T02:07:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_on_failed_host_check": { 2024-04-26T02:07:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_on_failed_host_check", 2024-04-26T02:07:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.322 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T02:07:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_on_stray_daemons": { 2024-04-26T02:07:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_on_stray_daemons", 2024-04-26T02:07:18.323 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T02:07:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_on_stray_hosts": { 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_on_stray_hosts", 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T02:07:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.326 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crash", 2024-04-26T02:07:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.328 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.329 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.330 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.331 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.332 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.333 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.334 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.335 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "retain_interval": { 2024-04-26T02:07:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "retain_interval", 2024-04-26T02:07:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "31536000", 2024-04-26T02:07:18.336 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T02:07:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_recent_interval": { 2024-04-26T02:07:18.337 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_recent_interval", 2024-04-26T02:07:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "1209600", 2024-04-26T02:07:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.338 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T02:07:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.339 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "dashboard", 2024-04-26T02:07:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T02:07:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T02:07:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.340 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T02:07:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T02:07:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.342 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T02:07:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T02:07:18.343 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.344 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "AUDIT_API_ENABLED": { 2024-04-26T02:07:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T02:07:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.345 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T02:07:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T02:07:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.347 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.348 INFO:teuthology.orchestra.run.smithi006.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T02:07:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T02:07:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.349 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T02:07:18.350 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T02:07:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.351 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T02:07:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T02:07:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.352 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.353 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T02:07:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T02:07:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.354 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T02:07:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T02:07:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.356 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T02:07:18.357 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T02:07:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.358 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T02:07:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T02:07:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.360 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T02:07:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T02:07:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.361 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T02:07:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T02:07:18.362 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.363 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T02:07:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T02:07:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.364 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "admin", 2024-04-26T02:07:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T02:07:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T02:07:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.366 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_API_URL": { 2024-04-26T02:07:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_API_URL", 2024-04-26T02:07:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.367 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.368 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T02:07:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T02:07:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "admin", 2024-04-26T02:07:18.369 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.370 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T02:07:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T02:07:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.371 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T02:07:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T02:07:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T02:07:18.373 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T02:07:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.374 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T02:07:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T02:07:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.375 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T02:07:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T02:07:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "[]", 2024-04-26T02:07:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.377 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T02:07:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T02:07:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.378 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "{}", 2024-04-26T02:07:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.379 INFO:teuthology.orchestra.run.smithi006.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T02:07:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T02:07:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.380 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T02:07:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T02:07:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.381 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.382 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T02:07:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T02:07:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.383 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T02:07:18.384 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T02:07:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.385 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T02:07:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T02:07:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.386 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.387 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T02:07:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T02:07:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.388 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T02:07:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T02:07:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.389 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.390 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T02:07:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T02:07:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.391 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T02:07:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T02:07:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.393 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T02:07:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T02:07:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.394 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.395 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T02:07:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T02:07:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.396 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T02:07:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.397 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T02:07:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T02:07:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T02:07:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T02:07:18.399 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8", 2024-04-26T02:07:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.400 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T02:07:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T02:07:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "45", 2024-04-26T02:07:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.401 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.402 INFO:teuthology.orchestra.run.smithi006.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T02:07:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T02:07:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.403 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T02:07:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T02:07:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.404 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "admin", 2024-04-26T02:07:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T02:07:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T02:07:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.406 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T02:07:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T02:07:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.407 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.408 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.408 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.408 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.408 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.408 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.408 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.408 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.408 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.408 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.409 INFO:teuthology.orchestra.run.smithi006.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T02:07:18.409 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T02:07:18.409 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.409 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.409 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.409 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.409 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.409 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.410 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.410 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.410 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.410 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.410 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.410 INFO:teuthology.orchestra.run.smithi006.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T02:07:18.411 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T02:07:18.411 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.411 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.411 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.411 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:18.411 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.411 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.411 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.411 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.412 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.412 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.412 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.412 INFO:teuthology.orchestra.run.smithi006.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T02:07:18.412 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T02:07:18.412 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.412 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.413 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.413 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:18.413 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.413 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.413 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.413 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.413 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.413 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.413 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.414 INFO:teuthology.orchestra.run.smithi006.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T02:07:18.414 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T02:07:18.414 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.414 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.414 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.414 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5", 2024-04-26T02:07:18.414 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.415 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.415 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.415 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.415 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.415 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.415 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.415 INFO:teuthology.orchestra.run.smithi006.stdout: "cross_origin_url": { 2024-04-26T02:07:18.415 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cross_origin_url", 2024-04-26T02:07:18.415 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.416 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.416 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.416 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.416 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.416 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.416 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.416 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.416 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.416 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.417 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.417 INFO:teuthology.orchestra.run.smithi006.stdout: "crt_file": { 2024-04-26T02:07:18.417 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "crt_file", 2024-04-26T02:07:18.417 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.417 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.417 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.417 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.417 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.418 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.418 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.418 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.418 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.418 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.418 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.418 INFO:teuthology.orchestra.run.smithi006.stdout: "debug": { 2024-04-26T02:07:18.418 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "debug", 2024-04-26T02:07:18.419 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.419 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.419 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.419 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.419 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.419 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.419 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.419 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Enable/disable debug options", 2024-04-26T02:07:18.419 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.420 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.420 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.420 INFO:teuthology.orchestra.run.smithi006.stdout: "jwt_token_ttl": { 2024-04-26T02:07:18.420 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "jwt_token_ttl", 2024-04-26T02:07:18.420 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.420 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.420 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.420 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "28800", 2024-04-26T02:07:18.420 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.421 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.421 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.421 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.421 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.421 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.421 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.421 INFO:teuthology.orchestra.run.smithi006.stdout: "key_file": { 2024-04-26T02:07:18.421 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "key_file", 2024-04-26T02:07:18.422 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.422 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.422 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.422 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.422 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.422 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.422 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.422 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.422 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.423 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.423 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.423 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.423 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.423 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.423 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.423 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.423 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.423 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.423 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.424 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.424 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.424 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.424 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.424 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.424 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.424 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.424 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.424 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.424 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.425 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.425 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.425 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.425 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.425 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.425 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.425 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.425 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.426 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.426 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.426 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.426 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.426 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.426 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.426 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.426 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.427 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.427 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.427 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.427 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.427 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.427 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.427 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.427 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.428 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.428 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.428 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.428 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.428 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.428 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.428 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.428 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.429 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.429 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.429 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.429 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.429 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.429 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.429 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.429 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.430 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.430 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.430 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.430 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.430 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.430 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.430 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.430 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.431 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.431 INFO:teuthology.orchestra.run.smithi006.stdout: "motd": { 2024-04-26T02:07:18.431 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "motd", 2024-04-26T02:07:18.431 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.431 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.431 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.431 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.431 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.432 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.432 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.432 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "The message of the day", 2024-04-26T02:07:18.432 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.432 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.432 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.432 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.432 INFO:teuthology.orchestra.run.smithi006.stdout: "redirect_resolve_ip_addr": { 2024-04-26T02:07:18.432 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T02:07:18.432 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.433 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.433 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.433 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.433 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.433 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.433 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.433 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.433 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.433 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.434 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.434 INFO:teuthology.orchestra.run.smithi006.stdout: "server_addr": { 2024-04-26T02:07:18.434 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_addr", 2024-04-26T02:07:18.434 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.434 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.434 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.434 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "::", 2024-04-26T02:07:18.434 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.435 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.435 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.435 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.435 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.435 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.435 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.435 INFO:teuthology.orchestra.run.smithi006.stdout: "server_port": { 2024-04-26T02:07:18.435 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_port", 2024-04-26T02:07:18.436 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.436 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.436 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.436 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8080", 2024-04-26T02:07:18.436 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.436 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.436 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.436 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.436 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: "ssl": { 2024-04-26T02:07:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ssl", 2024-04-26T02:07:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "ssl_server_port": { 2024-04-26T02:07:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ssl_server_port", 2024-04-26T02:07:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8443", 2024-04-26T02:07:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "standby_behaviour": { 2024-04-26T02:07:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "standby_behaviour", 2024-04-26T02:07:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "redirect", 2024-04-26T02:07:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "redirect" 2024-04-26T02:07:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "standby_error_status_code": { 2024-04-26T02:07:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "standby_error_status_code", 2024-04-26T02:07:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "500", 2024-04-26T02:07:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "400", 2024-04-26T02:07:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "599", 2024-04-26T02:07:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "url_prefix": { 2024-04-26T02:07:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "url_prefix", 2024-04-26T02:07:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "devicehealth", 2024-04-26T02:07:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "enable_monitoring": { 2024-04-26T02:07:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "enable_monitoring", 2024-04-26T02:07:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "monitor device health metrics", 2024-04-26T02:07:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.450 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.450 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.450 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.450 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.450 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.450 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.450 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.450 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.451 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.451 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.451 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.451 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.451 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.451 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.451 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.451 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.451 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.452 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.452 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.452 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.452 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.452 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.452 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.452 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.452 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.452 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.453 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.453 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.453 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.453 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.453 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.453 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.453 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.453 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.454 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.454 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.454 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.454 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.454 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.454 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.454 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.454 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.455 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.455 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.455 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.455 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.455 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.455 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.455 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.455 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.455 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.455 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.456 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.456 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.456 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.456 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.456 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.456 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.456 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.456 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.456 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.456 INFO:teuthology.orchestra.run.smithi006.stdout: "mark_out_threshold": { 2024-04-26T02:07:18.457 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mark_out_threshold", 2024-04-26T02:07:18.457 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.457 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.457 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.457 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "2419200", 2024-04-26T02:07:18.457 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.457 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.457 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.457 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T02:07:18.458 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.458 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.458 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.458 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.458 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": { 2024-04-26T02:07:18.458 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "pool_name", 2024-04-26T02:07:18.458 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.459 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.459 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.459 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "device_health_metrics", 2024-04-26T02:07:18.459 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.459 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.459 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.459 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T02:07:18.459 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.459 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.460 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.460 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.460 INFO:teuthology.orchestra.run.smithi006.stdout: "retention_period": { 2024-04-26T02:07:18.460 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "retention_period", 2024-04-26T02:07:18.460 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.460 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.460 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.460 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "15552000", 2024-04-26T02:07:18.461 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.461 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.461 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.461 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how long to retain device health metrics", 2024-04-26T02:07:18.461 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.461 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.462 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.462 INFO:teuthology.orchestra.run.smithi006.stdout: "scrape_frequency": { 2024-04-26T02:07:18.462 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "scrape_frequency", 2024-04-26T02:07:18.462 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.462 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.462 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.462 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "86400", 2024-04-26T02:07:18.463 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.463 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.463 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.463 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T02:07:18.463 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.463 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.463 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.463 INFO:teuthology.orchestra.run.smithi006.stdout: "self_heal": { 2024-04-26T02:07:18.464 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "self_heal", 2024-04-26T02:07:18.464 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.464 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.464 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.464 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.464 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.464 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.464 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.465 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T02:07:18.465 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.465 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.465 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.465 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.465 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:18.465 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:18.465 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.465 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.466 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.466 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:18.466 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.466 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.466 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.466 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T02:07:18.466 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.466 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.466 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.467 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.467 INFO:teuthology.orchestra.run.smithi006.stdout: "warn_threshold": { 2024-04-26T02:07:18.467 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "warn_threshold", 2024-04-26T02:07:18.467 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.467 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.467 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.467 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "7257600", 2024-04-26T02:07:18.467 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.467 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.468 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.468 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T02:07:18.468 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.468 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.468 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.468 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.468 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.469 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "diskprediction_local", 2024-04-26T02:07:18.469 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.469 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.469 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.469 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.469 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.469 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.469 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.470 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.470 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.470 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.470 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.470 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.470 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.470 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.470 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.471 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.471 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.471 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.471 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.471 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.471 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.471 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.471 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.472 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.472 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.472 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.472 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.472 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.472 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.472 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.472 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.473 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.473 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.473 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.473 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.473 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.473 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.473 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.473 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.473 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.473 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.475 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.475 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.475 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.475 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.475 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.475 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.475 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.475 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.475 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.476 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.476 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.476 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.476 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.476 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.476 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.476 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.476 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.477 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.477 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.477 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.477 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.477 INFO:teuthology.orchestra.run.smithi006.stdout: "predict_interval": { 2024-04-26T02:07:18.477 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "predict_interval", 2024-04-26T02:07:18.477 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.478 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.478 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.478 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "86400", 2024-04-26T02:07:18.478 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.478 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.478 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.478 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.478 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.478 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.479 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.479 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.479 INFO:teuthology.orchestra.run.smithi006.stdout: "predictor_model": { 2024-04-26T02:07:18.479 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "predictor_model", 2024-04-26T02:07:18.479 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.479 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.479 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.480 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "prophetstor", 2024-04-26T02:07:18.480 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.480 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.480 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.480 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.480 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.480 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.480 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.480 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.481 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:18.481 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:18.481 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.481 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.481 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.481 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "600", 2024-04-26T02:07:18.481 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.481 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.481 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.481 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.482 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.482 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.482 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.482 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.482 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.482 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "influx", 2024-04-26T02:07:18.482 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": false, 2024-04-26T02:07:18.483 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "influxdb python module not found", 2024-04-26T02:07:18.483 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.483 INFO:teuthology.orchestra.run.smithi006.stdout: "batch_size": { 2024-04-26T02:07:18.483 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "batch_size", 2024-04-26T02:07:18.483 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.483 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.483 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.483 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5000", 2024-04-26T02:07:18.483 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.484 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.484 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.484 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T02:07:18.484 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.484 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.484 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.484 INFO:teuthology.orchestra.run.smithi006.stdout: "database": { 2024-04-26T02:07:18.484 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "database", 2024-04-26T02:07:18.485 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.485 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.485 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.485 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "ceph", 2024-04-26T02:07:18.485 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.485 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.485 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.485 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T02:07:18.485 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.486 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.486 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.486 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.486 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": { 2024-04-26T02:07:18.486 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "hostname", 2024-04-26T02:07:18.486 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.486 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.486 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.487 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.487 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.487 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.487 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.487 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "InfluxDB server hostname", 2024-04-26T02:07:18.487 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.487 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.488 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.488 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:18.488 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:18.488 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.488 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.488 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.488 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "30", 2024-04-26T02:07:18.488 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "5", 2024-04-26T02:07:18.489 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.489 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.489 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T02:07:18.489 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.489 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.489 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.489 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.489 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.489 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.490 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.490 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.490 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.490 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.490 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.490 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.490 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.490 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.491 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.491 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.491 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.491 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.491 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.491 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.491 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.491 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.492 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.492 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.492 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.492 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.492 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.492 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.492 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.492 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.493 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.493 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.493 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.493 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.493 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.493 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.493 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.493 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.494 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.494 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.494 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.494 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.494 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.494 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.494 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.495 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.495 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.495 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.495 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.495 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.495 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.495 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.495 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.495 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.496 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.496 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.496 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.496 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.496 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.496 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.496 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.496 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.497 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.497 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.497 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.497 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.497 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.497 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.497 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.497 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.498 INFO:teuthology.orchestra.run.smithi006.stdout: "password": { 2024-04-26T02:07:18.498 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "password", 2024-04-26T02:07:18.498 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.498 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.498 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.498 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.498 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.499 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.499 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.499 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "password of InfluxDB server user", 2024-04-26T02:07:18.499 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.499 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.499 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.499 INFO:teuthology.orchestra.run.smithi006.stdout: "port": { 2024-04-26T02:07:18.499 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "port", 2024-04-26T02:07:18.500 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.500 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.500 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.500 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "8086", 2024-04-26T02:07:18.500 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.500 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.500 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.500 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "InfluxDB server port", 2024-04-26T02:07:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: "ssl": { 2024-04-26T02:07:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ssl", 2024-04-26T02:07:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.501 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.502 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "false", 2024-04-26T02:07:18.502 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.502 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.502 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.502 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T02:07:18.502 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.502 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.502 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.503 INFO:teuthology.orchestra.run.smithi006.stdout: "threads": { 2024-04-26T02:07:18.503 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "threads", 2024-04-26T02:07:18.503 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.503 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.503 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.503 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5", 2024-04-26T02:07:18.503 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1", 2024-04-26T02:07:18.503 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "32", 2024-04-26T02:07:18.504 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.504 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T02:07:18.504 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.504 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.504 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.504 INFO:teuthology.orchestra.run.smithi006.stdout: "username": { 2024-04-26T02:07:18.504 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "username", 2024-04-26T02:07:18.504 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.505 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.505 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.505 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.505 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.505 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.505 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.505 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "username of InfluxDB server user", 2024-04-26T02:07:18.505 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.506 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.506 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.506 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.506 INFO:teuthology.orchestra.run.smithi006.stdout: "verify_ssl": { 2024-04-26T02:07:18.506 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "verify_ssl", 2024-04-26T02:07:18.506 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.506 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.506 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.506 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "true", 2024-04-26T02:07:18.507 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.507 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.507 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.507 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T02:07:18.507 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.507 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.507 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.507 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.507 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.508 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.508 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.508 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "insights", 2024-04-26T02:07:18.508 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.508 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.508 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.508 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.508 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.508 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.509 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.509 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.509 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.509 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.509 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.509 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.510 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.510 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.510 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.510 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.510 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.510 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.510 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.510 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.511 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.511 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.511 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.511 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.511 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.511 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.511 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.511 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.511 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.512 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.512 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.512 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.512 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.512 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.512 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.512 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.512 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.512 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.512 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.513 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.513 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.513 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.513 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.513 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.513 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.513 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.513 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.513 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.514 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.514 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.514 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.514 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.514 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.514 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.514 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.514 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.515 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.515 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.515 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.515 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.515 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.515 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.515 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.515 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.515 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.515 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.516 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.516 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.516 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.516 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.516 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.516 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.516 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.516 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.516 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.517 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "iostat", 2024-04-26T02:07:18.517 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.517 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.517 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.517 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.517 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.517 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.517 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.518 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.518 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.518 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.518 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.518 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.518 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.518 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.518 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.518 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.519 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.519 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.519 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.519 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.519 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.519 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.519 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.520 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.520 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.520 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.520 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.520 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.520 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.520 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.520 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.521 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.521 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.521 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.521 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.521 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.521 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.521 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.521 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.521 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.522 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.522 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.522 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.522 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.522 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.522 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.522 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.522 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.523 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.523 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.523 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.523 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.523 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.523 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.523 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.523 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.523 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.523 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.524 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.524 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.524 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.524 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.524 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.524 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.524 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.524 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.524 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.525 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.525 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.525 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.525 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.525 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.525 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.525 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.525 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "k8sevents", 2024-04-26T02:07:18.526 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.526 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.526 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.526 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_event_retention_days": { 2024-04-26T02:07:18.526 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "ceph_event_retention_days", 2024-04-26T02:07:18.526 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.526 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.526 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.527 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "7", 2024-04-26T02:07:18.527 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.527 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.527 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.527 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T02:07:18.527 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.527 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.527 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.528 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.528 INFO:teuthology.orchestra.run.smithi006.stdout: "config_check_secs": { 2024-04-26T02:07:18.528 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "config_check_secs", 2024-04-26T02:07:18.528 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.528 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.528 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.528 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:18.528 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "10", 2024-04-26T02:07:18.529 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.529 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.529 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T02:07:18.529 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.529 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.529 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.529 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.529 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.530 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.530 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.530 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.530 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.530 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.530 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.530 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.530 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.531 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.531 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.531 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.531 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.531 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.531 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.531 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.531 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.532 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.532 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.532 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.532 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.532 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.532 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.532 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.532 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.533 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.533 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.533 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.533 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.533 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.533 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.533 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.534 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.534 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.534 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.534 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.534 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.534 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.534 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.534 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.535 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.535 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.535 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.535 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.535 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.535 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.535 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.535 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.536 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.536 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.536 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.536 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.536 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.536 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.536 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.537 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.537 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.537 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.537 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.537 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.537 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.537 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.537 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.537 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.538 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.538 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.538 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.538 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.538 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.538 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.538 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "localpool", 2024-04-26T02:07:18.538 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.538 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.539 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.539 INFO:teuthology.orchestra.run.smithi006.stdout: "failure_domain": { 2024-04-26T02:07:18.539 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "failure_domain", 2024-04-26T02:07:18.539 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.539 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.539 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.539 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "host", 2024-04-26T02:07:18.539 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.540 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.540 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.540 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "failure domain for any created local pool", 2024-04-26T02:07:18.540 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T02:07:18.540 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.540 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.540 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.540 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.540 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.541 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.541 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.541 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.541 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.541 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.541 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.541 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.541 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.542 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.542 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.542 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.542 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.542 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.542 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.542 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.542 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.542 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.543 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.543 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.543 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.543 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.543 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.543 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.543 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.544 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.544 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.544 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.544 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.544 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.544 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.544 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.544 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.545 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.545 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.545 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.545 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.545 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.545 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.545 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.545 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.545 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.546 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.546 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.546 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.546 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.546 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.546 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.546 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.546 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.547 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.547 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.547 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.547 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.547 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.547 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.547 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.548 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.548 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.548 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.548 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.548 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.548 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.548 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.548 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.549 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": { 2024-04-26T02:07:18.549 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "min_size", 2024-04-26T02:07:18.549 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.549 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.549 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.549 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.549 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.549 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.549 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.550 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "default min_size for any created local pool", 2024-04-26T02:07:18.550 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T02:07:18.550 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.550 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.550 INFO:teuthology.orchestra.run.smithi006.stdout: "num_rep": { 2024-04-26T02:07:18.550 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "num_rep", 2024-04-26T02:07:18.550 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.551 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.551 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.551 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "3", 2024-04-26T02:07:18.551 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.551 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.551 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.551 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "default replica count for any created local pool", 2024-04-26T02:07:18.551 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.551 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.552 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.552 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": { 2024-04-26T02:07:18.552 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "pg_num", 2024-04-26T02:07:18.552 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.552 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.552 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.552 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "128", 2024-04-26T02:07:18.552 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.553 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.553 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.553 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T02:07:18.553 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.553 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.553 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.553 INFO:teuthology.orchestra.run.smithi006.stdout: "prefix": { 2024-04-26T02:07:18.553 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "prefix", 2024-04-26T02:07:18.554 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.554 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.554 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.554 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.554 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.554 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.554 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.554 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "name prefix for any created local pool", 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi006.stdout: "subtree": { 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "subtree", 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.556 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.556 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "rack", 2024-04-26T02:07:18.556 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.556 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.556 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.556 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T02:07:18.556 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T02:07:18.556 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mds_autoscaler", 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.561 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.561 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.561 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.561 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.561 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.561 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.561 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.561 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.561 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.562 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.562 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.562 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.562 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.562 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.562 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.562 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.563 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.563 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.563 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.563 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.563 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.563 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.563 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.563 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.563 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.564 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.564 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.564 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.564 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.564 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.564 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.564 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.565 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.565 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.565 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.565 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.565 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.565 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.565 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.565 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.565 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.566 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.566 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.566 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.566 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mirroring", 2024-04-26T02:07:18.566 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.566 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.567 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.567 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.567 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.567 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.567 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.567 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.567 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.567 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.567 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.568 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.568 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.568 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.568 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.568 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.568 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.568 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.568 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.569 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.569 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.569 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.569 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.569 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.569 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.569 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.569 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.570 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.570 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.570 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.570 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.570 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.570 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.570 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.570 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.571 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.571 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.571 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.571 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.571 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.571 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.571 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.571 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.572 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.572 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.572 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.572 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.572 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.572 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.572 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.572 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.572 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.573 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.573 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.573 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.573 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.573 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.573 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.573 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.574 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.574 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.574 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.574 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.574 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.574 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.574 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.574 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.574 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.575 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.575 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.575 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.575 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "nfs", 2024-04-26T02:07:18.575 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.575 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.575 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.576 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.576 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.576 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.576 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.576 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.576 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.576 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.576 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.576 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.577 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.577 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.577 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.577 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.577 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.577 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.577 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.577 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.577 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.578 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.578 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.578 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.578 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.578 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.578 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.579 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.579 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.579 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.579 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.579 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.579 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.579 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.579 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.580 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.580 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.580 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.580 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.580 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.580 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.580 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.581 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.581 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.581 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.581 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.581 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.581 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.581 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.581 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.582 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.582 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.582 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.582 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.582 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.582 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.582 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.583 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.583 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.583 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.583 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.583 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.583 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.583 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.583 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.583 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.584 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.584 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.584 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.584 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.584 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.584 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "orchestrator", 2024-04-26T02:07:18.585 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.585 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.585 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.585 INFO:teuthology.orchestra.run.smithi006.stdout: "fail_fs": { 2024-04-26T02:07:18.585 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "fail_fs", 2024-04-26T02:07:18.585 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.585 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.585 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.586 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.586 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.586 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.586 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.586 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T02:07:18.586 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.586 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.586 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.587 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.587 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.587 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.587 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.587 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.587 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.587 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.588 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.588 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.588 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.588 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.588 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.588 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.588 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.588 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.589 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.589 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.589 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.589 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.589 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.589 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.589 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.589 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.590 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.590 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.590 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.590 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.590 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.590 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.590 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.590 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.591 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.591 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.591 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.591 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.591 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.591 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.591 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.592 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.592 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.592 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.592 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.592 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.592 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.592 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.592 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.592 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.592 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.593 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.593 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.593 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.593 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.593 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.593 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.593 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.593 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "orchestrator": { 2024-04-26T02:07:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "orchestrator", 2024-04-26T02:07:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm", 2024-04-26T02:07:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "rook", 2024-04-26T02:07:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "test_orchestrator" 2024-04-26T02:07:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Orchestrator backend", 2024-04-26T02:07:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "osd_perf_query", 2024-04-26T02:07:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "osd_support", 2024-04-26T02:07:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.607 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.607 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.607 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.607 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.607 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.607 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.607 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.607 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.607 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.608 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.608 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.608 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.608 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.608 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.608 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.608 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.608 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.608 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.608 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.609 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.609 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.609 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.609 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.609 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.609 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.609 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.609 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.610 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.610 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.610 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.610 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.610 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.610 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.610 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.610 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.611 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.611 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.611 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.611 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.611 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.611 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.611 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.611 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.611 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.612 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.612 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.612 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.612 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.612 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.612 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.612 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.612 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.612 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.613 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.613 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.613 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.613 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.613 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.613 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.613 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.613 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.613 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.613 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.614 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.614 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.614 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.614 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.614 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.614 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "pg_autoscaler", 2024-04-26T02:07:18.614 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.615 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.615 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.615 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.615 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.615 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.615 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.615 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.615 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.615 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.616 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.616 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.616 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.616 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.616 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.616 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.616 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.616 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.616 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.616 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.617 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.617 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.617 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.617 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.617 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.617 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.617 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.617 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.617 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.618 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.618 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.618 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.618 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.618 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.618 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.618 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.618 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.619 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.619 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.619 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.619 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.619 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.619 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.619 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.619 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.619 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.620 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.620 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.620 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.620 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.620 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.620 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.620 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.620 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.620 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.621 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.621 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.621 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.621 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.621 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.621 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.621 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.621 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.622 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.622 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.622 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.622 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.622 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.622 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.622 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.623 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:18.623 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:18.623 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.623 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.623 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.623 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:18.623 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.623 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.624 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.624 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.624 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.624 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.624 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.624 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.624 INFO:teuthology.orchestra.run.smithi006.stdout: "threshold": { 2024-04-26T02:07:18.624 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "threshold", 2024-04-26T02:07:18.625 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:18.625 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.625 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.625 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "3.0", 2024-04-26T02:07:18.625 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1.0", 2024-04-26T02:07:18.625 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.625 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.625 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "scaling threshold", 2024-04-26T02:07:18.626 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T02:07:18.626 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.626 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.626 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.626 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.626 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "progress", 2024-04-26T02:07:18.626 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.627 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.627 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.627 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_pg_recovery_event": { 2024-04-26T02:07:18.627 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "allow_pg_recovery_event", 2024-04-26T02:07:18.627 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.627 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.628 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.628 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.628 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.628 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.628 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.628 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T02:07:18.628 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.628 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.628 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.629 INFO:teuthology.orchestra.run.smithi006.stdout: "enabled": { 2024-04-26T02:07:18.629 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "enabled", 2024-04-26T02:07:18.629 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.629 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.629 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.629 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.630 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.630 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.630 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.630 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.630 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.630 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.630 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.631 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.631 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.631 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.631 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.631 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.631 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.631 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.631 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.632 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.632 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.632 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.632 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.632 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.632 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.632 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.632 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.632 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.632 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.633 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.633 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.633 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.633 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.633 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.633 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.633 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.633 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.635 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.635 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.635 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.635 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.635 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.635 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.635 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.635 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.635 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.635 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.636 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.636 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.636 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.636 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.636 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.636 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.636 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.636 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.636 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.637 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.637 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.637 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.637 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.637 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.637 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.637 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.637 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.638 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.638 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.638 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.638 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.638 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.638 INFO:teuthology.orchestra.run.smithi006.stdout: "max_completed_events": { 2024-04-26T02:07:18.638 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_completed_events", 2024-04-26T02:07:18.638 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.639 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.639 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.639 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "50", 2024-04-26T02:07:18.639 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.639 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.639 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.639 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "number of past completed events to remember", 2024-04-26T02:07:18.639 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.640 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.640 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.640 INFO:teuthology.orchestra.run.smithi006.stdout: "sleep_interval": { 2024-04-26T02:07:18.640 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "sleep_interval", 2024-04-26T02:07:18.640 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.640 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.640 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.641 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "5", 2024-04-26T02:07:18.641 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.641 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.641 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.641 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "how long the module is going to sleep", 2024-04-26T02:07:18.641 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.641 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.641 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.642 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.642 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.642 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "prometheus", 2024-04-26T02:07:18.642 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.642 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.642 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.642 INFO:teuthology.orchestra.run.smithi006.stdout: "cache": { 2024-04-26T02:07:18.643 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cache", 2024-04-26T02:07:18.643 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.643 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.643 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.643 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.643 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.643 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.643 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.644 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.644 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.644 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.644 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.644 INFO:teuthology.orchestra.run.smithi006.stdout: "exclude_perf_counters": { 2024-04-26T02:07:18.644 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "exclude_perf_counters", 2024-04-26T02:07:18.644 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.644 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.644 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.645 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.645 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.645 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.645 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.645 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T02:07:18.645 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T02:07:18.645 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.646 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.646 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.646 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.646 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.646 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.646 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.647 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.647 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.647 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.647 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.647 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.647 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.647 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.647 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.647 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.648 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.648 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.648 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.648 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.648 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.648 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.648 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.648 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.649 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.649 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.649 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.649 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.649 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.649 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.649 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.649 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.650 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.650 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.650 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.650 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.650 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.650 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.650 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.650 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.651 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.651 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.651 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.651 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.651 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.651 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.651 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.651 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.652 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.652 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.652 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.652 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.652 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.652 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.652 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.652 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.653 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.653 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.653 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.653 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.653 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.653 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.653 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.653 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.654 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.654 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.654 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.654 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.654 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_stats_pools": { 2024-04-26T02:07:18.654 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rbd_stats_pools", 2024-04-26T02:07:18.654 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.655 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.655 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.655 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.655 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.655 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.655 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.655 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.655 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.656 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.656 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.656 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.656 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T02:07:18.656 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T02:07:18.656 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.656 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.656 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.657 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "300", 2024-04-26T02:07:18.657 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.657 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.657 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.657 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.657 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.657 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.657 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.658 INFO:teuthology.orchestra.run.smithi006.stdout: "scrape_interval": { 2024-04-26T02:07:18.658 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "scrape_interval", 2024-04-26T02:07:18.658 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:18.658 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.658 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.658 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "15.0", 2024-04-26T02:07:18.658 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.659 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.659 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.659 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.659 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.659 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.659 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.659 INFO:teuthology.orchestra.run.smithi006.stdout: "server_addr": { 2024-04-26T02:07:18.659 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_addr", 2024-04-26T02:07:18.661 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.661 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.661 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.661 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "::", 2024-04-26T02:07:18.662 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.662 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.662 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.662 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T02:07:18.662 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.662 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.662 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.662 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.663 INFO:teuthology.orchestra.run.smithi006.stdout: "server_port": { 2024-04-26T02:07:18.663 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_port", 2024-04-26T02:07:18.663 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.663 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.663 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.663 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "9283", 2024-04-26T02:07:18.663 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.663 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.664 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.664 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T02:07:18.664 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.664 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.664 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.664 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.664 INFO:teuthology.orchestra.run.smithi006.stdout: "stale_cache_strategy": { 2024-04-26T02:07:18.664 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "stale_cache_strategy", 2024-04-26T02:07:18.664 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.665 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.665 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.665 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "log", 2024-04-26T02:07:18.665 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.665 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.665 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.665 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.665 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.666 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.666 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.666 INFO:teuthology.orchestra.run.smithi006.stdout: "standby_behaviour": { 2024-04-26T02:07:18.666 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "standby_behaviour", 2024-04-26T02:07:18.666 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.666 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.666 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.667 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "default", 2024-04-26T02:07:18.667 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.667 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.667 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.667 INFO:teuthology.orchestra.run.smithi006.stdout: "default", 2024-04-26T02:07:18.667 INFO:teuthology.orchestra.run.smithi006.stdout: "error" 2024-04-26T02:07:18.667 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.667 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.667 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.668 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.668 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.668 INFO:teuthology.orchestra.run.smithi006.stdout: "standby_error_status_code": { 2024-04-26T02:07:18.668 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "standby_error_status_code", 2024-04-26T02:07:18.668 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.668 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.668 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.668 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "500", 2024-04-26T02:07:18.669 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "400", 2024-04-26T02:07:18.669 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "599", 2024-04-26T02:07:18.669 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.669 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.669 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.669 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.669 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.669 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.670 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.670 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.670 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rbd_support", 2024-04-26T02:07:18.670 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.670 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.670 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.670 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.671 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.671 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.671 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.671 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.671 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.671 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.671 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.671 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.671 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.672 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.672 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.672 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.672 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.672 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.672 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.672 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.672 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.672 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.672 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.673 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.673 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.673 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.673 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.673 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.673 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.673 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.673 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.673 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.673 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.674 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.674 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.674 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.674 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.674 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.674 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.674 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.674 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.674 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.675 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.675 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.675 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.675 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.675 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.675 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.675 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.675 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.675 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.676 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.676 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.676 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.676 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.676 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.676 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.676 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.676 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.676 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.677 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.677 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.677 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.677 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.677 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.677 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.677 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.677 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.677 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.677 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.678 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.678 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.678 INFO:teuthology.orchestra.run.smithi006.stdout: "max_concurrent_snap_create": { 2024-04-26T02:07:18.678 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_concurrent_snap_create", 2024-04-26T02:07:18.678 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.678 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.678 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.678 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10", 2024-04-26T02:07:18.678 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.679 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.679 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.679 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.679 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.679 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.679 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.679 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mirror_snapshot_schedule": { 2024-04-26T02:07:18.679 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T02:07:18.679 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.680 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.680 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.680 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.680 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.680 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.680 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.680 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.680 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.681 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.681 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.681 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.681 INFO:teuthology.orchestra.run.smithi006.stdout: "trash_purge_schedule": { 2024-04-26T02:07:18.681 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "trash_purge_schedule", 2024-04-26T02:07:18.681 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.681 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.681 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.681 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.682 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.682 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.682 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.682 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.682 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.682 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.682 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.682 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.683 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.683 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.683 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.683 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "restful", 2024-04-26T02:07:18.683 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.683 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.683 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.683 INFO:teuthology.orchestra.run.smithi006.stdout: "enable_auth": { 2024-04-26T02:07:18.683 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "enable_auth", 2024-04-26T02:07:18.683 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.684 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.684 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.684 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.684 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.684 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.684 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.684 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.684 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.684 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.684 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.685 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.685 INFO:teuthology.orchestra.run.smithi006.stdout: "key_file": { 2024-04-26T02:07:18.685 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "key_file", 2024-04-26T02:07:18.685 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.685 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.685 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.685 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.685 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.685 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.685 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.686 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.686 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.686 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.686 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.686 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.686 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.686 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.686 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.687 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.687 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.687 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.687 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.687 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.687 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.687 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.687 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.687 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.688 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.688 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.688 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.688 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.688 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.688 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.688 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.688 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.688 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.689 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.689 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.689 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.689 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.689 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.689 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.689 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.689 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.690 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.690 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.690 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.690 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.690 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.690 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.690 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.690 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.691 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.691 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.691 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.691 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.691 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.691 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.691 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.691 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.691 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.691 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.692 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.692 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.692 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.692 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.692 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.692 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.692 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.692 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.692 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.693 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.693 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.693 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.693 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.693 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.693 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.693 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.693 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.693 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.693 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.694 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.694 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.694 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.694 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.694 INFO:teuthology.orchestra.run.smithi006.stdout: "server_addr": { 2024-04-26T02:07:18.694 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_addr", 2024-04-26T02:07:18.694 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.694 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.695 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.695 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.695 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.695 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.695 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.695 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.695 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.695 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.696 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.696 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.696 INFO:teuthology.orchestra.run.smithi006.stdout: "server_port": { 2024-04-26T02:07:18.696 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "server_port", 2024-04-26T02:07:18.696 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.696 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.696 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.696 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.696 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.696 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.697 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.697 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.697 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.697 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.697 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.697 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.697 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.697 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.697 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.698 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rgw", 2024-04-26T02:07:18.698 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.698 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.698 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.698 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.698 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.698 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.698 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.698 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.699 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.699 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.699 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.699 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.699 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.699 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.699 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.699 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.699 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.699 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.700 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.700 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.700 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.700 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.700 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.700 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.700 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.700 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.700 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.700 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.703 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.703 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.703 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.703 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.703 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.703 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.703 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.703 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.703 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.704 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.704 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.704 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.704 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.704 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.704 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.704 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.704 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.704 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.704 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.705 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.705 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.705 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.705 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.705 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.705 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.705 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.705 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.705 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.705 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.706 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rook", 2024-04-26T02:07:18.706 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.706 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.706 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.706 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.706 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.706 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.706 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.707 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.707 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.707 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.707 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.707 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.707 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.707 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.707 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.707 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.708 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.708 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.708 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.708 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.708 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.708 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.708 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.708 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.708 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.709 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.709 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.709 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.709 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.709 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.709 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.709 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.709 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.709 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.709 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.710 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.710 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.710 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.710 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.710 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.710 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.710 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.710 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.711 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.711 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.711 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.711 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.711 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.711 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.711 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.711 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.711 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.712 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.712 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.712 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.712 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.712 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.712 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.712 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.712 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.712 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.712 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.713 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.713 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.713 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.713 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.713 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.713 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.713 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.714 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.714 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.714 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.714 INFO:teuthology.orchestra.run.smithi006.stdout: "storage_class": { 2024-04-26T02:07:18.714 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "storage_class", 2024-04-26T02:07:18.714 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.714 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.714 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.715 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "local", 2024-04-26T02:07:18.715 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.715 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.715 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.715 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T02:07:18.715 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.715 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.715 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.715 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.716 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.716 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.716 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.716 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "selftest", 2024-04-26T02:07:18.716 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.716 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.716 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.716 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.717 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.717 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.717 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.717 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.717 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.717 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.717 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.717 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.718 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.718 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.718 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.718 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.718 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.718 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.718 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.718 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.718 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.719 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.719 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.719 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.719 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.719 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.719 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.719 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.719 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.720 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.720 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.720 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.720 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.720 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.720 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.720 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.720 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.721 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.721 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.721 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.721 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.721 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.721 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.721 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.721 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.721 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.722 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.722 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.722 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.722 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.722 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.722 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.722 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.722 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.723 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.723 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.723 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.723 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.723 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.723 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.723 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.723 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.723 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.724 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.724 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.724 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.724 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.724 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.724 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.724 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.724 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.724 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.725 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.725 INFO:teuthology.orchestra.run.smithi006.stdout: "roption1": { 2024-04-26T02:07:18.725 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "roption1", 2024-04-26T02:07:18.725 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.725 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.725 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.725 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.725 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.725 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.726 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.726 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.726 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.726 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.726 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.726 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.726 INFO:teuthology.orchestra.run.smithi006.stdout: "roption2": { 2024-04-26T02:07:18.726 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "roption2", 2024-04-26T02:07:18.726 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.727 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.727 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.727 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "xyz", 2024-04-26T02:07:18.727 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.727 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.727 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.727 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.727 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.727 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.728 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.728 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.728 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption1": { 2024-04-26T02:07:18.728 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption1", 2024-04-26T02:07:18.728 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.728 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.728 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.728 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.728 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.729 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.729 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.729 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.729 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.729 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.729 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.729 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.729 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption2": { 2024-04-26T02:07:18.730 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption2", 2024-04-26T02:07:18.730 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.730 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.730 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.730 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.730 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.730 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.730 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.731 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.731 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.731 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.731 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.731 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.731 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption3": { 2024-04-26T02:07:18.731 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption3", 2024-04-26T02:07:18.732 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "float", 2024-04-26T02:07:18.732 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.732 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.732 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.732 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.732 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.732 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.732 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.732 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.732 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.733 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.733 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.733 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption4": { 2024-04-26T02:07:18.733 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption4", 2024-04-26T02:07:18.733 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.733 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.733 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.733 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.733 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.733 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.734 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.734 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.734 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.734 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.734 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.734 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.734 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption5": { 2024-04-26T02:07:18.734 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption5", 2024-04-26T02:07:18.734 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.735 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.735 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.735 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.735 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.735 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.735 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.735 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.735 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.736 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.736 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.736 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.736 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption6": { 2024-04-26T02:07:18.736 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption6", 2024-04-26T02:07:18.736 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.736 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.736 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.736 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.737 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.737 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.737 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.737 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.737 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.737 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.737 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.738 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rwoption7": { 2024-04-26T02:07:18.738 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rwoption7", 2024-04-26T02:07:18.738 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.738 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.738 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.738 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.738 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "1", 2024-04-26T02:07:18.738 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "42", 2024-04-26T02:07:18.739 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.739 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.739 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.739 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.739 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.739 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.739 INFO:teuthology.orchestra.run.smithi006.stdout: "testkey": { 2024-04-26T02:07:18.739 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "testkey", 2024-04-26T02:07:18.739 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.740 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.740 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.740 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.740 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.740 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.740 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.740 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.740 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.741 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.741 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.741 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.741 INFO:teuthology.orchestra.run.smithi006.stdout: "testlkey": { 2024-04-26T02:07:18.741 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "testlkey", 2024-04-26T02:07:18.741 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.741 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.741 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.741 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.742 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.742 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.742 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.742 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.742 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.742 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.742 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.742 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.742 INFO:teuthology.orchestra.run.smithi006.stdout: "testnewline": { 2024-04-26T02:07:18.743 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "testnewline", 2024-04-26T02:07:18.743 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.743 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.743 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.743 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.743 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.743 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.743 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.743 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.744 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.744 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.744 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.744 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.744 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.744 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.744 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.744 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "snap_schedule", 2024-04-26T02:07:18.744 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.745 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.745 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.745 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_m_granularity": { 2024-04-26T02:07:18.745 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "allow_m_granularity", 2024-04-26T02:07:18.745 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.745 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.745 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.745 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.745 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.746 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.746 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.746 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T02:07:18.746 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.746 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.746 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.746 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.746 INFO:teuthology.orchestra.run.smithi006.stdout: "dump_on_update": { 2024-04-26T02:07:18.747 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "dump_on_update", 2024-04-26T02:07:18.747 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.747 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.747 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.747 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.747 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.747 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.747 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.747 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "dump database to debug log on update", 2024-04-26T02:07:18.748 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.748 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.748 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.748 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.748 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.748 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.748 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.748 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.748 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.749 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.749 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.749 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.749 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.749 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.749 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.749 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.749 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.750 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.750 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.750 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.750 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.750 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.750 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.750 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.750 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.751 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.751 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.751 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.751 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.751 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.751 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.751 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.752 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.752 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.752 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.752 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.752 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.752 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.752 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.752 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.752 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.753 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.753 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.753 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.753 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.753 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.753 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.753 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.753 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.754 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.754 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.754 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.754 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.754 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.754 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.754 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.754 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.754 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.755 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.755 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.755 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.755 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.755 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.755 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.755 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.755 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.756 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.756 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.756 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.756 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.756 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.756 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.756 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.756 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.756 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.756 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.757 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.757 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "stats", 2024-04-26T02:07:18.757 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.757 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.757 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.757 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.757 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.757 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.757 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.757 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.758 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.758 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.758 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.758 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.758 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.758 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.758 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.758 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.758 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.759 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.759 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.759 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.759 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.759 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.759 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.759 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.759 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.759 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.760 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.760 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.760 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.760 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.760 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.760 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.760 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.760 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.760 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.761 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.761 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.761 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.761 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.761 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.761 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.761 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.761 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.761 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.762 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.762 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.762 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.762 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.762 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.762 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.762 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.762 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.763 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.763 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.763 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.763 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.763 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.763 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.763 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.764 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.764 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.764 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.764 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.764 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.764 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.764 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.764 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.764 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.765 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.765 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.765 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.765 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.765 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.765 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.765 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.765 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.765 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "status", 2024-04-26T02:07:18.766 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.766 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.766 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.766 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.766 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.766 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.766 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.766 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.766 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.767 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.767 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.767 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.767 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.767 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.767 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.767 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.767 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.767 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.768 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.768 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.768 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.768 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.768 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.768 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.768 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.768 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.768 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.769 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.769 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.769 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.769 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.769 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.769 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.769 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.769 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.769 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.770 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.770 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.770 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.770 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.770 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.770 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.770 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.770 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.771 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.771 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.771 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.771 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.771 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.771 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.771 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.771 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.771 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.772 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.772 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.772 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.772 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.772 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.772 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.772 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.772 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.773 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.773 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.773 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.773 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.773 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.773 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.773 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.773 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.773 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.774 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.774 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.774 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.774 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.774 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.774 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.774 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "telegraf", 2024-04-26T02:07:18.774 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.775 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.775 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.775 INFO:teuthology.orchestra.run.smithi006.stdout: "address": { 2024-04-26T02:07:18.775 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "address", 2024-04-26T02:07:18.775 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.775 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.775 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.775 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T02:07:18.776 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.776 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.776 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.776 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.776 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.776 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.776 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.777 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:18.777 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:18.777 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.777 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.777 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.777 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "15", 2024-04-26T02:07:18.777 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.777 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.777 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.778 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.778 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.778 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.778 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.778 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.778 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.778 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.778 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.779 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.779 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.779 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.779 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.779 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.779 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.779 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.779 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.780 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.780 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.780 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.780 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.780 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.780 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.780 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.780 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "telemetry", 2024-04-26T02:07:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_basic": { 2024-04-26T02:07:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_basic", 2024-04-26T02:07:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T02:07:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_crash": { 2024-04-26T02:07:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_crash", 2024-04-26T02:07:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T02:07:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_device": { 2024-04-26T02:07:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_device", 2024-04-26T02:07:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T02:07:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_ident": { 2024-04-26T02:07:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_ident", 2024-04-26T02:07:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T02:07:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: "channel_perf": { 2024-04-26T02:07:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "channel_perf", 2024-04-26T02:07:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.794 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.794 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.794 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.794 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T02:07:18.794 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.794 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.794 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.794 INFO:teuthology.orchestra.run.smithi006.stdout: "contact": { 2024-04-26T02:07:18.795 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "contact", 2024-04-26T02:07:18.795 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.795 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.795 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.795 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.795 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.795 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.795 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.795 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.795 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.796 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.796 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.796 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.796 INFO:teuthology.orchestra.run.smithi006.stdout: "description": { 2024-04-26T02:07:18.796 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "description", 2024-04-26T02:07:18.796 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.796 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.796 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.796 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.796 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.797 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.797 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.797 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.797 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.797 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.797 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.797 INFO:teuthology.orchestra.run.smithi006.stdout: "device_url": { 2024-04-26T02:07:18.797 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "device_url", 2024-04-26T02:07:18.797 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.798 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.798 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.798 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T02:07:18.798 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.798 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.798 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.798 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.798 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.798 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.799 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.799 INFO:teuthology.orchestra.run.smithi006.stdout: "enabled": { 2024-04-26T02:07:18.799 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "enabled", 2024-04-26T02:07:18.799 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.799 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.799 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.799 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.799 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.800 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.800 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.800 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.800 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.800 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.800 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.800 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.800 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:18.800 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:18.801 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.801 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.801 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.801 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "24", 2024-04-26T02:07:18.801 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "8", 2024-04-26T02:07:18.801 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.801 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.801 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.801 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: "last_opt_revision": { 2024-04-26T02:07:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "last_opt_revision", 2024-04-26T02:07:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.802 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.803 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "1", 2024-04-26T02:07:18.803 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.803 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.803 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.803 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.803 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.803 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.803 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.804 INFO:teuthology.orchestra.run.smithi006.stdout: "leaderboard": { 2024-04-26T02:07:18.804 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "leaderboard", 2024-04-26T02:07:18.804 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.804 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.804 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.804 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.804 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.804 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.804 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.804 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.805 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.805 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.805 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.805 INFO:teuthology.orchestra.run.smithi006.stdout: "leaderboard_description": { 2024-04-26T02:07:18.805 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "leaderboard_description", 2024-04-26T02:07:18.805 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.805 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.806 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.806 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.806 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.806 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.806 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.806 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.806 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.806 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.807 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.807 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.807 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.807 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.807 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.807 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.807 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.807 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.808 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.808 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.808 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.808 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.808 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.808 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.808 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.808 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.808 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.809 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.809 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.809 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.809 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.809 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.809 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.809 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.809 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.809 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.809 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.810 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.810 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.810 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.810 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.810 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.810 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.810 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.810 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.811 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.811 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.811 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.811 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.811 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.811 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.811 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.811 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.811 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.812 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.812 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.812 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.812 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.812 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.812 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.812 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.812 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.812 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.812 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.813 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.813 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.813 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.813 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.813 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.813 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.813 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.813 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.813 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.813 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.814 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.814 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.814 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.814 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.814 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.814 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.814 INFO:teuthology.orchestra.run.smithi006.stdout: "organization": { 2024-04-26T02:07:18.815 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "organization", 2024-04-26T02:07:18.815 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.815 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.815 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.815 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.815 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.815 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.815 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.815 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.815 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.816 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.816 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.816 INFO:teuthology.orchestra.run.smithi006.stdout: "proxy": { 2024-04-26T02:07:18.816 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "proxy", 2024-04-26T02:07:18.816 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.816 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.816 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.816 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.817 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.817 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.817 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.817 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.817 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.817 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.817 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.817 INFO:teuthology.orchestra.run.smithi006.stdout: "url": { 2024-04-26T02:07:18.817 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "url", 2024-04-26T02:07:18.818 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.818 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.818 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.818 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T02:07:18.818 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.818 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.818 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.818 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.818 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.819 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.819 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.819 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.819 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.819 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.819 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.819 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "test_orchestrator", 2024-04-26T02:07:18.819 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.819 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.820 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.820 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.820 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.820 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.820 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.820 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.820 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.820 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.820 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.821 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.821 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.821 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.821 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.821 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.821 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.821 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.821 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.821 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.822 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.822 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.822 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.822 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.822 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.822 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.822 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.822 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.822 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.823 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.823 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.823 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.823 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.823 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.823 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.823 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.823 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.823 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.824 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.824 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.824 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.824 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.824 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.824 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.824 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.824 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.824 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.825 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.825 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.825 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.825 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.825 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.825 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.825 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.825 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.825 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.825 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.826 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.826 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.826 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.826 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.826 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.826 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.826 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.826 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.827 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.827 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.827 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.827 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.827 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.827 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.827 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.827 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.827 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.827 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.828 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "volumes", 2024-04-26T02:07:18.828 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.828 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.828 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.828 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.828 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.828 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.828 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.829 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.829 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.829 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.829 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.829 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.829 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.829 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.829 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.829 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.830 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.830 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.830 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.830 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.830 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.830 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.830 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.830 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.831 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.831 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.831 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.831 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.831 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.831 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.831 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.831 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.831 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.831 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.832 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.832 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.832 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.832 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.832 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.832 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.832 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.832 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.833 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.833 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.833 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.833 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.833 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.833 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.833 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.833 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.833 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.833 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.834 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.834 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.834 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.834 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.834 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.834 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.834 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.834 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.834 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.835 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.835 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.835 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.835 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.835 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.835 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.835 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.835 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.835 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.835 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.836 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.836 INFO:teuthology.orchestra.run.smithi006.stdout: "max_concurrent_clones": { 2024-04-26T02:07:18.836 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "max_concurrent_clones", 2024-04-26T02:07:18.836 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.836 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.836 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.836 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "4", 2024-04-26T02:07:18.836 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.836 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.837 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.837 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T02:07:18.837 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.837 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.837 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.837 INFO:teuthology.orchestra.run.smithi006.stdout: "periodic_async_work": { 2024-04-26T02:07:18.837 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "periodic_async_work", 2024-04-26T02:07:18.837 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.837 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.838 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.838 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.838 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.838 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.838 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.838 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Periodically check for async work", 2024-04-26T02:07:18.838 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.838 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.839 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.839 INFO:teuthology.orchestra.run.smithi006.stdout: "snapshot_clone_delay": { 2024-04-26T02:07:18.839 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "snapshot_clone_delay", 2024-04-26T02:07:18.839 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.839 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.839 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.839 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "0", 2024-04-26T02:07:18.839 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.840 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.840 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.840 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T02:07:18.840 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.840 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.840 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.840 INFO:teuthology.orchestra.run.smithi006.stdout: "snapshot_clone_no_wait": { 2024-04-26T02:07:18.840 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T02:07:18.841 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.841 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.841 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.841 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "True", 2024-04-26T02:07:18.841 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.841 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.841 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.841 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T02:07:18.841 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.842 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.842 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.842 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.842 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.842 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.842 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "zabbix", 2024-04-26T02:07:18.842 INFO:teuthology.orchestra.run.smithi006.stdout: "can_run": true, 2024-04-26T02:07:18.842 INFO:teuthology.orchestra.run.smithi006.stdout: "error_string": "", 2024-04-26T02:07:18.843 INFO:teuthology.orchestra.run.smithi006.stdout: "module_options": { 2024-04-26T02:07:18.843 INFO:teuthology.orchestra.run.smithi006.stdout: "discovery_interval": { 2024-04-26T02:07:18.843 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "discovery_interval", 2024-04-26T02:07:18.843 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "uint", 2024-04-26T02:07:18.843 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.843 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.843 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "100", 2024-04-26T02:07:18.843 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.843 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.843 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.844 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.844 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.844 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.844 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.844 INFO:teuthology.orchestra.run.smithi006.stdout: "identifier": { 2024-04-26T02:07:18.844 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "identifier", 2024-04-26T02:07:18.844 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.844 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.844 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.845 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.845 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.845 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.845 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.845 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.845 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.845 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.845 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.845 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.845 INFO:teuthology.orchestra.run.smithi006.stdout: "interval": { 2024-04-26T02:07:18.846 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "interval", 2024-04-26T02:07:18.846 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "secs", 2024-04-26T02:07:18.846 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.846 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.846 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "60", 2024-04-26T02:07:18.846 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.846 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.846 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.847 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.847 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.847 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.847 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.847 INFO:teuthology.orchestra.run.smithi006.stdout: "log_level": { 2024-04-26T02:07:18.847 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_level", 2024-04-26T02:07:18.847 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.847 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.848 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.848 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.848 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.848 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.848 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.848 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.848 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.848 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.848 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.849 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.849 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.849 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.849 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.849 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.849 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.849 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.849 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster": { 2024-04-26T02:07:18.850 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster", 2024-04-26T02:07:18.850 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.850 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.850 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.850 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.850 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.850 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.850 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.850 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.851 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.851 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.851 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.851 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_cluster_level": { 2024-04-26T02:07:18.851 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:18.851 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.851 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.851 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.851 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "info", 2024-04-26T02:07:18.852 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.852 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.852 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [ 2024-04-26T02:07:18.852 INFO:teuthology.orchestra.run.smithi006.stdout: "", 2024-04-26T02:07:18.852 INFO:teuthology.orchestra.run.smithi006.stdout: "critical", 2024-04-26T02:07:18.852 INFO:teuthology.orchestra.run.smithi006.stdout: "debug", 2024-04-26T02:07:18.852 INFO:teuthology.orchestra.run.smithi006.stdout: "error", 2024-04-26T02:07:18.852 INFO:teuthology.orchestra.run.smithi006.stdout: "info", 2024-04-26T02:07:18.852 INFO:teuthology.orchestra.run.smithi006.stdout: "warning" 2024-04-26T02:07:18.852 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.853 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.853 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.853 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.853 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.853 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": { 2024-04-26T02:07:18.853 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "log_to_file", 2024-04-26T02:07:18.853 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "bool", 2024-04-26T02:07:18.854 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.854 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-04-26T02:07:18.854 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "False", 2024-04-26T02:07:18.854 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.854 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.855 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.855 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.855 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.855 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.855 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.856 INFO:teuthology.orchestra.run.smithi006.stdout: "zabbix_host": { 2024-04-26T02:07:18.856 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "zabbix_host", 2024-04-26T02:07:18.856 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.856 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.857 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.857 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "", 2024-04-26T02:07:18.857 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.857 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.857 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.857 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.858 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.858 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.858 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.858 INFO:teuthology.orchestra.run.smithi006.stdout: "zabbix_port": { 2024-04-26T02:07:18.858 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "zabbix_port", 2024-04-26T02:07:18.858 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "int", 2024-04-26T02:07:18.858 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.859 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.859 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "10051", 2024-04-26T02:07:18.859 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.859 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.859 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.859 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.860 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.860 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.860 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.860 INFO:teuthology.orchestra.run.smithi006.stdout: "zabbix_sender": { 2024-04-26T02:07:18.860 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "zabbix_sender", 2024-04-26T02:07:18.861 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "str", 2024-04-26T02:07:18.861 INFO:teuthology.orchestra.run.smithi006.stdout: "level": "advanced", 2024-04-26T02:07:18.861 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 0, 2024-04-26T02:07:18.861 INFO:teuthology.orchestra.run.smithi006.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T02:07:18.861 INFO:teuthology.orchestra.run.smithi006.stdout: "min": "", 2024-04-26T02:07:18.862 INFO:teuthology.orchestra.run.smithi006.stdout: "max": "", 2024-04-26T02:07:18.862 INFO:teuthology.orchestra.run.smithi006.stdout: "enum_allowed": [], 2024-04-26T02:07:18.862 INFO:teuthology.orchestra.run.smithi006.stdout: "desc": "", 2024-04-26T02:07:18.862 INFO:teuthology.orchestra.run.smithi006.stdout: "long_desc": "", 2024-04-26T02:07:18.862 INFO:teuthology.orchestra.run.smithi006.stdout: "tags": [], 2024-04-26T02:07:18.862 INFO:teuthology.orchestra.run.smithi006.stdout: "see_also": [] 2024-04-26T02:07:18.862 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.863 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.863 INFO:teuthology.orchestra.run.smithi006.stdout: "services": { 2024-04-26T02:07:18.863 INFO:teuthology.orchestra.run.smithi006.stdout: "dashboard": "https://172.21.15.6:8443/" 2024-04-26T02:07:18.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.863 INFO:teuthology.orchestra.run.smithi006.stdout: "always_on_modules": { 2024-04-26T02:07:18.863 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus": [ 2024-04-26T02:07:18.863 INFO:teuthology.orchestra.run.smithi006.stdout: "balancer", 2024-04-26T02:07:18.864 INFO:teuthology.orchestra.run.smithi006.stdout: "crash", 2024-04-26T02:07:18.864 INFO:teuthology.orchestra.run.smithi006.stdout: "devicehealth", 2024-04-26T02:07:18.864 INFO:teuthology.orchestra.run.smithi006.stdout: "orchestrator", 2024-04-26T02:07:18.864 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscaler", 2024-04-26T02:07:18.864 INFO:teuthology.orchestra.run.smithi006.stdout: "progress", 2024-04-26T02:07:18.864 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_support", 2024-04-26T02:07:18.864 INFO:teuthology.orchestra.run.smithi006.stdout: "status", 2024-04-26T02:07:18.864 INFO:teuthology.orchestra.run.smithi006.stdout: "telemetry", 2024-04-26T02:07:18.864 INFO:teuthology.orchestra.run.smithi006.stdout: "volumes" 2024-04-26T02:07:18.864 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.865 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific": [ 2024-04-26T02:07:18.865 INFO:teuthology.orchestra.run.smithi006.stdout: "balancer", 2024-04-26T02:07:18.865 INFO:teuthology.orchestra.run.smithi006.stdout: "crash", 2024-04-26T02:07:18.865 INFO:teuthology.orchestra.run.smithi006.stdout: "devicehealth", 2024-04-26T02:07:18.865 INFO:teuthology.orchestra.run.smithi006.stdout: "orchestrator", 2024-04-26T02:07:18.865 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscaler", 2024-04-26T02:07:18.865 INFO:teuthology.orchestra.run.smithi006.stdout: "progress", 2024-04-26T02:07:18.865 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_support", 2024-04-26T02:07:18.866 INFO:teuthology.orchestra.run.smithi006.stdout: "status", 2024-04-26T02:07:18.866 INFO:teuthology.orchestra.run.smithi006.stdout: "telemetry", 2024-04-26T02:07:18.866 INFO:teuthology.orchestra.run.smithi006.stdout: "volumes" 2024-04-26T02:07:18.866 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.866 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy": [ 2024-04-26T02:07:18.866 INFO:teuthology.orchestra.run.smithi006.stdout: "balancer", 2024-04-26T02:07:18.866 INFO:teuthology.orchestra.run.smithi006.stdout: "crash", 2024-04-26T02:07:18.866 INFO:teuthology.orchestra.run.smithi006.stdout: "devicehealth", 2024-04-26T02:07:18.867 INFO:teuthology.orchestra.run.smithi006.stdout: "orchestrator", 2024-04-26T02:07:18.867 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscaler", 2024-04-26T02:07:18.867 INFO:teuthology.orchestra.run.smithi006.stdout: "progress", 2024-04-26T02:07:18.867 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_support", 2024-04-26T02:07:18.867 INFO:teuthology.orchestra.run.smithi006.stdout: "status", 2024-04-26T02:07:18.867 INFO:teuthology.orchestra.run.smithi006.stdout: "telemetry", 2024-04-26T02:07:18.867 INFO:teuthology.orchestra.run.smithi006.stdout: "volumes" 2024-04-26T02:07:18.867 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.867 INFO:teuthology.orchestra.run.smithi006.stdout: "reef": [ 2024-04-26T02:07:18.868 INFO:teuthology.orchestra.run.smithi006.stdout: "balancer", 2024-04-26T02:07:18.868 INFO:teuthology.orchestra.run.smithi006.stdout: "crash", 2024-04-26T02:07:18.868 INFO:teuthology.orchestra.run.smithi006.stdout: "devicehealth", 2024-04-26T02:07:18.868 INFO:teuthology.orchestra.run.smithi006.stdout: "orchestrator", 2024-04-26T02:07:18.868 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscaler", 2024-04-26T02:07:18.868 INFO:teuthology.orchestra.run.smithi006.stdout: "progress", 2024-04-26T02:07:18.868 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_support", 2024-04-26T02:07:18.868 INFO:teuthology.orchestra.run.smithi006.stdout: "status", 2024-04-26T02:07:18.869 INFO:teuthology.orchestra.run.smithi006.stdout: "telemetry", 2024-04-26T02:07:18.869 INFO:teuthology.orchestra.run.smithi006.stdout: "volumes" 2024-04-26T02:07:18.869 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-26T02:07:18.869 INFO:teuthology.orchestra.run.smithi006.stdout: "squid": [ 2024-04-26T02:07:18.869 INFO:teuthology.orchestra.run.smithi006.stdout: "balancer", 2024-04-26T02:07:18.869 INFO:teuthology.orchestra.run.smithi006.stdout: "crash", 2024-04-26T02:07:18.869 INFO:teuthology.orchestra.run.smithi006.stdout: "devicehealth", 2024-04-26T02:07:18.869 INFO:teuthology.orchestra.run.smithi006.stdout: "orchestrator", 2024-04-26T02:07:18.870 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscaler", 2024-04-26T02:07:18.870 INFO:teuthology.orchestra.run.smithi006.stdout: "progress", 2024-04-26T02:07:18.870 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_support", 2024-04-26T02:07:18.870 INFO:teuthology.orchestra.run.smithi006.stdout: "status", 2024-04-26T02:07:18.870 INFO:teuthology.orchestra.run.smithi006.stdout: "telemetry", 2024-04-26T02:07:18.870 INFO:teuthology.orchestra.run.smithi006.stdout: "volumes" 2024-04-26T02:07:18.870 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-26T02:07:18.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.871 INFO:teuthology.orchestra.run.smithi006.stdout: "last_failure_osd_epoch": 96, 2024-04-26T02:07:18.871 INFO:teuthology.orchestra.run.smithi006.stdout: "active_clients": [ 2024-04-26T02:07:18.871 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.871 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "devicehealth", 2024-04-26T02:07:18.871 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-26T02:07:18.871 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.871 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-04-26T02:07:18.871 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:0", 2024-04-26T02:07:18.871 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3936292144 2024-04-26T02:07:18.872 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.872 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-26T02:07:18.872 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.872 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.872 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "libcephsqlite", 2024-04-26T02:07:18.872 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-26T02:07:18.872 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.872 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-04-26T02:07:18.872 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:0", 2024-04-26T02:07:18.873 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3177901219 2024-04-26T02:07:18.873 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.873 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-26T02:07:18.873 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.873 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "rbd_support", 2024-04-26T02:07:18.873 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-26T02:07:18.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.874 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-04-26T02:07:18.874 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:0", 2024-04-26T02:07:18.874 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3874938334 2024-04-26T02:07:18.874 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.874 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-26T02:07:18.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.874 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "snap_schedule", 2024-04-26T02:07:18.875 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-26T02:07:18.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.875 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-04-26T02:07:18.875 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:0", 2024-04-26T02:07:18.875 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1735873849 2024-04-26T02:07:18.875 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.875 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-26T02:07:18.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.876 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "snap_schedule", 2024-04-26T02:07:18.876 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-26T02:07:18.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.876 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-04-26T02:07:18.876 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:0", 2024-04-26T02:07:18.876 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2513958245 2024-04-26T02:07:18.876 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.876 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-26T02:07:18.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:18.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.877 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "volumes", 2024-04-26T02:07:18.877 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-26T02:07:18.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-26T02:07:18.877 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-04-26T02:07:18.877 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:0", 2024-04-26T02:07:18.877 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3356156087 2024-04-26T02:07:18.877 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.878 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-26T02:07:18.878 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:18.878 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-26T02:07:18.878 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:07:18.879 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:16 smithi188 ceph-mon[32751]: pgmap v678: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 281 op/s 2024-04-26T02:07:18.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:16 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1121633358' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T02:07:18.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:16 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/523553894' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T02:07:18.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:17 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2191683823' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T02:07:18.880 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:16 smithi029 ceph-mon[32775]: pgmap v678: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 281 op/s 2024-04-26T02:07:18.880 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:16 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1121633358' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T02:07:18.880 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:16 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/523553894' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T02:07:18.880 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:17 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2191683823' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T02:07:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.783+0000 7f94f3e00640 1 -- 172.21.15.6:0/2191683823 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f94d80a3d90 msgr2=0x7f94d80a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.783+0000 7f94f3e00640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f94d80a3d90 0x7f94d80a6250 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f94f419bf10 tx=0x7f94e0005f90 comp rx=0 tx=0).stop 2024-04-26T02:07:18.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.783+0000 7f94f3e00640 1 -- 172.21.15.6:0/2191683823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94f4104290 msgr2=0x7f94f41a0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:18.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.783+0000 7f94f3e00640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94f4104290 0x7f94f41a0600 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f94e4035040 tx=0x7f94e4032d50 comp rx=0 tx=0).stop 2024-04-26T02:07:18.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.783+0000 7f94f3e00640 1 -- 172.21.15.6:0/2191683823 shutdown_connections 2024-04-26T02:07:18.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.783+0000 7f94f3e00640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f94d80a3d90 0x7f94d80a6250 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.783+0000 7f94f3e00640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94f419a6c0 0x7f94f419ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.783+0000 7f94f3e00640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f4109360 0x7f94f41a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.783+0000 7f94f3e00640 1 --2- 172.21.15.6:0/2191683823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94f4104290 0x7f94f41a0600 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.783+0000 7f94f3e00640 1 -- 172.21.15.6:0/2191683823 >> 172.21.15.6:0/2191683823 conn(0x7f94f40ffad0 msgr2=0x7f94f4100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:18.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.787+0000 7f94f3e00640 1 -- 172.21.15.6:0/2191683823 shutdown_connections 2024-04-26T02:07:18.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:16.787+0000 7f94f3e00640 1 -- 172.21.15.6:0/2191683823 wait complete. 2024-04-26T02:07:18.890 DEBUG:tasks.check_counter:Getting stats from a 2024-04-26T02:07:18.890 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-26T02:07:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:16 smithi006 ceph-mon[28419]: pgmap v678: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 281 op/s 2024-04-26T02:07:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:16 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1121633358' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T02:07:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:16 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/523553894' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T02:07:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:17 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2191683823' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T02:07:18.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:18 smithi029 ceph-mon[32775]: pgmap v680: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 4.3 MiB/s wr, 154 op/s 2024-04-26T02:07:18.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:18 smithi006 ceph-mon[28419]: pgmap v680: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 4.3 MiB/s wr, 154 op/s 2024-04-26T02:07:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:18 smithi188 ceph-mon[32751]: pgmap v680: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 4.3 MiB/s wr, 154 op/s 2024-04-26T02:07:19.346 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:20.913 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:20 smithi006 ceph-mon[28419]: pgmap v681: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 4.3 MiB/s wr, 154 op/s 2024-04-26T02:07:20.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:20 smithi029 ceph-mon[32775]: pgmap v681: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 4.3 MiB/s wr, 154 op/s 2024-04-26T02:07:21.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:20 smithi188 ceph-mon[32751]: pgmap v681: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 4.3 MiB/s wr, 154 op/s 2024-04-26T02:07:21.813 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi006.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_messages": 4, 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages": 4, 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_bytes": 2992, 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_bytes": 448, 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_active_connections": 0, 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_total_time": 0.005006038, 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_send_time": 0.000093873, 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_recv_time": 0.001425054, 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 3, 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.001851992, 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000617330 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi006.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_messages": 419, 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages": 407, 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_bytes": 1784688, 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_bytes": 251936, 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_created_connections": 2, 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_active_connections": 1, 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_total_time": 0.156376853, 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_send_time": 0.063956098, 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_recv_time": 0.084863196, 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_fast_dispatch_time": 0.019676672, 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 406, 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.042838558, 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000105513 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_encrypted_bytes": 1784688, 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_encrypted_bytes": 251936 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi006.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_messages": 3, 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages": 312, 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_bytes": 480, 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_bytes": 289296, 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_total_time": 0.061840917, 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_send_time": 0.048102782, 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_recv_time": 0.004466310, 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 309, 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.039918157, 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000129184 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_encrypted_bytes": 289296 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi006.stdout: "cct": { 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi006.stdout: "total_workers": 0, 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi006.stdout: "unhealthy_workers": 0 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi006.stdout: "mempool": { 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi006.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi006.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_items": 0, 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_anon_bytes": 149745, 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_anon_items": 31, 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bytes": 0, 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_items": 0, 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_bytes": 0, 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_items": 0, 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi006.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi006.stdout: "pgmap_items": 0, 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_co_bytes": 928, 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_co_items": 2, 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_1_items": 0, 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_2_items": 0 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi006.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi006.stdout: "val": 0, 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi006.stdout: "max": 104857600, 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi006.stdout: "get_started": 0, 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi006.stdout: "get": 422, 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi006.stdout: "get_sum": 1725509, 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_success": 422, 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi006.stdout: "take": 0, 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi006.stdout: "take_sum": 0, 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi006.stdout: "put": 422, 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi006.stdout: "put_sum": 1725509, 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi006.stdout: "wait": { 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:21.831 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:07:22.693 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-26T02:07:22.694 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-26T02:07:22.694 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-04-26T02:07:22.694 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-04-26T02:07:22.694 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-26T02:07:22.694 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-26T02:07:22.694 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-26T02:07:22.694 DEBUG:tasks.check_counter:Getting stats from d 2024-04-26T02:07:22.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-26T02:07:22.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:22 smithi029 ceph-mon[32775]: pgmap v682: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.5 MiB/s wr, 133 op/s 2024-04-26T02:07:23.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:22 smithi006 ceph-mon[28419]: pgmap v682: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.5 MiB/s wr, 133 op/s 2024-04-26T02:07:23.131 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:23.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:22 smithi188 ceph-mon[32751]: pgmap v682: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.5 MiB/s wr, 133 op/s 2024-04-26T02:07:24.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:24 smithi006 ceph-mon[28419]: pgmap v683: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.4 MiB/s wr, 133 op/s 2024-04-26T02:07:24.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:24 smithi029 ceph-mon[32775]: pgmap v683: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.4 MiB/s wr, 133 op/s 2024-04-26T02:07:25.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:24 smithi188 ceph-mon[32751]: pgmap v683: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.4 MiB/s wr, 133 op/s 2024-04-26T02:07:25.431 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:07:25.432 INFO:teuthology.orchestra.run.smithi006.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:25.432 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_messages": 4, 2024-04-26T02:07:25.432 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages": 4, 2024-04-26T02:07:25.432 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_bytes": 2992, 2024-04-26T02:07:25.432 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_bytes": 448, 2024-04-26T02:07:25.432 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:25.432 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_active_connections": 0, 2024-04-26T02:07:25.433 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_total_time": 0.006534367, 2024-04-26T02:07:25.433 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_send_time": 0.000078008, 2024-04-26T02:07:25.433 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_recv_time": 0.001371118, 2024-04-26T02:07:25.433 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:25.433 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:25.433 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 3, 2024-04-26T02:07:25.433 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.001791111, 2024-04-26T02:07:25.433 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000597037 2024-04-26T02:07:25.433 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:25.434 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:25.434 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:25.434 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.434 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:25.434 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T02:07:25.434 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T02:07:25.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:25.434 INFO:teuthology.orchestra.run.smithi006.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:25.435 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_messages": 427, 2024-04-26T02:07:25.435 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages": 415, 2024-04-26T02:07:25.435 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_bytes": 1786864, 2024-04-26T02:07:25.435 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_bytes": 257056, 2024-04-26T02:07:25.435 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_created_connections": 2, 2024-04-26T02:07:25.435 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_active_connections": 1, 2024-04-26T02:07:25.435 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_total_time": 0.178263909, 2024-04-26T02:07:25.435 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_send_time": 0.046423523, 2024-04-26T02:07:25.435 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_recv_time": 0.143529424, 2024-04-26T02:07:25.436 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_fast_dispatch_time": 0.020982970, 2024-04-26T02:07:25.436 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:25.436 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 414, 2024-04-26T02:07:25.436 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.049674981, 2024-04-26T02:07:25.436 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000119987 2024-04-26T02:07:25.436 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:25.436 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:25.436 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:25.436 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.437 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.437 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:25.437 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_encrypted_bytes": 1786864, 2024-04-26T02:07:25.437 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_encrypted_bytes": 257056 2024-04-26T02:07:25.437 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:25.437 INFO:teuthology.orchestra.run.smithi006.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:25.437 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_messages": 3, 2024-04-26T02:07:25.437 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages": 317, 2024-04-26T02:07:25.437 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_bytes": 480, 2024-04-26T02:07:25.438 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_bytes": 290176, 2024-04-26T02:07:25.438 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:25.438 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:25.438 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_total_time": 0.061760817, 2024-04-26T02:07:25.438 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_send_time": 0.047816666, 2024-04-26T02:07:25.438 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_recv_time": 0.005118555, 2024-04-26T02:07:25.438 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:25.438 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:25.438 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 314, 2024-04-26T02:07:25.439 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.045757212, 2024-04-26T02:07:25.439 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000145723 2024-04-26T02:07:25.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:25.439 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:25.439 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:25.439 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.439 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:25.439 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T02:07:25.440 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_encrypted_bytes": 290176 2024-04-26T02:07:25.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:25.440 INFO:teuthology.orchestra.run.smithi006.stdout: "cct": { 2024-04-26T02:07:25.440 INFO:teuthology.orchestra.run.smithi006.stdout: "total_workers": 0, 2024-04-26T02:07:25.440 INFO:teuthology.orchestra.run.smithi006.stdout: "unhealthy_workers": 0 2024-04-26T02:07:25.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:25.440 INFO:teuthology.orchestra.run.smithi006.stdout: "mempool": { 2024-04-26T02:07:25.440 INFO:teuthology.orchestra.run.smithi006.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:25.440 INFO:teuthology.orchestra.run.smithi006.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:25.441 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:25.441 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:25.441 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:25.441 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:25.441 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:25.441 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:25.441 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:25.441 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:25.441 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:25.442 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:25.442 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:25.442 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:25.442 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:25.442 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:25.442 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:25.442 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:25.442 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:25.442 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:25.443 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:25.443 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:25.443 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:25.443 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:25.443 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:25.443 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:25.443 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:25.443 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:25.443 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:25.444 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:25.444 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:25.444 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_items": 0, 2024-04-26T02:07:25.444 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:25.444 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:25.444 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:25.444 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:25.444 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_anon_bytes": 149745, 2024-04-26T02:07:25.444 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_anon_items": 31, 2024-04-26T02:07:25.444 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:25.445 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:25.445 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bytes": 0, 2024-04-26T02:07:25.445 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_items": 0, 2024-04-26T02:07:25.445 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:25.445 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:25.445 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:25.445 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:25.446 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_bytes": 0, 2024-04-26T02:07:25.446 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_items": 0, 2024-04-26T02:07:25.446 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:25.446 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:25.446 INFO:teuthology.orchestra.run.smithi006.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:25.446 INFO:teuthology.orchestra.run.smithi006.stdout: "pgmap_items": 0, 2024-04-26T02:07:25.446 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_co_bytes": 928, 2024-04-26T02:07:25.447 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_co_items": 2, 2024-04-26T02:07:25.447 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:25.447 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_1_items": 0, 2024-04-26T02:07:25.447 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:25.447 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_2_items": 0 2024-04-26T02:07:25.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:25.447 INFO:teuthology.orchestra.run.smithi006.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:25.447 INFO:teuthology.orchestra.run.smithi006.stdout: "val": 0, 2024-04-26T02:07:25.447 INFO:teuthology.orchestra.run.smithi006.stdout: "max": 104857600, 2024-04-26T02:07:25.448 INFO:teuthology.orchestra.run.smithi006.stdout: "get_started": 0, 2024-04-26T02:07:25.448 INFO:teuthology.orchestra.run.smithi006.stdout: "get": 430, 2024-04-26T02:07:25.448 INFO:teuthology.orchestra.run.smithi006.stdout: "get_sum": 1726549, 2024-04-26T02:07:25.448 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:25.448 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_success": 430, 2024-04-26T02:07:25.448 INFO:teuthology.orchestra.run.smithi006.stdout: "take": 0, 2024-04-26T02:07:25.448 INFO:teuthology.orchestra.run.smithi006.stdout: "take_sum": 0, 2024-04-26T02:07:25.448 INFO:teuthology.orchestra.run.smithi006.stdout: "put": 430, 2024-04-26T02:07:25.448 INFO:teuthology.orchestra.run.smithi006.stdout: "put_sum": 1726549, 2024-04-26T02:07:25.449 INFO:teuthology.orchestra.run.smithi006.stdout: "wait": { 2024-04-26T02:07:25.449 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:25.449 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.449 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.449 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:25.449 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:25.449 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:07:26.465 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-26T02:07:26.465 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-26T02:07:26.465 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-26T02:07:26.465 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-26T02:07:26.465 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-26T02:07:26.465 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-26T02:07:26.465 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-26T02:07:26.465 DEBUG:tasks.check_counter:Getting stats from g 2024-04-26T02:07:26.466 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-26T02:07:26.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:26 smithi029 ceph-mon[32775]: pgmap v684: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.4 MiB/s wr, 133 op/s 2024-04-26T02:07:26.942 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:27.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:26 smithi006 ceph-mon[28419]: pgmap v684: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.4 MiB/s wr, 133 op/s 2024-04-26T02:07:27.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:26 smithi188 ceph-mon[32751]: pgmap v684: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 3.4 MiB/s wr, 133 op/s 2024-04-26T02:07:28.909 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:28 smithi006 ceph-mon[28419]: pgmap v685: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 5.3 MiB/s wr, 179 op/s 2024-04-26T02:07:28.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:28 smithi029 ceph-mon[32775]: pgmap v685: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 5.3 MiB/s wr, 179 op/s 2024-04-26T02:07:29.138 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:07:29.138 INFO:teuthology.orchestra.run.smithi006.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:29.139 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_messages": 3643, 2024-04-26T02:07:29.139 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages": 3645, 2024-04-26T02:07:29.139 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_bytes": 214966712, 2024-04-26T02:07:29.139 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_bytes": 1158481, 2024-04-26T02:07:29.139 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_created_connections": 8, 2024-04-26T02:07:29.139 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_active_connections": 4, 2024-04-26T02:07:29.139 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_total_time": 1.294844978, 2024-04-26T02:07:29.139 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_send_time": 0.236076866, 2024-04-26T02:07:29.140 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_recv_time": 0.760702385, 2024-04-26T02:07:29.140 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_fast_dispatch_time": 0.229689120, 2024-04-26T02:07:29.140 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:29.140 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 3643, 2024-04-26T02:07:29.140 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.325066390, 2024-04-26T02:07:29.140 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000089230 2024-04-26T02:07:29.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.140 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T02:07:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T02:07:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_messages": 1609, 2024-04-26T02:07:29.142 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages": 1560, 2024-04-26T02:07:29.142 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_bytes": 158864520, 2024-04-26T02:07:29.142 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_bytes": 613172, 2024-04-26T02:07:29.142 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:29.142 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_active_connections": 5, 2024-04-26T02:07:29.142 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_total_time": 0.745166328, 2024-04-26T02:07:29.142 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_send_time": 0.128386991, 2024-04-26T02:07:29.142 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_recv_time": 0.535187822, 2024-04-26T02:07:29.142 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_fast_dispatch_time": 0.093975894, 2024-04-26T02:07:29.143 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:29.143 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 1560, 2024-04-26T02:07:29.143 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.153839363, 2024-04-26T02:07:29.143 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000098614 2024-04-26T02:07:29.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.143 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:29.143 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.143 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.143 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.144 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_encrypted_bytes": 1976960, 2024-04-26T02:07:29.144 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_encrypted_bytes": 262992 2024-04-26T02:07:29.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.144 INFO:teuthology.orchestra.run.smithi006.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:29.144 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_messages": 974, 2024-04-26T02:07:29.144 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages": 1285, 2024-04-26T02:07:29.145 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_bytes": 193241462, 2024-04-26T02:07:29.145 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_bytes": 1020392, 2024-04-26T02:07:29.145 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_created_connections": 9, 2024-04-26T02:07:29.145 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_active_connections": 6, 2024-04-26T02:07:29.145 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_total_time": 0.712351585, 2024-04-26T02:07:29.145 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_send_time": 0.117369656, 2024-04-26T02:07:29.145 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_recv_time": 0.508796218, 2024-04-26T02:07:29.145 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_fast_dispatch_time": 0.058775384, 2024-04-26T02:07:29.145 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:29.146 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 1282, 2024-04-26T02:07:29.146 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.132556215, 2024-04-26T02:07:29.146 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000103397 2024-04-26T02:07:29.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.146 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:29.146 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.146 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.146 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.147 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T02:07:29.147 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_encrypted_bytes": 711296 2024-04-26T02:07:29.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.147 INFO:teuthology.orchestra.run.smithi006.stdout: "cct": { 2024-04-26T02:07:29.147 INFO:teuthology.orchestra.run.smithi006.stdout: "total_workers": 1, 2024-04-26T02:07:29.147 INFO:teuthology.orchestra.run.smithi006.stdout: "unhealthy_workers": 0 2024-04-26T02:07:29.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.147 INFO:teuthology.orchestra.run.smithi006.stdout: "finisher-MDSRank": { 2024-04-26T02:07:29.147 INFO:teuthology.orchestra.run.smithi006.stdout: "queue_len": 0, 2024-04-26T02:07:29.147 INFO:teuthology.orchestra.run.smithi006.stdout: "complete_latency": { 2024-04-26T02:07:29.148 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 9172, 2024-04-26T02:07:29.148 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.531372288, 2024-04-26T02:07:29.148 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000057934 2024-04-26T02:07:29.148 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:29.148 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.148 INFO:teuthology.orchestra.run.smithi006.stdout: "finisher-PurgeQueue": { 2024-04-26T02:07:29.148 INFO:teuthology.orchestra.run.smithi006.stdout: "queue_len": 0, 2024-04-26T02:07:29.148 INFO:teuthology.orchestra.run.smithi006.stdout: "complete_latency": { 2024-04-26T02:07:29.148 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.148 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.148 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.149 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:29.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.149 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": { 2024-04-26T02:07:29.149 INFO:teuthology.orchestra.run.smithi006.stdout: "request": 0, 2024-04-26T02:07:29.149 INFO:teuthology.orchestra.run.smithi006.stdout: "reply": 0, 2024-04-26T02:07:29.149 INFO:teuthology.orchestra.run.smithi006.stdout: "reply_latency": { 2024-04-26T02:07:29.149 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.149 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.149 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.150 INFO:teuthology.orchestra.run.smithi006.stdout: "slow_reply": 0, 2024-04-26T02:07:29.150 INFO:teuthology.orchestra.run.smithi006.stdout: "forward": 0, 2024-04-26T02:07:29.150 INFO:teuthology.orchestra.run.smithi006.stdout: "dir_fetch_complete": 0, 2024-04-26T02:07:29.150 INFO:teuthology.orchestra.run.smithi006.stdout: "dir_fetch_keys": 0, 2024-04-26T02:07:29.150 INFO:teuthology.orchestra.run.smithi006.stdout: "dir_commit": 0, 2024-04-26T02:07:29.150 INFO:teuthology.orchestra.run.smithi006.stdout: "dir_split": 0, 2024-04-26T02:07:29.150 INFO:teuthology.orchestra.run.smithi006.stdout: "dir_merge": 0, 2024-04-26T02:07:29.151 INFO:teuthology.orchestra.run.smithi006.stdout: "inodes": 22939, 2024-04-26T02:07:29.151 INFO:teuthology.orchestra.run.smithi006.stdout: "inodes_top": 2468, 2024-04-26T02:07:29.151 INFO:teuthology.orchestra.run.smithi006.stdout: "inodes_bottom": 20471, 2024-04-26T02:07:29.151 INFO:teuthology.orchestra.run.smithi006.stdout: "inodes_pin_tail": 0, 2024-04-26T02:07:29.151 INFO:teuthology.orchestra.run.smithi006.stdout: "inodes_pinned": 19414, 2024-04-26T02:07:29.151 INFO:teuthology.orchestra.run.smithi006.stdout: "inodes_expired": 0, 2024-04-26T02:07:29.151 INFO:teuthology.orchestra.run.smithi006.stdout: "inodes_with_caps": 0, 2024-04-26T02:07:29.151 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": 0, 2024-04-26T02:07:29.151 INFO:teuthology.orchestra.run.smithi006.stdout: "subtrees": 9, 2024-04-26T02:07:29.152 INFO:teuthology.orchestra.run.smithi006.stdout: "traverse": 0, 2024-04-26T02:07:29.152 INFO:teuthology.orchestra.run.smithi006.stdout: "traverse_hit": 0, 2024-04-26T02:07:29.152 INFO:teuthology.orchestra.run.smithi006.stdout: "traverse_forward": 0, 2024-04-26T02:07:29.152 INFO:teuthology.orchestra.run.smithi006.stdout: "traverse_discover": 0, 2024-04-26T02:07:29.152 INFO:teuthology.orchestra.run.smithi006.stdout: "traverse_dir_fetch": 0, 2024-04-26T02:07:29.152 INFO:teuthology.orchestra.run.smithi006.stdout: "traverse_remote_ino": 0, 2024-04-26T02:07:29.152 INFO:teuthology.orchestra.run.smithi006.stdout: "traverse_lock": 0, 2024-04-26T02:07:29.152 INFO:teuthology.orchestra.run.smithi006.stdout: "load_cent": 0, 2024-04-26T02:07:29.152 INFO:teuthology.orchestra.run.smithi006.stdout: "q": 0, 2024-04-26T02:07:29.153 INFO:teuthology.orchestra.run.smithi006.stdout: "exported": 0, 2024-04-26T02:07:29.153 INFO:teuthology.orchestra.run.smithi006.stdout: "exported_inodes": 0, 2024-04-26T02:07:29.153 INFO:teuthology.orchestra.run.smithi006.stdout: "imported": 0, 2024-04-26T02:07:29.153 INFO:teuthology.orchestra.run.smithi006.stdout: "imported_inodes": 0, 2024-04-26T02:07:29.153 INFO:teuthology.orchestra.run.smithi006.stdout: "openino_dir_fetch": 0, 2024-04-26T02:07:29.153 INFO:teuthology.orchestra.run.smithi006.stdout: "openino_backtrace_fetch": 0, 2024-04-26T02:07:29.153 INFO:teuthology.orchestra.run.smithi006.stdout: "openino_peer_discover": 0, 2024-04-26T02:07:29.153 INFO:teuthology.orchestra.run.smithi006.stdout: "root_rfiles": 0, 2024-04-26T02:07:29.153 INFO:teuthology.orchestra.run.smithi006.stdout: "root_rbytes": 0, 2024-04-26T02:07:29.153 INFO:teuthology.orchestra.run.smithi006.stdout: "root_rsnaps": 0, 2024-04-26T02:07:29.154 INFO:teuthology.orchestra.run.smithi006.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T02:07:29.154 INFO:teuthology.orchestra.run.smithi006.stdout: "scrub_set_tag": 0, 2024-04-26T02:07:29.154 INFO:teuthology.orchestra.run.smithi006.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T02:07:29.154 INFO:teuthology.orchestra.run.smithi006.stdout: "scrub_inotable_repaired": 0, 2024-04-26T02:07:29.154 INFO:teuthology.orchestra.run.smithi006.stdout: "scrub_dir_inodes": 0, 2024-04-26T02:07:29.154 INFO:teuthology.orchestra.run.smithi006.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T02:07:29.154 INFO:teuthology.orchestra.run.smithi006.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T02:07:29.154 INFO:teuthology.orchestra.run.smithi006.stdout: "scrub_file_inodes": 0, 2024-04-26T02:07:29.155 INFO:teuthology.orchestra.run.smithi006.stdout: "handle_inode_file_caps": 0, 2024-04-26T02:07:29.155 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T02:07:29.155 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_cap_op_grant": 0, 2024-04-26T02:07:29.155 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T02:07:29.155 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T02:07:29.155 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T02:07:29.155 INFO:teuthology.orchestra.run.smithi006.stdout: "handle_client_caps": 0, 2024-04-26T02:07:29.156 INFO:teuthology.orchestra.run.smithi006.stdout: "handle_client_caps_dirty": 0, 2024-04-26T02:07:29.156 INFO:teuthology.orchestra.run.smithi006.stdout: "handle_client_cap_release": 0, 2024-04-26T02:07:29.156 INFO:teuthology.orchestra.run.smithi006.stdout: "process_request_cap_release": 0 2024-04-26T02:07:29.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.156 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache": { 2024-04-26T02:07:29.156 INFO:teuthology.orchestra.run.smithi006.stdout: "dir_update": 0, 2024-04-26T02:07:29.156 INFO:teuthology.orchestra.run.smithi006.stdout: "dir_update_receipt": 0, 2024-04-26T02:07:29.156 INFO:teuthology.orchestra.run.smithi006.stdout: "dir_try_discover": 0, 2024-04-26T02:07:29.156 INFO:teuthology.orchestra.run.smithi006.stdout: "dir_send_discover": 0, 2024-04-26T02:07:29.157 INFO:teuthology.orchestra.run.smithi006.stdout: "dir_handle_discover": 0, 2024-04-26T02:07:29.157 INFO:teuthology.orchestra.run.smithi006.stdout: "num_strays": 0, 2024-04-26T02:07:29.157 INFO:teuthology.orchestra.run.smithi006.stdout: "num_strays_delayed": 0, 2024-04-26T02:07:29.157 INFO:teuthology.orchestra.run.smithi006.stdout: "num_strays_enqueuing": 0, 2024-04-26T02:07:29.157 INFO:teuthology.orchestra.run.smithi006.stdout: "strays_created": 0, 2024-04-26T02:07:29.157 INFO:teuthology.orchestra.run.smithi006.stdout: "strays_enqueued": 0, 2024-04-26T02:07:29.157 INFO:teuthology.orchestra.run.smithi006.stdout: "strays_reintegrated": 0, 2024-04-26T02:07:29.157 INFO:teuthology.orchestra.run.smithi006.stdout: "strays_migrated": 0, 2024-04-26T02:07:29.157 INFO:teuthology.orchestra.run.smithi006.stdout: "num_recovering_processing": 0, 2024-04-26T02:07:29.158 INFO:teuthology.orchestra.run.smithi006.stdout: "num_recovering_enqueued": 0, 2024-04-26T02:07:29.158 INFO:teuthology.orchestra.run.smithi006.stdout: "num_recovering_prioritized": 0, 2024-04-26T02:07:29.158 INFO:teuthology.orchestra.run.smithi006.stdout: "recovery_started": 0, 2024-04-26T02:07:29.158 INFO:teuthology.orchestra.run.smithi006.stdout: "recovery_completed": 0, 2024-04-26T02:07:29.158 INFO:teuthology.orchestra.run.smithi006.stdout: "ireq_quiesce_path": 0, 2024-04-26T02:07:29.158 INFO:teuthology.orchestra.run.smithi006.stdout: "ireq_quiesce_inode": 0, 2024-04-26T02:07:29.158 INFO:teuthology.orchestra.run.smithi006.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T02:07:29.158 INFO:teuthology.orchestra.run.smithi006.stdout: "ireq_exportdir": 0, 2024-04-26T02:07:29.158 INFO:teuthology.orchestra.run.smithi006.stdout: "ireq_flush": 0, 2024-04-26T02:07:29.159 INFO:teuthology.orchestra.run.smithi006.stdout: "ireq_fragmentdir": 0, 2024-04-26T02:07:29.159 INFO:teuthology.orchestra.run.smithi006.stdout: "ireq_fragstats": 0, 2024-04-26T02:07:29.159 INFO:teuthology.orchestra.run.smithi006.stdout: "ireq_inodestats": 0 2024-04-26T02:07:29.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.159 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log": { 2024-04-26T02:07:29.159 INFO:teuthology.orchestra.run.smithi006.stdout: "evlrg": 0, 2024-04-26T02:07:29.159 INFO:teuthology.orchestra.run.smithi006.stdout: "evadd": 0, 2024-04-26T02:07:29.159 INFO:teuthology.orchestra.run.smithi006.stdout: "evex": 0, 2024-04-26T02:07:29.159 INFO:teuthology.orchestra.run.smithi006.stdout: "evtrm": 0, 2024-04-26T02:07:29.160 INFO:teuthology.orchestra.run.smithi006.stdout: "ev": 127543, 2024-04-26T02:07:29.160 INFO:teuthology.orchestra.run.smithi006.stdout: "evexg": 0, 2024-04-26T02:07:29.160 INFO:teuthology.orchestra.run.smithi006.stdout: "evexd": 0, 2024-04-26T02:07:29.160 INFO:teuthology.orchestra.run.smithi006.stdout: "segadd": 0, 2024-04-26T02:07:29.160 INFO:teuthology.orchestra.run.smithi006.stdout: "segex": 0, 2024-04-26T02:07:29.160 INFO:teuthology.orchestra.run.smithi006.stdout: "segtrm": 0, 2024-04-26T02:07:29.160 INFO:teuthology.orchestra.run.smithi006.stdout: "seg": 140, 2024-04-26T02:07:29.160 INFO:teuthology.orchestra.run.smithi006.stdout: "segmjr": 12, 2024-04-26T02:07:29.160 INFO:teuthology.orchestra.run.smithi006.stdout: "segexg": 0, 2024-04-26T02:07:29.161 INFO:teuthology.orchestra.run.smithi006.stdout: "segexd": 0, 2024-04-26T02:07:29.161 INFO:teuthology.orchestra.run.smithi006.stdout: "expos": 60987628, 2024-04-26T02:07:29.161 INFO:teuthology.orchestra.run.smithi006.stdout: "wrpos": 567629334, 2024-04-26T02:07:29.161 INFO:teuthology.orchestra.run.smithi006.stdout: "rdpos": 567623595, 2024-04-26T02:07:29.161 INFO:teuthology.orchestra.run.smithi006.stdout: "jlat": { 2024-04-26T02:07:29.161 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.161 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.161 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.162 INFO:teuthology.orchestra.run.smithi006.stdout: "replayed": 127543 2024-04-26T02:07:29.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.162 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_mem": { 2024-04-26T02:07:29.162 INFO:teuthology.orchestra.run.smithi006.stdout: "ino": 2064, 2024-04-26T02:07:29.162 INFO:teuthology.orchestra.run.smithi006.stdout: "ino+": 22911, 2024-04-26T02:07:29.162 INFO:teuthology.orchestra.run.smithi006.stdout: "ino-": 20847, 2024-04-26T02:07:29.162 INFO:teuthology.orchestra.run.smithi006.stdout: "dir": 113, 2024-04-26T02:07:29.162 INFO:teuthology.orchestra.run.smithi006.stdout: "dir+": 176, 2024-04-26T02:07:29.163 INFO:teuthology.orchestra.run.smithi006.stdout: "dir-": 63, 2024-04-26T02:07:29.163 INFO:teuthology.orchestra.run.smithi006.stdout: "dn": 22939, 2024-04-26T02:07:29.163 INFO:teuthology.orchestra.run.smithi006.stdout: "dn+": 23137, 2024-04-26T02:07:29.163 INFO:teuthology.orchestra.run.smithi006.stdout: "dn-": 198, 2024-04-26T02:07:29.163 INFO:teuthology.orchestra.run.smithi006.stdout: "cap": 0, 2024-04-26T02:07:29.163 INFO:teuthology.orchestra.run.smithi006.stdout: "cap+": 0, 2024-04-26T02:07:29.163 INFO:teuthology.orchestra.run.smithi006.stdout: "cap-": 0, 2024-04-26T02:07:29.163 INFO:teuthology.orchestra.run.smithi006.stdout: "rss": 79824, 2024-04-26T02:07:29.164 INFO:teuthology.orchestra.run.smithi006.stdout: "heap": 215416 2024-04-26T02:07:29.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.164 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_server": { 2024-04-26T02:07:29.164 INFO:teuthology.orchestra.run.smithi006.stdout: "dispatch_client_request": 0, 2024-04-26T02:07:29.164 INFO:teuthology.orchestra.run.smithi006.stdout: "dispatch_server_request": 0, 2024-04-26T02:07:29.164 INFO:teuthology.orchestra.run.smithi006.stdout: "handle_client_request": 0, 2024-04-26T02:07:29.164 INFO:teuthology.orchestra.run.smithi006.stdout: "handle_client_session": 0, 2024-04-26T02:07:29.164 INFO:teuthology.orchestra.run.smithi006.stdout: "handle_peer_request": 0, 2024-04-26T02:07:29.164 INFO:teuthology.orchestra.run.smithi006.stdout: "req_create_latency": { 2024-04-26T02:07:29.164 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.165 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.165 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.165 INFO:teuthology.orchestra.run.smithi006.stdout: "req_getattr_latency": { 2024-04-26T02:07:29.165 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.165 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.165 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.165 INFO:teuthology.orchestra.run.smithi006.stdout: "req_getfilelock_latency": { 2024-04-26T02:07:29.166 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.166 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.166 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.166 INFO:teuthology.orchestra.run.smithi006.stdout: "req_link_latency": { 2024-04-26T02:07:29.166 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.166 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.166 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.167 INFO:teuthology.orchestra.run.smithi006.stdout: "req_lookup_latency": { 2024-04-26T02:07:29.167 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.167 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.167 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.167 INFO:teuthology.orchestra.run.smithi006.stdout: "req_lookuphash_latency": { 2024-04-26T02:07:29.167 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.167 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.168 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.168 INFO:teuthology.orchestra.run.smithi006.stdout: "req_lookupino_latency": { 2024-04-26T02:07:29.168 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.168 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.168 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.168 INFO:teuthology.orchestra.run.smithi006.stdout: "req_lookupname_latency": { 2024-04-26T02:07:29.169 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.169 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.169 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.169 INFO:teuthology.orchestra.run.smithi006.stdout: "req_lookupparent_latency": { 2024-04-26T02:07:29.169 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.169 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.169 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.170 INFO:teuthology.orchestra.run.smithi006.stdout: "req_lookupsnap_latency": { 2024-04-26T02:07:29.170 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.170 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.170 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.170 INFO:teuthology.orchestra.run.smithi006.stdout: "req_lssnap_latency": { 2024-04-26T02:07:29.170 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.170 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.170 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:28 smithi188 ceph-mon[32751]: pgmap v685: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 5.3 MiB/s wr, 179 op/s 2024-04-26T02:07:29.172 INFO:teuthology.orchestra.run.smithi006.stdout: "req_mkdir_latency": { 2024-04-26T02:07:29.172 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.172 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.172 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.172 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.172 INFO:teuthology.orchestra.run.smithi006.stdout: "req_mknod_latency": { 2024-04-26T02:07:29.172 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.173 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.173 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.173 INFO:teuthology.orchestra.run.smithi006.stdout: "req_mksnap_latency": { 2024-04-26T02:07:29.173 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.173 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.173 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.174 INFO:teuthology.orchestra.run.smithi006.stdout: "req_open_latency": { 2024-04-26T02:07:29.174 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.174 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.174 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.174 INFO:teuthology.orchestra.run.smithi006.stdout: "req_readdir_latency": { 2024-04-26T02:07:29.174 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.175 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.175 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.175 INFO:teuthology.orchestra.run.smithi006.stdout: "req_rename_latency": { 2024-04-26T02:07:29.175 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.175 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.175 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.175 INFO:teuthology.orchestra.run.smithi006.stdout: "req_renamesnap_latency": { 2024-04-26T02:07:29.176 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.176 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.176 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.176 INFO:teuthology.orchestra.run.smithi006.stdout: "req_snapdiff_latency": { 2024-04-26T02:07:29.176 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.176 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.176 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.177 INFO:teuthology.orchestra.run.smithi006.stdout: "req_rmdir_latency": { 2024-04-26T02:07:29.177 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.177 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.177 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.177 INFO:teuthology.orchestra.run.smithi006.stdout: "req_rmsnap_latency": { 2024-04-26T02:07:29.177 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.177 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.177 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.178 INFO:teuthology.orchestra.run.smithi006.stdout: "req_rmxattr_latency": { 2024-04-26T02:07:29.178 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.178 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.178 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.178 INFO:teuthology.orchestra.run.smithi006.stdout: "req_setattr_latency": { 2024-04-26T02:07:29.178 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.179 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.179 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.179 INFO:teuthology.orchestra.run.smithi006.stdout: "req_setdirlayout_latency": { 2024-04-26T02:07:29.179 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.179 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.179 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.179 INFO:teuthology.orchestra.run.smithi006.stdout: "req_setfilelock_latency": { 2024-04-26T02:07:29.180 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.180 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.180 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.180 INFO:teuthology.orchestra.run.smithi006.stdout: "req_setlayout_latency": { 2024-04-26T02:07:29.180 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.180 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.180 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.181 INFO:teuthology.orchestra.run.smithi006.stdout: "req_setxattr_latency": { 2024-04-26T02:07:29.181 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.181 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.181 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.181 INFO:teuthology.orchestra.run.smithi006.stdout: "req_symlink_latency": { 2024-04-26T02:07:29.181 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.181 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.182 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.182 INFO:teuthology.orchestra.run.smithi006.stdout: "req_unlink_latency": { 2024-04-26T02:07:29.182 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.182 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.182 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.182 INFO:teuthology.orchestra.run.smithi006.stdout: "cap_revoke_eviction": 0, 2024-04-26T02:07:29.183 INFO:teuthology.orchestra.run.smithi006.stdout: "cap_acquisition_throttle": 0, 2024-04-26T02:07:29.183 INFO:teuthology.orchestra.run.smithi006.stdout: "req_getvxattr_latency": { 2024-04-26T02:07:29.183 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.183 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.183 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.183 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:29.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.183 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_sessions": { 2024-04-26T02:07:29.183 INFO:teuthology.orchestra.run.smithi006.stdout: "session_count": 1, 2024-04-26T02:07:29.184 INFO:teuthology.orchestra.run.smithi006.stdout: "session_add": 3, 2024-04-26T02:07:29.184 INFO:teuthology.orchestra.run.smithi006.stdout: "session_remove": 2, 2024-04-26T02:07:29.184 INFO:teuthology.orchestra.run.smithi006.stdout: "sessions_open": 2, 2024-04-26T02:07:29.184 INFO:teuthology.orchestra.run.smithi006.stdout: "sessions_stale": 0, 2024-04-26T02:07:29.184 INFO:teuthology.orchestra.run.smithi006.stdout: "total_load": 0, 2024-04-26T02:07:29.184 INFO:teuthology.orchestra.run.smithi006.stdout: "average_load": 0, 2024-04-26T02:07:29.184 INFO:teuthology.orchestra.run.smithi006.stdout: "avg_session_uptime": 4071, 2024-04-26T02:07:29.184 INFO:teuthology.orchestra.run.smithi006.stdout: "mdthresh_evicted": 0 2024-04-26T02:07:29.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.185 INFO:teuthology.orchestra.run.smithi006.stdout: "mempool": { 2024-04-26T02:07:29.185 INFO:teuthology.orchestra.run.smithi006.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:29.185 INFO:teuthology.orchestra.run.smithi006.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:29.185 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:29.185 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:29.185 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:29.185 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:29.185 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:29.185 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:29.186 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:29.186 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:29.186 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:29.186 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:29.186 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:29.186 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:29.186 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:29.186 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:29.187 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:29.187 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:29.187 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:29.187 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:29.187 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:29.187 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:29.187 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:29.187 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:29.187 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:29.188 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:29.188 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:29.188 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:29.188 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:29.188 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:29.188 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:29.188 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_items": 0, 2024-04-26T02:07:29.188 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:29.188 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:29.189 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:29.189 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:29.189 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_anon_bytes": 328093, 2024-04-26T02:07:29.189 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_anon_items": 2390, 2024-04-26T02:07:29.189 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:29.189 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:29.189 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bytes": 0, 2024-04-26T02:07:29.189 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_items": 0, 2024-04-26T02:07:29.189 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:29.190 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:29.190 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:29.190 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:29.190 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_bytes": 8320, 2024-04-26T02:07:29.190 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_items": 173, 2024-04-26T02:07:29.190 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:29.190 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:29.190 INFO:teuthology.orchestra.run.smithi006.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:29.191 INFO:teuthology.orchestra.run.smithi006.stdout: "pgmap_items": 0, 2024-04-26T02:07:29.191 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_co_bytes": 21458629, 2024-04-26T02:07:29.191 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_co_items": 173382, 2024-04-26T02:07:29.191 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:29.191 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_1_items": 0, 2024-04-26T02:07:29.191 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:29.191 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_2_items": 0 2024-04-26T02:07:29.191 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.191 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter": { 2024-04-26T02:07:29.192 INFO:teuthology.orchestra.run.smithi006.stdout: "op_active": 0, 2024-04-26T02:07:29.192 INFO:teuthology.orchestra.run.smithi006.stdout: "op_laggy": 0, 2024-04-26T02:07:29.192 INFO:teuthology.orchestra.run.smithi006.stdout: "op_send": 5716, 2024-04-26T02:07:29.192 INFO:teuthology.orchestra.run.smithi006.stdout: "op_send_bytes": 21, 2024-04-26T02:07:29.192 INFO:teuthology.orchestra.run.smithi006.stdout: "op_resend": 3, 2024-04-26T02:07:29.192 INFO:teuthology.orchestra.run.smithi006.stdout: "op_reply": 5716, 2024-04-26T02:07:29.192 INFO:teuthology.orchestra.run.smithi006.stdout: "op_latency": { 2024-04-26T02:07:29.192 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 5716, 2024-04-26T02:07:29.193 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 9.141897286, 2024-04-26T02:07:29.193 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.001599352 2024-04-26T02:07:29.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.193 INFO:teuthology.orchestra.run.smithi006.stdout: "op_inflight": 0, 2024-04-26T02:07:29.193 INFO:teuthology.orchestra.run.smithi006.stdout: "oplen_avg": { 2024-04-26T02:07:29.193 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 5716, 2024-04-26T02:07:29.193 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 5717 2024-04-26T02:07:29.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.193 INFO:teuthology.orchestra.run.smithi006.stdout: "op": 5716, 2024-04-26T02:07:29.194 INFO:teuthology.orchestra.run.smithi006.stdout: "op_r": 5716, 2024-04-26T02:07:29.194 INFO:teuthology.orchestra.run.smithi006.stdout: "op_w": 0, 2024-04-26T02:07:29.194 INFO:teuthology.orchestra.run.smithi006.stdout: "op_rmw": 0, 2024-04-26T02:07:29.194 INFO:teuthology.orchestra.run.smithi006.stdout: "op_pg": 0, 2024-04-26T02:07:29.194 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_stat": 3419, 2024-04-26T02:07:29.194 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_create": 0, 2024-04-26T02:07:29.194 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_read": 2295, 2024-04-26T02:07:29.194 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_write": 0, 2024-04-26T02:07:29.194 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_writefull": 0, 2024-04-26T02:07:29.195 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_writesame": 0, 2024-04-26T02:07:29.195 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_append": 0, 2024-04-26T02:07:29.195 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_zero": 0, 2024-04-26T02:07:29.195 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_truncate": 0, 2024-04-26T02:07:29.195 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_delete": 0, 2024-04-26T02:07:29.195 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_mapext": 0, 2024-04-26T02:07:29.195 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_sparse_read": 0, 2024-04-26T02:07:29.195 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_clonerange": 0, 2024-04-26T02:07:29.195 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_getxattr": 1, 2024-04-26T02:07:29.196 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_setxattr": 0, 2024-04-26T02:07:29.196 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_cmpxattr": 0, 2024-04-26T02:07:29.196 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_rmxattr": 0, 2024-04-26T02:07:29.196 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_resetxattrs": 0, 2024-04-26T02:07:29.196 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_call": 0, 2024-04-26T02:07:29.196 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_watch": 0, 2024-04-26T02:07:29.196 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_notify": 0, 2024-04-26T02:07:29.196 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T02:07:29.196 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_pgls": 0, 2024-04-26T02:07:29.196 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_pgls_filter": 0, 2024-04-26T02:07:29.197 INFO:teuthology.orchestra.run.smithi006.stdout: "osdop_other": 0, 2024-04-26T02:07:29.197 INFO:teuthology.orchestra.run.smithi006.stdout: "linger_active": 0, 2024-04-26T02:07:29.197 INFO:teuthology.orchestra.run.smithi006.stdout: "linger_send": 0, 2024-04-26T02:07:29.197 INFO:teuthology.orchestra.run.smithi006.stdout: "linger_resend": 0, 2024-04-26T02:07:29.197 INFO:teuthology.orchestra.run.smithi006.stdout: "linger_ping": 0, 2024-04-26T02:07:29.197 INFO:teuthology.orchestra.run.smithi006.stdout: "poolop_active": 0, 2024-04-26T02:07:29.197 INFO:teuthology.orchestra.run.smithi006.stdout: "poolop_send": 0, 2024-04-26T02:07:29.197 INFO:teuthology.orchestra.run.smithi006.stdout: "poolop_resend": 0, 2024-04-26T02:07:29.197 INFO:teuthology.orchestra.run.smithi006.stdout: "poolstat_active": 0, 2024-04-26T02:07:29.198 INFO:teuthology.orchestra.run.smithi006.stdout: "poolstat_send": 0, 2024-04-26T02:07:29.198 INFO:teuthology.orchestra.run.smithi006.stdout: "poolstat_resend": 0, 2024-04-26T02:07:29.198 INFO:teuthology.orchestra.run.smithi006.stdout: "statfs_active": 0, 2024-04-26T02:07:29.198 INFO:teuthology.orchestra.run.smithi006.stdout: "statfs_send": 0, 2024-04-26T02:07:29.198 INFO:teuthology.orchestra.run.smithi006.stdout: "statfs_resend": 0, 2024-04-26T02:07:29.198 INFO:teuthology.orchestra.run.smithi006.stdout: "command_active": 0, 2024-04-26T02:07:29.198 INFO:teuthology.orchestra.run.smithi006.stdout: "command_send": 0, 2024-04-26T02:07:29.198 INFO:teuthology.orchestra.run.smithi006.stdout: "command_resend": 0, 2024-04-26T02:07:29.198 INFO:teuthology.orchestra.run.smithi006.stdout: "map_epoch": 132, 2024-04-26T02:07:29.199 INFO:teuthology.orchestra.run.smithi006.stdout: "map_full": 0, 2024-04-26T02:07:29.199 INFO:teuthology.orchestra.run.smithi006.stdout: "map_inc": 41, 2024-04-26T02:07:29.199 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_sessions": 11, 2024-04-26T02:07:29.199 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_session_open": 12, 2024-04-26T02:07:29.199 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_session_close": 1, 2024-04-26T02:07:29.199 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_laggy": 0, 2024-04-26T02:07:29.199 INFO:teuthology.orchestra.run.smithi006.stdout: "omap_wr": 0, 2024-04-26T02:07:29.199 INFO:teuthology.orchestra.run.smithi006.stdout: "omap_rd": 2, 2024-04-26T02:07:29.199 INFO:teuthology.orchestra.run.smithi006.stdout: "omap_del": 0 2024-04-26T02:07:29.199 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.199 INFO:teuthology.orchestra.run.smithi006.stdout: "oft": { 2024-04-26T02:07:29.200 INFO:teuthology.orchestra.run.smithi006.stdout: "omap_total_objs": 0, 2024-04-26T02:07:29.200 INFO:teuthology.orchestra.run.smithi006.stdout: "omap_total_kv_pairs": 0, 2024-04-26T02:07:29.200 INFO:teuthology.orchestra.run.smithi006.stdout: "omap_total_updates": 0, 2024-04-26T02:07:29.200 INFO:teuthology.orchestra.run.smithi006.stdout: "omap_total_removes": 0 2024-04-26T02:07:29.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.200 INFO:teuthology.orchestra.run.smithi006.stdout: "osd-slow-ops": { 2024-04-26T02:07:29.200 INFO:teuthology.orchestra.run.smithi006.stdout: "slow_ops_count": 0 2024-04-26T02:07:29.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.200 INFO:teuthology.orchestra.run.smithi006.stdout: "purge_queue": { 2024-04-26T02:07:29.201 INFO:teuthology.orchestra.run.smithi006.stdout: "pq_executing_ops": 0, 2024-04-26T02:07:29.201 INFO:teuthology.orchestra.run.smithi006.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T02:07:29.201 INFO:teuthology.orchestra.run.smithi006.stdout: "pq_executing": 0, 2024-04-26T02:07:29.201 INFO:teuthology.orchestra.run.smithi006.stdout: "pq_executing_high_water": 0, 2024-04-26T02:07:29.201 INFO:teuthology.orchestra.run.smithi006.stdout: "pq_executed_ops": 0, 2024-04-26T02:07:29.201 INFO:teuthology.orchestra.run.smithi006.stdout: "pq_executed": 0, 2024-04-26T02:07:29.201 INFO:teuthology.orchestra.run.smithi006.stdout: "pq_item_in_journal": 0 2024-04-26T02:07:29.201 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.201 INFO:teuthology.orchestra.run.smithi006.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:29.202 INFO:teuthology.orchestra.run.smithi006.stdout: "val": 0, 2024-04-26T02:07:29.202 INFO:teuthology.orchestra.run.smithi006.stdout: "max": 104857600, 2024-04-26T02:07:29.202 INFO:teuthology.orchestra.run.smithi006.stdout: "get_started": 0, 2024-04-26T02:07:29.202 INFO:teuthology.orchestra.run.smithi006.stdout: "get": 6222, 2024-04-26T02:07:29.202 INFO:teuthology.orchestra.run.smithi006.stdout: "get_sum": 566484105, 2024-04-26T02:07:29.202 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:29.202 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_success": 6222, 2024-04-26T02:07:29.202 INFO:teuthology.orchestra.run.smithi006.stdout: "take": 0, 2024-04-26T02:07:29.203 INFO:teuthology.orchestra.run.smithi006.stdout: "take_sum": 0, 2024-04-26T02:07:29.203 INFO:teuthology.orchestra.run.smithi006.stdout: "put": 6222, 2024-04-26T02:07:29.203 INFO:teuthology.orchestra.run.smithi006.stdout: "put_sum": 566484105, 2024-04-26T02:07:29.203 INFO:teuthology.orchestra.run.smithi006.stdout: "wait": { 2024-04-26T02:07:29.203 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.203 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.203 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.203 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:29.203 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.203 INFO:teuthology.orchestra.run.smithi006.stdout: "throttle-objecter_bytes": { 2024-04-26T02:07:29.204 INFO:teuthology.orchestra.run.smithi006.stdout: "val": 0, 2024-04-26T02:07:29.204 INFO:teuthology.orchestra.run.smithi006.stdout: "max": 104857600, 2024-04-26T02:07:29.204 INFO:teuthology.orchestra.run.smithi006.stdout: "get_started": 0, 2024-04-26T02:07:29.204 INFO:teuthology.orchestra.run.smithi006.stdout: "get": 0, 2024-04-26T02:07:29.204 INFO:teuthology.orchestra.run.smithi006.stdout: "get_sum": 0, 2024-04-26T02:07:29.204 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:29.204 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:29.205 INFO:teuthology.orchestra.run.smithi006.stdout: "take": 5716, 2024-04-26T02:07:29.205 INFO:teuthology.orchestra.run.smithi006.stdout: "take_sum": 563516758, 2024-04-26T02:07:29.205 INFO:teuthology.orchestra.run.smithi006.stdout: "put": 5716, 2024-04-26T02:07:29.205 INFO:teuthology.orchestra.run.smithi006.stdout: "put_sum": 563516758, 2024-04-26T02:07:29.205 INFO:teuthology.orchestra.run.smithi006.stdout: "wait": { 2024-04-26T02:07:29.205 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.205 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.205 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.205 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:29.206 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.206 INFO:teuthology.orchestra.run.smithi006.stdout: "throttle-objecter_ops": { 2024-04-26T02:07:29.206 INFO:teuthology.orchestra.run.smithi006.stdout: "val": 0, 2024-04-26T02:07:29.206 INFO:teuthology.orchestra.run.smithi006.stdout: "max": 1024, 2024-04-26T02:07:29.206 INFO:teuthology.orchestra.run.smithi006.stdout: "get_started": 0, 2024-04-26T02:07:29.206 INFO:teuthology.orchestra.run.smithi006.stdout: "get": 0, 2024-04-26T02:07:29.206 INFO:teuthology.orchestra.run.smithi006.stdout: "get_sum": 0, 2024-04-26T02:07:29.206 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:29.206 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:29.207 INFO:teuthology.orchestra.run.smithi006.stdout: "take": 5716, 2024-04-26T02:07:29.207 INFO:teuthology.orchestra.run.smithi006.stdout: "take_sum": 5716, 2024-04-26T02:07:29.207 INFO:teuthology.orchestra.run.smithi006.stdout: "put": 5716, 2024-04-26T02:07:29.207 INFO:teuthology.orchestra.run.smithi006.stdout: "put_sum": 5716, 2024-04-26T02:07:29.207 INFO:teuthology.orchestra.run.smithi006.stdout: "wait": { 2024-04-26T02:07:29.207 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.207 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.207 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.208 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:29.208 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.208 INFO:teuthology.orchestra.run.smithi006.stdout: "throttle-write_buf_throttle": { 2024-04-26T02:07:29.208 INFO:teuthology.orchestra.run.smithi006.stdout: "val": 0, 2024-04-26T02:07:29.208 INFO:teuthology.orchestra.run.smithi006.stdout: "max": 3758096384, 2024-04-26T02:07:29.208 INFO:teuthology.orchestra.run.smithi006.stdout: "get_started": 0, 2024-04-26T02:07:29.208 INFO:teuthology.orchestra.run.smithi006.stdout: "get": 0, 2024-04-26T02:07:29.208 INFO:teuthology.orchestra.run.smithi006.stdout: "get_sum": 0, 2024-04-26T02:07:29.208 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:29.209 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:29.209 INFO:teuthology.orchestra.run.smithi006.stdout: "take": 0, 2024-04-26T02:07:29.209 INFO:teuthology.orchestra.run.smithi006.stdout: "take_sum": 0, 2024-04-26T02:07:29.209 INFO:teuthology.orchestra.run.smithi006.stdout: "put": 0, 2024-04-26T02:07:29.209 INFO:teuthology.orchestra.run.smithi006.stdout: "put_sum": 0, 2024-04-26T02:07:29.209 INFO:teuthology.orchestra.run.smithi006.stdout: "wait": { 2024-04-26T02:07:29.209 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.209 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.210 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.210 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:29.210 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:29.210 INFO:teuthology.orchestra.run.smithi006.stdout: "throttle-write_buf_throttle-0x562e9e486780": { 2024-04-26T02:07:29.210 INFO:teuthology.orchestra.run.smithi006.stdout: "val": 0, 2024-04-26T02:07:29.210 INFO:teuthology.orchestra.run.smithi006.stdout: "max": 3758096384, 2024-04-26T02:07:29.210 INFO:teuthology.orchestra.run.smithi006.stdout: "get_started": 0, 2024-04-26T02:07:29.210 INFO:teuthology.orchestra.run.smithi006.stdout: "get": 0, 2024-04-26T02:07:29.211 INFO:teuthology.orchestra.run.smithi006.stdout: "get_sum": 0, 2024-04-26T02:07:29.211 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:29.211 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:29.211 INFO:teuthology.orchestra.run.smithi006.stdout: "take": 0, 2024-04-26T02:07:29.211 INFO:teuthology.orchestra.run.smithi006.stdout: "take_sum": 0, 2024-04-26T02:07:29.211 INFO:teuthology.orchestra.run.smithi006.stdout: "put": 0, 2024-04-26T02:07:29.211 INFO:teuthology.orchestra.run.smithi006.stdout: "put_sum": 0, 2024-04-26T02:07:29.211 INFO:teuthology.orchestra.run.smithi006.stdout: "wait": { 2024-04-26T02:07:29.211 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:29.212 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.212 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.212 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:29.212 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:29.212 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:07:30.154 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-26T02:07:30.154 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-26T02:07:30.155 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-04-26T02:07:30.155 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-04-26T02:07:30.155 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-04-26T02:07:30.155 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-04-26T02:07:30.155 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T02:07:30.155 DEBUG:tasks.check_counter:Getting stats from j 2024-04-26T02:07:30.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-26T02:07:30.635 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:30.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:30 smithi029 ceph-mon[32775]: pgmap v686: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 3.4 MiB/s wr, 115 op/s 2024-04-26T02:07:30.943 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:30 smithi188 ceph-mon[32751]: pgmap v686: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 3.4 MiB/s wr, 115 op/s 2024-04-26T02:07:30.952 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:30 smithi006 ceph-mon[28419]: pgmap v686: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 3.4 MiB/s wr, 115 op/s 2024-04-26T02:07:33.030 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-26T02:07:33.030 INFO:teuthology.orchestra.run.smithi006.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:33.030 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_messages": 430, 2024-04-26T02:07:33.031 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages": 418, 2024-04-26T02:07:33.031 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_bytes": 1787680, 2024-04-26T02:07:33.031 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_bytes": 258976, 2024-04-26T02:07:33.031 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:33.031 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_active_connections": 0, 2024-04-26T02:07:33.031 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_total_time": 0.164916953, 2024-04-26T02:07:33.031 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_send_time": 0.067610642, 2024-04-26T02:07:33.031 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_recv_time": 0.082673097, 2024-04-26T02:07:33.032 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_fast_dispatch_time": 0.020341963, 2024-04-26T02:07:33.032 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:33.032 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 417, 2024-04-26T02:07:33.032 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.043557710, 2024-04-26T02:07:33.032 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000104454 2024-04-26T02:07:33.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:33.032 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:33.032 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:33.033 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:33.033 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:33.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:33.033 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_encrypted_bytes": 1787680, 2024-04-26T02:07:33.033 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_encrypted_bytes": 258976 2024-04-26T02:07:33.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:33.033 INFO:teuthology.orchestra.run.smithi006.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:33.033 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_messages": 3, 2024-04-26T02:07:33.033 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages": 320, 2024-04-26T02:07:33.034 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_bytes": 480, 2024-04-26T02:07:33.034 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_bytes": 290704, 2024-04-26T02:07:33.034 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:33.034 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:33.034 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_total_time": 0.061453908, 2024-04-26T02:07:33.034 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_send_time": 0.047721881, 2024-04-26T02:07:33.035 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_recv_time": 0.004807591, 2024-04-26T02:07:33.035 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:33.035 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:33.035 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 317, 2024-04-26T02:07:33.035 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.048253160, 2024-04-26T02:07:33.035 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000152218 2024-04-26T02:07:33.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:33.036 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:33.036 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:33.036 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:33.036 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:33.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:33.036 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T02:07:33.036 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_encrypted_bytes": 290704 2024-04-26T02:07:33.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:33.038 INFO:teuthology.orchestra.run.smithi006.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:33.038 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_messages": 4, 2024-04-26T02:07:33.038 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages": 4, 2024-04-26T02:07:33.038 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_bytes": 2992, 2024-04-26T02:07:33.038 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_bytes": 448, 2024-04-26T02:07:33.039 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_created_connections": 2, 2024-04-26T02:07:33.039 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_active_connections": 1, 2024-04-26T02:07:33.039 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_total_time": 0.001563189, 2024-04-26T02:07:33.039 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_send_time": 0.000092723, 2024-04-26T02:07:33.039 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_recv_time": 0.001416579, 2024-04-26T02:07:33.039 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:33.039 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:33.039 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 3, 2024-04-26T02:07:33.039 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.001652107, 2024-04-26T02:07:33.040 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000550702 2024-04-26T02:07:33.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:33.040 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:33.040 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:33.040 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:33.040 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:33.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:33.040 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T02:07:33.041 INFO:teuthology.orchestra.run.smithi006.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T02:07:33.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:33.041 INFO:teuthology.orchestra.run.smithi006.stdout: "cct": { 2024-04-26T02:07:33.041 INFO:teuthology.orchestra.run.smithi006.stdout: "total_workers": 0, 2024-04-26T02:07:33.041 INFO:teuthology.orchestra.run.smithi006.stdout: "unhealthy_workers": 0 2024-04-26T02:07:33.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:33.041 INFO:teuthology.orchestra.run.smithi006.stdout: "mempool": { 2024-04-26T02:07:33.041 INFO:teuthology.orchestra.run.smithi006.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:33.041 INFO:teuthology.orchestra.run.smithi006.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:33.042 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:33.042 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:33.042 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:33.042 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:33.042 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:33.042 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:33.042 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:33.043 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:33.043 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:33.043 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:33.043 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:33.044 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:33.044 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:33.044 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:33.044 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:33.045 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:33.045 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:33.045 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:33.045 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:33.045 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:33.046 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:33.046 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:33.046 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:33.046 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:33.047 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:33.047 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:33.047 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:33.047 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:33.047 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:33.047 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_items": 0, 2024-04-26T02:07:33.047 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:33.048 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:33.048 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:33.048 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:33.048 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_anon_bytes": 149745, 2024-04-26T02:07:33.048 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_anon_items": 31, 2024-04-26T02:07:33.048 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:33.048 INFO:teuthology.orchestra.run.smithi006.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:33.048 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bytes": 0, 2024-04-26T02:07:33.048 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_items": 0, 2024-04-26T02:07:33.049 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:33.049 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:33.049 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:33.049 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:33.049 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_bytes": 0, 2024-04-26T02:07:33.049 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_items": 0, 2024-04-26T02:07:33.049 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:33.049 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:33.050 INFO:teuthology.orchestra.run.smithi006.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:33.050 INFO:teuthology.orchestra.run.smithi006.stdout: "pgmap_items": 0, 2024-04-26T02:07:33.050 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_co_bytes": 928, 2024-04-26T02:07:33.050 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_co_items": 2, 2024-04-26T02:07:33.050 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:33.050 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_1_items": 0, 2024-04-26T02:07:33.050 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:33.050 INFO:teuthology.orchestra.run.smithi006.stdout: "unittest_2_items": 0 2024-04-26T02:07:33.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-26T02:07:33.051 INFO:teuthology.orchestra.run.smithi006.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:33.051 INFO:teuthology.orchestra.run.smithi006.stdout: "val": 0, 2024-04-26T02:07:33.051 INFO:teuthology.orchestra.run.smithi006.stdout: "max": 104857600, 2024-04-26T02:07:33.051 INFO:teuthology.orchestra.run.smithi006.stdout: "get_started": 0, 2024-04-26T02:07:33.051 INFO:teuthology.orchestra.run.smithi006.stdout: "get": 433, 2024-04-26T02:07:33.051 INFO:teuthology.orchestra.run.smithi006.stdout: "get_sum": 1726939, 2024-04-26T02:07:33.051 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:33.051 INFO:teuthology.orchestra.run.smithi006.stdout: "get_or_fail_success": 433, 2024-04-26T02:07:33.052 INFO:teuthology.orchestra.run.smithi006.stdout: "take": 0, 2024-04-26T02:07:33.052 INFO:teuthology.orchestra.run.smithi006.stdout: "take_sum": 0, 2024-04-26T02:07:33.052 INFO:teuthology.orchestra.run.smithi006.stdout: "put": 433, 2024-04-26T02:07:33.052 INFO:teuthology.orchestra.run.smithi006.stdout: "put_sum": 1726939, 2024-04-26T02:07:33.052 INFO:teuthology.orchestra.run.smithi006.stdout: "wait": { 2024-04-26T02:07:33.052 INFO:teuthology.orchestra.run.smithi006.stdout: "avgcount": 0, 2024-04-26T02:07:33.052 INFO:teuthology.orchestra.run.smithi006.stdout: "sum": 0.000000000, 2024-04-26T02:07:33.052 INFO:teuthology.orchestra.run.smithi006.stdout: "avgtime": 0.000000000 2024-04-26T02:07:33.052 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:33.053 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-26T02:07:33.053 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-26T02:07:33.053 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:32 smithi006 ceph-mon[28419]: pgmap v687: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 5.2 MiB/s wr, 171 op/s 2024-04-26T02:07:33.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:32 smithi188 ceph-mon[32751]: pgmap v687: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 5.2 MiB/s wr, 171 op/s 2024-04-26T02:07:33.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:32 smithi029 ceph-mon[32775]: pgmap v687: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 5.2 MiB/s wr, 171 op/s 2024-04-26T02:07:33.840 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-26T02:07:33.841 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-26T02:07:33.841 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-26T02:07:33.841 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-26T02:07:33.841 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-26T02:07:33.841 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-26T02:07:33.841 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-26T02:07:33.841 DEBUG:tasks.check_counter:Getting stats from b 2024-04-26T02:07:33.841 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-26T02:07:34.098 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T02:07:34.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:34 smithi029 ceph-mon[32775]: pgmap v688: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 4.4 MiB/s wr, 135 op/s 2024-04-26T02:07:35.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:34 smithi006 ceph-mon[28419]: pgmap v688: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 4.4 MiB/s wr, 135 op/s 2024-04-26T02:07:35.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:34 smithi188 ceph-mon[32751]: pgmap v688: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 4.4 MiB/s wr, 135 op/s 2024-04-26T02:07:35.988 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-26T02:07:35.989 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:35.989 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 236836, 2024-04-26T02:07:35.989 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 198604, 2024-04-26T02:07:35.989 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 73818744, 2024-04-26T02:07:35.989 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 297475866, 2024-04-26T02:07:35.989 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 14, 2024-04-26T02:07:35.989 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:35.989 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 12.738133064, 2024-04-26T02:07:35.990 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 5.008350440, 2024-04-26T02:07:35.990 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 8.785251960, 2024-04-26T02:07:35.990 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.756499114, 2024-04-26T02:07:35.990 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:35.990 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 198602, 2024-04-26T02:07:35.990 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 4.763681217, 2024-04-26T02:07:35.990 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000023986 2024-04-26T02:07:35.990 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:35.991 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:35.991 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 387548, 2024-04-26T02:07:35.991 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.059272714, 2024-04-26T02:07:35.991 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000152 2024-04-26T02:07:35.991 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:35.991 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T02:07:35.991 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T02:07:35.991 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:35.992 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:35.992 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 278675, 2024-04-26T02:07:35.992 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 376777, 2024-04-26T02:07:35.992 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 90709785, 2024-04-26T02:07:35.992 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 445986668, 2024-04-26T02:07:35.992 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 17, 2024-04-26T02:07:35.992 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 5, 2024-04-26T02:07:35.992 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 16.480330820, 2024-04-26T02:07:35.992 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 6.965170169, 2024-04-26T02:07:35.993 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 10.843255642, 2024-04-26T02:07:35.993 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.867636041, 2024-04-26T02:07:35.993 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:35.993 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 376775, 2024-04-26T02:07:35.993 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 10.148209625, 2024-04-26T02:07:35.993 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000026934 2024-04-26T02:07:35.993 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:35.993 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:35.993 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 514949, 2024-04-26T02:07:35.994 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.101543889, 2024-04-26T02:07:35.994 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000197 2024-04-26T02:07:35.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:35.994 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 7216, 2024-04-26T02:07:35.994 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 168112 2024-04-26T02:07:35.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:35.994 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:35.994 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 137433, 2024-04-26T02:07:35.994 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 126328, 2024-04-26T02:07:35.995 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 49069065, 2024-04-26T02:07:35.995 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 223666795, 2024-04-26T02:07:35.995 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 7, 2024-04-26T02:07:35.995 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 6, 2024-04-26T02:07:35.995 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 7.539367452, 2024-04-26T02:07:35.995 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 2.952045991, 2024-04-26T02:07:35.995 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 4.644055158, 2024-04-26T02:07:35.995 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.763536398, 2024-04-26T02:07:35.995 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:35.995 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 126327, 2024-04-26T02:07:35.996 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 3.391262769, 2024-04-26T02:07:35.996 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000026845 2024-04-26T02:07:35.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:35.996 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:35.996 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 229334, 2024-04-26T02:07:35.996 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.035349387, 2024-04-26T02:07:35.996 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000154 2024-04-26T02:07:35.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:35.996 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 1990704, 2024-04-26T02:07:35.996 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 822736 2024-04-26T02:07:35.997 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:35.997 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-04-26T02:07:35.997 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 1, 2024-04-26T02:07:35.997 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-04-26T02:07:35.997 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:35.997 INFO:teuthology.orchestra.run.smithi029.stdout: "finisher-MDSRank": { 2024-04-26T02:07:35.997 INFO:teuthology.orchestra.run.smithi029.stdout: "queue_len": 0, 2024-04-26T02:07:35.997 INFO:teuthology.orchestra.run.smithi029.stdout: "complete_latency": { 2024-04-26T02:07:35.997 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 54227, 2024-04-26T02:07:35.998 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 82.897363404, 2024-04-26T02:07:35.998 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.001528710 2024-04-26T02:07:35.998 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:35.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:35.998 INFO:teuthology.orchestra.run.smithi029.stdout: "finisher-PurgeQueue": { 2024-04-26T02:07:35.998 INFO:teuthology.orchestra.run.smithi029.stdout: "queue_len": 0, 2024-04-26T02:07:35.998 INFO:teuthology.orchestra.run.smithi029.stdout: "complete_latency": { 2024-04-26T02:07:35.998 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 38438, 2024-04-26T02:07:35.999 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 5.615807665, 2024-04-26T02:07:35.999 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000146100 2024-04-26T02:07:35.999 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:35.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:35.999 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": { 2024-04-26T02:07:35.999 INFO:teuthology.orchestra.run.smithi029.stdout: "request": 193832, 2024-04-26T02:07:36.000 INFO:teuthology.orchestra.run.smithi029.stdout: "reply": 167809, 2024-04-26T02:07:36.000 INFO:teuthology.orchestra.run.smithi029.stdout: "reply_latency": { 2024-04-26T02:07:36.000 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 167809, 2024-04-26T02:07:36.000 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 1084.908191653, 2024-04-26T02:07:36.000 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.006465137 2024-04-26T02:07:36.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.000 INFO:teuthology.orchestra.run.smithi029.stdout: "slow_reply": 0, 2024-04-26T02:07:36.000 INFO:teuthology.orchestra.run.smithi029.stdout: "forward": 26023, 2024-04-26T02:07:36.000 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_fetch_complete": 11, 2024-04-26T02:07:36.000 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_fetch_keys": 0, 2024-04-26T02:07:36.001 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_commit": 162, 2024-04-26T02:07:36.001 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_split": 14, 2024-04-26T02:07:36.001 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_merge": 8, 2024-04-26T02:07:36.001 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes": 3415, 2024-04-26T02:07:36.002 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_top": 1432, 2024-04-26T02:07:36.002 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_bottom": 1971, 2024-04-26T02:07:36.002 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_pin_tail": 12, 2024-04-26T02:07:36.002 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_pinned": 1369, 2024-04-26T02:07:36.003 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_expired": 277, 2024-04-26T02:07:36.003 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_with_caps": 195, 2024-04-26T02:07:36.003 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": 195, 2024-04-26T02:07:36.003 INFO:teuthology.orchestra.run.smithi029.stdout: "subtrees": 14, 2024-04-26T02:07:36.004 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse": 299521, 2024-04-26T02:07:36.004 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_hit": 162432, 2024-04-26T02:07:36.004 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_forward": 26015, 2024-04-26T02:07:36.004 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_discover": 179, 2024-04-26T02:07:36.004 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_dir_fetch": 0, 2024-04-26T02:07:36.004 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_remote_ino": 0, 2024-04-26T02:07:36.004 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_lock": 2838, 2024-04-26T02:07:36.004 INFO:teuthology.orchestra.run.smithi029.stdout: "load_cent": 32629, 2024-04-26T02:07:36.005 INFO:teuthology.orchestra.run.smithi029.stdout: "q": 0, 2024-04-26T02:07:36.005 INFO:teuthology.orchestra.run.smithi029.stdout: "exported": 5, 2024-04-26T02:07:36.005 INFO:teuthology.orchestra.run.smithi029.stdout: "exported_inodes": 196, 2024-04-26T02:07:36.005 INFO:teuthology.orchestra.run.smithi029.stdout: "imported": 5, 2024-04-26T02:07:36.006 INFO:teuthology.orchestra.run.smithi029.stdout: "imported_inodes": 501, 2024-04-26T02:07:36.006 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_dir_fetch": 0, 2024-04-26T02:07:36.006 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_backtrace_fetch": 0, 2024-04-26T02:07:36.006 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_peer_discover": 0, 2024-04-26T02:07:36.006 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rfiles": 727, 2024-04-26T02:07:36.006 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rbytes": 212049877, 2024-04-26T02:07:36.006 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rsnaps": 7, 2024-04-26T02:07:36.007 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T02:07:36.007 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_set_tag": 0, 2024-04-26T02:07:36.007 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T02:07:36.007 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_inotable_repaired": 0, 2024-04-26T02:07:36.007 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dir_inodes": 0, 2024-04-26T02:07:36.007 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T02:07:36.007 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T02:07:36.007 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_file_inodes": 0, 2024-04-26T02:07:36.008 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_inode_file_caps": 20, 2024-04-26T02:07:36.008 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_revoke": 10713, 2024-04-26T02:07:36.008 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_grant": 83166, 2024-04-26T02:07:36.008 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T02:07:36.008 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T02:07:36.008 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T02:07:36.008 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_caps": 49385, 2024-04-26T02:07:36.008 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_caps_dirty": 18794, 2024-04-26T02:07:36.008 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_cap_release": 9570, 2024-04-26T02:07:36.009 INFO:teuthology.orchestra.run.smithi029.stdout: "process_request_cap_release": 130336 2024-04-26T02:07:36.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.009 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache": { 2024-04-26T02:07:36.009 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_update": 559, 2024-04-26T02:07:36.009 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_update_receipt": 1427, 2024-04-26T02:07:36.009 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_try_discover": 772, 2024-04-26T02:07:36.009 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_send_discover": 1440, 2024-04-26T02:07:36.009 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_handle_discover": 633, 2024-04-26T02:07:36.009 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays": 0, 2024-04-26T02:07:36.010 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays_delayed": 0, 2024-04-26T02:07:36.010 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays_enqueuing": 0, 2024-04-26T02:07:36.010 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_created": 20947, 2024-04-26T02:07:36.010 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_enqueued": 20947, 2024-04-26T02:07:36.010 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_reintegrated": 0, 2024-04-26T02:07:36.010 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_migrated": 0, 2024-04-26T02:07:36.010 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_processing": 0, 2024-04-26T02:07:36.010 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_enqueued": 0, 2024-04-26T02:07:36.011 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_prioritized": 0, 2024-04-26T02:07:36.011 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_started": 0, 2024-04-26T02:07:36.011 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_completed": 0, 2024-04-26T02:07:36.011 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_quiesce_path": 14, 2024-04-26T02:07:36.011 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_quiesce_inode": 1779, 2024-04-26T02:07:36.011 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T02:07:36.011 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_exportdir": 11, 2024-04-26T02:07:36.011 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_flush": 0, 2024-04-26T02:07:36.012 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_fragmentdir": 22, 2024-04-26T02:07:36.012 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_fragstats": 0, 2024-04-26T02:07:36.012 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_inodestats": 0 2024-04-26T02:07:36.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.012 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log": { 2024-04-26T02:07:36.012 INFO:teuthology.orchestra.run.smithi029.stdout: "evlrg": 0, 2024-04-26T02:07:36.012 INFO:teuthology.orchestra.run.smithi029.stdout: "evadd": 128136, 2024-04-26T02:07:36.013 INFO:teuthology.orchestra.run.smithi029.stdout: "evex": 45864, 2024-04-26T02:07:36.013 INFO:teuthology.orchestra.run.smithi029.stdout: "evtrm": 36871, 2024-04-26T02:07:36.013 INFO:teuthology.orchestra.run.smithi029.stdout: "ev": 91265, 2024-04-26T02:07:36.013 INFO:teuthology.orchestra.run.smithi029.stdout: "evexg": 0, 2024-04-26T02:07:36.013 INFO:teuthology.orchestra.run.smithi029.stdout: "evexd": 8993, 2024-04-26T02:07:36.013 INFO:teuthology.orchestra.run.smithi029.stdout: "segadd": 186, 2024-04-26T02:07:36.013 INFO:teuthology.orchestra.run.smithi029.stdout: "segex": 58, 2024-04-26T02:07:36.013 INFO:teuthology.orchestra.run.smithi029.stdout: "segtrm": 45, 2024-04-26T02:07:36.014 INFO:teuthology.orchestra.run.smithi029.stdout: "seg": 141, 2024-04-26T02:07:36.014 INFO:teuthology.orchestra.run.smithi029.stdout: "segmjr": 8, 2024-04-26T02:07:36.014 INFO:teuthology.orchestra.run.smithi029.stdout: "segexg": 0, 2024-04-26T02:07:36.014 INFO:teuthology.orchestra.run.smithi029.stdout: "segexd": 13, 2024-04-26T02:07:36.014 INFO:teuthology.orchestra.run.smithi029.stdout: "expos": 60987628, 2024-04-26T02:07:36.014 INFO:teuthology.orchestra.run.smithi029.stdout: "wrpos": 570634192, 2024-04-26T02:07:36.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rdpos": 0, 2024-04-26T02:07:36.015 INFO:teuthology.orchestra.run.smithi029.stdout: "jlat": { 2024-04-26T02:07:36.015 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 29880, 2024-04-26T02:07:36.015 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 197.741619746, 2024-04-26T02:07:36.015 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.006617858 2024-04-26T02:07:36.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.015 INFO:teuthology.orchestra.run.smithi029.stdout: "replayed": 0 2024-04-26T02:07:36.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.015 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_mem": { 2024-04-26T02:07:36.015 INFO:teuthology.orchestra.run.smithi029.stdout: "ino": 1983, 2024-04-26T02:07:36.016 INFO:teuthology.orchestra.run.smithi029.stdout: "ino+": 23087, 2024-04-26T02:07:36.016 INFO:teuthology.orchestra.run.smithi029.stdout: "ino-": 21104, 2024-04-26T02:07:36.016 INFO:teuthology.orchestra.run.smithi029.stdout: "dir": 264, 2024-04-26T02:07:36.016 INFO:teuthology.orchestra.run.smithi029.stdout: "dir+": 992, 2024-04-26T02:07:36.016 INFO:teuthology.orchestra.run.smithi029.stdout: "dir-": 728, 2024-04-26T02:07:36.016 INFO:teuthology.orchestra.run.smithi029.stdout: "dn": 3415, 2024-04-26T02:07:36.016 INFO:teuthology.orchestra.run.smithi029.stdout: "dn+": 44423, 2024-04-26T02:07:36.016 INFO:teuthology.orchestra.run.smithi029.stdout: "dn-": 41008, 2024-04-26T02:07:36.016 INFO:teuthology.orchestra.run.smithi029.stdout: "cap": 187, 2024-04-26T02:07:36.017 INFO:teuthology.orchestra.run.smithi029.stdout: "cap+": 25148, 2024-04-26T02:07:36.017 INFO:teuthology.orchestra.run.smithi029.stdout: "cap-": 24961, 2024-04-26T02:07:36.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rss": 88896, 2024-04-26T02:07:36.017 INFO:teuthology.orchestra.run.smithi029.stdout: "heap": 223608 2024-04-26T02:07:36.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.017 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_server": { 2024-04-26T02:07:36.017 INFO:teuthology.orchestra.run.smithi029.stdout: "dispatch_client_request": 265442, 2024-04-26T02:07:36.017 INFO:teuthology.orchestra.run.smithi029.stdout: "dispatch_server_request": 253, 2024-04-26T02:07:36.017 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_request": 193832, 2024-04-26T02:07:36.017 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_session": 2588, 2024-04-26T02:07:36.018 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_peer_request": 8457, 2024-04-26T02:07:36.018 INFO:teuthology.orchestra.run.smithi029.stdout: "req_create_latency": { 2024-04-26T02:07:36.018 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 20898, 2024-04-26T02:07:36.018 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 278.438361844, 2024-04-26T02:07:36.018 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.013323684 2024-04-26T02:07:36.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.021 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getattr_latency": { 2024-04-26T02:07:36.021 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 5108, 2024-04-26T02:07:36.022 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 50.399565151, 2024-04-26T02:07:36.022 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.009866790 2024-04-26T02:07:36.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.022 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getfilelock_latency": { 2024-04-26T02:07:36.022 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.022 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.022 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.023 INFO:teuthology.orchestra.run.smithi029.stdout: "req_link_latency": { 2024-04-26T02:07:36.023 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.023 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.023 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.023 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.023 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookup_latency": { 2024-04-26T02:07:36.023 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 44717, 2024-04-26T02:07:36.023 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 148.950384309, 2024-04-26T02:07:36.023 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.003330956 2024-04-26T02:07:36.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.024 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookuphash_latency": { 2024-04-26T02:07:36.024 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.024 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.024 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.024 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupino_latency": { 2024-04-26T02:07:36.024 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.024 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.025 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.025 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupname_latency": { 2024-04-26T02:07:36.025 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.025 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.025 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.025 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupparent_latency": { 2024-04-26T02:07:36.025 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.025 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.026 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.026 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupsnap_latency": { 2024-04-26T02:07:36.027 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.027 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.027 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.027 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lssnap_latency": { 2024-04-26T02:07:36.027 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.027 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.027 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.028 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mkdir_latency": { 2024-04-26T02:07:36.028 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.028 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.028 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.028 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mknod_latency": { 2024-04-26T02:07:36.028 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.028 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.029 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.029 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mksnap_latency": { 2024-04-26T02:07:36.029 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.029 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.029 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.029 INFO:teuthology.orchestra.run.smithi029.stdout: "req_open_latency": { 2024-04-26T02:07:36.030 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 2, 2024-04-26T02:07:36.030 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.001112713, 2024-04-26T02:07:36.030 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000556356 2024-04-26T02:07:36.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.030 INFO:teuthology.orchestra.run.smithi029.stdout: "req_readdir_latency": { 2024-04-26T02:07:36.030 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 70968, 2024-04-26T02:07:36.030 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 230.798445852, 2024-04-26T02:07:36.030 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.003252148 2024-04-26T02:07:36.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.031 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rename_latency": { 2024-04-26T02:07:36.031 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 4556, 2024-04-26T02:07:36.031 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 151.352894963, 2024-04-26T02:07:36.031 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.033220565 2024-04-26T02:07:36.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.031 INFO:teuthology.orchestra.run.smithi029.stdout: "req_renamesnap_latency": { 2024-04-26T02:07:36.031 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.031 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.031 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.032 INFO:teuthology.orchestra.run.smithi029.stdout: "req_snapdiff_latency": { 2024-04-26T02:07:36.032 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.032 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.032 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.032 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmdir_latency": { 2024-04-26T02:07:36.032 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.032 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.032 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.033 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmsnap_latency": { 2024-04-26T02:07:36.033 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.033 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.033 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.033 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmxattr_latency": { 2024-04-26T02:07:36.033 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.033 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.034 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.034 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setattr_latency": { 2024-04-26T02:07:36.034 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 41, 2024-04-26T02:07:36.034 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.363038006, 2024-04-26T02:07:36.034 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.008854585 2024-04-26T02:07:36.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.034 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setdirlayout_latency": { 2024-04-26T02:07:36.034 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.035 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.035 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.035 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setfilelock_latency": { 2024-04-26T02:07:36.035 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 572, 2024-04-26T02:07:36.035 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 1.034395498, 2024-04-26T02:07:36.035 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.001808383 2024-04-26T02:07:36.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.036 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setlayout_latency": { 2024-04-26T02:07:36.036 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.036 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.036 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.036 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setxattr_latency": { 2024-04-26T02:07:36.036 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.037 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.037 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.037 INFO:teuthology.orchestra.run.smithi029.stdout: "req_symlink_latency": { 2024-04-26T02:07:36.037 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.037 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.037 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.037 INFO:teuthology.orchestra.run.smithi029.stdout: "req_unlink_latency": { 2024-04-26T02:07:36.037 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 20947, 2024-04-26T02:07:36.038 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 223.569993317, 2024-04-26T02:07:36.038 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.010673127 2024-04-26T02:07:36.038 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.038 INFO:teuthology.orchestra.run.smithi029.stdout: "cap_revoke_eviction": 0, 2024-04-26T02:07:36.039 INFO:teuthology.orchestra.run.smithi029.stdout: "cap_acquisition_throttle": 0, 2024-04-26T02:07:36.039 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getvxattr_latency": { 2024-04-26T02:07:36.039 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.039 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.040 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.040 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:36.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.040 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_sessions": { 2024-04-26T02:07:36.041 INFO:teuthology.orchestra.run.smithi029.stdout: "session_count": 1, 2024-04-26T02:07:36.041 INFO:teuthology.orchestra.run.smithi029.stdout: "session_add": 3, 2024-04-26T02:07:36.041 INFO:teuthology.orchestra.run.smithi029.stdout: "session_remove": 2, 2024-04-26T02:07:36.041 INFO:teuthology.orchestra.run.smithi029.stdout: "sessions_open": 1, 2024-04-26T02:07:36.042 INFO:teuthology.orchestra.run.smithi029.stdout: "sessions_stale": 0, 2024-04-26T02:07:36.042 INFO:teuthology.orchestra.run.smithi029.stdout: "total_load": 33561, 2024-04-26T02:07:36.042 INFO:teuthology.orchestra.run.smithi029.stdout: "average_load": 33561, 2024-04-26T02:07:36.042 INFO:teuthology.orchestra.run.smithi029.stdout: "avg_session_uptime": 1300, 2024-04-26T02:07:36.042 INFO:teuthology.orchestra.run.smithi029.stdout: "mdthresh_evicted": 0 2024-04-26T02:07:36.042 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.042 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-04-26T02:07:36.042 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:36.043 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:36.043 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:36.043 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:36.043 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:36.043 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:36.043 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:36.043 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:36.043 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:36.045 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:36.045 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:36.045 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:36.045 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:36.045 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:36.045 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:36.045 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:36.045 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:36.046 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:36.046 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:36.046 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:36.046 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:36.046 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:36.046 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-04-26T02:07:36.046 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:36.046 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:36.047 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:36.047 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:36.047 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 745380, 2024-04-26T02:07:36.047 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 893, 2024-04-26T02:07:36.047 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:36.047 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:36.047 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-04-26T02:07:36.047 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-04-26T02:07:36.047 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:36.047 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:36.048 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:36.048 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:36.048 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 8320, 2024-04-26T02:07:36.048 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 173, 2024-04-26T02:07:36.048 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:36.048 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:36.048 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:36.048 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-04-26T02:07:36.049 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 7762470, 2024-04-26T02:07:36.049 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 68631, 2024-04-26T02:07:36.049 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:36.049 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-04-26T02:07:36.049 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:36.049 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-04-26T02:07:36.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.050 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter": { 2024-04-26T02:07:36.050 INFO:teuthology.orchestra.run.smithi029.stdout: "op_active": 0, 2024-04-26T02:07:36.050 INFO:teuthology.orchestra.run.smithi029.stdout: "op_laggy": 0, 2024-04-26T02:07:36.050 INFO:teuthology.orchestra.run.smithi029.stdout: "op_send": 54004, 2024-04-26T02:07:36.050 INFO:teuthology.orchestra.run.smithi029.stdout: "op_send_bytes": 572773377, 2024-04-26T02:07:36.050 INFO:teuthology.orchestra.run.smithi029.stdout: "op_resend": 33, 2024-04-26T02:07:36.050 INFO:teuthology.orchestra.run.smithi029.stdout: "op_reply": 54004, 2024-04-26T02:07:36.050 INFO:teuthology.orchestra.run.smithi029.stdout: "op_latency": { 2024-04-26T02:07:36.051 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 54004, 2024-04-26T02:07:36.051 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 369.806836393, 2024-04-26T02:07:36.051 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.006847767 2024-04-26T02:07:36.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.051 INFO:teuthology.orchestra.run.smithi029.stdout: "op_inflight": 0, 2024-04-26T02:07:36.051 INFO:teuthology.orchestra.run.smithi029.stdout: "oplen_avg": { 2024-04-26T02:07:36.051 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 54004, 2024-04-26T02:07:36.052 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 55420 2024-04-26T02:07:36.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.052 INFO:teuthology.orchestra.run.smithi029.stdout: "op": 54004, 2024-04-26T02:07:36.052 INFO:teuthology.orchestra.run.smithi029.stdout: "op_r": 793, 2024-04-26T02:07:36.052 INFO:teuthology.orchestra.run.smithi029.stdout: "op_w": 53211, 2024-04-26T02:07:36.052 INFO:teuthology.orchestra.run.smithi029.stdout: "op_rmw": 0, 2024-04-26T02:07:36.052 INFO:teuthology.orchestra.run.smithi029.stdout: "op_pg": 0, 2024-04-26T02:07:36.052 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_stat": 31, 2024-04-26T02:07:36.052 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_create": 33, 2024-04-26T02:07:36.053 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_read": 782, 2024-04-26T02:07:36.053 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_write": 30662, 2024-04-26T02:07:36.053 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_writefull": 466, 2024-04-26T02:07:36.053 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_writesame": 0, 2024-04-26T02:07:36.053 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_append": 0, 2024-04-26T02:07:36.053 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_zero": 0, 2024-04-26T02:07:36.053 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_truncate": 14, 2024-04-26T02:07:36.053 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_delete": 21151, 2024-04-26T02:07:36.053 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_mapext": 0, 2024-04-26T02:07:36.053 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_sparse_read": 0, 2024-04-26T02:07:36.054 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_clonerange": 0, 2024-04-26T02:07:36.054 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_getxattr": 0, 2024-04-26T02:07:36.054 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_setxattr": 66, 2024-04-26T02:07:36.054 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_cmpxattr": 0, 2024-04-26T02:07:36.054 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_rmxattr": 0, 2024-04-26T02:07:36.054 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_resetxattrs": 0, 2024-04-26T02:07:36.054 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_call": 0, 2024-04-26T02:07:36.055 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_watch": 0, 2024-04-26T02:07:36.055 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_notify": 0, 2024-04-26T02:07:36.055 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T02:07:36.055 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_pgls": 0, 2024-04-26T02:07:36.055 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_pgls_filter": 0, 2024-04-26T02:07:36.055 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_other": 0, 2024-04-26T02:07:36.055 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_active": 0, 2024-04-26T02:07:36.055 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_send": 0, 2024-04-26T02:07:36.055 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_resend": 0, 2024-04-26T02:07:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_ping": 0, 2024-04-26T02:07:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_active": 0, 2024-04-26T02:07:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_send": 0, 2024-04-26T02:07:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_resend": 0, 2024-04-26T02:07:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_active": 0, 2024-04-26T02:07:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_send": 0, 2024-04-26T02:07:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_resend": 0, 2024-04-26T02:07:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_active": 0, 2024-04-26T02:07:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_send": 0, 2024-04-26T02:07:36.057 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_resend": 0, 2024-04-26T02:07:36.057 INFO:teuthology.orchestra.run.smithi029.stdout: "command_active": 0, 2024-04-26T02:07:36.057 INFO:teuthology.orchestra.run.smithi029.stdout: "command_send": 0, 2024-04-26T02:07:36.057 INFO:teuthology.orchestra.run.smithi029.stdout: "command_resend": 0, 2024-04-26T02:07:36.057 INFO:teuthology.orchestra.run.smithi029.stdout: "map_epoch": 132, 2024-04-26T02:07:36.057 INFO:teuthology.orchestra.run.smithi029.stdout: "map_full": 0, 2024-04-26T02:07:36.057 INFO:teuthology.orchestra.run.smithi029.stdout: "map_inc": 41, 2024-04-26T02:07:36.057 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_sessions": 12, 2024-04-26T02:07:36.057 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_session_open": 12, 2024-04-26T02:07:36.058 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_session_close": 0, 2024-04-26T02:07:36.058 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_laggy": 0, 2024-04-26T02:07:36.058 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_wr": 1659, 2024-04-26T02:07:36.058 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_rd": 22, 2024-04-26T02:07:36.058 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_del": 534 2024-04-26T02:07:36.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.058 INFO:teuthology.orchestra.run.smithi029.stdout: "oft": { 2024-04-26T02:07:36.058 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_objs": 1, 2024-04-26T02:07:36.058 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_kv_pairs": 0, 2024-04-26T02:07:36.059 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_updates": 0, 2024-04-26T02:07:36.059 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_removes": 0 2024-04-26T02:07:36.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.059 INFO:teuthology.orchestra.run.smithi029.stdout: "osd-slow-ops": { 2024-04-26T02:07:36.059 INFO:teuthology.orchestra.run.smithi029.stdout: "slow_ops_count": 2051 2024-04-26T02:07:36.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.059 INFO:teuthology.orchestra.run.smithi029.stdout: "purge_queue": { 2024-04-26T02:07:36.059 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_ops": 0, 2024-04-26T02:07:36.059 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_ops_high_water": 10, 2024-04-26T02:07:36.060 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing": 0, 2024-04-26T02:07:36.060 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_high_water": 10, 2024-04-26T02:07:36.060 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executed_ops": 20947, 2024-04-26T02:07:36.060 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executed": 20947, 2024-04-26T02:07:36.060 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_item_in_journal": 0 2024-04-26T02:07:36.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.060 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:36.060 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-04-26T02:07:36.060 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-04-26T02:07:36.061 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-04-26T02:07:36.061 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 652940, 2024-04-26T02:07:36.061 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 154802090, 2024-04-26T02:07:36.061 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:36.061 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 652940, 2024-04-26T02:07:36.061 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-04-26T02:07:36.061 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-04-26T02:07:36.061 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 652940, 2024-04-26T02:07:36.061 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 154802090, 2024-04-26T02:07:36.062 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-04-26T02:07:36.062 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.062 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.062 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.062 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:36.062 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.062 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-objecter_bytes": { 2024-04-26T02:07:36.062 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-04-26T02:07:36.063 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-04-26T02:07:36.063 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-04-26T02:07:36.063 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-04-26T02:07:36.063 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-04-26T02:07:36.063 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:36.063 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:36.063 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 54004, 2024-04-26T02:07:36.063 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 574892160, 2024-04-26T02:07:36.063 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 54004, 2024-04-26T02:07:36.064 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 574892160, 2024-04-26T02:07:36.064 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-04-26T02:07:36.064 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.064 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.064 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.064 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:36.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.064 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-objecter_ops": { 2024-04-26T02:07:36.064 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-04-26T02:07:36.065 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 1024, 2024-04-26T02:07:36.065 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-04-26T02:07:36.065 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-04-26T02:07:36.065 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-04-26T02:07:36.065 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:36.065 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:36.065 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 54004, 2024-04-26T02:07:36.065 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 54004, 2024-04-26T02:07:36.065 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 54004, 2024-04-26T02:07:36.065 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 54004, 2024-04-26T02:07:36.066 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-04-26T02:07:36.066 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.066 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.066 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.066 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:36.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.066 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-write_buf_throttle": { 2024-04-26T02:07:36.066 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-04-26T02:07:36.067 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 3758096384, 2024-04-26T02:07:36.067 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-04-26T02:07:36.067 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 20947, 2024-04-26T02:07:36.067 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 2118959, 2024-04-26T02:07:36.067 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:36.067 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 20947, 2024-04-26T02:07:36.067 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-04-26T02:07:36.067 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-04-26T02:07:36.067 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 782, 2024-04-26T02:07:36.068 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 2118959, 2024-04-26T02:07:36.068 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-04-26T02:07:36.068 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.068 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.068 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.068 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:36.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:36.068 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-write_buf_throttle-0x5618d792e460": { 2024-04-26T02:07:36.068 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 37071, 2024-04-26T02:07:36.068 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 3758096384, 2024-04-26T02:07:36.069 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-04-26T02:07:36.069 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 128136, 2024-04-26T02:07:36.069 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 566439888, 2024-04-26T02:07:36.069 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:36.069 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 128136, 2024-04-26T02:07:36.069 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-04-26T02:07:36.069 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-04-26T02:07:36.069 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 29880, 2024-04-26T02:07:36.070 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 566402817, 2024-04-26T02:07:36.070 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-04-26T02:07:36.070 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:36.070 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.070 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.070 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:36.070 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:36.070 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-26T02:07:36.850 INFO:tasks.check_counter:Daemon mds.b mds.exported=5 2024-04-26T02:07:36.851 INFO:tasks.check_counter:Daemon mds.b mds.imported=5 2024-04-26T02:07:36.851 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=559 2024-04-26T02:07:36.851 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=1427 2024-04-26T02:07:36.851 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=7 2024-04-26T02:07:36.851 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=0 2024-04-26T02:07:36.851 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T02:07:36.851 DEBUG:tasks.check_counter:Getting stats from e 2024-04-26T02:07:36.852 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-26T02:07:37.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:36 smithi006 ceph-mon[28419]: pgmap v689: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 4.4 MiB/s wr, 130 op/s 2024-04-26T02:07:37.111 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T02:07:37.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:36 smithi188 ceph-mon[32751]: pgmap v689: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 4.4 MiB/s wr, 130 op/s 2024-04-26T02:07:37.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:36 smithi029 ceph-mon[32775]: pgmap v689: 97 pgs: 97 active+clean; 1.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 4.4 MiB/s wr, 130 op/s 2024-04-26T02:07:38.898 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:38 smithi029 ceph-mon[32775]: pgmap v690: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 5.8 MiB/s wr, 163 op/s 2024-04-26T02:07:38.913 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-26T02:07:38.913 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:38.913 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 4, 2024-04-26T02:07:38.913 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 4, 2024-04-26T02:07:38.914 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 2992, 2024-04-26T02:07:38.914 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 448, 2024-04-26T02:07:38.914 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:38.914 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 0, 2024-04-26T02:07:38.914 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.001701966, 2024-04-26T02:07:38.914 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.000040501, 2024-04-26T02:07:38.914 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.000633200, 2024-04-26T02:07:38.914 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:38.915 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:38.915 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 3, 2024-04-26T02:07:38.915 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000752927, 2024-04-26T02:07:38.915 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000250975 2024-04-26T02:07:38.915 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:38.915 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:38.915 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:38.915 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:38.916 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:38.916 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:38.916 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T02:07:38.916 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T02:07:38.916 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:38.916 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:38.916 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 429, 2024-04-26T02:07:38.916 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 417, 2024-04-26T02:07:38.916 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 1787408, 2024-04-26T02:07:38.917 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 258336, 2024-04-26T02:07:38.917 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 2, 2024-04-26T02:07:38.917 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 1, 2024-04-26T02:07:38.917 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.070194101, 2024-04-26T02:07:38.917 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.023543907, 2024-04-26T02:07:38.917 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.044207485, 2024-04-26T02:07:38.917 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.009755142, 2024-04-26T02:07:38.917 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:38.917 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 416, 2024-04-26T02:07:38.918 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.018918151, 2024-04-26T02:07:38.918 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000045476 2024-04-26T02:07:38.918 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:38.918 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:38.918 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:38.918 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:38.918 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:38.918 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:38.918 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 1787408, 2024-04-26T02:07:38.919 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 258336 2024-04-26T02:07:38.919 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:38.919 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:38.919 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 4, 2024-04-26T02:07:38.919 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 322, 2024-04-26T02:07:38.919 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 640, 2024-04-26T02:07:38.919 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 369360, 2024-04-26T02:07:38.919 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 7, 2024-04-26T02:07:38.919 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:38.920 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.019767985, 2024-04-26T02:07:38.920 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.013623684, 2024-04-26T02:07:38.920 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.002355176, 2024-04-26T02:07:38.920 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:38.920 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:38.920 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 319, 2024-04-26T02:07:38.920 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.025965601, 2024-04-26T02:07:38.920 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000081396 2024-04-26T02:07:38.920 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:38.920 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:38.921 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:38.921 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:38.921 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:38.921 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:38.921 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 640, 2024-04-26T02:07:38.921 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 369360 2024-04-26T02:07:38.921 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:38.921 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-04-26T02:07:38.922 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 0, 2024-04-26T02:07:38.922 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-04-26T02:07:38.922 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:38.922 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-04-26T02:07:38.922 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:38.922 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:38.922 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:38.922 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:38.922 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:38.923 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:38.923 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:38.923 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:38.923 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:38.923 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:38.923 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:38.923 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:38.923 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:38.923 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:38.924 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:38.924 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:38.924 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:38.924 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:38.924 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:38.924 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:38.924 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:38.924 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:38.924 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:38.924 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:38.925 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:38.925 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:38.925 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:38.925 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:38.925 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:38.925 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:38.925 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:38.925 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-04-26T02:07:38.925 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:38.925 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:38.926 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:38.926 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:38.926 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 148692, 2024-04-26T02:07:38.926 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 30, 2024-04-26T02:07:38.926 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:38.926 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:38.926 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-04-26T02:07:38.926 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-04-26T02:07:38.926 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:38.927 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:38.927 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:38.927 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:38.927 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 0, 2024-04-26T02:07:38.927 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 0, 2024-04-26T02:07:38.927 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:38.927 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:38.927 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:38.927 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-04-26T02:07:38.927 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 928, 2024-04-26T02:07:38.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 2, 2024-04-26T02:07:38.928 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:38.928 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-04-26T02:07:38.928 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:38.928 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-04-26T02:07:38.928 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:38.928 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:38.928 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-04-26T02:07:38.928 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-04-26T02:07:38.928 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-04-26T02:07:38.929 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 433, 2024-04-26T02:07:38.929 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 1726836, 2024-04-26T02:07:38.929 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:38.929 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 433, 2024-04-26T02:07:38.929 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-04-26T02:07:38.929 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-04-26T02:07:38.929 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 433, 2024-04-26T02:07:38.929 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 1726836, 2024-04-26T02:07:38.929 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-04-26T02:07:38.930 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:38.930 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:38.930 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:38.930 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:38.930 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:38.930 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-26T02:07:39.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:38 smithi006 ceph-mon[28419]: pgmap v690: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 5.8 MiB/s wr, 163 op/s 2024-04-26T02:07:39.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:38 smithi188 ceph-mon[32751]: pgmap v690: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 5.8 MiB/s wr, 163 op/s 2024-04-26T02:07:39.857 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-04-26T02:07:39.857 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-04-26T02:07:39.858 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.e 2024-04-26T02:07:39.858 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.e 2024-04-26T02:07:39.858 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-04-26T02:07:39.858 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-04-26T02:07:39.858 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-04-26T02:07:39.858 DEBUG:tasks.check_counter:Getting stats from h 2024-04-26T02:07:39.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-26T02:07:40.112 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T02:07:40.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:40 smithi029 ceph-mon[32775]: pgmap v691: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 3.4 MiB/s wr, 96 op/s 2024-04-26T02:07:41.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:40 smithi006 ceph-mon[28419]: pgmap v691: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 3.4 MiB/s wr, 96 op/s 2024-04-26T02:07:41.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:40 smithi188 ceph-mon[32751]: pgmap v691: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 3.4 MiB/s wr, 96 op/s 2024-04-26T02:07:41.975 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-26T02:07:41.976 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:41.976 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 1459, 2024-04-26T02:07:41.976 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 1459, 2024-04-26T02:07:41.976 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 197524215, 2024-04-26T02:07:41.976 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 463202, 2024-04-26T02:07:41.976 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 9, 2024-04-26T02:07:41.976 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 4, 2024-04-26T02:07:41.976 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.211038103, 2024-04-26T02:07:41.977 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.036965725, 2024-04-26T02:07:41.977 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.125982888, 2024-04-26T02:07:41.977 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.035888301, 2024-04-26T02:07:41.977 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:41.977 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 1458, 2024-04-26T02:07:41.977 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.047931948, 2024-04-26T02:07:41.977 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000032875 2024-04-26T02:07:41.977 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.978 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:41.978 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:41.978 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:41.978 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:41.978 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.978 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T02:07:41.978 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T02:07:41.979 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.979 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:41.979 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 2051, 2024-04-26T02:07:41.979 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 2004, 2024-04-26T02:07:41.979 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 186927891, 2024-04-26T02:07:41.979 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 756105, 2024-04-26T02:07:41.979 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:41.979 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 5, 2024-04-26T02:07:41.979 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.314543992, 2024-04-26T02:07:41.980 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.058107632, 2024-04-26T02:07:41.980 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.202043819, 2024-04-26T02:07:41.980 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.052061778, 2024-04-26T02:07:41.980 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:41.980 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 2003, 2024-04-26T02:07:41.980 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.069167527, 2024-04-26T02:07:41.980 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000034531 2024-04-26T02:07:41.981 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.981 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:41.981 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:41.981 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:41.981 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:41.981 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.981 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 1975328, 2024-04-26T02:07:41.981 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 265744 2024-04-26T02:07:41.982 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.982 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:41.982 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 2812, 2024-04-26T02:07:41.982 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 3129, 2024-04-26T02:07:41.982 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 196698015, 2024-04-26T02:07:41.982 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 1690382, 2024-04-26T02:07:41.982 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 9, 2024-04-26T02:07:41.982 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 6, 2024-04-26T02:07:41.983 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.385571086, 2024-04-26T02:07:41.983 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.084290887, 2024-04-26T02:07:41.983 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.185871143, 2024-04-26T02:07:41.983 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.088472539, 2024-04-26T02:07:41.983 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:41.983 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 3125, 2024-04-26T02:07:41.983 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.126170185, 2024-04-26T02:07:41.983 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000040374 2024-04-26T02:07:41.984 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.984 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:41.984 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:41.984 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:41.984 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:41.984 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.984 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T02:07:41.984 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 797120 2024-04-26T02:07:41.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.985 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-04-26T02:07:41.985 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 1, 2024-04-26T02:07:41.985 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-04-26T02:07:41.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.985 INFO:teuthology.orchestra.run.smithi029.stdout: "finisher-MDSRank": { 2024-04-26T02:07:41.985 INFO:teuthology.orchestra.run.smithi029.stdout: "queue_len": 0, 2024-04-26T02:07:41.986 INFO:teuthology.orchestra.run.smithi029.stdout: "complete_latency": { 2024-04-26T02:07:41.986 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 9462, 2024-04-26T02:07:41.986 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.252229915, 2024-04-26T02:07:41.986 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000026657 2024-04-26T02:07:41.986 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:41.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.986 INFO:teuthology.orchestra.run.smithi029.stdout: "finisher-PurgeQueue": { 2024-04-26T02:07:41.986 INFO:teuthology.orchestra.run.smithi029.stdout: "queue_len": 0, 2024-04-26T02:07:41.987 INFO:teuthology.orchestra.run.smithi029.stdout: "complete_latency": { 2024-04-26T02:07:41.987 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:41.987 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:41.987 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:41.987 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:41.987 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.987 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": { 2024-04-26T02:07:41.987 INFO:teuthology.orchestra.run.smithi029.stdout: "request": 0, 2024-04-26T02:07:41.988 INFO:teuthology.orchestra.run.smithi029.stdout: "reply": 0, 2024-04-26T02:07:41.988 INFO:teuthology.orchestra.run.smithi029.stdout: "reply_latency": { 2024-04-26T02:07:41.988 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:41.988 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:41.988 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:41.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.988 INFO:teuthology.orchestra.run.smithi029.stdout: "slow_reply": 0, 2024-04-26T02:07:41.988 INFO:teuthology.orchestra.run.smithi029.stdout: "forward": 0, 2024-04-26T02:07:41.988 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_fetch_complete": 0, 2024-04-26T02:07:41.989 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_fetch_keys": 0, 2024-04-26T02:07:41.989 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_commit": 0, 2024-04-26T02:07:41.989 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_split": 0, 2024-04-26T02:07:41.989 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_merge": 0, 2024-04-26T02:07:41.989 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes": 27435, 2024-04-26T02:07:41.989 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_top": 5304, 2024-04-26T02:07:41.989 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_bottom": 22131, 2024-04-26T02:07:41.989 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_pin_tail": 0, 2024-04-26T02:07:41.990 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_pinned": 19858, 2024-04-26T02:07:41.990 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_expired": 0, 2024-04-26T02:07:41.990 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_with_caps": 0, 2024-04-26T02:07:41.990 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": 0, 2024-04-26T02:07:41.990 INFO:teuthology.orchestra.run.smithi029.stdout: "subtrees": 15, 2024-04-26T02:07:41.990 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse": 0, 2024-04-26T02:07:41.990 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_hit": 0, 2024-04-26T02:07:41.990 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_forward": 0, 2024-04-26T02:07:41.991 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_discover": 0, 2024-04-26T02:07:41.991 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_dir_fetch": 0, 2024-04-26T02:07:41.991 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_remote_ino": 0, 2024-04-26T02:07:41.991 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_lock": 0, 2024-04-26T02:07:41.991 INFO:teuthology.orchestra.run.smithi029.stdout: "load_cent": 0, 2024-04-26T02:07:41.991 INFO:teuthology.orchestra.run.smithi029.stdout: "q": 0, 2024-04-26T02:07:41.991 INFO:teuthology.orchestra.run.smithi029.stdout: "exported": 0, 2024-04-26T02:07:41.991 INFO:teuthology.orchestra.run.smithi029.stdout: "exported_inodes": 0, 2024-04-26T02:07:41.991 INFO:teuthology.orchestra.run.smithi029.stdout: "imported": 0, 2024-04-26T02:07:41.992 INFO:teuthology.orchestra.run.smithi029.stdout: "imported_inodes": 0, 2024-04-26T02:07:41.992 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_dir_fetch": 0, 2024-04-26T02:07:41.992 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_backtrace_fetch": 0, 2024-04-26T02:07:41.992 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_peer_discover": 0, 2024-04-26T02:07:41.992 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rfiles": 0, 2024-04-26T02:07:41.992 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rbytes": 0, 2024-04-26T02:07:41.992 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rsnaps": 0, 2024-04-26T02:07:41.993 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T02:07:41.993 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_set_tag": 0, 2024-04-26T02:07:41.993 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T02:07:41.993 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_inotable_repaired": 0, 2024-04-26T02:07:41.993 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dir_inodes": 0, 2024-04-26T02:07:41.993 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T02:07:41.993 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T02:07:41.993 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_file_inodes": 0, 2024-04-26T02:07:41.993 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_inode_file_caps": 0, 2024-04-26T02:07:41.994 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T02:07:41.994 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_grant": 0, 2024-04-26T02:07:41.994 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T02:07:41.994 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T02:07:41.994 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T02:07:41.994 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_caps": 0, 2024-04-26T02:07:41.994 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_caps_dirty": 0, 2024-04-26T02:07:41.994 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_cap_release": 0, 2024-04-26T02:07:41.995 INFO:teuthology.orchestra.run.smithi029.stdout: "process_request_cap_release": 0 2024-04-26T02:07:41.995 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.995 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache": { 2024-04-26T02:07:41.995 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_update": 0, 2024-04-26T02:07:41.995 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_update_receipt": 0, 2024-04-26T02:07:41.995 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_try_discover": 0, 2024-04-26T02:07:41.995 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_send_discover": 0, 2024-04-26T02:07:41.995 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_handle_discover": 0, 2024-04-26T02:07:41.995 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays": 0, 2024-04-26T02:07:41.996 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays_delayed": 0, 2024-04-26T02:07:41.996 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays_enqueuing": 0, 2024-04-26T02:07:41.996 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_created": 0, 2024-04-26T02:07:41.996 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_enqueued": 0, 2024-04-26T02:07:41.996 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_reintegrated": 0, 2024-04-26T02:07:41.997 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_migrated": 0, 2024-04-26T02:07:41.997 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_processing": 0, 2024-04-26T02:07:41.997 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_enqueued": 0, 2024-04-26T02:07:41.997 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_prioritized": 0, 2024-04-26T02:07:41.997 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_started": 0, 2024-04-26T02:07:41.997 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_completed": 0, 2024-04-26T02:07:41.997 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_quiesce_path": 0, 2024-04-26T02:07:41.998 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_quiesce_inode": 0, 2024-04-26T02:07:41.998 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T02:07:41.998 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_exportdir": 0, 2024-04-26T02:07:41.998 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_flush": 0, 2024-04-26T02:07:41.998 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_fragmentdir": 0, 2024-04-26T02:07:41.998 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_fragstats": 0, 2024-04-26T02:07:41.999 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_inodestats": 0 2024-04-26T02:07:41.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:41.999 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log": { 2024-04-26T02:07:41.999 INFO:teuthology.orchestra.run.smithi029.stdout: "evlrg": 0, 2024-04-26T02:07:41.999 INFO:teuthology.orchestra.run.smithi029.stdout: "evadd": 0, 2024-04-26T02:07:41.999 INFO:teuthology.orchestra.run.smithi029.stdout: "evex": 0, 2024-04-26T02:07:41.999 INFO:teuthology.orchestra.run.smithi029.stdout: "evtrm": 0, 2024-04-26T02:07:41.999 INFO:teuthology.orchestra.run.smithi029.stdout: "ev": 124830, 2024-04-26T02:07:42.000 INFO:teuthology.orchestra.run.smithi029.stdout: "evexg": 0, 2024-04-26T02:07:42.000 INFO:teuthology.orchestra.run.smithi029.stdout: "evexd": 0, 2024-04-26T02:07:42.000 INFO:teuthology.orchestra.run.smithi029.stdout: "segadd": 0, 2024-04-26T02:07:42.000 INFO:teuthology.orchestra.run.smithi029.stdout: "segex": 0, 2024-04-26T02:07:42.000 INFO:teuthology.orchestra.run.smithi029.stdout: "segtrm": 0, 2024-04-26T02:07:42.000 INFO:teuthology.orchestra.run.smithi029.stdout: "seg": 138, 2024-04-26T02:07:42.000 INFO:teuthology.orchestra.run.smithi029.stdout: "segmjr": 12, 2024-04-26T02:07:42.001 INFO:teuthology.orchestra.run.smithi029.stdout: "segexg": 0, 2024-04-26T02:07:42.001 INFO:teuthology.orchestra.run.smithi029.stdout: "segexd": 0, 2024-04-26T02:07:42.001 INFO:teuthology.orchestra.run.smithi029.stdout: "expos": 66598130, 2024-04-26T02:07:42.001 INFO:teuthology.orchestra.run.smithi029.stdout: "wrpos": 581734172, 2024-04-26T02:07:42.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rdpos": 581728433, 2024-04-26T02:07:42.001 INFO:teuthology.orchestra.run.smithi029.stdout: "jlat": { 2024-04-26T02:07:42.001 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.001 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.002 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.002 INFO:teuthology.orchestra.run.smithi029.stdout: "replayed": 124830 2024-04-26T02:07:42.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.002 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_mem": { 2024-04-26T02:07:42.002 INFO:teuthology.orchestra.run.smithi029.stdout: "ino": 1925, 2024-04-26T02:07:42.003 INFO:teuthology.orchestra.run.smithi029.stdout: "ino+": 27410, 2024-04-26T02:07:42.003 INFO:teuthology.orchestra.run.smithi029.stdout: "ino-": 25485, 2024-04-26T02:07:42.003 INFO:teuthology.orchestra.run.smithi029.stdout: "dir": 116, 2024-04-26T02:07:42.003 INFO:teuthology.orchestra.run.smithi029.stdout: "dir+": 191, 2024-04-26T02:07:42.003 INFO:teuthology.orchestra.run.smithi029.stdout: "dir-": 75, 2024-04-26T02:07:42.003 INFO:teuthology.orchestra.run.smithi029.stdout: "dn": 27435, 2024-04-26T02:07:42.003 INFO:teuthology.orchestra.run.smithi029.stdout: "dn+": 27815, 2024-04-26T02:07:42.004 INFO:teuthology.orchestra.run.smithi029.stdout: "dn-": 380, 2024-04-26T02:07:42.004 INFO:teuthology.orchestra.run.smithi029.stdout: "cap": 0, 2024-04-26T02:07:42.004 INFO:teuthology.orchestra.run.smithi029.stdout: "cap+": 0, 2024-04-26T02:07:42.004 INFO:teuthology.orchestra.run.smithi029.stdout: "cap-": 0, 2024-04-26T02:07:42.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rss": 84360, 2024-04-26T02:07:42.004 INFO:teuthology.orchestra.run.smithi029.stdout: "heap": 215416 2024-04-26T02:07:42.004 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.005 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_server": { 2024-04-26T02:07:42.005 INFO:teuthology.orchestra.run.smithi029.stdout: "dispatch_client_request": 0, 2024-04-26T02:07:42.005 INFO:teuthology.orchestra.run.smithi029.stdout: "dispatch_server_request": 0, 2024-04-26T02:07:42.005 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_request": 0, 2024-04-26T02:07:42.005 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_session": 0, 2024-04-26T02:07:42.005 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_peer_request": 0, 2024-04-26T02:07:42.005 INFO:teuthology.orchestra.run.smithi029.stdout: "req_create_latency": { 2024-04-26T02:07:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getattr_latency": { 2024-04-26T02:07:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.006 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.007 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getfilelock_latency": { 2024-04-26T02:07:42.007 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.007 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.007 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.007 INFO:teuthology.orchestra.run.smithi029.stdout: "req_link_latency": { 2024-04-26T02:07:42.008 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.008 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.008 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.008 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookup_latency": { 2024-04-26T02:07:42.008 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.008 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.008 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.009 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookuphash_latency": { 2024-04-26T02:07:42.009 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.009 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.009 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.009 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupino_latency": { 2024-04-26T02:07:42.009 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.010 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.010 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.010 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupname_latency": { 2024-04-26T02:07:42.010 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.011 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.011 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.011 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupparent_latency": { 2024-04-26T02:07:42.011 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.011 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.011 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.011 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupsnap_latency": { 2024-04-26T02:07:42.012 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.012 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.012 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.012 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lssnap_latency": { 2024-04-26T02:07:42.012 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.012 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.012 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.013 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.013 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mkdir_latency": { 2024-04-26T02:07:42.013 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.013 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.014 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.014 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mknod_latency": { 2024-04-26T02:07:42.014 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.014 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.014 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.015 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mksnap_latency": { 2024-04-26T02:07:42.015 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.015 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.015 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.015 INFO:teuthology.orchestra.run.smithi029.stdout: "req_open_latency": { 2024-04-26T02:07:42.016 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.016 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.016 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.016 INFO:teuthology.orchestra.run.smithi029.stdout: "req_readdir_latency": { 2024-04-26T02:07:42.016 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.016 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.016 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.017 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rename_latency": { 2024-04-26T02:07:42.017 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.017 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.017 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.017 INFO:teuthology.orchestra.run.smithi029.stdout: "req_renamesnap_latency": { 2024-04-26T02:07:42.017 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.018 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.018 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.018 INFO:teuthology.orchestra.run.smithi029.stdout: "req_snapdiff_latency": { 2024-04-26T02:07:42.018 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.018 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.018 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.019 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmdir_latency": { 2024-04-26T02:07:42.019 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.019 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.019 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.019 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmsnap_latency": { 2024-04-26T02:07:42.019 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.019 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.020 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.020 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmxattr_latency": { 2024-04-26T02:07:42.020 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.020 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.020 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.020 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setattr_latency": { 2024-04-26T02:07:42.020 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.021 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.021 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.021 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setdirlayout_latency": { 2024-04-26T02:07:42.021 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.021 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.021 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.022 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setfilelock_latency": { 2024-04-26T02:07:42.022 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.022 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.022 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.022 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setlayout_latency": { 2024-04-26T02:07:42.022 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.022 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.023 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.023 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.023 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setxattr_latency": { 2024-04-26T02:07:42.023 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.023 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.023 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.023 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.023 INFO:teuthology.orchestra.run.smithi029.stdout: "req_symlink_latency": { 2024-04-26T02:07:42.024 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.024 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.024 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.024 INFO:teuthology.orchestra.run.smithi029.stdout: "req_unlink_latency": { 2024-04-26T02:07:42.024 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.024 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.024 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.025 INFO:teuthology.orchestra.run.smithi029.stdout: "cap_revoke_eviction": 0, 2024-04-26T02:07:42.025 INFO:teuthology.orchestra.run.smithi029.stdout: "cap_acquisition_throttle": 0, 2024-04-26T02:07:42.025 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getvxattr_latency": { 2024-04-26T02:07:42.025 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.025 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.025 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.025 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:42.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.026 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_sessions": { 2024-04-26T02:07:42.026 INFO:teuthology.orchestra.run.smithi029.stdout: "session_count": 2, 2024-04-26T02:07:42.026 INFO:teuthology.orchestra.run.smithi029.stdout: "session_add": 4, 2024-04-26T02:07:42.026 INFO:teuthology.orchestra.run.smithi029.stdout: "session_remove": 2, 2024-04-26T02:07:42.026 INFO:teuthology.orchestra.run.smithi029.stdout: "sessions_open": 3, 2024-04-26T02:07:42.026 INFO:teuthology.orchestra.run.smithi029.stdout: "sessions_stale": 0, 2024-04-26T02:07:42.026 INFO:teuthology.orchestra.run.smithi029.stdout: "total_load": 0, 2024-04-26T02:07:42.026 INFO:teuthology.orchestra.run.smithi029.stdout: "average_load": 0, 2024-04-26T02:07:42.027 INFO:teuthology.orchestra.run.smithi029.stdout: "avg_session_uptime": 3729, 2024-04-26T02:07:42.027 INFO:teuthology.orchestra.run.smithi029.stdout: "mdthresh_evicted": 0 2024-04-26T02:07:42.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.027 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-04-26T02:07:42.027 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:42.027 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:42.027 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:42.027 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:42.027 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:42.028 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:42.028 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:42.028 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:42.028 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:42.028 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:42.028 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:42.028 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:42.028 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-04-26T02:07:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 326985, 2024-04-26T02:07:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 2349, 2024-04-26T02:07:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-04-26T02:07:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-04-26T02:07:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 8320, 2024-04-26T02:07:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 173, 2024-04-26T02:07:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-04-26T02:07:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 24354136, 2024-04-26T02:07:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 160517, 2024-04-26T02:07:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-04-26T02:07:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-04-26T02:07:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter": { 2024-04-26T02:07:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: "op_active": 0, 2024-04-26T02:07:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: "op_laggy": 0, 2024-04-26T02:07:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: "op_send": 5806, 2024-04-26T02:07:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: "op_send_bytes": 21, 2024-04-26T02:07:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: "op_resend": 3, 2024-04-26T02:07:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: "op_reply": 5806, 2024-04-26T02:07:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: "op_latency": { 2024-04-26T02:07:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 5806, 2024-04-26T02:07:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 7.752870359, 2024-04-26T02:07:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.001335320 2024-04-26T02:07:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: "op_inflight": 0, 2024-04-26T02:07:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: "oplen_avg": { 2024-04-26T02:07:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 5806, 2024-04-26T02:07:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 5807 2024-04-26T02:07:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: "op": 5806, 2024-04-26T02:07:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: "op_r": 5806, 2024-04-26T02:07:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: "op_w": 0, 2024-04-26T02:07:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: "op_rmw": 0, 2024-04-26T02:07:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: "op_pg": 0, 2024-04-26T02:07:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_stat": 3462, 2024-04-26T02:07:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_create": 0, 2024-04-26T02:07:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_read": 2342, 2024-04-26T02:07:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_write": 0, 2024-04-26T02:07:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_writefull": 0, 2024-04-26T02:07:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_writesame": 0, 2024-04-26T02:07:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_append": 0, 2024-04-26T02:07:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_zero": 0, 2024-04-26T02:07:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_truncate": 0, 2024-04-26T02:07:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_delete": 0, 2024-04-26T02:07:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_mapext": 0, 2024-04-26T02:07:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_sparse_read": 0, 2024-04-26T02:07:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_clonerange": 0, 2024-04-26T02:07:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_getxattr": 1, 2024-04-26T02:07:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_setxattr": 0, 2024-04-26T02:07:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_cmpxattr": 0, 2024-04-26T02:07:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_rmxattr": 0, 2024-04-26T02:07:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_resetxattrs": 0, 2024-04-26T02:07:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_call": 0, 2024-04-26T02:07:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_watch": 0, 2024-04-26T02:07:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_notify": 0, 2024-04-26T02:07:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T02:07:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_pgls": 0, 2024-04-26T02:07:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_pgls_filter": 0, 2024-04-26T02:07:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_other": 0, 2024-04-26T02:07:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_active": 0, 2024-04-26T02:07:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_send": 0, 2024-04-26T02:07:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_resend": 0, 2024-04-26T02:07:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_ping": 0, 2024-04-26T02:07:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_active": 0, 2024-04-26T02:07:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_send": 0, 2024-04-26T02:07:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_resend": 0, 2024-04-26T02:07:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_active": 0, 2024-04-26T02:07:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_send": 0, 2024-04-26T02:07:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_resend": 0, 2024-04-26T02:07:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_active": 0, 2024-04-26T02:07:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_send": 0, 2024-04-26T02:07:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_resend": 0, 2024-04-26T02:07:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: "command_active": 0, 2024-04-26T02:07:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: "command_send": 0, 2024-04-26T02:07:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: "command_resend": 0, 2024-04-26T02:07:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: "map_epoch": 132, 2024-04-26T02:07:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: "map_full": 0, 2024-04-26T02:07:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: "map_inc": 41, 2024-04-26T02:07:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_sessions": 11, 2024-04-26T02:07:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_session_open": 12, 2024-04-26T02:07:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_session_close": 1, 2024-04-26T02:07:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_laggy": 0, 2024-04-26T02:07:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_wr": 0, 2024-04-26T02:07:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_rd": 2, 2024-04-26T02:07:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_del": 0 2024-04-26T02:07:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "oft": { 2024-04-26T02:07:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_objs": 0, 2024-04-26T02:07:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_kv_pairs": 0, 2024-04-26T02:07:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_updates": 0, 2024-04-26T02:07:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_removes": 0 2024-04-26T02:07:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: "osd-slow-ops": { 2024-04-26T02:07:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: "slow_ops_count": 0 2024-04-26T02:07:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: "purge_queue": { 2024-04-26T02:07:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_ops": 0, 2024-04-26T02:07:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T02:07:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing": 0, 2024-04-26T02:07:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_high_water": 0, 2024-04-26T02:07:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executed_ops": 0, 2024-04-26T02:07:42.046 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executed": 0, 2024-04-26T02:07:42.046 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_item_in_journal": 0 2024-04-26T02:07:42.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.046 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:42.046 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-04-26T02:07:42.046 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-04-26T02:07:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-04-26T02:07:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 6318, 2024-04-26T02:07:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 580552589, 2024-04-26T02:07:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 6318, 2024-04-26T02:07:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-04-26T02:07:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-04-26T02:07:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 6318, 2024-04-26T02:07:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 580552589, 2024-04-26T02:07:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-04-26T02:07:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-objecter_bytes": { 2024-04-26T02:07:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-04-26T02:07:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-04-26T02:07:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-04-26T02:07:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-04-26T02:07:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-04-26T02:07:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:42.050 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 5806, 2024-04-26T02:07:42.050 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 577568077, 2024-04-26T02:07:42.050 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 5806, 2024-04-26T02:07:42.050 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 577568077, 2024-04-26T02:07:42.050 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-04-26T02:07:42.050 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.050 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.051 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.051 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:42.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.051 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-objecter_ops": { 2024-04-26T02:07:42.051 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-04-26T02:07:42.051 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 1024, 2024-04-26T02:07:42.051 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-04-26T02:07:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-04-26T02:07:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-04-26T02:07:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 5806, 2024-04-26T02:07:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 5806, 2024-04-26T02:07:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 5806, 2024-04-26T02:07:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 5806, 2024-04-26T02:07:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-04-26T02:07:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-write_buf_throttle": { 2024-04-26T02:07:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-04-26T02:07:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 3758096384, 2024-04-26T02:07:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-04-26T02:07:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-04-26T02:07:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-04-26T02:07:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-04-26T02:07:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-04-26T02:07:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 0, 2024-04-26T02:07:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 0, 2024-04-26T02:07:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-04-26T02:07:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-write_buf_throttle-0x564c81cca6e0": { 2024-04-26T02:07:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-04-26T02:07:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 3758096384, 2024-04-26T02:07:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-04-26T02:07:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-04-26T02:07:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-04-26T02:07:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-04-26T02:07:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-04-26T02:07:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 0, 2024-04-26T02:07:42.057 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 0, 2024-04-26T02:07:42.057 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-04-26T02:07:42.057 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:42.057 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.057 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.057 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:42.057 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:42.057 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-26T02:07:42.898 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:42 smithi029 ceph-mon[32775]: pgmap v692: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 3.4 MiB/s wr, 101 op/s 2024-04-26T02:07:42.934 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-26T02:07:42.934 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-26T02:07:42.934 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-04-26T02:07:42.934 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-04-26T02:07:42.934 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-04-26T02:07:42.934 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-26T02:07:42.935 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T02:07:42.935 DEBUG:tasks.check_counter:Getting stats from k 2024-04-26T02:07:42.935 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-26T02:07:43.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:42 smithi006 ceph-mon[28419]: pgmap v692: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 3.4 MiB/s wr, 101 op/s 2024-04-26T02:07:43.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:42 smithi188 ceph-mon[32751]: pgmap v692: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 3.4 MiB/s wr, 101 op/s 2024-04-26T02:07:43.192 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T02:07:44.931 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:44 smithi029 ceph-mon[32775]: pgmap v693: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.6 MiB/s wr, 46 op/s 2024-04-26T02:07:44.953 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-26T02:07:44.954 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:44.954 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 4, 2024-04-26T02:07:44.954 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 4, 2024-04-26T02:07:44.954 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 2992, 2024-04-26T02:07:44.954 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 448, 2024-04-26T02:07:44.954 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:44.954 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 0, 2024-04-26T02:07:44.955 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.001650509, 2024-04-26T02:07:44.955 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.000041966, 2024-04-26T02:07:44.955 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.000626937, 2024-04-26T02:07:44.955 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:44.955 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:44.955 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 3, 2024-04-26T02:07:44.956 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000705502, 2024-04-26T02:07:44.956 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000235167 2024-04-26T02:07:44.956 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:44.956 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:44.956 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:44.956 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:44.956 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:44.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:44.957 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T02:07:44.957 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T02:07:44.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:44.957 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:44.957 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 3, 2024-04-26T02:07:44.957 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 323, 2024-04-26T02:07:44.957 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 480, 2024-04-26T02:07:44.958 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 291232, 2024-04-26T02:07:44.958 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:44.958 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:44.958 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.018187442, 2024-04-26T02:07:44.958 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.012753453, 2024-04-26T02:07:44.958 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.001991806, 2024-04-26T02:07:44.958 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:44.958 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:44.959 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 320, 2024-04-26T02:07:44.959 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.040445982, 2024-04-26T02:07:44.959 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000126393 2024-04-26T02:07:44.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:44.959 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:44.959 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:44.959 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:44.959 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:44.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:44.960 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T02:07:44.960 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 291232 2024-04-26T02:07:44.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:44.960 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:44.960 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 434, 2024-04-26T02:07:44.960 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 422, 2024-04-26T02:07:44.961 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 1788768, 2024-04-26T02:07:44.961 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 261536, 2024-04-26T02:07:44.961 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 2, 2024-04-26T02:07:44.961 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 1, 2024-04-26T02:07:44.961 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.073430026, 2024-04-26T02:07:44.961 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.023784352, 2024-04-26T02:07:44.961 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.046522011, 2024-04-26T02:07:44.961 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.010280389, 2024-04-26T02:07:44.962 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:44.962 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 421, 2024-04-26T02:07:44.962 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.018711677, 2024-04-26T02:07:44.962 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000044445 2024-04-26T02:07:44.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:44.962 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:44.962 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:44.963 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:44.963 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:44.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:44.963 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 1788768, 2024-04-26T02:07:44.963 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 261536 2024-04-26T02:07:44.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:44.963 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-04-26T02:07:44.963 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 0, 2024-04-26T02:07:44.963 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-04-26T02:07:44.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:44.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-04-26T02:07:44.964 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:44.964 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:44.964 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:44.964 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:44.964 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:44.964 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:44.965 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:44.965 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:44.965 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:44.965 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:44.965 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:44.965 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:44.965 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:44.965 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:44.965 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:44.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:44.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:44.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:44.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:44.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:44.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:44.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:44.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:44.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:44.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:44.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:44.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:44.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:44.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:44.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:44.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:44.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-04-26T02:07:44.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:44.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:44.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:44.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:44.968 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 148732, 2024-04-26T02:07:44.968 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 31, 2024-04-26T02:07:44.969 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:44.969 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:44.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-04-26T02:07:44.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-04-26T02:07:44.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:44.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:44.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:44.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:44.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 0, 2024-04-26T02:07:44.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 0, 2024-04-26T02:07:44.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:44.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:44.971 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:44.971 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-04-26T02:07:44.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 928, 2024-04-26T02:07:44.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 2, 2024-04-26T02:07:44.971 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:44.971 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-04-26T02:07:44.971 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:44.971 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-04-26T02:07:44.972 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-26T02:07:44.972 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:44.972 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-04-26T02:07:44.972 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-04-26T02:07:44.972 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-04-26T02:07:44.972 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 437, 2024-04-26T02:07:44.972 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 1727459, 2024-04-26T02:07:44.973 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:44.973 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 437, 2024-04-26T02:07:44.973 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-04-26T02:07:44.973 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-04-26T02:07:44.973 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 437, 2024-04-26T02:07:44.973 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 1727459, 2024-04-26T02:07:44.973 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-04-26T02:07:44.973 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-04-26T02:07:44.974 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-04-26T02:07:44.974 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-04-26T02:07:44.974 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:44.974 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-26T02:07:44.974 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-26T02:07:45.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:44 smithi006 ceph-mon[28419]: pgmap v693: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.6 MiB/s wr, 46 op/s 2024-04-26T02:07:45.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:44 smithi188 ceph-mon[32751]: pgmap v693: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.6 MiB/s wr, 46 op/s 2024-04-26T02:07:45.968 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-26T02:07:45.968 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-26T02:07:45.968 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-04-26T02:07:45.968 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-04-26T02:07:45.969 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-26T02:07:45.969 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-26T02:07:45.969 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-26T02:07:45.969 DEBUG:tasks.check_counter:Getting stats from c 2024-04-26T02:07:45.969 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-26T02:07:46.229 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T02:07:46.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:46 smithi188 ceph-mon[32751]: pgmap v694: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.5 MiB/s wr, 38 op/s 2024-04-26T02:07:47.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:46 smithi006 ceph-mon[28419]: pgmap v694: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.5 MiB/s wr, 38 op/s 2024-04-26T02:07:47.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:46 smithi029 ceph-mon[32775]: pgmap v694: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.5 MiB/s wr, 38 op/s 2024-04-26T02:07:48.014 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-26T02:07:48.014 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:48.015 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 524055, 2024-04-26T02:07:48.015 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 640786, 2024-04-26T02:07:48.015 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 168794646, 2024-04-26T02:07:48.015 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 481746032, 2024-04-26T02:07:48.015 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 9, 2024-04-26T02:07:48.015 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 4, 2024-04-26T02:07:48.015 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 28.874763791, 2024-04-26T02:07:48.015 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 13.341159528, 2024-04-26T02:07:48.015 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 19.105802931, 2024-04-26T02:07:48.015 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.569530553, 2024-04-26T02:07:48.016 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:48.016 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 640789, 2024-04-26T02:07:48.016 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 21.308512820, 2024-04-26T02:07:48.016 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000033253 2024-04-26T02:07:48.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.016 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:48.016 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1006456, 2024-04-26T02:07:48.016 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.176832344, 2024-04-26T02:07:48.016 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000175 2024-04-26T02:07:48.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.017 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 1988480, 2024-04-26T02:07:48.017 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 273200 2024-04-26T02:07:48.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.017 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:48.017 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 29532, 2024-04-26T02:07:48.017 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 29663, 2024-04-26T02:07:48.017 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 7410080, 2024-04-26T02:07:48.018 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 330519887, 2024-04-26T02:07:48.018 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 16, 2024-04-26T02:07:48.018 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 7, 2024-04-26T02:07:48.018 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 3.836028369, 2024-04-26T02:07:48.018 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 1.664379056, 2024-04-26T02:07:48.018 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.501278987, 2024-04-26T02:07:48.018 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 1.511441052, 2024-04-26T02:07:48.018 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:48.018 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 29659, 2024-04-26T02:07:48.019 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 1.630884440, 2024-04-26T02:07:48.019 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000054987 2024-04-26T02:07:48.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.019 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:48.019 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 106, 2024-04-26T02:07:48.019 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000025288, 2024-04-26T02:07:48.019 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000238 2024-04-26T02:07:48.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.019 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 9248, 2024-04-26T02:07:48.020 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 188912 2024-04-26T02:07:48.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.020 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:48.020 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 131367, 2024-04-26T02:07:48.020 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 142958, 2024-04-26T02:07:48.020 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 38002546, 2024-04-26T02:07:48.020 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 195572656, 2024-04-26T02:07:48.020 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 14, 2024-04-26T02:07:48.020 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 6, 2024-04-26T02:07:48.021 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 8.388399734, 2024-04-26T02:07:48.021 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 3.009175322, 2024-04-26T02:07:48.021 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 4.389269965, 2024-04-26T02:07:48.021 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 1.537069881, 2024-04-26T02:07:48.021 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:48.021 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 142954, 2024-04-26T02:07:48.021 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 6.527912391, 2024-04-26T02:07:48.021 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000045664 2024-04-26T02:07:48.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.022 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:48.022 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 229653, 2024-04-26T02:07:48.022 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.040404874, 2024-04-26T02:07:48.022 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000175 2024-04-26T02:07:48.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.022 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-04-26T02:07:48.022 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 559952 2024-04-26T02:07:48.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2024-04-26T02:07:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2024-04-26T02:07:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2024-04-26T02:07:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2024-04-26T02:07:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-26T02:07:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-26T02:07:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 59682, 2024-04-26T02:07:48.023 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 93.734820614, 2024-04-26T02:07:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001570571 2024-04-26T02:07:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2024-04-26T02:07:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-26T02:07:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-26T02:07:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 43699, 2024-04-26T02:07:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 7.623142007, 2024-04-26T02:07:48.024 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000174446 2024-04-26T02:07:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2024-04-26T02:07:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 193708, 2024-04-26T02:07:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 167470, 2024-04-26T02:07:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2024-04-26T02:07:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 167470, 2024-04-26T02:07:48.025 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 1403.049701693, 2024-04-26T02:07:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.008377916 2024-04-26T02:07:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2024-04-26T02:07:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 26237, 2024-04-26T02:07:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 0, 2024-04-26T02:07:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2024-04-26T02:07:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 151, 2024-04-26T02:07:48.026 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 12, 2024-04-26T02:07:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 10, 2024-04-26T02:07:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 3135, 2024-04-26T02:07:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 1499, 2024-04-26T02:07:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 1620, 2024-04-26T02:07:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 16, 2024-04-26T02:07:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 994, 2024-04-26T02:07:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 465, 2024-04-26T02:07:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 226, 2024-04-26T02:07:48.027 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 227, 2024-04-26T02:07:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 23, 2024-04-26T02:07:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 325874, 2024-04-26T02:07:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 164025, 2024-04-26T02:07:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 26224, 2024-04-26T02:07:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 146, 2024-04-26T02:07:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 0, 2024-04-26T02:07:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2024-04-26T02:07:48.028 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 3644, 2024-04-26T02:07:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 29009, 2024-04-26T02:07:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 0, 2024-04-26T02:07:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 6, 2024-04-26T02:07:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 365, 2024-04-26T02:07:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 8, 2024-04-26T02:07:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 756, 2024-04-26T02:07:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2024-04-26T02:07:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 0, 2024-04-26T02:07:48.029 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2024-04-26T02:07:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 742, 2024-04-26T02:07:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 213002059, 2024-04-26T02:07:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 7, 2024-04-26T02:07:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T02:07:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2024-04-26T02:07:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T02:07:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2024-04-26T02:07:48.030 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 0, 2024-04-26T02:07:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T02:07:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T02:07:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 0, 2024-04-26T02:07:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 21, 2024-04-26T02:07:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 23333, 2024-04-26T02:07:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 103511, 2024-04-26T02:07:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T02:07:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T02:07:48.031 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T02:07:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 55721, 2024-04-26T02:07:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 16748, 2024-04-26T02:07:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 7541, 2024-04-26T02:07:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 130628 2024-04-26T02:07:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2024-04-26T02:07:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 275, 2024-04-26T02:07:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 1438, 2024-04-26T02:07:48.032 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 780, 2024-04-26T02:07:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 1350, 2024-04-26T02:07:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 682, 2024-04-26T02:07:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 0, 2024-04-26T02:07:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2024-04-26T02:07:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2024-04-26T02:07:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 25207, 2024-04-26T02:07:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 25207, 2024-04-26T02:07:48.033 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2024-04-26T02:07:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2024-04-26T02:07:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2024-04-26T02:07:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2024-04-26T02:07:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2024-04-26T02:07:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2024-04-26T02:07:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2024-04-26T02:07:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_path": 14, 2024-04-26T02:07:48.034 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_inode": 1922, 2024-04-26T02:07:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T02:07:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 8, 2024-04-26T02:07:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2024-04-26T02:07:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 22, 2024-04-26T02:07:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2024-04-26T02:07:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2024-04-26T02:07:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2024-04-26T02:07:48.035 INFO:teuthology.orchestra.run.smithi188.stdout: "evlrg": 0, 2024-04-26T02:07:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 124850, 2024-04-26T02:07:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 43248, 2024-04-26T02:07:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 36871, 2024-04-26T02:07:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 87979, 2024-04-26T02:07:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2024-04-26T02:07:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 6377, 2024-04-26T02:07:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 185, 2024-04-26T02:07:48.036 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 57, 2024-04-26T02:07:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 47, 2024-04-26T02:07:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 138, 2024-04-26T02:07:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "segmjr": 8, 2024-04-26T02:07:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2024-04-26T02:07:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 10, 2024-04-26T02:07:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 66598130, 2024-04-26T02:07:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 581901688, 2024-04-26T02:07:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 0, 2024-04-26T02:07:48.037 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2024-04-26T02:07:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 32776, 2024-04-26T02:07:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 219.099134304, 2024-04-26T02:07:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.006684742 2024-04-26T02:07:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 0 2024-04-26T02:07:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2024-04-26T02:07:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 2030, 2024-04-26T02:07:48.038 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 27605, 2024-04-26T02:07:48.039 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 25575, 2024-04-26T02:07:48.039 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 250, 2024-04-26T02:07:48.039 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 947, 2024-04-26T02:07:48.039 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 697, 2024-04-26T02:07:48.039 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 3138, 2024-04-26T02:07:48.039 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 47851, 2024-04-26T02:07:48.039 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 44713, 2024-04-26T02:07:48.039 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 217, 2024-04-26T02:07:48.040 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 27712, 2024-04-26T02:07:48.040 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 27495, 2024-04-26T02:07:48.040 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 87256, 2024-04-26T02:07:48.040 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 223608 2024-04-26T02:07:48.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.040 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2024-04-26T02:07:48.040 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 294408, 2024-04-26T02:07:48.040 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 2430, 2024-04-26T02:07:48.040 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 193708, 2024-04-26T02:07:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 3067, 2024-04-26T02:07:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 22428, 2024-04-26T02:07:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2024-04-26T02:07:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 25169, 2024-04-26T02:07:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 516.763545691, 2024-04-26T02:07:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.020531747 2024-04-26T02:07:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2024-04-26T02:07:48.041 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 7152, 2024-04-26T02:07:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 101.005290503, 2024-04-26T02:07:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.014122663 2024-04-26T02:07:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2024-04-26T02:07:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.043 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2024-04-26T02:07:48.043 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.043 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.043 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.043 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2024-04-26T02:07:48.043 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 48528, 2024-04-26T02:07:48.043 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 262.652538743, 2024-04-26T02:07:48.044 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.005412391 2024-04-26T02:07:48.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.044 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2024-04-26T02:07:48.044 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.044 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.044 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.044 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2024-04-26T02:07:48.044 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 2, 2024-04-26T02:07:48.045 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000215843, 2024-04-26T02:07:48.045 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000107921 2024-04-26T02:07:48.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.045 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2024-04-26T02:07:48.045 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.045 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.045 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.046 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2024-04-26T02:07:48.046 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.046 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.046 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.046 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2024-04-26T02:07:48.047 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.047 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.047 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.047 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2024-04-26T02:07:48.048 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.048 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.048 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.048 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2024-04-26T02:07:48.048 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.048 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.048 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2024-04-26T02:07:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2024-04-26T02:07:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.049 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.050 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.050 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2024-04-26T02:07:48.050 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.050 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.050 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.050 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2024-04-26T02:07:48.051 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 55461, 2024-04-26T02:07:48.051 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 272.570225093, 2024-04-26T02:07:48.051 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.004914628 2024-04-26T02:07:48.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.051 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2024-04-26T02:07:48.051 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 4795, 2024-04-26T02:07:48.051 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 33.485162605, 2024-04-26T02:07:48.051 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.006983349 2024-04-26T02:07:48.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.051 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2024-04-26T02:07:48.052 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.052 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.052 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.052 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2024-04-26T02:07:48.052 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.052 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.053 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.053 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2024-04-26T02:07:48.053 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.053 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.053 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.053 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2024-04-26T02:07:48.053 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.054 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.054 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.054 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2024-04-26T02:07:48.054 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.054 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.054 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.054 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2024-04-26T02:07:48.055 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 84, 2024-04-26T02:07:48.055 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 33.186339217, 2024-04-26T02:07:48.055 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.395075466 2024-04-26T02:07:48.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.055 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2024-04-26T02:07:48.055 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.055 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.055 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.056 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2024-04-26T02:07:48.056 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1072, 2024-04-26T02:07:48.056 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 1.811310042, 2024-04-26T02:07:48.056 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001689654 2024-04-26T02:07:48.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.056 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2024-04-26T02:07:48.056 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.056 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.057 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.057 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2024-04-26T02:07:48.057 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.057 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.057 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.057 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2024-04-26T02:07:48.058 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.058 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.058 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.058 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2024-04-26T02:07:48.058 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 25207, 2024-04-26T02:07:48.058 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 181.575073956, 2024-04-26T02:07:48.058 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.007203359 2024-04-26T02:07:48.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.059 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2024-04-26T02:07:48.059 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2024-04-26T02:07:48.059 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2024-04-26T02:07:48.059 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.059 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.059 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.059 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:48.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.060 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2024-04-26T02:07:48.060 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 2, 2024-04-26T02:07:48.060 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 4, 2024-04-26T02:07:48.060 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 2, 2024-04-26T02:07:48.060 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 2, 2024-04-26T02:07:48.060 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2024-04-26T02:07:48.060 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 32279, 2024-04-26T02:07:48.060 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 16139, 2024-04-26T02:07:48.061 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 1291, 2024-04-26T02:07:48.061 INFO:teuthology.orchestra.run.smithi188.stdout: "mdthresh_evicted": 0 2024-04-26T02:07:48.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.061 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2024-04-26T02:07:48.061 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:48.061 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:48.061 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:48.061 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:48.062 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:48.062 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:48.062 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:48.062 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:48.062 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:48.062 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:48.062 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:48.063 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:48.063 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:48.063 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:48.063 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:48.063 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:48.063 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:48.063 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:48.063 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:48.063 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:48.064 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:48.064 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:48.064 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:48.064 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:48.064 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:48.064 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:48.064 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:48.064 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:48.064 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:48.065 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:48.065 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:48.065 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2024-04-26T02:07:48.065 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:48.065 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:48.065 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:48.065 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:48.065 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 645746, 2024-04-26T02:07:48.065 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 986, 2024-04-26T02:07:48.066 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 88, 2024-04-26T02:07:48.066 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 1, 2024-04-26T02:07:48.066 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2024-04-26T02:07:48.066 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2024-04-26T02:07:48.066 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:48.066 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:48.066 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:48.066 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:48.067 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 8320, 2024-04-26T02:07:48.067 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 173, 2024-04-26T02:07:48.067 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:48.067 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:48.067 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:48.067 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2024-04-26T02:07:48.067 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 7729909, 2024-04-26T02:07:48.067 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 63299, 2024-04-26T02:07:48.068 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:48.068 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2024-04-26T02:07:48.068 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:48.068 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2024-04-26T02:07:48.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.068 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2024-04-26T02:07:48.068 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2024-04-26T02:07:48.068 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2024-04-26T02:07:48.069 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 61477, 2024-04-26T02:07:48.069 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 584439179, 2024-04-26T02:07:48.069 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 33, 2024-04-26T02:07:48.069 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 61477, 2024-04-26T02:07:48.069 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2024-04-26T02:07:48.069 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 61477, 2024-04-26T02:07:48.069 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 410.227302464, 2024-04-26T02:07:48.069 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.006672858 2024-04-26T02:07:48.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.070 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2024-04-26T02:07:48.070 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2024-04-26T02:07:48.070 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 61477, 2024-04-26T02:07:48.070 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 62932 2024-04-26T02:07:48.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.070 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 61477, 2024-04-26T02:07:48.070 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 915, 2024-04-26T02:07:48.070 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 60562, 2024-04-26T02:07:48.071 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2024-04-26T02:07:48.071 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2024-04-26T02:07:48.071 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 34, 2024-04-26T02:07:48.071 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 21, 2024-04-26T02:07:48.071 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 915, 2024-04-26T02:07:48.071 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 33691, 2024-04-26T02:07:48.071 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 541, 2024-04-26T02:07:48.071 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2024-04-26T02:07:48.071 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2024-04-26T02:07:48.072 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2024-04-26T02:07:48.072 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 12, 2024-04-26T02:07:48.072 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 25418, 2024-04-26T02:07:48.072 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2024-04-26T02:07:48.072 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2024-04-26T02:07:48.072 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2024-04-26T02:07:48.072 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 0, 2024-04-26T02:07:48.072 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 42, 2024-04-26T02:07:48.072 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2024-04-26T02:07:48.073 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2024-04-26T02:07:48.073 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2024-04-26T02:07:48.073 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2024-04-26T02:07:48.073 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2024-04-26T02:07:48.073 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2024-04-26T02:07:48.073 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T02:07:48.073 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2024-04-26T02:07:48.073 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2024-04-26T02:07:48.073 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2024-04-26T02:07:48.074 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2024-04-26T02:07:48.074 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2024-04-26T02:07:48.074 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2024-04-26T02:07:48.074 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2024-04-26T02:07:48.074 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2024-04-26T02:07:48.074 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2024-04-26T02:07:48.074 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2024-04-26T02:07:48.074 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2024-04-26T02:07:48.075 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2024-04-26T02:07:48.075 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2024-04-26T02:07:48.075 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2024-04-26T02:07:48.075 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2024-04-26T02:07:48.075 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2024-04-26T02:07:48.075 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2024-04-26T02:07:48.075 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2024-04-26T02:07:48.075 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2024-04-26T02:07:48.075 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 132, 2024-04-26T02:07:48.076 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2024-04-26T02:07:48.076 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 41, 2024-04-26T02:07:48.076 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 12, 2024-04-26T02:07:48.076 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 12, 2024-04-26T02:07:48.076 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 0, 2024-04-26T02:07:48.076 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2024-04-26T02:07:48.076 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 1645, 2024-04-26T02:07:48.076 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 0, 2024-04-26T02:07:48.077 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 613 2024-04-26T02:07:48.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.077 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2024-04-26T02:07:48.077 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 1, 2024-04-26T02:07:48.077 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2024-04-26T02:07:48.077 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2024-04-26T02:07:48.077 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2024-04-26T02:07:48.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.078 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2024-04-26T02:07:48.078 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 2252 2024-04-26T02:07:48.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.078 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2024-04-26T02:07:48.078 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2024-04-26T02:07:48.078 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 10, 2024-04-26T02:07:48.078 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2024-04-26T02:07:48.079 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 10, 2024-04-26T02:07:48.079 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed_ops": 25207, 2024-04-26T02:07:48.079 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 25207, 2024-04-26T02:07:48.079 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2024-04-26T02:07:48.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.079 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:48.079 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:48.079 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-26T02:07:48.080 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:48.080 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 684950, 2024-04-26T02:07:48.080 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 152529875, 2024-04-26T02:07:48.080 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:48.080 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 684950, 2024-04-26T02:07:48.080 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-26T02:07:48.080 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-26T02:07:48.080 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 684950, 2024-04-26T02:07:48.081 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 152529875, 2024-04-26T02:07:48.081 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:48.081 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.081 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.081 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.081 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:48.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.081 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2024-04-26T02:07:48.082 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:48.082 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-26T02:07:48.082 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:48.082 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-26T02:07:48.082 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-26T02:07:48.082 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:48.082 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:48.082 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 61477, 2024-04-26T02:07:48.083 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 586989182, 2024-04-26T02:07:48.083 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 61477, 2024-04-26T02:07:48.083 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 586989182, 2024-04-26T02:07:48.083 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:48.083 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.083 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.083 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.084 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:48.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.084 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2024-04-26T02:07:48.084 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:48.084 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2024-04-26T02:07:48.084 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:48.084 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-26T02:07:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-26T02:07:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 61477, 2024-04-26T02:07:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 61477, 2024-04-26T02:07:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 61477, 2024-04-26T02:07:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 61477, 2024-04-26T02:07:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2024-04-26T02:07:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-26T02:07:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 25207, 2024-04-26T02:07:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 2550003, 2024-04-26T02:07:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 25207, 2024-04-26T02:07:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-26T02:07:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-26T02:07:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 915, 2024-04-26T02:07:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 2550003, 2024-04-26T02:07:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x564da9c1e460": { 2024-04-26T02:07:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 29970, 2024-04-26T02:07:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-26T02:07:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 124850, 2024-04-26T02:07:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 577707384, 2024-04-26T02:07:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 124850, 2024-04-26T02:07:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-26T02:07:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-26T02:07:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 32776, 2024-04-26T02:07:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 577677414, 2024-04-26T02:07:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:48.090 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:48.090 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:48.090 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:48.090 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:48.090 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:48.090 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-26T02:07:49.005 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:48 smithi188 ceph-mon[32751]: pgmap v695: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.5 MiB/s wr, 43 op/s 2024-04-26T02:07:49.036 INFO:tasks.check_counter:Daemon mds.c mds.exported=6 2024-04-26T02:07:49.036 INFO:tasks.check_counter:Daemon mds.c mds.imported=8 2024-04-26T02:07:49.037 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=275 2024-04-26T02:07:49.037 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=1438 2024-04-26T02:07:49.037 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=7 2024-04-26T02:07:49.037 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-26T02:07:49.037 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T02:07:49.037 DEBUG:tasks.check_counter:Getting stats from f 2024-04-26T02:07:49.037 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-26T02:07:49.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:48 smithi006 ceph-mon[28419]: pgmap v695: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.5 MiB/s wr, 43 op/s 2024-04-26T02:07:49.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:48 smithi029 ceph-mon[32775]: pgmap v695: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.5 MiB/s wr, 43 op/s 2024-04-26T02:07:49.289 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T02:07:50.959 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-26T02:07:50.959 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:50.959 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 182, 2024-04-26T02:07:50.959 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 184, 2024-04-26T02:07:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 12645497, 2024-04-26T02:07:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 57886, 2024-04-26T02:07:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 7, 2024-04-26T02:07:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.025137427, 2024-04-26T02:07:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.004643372, 2024-04-26T02:07:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.013115945, 2024-04-26T02:07:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.004983824, 2024-04-26T02:07:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 182, 2024-04-26T02:07:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.010698792, 2024-04-26T02:07:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000058784 2024-04-26T02:07:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T02:07:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T02:07:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 75, 2024-04-26T02:07:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 90, 2024-04-26T02:07:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 8584863, 2024-04-26T02:07:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 137405, 2024-04-26T02:07:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 5, 2024-04-26T02:07:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.012270104, 2024-04-26T02:07:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.002588513, 2024-04-26T02:07:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.007317899, 2024-04-26T02:07:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.001774755, 2024-04-26T02:07:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 89, 2024-04-26T02:07:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.009061911, 2024-04-26T02:07:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000101819 2024-04-26T02:07:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-26T02:07:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 115088 2024-04-26T02:07:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 86, 2024-04-26T02:07:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 84, 2024-04-26T02:07:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 12764968, 2024-04-26T02:07:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 31348, 2024-04-26T02:07:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.016517559, 2024-04-26T02:07:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.002307755, 2024-04-26T02:07:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.014519164, 2024-04-26T02:07:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.001709634, 2024-04-26T02:07:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 83, 2024-04-26T02:07:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.007255867, 2024-04-26T02:07:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000087420 2024-04-26T02:07:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 167760, 2024-04-26T02:07:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 15120 2024-04-26T02:07:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2024-04-26T02:07:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2024-04-26T02:07:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2024-04-26T02:07:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2024-04-26T02:07:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-26T02:07:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-26T02:07:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 489, 2024-04-26T02:07:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.025071211, 2024-04-26T02:07:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000051270 2024-04-26T02:07:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2024-04-26T02:07:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-26T02:07:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-26T02:07:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2024-04-26T02:07:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 0, 2024-04-26T02:07:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 0, 2024-04-26T02:07:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2024-04-26T02:07:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2024-04-26T02:07:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 0, 2024-04-26T02:07:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 0, 2024-04-26T02:07:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2024-04-26T02:07:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 0, 2024-04-26T02:07:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 0, 2024-04-26T02:07:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 0, 2024-04-26T02:07:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 2557, 2024-04-26T02:07:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 1105, 2024-04-26T02:07:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 1452, 2024-04-26T02:07:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 0, 2024-04-26T02:07:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 979, 2024-04-26T02:07:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 0, 2024-04-26T02:07:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 0, 2024-04-26T02:07:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 0, 2024-04-26T02:07:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 22, 2024-04-26T02:07:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 0, 2024-04-26T02:07:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 0, 2024-04-26T02:07:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 0, 2024-04-26T02:07:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 0, 2024-04-26T02:07:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 0, 2024-04-26T02:07:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2024-04-26T02:07:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 0, 2024-04-26T02:07:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 0, 2024-04-26T02:07:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 0, 2024-04-26T02:07:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 0, 2024-04-26T02:07:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 0, 2024-04-26T02:07:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 0, 2024-04-26T02:07:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 0, 2024-04-26T02:07:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2024-04-26T02:07:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 0, 2024-04-26T02:07:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2024-04-26T02:07:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 742, 2024-04-26T02:07:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 225232701, 2024-04-26T02:07:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 7, 2024-04-26T02:07:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T02:07:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2024-04-26T02:07:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T02:07:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2024-04-26T02:07:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 0, 2024-04-26T02:07:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T02:07:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T02:07:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 0, 2024-04-26T02:07:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 0, 2024-04-26T02:07:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T02:07:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 0, 2024-04-26T02:07:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T02:07:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T02:07:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T02:07:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 0, 2024-04-26T02:07:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 0, 2024-04-26T02:07:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 0, 2024-04-26T02:07:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 0 2024-04-26T02:07:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2024-04-26T02:07:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 0, 2024-04-26T02:07:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 0, 2024-04-26T02:07:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 0, 2024-04-26T02:07:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 0, 2024-04-26T02:07:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 0, 2024-04-26T02:07:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 0, 2024-04-26T02:07:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2024-04-26T02:07:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2024-04-26T02:07:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 0, 2024-04-26T02:07:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 0, 2024-04-26T02:07:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2024-04-26T02:07:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2024-04-26T02:07:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2024-04-26T02:07:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2024-04-26T02:07:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2024-04-26T02:07:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2024-04-26T02:07:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2024-04-26T02:07:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_path": 0, 2024-04-26T02:07:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_inode": 0, 2024-04-26T02:07:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T02:07:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 0, 2024-04-26T02:07:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2024-04-26T02:07:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 0, 2024-04-26T02:07:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2024-04-26T02:07:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2024-04-26T02:07:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2024-04-26T02:07:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: "evlrg": 0, 2024-04-26T02:07:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 0, 2024-04-26T02:07:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 0, 2024-04-26T02:07:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 0, 2024-04-26T02:07:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 8434, 2024-04-26T02:07:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2024-04-26T02:07:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 0, 2024-04-26T02:07:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 0, 2024-04-26T02:07:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 0, 2024-04-26T02:07:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 0, 2024-04-26T02:07:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 4, 2024-04-26T02:07:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: "segmjr": 3, 2024-04-26T02:07:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2024-04-26T02:07:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 0, 2024-04-26T02:07:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 594376683, 2024-04-26T02:07:50.982 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 600144274, 2024-04-26T02:07:50.982 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 600132280, 2024-04-26T02:07:50.982 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2024-04-26T02:07:50.982 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.982 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.982 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 8434 2024-04-26T02:07:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2024-04-26T02:07:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 429, 2024-04-26T02:07:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 2290, 2024-04-26T02:07:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 1861, 2024-04-26T02:07:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 117, 2024-04-26T02:07:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 119, 2024-04-26T02:07:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 2, 2024-04-26T02:07:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 2557, 2024-04-26T02:07:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 2557, 2024-04-26T02:07:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 0, 2024-04-26T02:07:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 0, 2024-04-26T02:07:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 0, 2024-04-26T02:07:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 0, 2024-04-26T02:07:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 55264, 2024-04-26T02:07:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 215392 2024-04-26T02:07:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2024-04-26T02:07:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 0, 2024-04-26T02:07:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 0, 2024-04-26T02:07:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 0, 2024-04-26T02:07:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 0, 2024-04-26T02:07:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 0, 2024-04-26T02:07:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2024-04-26T02:07:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2024-04-26T02:07:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2024-04-26T02:07:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2024-04-26T02:07:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2024-04-26T02:07:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2024-04-26T02:07:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2024-04-26T02:07:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2024-04-26T02:07:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2024-04-26T02:07:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2024-04-26T02:07:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2024-04-26T02:07:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2024-04-26T02:07:50.992 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.992 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2024-04-26T02:07:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2024-04-26T02:07:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2024-04-26T02:07:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.995 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2024-04-26T02:07:50.995 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.995 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.995 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.995 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2024-04-26T02:07:50.995 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.996 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.996 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.996 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2024-04-26T02:07:50.996 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.996 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.996 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.996 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2024-04-26T02:07:50.997 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.997 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.997 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.997 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2024-04-26T02:07:50.997 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.997 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.997 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2024-04-26T02:07:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2024-04-26T02:07:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2024-04-26T02:07:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2024-04-26T02:07:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2024-04-26T02:07:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2024-04-26T02:07:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2024-04-26T02:07:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2024-04-26T02:07:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2024-04-26T02:07:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2024-04-26T02:07:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2024-04-26T02:07:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2024-04-26T02:07:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2024-04-26T02:07:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 2, 2024-04-26T02:07:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 2, 2024-04-26T02:07:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 0, 2024-04-26T02:07:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 0, 2024-04-26T02:07:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2024-04-26T02:07:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 0, 2024-04-26T02:07:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 0, 2024-04-26T02:07:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 2949, 2024-04-26T02:07:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: "mdthresh_evicted": 0 2024-04-26T02:07:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2024-04-26T02:07:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:51.007 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:51.007 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:51.007 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:51.007 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:51.007 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:51.007 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:51.007 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2024-04-26T02:07:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 240037, 2024-04-26T02:07:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 797, 2024-04-26T02:07:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2024-04-26T02:07:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2024-04-26T02:07:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 8320, 2024-04-26T02:07:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 173, 2024-04-26T02:07:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2024-04-26T02:07:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 3278393, 2024-04-26T02:07:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 26509, 2024-04-26T02:07:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2024-04-26T02:07:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2024-04-26T02:07:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2024-04-26T02:07:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2024-04-26T02:07:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2024-04-26T02:07:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 303, 2024-04-26T02:07:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 26, 2024-04-26T02:07:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 4, 2024-04-26T02:07:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 303, 2024-04-26T02:07:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2024-04-26T02:07:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 303, 2024-04-26T02:07:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.630988255, 2024-04-26T02:07:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.002082469 2024-04-26T02:07:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2024-04-26T02:07:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2024-04-26T02:07:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 303, 2024-04-26T02:07:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 304 2024-04-26T02:07:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 303, 2024-04-26T02:07:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 303, 2024-04-26T02:07:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 0, 2024-04-26T02:07:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2024-04-26T02:07:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2024-04-26T02:07:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 166, 2024-04-26T02:07:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 0, 2024-04-26T02:07:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 134, 2024-04-26T02:07:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 0, 2024-04-26T02:07:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 0, 2024-04-26T02:07:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2024-04-26T02:07:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2024-04-26T02:07:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2024-04-26T02:07:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 0, 2024-04-26T02:07:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 0, 2024-04-26T02:07:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2024-04-26T02:07:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2024-04-26T02:07:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2024-04-26T02:07:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 2, 2024-04-26T02:07:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 0, 2024-04-26T02:07:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2024-04-26T02:07:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2024-04-26T02:07:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2024-04-26T02:07:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2024-04-26T02:07:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2024-04-26T02:07:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2024-04-26T02:07:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T02:07:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2024-04-26T02:07:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2024-04-26T02:07:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2024-04-26T02:07:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2024-04-26T02:07:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2024-04-26T02:07:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2024-04-26T02:07:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2024-04-26T02:07:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2024-04-26T02:07:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2024-04-26T02:07:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2024-04-26T02:07:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2024-04-26T02:07:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2024-04-26T02:07:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2024-04-26T02:07:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2024-04-26T02:07:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2024-04-26T02:07:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2024-04-26T02:07:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2024-04-26T02:07:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2024-04-26T02:07:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2024-04-26T02:07:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 132, 2024-04-26T02:07:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2024-04-26T02:07:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 2, 2024-04-26T02:07:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 8, 2024-04-26T02:07:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 8, 2024-04-26T02:07:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 0, 2024-04-26T02:07:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2024-04-26T02:07:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 0, 2024-04-26T02:07:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 2, 2024-04-26T02:07:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 0 2024-04-26T02:07:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2024-04-26T02:07:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 0, 2024-04-26T02:07:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2024-04-26T02:07:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2024-04-26T02:07:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2024-04-26T02:07:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2024-04-26T02:07:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 0 2024-04-26T02:07:51.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.022 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2024-04-26T02:07:51.022 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2024-04-26T02:07:51.022 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T02:07:51.022 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2024-04-26T02:07:51.022 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 0, 2024-04-26T02:07:51.022 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed_ops": 0, 2024-04-26T02:07:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 0, 2024-04-26T02:07:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2024-04-26T02:07:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-26T02:07:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 339, 2024-04-26T02:07:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 33960056, 2024-04-26T02:07:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 339, 2024-04-26T02:07:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-26T02:07:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-26T02:07:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 339, 2024-04-26T02:07:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 33960056, 2024-04-26T02:07:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2024-04-26T02:07:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-26T02:07:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-26T02:07:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-26T02:07:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 303, 2024-04-26T02:07:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 33734528, 2024-04-26T02:07:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 303, 2024-04-26T02:07:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 33734528, 2024-04-26T02:07:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2024-04-26T02:07:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2024-04-26T02:07:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-26T02:07:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-26T02:07:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:51.028 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 303, 2024-04-26T02:07:51.028 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 303, 2024-04-26T02:07:51.028 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 303, 2024-04-26T02:07:51.028 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 303, 2024-04-26T02:07:51.028 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:51.028 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:51.028 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:51.028 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:51.028 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:51.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.029 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2024-04-26T02:07:51.029 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:51.029 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-26T02:07:51.029 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:51.029 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-26T02:07:51.029 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-26T02:07:51.029 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:51.029 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:51.030 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-26T02:07:51.030 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-26T02:07:51.030 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 0, 2024-04-26T02:07:51.030 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 0, 2024-04-26T02:07:51.030 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:51.030 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:51.030 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:51.030 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x557c392586e0": { 2024-04-26T02:07:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-26T02:07:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-26T02:07:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-26T02:07:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-26T02:07:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-26T02:07:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 0, 2024-04-26T02:07:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 0, 2024-04-26T02:07:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:51.033 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:51.033 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:51.033 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-26T02:07:51.033 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:50 smithi188 ceph-mon[32751]: pgmap v696: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 10 op/s 2024-04-26T02:07:51.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:50 smithi006 ceph-mon[28419]: pgmap v696: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 10 op/s 2024-04-26T02:07:51.194 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:50 smithi029 ceph-mon[32775]: pgmap v696: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 10 op/s 2024-04-26T02:07:51.683 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-26T02:07:51.684 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-26T02:07:51.684 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-26T02:07:51.684 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-04-26T02:07:51.684 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=7 2024-04-26T02:07:51.684 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-26T02:07:51.684 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T02:07:51.684 DEBUG:tasks.check_counter:Getting stats from i 2024-04-26T02:07:51.684 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-26T02:07:51.944 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T02:07:52.016 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:51 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:07:52.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:51 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:07:52.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:51 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:07:53.013 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:52 smithi188 ceph-mon[32751]: pgmap v697: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 16 op/s 2024-04-26T02:07:53.014 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:52 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:07:53.014 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:52 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:07:53.014 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:52 smithi188 ceph-mon[32751]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:07:53.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:52 smithi006 ceph-mon[28419]: pgmap v697: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 16 op/s 2024-04-26T02:07:53.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:52 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:07:53.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:52 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:07:53.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:52 smithi006 ceph-mon[28419]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:07:53.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:52 smithi029 ceph-mon[32775]: pgmap v697: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 16 op/s 2024-04-26T02:07:53.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:52 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:07:53.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:52 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:07:53.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:52 smithi029 ceph-mon[32775]: from='mgr.25037 ' entity='mgr.x' 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 211252, 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 249399, 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 58564603, 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 335309580, 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 251, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 5, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 13.516159755, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 5.480866296, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 7.183841922, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 1.667187685, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 249401, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 18.397082575, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000073765 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 388871, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.067657471, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000173 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 2042752, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 279664 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 353165, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 480008, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 116929710, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 532484477, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 19, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 5, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 20.388998469, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 8.779455202, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 12.590501493, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 1.710384571, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 480005, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 31.282559966, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000065171 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 610990, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.116183436, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000190 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 108032, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 768320 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 260009, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 275117, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 75166323, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 269808987, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 10, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 8, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 17.239636522, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 7.811499405, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 9.383701062, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 1.684697875, 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 275113, 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 23.495494990, 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000085403 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 468036, 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.073387060, 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000156 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 707552, 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 230784 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 97915, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 115.090625430, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001175413 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 42063, 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 7.230292986, 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000171891 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 219294, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 191024, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 191024, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 1501.889572369, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.007862308 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 28270, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 20, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 3759, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 21, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 17, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 3670, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 1825, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 1835, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 10, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 1064, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 1885, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 217, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 217, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 24, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 338271, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 198984, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 28245, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 108, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 12, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 3325, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 12317, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 6, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 13, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 1257, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 11, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 561, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 0, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 742, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 225232701, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 7, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 0, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 0, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 51, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 37910, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 103493, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 79203, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 23895, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 9785, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 129510 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2024-04-26T02:07:53.592 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 517, 2024-04-26T02:07:53.592 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 109, 2024-04-26T02:07:53.592 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 71, 2024-04-26T02:07:53.592 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 146, 2024-04-26T02:07:53.592 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 1621, 2024-04-26T02:07:53.592 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 0, 2024-04-26T02:07:53.592 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2024-04-26T02:07:53.592 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2024-04-26T02:07:53.592 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 24065, 2024-04-26T02:07:53.593 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 24065, 2024-04-26T02:07:53.593 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2024-04-26T02:07:53.593 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2024-04-26T02:07:53.593 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2024-04-26T02:07:53.593 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2024-04-26T02:07:53.593 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2024-04-26T02:07:53.593 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2024-04-26T02:07:53.593 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2024-04-26T02:07:53.593 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_path": 14, 2024-04-26T02:07:53.594 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_inode": 1691, 2024-04-26T02:07:53.594 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T02:07:53.594 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 13, 2024-04-26T02:07:53.594 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2024-04-26T02:07:53.594 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 38, 2024-04-26T02:07:53.594 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2024-04-26T02:07:53.594 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2024-04-26T02:07:53.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.595 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2024-04-26T02:07:53.595 INFO:teuthology.orchestra.run.smithi188.stdout: "evlrg": 0, 2024-04-26T02:07:53.595 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 143923, 2024-04-26T02:07:53.595 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 142177, 2024-04-26T02:07:53.595 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 142177, 2024-04-26T02:07:53.595 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 1746, 2024-04-26T02:07:53.595 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2024-04-26T02:07:53.595 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 3226, 2024-04-26T02:07:53.595 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 285, 2024-04-26T02:07:53.596 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 281, 2024-04-26T02:07:53.596 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 281, 2024-04-26T02:07:53.596 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 4, 2024-04-26T02:07:53.596 INFO:teuthology.orchestra.run.smithi188.stdout: "segmjr": 1, 2024-04-26T02:07:53.596 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2024-04-26T02:07:53.596 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 6, 2024-04-26T02:07:53.596 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 594376683, 2024-04-26T02:07:53.596 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 600245323, 2024-04-26T02:07:53.596 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 0, 2024-04-26T02:07:53.597 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2024-04-26T02:07:53.597 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 51946, 2024-04-26T02:07:53.597 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 327.280038319, 2024-04-26T02:07:53.597 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.006300389 2024-04-26T02:07:53.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.597 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 0 2024-04-26T02:07:53.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.597 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2024-04-26T02:07:53.597 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 2238, 2024-04-26T02:07:53.598 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 27537, 2024-04-26T02:07:53.598 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 25299, 2024-04-26T02:07:53.598 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 287, 2024-04-26T02:07:53.598 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 546, 2024-04-26T02:07:53.598 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 259, 2024-04-26T02:07:53.598 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 3675, 2024-04-26T02:07:53.598 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 55497, 2024-04-26T02:07:53.598 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 51822, 2024-04-26T02:07:53.598 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 208, 2024-04-26T02:07:53.599 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 27469, 2024-04-26T02:07:53.599 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 27261, 2024-04-26T02:07:53.599 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 94492, 2024-04-26T02:07:53.599 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 231800 2024-04-26T02:07:53.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.599 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2024-04-26T02:07:53.599 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 310490, 2024-04-26T02:07:53.599 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 25352, 2024-04-26T02:07:53.600 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 219294, 2024-04-26T02:07:53.600 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 4121, 2024-04-26T02:07:53.600 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 52737, 2024-04-26T02:07:53.600 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2024-04-26T02:07:53.600 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 24894, 2024-04-26T02:07:53.600 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 281.027843776, 2024-04-26T02:07:53.600 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.011288979 2024-04-26T02:07:53.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.600 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2024-04-26T02:07:53.601 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 22821, 2024-04-26T02:07:53.601 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 189.063335450, 2024-04-26T02:07:53.601 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.008284620 2024-04-26T02:07:53.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.601 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2024-04-26T02:07:53.601 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.601 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.601 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.602 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2024-04-26T02:07:53.602 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.602 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.602 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.602 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2024-04-26T02:07:53.602 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 51830, 2024-04-26T02:07:53.602 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 262.990893766, 2024-04-26T02:07:53.603 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.005074105 2024-04-26T02:07:53.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.603 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2024-04-26T02:07:53.603 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.603 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.603 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.603 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2024-04-26T02:07:53.604 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.604 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.604 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.604 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2024-04-26T02:07:53.604 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.604 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.604 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.605 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2024-04-26T02:07:53.605 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.605 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.605 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.605 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2024-04-26T02:07:53.605 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.605 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.605 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.606 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2024-04-26T02:07:53.606 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 22, 2024-04-26T02:07:53.606 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.008823868, 2024-04-26T02:07:53.606 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000401084 2024-04-26T02:07:53.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.606 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2024-04-26T02:07:53.606 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 119, 2024-04-26T02:07:53.606 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 2.684777058, 2024-04-26T02:07:53.607 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.022561151 2024-04-26T02:07:53.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.607 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2024-04-26T02:07:53.607 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.607 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.607 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.607 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2024-04-26T02:07:53.607 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 22, 2024-04-26T02:07:53.608 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 5.938290993, 2024-04-26T02:07:53.608 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.269922317 2024-04-26T02:07:53.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.608 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2024-04-26T02:07:53.608 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 16, 2024-04-26T02:07:53.608 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.031552158, 2024-04-26T02:07:53.608 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001972009 2024-04-26T02:07:53.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.609 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2024-04-26T02:07:53.609 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 60950, 2024-04-26T02:07:53.609 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 292.864986050, 2024-04-26T02:07:53.609 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.004805003 2024-04-26T02:07:53.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.609 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2024-04-26T02:07:53.609 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 5443, 2024-04-26T02:07:53.609 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 69.477793112, 2024-04-26T02:07:53.610 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.012764613 2024-04-26T02:07:53.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.610 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2024-04-26T02:07:53.610 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.610 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.610 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.610 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2024-04-26T02:07:53.610 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.611 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.611 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.611 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2024-04-26T02:07:53.611 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1, 2024-04-26T02:07:53.611 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.001251497, 2024-04-26T02:07:53.611 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001251497 2024-04-26T02:07:53.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.611 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2024-04-26T02:07:53.611 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 15, 2024-04-26T02:07:53.612 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 5.986951409, 2024-04-26T02:07:53.612 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.399130093 2024-04-26T02:07:53.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.612 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2024-04-26T02:07:53.612 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.612 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.612 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.612 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2024-04-26T02:07:53.613 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 191, 2024-04-26T02:07:53.613 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 160.443942169, 2024-04-26T02:07:53.613 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.840020639 2024-04-26T02:07:53.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.613 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2024-04-26T02:07:53.613 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.613 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.613 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.614 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2024-04-26T02:07:53.614 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 628, 2024-04-26T02:07:53.614 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 2.085244343, 2024-04-26T02:07:53.614 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.003320452 2024-04-26T02:07:53.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.614 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2024-04-26T02:07:53.614 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.614 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.614 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.615 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2024-04-26T02:07:53.615 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 7, 2024-04-26T02:07:53.615 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.021704838, 2024-04-26T02:07:53.615 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.003100691 2024-04-26T02:07:53.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.615 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2024-04-26T02:07:53.615 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.615 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.615 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.616 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2024-04-26T02:07:53.616 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 24065, 2024-04-26T02:07:53.616 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 229.262181882, 2024-04-26T02:07:53.616 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.009526789 2024-04-26T02:07:53.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.616 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2024-04-26T02:07:53.616 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2024-04-26T02:07:53.616 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2024-04-26T02:07:53.617 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.617 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.617 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.617 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:53.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.617 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2024-04-26T02:07:53.617 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 2, 2024-04-26T02:07:53.617 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 5, 2024-04-26T02:07:53.617 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 3, 2024-04-26T02:07:53.617 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 2, 2024-04-26T02:07:53.618 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2024-04-26T02:07:53.618 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 31083, 2024-04-26T02:07:53.618 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 15541, 2024-04-26T02:07:53.618 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 1312, 2024-04-26T02:07:53.618 INFO:teuthology.orchestra.run.smithi188.stdout: "mdthresh_evicted": 0 2024-04-26T02:07:53.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.618 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2024-04-26T02:07:53.618 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 360, 2024-04-26T02:07:53.618 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 360, 2024-04-26T02:07:53.619 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:53.619 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:53.619 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:53.619 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:53.619 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:53.619 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:53.619 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:53.619 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:53.620 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:53.620 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:53.620 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:53.620 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:53.620 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:53.620 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:53.620 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:53.620 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:53.620 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:53.621 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:53.621 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:53.621 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:53.621 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:53.621 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:53.621 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:53.621 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:53.621 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:53.621 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:53.621 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:53.622 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:53.622 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:53.622 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2024-04-26T02:07:53.622 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:53.622 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:53.622 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:53.622 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:53.622 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 329445, 2024-04-26T02:07:53.623 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 1212, 2024-04-26T02:07:53.623 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:53.623 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:53.623 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2024-04-26T02:07:53.623 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2024-04-26T02:07:53.623 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:53.623 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:53.623 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:53.623 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:53.623 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 8320, 2024-04-26T02:07:53.624 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 173, 2024-04-26T02:07:53.624 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:53.624 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:53.624 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:53.624 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2024-04-26T02:07:53.624 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 8633967, 2024-04-26T02:07:53.624 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 70584, 2024-04-26T02:07:53.624 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:53.624 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2024-04-26T02:07:53.624 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:53.625 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2024-04-26T02:07:53.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.625 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2024-04-26T02:07:53.625 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2024-04-26T02:07:53.625 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2024-04-26T02:07:53.625 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 98218, 2024-04-26T02:07:53.625 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 628039761, 2024-04-26T02:07:53.625 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 31, 2024-04-26T02:07:53.625 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 98218, 2024-04-26T02:07:53.626 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2024-04-26T02:07:53.626 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 98218, 2024-04-26T02:07:53.626 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 1769.122717142, 2024-04-26T02:07:53.626 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.018012204 2024-04-26T02:07:53.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.626 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2024-04-26T02:07:53.626 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2024-04-26T02:07:53.626 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 98218, 2024-04-26T02:07:53.626 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 135699 2024-04-26T02:07:53.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.627 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 98218, 2024-04-26T02:07:53.627 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 923, 2024-04-26T02:07:53.627 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 97295, 2024-04-26T02:07:53.627 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2024-04-26T02:07:53.627 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2024-04-26T02:07:53.627 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 3443, 2024-04-26T02:07:53.627 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 13796, 2024-04-26T02:07:53.627 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 903, 2024-04-26T02:07:53.627 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 52849, 2024-04-26T02:07:53.628 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 878, 2024-04-26T02:07:53.628 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2024-04-26T02:07:53.628 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2024-04-26T02:07:53.628 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2024-04-26T02:07:53.628 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 21, 2024-04-26T02:07:53.628 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 24975, 2024-04-26T02:07:53.628 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2024-04-26T02:07:53.628 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2024-04-26T02:07:53.628 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2024-04-26T02:07:53.628 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 14, 2024-04-26T02:07:53.629 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 27592, 2024-04-26T02:07:53.629 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2024-04-26T02:07:53.629 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2024-04-26T02:07:53.629 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2024-04-26T02:07:53.629 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2024-04-26T02:07:53.629 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2024-04-26T02:07:53.629 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2024-04-26T02:07:53.629 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T02:07:53.629 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2024-04-26T02:07:53.629 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2024-04-26T02:07:53.630 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2024-04-26T02:07:53.630 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2024-04-26T02:07:53.630 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2024-04-26T02:07:53.630 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2024-04-26T02:07:53.630 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2024-04-26T02:07:53.630 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2024-04-26T02:07:53.630 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2024-04-26T02:07:53.630 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2024-04-26T02:07:53.631 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2024-04-26T02:07:53.631 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2024-04-26T02:07:53.631 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2024-04-26T02:07:53.631 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2024-04-26T02:07:53.631 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2024-04-26T02:07:53.631 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2024-04-26T02:07:53.631 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2024-04-26T02:07:53.631 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2024-04-26T02:07:53.631 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2024-04-26T02:07:53.632 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 132, 2024-04-26T02:07:53.632 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2024-04-26T02:07:53.632 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 41, 2024-04-26T02:07:53.632 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 12, 2024-04-26T02:07:53.632 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 12, 2024-04-26T02:07:53.632 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 0, 2024-04-26T02:07:53.632 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2024-04-26T02:07:53.632 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 8724, 2024-04-26T02:07:53.632 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 40, 2024-04-26T02:07:53.633 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 2464 2024-04-26T02:07:53.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.633 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2024-04-26T02:07:53.633 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 1, 2024-04-26T02:07:53.633 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2024-04-26T02:07:53.633 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2024-04-26T02:07:53.633 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2024-04-26T02:07:53.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.633 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2024-04-26T02:07:53.633 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 2583 2024-04-26T02:07:53.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.634 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2024-04-26T02:07:53.634 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2024-04-26T02:07:53.634 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 10, 2024-04-26T02:07:53.634 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2024-04-26T02:07:53.634 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 10, 2024-04-26T02:07:53.634 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed_ops": 24065, 2024-04-26T02:07:53.634 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 24065, 2024-04-26T02:07:53.634 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2024-04-26T02:07:53.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.635 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:53.635 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:53.635 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-26T02:07:53.635 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:53.635 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 824422, 2024-04-26T02:07:53.635 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 176341522, 2024-04-26T02:07:53.635 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:53.635 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 824422, 2024-04-26T02:07:53.635 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-26T02:07:53.636 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-26T02:07:53.636 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 824422, 2024-04-26T02:07:53.636 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 176341522, 2024-04-26T02:07:53.636 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:53.636 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.636 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.636 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.636 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:53.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.637 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2024-04-26T02:07:53.637 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:53.637 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-26T02:07:53.637 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:53.637 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-26T02:07:53.637 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-26T02:07:53.637 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:53.637 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:53.637 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 98218, 2024-04-26T02:07:53.637 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 630474022, 2024-04-26T02:07:53.638 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 98218, 2024-04-26T02:07:53.638 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 630474022, 2024-04-26T02:07:53.638 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:53.638 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.638 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.638 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.638 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:53.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.639 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2024-04-26T02:07:53.642 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:53.642 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2024-04-26T02:07:53.642 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:53.642 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-26T02:07:53.642 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-26T02:07:53.642 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:53.642 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:53.642 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 98218, 2024-04-26T02:07:53.643 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 98218, 2024-04-26T02:07:53.643 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 98218, 2024-04-26T02:07:53.643 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 98218, 2024-04-26T02:07:53.643 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:53.643 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.643 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.643 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.643 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:53.643 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.644 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2024-04-26T02:07:53.644 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:53.644 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-26T02:07:53.644 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:53.644 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 24065, 2024-04-26T02:07:53.644 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 2434581, 2024-04-26T02:07:53.644 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:53.644 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 24065, 2024-04-26T02:07:53.644 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-26T02:07:53.645 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-26T02:07:53.645 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 903, 2024-04-26T02:07:53.645 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 2434581, 2024-04-26T02:07:53.645 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:53.645 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.645 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.645 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.645 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:53.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:53.646 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x5596ff0c2460": { 2024-04-26T02:07:53.646 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 101049, 2024-04-26T02:07:53.646 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-26T02:07:53.646 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:53.646 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 143923, 2024-04-26T02:07:53.646 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 596051019, 2024-04-26T02:07:53.646 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:53.646 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 143923, 2024-04-26T02:07:53.646 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-26T02:07:53.646 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-26T02:07:53.647 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 51946, 2024-04-26T02:07:53.647 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 595949970, 2024-04-26T02:07:53.647 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:53.647 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:53.647 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.647 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.647 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:53.647 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:53.647 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-26T02:07:54.324 INFO:tasks.check_counter:Daemon mds.i mds.exported=13 2024-04-26T02:07:54.324 INFO:tasks.check_counter:Daemon mds.i mds.imported=11 2024-04-26T02:07:54.324 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=517 2024-04-26T02:07:54.324 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=109 2024-04-26T02:07:54.324 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=7 2024-04-26T02:07:54.324 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=22 2024-04-26T02:07:54.324 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=15 2024-04-26T02:07:54.324 DEBUG:tasks.check_counter:Getting stats from l 2024-04-26T02:07:54.325 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-26T02:07:54.560 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T02:07:55.016 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:54 smithi188 ceph-mon[32751]: pgmap v698: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 10 op/s 2024-04-26T02:07:55.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:54 smithi006 ceph-mon[28419]: pgmap v698: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 10 op/s 2024-04-26T02:07:55.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:54 smithi029 ceph-mon[32775]: pgmap v698: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 10 op/s 2024-04-26T02:07:56.149 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-26T02:07:56.149 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:56.149 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 438, 2024-04-26T02:07:56.150 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 426, 2024-04-26T02:07:56.150 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 1789856, 2024-04-26T02:07:56.150 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 264096, 2024-04-26T02:07:56.150 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:56.150 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 0, 2024-04-26T02:07:56.150 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.071864443, 2024-04-26T02:07:56.150 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.023337447, 2024-04-26T02:07:56.150 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.045184336, 2024-04-26T02:07:56.151 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.010125597, 2024-04-26T02:07:56.151 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:56.151 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 425, 2024-04-26T02:07:56.151 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.018526895, 2024-04-26T02:07:56.151 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000043592 2024-04-26T02:07:56.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:56.151 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:56.151 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:56.152 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.152 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:56.152 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 1789856, 2024-04-26T02:07:56.152 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 264096 2024-04-26T02:07:56.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:56.152 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:56.152 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 2, 2024-04-26T02:07:56.152 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 326, 2024-04-26T02:07:56.153 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 320, 2024-04-26T02:07:56.153 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 213456, 2024-04-26T02:07:56.153 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 5, 2024-04-26T02:07:56.153 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:56.153 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.017306269, 2024-04-26T02:07:56.153 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.012617130, 2024-04-26T02:07:56.153 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.001292696, 2024-04-26T02:07:56.153 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:56.153 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:56.154 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 323, 2024-04-26T02:07:56.154 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.033572869, 2024-04-26T02:07:56.154 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000103940 2024-04-26T02:07:56.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:56.154 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:56.154 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:56.154 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.154 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:56.155 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T02:07:56.155 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 213456 2024-04-26T02:07:56.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:56.155 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:56.155 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 4, 2024-04-26T02:07:56.155 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 4, 2024-04-26T02:07:56.155 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 2992, 2024-04-26T02:07:56.155 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 448, 2024-04-26T02:07:56.156 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 2, 2024-04-26T02:07:56.156 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 1, 2024-04-26T02:07:56.156 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.000638034, 2024-04-26T02:07:56.156 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.000046126, 2024-04-26T02:07:56.156 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.000550728, 2024-04-26T02:07:56.156 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:56.156 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:56.156 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 3, 2024-04-26T02:07:56.156 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000748094, 2024-04-26T02:07:56.156 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000249364 2024-04-26T02:07:56.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:56.157 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:56.157 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:56.157 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.157 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:56.157 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T02:07:56.157 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T02:07:56.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:56.158 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2024-04-26T02:07:56.158 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 0, 2024-04-26T02:07:56.158 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2024-04-26T02:07:56.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:56.158 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2024-04-26T02:07:56.158 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:56.158 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:56.158 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:56.158 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:56.159 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:56.159 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:56.159 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:56.159 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:56.159 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:56.159 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:56.159 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:56.159 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:56.159 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:56.160 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:56.160 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:56.160 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:56.160 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:56.160 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:56.160 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:56.160 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:56.160 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:56.161 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:56.161 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:56.161 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:56.161 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:56.161 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:56.161 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:56.161 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:56.161 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:56.161 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:56.161 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:56.162 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2024-04-26T02:07:56.162 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:56.162 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:56.162 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:56.162 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:56.162 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 148692, 2024-04-26T02:07:56.162 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 30, 2024-04-26T02:07:56.162 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:56.162 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:56.163 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2024-04-26T02:07:56.163 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2024-04-26T02:07:56.163 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:56.163 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:56.163 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:56.163 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:56.163 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 0, 2024-04-26T02:07:56.163 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 0, 2024-04-26T02:07:56.164 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:56.164 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:56.164 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:56.164 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2024-04-26T02:07:56.164 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 928, 2024-04-26T02:07:56.164 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 2, 2024-04-26T02:07:56.165 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:56.165 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2024-04-26T02:07:56.165 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:56.165 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2024-04-26T02:07:56.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-26T02:07:56.165 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:56.165 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-26T02:07:56.165 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-26T02:07:56.165 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-26T02:07:56.165 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 440, 2024-04-26T02:07:56.166 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 1727952, 2024-04-26T02:07:56.166 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:56.166 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 440, 2024-04-26T02:07:56.166 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-26T02:07:56.166 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-26T02:07:56.166 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 440, 2024-04-26T02:07:56.166 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 1727952, 2024-04-26T02:07:56.166 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-26T02:07:56.167 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-26T02:07:56.167 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.167 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.167 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:56.167 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-26T02:07:56.167 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-26T02:07:56.705 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-26T02:07:56.705 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-26T02:07:56.705 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-04-26T02:07:56.706 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-04-26T02:07:56.706 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-26T02:07:56.706 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-26T02:07:56.706 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-26T02:07:56.706 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-26T02:07:56.723 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T02:07:56.723 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T02:07:56.723 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T02:07:56.785 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T02:07:56.785 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T02:07:56.844 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd blocklist ls 2024-04-26T02:07:57.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:56 smithi006 ceph-mon[28419]: pgmap v699: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 10 op/s 2024-04-26T02:07:57.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:56 smithi188 ceph-mon[32751]: pgmap v699: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 10 op/s 2024-04-26T02:07:57.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:56 smithi029 ceph-mon[32775]: pgmap v699: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 10 op/s 2024-04-26T02:07:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:59.012 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:07:58 smithi006 ceph-mon[28419]: pgmap v700: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 128 KiB/s wr, 15 op/s 2024-04-26T02:07:59.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:07:58 smithi188 ceph-mon[32751]: pgmap v700: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 128 KiB/s wr, 15 op/s 2024-04-26T02:07:59.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:07:58 smithi029 ceph-mon[32775]: pgmap v700: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 128 KiB/s wr, 15 op/s 2024-04-26T02:07:59.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.553+0000 7f6154600700 1 -- 172.21.15.6:0/3687002063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f614c10f4f0 msgr2=0x7f614c1118e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:59.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.553+0000 7f6154600700 1 --2- 172.21.15.6:0/3687002063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f614c10f4f0 0x7f614c1118e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f614400b0a0 tx=0x7f614400b3b0 comp rx=0 tx=0).stop 2024-04-26T02:07:59.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.553+0000 7f6154600700 1 -- 172.21.15.6:0/3687002063 shutdown_connections 2024-04-26T02:07:59.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.553+0000 7f6154600700 1 --2- 172.21.15.6:0/3687002063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f614c10f4f0 0x7f614c1118e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:59.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.553+0000 7f6154600700 1 --2- 172.21.15.6:0/3687002063 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f614c06a750 0x7f614c06ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:59.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.553+0000 7f6154600700 1 --2- 172.21.15.6:0/3687002063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614c06b070 0x7f614c10efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:59.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.553+0000 7f6154600700 1 -- 172.21.15.6:0/3687002063 >> 172.21.15.6:0/3687002063 conn(0x7f614c0788f0 msgr2=0x7f614c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:59.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.553+0000 7f6154600700 1 -- 172.21.15.6:0/3687002063 shutdown_connections 2024-04-26T02:07:59.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.553+0000 7f6154600700 1 -- 172.21.15.6:0/3687002063 wait complete. 2024-04-26T02:07:59.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.554+0000 7f6154600700 1 Processor -- start 2024-04-26T02:07:59.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.555+0000 7f6154600700 1 -- start start 2024-04-26T02:07:59.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.555+0000 7f6154600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f614c06a750 0x7f614c19b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:59.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.555+0000 7f6154600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614c06b070 0x7f614c19b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:59.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.555+0000 7f6154600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f614c10f4f0 0x7f614c19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:59.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.555+0000 7f6154600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f614c114e40 con 0x7f614c06a750 2024-04-26T02:07:59.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.555+0000 7f6154600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f614c114cc0 con 0x7f614c06b070 2024-04-26T02:07:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.555+0000 7f6154600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f614c114fc0 con 0x7f614c10f4f0 2024-04-26T02:07:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.555+0000 7f6151c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f614c06a750 0x7f614c19b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6151200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614c06b070 0x7f614c19b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6151c00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f614c06a750 0x7f614c19b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37952/0 (socket says 172.21.15.6:37952) 2024-04-26T02:07:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6151200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614c06b070 0x7f614c19b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59560/0 (socket says 172.21.15.6:59560) 2024-04-26T02:07:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6151c00700 1 -- 172.21.15.6:0/1030902170 learned_addr learned my addr 172.21.15.6:0/1030902170 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:07:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6152600700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f614c10f4f0 0x7f614c19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6151c00700 1 -- 172.21.15.6:0/1030902170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f614c10f4f0 msgr2=0x7f614c19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6151c00700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f614c10f4f0 0x7f614c19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:59.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6151c00700 1 -- 172.21.15.6:0/1030902170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614c06b070 msgr2=0x7f614c19b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:59.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6151c00700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614c06b070 0x7f614c19b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:59.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6151c00700 1 -- 172.21.15.6:0/1030902170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614c1a0330 con 0x7f614c06a750 2024-04-26T02:07:59.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6151200700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614c06b070 0x7f614c19b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:59.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6152600700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f614c10f4f0 0x7f614c19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:07:59.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.556+0000 7f6151c00700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f614c06a750 0x7f614c19b370 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f614000e9a0 tx=0x7f614000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:59.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.557+0000 7f614a000700 1 -- 172.21.15.6:0/1030902170 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614000cbd0 con 0x7f614c06a750 2024-04-26T02:07:59.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.557+0000 7f6154600700 1 -- 172.21.15.6:0/1030902170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f614c1a0620 con 0x7f614c06a750 2024-04-26T02:07:59.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.557+0000 7f614a000700 1 -- 172.21.15.6:0/1030902170 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f614000cd70 con 0x7f614c06a750 2024-04-26T02:07:59.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.557+0000 7f614a000700 1 -- 172.21.15.6:0/1030902170 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6140018680 con 0x7f614c06a750 2024-04-26T02:07:59.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.557+0000 7f6154600700 1 -- 172.21.15.6:0/1030902170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f614c1a7ea0 con 0x7f614c06a750 2024-04-26T02:07:59.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.558+0000 7f6154600700 1 -- 172.21.15.6:0/1030902170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f614c069730 con 0x7f614c06a750 2024-04-26T02:07:59.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.560+0000 7f614a000700 1 -- 172.21.15.6:0/1030902170 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f6140010430 con 0x7f614c06a750 2024-04-26T02:07:59.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.562+0000 7f614a000700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f61340a3e30 0x7f61340a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:59.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.562+0000 7f614a000700 1 -- 172.21.15.6:0/1030902170 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8600+0+0 (secure 0 0 0) 0x7f61400d0a80 con 0x7f614c06a750 2024-04-26T02:07:59.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.562+0000 7f6151200700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f61340a3e30 0x7f61340a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:59.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.563+0000 7f6151200700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f61340a3e30 0x7f61340a62f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f614c102e30 tx=0x7f6138009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:59.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.567+0000 7f614a000700 1 -- 172.21.15.6:0/1030902170 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61400d2e20 con 0x7f614c06a750 2024-04-26T02:07:59.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.815+0000 7f6154600700 1 -- 172.21.15.6:0/1030902170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f614c19c680 con 0x7f614c06a750 2024-04-26T02:07:59.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.816+0000 7f614a000700 1 -- 172.21.15.6:0/1030902170 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v132) ==== 81+0+920 (secure 0 0 0) 0x7f614009d070 con 0x7f614c06a750 2024-04-26T02:07:59.816 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6801/809793211 2024-04-27T01:45:54.919747+0000 2024-04-26T02:07:59.816 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6800/809793211 2024-04-27T01:45:54.919747+0000 2024-04-26T02:07:59.816 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:0/286280546 2024-04-27T01:45:54.919747+0000 2024-04-26T02:07:59.817 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:0/402969404 2024-04-27T01:45:54.919747+0000 2024-04-26T02:07:59.817 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:0/2707582744 2024-04-27T01:45:54.919747+0000 2024-04-26T02:07:59.817 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:0/3143272062 2024-04-27T01:33:47.021882+0000 2024-04-26T02:07:59.817 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:0/1325707596 2024-04-27T01:33:47.021882+0000 2024-04-26T02:07:59.817 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6800/635066163 2024-04-27T01:33:47.021882+0000 2024-04-26T02:07:59.817 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6801/635066163 2024-04-27T01:33:47.021882+0000 2024-04-26T02:07:59.817 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:0/1359518071 2024-04-27T01:33:11.534520+0000 2024-04-26T02:07:59.817 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:0/1823104535 2024-04-27T01:45:54.919747+0000 2024-04-26T02:07:59.817 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6801/801052348 2024-04-27T01:33:11.534520+0000 2024-04-26T02:07:59.818 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:0/2813598711 2024-04-27T01:33:11.534520+0000 2024-04-26T02:07:59.818 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:0/3140150861 2024-04-27T01:33:11.534520+0000 2024-04-26T02:07:59.818 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6800/801052348 2024-04-27T01:33:11.534520+0000 2024-04-26T02:07:59.818 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:0/65701414 2024-04-27T01:33:47.021882+0000 2024-04-26T02:07:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.819+0000 7f6154600700 1 -- 172.21.15.6:0/1030902170 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f61340a3e30 msgr2=0x7f61340a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.819+0000 7f6154600700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f61340a3e30 0x7f61340a62f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f614c102e30 tx=0x7f6138009260 comp rx=0 tx=0).stop 2024-04-26T02:07:59.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.819+0000 7f6154600700 1 -- 172.21.15.6:0/1030902170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f614c06a750 msgr2=0x7f614c19b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:59.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.819+0000 7f6154600700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f614c06a750 0x7f614c19b370 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f614000e9a0 tx=0x7f614000ecb0 comp rx=0 tx=0).stop 2024-04-26T02:07:59.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.819+0000 7f6154600700 1 -- 172.21.15.6:0/1030902170 shutdown_connections 2024-04-26T02:07:59.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.819+0000 7f6154600700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f61340a3e30 0x7f61340a62f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:59.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.819+0000 7f6154600700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f614c10f4f0 0x7f614c19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:59.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.819+0000 7f6154600700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f614c06a750 0x7f614c19b370 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:59.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.819+0000 7f6154600700 1 --2- 172.21.15.6:0/1030902170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614c06b070 0x7f614c19b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:59.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.819+0000 7f6154600700 1 -- 172.21.15.6:0/1030902170 >> 172.21.15.6:0/1030902170 conn(0x7f614c0788f0 msgr2=0x7f614c1109e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:59.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.820+0000 7f6154600700 1 -- 172.21.15.6:0/1030902170 shutdown_connections 2024-04-26T02:07:59.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:07:59.820+0000 7f6154600700 1 -- 172.21.15.6:0/1030902170 wait complete. 2024-04-26T02:07:59.821 INFO:teuthology.orchestra.run.smithi006.stderr:listed 16 entries 2024-04-26T02:08:00.590 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T02:08:00.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:00.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T02:08:00.917 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-04-26T02:08:00.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:00.917 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T02:08:01.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:01.010 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-04-26T02:08:01.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:00 smithi006 ceph-mon[28419]: pgmap v701: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 82 KiB/s wr, 9 op/s 2024-04-26T02:08:01.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:00 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1030902170' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T02:08:01.089 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T02:08:01.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:01.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T02:08:01.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:00 smithi188 ceph-mon[32751]: pgmap v701: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 82 KiB/s wr, 9 op/s 2024-04-26T02:08:01.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:00 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1030902170' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T02:08:01.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:00 smithi029 ceph-mon[32775]: pgmap v701: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 82 KiB/s wr, 9 op/s 2024-04-26T02:08:01.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:00 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1030902170' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T02:08:01.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:01.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link delete ceph-brx 2024-04-26T02:08:01.338 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T02:08:01.339 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-26T02:08:01.339 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T02:08:01.383 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/df4031ac8b4ff51df5720fbda651344ee2016ebfa78140fb31c50d6a2397874b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/LT2DWPBZMP6YTYEPYABBSVDWAY:/var/lib/containers/storage/overlay/l/7NC26RIQYI3KHATYHEYQHKP2QN,upperdir=/var/lib/containers/storage/overlay/df4031ac8b4ff51df5720fbda651344ee2016ebfa78140fb31c50d6a2397874b/diff,workdir=/var/lib/containers/storage/overlay/df4031ac8b4ff51df5720fbda651344ee2016ebfa78140fb31c50d6a2397874b/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/57723aa746490ae4f8f2699a7c34fcbb86e88f4567c0c0ccc57b561827a1836c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/LT2DWPBZMP6YTYEPYABBSVDWAY:/var/lib/containers/storage/overlay/l/7NC26RIQYI3KHATYHEYQHKP2QN,upperdir=/var/lib/containers/storage/overlay/57723aa746490ae4f8f2699a7c34fcbb86e88f4567c0c0ccc57b561827a1836c/diff,workdir=/var/lib/containers/storage/overlay/57723aa746490ae4f8f2699a7c34fcbb86e88f4567c0c0ccc57b561827a1836c/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/04ba3e92ada1af60b6ed95f45fdf5a76a4724e3248655de931d897dce6c5c0d3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/LT2DWPBZMP6YTYEPYABBSVDWAY:/var/lib/containers/storage/overlay/l/7NC26RIQYI3KHATYHEYQHKP2QN,upperdir=/var/lib/containers/storage/overlay/04ba3e92ada1af60b6ed95f45fdf5a76a4724e3248655de931d897dce6c5c0d3/diff,workdir=/var/lib/containers/storage/overlay/04ba3e92ada1af60b6ed95f45fdf5a76a4724e3248655de931d897dce6c5c0d3/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/bd473edd235fbf10e3032b5ff4f00e968da27f22732cc0b19010a7a500540bd3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/LT2DWPBZMP6YTYEPYABBSVDWAY:/var/lib/containers/storage/overlay/l/7NC26RIQYI3KHATYHEYQHKP2QN,upperdir=/var/lib/containers/storage/overlay/bd473edd235fbf10e3032b5ff4f00e968da27f22732cc0b19010a7a500540bd3/diff,workdir=/var/lib/containers/storage/overlay/bd473edd235fbf10e3032b5ff4f00e968da27f22732cc0b19010a7a500540bd3/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9cb5cbce47c99ff41ea18398ad4d25db4c9e502aa900de919eda56d4c0c34613/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/LT2DWPBZMP6YTYEPYABBSVDWAY:/var/lib/containers/storage/overlay/l/7NC26RIQYI3KHATYHEYQHKP2QN,upperdir=/var/lib/containers/storage/overlay/9cb5cbce47c99ff41ea18398ad4d25db4c9e502aa900de919eda56d4c0c34613/diff,workdir=/var/lib/containers/storage/overlay/9cb5cbce47c99ff41ea18398ad4d25db4c9e502aa900de919eda56d4c0c34613/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d420b45631e08b63f6b4ab52e1622af41e2869106025e216ad1cbc1feb63ca51/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/LT2DWPBZMP6YTYEPYABBSVDWAY:/var/lib/containers/storage/overlay/l/7NC26RIQYI3KHATYHEYQHKP2QN,upperdir=/var/lib/containers/storage/overlay/d420b45631e08b63f6b4ab52e1622af41e2869106025e216ad1cbc1feb63ca51/diff,workdir=/var/lib/containers/storage/overlay/d420b45631e08b63f6b4ab52e1622af41e2869106025e216ad1cbc1feb63ca51/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0b7e392683879e6c67b0ca9254e47038cabda9a15186fc7e9c8fdd7532ca9801/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/LT2DWPBZMP6YTYEPYABBSVDWAY:/var/lib/containers/storage/overlay/l/7NC26RIQYI3KHATYHEYQHKP2QN,upperdir=/var/lib/containers/storage/overlay/0b7e392683879e6c67b0ca9254e47038cabda9a15186fc7e9c8fdd7532ca9801/diff,workdir=/var/lib/containers/storage/overlay/0b7e392683879e6c67b0ca9254e47038cabda9a15186fc7e9c8fdd7532ca9801/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a69127c671d16e0ebe74296b855fc33f687c58e871c106dbd27def8b7788365f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/LT2DWPBZMP6YTYEPYABBSVDWAY:/var/lib/containers/storage/overlay/l/7NC26RIQYI3KHATYHEYQHKP2QN,upperdir=/var/lib/containers/storage/overlay/a69127c671d16e0ebe74296b855fc33f687c58e871c106dbd27def8b7788365f/diff,workdir=/var/lib/containers/storage/overlay/a69127c671d16e0ebe74296b855fc33f687c58e871c106dbd27def8b7788365f/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/19496f4aa8d0976ec7722d7406940705ec907dae0b67bcc65b67f1120d52849b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/LT2DWPBZMP6YTYEPYABBSVDWAY:/var/lib/containers/storage/overlay/l/7NC26RIQYI3KHATYHEYQHKP2QN,upperdir=/var/lib/containers/storage/overlay/19496f4aa8d0976ec7722d7406940705ec907dae0b67bcc65b67f1120d52849b/diff,workdir=/var/lib/containers/storage/overlay/19496f4aa8d0976ec7722d7406940705ec907dae0b67bcc65b67f1120d52849b/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/fd04cdeaee9e44eb6675b4aaf33b3e7f2bb57a3bebcf23063843e62a31c34b7a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/LT2DWPBZMP6YTYEPYABBSVDWAY:/var/lib/containers/storage/overlay/l/7NC26RIQYI3KHATYHEYQHKP2QN,upperdir=/var/lib/containers/storage/overlay/fd04cdeaee9e44eb6675b4aaf33b3e7f2bb57a3bebcf23063843e62a31c34b7a/diff,workdir=/var/lib/containers/storage/overlay/fd04cdeaee9e44eb6675b4aaf33b3e7f2bb57a3bebcf23063843e62a31c34b7a/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 2024-04-26T02:08:01.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:01.384 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-04-26T02:08:01.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:01.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link delete ceph-brx 2024-04-26T02:08:01.584 INFO:teuthology.orchestra.run.smithi006.stderr:Cannot find device "ceph-brx" 2024-04-26T02:08:01.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:08:01.588 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-26T02:08:01.600 INFO:tasks.cephadm:Teardown begin 2024-04-26T02:08:01.600 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T02:08:01.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:08:02.031 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:03.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:02 smithi006 ceph-mon[28419]: pgmap v702: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 197 KiB/s wr, 31 op/s 2024-04-26T02:08:03.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:02 smithi188 ceph-mon[32751]: pgmap v702: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 197 KiB/s wr, 31 op/s 2024-04-26T02:08:03.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:02 smithi029 ceph-mon[32775]: pgmap v702: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 197 KiB/s wr, 31 op/s 2024-04-26T02:08:04.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.058+0000 7f236d600700 1 -- 172.21.15.6:0/1609507711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2368109fa0 msgr2=0x7f2368111b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:04.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.058+0000 7f236d600700 1 --2- 172.21.15.6:0/1609507711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2368109fa0 0x7f2368111b70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f236000b0d0 tx=0x7f236000b3e0 comp rx=0 tx=0).stop 2024-04-26T02:08:04.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.058+0000 7f236d600700 1 -- 172.21.15.6:0/1609507711 shutdown_connections 2024-04-26T02:08:04.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.058+0000 7f236d600700 1 --2- 172.21.15.6:0/1609507711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2368109fa0 0x7f2368111b70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:04.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.058+0000 7f236d600700 1 --2- 172.21.15.6:0/1609507711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2368104ec0 0x7f23681052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:04.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.058+0000 7f236d600700 1 --2- 172.21.15.6:0/1609507711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368105870 0x7f23681098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:04.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.058+0000 7f236d600700 1 -- 172.21.15.6:0/1609507711 >> 172.21.15.6:0/1609507711 conn(0x7f2368100750 msgr2=0x7f2368102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:04.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.058+0000 7f236d600700 1 -- 172.21.15.6:0/1609507711 shutdown_connections 2024-04-26T02:08:04.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.058+0000 7f236d600700 1 -- 172.21.15.6:0/1609507711 wait complete. 2024-04-26T02:08:04.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.059+0000 7f236d600700 1 Processor -- start 2024-04-26T02:08:04.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.059+0000 7f236d600700 1 -- start start 2024-04-26T02:08:04.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f236d600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2368104ec0 0x7f236819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:04.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f236d600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2368105870 0x7f236819b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:04.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f236d600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368109fa0 0x7f236819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:04.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f236d600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2368114eb0 con 0x7f2368104ec0 2024-04-26T02:08:04.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f236d600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2368114d30 con 0x7f2368109fa0 2024-04-26T02:08:04.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f236d600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2368115030 con 0x7f2368105870 2024-04-26T02:08:04.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f2366a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2368104ec0 0x7f236819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:04.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f2366a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2368104ec0 0x7f236819b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37972/0 (socket says 172.21.15.6:37972) 2024-04-26T02:08:04.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f2366a00700 1 -- 172.21.15.6:0/2221217034 learned_addr learned my addr 172.21.15.6:0/2221217034 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:08:04.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f235fe00700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2368105870 0x7f236819b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:04.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f2367400700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368109fa0 0x7f236819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:04.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f235fe00700 1 -- 172.21.15.6:0/2221217034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368109fa0 msgr2=0x7f236819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:04.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f235fe00700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368109fa0 0x7f236819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:04.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f235fe00700 1 -- 172.21.15.6:0/2221217034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2368104ec0 msgr2=0x7f236819b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:04.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f235fe00700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2368104ec0 0x7f236819b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:04.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.060+0000 7f235fe00700 1 -- 172.21.15.6:0/2221217034 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23681a0300 con 0x7f2368105870 2024-04-26T02:08:04.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.061+0000 7f235fe00700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2368105870 0x7f236819b880 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f235000e9c0 tx=0x7f235000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:04.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.061+0000 7f235f400700 1 -- 172.21.15.6:0/2221217034 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235000cb70 con 0x7f2368105870 2024-04-26T02:08:04.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.061+0000 7f236d600700 1 -- 172.21.15.6:0/2221217034 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23681a05f0 con 0x7f2368105870 2024-04-26T02:08:04.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.061+0000 7f235f400700 1 -- 172.21.15.6:0/2221217034 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f235000ee60 con 0x7f2368105870 2024-04-26T02:08:04.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.061+0000 7f235f400700 1 -- 172.21.15.6:0/2221217034 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350003d60 con 0x7f2368105870 2024-04-26T02:08:04.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.061+0000 7f236d600700 1 -- 172.21.15.6:0/2221217034 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23681a7e70 con 0x7f2368105870 2024-04-26T02:08:04.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.062+0000 7f236d600700 1 -- 172.21.15.6:0/2221217034 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2368069730 con 0x7f2368105870 2024-04-26T02:08:04.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.065+0000 7f235f400700 1 -- 172.21.15.6:0/2221217034 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f235000cd10 con 0x7f2368105870 2024-04-26T02:08:04.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.066+0000 7f235f400700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f234c0a3db0 0x7f234c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:04.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.066+0000 7f235f400700 1 -- 172.21.15.6:0/2221217034 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8600+0+0 (secure 0 0 0) 0x7f23500d0a90 con 0x7f2368105870 2024-04-26T02:08:04.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.066+0000 7f2366a00700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f234c0a3db0 0x7f234c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:04.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.067+0000 7f2366a00700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f234c0a3db0 0x7f234c0a6270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2358009d60 tx=0x7f2358009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:04.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.068+0000 7f235f400700 1 -- 172.21.15.6:0/2221217034 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f235009d080 con 0x7f2368105870 2024-04-26T02:08:04.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.355+0000 7f236d600700 1 -- 172.21.15.6:0/2221217034 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f236819c650 con 0x7f2368105870 2024-04-26T02:08:04.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.357+0000 7f235f400700 1 -- 172.21.15.6:0/2221217034 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+11850 (secure 0 0 0) 0x7f235009c980 con 0x7f2368105870 2024-04-26T02:08:04.357 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:08:04.357 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":59,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T02:06:53.954620+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27562":{"gid":27562,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1973934809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1973934809},{"type":"v1","addr":"172.21.15.188:6835","nonce":1973934809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24505,24512]},"id":1}]} 2024-04-26T02:08:04.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.360+0000 7f236d600700 1 -- 172.21.15.6:0/2221217034 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f234c0a3db0 msgr2=0x7f234c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:04.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.360+0000 7f236d600700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f234c0a3db0 0x7f234c0a6270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2358009d60 tx=0x7f2358009260 comp rx=0 tx=0).stop 2024-04-26T02:08:04.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.360+0000 7f236d600700 1 -- 172.21.15.6:0/2221217034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2368105870 msgr2=0x7f236819b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:04.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.360+0000 7f236d600700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2368105870 0x7f236819b880 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f235000e9c0 tx=0x7f235000ecd0 comp rx=0 tx=0).stop 2024-04-26T02:08:04.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.360+0000 7f236d600700 1 -- 172.21.15.6:0/2221217034 shutdown_connections 2024-04-26T02:08:04.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.360+0000 7f236d600700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f234c0a3db0 0x7f234c0a6270 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:04.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.360+0000 7f236d600700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368109fa0 0x7f236819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:04.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.360+0000 7f236d600700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2368104ec0 0x7f236819b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:04.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.360+0000 7f236d600700 1 --2- 172.21.15.6:0/2221217034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2368105870 0x7f236819b880 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:04.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.360+0000 7f236d600700 1 -- 172.21.15.6:0/2221217034 >> 172.21.15.6:0/2221217034 conn(0x7f2368100750 msgr2=0x7f23681012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:04.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.360+0000 7f236d600700 1 -- 172.21.15.6:0/2221217034 shutdown_connections 2024-04-26T02:08:04.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:04.361+0000 7f236d600700 1 -- 172.21.15.6:0/2221217034 wait complete. 2024-04-26T02:08:04.362 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 59 2024-04-26T02:08:05.041 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:04 smithi006 ceph-mon[28419]: pgmap v703: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 163 KiB/s wr, 25 op/s 2024-04-26T02:08:05.041 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:04 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2221217034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:08:05.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:08:05.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:04 smithi188 ceph-mon[32751]: pgmap v703: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 163 KiB/s wr, 25 op/s 2024-04-26T02:08:05.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:04 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2221217034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:08:05.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:04 smithi029 ceph-mon[32775]: pgmap v703: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 163 KiB/s wr, 25 op/s 2024-04-26T02:08:05.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:04 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2221217034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:08:05.775 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:06.115 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:05 smithi006 ceph-mon[28419]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:08:06.115 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:05 smithi006 ceph-mon[28419]: osdmap e133: 12 total, 12 up, 12 in 2024-04-26T02:08:06.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:05 smithi188 ceph-mon[32751]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:08:06.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:05 smithi188 ceph-mon[32751]: osdmap e133: 12 total, 12 up, 12 in 2024-04-26T02:08:06.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:05 smithi029 ceph-mon[32775]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:08:06.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:05 smithi029 ceph-mon[32775]: osdmap e133: 12 total, 12 up, 12 in 2024-04-26T02:08:07.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:06 smithi188 ceph-mon[32751]: pgmap v705: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 196 KiB/s wr, 30 op/s 2024-04-26T02:08:07.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:06 smithi029 ceph-mon[32775]: pgmap v705: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 196 KiB/s wr, 30 op/s 2024-04-26T02:08:07.231 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:06 smithi006 ceph-mon[28419]: pgmap v705: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 196 KiB/s wr, 30 op/s 2024-04-26T02:08:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.062+0000 7f5e80a00700 1 -- 172.21.15.6:0/2005461933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e7c109fa0 msgr2=0x7f5e7c111b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.063+0000 7f5e80a00700 1 --2- 172.21.15.6:0/2005461933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e7c109fa0 0x7f5e7c111b70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009990 tx=0x7f5e74009ca0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.063+0000 7f5e80a00700 1 -- 172.21.15.6:0/2005461933 shutdown_connections 2024-04-26T02:08:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.063+0000 7f5e80a00700 1 --2- 172.21.15.6:0/2005461933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e7c109fa0 0x7f5e7c111b70 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.063+0000 7f5e80a00700 1 --2- 172.21.15.6:0/2005461933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e7c104ec0 0x7f5e7c1052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.063+0000 7f5e80a00700 1 --2- 172.21.15.6:0/2005461933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c105870 0x7f5e7c1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.063+0000 7f5e80a00700 1 -- 172.21.15.6:0/2005461933 >> 172.21.15.6:0/2005461933 conn(0x7f5e7c100750 msgr2=0x7f5e7c102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.063+0000 7f5e80a00700 1 -- 172.21.15.6:0/2005461933 shutdown_connections 2024-04-26T02:08:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.063+0000 7f5e80a00700 1 -- 172.21.15.6:0/2005461933 wait complete. 2024-04-26T02:08:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.064+0000 7f5e80a00700 1 Processor -- start 2024-04-26T02:08:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.065+0000 7f5e80a00700 1 -- start start 2024-04-26T02:08:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.065+0000 7f5e80a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e7c104ec0 0x7f5e7c19b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.065+0000 7f5e80a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c105870 0x7f5e7c19b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.065+0000 7f5e80a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e7c109fa0 0x7f5e7c19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.065+0000 7f5e80a00700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c114e50 con 0x7f5e7c109fa0 2024-04-26T02:08:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.065+0000 7f5e80a00700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c114cd0 con 0x7f5e7c104ec0 2024-04-26T02:08:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.065+0000 7f5e80a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c114fd0 con 0x7f5e7c105870 2024-04-26T02:08:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.065+0000 7f5e79e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e7c104ec0 0x7f5e7c19b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.065+0000 7f5e7a800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e7c109fa0 0x7f5e7c19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.065+0000 7f5e79e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e7c104ec0 0x7f5e7c19b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59094/0 (socket says 172.21.15.6:59094) 2024-04-26T02:08:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.065+0000 7f5e79e00700 1 -- 172.21.15.6:0/1123409897 learned_addr learned my addr 172.21.15.6:0/1123409897 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:08:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.066+0000 7f5e79400700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c105870 0x7f5e7c19b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.066+0000 7f5e79e00700 1 -- 172.21.15.6:0/1123409897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c105870 msgr2=0x7f5e7c19b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.066+0000 7f5e79e00700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c105870 0x7f5e7c19b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.066+0000 7f5e79e00700 1 -- 172.21.15.6:0/1123409897 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e7c109fa0 msgr2=0x7f5e7c19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.066+0000 7f5e79e00700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e7c109fa0 0x7f5e7c19fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.066+0000 7f5e79e00700 1 -- 172.21.15.6:0/1123409897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e7c1a0340 con 0x7f5e7c104ec0 2024-04-26T02:08:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.066+0000 7f5e79e00700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e7c104ec0 0x7f5e7c19b380 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00ef10 tx=0x7f5e6c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.066+0000 7f5e72a00700 1 -- 172.21.15.6:0/1123409897 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c00c9c0 con 0x7f5e7c104ec0 2024-04-26T02:08:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.067+0000 7f5e72a00700 1 -- 172.21.15.6:0/1123409897 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e6c019070 con 0x7f5e7c104ec0 2024-04-26T02:08:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.067+0000 7f5e80a00700 1 -- 172.21.15.6:0/1123409897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e7c1a05d0 con 0x7f5e7c104ec0 2024-04-26T02:08:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.067+0000 7f5e72a00700 1 -- 172.21.15.6:0/1123409897 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c014480 con 0x7f5e7c104ec0 2024-04-26T02:08:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.067+0000 7f5e80a00700 1 -- 172.21.15.6:0/1123409897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e7c1a7eb0 con 0x7f5e7c104ec0 2024-04-26T02:08:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.068+0000 7f5e80a00700 1 -- 172.21.15.6:0/1123409897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e7c069730 con 0x7f5e7c104ec0 2024-04-26T02:08:08.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.070+0000 7f5e72a00700 1 -- 172.21.15.6:0/1123409897 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f5e6c012050 con 0x7f5e7c104ec0 2024-04-26T02:08:08.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.071+0000 7f5e72a00700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5e600a3e30 0x7f5e600a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:08.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.071+0000 7f5e72a00700 1 -- 172.21.15.6:0/1123409897 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 8644+0+0 (secure 0 0 0) 0x7f5e6c0d0e70 con 0x7f5e7c104ec0 2024-04-26T02:08:08.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.072+0000 7f5e79400700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5e600a3e30 0x7f5e600a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:08.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.072+0000 7f5e79400700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5e600a3e30 0x7f5e600a62f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c10b5a0 tx=0x7f5e68009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:08.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.074+0000 7f5e72a00700 1 -- 172.21.15.6:0/1123409897 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5e6c010040 con 0x7f5e7c104ec0 2024-04-26T02:08:08.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:07 smithi188 ceph-mon[32751]: pgmap v706: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 273 KiB/s wr, 37 op/s 2024-04-26T02:08:08.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:07 smithi029 ceph-mon[32775]: pgmap v706: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 273 KiB/s wr, 37 op/s 2024-04-26T02:08:08.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:07 smithi006 ceph-mon[28419]: pgmap v706: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 273 KiB/s wr, 37 op/s 2024-04-26T02:08:08.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.358+0000 7f5e80a00700 1 -- 172.21.15.6:0/1123409897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e7c19c690 con 0x7f5e7c104ec0 2024-04-26T02:08:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.359+0000 7f5e72a00700 1 -- 172.21.15.6:0/1123409897 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+11851 (secure 0 0 0) 0x7f5e6c09d430 con 0x7f5e7c104ec0 2024-04-26T02:08:08.359 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:08:08.359 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3962248183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3962248183},{"type":"v1","addr":"172.21.15.6:6835","nonce":3962248183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6841/1062358442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6840","nonce":1062358442},{"type":"v1","addr":"172.21.15.6:6841","nonce":1062358442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/3500196993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3500196993},{"type":"v1","addr":"172.21.15.29:6833","nonce":3500196993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3510817210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3510817210},{"type":"v1","addr":"172.21.15.188:6833","nonce":3510817210}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2373773979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2373773979},{"type":"v1","addr":"172.21.15.29:6835","nonce":2373773979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/680270961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":680270961},{"type":"v1","addr":"172.21.15.6:6837","nonce":680270961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":60,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:42:08.299563+0000","modified":"2024-04-26T02:08:04.562501+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24523,"mds_1":24512,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6837/402171153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":402171153},{"type":"v1","addr":"172.21.15.29:6837","nonce":402171153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.6:6839/3936238112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6838","nonce":3936238112},{"type":"v1","addr":"172.21.15.6:6839","nonce":3936238112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/489767141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":489767141},{"type":"v1","addr":"172.21.15.188:6837","nonce":489767141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.29:6839/2325582359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":2325582359},{"type":"v1","addr":"172.21.15.29:6839","nonce":2325582359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.188:6839/674623421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":674623421},{"type":"v1","addr":"172.21.15.188:6839","nonce":674623421}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27562":{"gid":27562,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1973934809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1973934809},{"type":"v1","addr":"172.21.15.188:6835","nonce":1973934809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24523,"qdb_cluster":[24523,24512,24505]},"id":1}]} 2024-04-26T02:08:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.362+0000 7f5e80a00700 1 -- 172.21.15.6:0/1123409897 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5e600a3e30 msgr2=0x7f5e600a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.362+0000 7f5e80a00700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5e600a3e30 0x7f5e600a62f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c10b5a0 tx=0x7f5e68009260 comp rx=0 tx=0).stop 2024-04-26T02:08:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.362+0000 7f5e80a00700 1 -- 172.21.15.6:0/1123409897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e7c104ec0 msgr2=0x7f5e7c19b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.362+0000 7f5e80a00700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e7c104ec0 0x7f5e7c19b380 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00ef10 tx=0x7f5e6c00c3a0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.362+0000 7f5e80a00700 1 -- 172.21.15.6:0/1123409897 shutdown_connections 2024-04-26T02:08:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.362+0000 7f5e80a00700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f5e600a3e30 0x7f5e600a62f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.362+0000 7f5e80a00700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e7c109fa0 0x7f5e7c19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.362+0000 7f5e80a00700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e7c104ec0 0x7f5e7c19b380 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.362+0000 7f5e80a00700 1 --2- 172.21.15.6:0/1123409897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c105870 0x7f5e7c19b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.362+0000 7f5e80a00700 1 -- 172.21.15.6:0/1123409897 >> 172.21.15.6:0/1123409897 conn(0x7f5e7c100750 msgr2=0x7f5e7c102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.362+0000 7f5e80a00700 1 -- 172.21.15.6:0/1123409897 shutdown_connections 2024-04-26T02:08:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:08.363+0000 7f5e80a00700 1 -- 172.21.15.6:0/1123409897 wait complete. 2024-04-26T02:08:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 60 2024-04-26T02:08:09.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:08 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1123409897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:08:09.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:08 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1123409897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:08:09.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:08 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1123409897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:08:09.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T02:08:09.677 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:10.093 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:09 smithi029 ceph-mon[32775]: pgmap v707: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 273 KiB/s wr, 37 op/s 2024-04-26T02:08:10.128 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:09 smithi006 ceph-mon[28419]: pgmap v707: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 273 KiB/s wr, 37 op/s 2024-04-26T02:08:10.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:09 smithi188 ceph-mon[32751]: pgmap v707: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 273 KiB/s wr, 37 op/s 2024-04-26T02:08:11.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.789+0000 7f90c2600700 1 -- 172.21.15.6:0/74082395 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc102a90 msgr2=0x7f90bc102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:11.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.789+0000 7f90c2600700 1 --2- 172.21.15.6:0/74082395 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc102a90 0x7f90bc102f10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f90a4009990 tx=0x7f90a4009ca0 comp rx=0 tx=0).stop 2024-04-26T02:08:11.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.790+0000 7f90c2600700 1 -- 172.21.15.6:0/74082395 shutdown_connections 2024-04-26T02:08:11.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.790+0000 7f90c2600700 1 --2- 172.21.15.6:0/74082395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90bc103450 0x7f90bc10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:11.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.790+0000 7f90c2600700 1 --2- 172.21.15.6:0/74082395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc108a80 0x7f90bc108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:11.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.790+0000 7f90c2600700 1 --2- 172.21.15.6:0/74082395 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc102a90 0x7f90bc102f10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:11.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.790+0000 7f90c2600700 1 -- 172.21.15.6:0/74082395 >> 172.21.15.6:0/74082395 conn(0x7f90bc079bf0 msgr2=0x7f90bc07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:11.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.790+0000 7f90c2600700 1 -- 172.21.15.6:0/74082395 shutdown_connections 2024-04-26T02:08:11.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.790+0000 7f90c2600700 1 -- 172.21.15.6:0/74082395 wait complete. 2024-04-26T02:08:11.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.791+0000 7f90c2600700 1 Processor -- start 2024-04-26T02:08:11.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.791+0000 7f90c2600700 1 -- start start 2024-04-26T02:08:11.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.792+0000 7f90c2600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90bc102a90 0x7f90bc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:11.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.792+0000 7f90c2600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc103450 0x7f90bc075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:11.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.792+0000 7f90c2600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc108a80 0x7f90bc0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:11.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.792+0000 7f90c2600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f90bc112c30 con 0x7f90bc103450 2024-04-26T02:08:11.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.792+0000 7f90c2600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f90bc112ab0 con 0x7f90bc102a90 2024-04-26T02:08:11.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.792+0000 7f90c2600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f90bc112db0 con 0x7f90bc108a80 2024-04-26T02:08:11.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.792+0000 7f90b2a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc103450 0x7f90bc075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:11.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.792+0000 7f90bbe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc108a80 0x7f90bc0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:11.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.792+0000 7f90bb400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90bc102a90 0x7f90bc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:11.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.792+0000 7f90b2a00700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc103450 0x7f90bc075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56422/0 (socket says 172.21.15.6:56422) 2024-04-26T02:08:11.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.792+0000 7f90b2a00700 1 -- 172.21.15.6:0/782715300 learned_addr learned my addr 172.21.15.6:0/782715300 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:08:11.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.793+0000 7f90bbe00700 1 -- 172.21.15.6:0/782715300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90bc102a90 msgr2=0x7f90bc074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:11.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.793+0000 7f90bbe00700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90bc102a90 0x7f90bc074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:11.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.793+0000 7f90bbe00700 1 -- 172.21.15.6:0/782715300 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc103450 msgr2=0x7f90bc075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:11.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.793+0000 7f90bbe00700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc103450 0x7f90bc075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:11.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.793+0000 7f90bbe00700 1 -- 172.21.15.6:0/782715300 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90bc1a76f0 con 0x7f90bc108a80 2024-04-26T02:08:11.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.793+0000 7f90b2a00700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc103450 0x7f90bc075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:08:11.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.793+0000 7f90bbe00700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc108a80 0x7f90bc0757c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f90b400efc0 tx=0x7f90b400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:11.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.793+0000 7f90b8c00700 1 -- 172.21.15.6:0/782715300 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b40092e0 con 0x7f90bc108a80 2024-04-26T02:08:11.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.793+0000 7f90b8c00700 1 -- 172.21.15.6:0/782715300 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90b4004bb0 con 0x7f90bc108a80 2024-04-26T02:08:11.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.793+0000 7f90c2600700 1 -- 172.21.15.6:0/782715300 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90bc1a7980 con 0x7f90bc108a80 2024-04-26T02:08:11.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.793+0000 7f90b8c00700 1 -- 172.21.15.6:0/782715300 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b4004020 con 0x7f90bc108a80 2024-04-26T02:08:11.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.793+0000 7f90c2600700 1 -- 172.21.15.6:0/782715300 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90bc1a7ec0 con 0x7f90bc108a80 2024-04-26T02:08:11.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.794+0000 7f90c2600700 1 -- 172.21.15.6:0/782715300 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90bc069730 con 0x7f90bc108a80 2024-04-26T02:08:11.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.796+0000 7f90b8c00700 1 -- 172.21.15.6:0/782715300 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f90b401b030 con 0x7f90bc108a80 2024-04-26T02:08:11.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.798+0000 7f90b8c00700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f90a00a3e30 0x7f90a00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:11.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.798+0000 7f90b8c00700 1 -- 172.21.15.6:0/782715300 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 8644+0+0 (secure 0 0 0) 0x7f90b40d0300 con 0x7f90bc108a80 2024-04-26T02:08:11.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.798+0000 7f90bb400700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f90a00a3e30 0x7f90a00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:11.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.799+0000 7f90bb400700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f90a00a3e30 0x7f90a00a62f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f90ac009be0 tx=0x7f90ac009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:11.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:11.803+0000 7f90b8c00700 1 -- 172.21.15.6:0/782715300 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f90b4022020 con 0x7f90bc108a80 2024-04-26T02:08:12.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.037+0000 7f90c2600700 1 -- 172.21.15.6:0/782715300 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90bc02df80 con 0x7f90bc108a80 2024-04-26T02:08:12.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.039+0000 7f90b8c00700 1 -- 172.21.15.6:0/782715300 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) ==== 74+0+23214 (secure 0 0 0) 0x7f90b4010040 con 0x7f90bc108a80 2024-04-26T02:08:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-26T02:08:12.040 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":133,"fsid":"a5b72bee-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:32:29.071544+0000","modified":"2024-04-26T02:08:04.789258+0000","last_up_change":"2024-04-26T01:40:49.915924+0000","last_in_change":"2024-04-26T01:40:29.559647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:36:35.182726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:41:59.612167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:42:04.106467+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":40,"snap_epoch":133,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f164c51d-ce5d-4140-b9e3-a4b84088f5ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6803","nonce":76518486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6805","nonce":76518486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6809","nonce":76518486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":76518486},{"type":"v1","addr":"172.21.15.6:6807","nonce":76518486}]},"public_addr":"172.21.15.6:6803/76518486","cluster_addr":"172.21.15.6:6805/76518486","heartbeat_back_addr":"172.21.15.6:6809/76518486","heartbeat_front_addr":"172.21.15.6:6807/76518486","state":["exists","up"]},{"osd":1,"uuid":"57e875dd-ddf1-4ee4-a2a7-9589d7f42818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6801","nonce":400769116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6803","nonce":400769116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6807","nonce":400769116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":400769116},{"type":"v1","addr":"172.21.15.29:6805","nonce":400769116}]},"public_addr":"172.21.15.29:6801/400769116","cluster_addr":"172.21.15.29:6803/400769116","heartbeat_back_addr":"172.21.15.29:6807/400769116","heartbeat_front_addr":"172.21.15.29:6805/400769116","state":["exists","up"]},{"osd":2,"uuid":"77e0c294-c48a-4318-938b-4604a533a2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6801","nonce":3508067672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6803","nonce":3508067672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6807","nonce":3508067672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":3508067672},{"type":"v1","addr":"172.21.15.188:6805","nonce":3508067672}]},"public_addr":"172.21.15.188:6801/3508067672","cluster_addr":"172.21.15.188:6803/3508067672","heartbeat_back_addr":"172.21.15.188:6807/3508067672","heartbeat_front_addr":"172.21.15.188:6805/3508067672","state":["exists","up"]},{"osd":3,"uuid":"6393dac5-996c-47af-9687-0f1393ecc483","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6811","nonce":1987311118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6813","nonce":1987311118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6817","nonce":1987311118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1987311118},{"type":"v1","addr":"172.21.15.6:6815","nonce":1987311118}]},"public_addr":"172.21.15.6:6811/1987311118","cluster_addr":"172.21.15.6:6813/1987311118","heartbeat_back_addr":"172.21.15.6:6817/1987311118","heartbeat_front_addr":"172.21.15.6:6815/1987311118","state":["exists","up"]},{"osd":4,"uuid":"f2c1a535-26a0-46c6-a6d0-c2529c786739","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6809","nonce":2125876609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6811","nonce":2125876609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6815","nonce":2125876609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2125876609},{"type":"v1","addr":"172.21.15.29:6813","nonce":2125876609}]},"public_addr":"172.21.15.29:6809/2125876609","cluster_addr":"172.21.15.29:6811/2125876609","heartbeat_back_addr":"172.21.15.29:6815/2125876609","heartbeat_front_addr":"172.21.15.29:6813/2125876609","state":["exists","up"]},{"osd":5,"uuid":"9788bb0a-0d79-4743-bce6-80f5342e8f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6809","nonce":2092077007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6811","nonce":2092077007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6815","nonce":2092077007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2092077007},{"type":"v1","addr":"172.21.15.188:6813","nonce":2092077007}]},"public_addr":"172.21.15.188:6809/2092077007","cluster_addr":"172.21.15.188:6811/2092077007","heartbeat_back_addr":"172.21.15.188:6815/2092077007","heartbeat_front_addr":"172.21.15.188:6813/2092077007","state":["exists","up"]},{"osd":6,"uuid":"fd8584e1-69ee-4c30-85e4-a637357d1b6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6819","nonce":3256016695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6821","nonce":3256016695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6825","nonce":3256016695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3256016695},{"type":"v1","addr":"172.21.15.6:6823","nonce":3256016695}]},"public_addr":"172.21.15.6:6819/3256016695","cluster_addr":"172.21.15.6:6821/3256016695","heartbeat_back_addr":"172.21.15.6:6825/3256016695","heartbeat_front_addr":"172.21.15.6:6823/3256016695","state":["exists","up"]},{"osd":7,"uuid":"6b7e6784-d574-42da-8416-a74a3b6b3755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6817","nonce":3478594037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6819","nonce":3478594037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6823","nonce":3478594037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3478594037},{"type":"v1","addr":"172.21.15.29:6821","nonce":3478594037}]},"public_addr":"172.21.15.29:6817/3478594037","cluster_addr":"172.21.15.29:6819/3478594037","heartbeat_back_addr":"172.21.15.29:6823/3478594037","heartbeat_front_addr":"172.21.15.29:6821/3478594037","state":["exists","up"]},{"osd":8,"uuid":"30cd2ea1-2767-4062-8bd9-47214e4b4b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6817","nonce":3290364376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6819","nonce":3290364376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6823","nonce":3290364376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3290364376},{"type":"v1","addr":"172.21.15.188:6821","nonce":3290364376}]},"public_addr":"172.21.15.188:6817/3290364376","cluster_addr":"172.21.15.188:6819/3290364376","heartbeat_back_addr":"172.21.15.188:6823/3290364376","heartbeat_front_addr":"172.21.15.188:6821/3290364376","state":["exists","up"]},{"osd":9,"uuid":"f033cdc0-1178-4a7f-a338-69ff32416087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6827","nonce":1918893912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6829","nonce":1918893912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6833","nonce":1918893912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1918893912},{"type":"v1","addr":"172.21.15.6:6831","nonce":1918893912}]},"public_addr":"172.21.15.6:6827/1918893912","cluster_addr":"172.21.15.6:6829/1918893912","heartbeat_back_addr":"172.21.15.6:6833/1918893912","heartbeat_front_addr":"172.21.15.6:6831/1918893912","state":["exists","up"]},{"osd":10,"uuid":"04e9b5ef-ac0c-482b-a69e-a7121b36e7da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6825","nonce":1395777729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6827","nonce":1395777729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6831","nonce":1395777729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1395777729},{"type":"v1","addr":"172.21.15.29:6829","nonce":1395777729}]},"public_addr":"172.21.15.29:6825/1395777729","cluster_addr":"172.21.15.29:6827/1395777729","heartbeat_back_addr":"172.21.15.29:6831/1395777729","heartbeat_front_addr":"172.21.15.29:6829/1395777729","state":["exists","up"]},{"osd":11,"uuid":"cb1638ce-4189-4241-b63e-b63c607e90ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6825","nonce":3322197880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6827","nonce":3322197880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6831","nonce":3322197880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":3322197880},{"type":"v1","addr":"172.21.15.188:6829","nonce":3322197880}]},"public_addr":"172.21.15.188:6825/3322197880","cluster_addr":"172.21.15.188:6827/3322197880","heartbeat_back_addr":"172.21.15.188:6831/3322197880","heartbeat_front_addr":"172.21.15.188:6829/3322197880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:41.553455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:07.641467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:31.748891+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:03.743553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:30.065262+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:54.976527+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:29.348206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:56.285928+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:21.246409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:54.468561+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:22.093015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:40:47.342379+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.19","mappings":[{"from":3,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6801/809793211":"2024-04-27T01:45:54.919747+0000","172.21.15.6:6800/809793211":"2024-04-27T01:45:54.919747+0000","172.21.15.6:0/286280546":"2024-04-27T01:45:54.919747+0000","172.21.15.6:0/402969404":"2024-04-27T01:45:54.919747+0000","172.21.15.6:0/2707582744":"2024-04-27T01:45:54.919747+0000","172.21.15.6:0/3143272062":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1325707596":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6800/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:6801/635066163":"2024-04-27T01:33:47.021882+0000","172.21.15.6:0/1359518071":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/1823104535":"2024-04-27T01:45:54.919747+0000","172.21.15.6:6801/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/2813598711":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/3140150861":"2024-04-27T01:33:11.534520+0000","172.21.15.6:6800/801052348":"2024-04-27T01:33:11.534520+0000","172.21.15.6:0/65701414":"2024-04-27T01:33:47.021882+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":28,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":28,"length":1},{"begin":40,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:08:12.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.042+0000 7f90c2600700 1 -- 172.21.15.6:0/782715300 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f90a00a3e30 msgr2=0x7f90a00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:12.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.042+0000 7f90c2600700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f90a00a3e30 0x7f90a00a62f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f90ac009be0 tx=0x7f90ac009260 comp rx=0 tx=0).stop 2024-04-26T02:08:12.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.042+0000 7f90c2600700 1 -- 172.21.15.6:0/782715300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc108a80 msgr2=0x7f90bc0757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:12.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.042+0000 7f90c2600700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc108a80 0x7f90bc0757c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f90b400efc0 tx=0x7f90b400c3a0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.043+0000 7f90c2600700 1 -- 172.21.15.6:0/782715300 shutdown_connections 2024-04-26T02:08:12.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.043+0000 7f90c2600700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f90a00a3e30 0x7f90a00a62f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.043+0000 7f90c2600700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc108a80 0x7f90bc0757c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.043+0000 7f90c2600700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90bc102a90 0x7f90bc074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.043+0000 7f90c2600700 1 --2- 172.21.15.6:0/782715300 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc103450 0x7f90bc075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.043+0000 7f90c2600700 1 -- 172.21.15.6:0/782715300 >> 172.21.15.6:0/782715300 conn(0x7f90bc079bf0 msgr2=0x7f90bc1017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:12.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.043+0000 7f90c2600700 1 -- 172.21.15.6:0/782715300 shutdown_connections 2024-04-26T02:08:12.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:12.043+0000 7f90c2600700 1 -- 172.21.15.6:0/782715300 wait complete. 2024-04-26T02:08:12.771 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:12 smithi006 ceph-mon[28419]: pgmap v708: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 153 KiB/s wr, 17 op/s 2024-04-26T02:08:12.771 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:12 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/782715300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T02:08:12.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs fail cephfs 2024-04-26T02:08:12.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:12 smithi188 ceph-mon[32751]: pgmap v708: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 153 KiB/s wr, 17 op/s 2024-04-26T02:08:12.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:12 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/782715300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T02:08:12.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:12 smithi029 ceph-mon[32775]: pgmap v708: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 153 KiB/s wr, 17 op/s 2024-04-26T02:08:12.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:12 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/782715300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T02:08:13.296 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:14.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:14 smithi006 ceph-mon[28419]: pgmap v709: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 153 KiB/s wr, 17 op/s 2024-04-26T02:08:14.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:14 smithi188 ceph-mon[32751]: pgmap v709: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 153 KiB/s wr, 17 op/s 2024-04-26T02:08:14.941 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:14 smithi029 ceph-mon[32775]: pgmap v709: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 153 KiB/s wr, 17 op/s 2024-04-26T02:08:15.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.354+0000 7fc6de600700 1 -- 172.21.15.6:0/2841642552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d8105860 msgr2=0x7fc6d81098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:15.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.354+0000 7fc6de600700 1 --2- 172.21.15.6:0/2841642552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d8105860 0x7fc6d81098b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40099c0 tx=0x7fc6c4009cd0 comp rx=0 tx=0).stop 2024-04-26T02:08:15.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.354+0000 7fc6de600700 1 -- 172.21.15.6:0/2841642552 shutdown_connections 2024-04-26T02:08:15.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.354+0000 7fc6de600700 1 --2- 172.21.15.6:0/2841642552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d8109f90 0x7fc6d8111b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:15.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.354+0000 7fc6de600700 1 --2- 172.21.15.6:0/2841642552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d8104eb0 0x7fc6d8105290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:15.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.354+0000 7fc6de600700 1 --2- 172.21.15.6:0/2841642552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d8105860 0x7fc6d81098b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:15.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.354+0000 7fc6de600700 1 -- 172.21.15.6:0/2841642552 >> 172.21.15.6:0/2841642552 conn(0x7fc6d8100740 msgr2=0x7fc6d8102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:15.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.354+0000 7fc6de600700 1 -- 172.21.15.6:0/2841642552 shutdown_connections 2024-04-26T02:08:15.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.354+0000 7fc6de600700 1 -- 172.21.15.6:0/2841642552 wait complete. 2024-04-26T02:08:15.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.355+0000 7fc6de600700 1 Processor -- start 2024-04-26T02:08:15.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.355+0000 7fc6de600700 1 -- start start 2024-04-26T02:08:15.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.356+0000 7fc6de600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d8104eb0 0x7fc6d819b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:15.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.356+0000 7fc6de600700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d8105860 0x7fc6d819b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:15.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.356+0000 7fc6de600700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d8109f90 0x7fc6d819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:15.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.356+0000 7fc6de600700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d8114e50 con 0x7fc6d8109f90 2024-04-26T02:08:15.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.356+0000 7fc6de600700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d8114cd0 con 0x7fc6d8105860 2024-04-26T02:08:15.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.356+0000 7fc6de600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d8114fd0 con 0x7fc6d8104eb0 2024-04-26T02:08:15.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.356+0000 7fc6d6a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d8105860 0x7fc6d819b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:15.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.356+0000 7fc6d6a00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d8105860 0x7fc6d819b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:59138/0 (socket says 172.21.15.6:59138) 2024-04-26T02:08:15.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.356+0000 7fc6d6a00700 1 -- 172.21.15.6:0/2243379903 learned_addr learned my addr 172.21.15.6:0/2243379903 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:08:15.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.356+0000 7fc6d7400700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d8104eb0 0x7fc6d819b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:15.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.356+0000 7fc6d7e00700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d8109f90 0x7fc6d819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:15.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.357+0000 7fc6d6a00700 1 -- 172.21.15.6:0/2243379903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d8104eb0 msgr2=0x7fc6d819b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:15.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.357+0000 7fc6d6a00700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d8104eb0 0x7fc6d819b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:15.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.357+0000 7fc6d6a00700 1 -- 172.21.15.6:0/2243379903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d8109f90 msgr2=0x7fc6d819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:15.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.357+0000 7fc6d6a00700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d8109f90 0x7fc6d819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:15.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.357+0000 7fc6d6a00700 1 -- 172.21.15.6:0/2243379903 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d81a0390 con 0x7fc6d8105860 2024-04-26T02:08:15.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.357+0000 7fc6d7400700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d8104eb0 0x7fc6d819b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:08:15.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.357+0000 7fc6d6a00700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d8105860 0x7fc6d819b8c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4005f50 tx=0x7fc6c4005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:15.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.358+0000 7fc6cfe00700 1 -- 172.21.15.6:0/2243379903 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4026070 con 0x7fc6d8105860 2024-04-26T02:08:15.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.358+0000 7fc6de600700 1 -- 172.21.15.6:0/2243379903 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d81a0620 con 0x7fc6d8105860 2024-04-26T02:08:15.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.358+0000 7fc6cfe00700 1 -- 172.21.15.6:0/2243379903 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6c4003c70 con 0x7fc6d8105860 2024-04-26T02:08:15.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.358+0000 7fc6de600700 1 -- 172.21.15.6:0/2243379903 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d81a7f00 con 0x7fc6d8105860 2024-04-26T02:08:15.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.358+0000 7fc6cfe00700 1 -- 172.21.15.6:0/2243379903 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c400f960 con 0x7fc6d8105860 2024-04-26T02:08:15.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.359+0000 7fc6de600700 1 -- 172.21.15.6:0/2243379903 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d8069730 con 0x7fc6d8105860 2024-04-26T02:08:15.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.361+0000 7fc6cfe00700 1 -- 172.21.15.6:0/2243379903 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fc6c4029070 con 0x7fc6d8105860 2024-04-26T02:08:15.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.363+0000 7fc6cfe00700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc6bc0a3e30 0x7fc6bc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:15.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.363+0000 7fc6cfe00700 1 -- 172.21.15.6:0/2243379903 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 8644+0+0 (secure 0 0 0) 0x7fc6c40d1400 con 0x7fc6d8105860 2024-04-26T02:08:15.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.363+0000 7fc6d7400700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc6bc0a3e30 0x7fc6bc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:15.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.364+0000 7fc6d7400700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc6bc0a3e30 0x7fc6bc0a62f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8011490 tx=0x7fc6c8009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:15.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.365+0000 7fc6cfe00700 1 -- 172.21.15.6:0/2243379903 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc6c4013070 con 0x7fc6d8105860 2024-04-26T02:08:15.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.652+0000 7fc6de600700 1 -- 172.21.15.6:0/2243379903 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc6d802df80 con 0x7fc6d8105860 2024-04-26T02:08:15.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.661+0000 7fc6cfe00700 1 -- 172.21.15.6:0/2243379903 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v61) ==== 161+0+0 (secure 0 0 0) 0x7fc6c409da50 con 0x7fc6d8105860 2024-04-26T02:08:15.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.666+0000 7fc6de600700 1 -- 172.21.15.6:0/2243379903 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc6bc0a3e30 msgr2=0x7fc6bc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:15.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.666+0000 7fc6de600700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc6bc0a3e30 0x7fc6bc0a62f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8011490 tx=0x7fc6c8009260 comp rx=0 tx=0).stop 2024-04-26T02:08:15.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.666+0000 7fc6de600700 1 -- 172.21.15.6:0/2243379903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d8105860 msgr2=0x7fc6d819b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:15.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.666+0000 7fc6de600700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d8105860 0x7fc6d819b8c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4005f50 tx=0x7fc6c4005280 comp rx=0 tx=0).stop 2024-04-26T02:08:15.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.668+0000 7fc6de600700 1 -- 172.21.15.6:0/2243379903 shutdown_connections 2024-04-26T02:08:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.668+0000 7fc6de600700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fc6bc0a3e30 0x7fc6bc0a62f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.668+0000 7fc6de600700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d8109f90 0x7fc6d819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.668+0000 7fc6de600700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d8104eb0 0x7fc6d819b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.668+0000 7fc6de600700 1 --2- 172.21.15.6:0/2243379903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6d8105860 0x7fc6d819b8c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.668+0000 7fc6de600700 1 -- 172.21.15.6:0/2243379903 >> 172.21.15.6:0/2243379903 conn(0x7fc6d8100740 msgr2=0x7fc6d8102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.669+0000 7fc6de600700 1 -- 172.21.15.6:0/2243379903 shutdown_connections 2024-04-26T02:08:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:15.669+0000 7fc6de600700 1 -- 172.21.15.6:0/2243379903 wait complete. 2024-04-26T02:08:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T02:08:16.048 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:15 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a[28415]: 2024-04-26T02:08:15.654+0000 7f63da600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T02:08:16.048 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:08:15 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g[70436]: ignoring --setuser ceph since I am not root 2024-04-26T02:08:16.049 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:08:15 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g[70436]: ignoring --setgroup ceph since I am not root 2024-04-26T02:08:16.049 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:08:15 smithi006 ceph-mds[70440]: -- 172.21.15.6:0/2304331309 <== mon.2 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560af3159a40 con 0x560af312c800 2024-04-26T02:08:16.049 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:08:15 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g[70436]: starting mds.g at 2024-04-26T02:08:16.141 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 02:08:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-c[63920]: ignoring --setuser ceph since I am not root 2024-04-26T02:08:16.141 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 02:08:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-c[63920]: ignoring --setgroup ceph since I am not root 2024-04-26T02:08:16.141 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 02:08:15 smithi188 ceph-mds[63924]: -- 172.21.15.188:0/2744486468 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5590f38dfa40 con 0x5590f38b2800 2024-04-26T02:08:16.141 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 02:08:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-c[63920]: starting mds.c at 2024-04-26T02:08:16.142 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 02:08:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i[64227]: ignoring --setuser ceph since I am not root 2024-04-26T02:08:16.142 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 02:08:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i[64227]: ignoring --setgroup ceph since I am not root 2024-04-26T02:08:16.142 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 02:08:15 smithi188 ceph-mds[64231]: -- 172.21.15.188:0/493672223 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a714793a40 con 0x55a714767000 2024-04-26T02:08:16.142 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 02:08:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i[64227]: starting mds.i at 2024-04-26T02:08:16.143 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:08:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: ignoring --setuser ceph since I am not root 2024-04-26T02:08:16.143 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:08:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: ignoring --setgroup ceph since I am not root 2024-04-26T02:08:16.143 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:08:15 smithi188 ceph-mds[68153]: -- 172.21.15.188:0/4174874784 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c97ea5ba40 con 0x55c97ea2ec00 2024-04-26T02:08:16.143 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:08:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: starting mds.f at 2024-04-26T02:08:16.192 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 02:08:15 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-h[63311]: ignoring --setuser ceph since I am not root 2024-04-26T02:08:16.192 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 02:08:15 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-h[63311]: ignoring --setgroup ceph since I am not root 2024-04-26T02:08:16.192 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 02:08:15 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-h[63311]: starting mds.h at 2024-04-26T02:08:16.192 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 02:08:15 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-b[63628]: ignoring --setuser ceph since I am not root 2024-04-26T02:08:16.192 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 02:08:15 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-b[63628]: ignoring --setgroup ceph since I am not root 2024-04-26T02:08:16.192 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 02:08:15 smithi029 ceph-mds[63632]: -- 172.21.15.29:0/1112697887 <== mon.2 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55af0bfaba40 con 0x55af0bf7ec00 2024-04-26T02:08:16.193 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 02:08:15 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-b[63628]: starting mds.b at 2024-04-26T02:08:16.483 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T02:08:16.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:16 smithi006 ceph-mon[28419]: pgmap v710: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 143 KiB/s wr, 15 op/s 2024-04-26T02:08:16.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:16 smithi006 ceph-mon[28419]: osdmap e134: 12 total, 12 up, 12 in 2024-04-26T02:08:16.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:16 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2243379903' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T02:08:16.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:16 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T02:08:16.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:16 smithi006 ceph-mon[28419]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T02:08:16.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:16 smithi006 ceph-mon[28419]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T02:08:16.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:16 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T02:08:16.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:16 smithi006 ceph-mon[28419]: osdmap e135: 12 total, 12 up, 12 in 2024-04-26T02:08:16.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:16 smithi006 ceph-mon[28419]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T02:08:16.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:16 smithi188 ceph-mon[32751]: pgmap v710: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 143 KiB/s wr, 15 op/s 2024-04-26T02:08:16.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:16 smithi188 ceph-mon[32751]: osdmap e134: 12 total, 12 up, 12 in 2024-04-26T02:08:16.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:16 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2243379903' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T02:08:16.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:16 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T02:08:16.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:16 smithi188 ceph-mon[32751]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T02:08:16.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:16 smithi188 ceph-mon[32751]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T02:08:16.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:16 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T02:08:16.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:16 smithi188 ceph-mon[32751]: osdmap e135: 12 total, 12 up, 12 in 2024-04-26T02:08:16.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:16 smithi188 ceph-mon[32751]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T02:08:16.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:16 smithi029 ceph-mon[32775]: pgmap v710: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 143 KiB/s wr, 15 op/s 2024-04-26T02:08:16.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:16 smithi029 ceph-mon[32775]: osdmap e134: 12 total, 12 up, 12 in 2024-04-26T02:08:16.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:16 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2243379903' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T02:08:16.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:16 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T02:08:16.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:16 smithi029 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T02:08:16.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:16 smithi029 ceph-mon[32775]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T02:08:16.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:16 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T02:08:16.943 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:16 smithi029 ceph-mon[32775]: osdmap e135: 12 total, 12 up, 12 in 2024-04-26T02:08:16.943 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:16 smithi029 ceph-mon[32775]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T02:08:16.960 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:17.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T02:08:17.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T02:08:17.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:08:17.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6834/2914125714,v1:172.21.15.188:6835/2914125714] up:boot 2024-04-26T02:08:17.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.29:6837/2315605512,v1:172.21.15.29:6839/2315605512] up:boot 2024-04-26T02:08:17.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.29:6836/602676512,v1:172.21.15.29:6838/602676512] up:boot 2024-04-26T02:08:17.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6836/2622025754,v1:172.21.15.188:6837/2622025754] up:boot 2024-04-26T02:08:17.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.188:6838/2782675182,v1:172.21.15.188:6839/2782675182] up:boot 2024-04-26T02:08:17.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: mds.? [v2:172.21.15.6:6838/3419831811,v1:172.21.15.6:6839/3419831811] up:boot 2024-04-26T02:08:17.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-26T02:08:17.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T02:08:17.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T02:08:17.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:17 smithi006 ceph-mon[28419]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T02:08:17.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T02:08:17.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T02:08:17.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:08:17.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6834/2914125714,v1:172.21.15.188:6835/2914125714] up:boot 2024-04-26T02:08:17.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.29:6837/2315605512,v1:172.21.15.29:6839/2315605512] up:boot 2024-04-26T02:08:17.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.29:6836/602676512,v1:172.21.15.29:6838/602676512] up:boot 2024-04-26T02:08:17.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6836/2622025754,v1:172.21.15.188:6837/2622025754] up:boot 2024-04-26T02:08:17.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.188:6838/2782675182,v1:172.21.15.188:6839/2782675182] up:boot 2024-04-26T02:08:17.943 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: mds.? [v2:172.21.15.6:6838/3419831811,v1:172.21.15.6:6839/3419831811] up:boot 2024-04-26T02:08:17.943 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-26T02:08:17.943 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T02:08:17.943 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T02:08:17.943 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:17 smithi029 ceph-mon[32775]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T02:08:18.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T02:08:18.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T02:08:18.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:08:18.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6834/2914125714,v1:172.21.15.188:6835/2914125714] up:boot 2024-04-26T02:08:18.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.29:6837/2315605512,v1:172.21.15.29:6839/2315605512] up:boot 2024-04-26T02:08:18.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.29:6836/602676512,v1:172.21.15.29:6838/602676512] up:boot 2024-04-26T02:08:18.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6836/2622025754,v1:172.21.15.188:6837/2622025754] up:boot 2024-04-26T02:08:18.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.188:6838/2782675182,v1:172.21.15.188:6839/2782675182] up:boot 2024-04-26T02:08:18.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: mds.? [v2:172.21.15.6:6838/3419831811,v1:172.21.15.6:6839/3419831811] up:boot 2024-04-26T02:08:18.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-26T02:08:18.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T02:08:18.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T02:08:18.143 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:17 smithi188 ceph-mon[32751]: from='mgr.25037 172.21.15.6:0/3038646264' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T02:08:18.874 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:18 smithi006 ceph-mon[28419]: pgmap v713: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 10 op/s 2024-04-26T02:08:18.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:18 smithi029 ceph-mon[32775]: pgmap v713: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 10 op/s 2024-04-26T02:08:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:18 smithi188 ceph-mon[32751]: pgmap v713: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 10 op/s 2024-04-26T02:08:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.184+0000 7f7403000700 1 -- 172.21.15.6:0/681916694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc1092d0 msgr2=0x7f73fc1096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.184+0000 7f7403000700 1 --2- 172.21.15.6:0/681916694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc1092d0 0x7f73fc1096b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f73ec009990 tx=0x7f73ec009ca0 comp rx=0 tx=0).stop 2024-04-26T02:08:19.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.184+0000 7f7403000700 1 -- 172.21.15.6:0/681916694 shutdown_connections 2024-04-26T02:08:19.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.184+0000 7f7403000700 1 --2- 172.21.15.6:0/681916694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73fc10e3b0 0x7f73fc115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:19.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.184+0000 7f7403000700 1 --2- 172.21.15.6:0/681916694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc1092d0 0x7f73fc1096b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:19.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.184+0000 7f7403000700 1 --2- 172.21.15.6:0/681916694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73fc109c80 0x7f73fc10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:19.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.184+0000 7f7403000700 1 -- 172.21.15.6:0/681916694 >> 172.21.15.6:0/681916694 conn(0x7f73fc076210 msgr2=0x7f73fc107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:19.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.184+0000 7f7403000700 1 -- 172.21.15.6:0/681916694 shutdown_connections 2024-04-26T02:08:19.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.185+0000 7f7403000700 1 -- 172.21.15.6:0/681916694 wait complete. 2024-04-26T02:08:19.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.186+0000 7f7403000700 1 Processor -- start 2024-04-26T02:08:19.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.186+0000 7f7403000700 1 -- start start 2024-04-26T02:08:19.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.186+0000 7f7403000700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc1092d0 0x7f73fc19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:19.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.186+0000 7f7403000700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73fc109c80 0x7f73fc19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:19.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f7403000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73fc10e3b0 0x7f73fc1a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:19.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f7403000700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc119240 con 0x7f73fc109c80 2024-04-26T02:08:19.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f7403000700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc1190c0 con 0x7f73fc1092d0 2024-04-26T02:08:19.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f7403000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f73fc1193c0 con 0x7f73fc10e3b0 2024-04-26T02:08:19.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f73fb400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73fc109c80 0x7f73fc19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:19.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f73fbe00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc1092d0 0x7f73fc19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:19.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f73fb400700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73fc109c80 0x7f73fc19fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56110/0 (socket says 172.21.15.6:56110) 2024-04-26T02:08:19.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f73fb400700 1 -- 172.21.15.6:0/1651052512 learned_addr learned my addr 172.21.15.6:0/1651052512 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:08:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f7401000700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73fc10e3b0 0x7f73fc1a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f73fbe00700 1 -- 172.21.15.6:0/1651052512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73fc10e3b0 msgr2=0x7f73fc1a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f73fbe00700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73fc10e3b0 0x7f73fc1a4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f73fbe00700 1 -- 172.21.15.6:0/1651052512 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73fc109c80 msgr2=0x7f73fc19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f73fbe00700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73fc109c80 0x7f73fc19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.187+0000 7f73fbe00700 1 -- 172.21.15.6:0/1651052512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73fc1a4740 con 0x7f73fc1092d0 2024-04-26T02:08:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.188+0000 7f7401000700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73fc10e3b0 0x7f73fc1a4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:08:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.188+0000 7f73fb400700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73fc109c80 0x7f73fc19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:08:19.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.188+0000 7f73fbe00700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc1092d0 0x7f73fc19f730 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f73ec006010 tx=0x7f73ec0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.188+0000 7f73f8c00700 1 -- 172.21.15.6:0/1651052512 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec027070 con 0x7f73fc1092d0 2024-04-26T02:08:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.188+0000 7f73f8c00700 1 -- 172.21.15.6:0/1651052512 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73ec00b610 con 0x7f73fc1092d0 2024-04-26T02:08:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.188+0000 7f7403000700 1 -- 172.21.15.6:0/1651052512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73fc1a49d0 con 0x7f73fc1092d0 2024-04-26T02:08:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.188+0000 7f73f8c00700 1 -- 172.21.15.6:0/1651052512 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec022e30 con 0x7f73fc1092d0 2024-04-26T02:08:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.188+0000 7f7403000700 1 -- 172.21.15.6:0/1651052512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73fc1ac2b0 con 0x7f73fc1092d0 2024-04-26T02:08:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.190+0000 7f7403000700 1 -- 172.21.15.6:0/1651052512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73fc069730 con 0x7f73fc1092d0 2024-04-26T02:08:19.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.192+0000 7f73f8c00700 1 -- 172.21.15.6:0/1651052512 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f73ec02a070 con 0x7f73fc1092d0 2024-04-26T02:08:19.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.193+0000 7f73f8c00700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f73e00a3e30 0x7f73e00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:19.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.193+0000 7f73f8c00700 1 -- 172.21.15.6:0/1651052512 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 8814+0+0 (secure 0 0 0) 0x7f73ec0d1780 con 0x7f73fc1092d0 2024-04-26T02:08:19.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.194+0000 7f73fb400700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f73e00a3e30 0x7f73e00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:19.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.194+0000 7f73fb400700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f73e00a3e30 0x7f73e00a62f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0782a0 tx=0x7f73e8009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:19.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.196+0000 7f73f8c00700 1 -- 172.21.15.6:0/1651052512 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f73ec013070 con 0x7f73fc1092d0 2024-04-26T02:08:19.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.480+0000 7f7403000700 1 -- 172.21.15.6:0/1651052512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f73fc1a0a90 con 0x7f73fc1092d0 2024-04-26T02:08:19.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.687+0000 7f73f8c00700 1 -- 172.21.15.6:0/1651052512 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v63) ==== 104+0+0 (secure 0 0 0) 0x7f73ec09dd30 con 0x7f73fc1092d0 2024-04-26T02:08:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.692+0000 7f7403000700 1 -- 172.21.15.6:0/1651052512 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f73e00a3e30 msgr2=0x7f73e00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.692+0000 7f7403000700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f73e00a3e30 0x7f73e00a62f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0782a0 tx=0x7f73e8009260 comp rx=0 tx=0).stop 2024-04-26T02:08:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.692+0000 7f7403000700 1 -- 172.21.15.6:0/1651052512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc1092d0 msgr2=0x7f73fc19f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.692+0000 7f7403000700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc1092d0 0x7f73fc19f730 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f73ec006010 tx=0x7f73ec0057c0 comp rx=0 tx=0).stop 2024-04-26T02:08:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.692+0000 7f7403000700 1 -- 172.21.15.6:0/1651052512 shutdown_connections 2024-04-26T02:08:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.692+0000 7f7403000700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f73e00a3e30 0x7f73e00a62f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.692+0000 7f7403000700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73fc10e3b0 0x7f73fc1a4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:19.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.692+0000 7f7403000700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc1092d0 0x7f73fc19f730 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:19.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.692+0000 7f7403000700 1 --2- 172.21.15.6:0/1651052512 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73fc109c80 0x7f73fc19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:19.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.692+0000 7f7403000700 1 -- 172.21.15.6:0/1651052512 >> 172.21.15.6:0/1651052512 conn(0x7f73fc076210 msgr2=0x7f73fc10cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:19.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.693+0000 7f7403000700 1 -- 172.21.15.6:0/1651052512 shutdown_connections 2024-04-26T02:08:19.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:19.693+0000 7f7403000700 1 -- 172.21.15.6:0/1651052512 wait complete. 2024-04-26T02:08:19.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:19 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1651052512' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T02:08:19.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:19 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T02:08:19.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:19 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1651052512' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T02:08:19.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:19 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T02:08:20.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:19 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1651052512' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T02:08:20.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:19 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T02:08:20.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T02:08:20.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:20 smithi006 ceph-mon[28419]: pgmap v714: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 3 op/s 2024-04-26T02:08:20.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:20 smithi006 ceph-mon[28419]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T02:08:20.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:20 smithi006 ceph-mon[28419]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T02:08:20.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:20 smithi006 ceph-mon[28419]: Cluster is now healthy 2024-04-26T02:08:20.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:20 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T02:08:20.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:20 smithi006 ceph-mon[28419]: fsmap 12 up:standby 2024-04-26T02:08:20.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:20 smithi029 ceph-mon[32775]: pgmap v714: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 3 op/s 2024-04-26T02:08:20.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:20 smithi029 ceph-mon[32775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T02:08:20.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:20 smithi029 ceph-mon[32775]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T02:08:20.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:20 smithi029 ceph-mon[32775]: Cluster is now healthy 2024-04-26T02:08:20.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:20 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T02:08:20.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:20 smithi029 ceph-mon[32775]: fsmap 12 up:standby 2024-04-26T02:08:21.050 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:21.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:20 smithi188 ceph-mon[32751]: pgmap v714: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 3 op/s 2024-04-26T02:08:21.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:20 smithi188 ceph-mon[32751]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T02:08:21.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:20 smithi188 ceph-mon[32751]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T02:08:21.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:20 smithi188 ceph-mon[32751]: Cluster is now healthy 2024-04-26T02:08:21.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:20 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T02:08:21.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:20 smithi188 ceph-mon[32751]: fsmap 12 up:standby 2024-04-26T02:08:22.917 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:22 smithi006 ceph-mon[28419]: pgmap v715: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 3 op/s 2024-04-26T02:08:22.942 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:22 smithi029 ceph-mon[32775]: pgmap v715: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 3 op/s 2024-04-26T02:08:23.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:22 smithi188 ceph-mon[32751]: pgmap v715: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 3 op/s 2024-04-26T02:08:23.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.667+0000 7fd2aa200700 1 -- 172.21.15.6:0/3108045375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2a4104dd0 msgr2=0x7fd2a41051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:23.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.667+0000 7fd2aa200700 1 --2- 172.21.15.6:0/3108045375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2a4104dd0 0x7fd2a41051b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd2980099c0 tx=0x7fd298009cd0 comp rx=0 tx=0).stop 2024-04-26T02:08:23.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.668+0000 7fd2aa200700 1 -- 172.21.15.6:0/3108045375 shutdown_connections 2024-04-26T02:08:23.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.668+0000 7fd2aa200700 1 --2- 172.21.15.6:0/3108045375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4109eb0 0x7fd2a4111a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:23.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.668+0000 7fd2aa200700 1 --2- 172.21.15.6:0/3108045375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2a4104dd0 0x7fd2a41051b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:23.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.668+0000 7fd2aa200700 1 --2- 172.21.15.6:0/3108045375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a4105780 0x7fd2a41097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:23.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.668+0000 7fd2aa200700 1 -- 172.21.15.6:0/3108045375 >> 172.21.15.6:0/3108045375 conn(0x7fd2a4100660 msgr2=0x7fd2a4102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:23.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.668+0000 7fd2aa200700 1 -- 172.21.15.6:0/3108045375 shutdown_connections 2024-04-26T02:08:23.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.668+0000 7fd2aa200700 1 -- 172.21.15.6:0/3108045375 wait complete. 2024-04-26T02:08:23.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.669+0000 7fd2aa200700 1 Processor -- start 2024-04-26T02:08:23.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.669+0000 7fd2aa200700 1 -- start start 2024-04-26T02:08:23.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.670+0000 7fd2aa200700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a4104dd0 0x7fd2a419b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:23.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.670+0000 7fd2aa200700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2a4105780 0x7fd2a419b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:23.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.670+0000 7fd2aa200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4109eb0 0x7fd2a419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:23.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.670+0000 7fd2aa200700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4114d50 con 0x7fd2a4105780 2024-04-26T02:08:23.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.670+0000 7fd2aa200700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4114bd0 con 0x7fd2a4104dd0 2024-04-26T02:08:23.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.670+0000 7fd2aa200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4114ed0 con 0x7fd2a4109eb0 2024-04-26T02:08:23.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.670+0000 7fd2a8e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a4104dd0 0x7fd2a419b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:23.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.670+0000 7fd2a8e00700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a4104dd0 0x7fd2a419b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.6:36874/0 (socket says 172.21.15.6:36874) 2024-04-26T02:08:23.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.670+0000 7fd2a8e00700 1 -- 172.21.15.6:0/1373483377 learned_addr learned my addr 172.21.15.6:0/1373483377 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:08:23.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.670+0000 7fd2a3e00700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2a4105780 0x7fd2a419b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:23.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.670+0000 7fd2a9800700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4109eb0 0x7fd2a419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:23.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.671+0000 7fd2a8e00700 1 -- 172.21.15.6:0/1373483377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4109eb0 msgr2=0x7fd2a419fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:23.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.671+0000 7fd2a8e00700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4109eb0 0x7fd2a419fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:23.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.671+0000 7fd2a8e00700 1 -- 172.21.15.6:0/1373483377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2a4105780 msgr2=0x7fd2a419b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:23.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.671+0000 7fd2a8e00700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2a4105780 0x7fd2a419b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:23.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.671+0000 7fd2a8e00700 1 -- 172.21.15.6:0/1373483377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a41a0240 con 0x7fd2a4104dd0 2024-04-26T02:08:23.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.671+0000 7fd2a9800700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4109eb0 0x7fd2a419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:08:23.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.671+0000 7fd2a8e00700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a4104dd0 0x7fd2a419b280 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd298006010 tx=0x7fd2980055f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:23.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.671+0000 7fd2a1600700 1 -- 172.21.15.6:0/1373483377 <== mon.2 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd298026070 con 0x7fd2a4104dd0 2024-04-26T02:08:23.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.672+0000 7fd2aa200700 1 -- 172.21.15.6:0/1373483377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a41a04d0 con 0x7fd2a4104dd0 2024-04-26T02:08:23.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.672+0000 7fd2a1600700 1 -- 172.21.15.6:0/1373483377 <== mon.2 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd29800f460 con 0x7fd2a4104dd0 2024-04-26T02:08:23.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.672+0000 7fd2a1600700 1 -- 172.21.15.6:0/1373483377 <== mon.2 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd298009e60 con 0x7fd2a4104dd0 2024-04-26T02:08:23.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.672+0000 7fd2aa200700 1 -- 172.21.15.6:0/1373483377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a41a7db0 con 0x7fd2a4104dd0 2024-04-26T02:08:23.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.673+0000 7fd2aa200700 1 -- 172.21.15.6:0/1373483377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a4069730 con 0x7fd2a4104dd0 2024-04-26T02:08:23.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.675+0000 7fd2a1600700 1 -- 172.21.15.6:0/1373483377 <== mon.2 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7fd298029070 con 0x7fd2a4104dd0 2024-04-26T02:08:23.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.676+0000 7fd2a1600700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd28c0a3e30 0x7fd28c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:23.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.677+0000 7fd2a1600700 1 -- 172.21.15.6:0/1373483377 <== mon.2 v2:172.21.15.29:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 8814+0+0 (secure 0 0 0) 0x7fd2980d1540 con 0x7fd2a4104dd0 2024-04-26T02:08:23.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.677+0000 7fd2a3e00700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd28c0a3e30 0x7fd28c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:23.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.677+0000 7fd2a3e00700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd28c0a3e30 0x7fd28c0a62f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd2a410b450 tx=0x7fd290009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.679+0000 7fd2a1600700 1 -- 172.21.15.6:0/1373483377 <== mon.2 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd298030020 con 0x7fd2a4104dd0 2024-04-26T02:08:23.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:23.916+0000 7fd2aa200700 1 -- 172.21.15.6:0/1373483377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd2a402df80 con 0x7fd2a4104dd0 2024-04-26T02:08:24.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.701+0000 7fd2a1600700 1 -- 172.21.15.6:0/1373483377 <== mon.2 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v136) ==== 181+0+0 (secure 0 0 0) 0x7fd298013070 con 0x7fd2a4104dd0 2024-04-26T02:08:24.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.707+0000 7fd2aa200700 1 -- 172.21.15.6:0/1373483377 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd28c0a3e30 msgr2=0x7fd28c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.707+0000 7fd2aa200700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd28c0a3e30 0x7fd28c0a62f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd2a410b450 tx=0x7fd290009260 comp rx=0 tx=0).stop 2024-04-26T02:08:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.707+0000 7fd2aa200700 1 -- 172.21.15.6:0/1373483377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a4104dd0 msgr2=0x7fd2a419b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.707+0000 7fd2aa200700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a4104dd0 0x7fd2a419b280 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd298006010 tx=0x7fd2980055f0 comp rx=0 tx=0).stop 2024-04-26T02:08:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.707+0000 7fd2aa200700 1 -- 172.21.15.6:0/1373483377 shutdown_connections 2024-04-26T02:08:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.707+0000 7fd2aa200700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7fd28c0a3e30 0x7fd28c0a62f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.707+0000 7fd2aa200700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4109eb0 0x7fd2a419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.707+0000 7fd2aa200700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a4104dd0 0x7fd2a419b280 secure :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd298006010 tx=0x7fd2980055f0 comp rx=0 tx=0).stop 2024-04-26T02:08:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.707+0000 7fd2aa200700 1 --2- 172.21.15.6:0/1373483377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2a4105780 0x7fd2a419b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.707+0000 7fd2aa200700 1 -- 172.21.15.6:0/1373483377 >> 172.21.15.6:0/1373483377 conn(0x7fd2a4100660 msgr2=0x7fd2a4101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.708+0000 7fd2aa200700 1 -- 172.21.15.6:0/1373483377 shutdown_connections 2024-04-26T02:08:24.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:24.708+0000 7fd2aa200700 1 -- 172.21.15.6:0/1373483377 wait complete. 2024-04-26T02:08:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-04-26T02:08:25.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:24 smithi006 ceph-mon[28419]: pgmap v716: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 3 op/s 2024-04-26T02:08:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:24 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/1373483377' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:25.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:24 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:25.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:24 smithi188 ceph-mon[32751]: pgmap v716: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 3 op/s 2024-04-26T02:08:25.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:24 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/1373483377' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:25.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:24 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:25.191 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:24 smithi029 ceph-mon[32775]: pgmap v716: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 3 op/s 2024-04-26T02:08:25.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:24 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/1373483377' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:25.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:24 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:25.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid a5b72bee-036c-11ef-bc93-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T02:08:25.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:25 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T02:08:25.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:25 smithi006 ceph-mon[28419]: osdmap e136: 12 total, 12 up, 12 in 2024-04-26T02:08:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:26.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:25 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T02:08:26.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:25 smithi188 ceph-mon[32751]: osdmap e136: 12 total, 12 up, 12 in 2024-04-26T02:08:26.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:25 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T02:08:26.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:25 smithi029 ceph-mon[32775]: osdmap e136: 12 total, 12 up, 12 in 2024-04-26T02:08:26.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:26 smithi006 ceph-mon[28419]: pgmap v718: 65 pgs: 65 active+clean; 313 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T02:08:27.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:26 smithi188 ceph-mon[32751]: pgmap v718: 65 pgs: 65 active+clean; 313 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T02:08:27.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:26 smithi029 ceph-mon[32775]: pgmap v718: 65 pgs: 65 active+clean; 313 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T02:08:28.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.195+0000 7f2036800700 1 -- 172.21.15.6:0/2286257436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f203006b980 msgr2=0x7f203010f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:28.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.195+0000 7f2036800700 1 --2- 172.21.15.6:0/2286257436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f203006b980 0x7f203010f8f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f202800b0a0 tx=0x7f202800b3b0 comp rx=0 tx=0).stop 2024-04-26T02:08:28.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.195+0000 7f2036800700 1 -- 172.21.15.6:0/2286257436 shutdown_connections 2024-04-26T02:08:28.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.195+0000 7f2036800700 1 --2- 172.21.15.6:0/2286257436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f203006b980 0x7f203010f8f0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:28.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.195+0000 7f2036800700 1 --2- 172.21.15.6:0/2286257436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2030108a20 0x7f2030108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.195+0000 7f2036800700 1 --2- 172.21.15.6:0/2286257436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f203006afe0 0x7f203006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.195+0000 7f2036800700 1 -- 172.21.15.6:0/2286257436 >> 172.21.15.6:0/2286257436 conn(0x7f20300788b0 msgr2=0x7f2030078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.195+0000 7f2036800700 1 -- 172.21.15.6:0/2286257436 shutdown_connections 2024-04-26T02:08:28.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.195+0000 7f2036800700 1 -- 172.21.15.6:0/2286257436 wait complete. 2024-04-26T02:08:28.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.196+0000 7f2036800700 1 Processor -- start 2024-04-26T02:08:28.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.196+0000 7f2036800700 1 -- start start 2024-04-26T02:08:28.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f2036800700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f203006afe0 0x7f203019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:28.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f2036800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f203006b980 0x7f203019b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:28.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f2036800700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2030108a20 0x7f203019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:28.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f2036800700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2030112c10 con 0x7f2030108a20 2024-04-26T02:08:28.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f2036800700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2030112a90 con 0x7f203006afe0 2024-04-26T02:08:28.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f2036800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2030112d90 con 0x7f203006b980 2024-04-26T02:08:28.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f202ea00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f203006b980 0x7f203019b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:28.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f202ea00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f203006b980 0x7f203019b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.6:46886/0 (socket says 172.21.15.6:46886) 2024-04-26T02:08:28.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f202ea00700 1 -- 172.21.15.6:0/2181711586 learned_addr learned my addr 172.21.15.6:0/2181711586 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-26T02:08:28.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f202fe00700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2030108a20 0x7f203019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:28.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f202f400700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f203006afe0 0x7f203019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:28.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f202ea00700 1 -- 172.21.15.6:0/2181711586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f203006afe0 msgr2=0x7f203019b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:28.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.197+0000 7f202ea00700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f203006afe0 0x7f203019b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:28.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.198+0000 7f202ea00700 1 -- 172.21.15.6:0/2181711586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2030108a20 msgr2=0x7f203019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:28.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.198+0000 7f202ea00700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2030108a20 0x7f203019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:28.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.198+0000 7f202ea00700 1 -- 172.21.15.6:0/2181711586 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20301a0350 con 0x7f203006b980 2024-04-26T02:08:28.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.198+0000 7f202ea00700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f203006b980 0x7f203019b880 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e9a0 tx=0x7f201c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:28.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.198+0000 7f2027e00700 1 -- 172.21.15.6:0/2181711586 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c00ca10 con 0x7f203006b980 2024-04-26T02:08:28.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.198+0000 7f2036800700 1 -- 172.21.15.6:0/2181711586 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20301a0640 con 0x7f203006b980 2024-04-26T02:08:28.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.198+0000 7f2027e00700 1 -- 172.21.15.6:0/2181711586 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f201c00cbb0 con 0x7f203006b980 2024-04-26T02:08:28.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.198+0000 7f2027e00700 1 -- 172.21.15.6:0/2181711586 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c018420 con 0x7f203006b980 2024-04-26T02:08:28.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.199+0000 7f2036800700 1 -- 172.21.15.6:0/2181711586 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20301a7ec0 con 0x7f203006b980 2024-04-26T02:08:28.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.200+0000 7f2036800700 1 -- 172.21.15.6:0/2181711586 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2030069730 con 0x7f203006b980 2024-04-26T02:08:28.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.202+0000 7f2027e00700 1 -- 172.21.15.6:0/2181711586 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138776+0+0 (secure 0 0 0) 0x7f201c010430 con 0x7f203006b980 2024-04-26T02:08:28.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.203+0000 7f2027e00700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f20140a3e30 0x7f20140a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.203+0000 7f2027e00700 1 -- 172.21.15.6:0/2181711586 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 8260+0+0 (secure 0 0 0) 0x7f201c0d0900 con 0x7f203006b980 2024-04-26T02:08:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.203+0000 7f202f400700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f20140a3e30 0x7f20140a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.204+0000 7f202f400700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f20140a3e30 0x7f20140a62f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2020009be0 tx=0x7f2020009260 comp rx=0 tx=0).ready entity=mgr.25037 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:28.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.207+0000 7f2027e00700 1 -- 172.21.15.6:0/2181711586 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f201c09d040 con 0x7f203006b980 2024-04-26T02:08:28.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.454+0000 7f2036800700 1 -- 172.21.15.6:0/2181711586 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f20301a80f0 con 0x7f203006b980 2024-04-26T02:08:28.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.738+0000 7f2027e00700 1 -- 172.21.15.6:0/2181711586 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v137) ==== 169+0+0 (secure 0 0 0) 0x7f201c00cd60 con 0x7f203006b980 2024-04-26T02:08:28.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.744+0000 7f2036800700 1 -- 172.21.15.6:0/2181711586 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f20140a3e30 msgr2=0x7f20140a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:28.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.744+0000 7f2036800700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f20140a3e30 0x7f20140a62f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2020009be0 tx=0x7f2020009260 comp rx=0 tx=0).stop 2024-04-26T02:08:28.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.744+0000 7f2036800700 1 -- 172.21.15.6:0/2181711586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f203006b980 msgr2=0x7f203019b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:28.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.744+0000 7f2036800700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f203006b980 0x7f203019b880 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e9a0 tx=0x7f201c00ecb0 comp rx=0 tx=0).stop 2024-04-26T02:08:28.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.746+0000 7f2036800700 1 -- 172.21.15.6:0/2181711586 shutdown_connections 2024-04-26T02:08:28.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.746+0000 7f2036800700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.6:6800/3353570381,v1:172.21.15.6:6801/3353570381] conn(0x7f20140a3e30 0x7f20140a62f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:28.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.746+0000 7f2036800700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2030108a20 0x7f203019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:28.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.746+0000 7f2036800700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f203006afe0 0x7f203019b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:28.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.746+0000 7f2036800700 1 --2- 172.21.15.6:0/2181711586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f203006b980 0x7f203019b880 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:28.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.746+0000 7f2036800700 1 -- 172.21.15.6:0/2181711586 >> 172.21.15.6:0/2181711586 conn(0x7f20300788b0 msgr2=0x7f20300ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:28.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.746+0000 7f2036800700 1 -- 172.21.15.6:0/2181711586 shutdown_connections 2024-04-26T02:08:28.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-26T02:08:28.746+0000 7f2036800700 1 -- 172.21.15.6:0/2181711586 wait complete. 2024-04-26T02:08:28.749 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-04-26T02:08:29.067 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:28 smithi006 ceph-mon[28419]: pgmap v719: 65 pgs: 65 active+clean; 313 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T02:08:29.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:28 smithi006 ceph-mon[28419]: from='client.? 172.21.15.6:0/2181711586' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:29.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:28 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:29.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:28 smithi188 ceph-mon[32751]: pgmap v719: 65 pgs: 65 active+clean; 313 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T02:08:29.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:28 smithi188 ceph-mon[32751]: from='client.? 172.21.15.6:0/2181711586' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:29.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:28 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:29.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:28 smithi029 ceph-mon[32775]: pgmap v719: 65 pgs: 65 active+clean; 313 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T02:08:29.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:28 smithi029 ceph-mon[32775]: from='client.? 172.21.15.6:0/2181711586' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:29.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:28 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:29.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:08:29.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:08:29.740 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:08:29.775 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-26T02:08:29.775 DEBUG:teuthology.orchestra.run.smithi006:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T02:08:29.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:29 smithi006 ceph-mon[28419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T02:08:29.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:29 smithi006 ceph-mon[28419]: osdmap e137: 12 total, 12 up, 12 in 2024-04-26T02:08:29.829 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T02:08:29.850 DEBUG:teuthology.orchestra.run.smithi188:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T02:08:29.865 INFO:tasks.cephadm:Stopping all daemons... 2024-04-26T02:08:29.866 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-26T02:08:29.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.a 2024-04-26T02:08:30.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:29 smithi188 ceph-mon[32751]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T02:08:30.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:29 smithi188 ceph-mon[32751]: osdmap e137: 12 total, 12 up, 12 in 2024-04-26T02:08:30.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:29 smithi029 ceph-mon[32775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T02:08:30.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:29 smithi029 ceph-mon[32775]: osdmap e137: 12 total, 12 up, 12 in 2024-04-26T02:08:30.318 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:29 smithi006 systemd[1]: Stopping Ceph mon.a for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:30.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:30 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a[28415]: 2024-04-26T02:08:30.489+0000 7f63e1400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:30.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:30 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a[28415]: 2024-04-26T02:08:30.489+0000 7f63e1400700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-26T02:08:30.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:30 smithi006 podman[110572]: 2024-04-26 02:08:30.676846873 +0000 UTC m=+0.341095040 container died 3d4cd5294d32443d4cb0bd7e3122659a74439ad64ce72f1163da9b03b96c02a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:08:31.496 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:31 smithi006 podman[110572]: 2024-04-26 02:08:31.210983383 +0000 UTC m=+0.875231570 container cleanup 3d4cd5294d32443d4cb0bd7e3122659a74439ad64ce72f1163da9b03b96c02a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-26T02:08:31.496 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:31 smithi006 bash[110572]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a 2024-04-26T02:08:31.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Apr 26 02:08:31 smithi006 podman[110586]: 2024-04-26 02:08:31.496100687 +0000 UTC m=+0.804059518 container remove 3d4cd5294d32443d4cb0bd7e3122659a74439ad64ce72f1163da9b03b96c02a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-26T02:08:32.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.a.service' 2024-04-26T02:08:32.147 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:32.148 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-26T02:08:32.148 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-26T02:08:32.148 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.b 2024-04-26T02:08:32.442 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:32 smithi029 systemd[1]: Stopping Ceph mon.b for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:32.807 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:32 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-b[32771]: 2024-04-26T02:08:32.480+0000 7fa02280e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:32.807 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:32 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-b[32771]: 2024-04-26T02:08:32.480+0000 7fa02280e700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-26T02:08:32.807 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:32 smithi029 podman[72453]: 2024-04-26 02:08:32.542622636 +0000 UTC m=+0.179435446 container died 3724a8af3fb8d03f6fe626b460877764586fdb88d3fffc1d458f1ae7fb27aaf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:08:33.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:32 smithi029 podman[72453]: 2024-04-26 02:08:32.925257117 +0000 UTC m=+0.562069927 container cleanup 3724a8af3fb8d03f6fe626b460877764586fdb88d3fffc1d458f1ae7fb27aaf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-b, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T02:08:33.192 INFO:journalctl@ceph.mon.b.smithi029.stdout:Apr 26 02:08:32 smithi029 bash[72453]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-b 2024-04-26T02:08:33.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.b.service' 2024-04-26T02:08:33.396 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:33.396 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-26T02:08:33.396 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-26T02:08:33.397 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.c 2024-04-26T02:08:33.738 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:33 smithi188 systemd[1]: Stopping Ceph mon.c for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:34.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:33 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-c[32747]: 2024-04-26T02:08:33.739+0000 7f1503d20700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:34.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:33 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-c[32747]: 2024-04-26T02:08:33.739+0000 7f1503d20700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-26T02:08:34.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:33 smithi188 podman[73666]: 2024-04-26 02:08:33.912426345 +0000 UTC m=+0.301165020 container died 344aa2641bc9ba02b95f7932791ffd516dca7f3b66847cb033e45af339296781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , ceph=True) 2024-04-26T02:08:34.606 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:34 smithi188 podman[73666]: 2024-04-26 02:08:34.272587193 +0000 UTC m=+0.661325875 container cleanup 344aa2641bc9ba02b95f7932791ffd516dca7f3b66847cb033e45af339296781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:08:34.606 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 26 02:08:34 smithi188 bash[73666]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mon-c 2024-04-26T02:08:34.720 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mon.c.service' 2024-04-26T02:08:34.779 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:34.780 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-26T02:08:34.780 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-26T02:08:34.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.x 2024-04-26T02:08:35.319 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 02:08:34 smithi006 systemd[1]: Stopping Ceph mgr.x for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:35.756 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 02:08:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T02:08:35.324+0000 7f018c400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:35.756 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 02:08:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x[28658]: 2024-04-26T02:08:35.324+0000 7f018c400700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-26T02:08:35.756 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 02:08:35 smithi006 podman[110690]: 2024-04-26 02:08:35.419108807 +0000 UTC m=+0.220090270 container died 879fdfd52bdac43e4d44cbd18863b628c21d8a988bb9b32ab81c1ca2d15dc47c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416) 2024-04-26T02:08:36.068 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 02:08:35 smithi006 podman[110690]: 2024-04-26 02:08:35.941278777 +0000 UTC m=+0.742260203 container cleanup 879fdfd52bdac43e4d44cbd18863b628c21d8a988bb9b32ab81c1ca2d15dc47c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-26T02:08:36.068 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 02:08:35 smithi006 bash[110690]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x 2024-04-26T02:08:36.568 INFO:journalctl@ceph.mgr.x.smithi006.stdout:Apr 26 02:08:36 smithi006 podman[110704]: 2024-04-26 02:08:36.176228766 +0000 UTC m=+0.737469823 container remove 879fdfd52bdac43e4d44cbd18863b628c21d8a988bb9b32ab81c1ca2d15dc47c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-26T02:08:36.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-26T02:08:36.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:36.858 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-26T02:08:36.858 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-26T02:08:36.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.y 2024-04-26T02:08:37.192 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 02:08:36 smithi029 systemd[1]: Stopping Ceph mgr.y for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:37.613 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 02:08:37 smithi029 podman[72575]: 2024-04-26 02:08:37.196765134 +0000 UTC m=+0.140106767 container died c9b317ef545a63c8d52fb625add9e3205763ff29c1fb9ea3242bdbcb53d89e1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:08:37.942 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 02:08:37 smithi029 podman[72575]: 2024-04-26 02:08:37.707272003 +0000 UTC m=+0.650613677 container cleanup c9b317ef545a63c8d52fb625add9e3205763ff29c1fb9ea3242bdbcb53d89e1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:08:37.942 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Apr 26 02:08:37 smithi029 bash[72575]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-y 2024-04-26T02:08:38.141 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-26T02:08:38.175 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:38.176 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-26T02:08:38.176 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-26T02:08:38.176 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.z 2024-04-26T02:08:38.488 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 02:08:38 smithi188 systemd[1]: Stopping Ceph mgr.z for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:38.801 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 02:08:38 smithi188 podman[73787]: 2024-04-26 02:08:38.489926293 +0000 UTC m=+0.129027453 container died 8a425eb7f7ea4ab302a05eaef0ee92d1a31b5ee8895443e0417e8fbe77a169fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:08:39.127 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 02:08:38 smithi188 podman[73787]: 2024-04-26 02:08:38.877546975 +0000 UTC m=+0.516648215 container cleanup 8a425eb7f7ea4ab302a05eaef0ee92d1a31b5ee8895443e0417e8fbe77a169fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T02:08:39.127 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 26 02:08:38 smithi188 bash[73787]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mgr-z 2024-04-26T02:08:39.305 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-26T02:08:39.338 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:39.338 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-26T02:08:39.338 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-26T02:08:39.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.0 2024-04-26T02:08:39.818 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:39 smithi006 systemd[1]: Stopping Ceph osd.0 for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:40.318 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:39 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0[43410]: 2024-04-26T02:08:39.868+0000 7f4c74c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:40.319 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:39 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0[43410]: 2024-04-26T02:08:39.868+0000 7f4c74c00700 -1 osd.0 137 *** Got signal Terminated *** 2024-04-26T02:08:40.319 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:39 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0[43410]: 2024-04-26T02:08:39.868+0000 7f4c74c00700 -1 osd.0 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:08:45.696 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:45 smithi006 podman[110807]: 2024-04-26 02:08:45.442623568 +0000 UTC m=+5.705163248 container died 675a86c10535bededc8563e651d9c66bc46ef235497e053e29711099ba29af04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0) 2024-04-26T02:08:45.950 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:45 smithi006 podman[110807]: 2024-04-26 02:08:45.938255274 +0000 UTC m=+6.200794964 container cleanup 675a86c10535bededc8563e651d9c66bc46ef235497e053e29711099ba29af04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:08:45.950 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:45 smithi006 bash[110807]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0 2024-04-26T02:08:46.568 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:46 smithi006 podman[110826]: 2024-04-26 02:08:46.15602481 +0000 UTC m=+0.696171436 container remove 675a86c10535bededc8563e651d9c66bc46ef235497e053e29711099ba29af04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-26T02:08:47.319 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:47 smithi006 podman[110914]: 2024-04-26 02:08:46.922415655 +0000 UTC m=+0.040566324 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:08:47.319 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:47 smithi006 podman[110914]: 2024-04-26 02:08:47.057298091 +0000 UTC m=+0.175448776 container create 087e5ba92a5cb3b91a0ca90cb9dac6d382aac88daf28c47827e1a34c0d3828da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-26T02:08:47.787 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:47 smithi006 podman[110914]: 2024-04-26 02:08:47.320770305 +0000 UTC m=+0.438921040 container init 087e5ba92a5cb3b91a0ca90cb9dac6d382aac88daf28c47827e1a34c0d3828da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5) 2024-04-26T02:08:47.787 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:47 smithi006 podman[110914]: 2024-04-26 02:08:47.333620687 +0000 UTC m=+0.451771372 container start 087e5ba92a5cb3b91a0ca90cb9dac6d382aac88daf28c47827e1a34c0d3828da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:08:47.787 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:47 smithi006 podman[110914]: 2024-04-26 02:08:47.409864863 +0000 UTC m=+0.528015555 container attach 087e5ba92a5cb3b91a0ca90cb9dac6d382aac88daf28c47827e1a34c0d3828da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-26T02:08:48.068 INFO:journalctl@ceph.osd.0.smithi006.stdout:Apr 26 02:08:47 smithi006 podman[110914]: 2024-04-26 02:08:47.788204775 +0000 UTC m=+0.906355487 container died 087e5ba92a5cb3b91a0ca90cb9dac6d382aac88daf28c47827e1a34c0d3828da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-26T02:08:48.458 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.0.service' 2024-04-26T02:08:48.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:48.581 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-26T02:08:48.582 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-26T02:08:48.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.1 2024-04-26T02:08:48.933 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:48 smithi029 systemd[1]: Stopping Ceph osd.1 for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:49.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1[37375]: 2024-04-26T02:08:48.933+0000 7f38defb9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:49.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1[37375]: 2024-04-26T02:08:48.933+0000 7f38defb9700 -1 osd.1 137 *** Got signal Terminated *** 2024-04-26T02:08:49.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1[37375]: 2024-04-26T02:08:48.933+0000 7f38defb9700 -1 osd.1 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:08:54.771 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:54 smithi029 podman[72713]: 2024-04-26 02:08:54.487342351 +0000 UTC m=+5.692776450 container died 4e4681024de18dbd3441372d3c11f01244a74d3eb338f492ba11fd3fa907480f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-26T02:08:55.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:54 smithi029 podman[72713]: 2024-04-26 02:08:54.872249381 +0000 UTC m=+6.077683468 container cleanup 4e4681024de18dbd3441372d3c11f01244a74d3eb338f492ba11fd3fa907480f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-26T02:08:55.115 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:54 smithi029 bash[72713]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1 2024-04-26T02:08:55.442 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:55 smithi029 podman[72730]: 2024-04-26 02:08:55.116730745 +0000 UTC m=+0.626732605 container remove 4e4681024de18dbd3441372d3c11f01244a74d3eb338f492ba11fd3fa907480f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:08:55.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:55 smithi029 podman[72823]: 2024-04-26 02:08:55.353359655 +0000 UTC m=+0.018498775 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:08:55.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:55 smithi029 podman[72823]: 2024-04-26 02:08:55.460188863 +0000 UTC m=+0.125327957 container create 04df33b6d3849c15536b3fbaf5876549e9c28e577b6c049537606e1a40baa552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-04-26T02:08:55.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:55 smithi029 podman[72823]: 2024-04-26 02:08:55.683015578 +0000 UTC m=+0.348154672 container init 04df33b6d3849c15536b3fbaf5876549e9c28e577b6c049537606e1a40baa552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-26T02:08:55.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:55 smithi029 podman[72823]: 2024-04-26 02:08:55.68625954 +0000 UTC m=+0.351398636 container start 04df33b6d3849c15536b3fbaf5876549e9c28e577b6c049537606e1a40baa552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-26T02:08:55.800 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:55 smithi029 podman[72823]: 2024-04-26 02:08:55.800104841 +0000 UTC m=+0.465243945 container attach 04df33b6d3849c15536b3fbaf5876549e9c28e577b6c049537606e1a40baa552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T02:08:56.192 INFO:journalctl@ceph.osd.1.smithi029.stdout:Apr 26 02:08:55 smithi029 podman[72823]: 2024-04-26 02:08:55.999694378 +0000 UTC m=+0.664833483 container died 04df33b6d3849c15536b3fbaf5876549e9c28e577b6c049537606e1a40baa552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-26T02:08:56.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.1.service' 2024-04-26T02:08:56.683 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:56.683 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-26T02:08:56.683 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-26T02:08:56.684 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.2 2024-04-26T02:08:56.978 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:08:56 smithi188 systemd[1]: Stopping Ceph osd.2 for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:57.391 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:08:56 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2[38071]: 2024-04-26T02:08:56.979+0000 7f528194c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:57.391 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:08:56 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2[38071]: 2024-04-26T02:08:56.979+0000 7f528194c700 -1 osd.2 137 *** Got signal Terminated *** 2024-04-26T02:08:57.391 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:08:56 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2[38071]: 2024-04-26T02:08:56.979+0000 7f528194c700 -1 osd.2 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:02.679 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:09:02 smithi188 podman[73911]: 2024-04-26 02:09:02.410659609 +0000 UTC m=+5.535995669 container died 17999d9593c44814337a93cea3c326b5a5998949df7940c79823deedb4122daf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-26T02:09:03.045 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:09:02 smithi188 podman[73911]: 2024-04-26 02:09:02.796313086 +0000 UTC m=+5.921649191 container cleanup 17999d9593c44814337a93cea3c326b5a5998949df7940c79823deedb4122daf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:09:03.046 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:09:02 smithi188 bash[73911]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2 2024-04-26T02:09:03.361 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:09:03 smithi188 podman[73928]: 2024-04-26 02:09:03.047372494 +0000 UTC m=+0.631311046 container remove 17999d9593c44814337a93cea3c326b5a5998949df7940c79823deedb4122daf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:09:03.641 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:09:03 smithi188 podman[74017]: 2024-04-26 02:09:03.265504269 +0000 UTC m=+0.017943665 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:03.641 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:09:03 smithi188 podman[74017]: 2024-04-26 02:09:03.365221423 +0000 UTC m=+0.117660835 container create ff9dfdb40ddfc6b23147cf1c57cb8f19f154299ab461af91a6384c16993dc6fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-26T02:09:03.641 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:09:03 smithi188 podman[74017]: 2024-04-26 02:09:03.567808783 +0000 UTC m=+0.320248152 container init ff9dfdb40ddfc6b23147cf1c57cb8f19f154299ab461af91a6384c16993dc6fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T02:09:03.642 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:09:03 smithi188 podman[74017]: 2024-04-26 02:09:03.571374423 +0000 UTC m=+0.323813793 container start ff9dfdb40ddfc6b23147cf1c57cb8f19f154299ab461af91a6384c16993dc6fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T02:09:04.110 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:09:03 smithi188 podman[74017]: 2024-04-26 02:09:03.693102746 +0000 UTC m=+0.445542130 container attach ff9dfdb40ddfc6b23147cf1c57cb8f19f154299ab461af91a6384c16993dc6fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:09:04.391 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 26 02:09:04 smithi188 podman[74017]: 2024-04-26 02:09:04.112388754 +0000 UTC m=+0.864828137 container died ff9dfdb40ddfc6b23147cf1c57cb8f19f154299ab461af91a6384c16993dc6fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True) 2024-04-26T02:09:04.709 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.2.service' 2024-04-26T02:09:04.746 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:04.746 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-26T02:09:04.747 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-26T02:09:04.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.3 2024-04-26T02:09:05.318 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:04 smithi006 systemd[1]: Stopping Ceph osd.3 for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:05.819 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:05 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3[50193]: 2024-04-26T02:09:05.372+0000 7f35ebc00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:05.819 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:05 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3[50193]: 2024-04-26T02:09:05.372+0000 7f35ebc00700 -1 osd.3 137 *** Got signal Terminated *** 2024-04-26T02:09:05.820 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:05 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3[50193]: 2024-04-26T02:09:05.372+0000 7f35ebc00700 -1 osd.3 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:06.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:06.116+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:06.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:06 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:06.656+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:07.068 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:06.749+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:07.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:07 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:07.068+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:07.568 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:07 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3[50193]: 2024-04-26T02:09:07.163+0000 7f35e7400700 -1 osd.3 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.080528+0000 front 2024-04-26T02:08:44.081071+0000 (oldest deadline 2024-04-26T02:09:06.980693+0000) 2024-04-26T02:09:07.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:07 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:07.212+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:07.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:07 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:07.637+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:08.087 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:07 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:07.776+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:08.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:08.088+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:08.568 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:08 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3[50193]: 2024-04-26T02:09:08.207+0000 7f35e7400700 -1 osd.3 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.080528+0000 front 2024-04-26T02:08:44.081071+0000 (oldest deadline 2024-04-26T02:09:06.980693+0000) 2024-04-26T02:09:08.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:08 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:08.249+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:08.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:08 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:08.672+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:09.107 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:08.745+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:09.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:09 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:09.107+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:09.568 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:09 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3[50193]: 2024-04-26T02:09:09.179+0000 7f35e7400700 -1 osd.3 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.080528+0000 front 2024-04-26T02:08:44.081071+0000 (oldest deadline 2024-04-26T02:09:06.980693+0000) 2024-04-26T02:09:09.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:09 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:09.245+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:09.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:09 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:09.383+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:09.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:09 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:09.690+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:10.141 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:09 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:09.784+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:10.568 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:10 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3[50193]: 2024-04-26T02:09:10.148+0000 7f35e7400700 -1 osd.3 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.080528+0000 front 2024-04-26T02:08:44.081071+0000 (oldest deadline 2024-04-26T02:09:06.980693+0000) 2024-04-26T02:09:10.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:10 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:10.231+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:10.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:10 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:10.361+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:10.641 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:10 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:10.144+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:10.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:10 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:10.692+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:11.097 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:10 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:10.780+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:11.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:11.097+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:11.546 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:11 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:11.204+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:11.546 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:11 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:11.317+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:11.546 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:11 smithi006 podman[111018]: 2024-04-26 02:09:11.093830769 +0000 UTC m=+5.873179191 container died 2ee7646876b863f5e6805e647e3bd4a565c4a662d2590e404994ca0e363907ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T02:09:11.818 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:11 smithi006 podman[111018]: 2024-04-26 02:09:11.690661613 +0000 UTC m=+6.470010025 container cleanup 2ee7646876b863f5e6805e647e3bd4a565c4a662d2590e404994ca0e363907ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:09:11.818 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:11 smithi006 bash[111018]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3 2024-04-26T02:09:11.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:11 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:11.650+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:12.063 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:11.747+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:12.229 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:11 smithi006 podman[111049]: 2024-04-26 02:09:11.875002437 +0000 UTC m=+0.765244624 container remove 2ee7646876b863f5e6805e647e3bd4a565c4a662d2590e404994ca0e363907ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:09:12.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:12 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:12.064+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:12.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:12 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:12.229+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:12.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:12 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:12.363+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:12.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:12 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:12.641+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:13.037 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:12 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:12.717+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:13.068 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:12 smithi006 podman[111140]: 2024-04-26 02:09:12.678051052 +0000 UTC m=+0.036340235 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:13.069 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:12 smithi006 podman[111140]: 2024-04-26 02:09:12.792845045 +0000 UTC m=+0.151134259 container create 11183d887bed90dc6fc563a83f712234e256b1a8dfeeb41d4c3b191907fcbd73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T02:09:13.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:13.037+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:13.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:13 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:13.202+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:13.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:13 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:13.322+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:13.569 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:13 smithi006 podman[111140]: 2024-04-26 02:09:13.088229904 +0000 UTC m=+0.446519125 container init 11183d887bed90dc6fc563a83f712234e256b1a8dfeeb41d4c3b191907fcbd73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:09:13.569 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:13 smithi006 podman[111140]: 2024-04-26 02:09:13.100227488 +0000 UTC m=+0.458516705 container start 11183d887bed90dc6fc563a83f712234e256b1a8dfeeb41d4c3b191907fcbd73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-26T02:09:13.569 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:13 smithi006 podman[111140]: 2024-04-26 02:09:13.303552697 +0000 UTC m=+0.661841967 container attach 11183d887bed90dc6fc563a83f712234e256b1a8dfeeb41d4c3b191907fcbd73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-26T02:09:13.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:13 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:13.648+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:13.943 INFO:journalctl@ceph.osd.3.smithi006.stdout:Apr 26 02:09:13 smithi006 podman[111140]: 2024-04-26 02:09:13.581104982 +0000 UTC m=+0.939394199 container died 11183d887bed90dc6fc563a83f712234e256b1a8dfeeb41d4c3b191907fcbd73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:09:14.061 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:13.726+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:14.282 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:14 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:14.168+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:14.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.3.service' 2024-04-26T02:09:14.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:14.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:14.440 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:14.441 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-26T02:09:14.441 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-26T02:09:14.441 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.4 2024-04-26T02:09:14.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:14 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:14.351+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:14.766 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:14 smithi029 systemd[1]: Stopping Ceph osd.4 for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:14.767 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:14 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:14.664+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:15.060 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:14.679+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:15.192 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:14 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4[44350]: 2024-04-26T02:09:14.767+0000 7f87b5820700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:15.192 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:14 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4[44350]: 2024-04-26T02:09:14.767+0000 7f87b5820700 -1 osd.4 137 *** Got signal Terminated *** 2024-04-26T02:09:15.192 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:14 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4[44350]: 2024-04-26T02:09:14.767+0000 7f87b5820700 -1 osd.4 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:15.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:15.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:15.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:15.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:15.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:15.304+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:15.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:15 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:15.165+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:15.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:15 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:15.370+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:15.891 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:15.670+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:15.891 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:15.670+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:15.942 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:15 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4[44350]: 2024-04-26T02:09:15.677+0000 7f87ade0c700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.116433+0000 front 2024-04-26T02:08:53.116467+0000 (oldest deadline 2024-04-26T02:09:15.416595+0000) 2024-04-26T02:09:15.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:15 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:15.629+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:15.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:15 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:15.629+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:16.316 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:16.023+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:16.317 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:16.023+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:16.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:16 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:16.140+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:16.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:16 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:16.358+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:16.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:16 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:16.358+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:16.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:16.317+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:16.942 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:16 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4[44350]: 2024-04-26T02:09:16.633+0000 7f87ade0c700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.116433+0000 front 2024-04-26T02:08:53.116467+0000 (oldest deadline 2024-04-26T02:09:15.416595+0000) 2024-04-26T02:09:16.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:16 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:16.636+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:16.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:16 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:16.636+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:17.023 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:16.698+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:17.023 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:16.698+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:17.290 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:17.024+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:17.290 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:17.024+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:17.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:17.097+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:17.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:17.316+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:17.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:17 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:17.316+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:17.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:17.290+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:17.942 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:17 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4[44350]: 2024-04-26T02:09:17.602+0000 7f87ade0c700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.116433+0000 front 2024-04-26T02:08:53.116467+0000 (oldest deadline 2024-04-26T02:09:15.416595+0000) 2024-04-26T02:09:17.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:17 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:17.666+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:17.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:17 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:17.666+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:18.065 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:17.686+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:18.065 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:17.686+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:18.329 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:18.066+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:18.329 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:18.066+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:18.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:18 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:18.071+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:18.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:18 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:18.071+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:18.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:18 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:18.315+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:18.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:18 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:18.315+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:18.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:18.329+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:18.942 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:18 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4[44350]: 2024-04-26T02:09:18.596+0000 7f87ade0c700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.116433+0000 front 2024-04-26T02:08:53.116467+0000 (oldest deadline 2024-04-26T02:09:15.416595+0000) 2024-04-26T02:09:18.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:18 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:18.714+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:18.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:18 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:18.714+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:19.050 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:18.707+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:19.050 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:18.707+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:19.320 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:19 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:19.051+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:19.320 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:19 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:19.051+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:19.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:19 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:19.085+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:19.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:19 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:19.085+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:19.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:19 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:19.308+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:19.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:19 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:19.308+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:19.638 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:19 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:19.302+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:19.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:19 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:19.320+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:19.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:19 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:19.667+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:19.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:19 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:19.667+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:19.943 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:19 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4[44350]: 2024-04-26T02:09:19.639+0000 7f87ade0c700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.116433+0000 front 2024-04-26T02:08:53.116467+0000 (oldest deadline 2024-04-26T02:09:15.416595+0000) 2024-04-26T02:09:20.075 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:19 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:19.741+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:20.075 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:19 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:19.741+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:20.391 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:20 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:20.306+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:20.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:20 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:20.076+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:20.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:20 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:20.076+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:20.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:20 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:20.072+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:20.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:20 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:20.072+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:20.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:20 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:20.302+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:20.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:20 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:20.302+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:20.627 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:20 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:20.301+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:20.628 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:20 smithi029 podman[72929]: 2024-04-26 02:09:20.347146581 +0000 UTC m=+5.708233187 container died 71819c45edc18fdec3fbf044e14fb2fe526373221b7ff1c86d2b5296e07929ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T02:09:20.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:20 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:20.664+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:20.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:20 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:20.664+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:20.943 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:20 smithi029 podman[72929]: 2024-04-26 02:09:20.73678004 +0000 UTC m=+6.097866652 container cleanup 71819c45edc18fdec3fbf044e14fb2fe526373221b7ff1c86d2b5296e07929ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-26T02:09:20.943 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:20 smithi029 bash[72929]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4 2024-04-26T02:09:21.141 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:20 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:20.786+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:21.141 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:20 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:20.786+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:21.142 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:21 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:21.028+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:21.142 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:21 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:21.028+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:21.417 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:21 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:21.265+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:21.417 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:21 smithi029 podman[72946]: 2024-04-26 02:09:21.06645339 +0000 UTC m=+0.716417531 container remove 71819c45edc18fdec3fbf044e14fb2fe526373221b7ff1c86d2b5296e07929ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-26T02:09:21.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:21 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:21.090+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:21.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:21 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:21.090+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:21.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:21 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:21.290+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:21.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:21 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:21.290+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:21.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:21 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:21.304+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:21.679 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:21 smithi029 podman[73040]: 2024-04-26 02:09:21.41825615 +0000 UTC m=+0.104091021 container create 45efce175f7ddcbddebfddc3c8b0e45738427af9513b84fbed1e633872015652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-26T02:09:21.679 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:21 smithi029 podman[73040]: 2024-04-26 02:09:21.332734732 +0000 UTC m=+0.018569581 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:21.680 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:21 smithi029 podman[73040]: 2024-04-26 02:09:21.680743327 +0000 UTC m=+0.366578157 container init 45efce175f7ddcbddebfddc3c8b0e45738427af9513b84fbed1e633872015652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-26T02:09:21.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:21 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:21.682+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:21.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:21 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:21.682+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:21.943 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:21 smithi029 podman[73040]: 2024-04-26 02:09:21.684485752 +0000 UTC m=+0.370320583 container start 45efce175f7ddcbddebfddc3c8b0e45738427af9513b84fbed1e633872015652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-26T02:09:21.943 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:21 smithi029 podman[73040]: 2024-04-26 02:09:21.77277308 +0000 UTC m=+0.458607939 container attach 45efce175f7ddcbddebfddc3c8b0e45738427af9513b84fbed1e633872015652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-26T02:09:22.070 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:21 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:21.795+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:22.070 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:21 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:21.795+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:22.294 INFO:journalctl@ceph.osd.4.smithi029.stdout:Apr 26 02:09:21 smithi029 podman[73040]: 2024-04-26 02:09:21.997120226 +0000 UTC m=+0.682955046 container died 45efce175f7ddcbddebfddc3c8b0e45738427af9513b84fbed1e633872015652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-26T02:09:22.321 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:22 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:22.070+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:22.322 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:22 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:22.070+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:22.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:22 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:22.125+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:22.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:22 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:22.125+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:22.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:22 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:22.277+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:22.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:22 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:22.277+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:22.638 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:22 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:22.315+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:22.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:22 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:22.322+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:22.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.4.service' 2024-04-26T02:09:22.704 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:22.705 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-26T02:09:22.705 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-26T02:09:22.705 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.5 2024-04-26T02:09:22.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:22 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:22.714+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:22.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:22 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:22.714+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:22.989 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:22 smithi188 systemd[1]: Stopping Ceph osd.5 for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:22.989 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:22 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:22.811+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:22.989 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:22 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:22.811+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:22.989 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:22 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:22.989+0000 7f34610ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:22.989 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:22 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:22.989+0000 7f34610ec700 -1 osd.5 137 *** Got signal Terminated *** 2024-04-26T02:09:22.989 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:22 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:22.989+0000 7f34610ec700 -1 osd.5 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:23.331 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:23 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:23.037+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:23.331 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:23 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:23.037+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:23.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:23 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:23.260+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:23.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:23 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:23.260+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:23.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:23 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:23.110+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:23.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:23 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:23.110+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:23.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:23 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:23.110+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:23.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:23 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:23.332+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:23.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:23 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:23.332+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:23.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:23 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:23.327+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:24.076 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:23 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:23.762+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:24.076 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:23 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:23.762+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:24.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:23 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:23.711+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:24.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:23 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:23.711+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:24.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:23 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:23.711+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:24.379 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:24 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:24.077+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:24.380 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:24 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:24.077+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:24.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:24 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:24.292+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:24.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:24 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:24.292+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:24.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:24 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:24.086+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:24.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:24 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:24.086+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:24.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:24 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:24.086+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:24.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:24 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:24.380+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:24.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:24 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:24.380+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:24.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:24 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:24.319+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:24.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:24 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:24.319+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:25.114 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:24 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:24.780+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:25.115 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:24 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:24.780+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:25.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:24 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:24.737+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:25.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:24 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:24.737+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:25.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:24 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:24.737+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:25.391 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:25 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:25.335+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:25.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:25 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:25.335+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:25.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:25 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:25.115+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:25.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:25 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:25.115+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:25.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:25 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:25.115+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:25.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:25 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:25.289+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:25.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:25 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:25.289+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:25.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:25 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:25.289+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:25.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:25 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:25.103+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:25.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:25 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:25.103+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:25.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:25 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:25.103+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:25.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:25 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:25.307+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:25.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:25 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:25.307+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:26.079 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:25 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:25.746+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:26.080 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:25 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:25.746+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:26.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:25 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:25.726+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:26.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:25 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:25.726+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:26.193 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:25 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:25.726+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:26.352 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:26 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:26.080+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:26.352 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:26 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:26.080+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:26.352 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:26 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:26.080+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:26.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:26 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:26.333+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:26.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:26 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:26.333+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:26.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:26 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:26.333+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:26.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:26 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:26.148+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:26.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:26 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:26.148+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:26.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:26 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:26.148+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:26.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:26 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:26.353+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:26.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:26 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:26.353+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:26.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:26 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:26.297+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:26.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:26 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:26.297+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:27.090 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:26 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:26.744+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:27.091 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:26 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:26.744+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:27.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:26 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:26.740+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:27.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:26 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:26.740+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:27.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:26 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:26.740+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:27.387 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:27 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:27.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:27.387 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:27 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:27.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:27.387 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:27 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:27.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:27.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:27 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:27.166+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:27.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:27 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:27.166+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:27.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:27 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:27.166+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:27.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:27 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:27.330+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:27.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:27 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:27.330+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:27.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:27 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:27.330+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:27.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:27 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:27.388+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:27.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:27 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:27.388+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:27.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:27 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:27.260+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:27.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:27 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:27.260+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:28.060 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:27 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:27.701+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.885643+0000 front 2024-04-26T02:08:42.885447+0000 (oldest deadline 2024-04-26T02:09:05.785420+0000) 2024-04-26T02:09:28.060 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:27 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:27.701+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.387453+0000 front 2024-04-26T02:08:50.387512+0000 (oldest deadline 2024-04-26T02:09:15.087545+0000) 2024-04-26T02:09:28.060 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:27 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5[44735]: 2024-04-26T02:09:27.701+0000 7f34596d8700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:01.489162+0000 front 2024-04-26T02:09:01.489186+0000 (oldest deadline 2024-04-26T02:09:26.789301+0000) 2024-04-26T02:09:28.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:27 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:27.709+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:28.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:27 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:27.709+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:28.193 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:27 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:27.709+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:28.354 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:28 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:28.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:28.354 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:28 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:28.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:28.354 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:28 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:28.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:28.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:28 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:28.191+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:28.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:28 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:28.191+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:28.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:28 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:28.191+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:28.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:28 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:28.281+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:28.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:28 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:28.281+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:28.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:28 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:28.281+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:28.641 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:28 smithi188 podman[74135]: 2024-04-26 02:09:28.445874344 +0000 UTC m=+5.546294163 container died e2c65132c774206b3a82a2cc34e961e07b7c93dc8c9b109dd4a7ce5819a58c05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-26T02:09:28.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:28 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:28.355+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:28.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:28 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:28.355+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:28.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:28 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:28.215+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:28.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:28 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:28.215+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:29.018 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:28 smithi188 podman[74135]: 2024-04-26 02:09:28.797243261 +0000 UTC m=+5.897663084 container cleanup e2c65132c774206b3a82a2cc34e961e07b7c93dc8c9b109dd4a7ce5819a58c05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-26T02:09:29.018 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:28 smithi188 bash[74135]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5 2024-04-26T02:09:29.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:28 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:28.732+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:29.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:28 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:28.732+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:29.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:28 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:28.732+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:29.345 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:29 smithi188 podman[74153]: 2024-04-26 02:09:29.073561601 +0000 UTC m=+0.631691331 container remove e2c65132c774206b3a82a2cc34e961e07b7c93dc8c9b109dd4a7ce5819a58c05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-26T02:09:29.345 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:29 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:29.019+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:29.345 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:29 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:29.019+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:29.346 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:29 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:29.019+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:29.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:29 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:29.329+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:29.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:29 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:29.329+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:29.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:29 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:29.329+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:29.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:29 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:29.168+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:29.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:29 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:29.168+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:29.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:29 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:29.168+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:29.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:29 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:29.346+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:29.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:29 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:29.346+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:29.642 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:29 smithi188 podman[74245]: 2024-04-26 02:09:29.357598685 +0000 UTC m=+0.096829943 container create 0f86ff7365c8f10c109fe4afedc7f2fb8c26cb42193729b0b1371b8dc143aa95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-26T02:09:29.642 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:29 smithi188 podman[74245]: 2024-04-26 02:09:29.278388062 +0000 UTC m=+0.017619350 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:29.642 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:29 smithi188 podman[74245]: 2024-04-26 02:09:29.560219726 +0000 UTC m=+0.299450991 container init 0f86ff7365c8f10c109fe4afedc7f2fb8c26cb42193729b0b1371b8dc143aa95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:09:29.642 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:29 smithi188 podman[74245]: 2024-04-26 02:09:29.56333779 +0000 UTC m=+0.302569057 container start 0f86ff7365c8f10c109fe4afedc7f2fb8c26cb42193729b0b1371b8dc143aa95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-26T02:09:29.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:29 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:29.253+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:29.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:29 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:29.253+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:30.053 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:29 smithi188 podman[74245]: 2024-04-26 02:09:29.661076392 +0000 UTC m=+0.400307650 container attach 0f86ff7365c8f10c109fe4afedc7f2fb8c26cb42193729b0b1371b8dc143aa95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:09:30.053 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 26 02:09:29 smithi188 podman[74245]: 2024-04-26 02:09:29.856244863 +0000 UTC m=+0.595476125 container died 0f86ff7365c8f10c109fe4afedc7f2fb8c26cb42193729b0b1371b8dc143aa95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-26T02:09:30.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:29 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:29.735+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:30.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:29 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:29.735+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:30.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:29 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:29.735+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:30.391 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:30 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:30.304+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:30.391 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:30 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:30.304+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:30.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:30 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:30.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:30.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:30 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:30.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:30.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:30 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:30.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:30.422 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.5.service' 2024-04-26T02:09:30.459 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:30.459 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-26T02:09:30.459 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-26T02:09:30.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.6 2024-04-26T02:09:30.541 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:30 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:30.160+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:30.541 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:30 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:30.160+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:30.541 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:30 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:30.160+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:30.541 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:30 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:30.318+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:30.542 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:30 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:30.318+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:30.542 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:30 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:30.318+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:30.688 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:30 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:30.209+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:30.688 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:30 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:30.209+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:30.818 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:30 smithi006 systemd[1]: Stopping Ceph osd.6 for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:30.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:30 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:30.688+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:30.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:30 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:30.688+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:30.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:30 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:30.688+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:31.333 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:31 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:31.039+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:31.333 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:31 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:31.039+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:31.333 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:31 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:31.039+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:31.348 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:31 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:31.074+0000 7fa7e9800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:31.348 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:31 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:31.074+0000 7fa7e9800700 -1 osd.6 137 *** Got signal Terminated *** 2024-04-26T02:09:31.349 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:31 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:31.074+0000 7fa7e9800700 -1 osd.6 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:31.349 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:31 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:31.112+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:31.349 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:31 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:31.112+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:31.349 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:31 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:31.112+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:31.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:31 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:31.334+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:31.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:31 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:31.334+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:31.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:31 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:31.193+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:31.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:31 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:31.193+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:31.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:31 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:31.347+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:31.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:31 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:31.347+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:31.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:31 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:31.347+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:31.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:31 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:31.347+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:32.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:31 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:31.703+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:32.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:31 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:31.703+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:32.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:31 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:31.703+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:32.348 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:32 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:32.010+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:32.348 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:32 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:32.010+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:32.348 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:32 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:32.010+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:32.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:32 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:32.152+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:32.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:32 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:32.152+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:32.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:32 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:32.152+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:32.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:32 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:32.370+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:32.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:32 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:32.370+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:32.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:32 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:32.370+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:32.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:32 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:32.370+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:32.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:32 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:32.349+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:32.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:32 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:32.349+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:32.670 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:32 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:32.197+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:32.670 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:32 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:32.197+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:32.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:32 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:32.670+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:32.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:32 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:32.670+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:32.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:32 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:32.670+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:33.365 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:33 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:33.046+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:33.365 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:33 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:33.046+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:33.366 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:33 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:33.046+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:33.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:33 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:33.182+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:33.568 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:33 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:33.182+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:33.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:33 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:33.182+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:33.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:33 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:33.182+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.589363+0000 front 2024-04-26T02:09:09.589540+0000 (oldest deadline 2024-04-26T02:09:33.089571+0000) 2024-04-26T02:09:33.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:33 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:33.332+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:33.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:33 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:33.332+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:33.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:33 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:33.332+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:33.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:33 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:33.332+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:33.620 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:33 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:33.390+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:33.620 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:33 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:33.390+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:33.631 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:33 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:33.232+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:33.631 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:33 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:33.232+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:33.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:33 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:33.631+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:33.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:33 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:33.631+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:33.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:33 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:33.631+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:34.359 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:34 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:34.011+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:34.359 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:34 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:34.011+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:34.360 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:34 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:34.011+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:34.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:34 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:34.311+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:34.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:34 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:34.311+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:34.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:34 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:34.311+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:34.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:34 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:34.311+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:34.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:34 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:34.200+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:34.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:34 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:34.200+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:34.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:34 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:34.200+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:34.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:34 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:34.200+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.589363+0000 front 2024-04-26T02:09:09.589540+0000 (oldest deadline 2024-04-26T02:09:33.089571+0000) 2024-04-26T02:09:34.582 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:34 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:34.224+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:34.582 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:34 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:34.224+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:34.582 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:34 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:34.224+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.887772+0000 front 2024-04-26T02:09:09.887848+0000 (oldest deadline 2024-04-26T02:09:33.888590+0000) 2024-04-26T02:09:34.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:34 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:34.360+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:34.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:34 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:34.360+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:34.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:34 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:34.360+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:34.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:34 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:34.582+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:34.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:34 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:34.582+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:34.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:34 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:34.582+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:35.335 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:35 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:35.057+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:35.335 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:35 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:35.057+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:35.335 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:35 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:35.057+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:35.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:35.309+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:35.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:35.309+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:35.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:35.309+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:35.569 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:35.309+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:35.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:35.248+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:41.577134+0000 front 2024-04-26T02:08:41.577726+0000 (oldest deadline 2024-04-26T02:09:06.276966+0000) 2024-04-26T02:09:35.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:35.248+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.281197+0000 front 2024-04-26T02:08:53.281396+0000 (oldest deadline 2024-04-26T02:09:17.882573+0000) 2024-04-26T02:09:35.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:35.248+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:57.883848+0000 front 2024-04-26T02:08:57.883952+0000 (oldest deadline 2024-04-26T02:09:22.584082+0000) 2024-04-26T02:09:35.569 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:35 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6[57024]: 2024-04-26T02:09:35.248+0000 7fa7e5000700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.589363+0000 front 2024-04-26T02:09:09.589540+0000 (oldest deadline 2024-04-26T02:09:33.089571+0000) 2024-04-26T02:09:35.587 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:35 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:35.254+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:35.588 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:35 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:35.254+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:35.588 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:35 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:35.254+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.887772+0000 front 2024-04-26T02:09:09.887848+0000 (oldest deadline 2024-04-26T02:09:33.888590+0000) 2024-04-26T02:09:35.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:35 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:35.336+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:35.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:35 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:35.336+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:35.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:35 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:35.336+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:35.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:35 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:35.587+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:35.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:35 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:35.587+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:35.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:35 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:35.587+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:35.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:35 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:35.587+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:36.350 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:36 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:36.030+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:36.350 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:36 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:36.030+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:36.350 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:36 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:36.030+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:36.602 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:36 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:36.273+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:36.603 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:36 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:36.273+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:36.603 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:36 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:36.273+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.887772+0000 front 2024-04-26T02:09:09.887848+0000 (oldest deadline 2024-04-26T02:09:33.888590+0000) 2024-04-26T02:09:36.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:36 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:36.351+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:36.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:36 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:36.351+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:36.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:36 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:36.351+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:36.798 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:36 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:36.338+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:36.799 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:36 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:36.338+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:36.799 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:36 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:36.338+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:36.799 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:36 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:36.338+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:36.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:36 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:36.603+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:36.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:36 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:36.603+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:36.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:36 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:36.603+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:36.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:36 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:36.603+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:37.068 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:36 smithi006 podman[111246]: 2024-04-26 02:09:36.799033356 +0000 UTC m=+5.858037727 container died 6655478b65ebbcb1597d8d947fa4bd7fc1d62272f07ec83b5e1530ba12b4e1a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD) 2024-04-26T02:09:37.328 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:37 smithi006 podman[111246]: 2024-04-26 02:09:37.275780735 +0000 UTC m=+6.334785099 container cleanup 6655478b65ebbcb1597d8d947fa4bd7fc1d62272f07ec83b5e1530ba12b4e1a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:09:37.329 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:37 smithi006 bash[111246]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6 2024-04-26T02:09:37.366 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:37 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:37.039+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:37.366 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:37 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:37.039+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:37.367 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:37 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:37.039+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:37.575 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:37 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:37.315+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:37.575 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:37 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:37.315+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:37.575 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:37 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:37.315+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.887772+0000 front 2024-04-26T02:09:09.887848+0000 (oldest deadline 2024-04-26T02:09:33.888590+0000) 2024-04-26T02:09:37.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:37 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:37.367+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:37.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:37 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:37.367+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:37.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:37 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:37.367+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:37.818 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:37 smithi006 podman[111265]: 2024-04-26 02:09:37.494243116 +0000 UTC m=+0.685495749 container remove 6655478b65ebbcb1597d8d947fa4bd7fc1d62272f07ec83b5e1530ba12b4e1a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:09:37.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:37 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:37.328+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:37.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:37 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:37.328+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:37.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:37 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:37.328+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:37.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:37 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:37.328+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:37.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:37 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:37.575+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:37.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:37 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:37.575+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:37.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:37 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:37.575+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:37.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:37 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:37.575+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:38.336 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:38 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:38.045+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:38.336 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:38 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:38.045+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:38.336 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:38 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:38.045+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:38.596 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:38 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:38.292+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:38.596 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:38 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:38.292+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:38.596 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:38 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:38.292+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.887772+0000 front 2024-04-26T02:09:09.887848+0000 (oldest deadline 2024-04-26T02:09:33.888590+0000) 2024-04-26T02:09:38.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:38 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:38.337+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:38.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:38 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:38.337+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:38.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:38 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:38.337+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:38.657 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:38 smithi006 podman[111353]: 2024-04-26 02:09:38.259228187 +0000 UTC m=+0.034199949 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:38.657 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:38 smithi006 podman[111353]: 2024-04-26 02:09:38.370060256 +0000 UTC m=+0.145032045 container create be8a640a79e1be772fbd76f30408a588a9a876c0b29b03ad6c277c200f08b016 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:09:38.657 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:38 smithi006 podman[111353]: 2024-04-26 02:09:38.657123758 +0000 UTC m=+0.432095563 container init be8a640a79e1be772fbd76f30408a588a9a876c0b29b03ad6c277c200f08b016 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:09:38.658 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:38 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:38.335+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:38.658 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:38 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:38.335+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:38.658 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:38 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:38.335+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:38.658 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:38 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:38.335+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:38.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:38 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:38.595+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:38.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:38 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:38.595+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:38.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:38 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:38.595+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:38.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:38 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:38.595+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:39.068 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:38 smithi006 podman[111353]: 2024-04-26 02:09:38.670526514 +0000 UTC m=+0.445498287 container start be8a640a79e1be772fbd76f30408a588a9a876c0b29b03ad6c277c200f08b016 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T02:09:39.069 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:38 smithi006 podman[111353]: 2024-04-26 02:09:38.747466459 +0000 UTC m=+0.522438278 container attach be8a640a79e1be772fbd76f30408a588a9a876c0b29b03ad6c277c200f08b016 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0) 2024-04-26T02:09:39.300 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:39 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:39.023+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:39.300 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:39 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:39.023+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:39.301 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:39 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:39.023+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:39.454 INFO:journalctl@ceph.osd.6.smithi006.stdout:Apr 26 02:09:39 smithi006 podman[111353]: 2024-04-26 02:09:39.145604769 +0000 UTC m=+0.920576628 container died be8a640a79e1be772fbd76f30408a588a9a876c0b29b03ad6c277c200f08b016 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-26T02:09:39.455 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:39 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:39.371+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:39.455 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:39 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:39.371+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:39.455 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:39 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:39.371+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:39.455 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:39 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:39.371+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:39.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:39 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:39.301+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:39.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:39 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:39.301+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:39.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:39 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:39.301+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:39.692 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:39 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:39.551+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:39.692 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:39 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:39.551+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:39.692 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:39 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:39.551+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:39.692 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:39 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:39.551+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:39.693 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:39 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:39.341+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:39.693 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:39 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:39.341+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:39.693 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:39 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:39.341+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.887772+0000 front 2024-04-26T02:09:09.887848+0000 (oldest deadline 2024-04-26T02:09:33.888590+0000) 2024-04-26T02:09:39.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.6.service' 2024-04-26T02:09:40.036 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:40.036 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-26T02:09:40.037 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-26T02:09:40.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.7 2024-04-26T02:09:40.303 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:40 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:40.052+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:40.303 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:40 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:40.052+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:40.303 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:40 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:40.052+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:40.337 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:40 smithi029 systemd[1]: Stopping Ceph osd.7 for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:40.337 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:40 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:40.302+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:40.337 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:40 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:40.302+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:40.337 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:40 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:40.302+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.887772+0000 front 2024-04-26T02:09:09.887848+0000 (oldest deadline 2024-04-26T02:09:33.888590+0000) 2024-04-26T02:09:40.337 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:40 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:40.337+0000 7f9044947700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:40.338 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:40 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:40.337+0000 7f9044947700 -1 osd.7 137 *** Got signal Terminated *** 2024-04-26T02:09:40.338 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:40 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:40.337+0000 7f9044947700 -1 osd.7 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:40.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:40 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:40.304+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:40.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:40 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:40.304+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:40.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:40 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:40.304+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:40.692 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:40 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:40.579+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:40.692 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:40 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:40.579+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:40.692 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:40 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:40.579+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:40.692 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:40 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:40.579+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:40.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:40 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:40.394+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:40.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:40 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:40.394+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:40.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:40 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:40.394+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:40.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:40 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:40.394+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:41.391 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:41 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:41.282+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:41.391 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:41 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:41.282+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:41.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:41 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:41.282+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:41.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:41 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:41.101+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:41.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:41 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:41.101+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:41.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:41 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:41.101+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:41.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:41 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:41.332+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:41.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:41 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:41.332+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:41.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:41 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:41.332+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.887772+0000 front 2024-04-26T02:09:09.887848+0000 (oldest deadline 2024-04-26T02:09:33.888590+0000) 2024-04-26T02:09:41.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:41 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:41.540+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:41.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:41 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:41.540+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:41.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:41 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:41.540+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:41.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:41 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:41.540+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:41.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:41 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:41.407+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:41.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:41 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:41.407+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:41.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:41 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:41.407+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:41.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:41 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:41.407+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:42.391 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:42 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:42.242+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:42.391 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:42 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:42.242+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:42.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:42 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:42.242+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:42.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:42 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:42.107+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:42.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:42 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:42.107+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:42.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:42 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:42.107+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:42.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:42 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:42.332+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:42.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:42 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:42.332+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:42.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:42 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:42.332+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.887772+0000 front 2024-04-26T02:09:09.887848+0000 (oldest deadline 2024-04-26T02:09:33.888590+0000) 2024-04-26T02:09:42.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:42 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:42.524+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:42.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:42 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:42.524+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:42.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:42 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:42.524+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:42.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:42 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:42.524+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:42.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:42 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:42.524+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:42.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:42 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:42.438+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:42.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:42 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:42.438+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:42.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:42 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:42.438+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:42.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:42 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:42.438+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:42.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:42 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:42.438+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:43.391 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:43 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:43.278+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:43.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:43 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:43.278+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:43.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:43 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:43.278+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:43.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:43 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:43.099+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:43.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:43 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:43.099+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:43.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:43 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:43.099+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:43.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:43 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:43.370+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:43.693 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:43 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:43.370+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:43.693 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:43 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:43.370+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.887772+0000 front 2024-04-26T02:09:09.887848+0000 (oldest deadline 2024-04-26T02:09:33.888590+0000) 2024-04-26T02:09:43.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:43 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:43.514+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:43.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:43 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:43.514+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:43.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:43 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:43.514+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:43.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:43 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:43.514+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:43.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:43 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:43.514+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:43.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:43 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:43.426+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:43.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:43 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:43.426+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:43.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:43 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:43.426+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:43.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:43 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:43.426+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:43.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:43 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:43.426+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:44.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:44 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:44.078+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:44.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:44 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:44.078+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:44.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:44 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:44.078+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:44.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:44 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:44.281+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:44.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:44 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:44.281+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:44.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:44 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:44.281+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:44.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:44 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:44.281+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.961233+0000 front 2024-04-26T02:09:18.961251+0000 (oldest deadline 2024-04-26T02:09:44.261391+0000) 2024-04-26T02:09:44.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:44 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:44.341+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:44.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:44 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:44.341+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:44.692 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:44 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:44.341+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.887772+0000 front 2024-04-26T02:09:09.887848+0000 (oldest deadline 2024-04-26T02:09:33.888590+0000) 2024-04-26T02:09:44.693 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:44 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:44.341+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.990954+0000 front 2024-04-26T02:09:18.990977+0000 (oldest deadline 2024-04-26T02:09:44.291141+0000) 2024-04-26T02:09:44.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:44 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:44.473+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:44.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:44 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:44.473+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:44.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:44 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:44.473+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:44.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:44 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:44.473+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:44.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:44 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:44.473+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:44.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:44 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:44.385+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:44.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:44 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:44.385+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:44.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:44 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:44.385+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:44.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:44 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:44.385+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:44.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:44 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:44.385+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:45.391 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:45.248+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:45.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:45.248+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:45.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:45.248+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:45.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:45.248+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.961233+0000 front 2024-04-26T02:09:18.961251+0000 (oldest deadline 2024-04-26T02:09:44.261391+0000) 2024-04-26T02:09:45.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:45.090+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:45.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:45.090+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:45.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:45.090+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:45.393 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:45 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:45.090+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:45.593 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:45.330+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.385588+0000 front 2024-04-26T02:08:53.385563+0000 (oldest deadline 2024-04-26T02:09:19.285730+0000) 2024-04-26T02:09:45.594 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:45.330+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:08:59.286385+0000 front 2024-04-26T02:08:59.286287+0000 (oldest deadline 2024-04-26T02:09:23.986350+0000) 2024-04-26T02:09:45.594 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:45.330+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.887772+0000 front 2024-04-26T02:09:09.887848+0000 (oldest deadline 2024-04-26T02:09:33.888590+0000) 2024-04-26T02:09:45.594 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7[51315]: 2024-04-26T02:09:45.330+0000 7f9040f3b700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.990954+0000 front 2024-04-26T02:09:18.990977+0000 (oldest deadline 2024-04-26T02:09:44.291141+0000) 2024-04-26T02:09:45.594 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:45.518+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:45.594 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:45.518+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:45.594 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:45.518+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:45.595 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:45.518+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:45.595 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:45 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:45.518+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:45.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:45.435+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:45.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:45.435+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:45.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:45.435+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:45.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:45.435+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:45.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:45 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:45.435+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:45.899 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:45 smithi029 podman[73147]: 2024-04-26 02:09:45.799473023 +0000 UTC m=+5.565246950 container died 1e3e25e668313c97d795b23280d42903147f4deb96596f54b2c05f736b6508cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:09:46.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:46 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:46.119+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:46.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:46 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:46.119+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:46.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:46 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:46.119+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:46.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:46 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:46.119+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:46.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:46 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:46.248+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:46.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:46 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:46.248+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:46.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:46 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:46.248+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:46.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:46 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:46.248+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.961233+0000 front 2024-04-26T02:09:18.961251+0000 (oldest deadline 2024-04-26T02:09:44.261391+0000) 2024-04-26T02:09:46.524 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:46 smithi029 podman[73147]: 2024-04-26 02:09:46.218495873 +0000 UTC m=+5.984269800 container cleanup 1e3e25e668313c97d795b23280d42903147f4deb96596f54b2c05f736b6508cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:09:46.524 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:46 smithi029 bash[73147]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7 2024-04-26T02:09:46.524 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:46 smithi029 podman[73165]: 2024-04-26 02:09:46.42085158 +0000 UTC m=+0.619146637 container remove 1e3e25e668313c97d795b23280d42903147f4deb96596f54b2c05f736b6508cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T02:09:46.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:46 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:46.435+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:46.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:46 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:46.435+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:46.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:46 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:46.435+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:46.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:46 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:46.435+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:46.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:46 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:46.435+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:46.843 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:46 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:46.523+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:46.844 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:46 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:46.523+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:46.844 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:46 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:46.523+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:46.844 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:46 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:46.523+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:46.844 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:46 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:46.523+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:47.120 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:46 smithi029 podman[73268]: 2024-04-26 02:09:46.749744208 +0000 UTC m=+0.020941638 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:47.120 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:46 smithi029 podman[73268]: 2024-04-26 02:09:46.848205337 +0000 UTC m=+0.119402757 container create 736ae297acee80b9c9b17bbab5f39be13f6ff4a64cc3f7130dbc5d47a23ea7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:09:47.121 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:47 smithi029 podman[73268]: 2024-04-26 02:09:47.12156644 +0000 UTC m=+0.392763857 container init 736ae297acee80b9c9b17bbab5f39be13f6ff4a64cc3f7130dbc5d47a23ea7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:09:47.429 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:47 smithi029 podman[73268]: 2024-04-26 02:09:47.125117524 +0000 UTC m=+0.396314938 container start 736ae297acee80b9c9b17bbab5f39be13f6ff4a64cc3f7130dbc5d47a23ea7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T02:09:47.430 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:47 smithi029 podman[73268]: 2024-04-26 02:09:47.213405378 +0000 UTC m=+0.484602798 container attach 736ae297acee80b9c9b17bbab5f39be13f6ff4a64cc3f7130dbc5d47a23ea7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True) 2024-04-26T02:09:47.641 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:47 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:47.166+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:47.641 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:47 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:47.166+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:47.641 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:47 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:47.166+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:47.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:47 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:47.166+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:47.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:47 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:47.255+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:47.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:47 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:47.255+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:47.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:47 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:47.255+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:47.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:47 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:47.255+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.961233+0000 front 2024-04-26T02:09:18.961251+0000 (oldest deadline 2024-04-26T02:09:44.261391+0000) 2024-04-26T02:09:47.694 INFO:journalctl@ceph.osd.7.smithi029.stdout:Apr 26 02:09:47 smithi029 podman[73268]: 2024-04-26 02:09:47.431368358 +0000 UTC m=+0.702565775 container died 736ae297acee80b9c9b17bbab5f39be13f6ff4a64cc3f7130dbc5d47a23ea7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-26T02:09:47.694 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:47 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:47.545+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:47.695 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:47 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:47.545+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:47.695 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:47 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:47.545+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:47.695 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:47 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:47.545+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:47.695 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:47 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:47.545+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:47.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:47 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:47.465+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:47.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:47 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:47.465+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:47.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:47 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:47.465+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:47.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:47 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:47.465+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:47.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:47 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:47.465+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:48.149 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.7.service' 2024-04-26T02:09:48.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:48.197 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-26T02:09:48.198 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-26T02:09:48.198 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.8 2024-04-26T02:09:48.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:48.139+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:48.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:48.139+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:48.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:48.139+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:48.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:48.139+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:48.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:48.229+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:48.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:48.229+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:48.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:48.229+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:48.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:48.229+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.961233+0000 front 2024-04-26T02:09:18.961251+0000 (oldest deadline 2024-04-26T02:09:44.261391+0000) 2024-04-26T02:09:48.392 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:48 smithi188 systemd[1]: Stopping Ceph osd.8 for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:48.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:48 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:48.448+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:48.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:48 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:48.448+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:48.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:48 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:48.448+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:48.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:48 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:48.448+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:48.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:48 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:48.448+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:48.891 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:48.492+0000 7fd4af65b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:48.891 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:48.492+0000 7fd4af65b700 -1 osd.8 137 *** Got signal Terminated *** 2024-04-26T02:09:48.891 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:48 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:48.492+0000 7fd4af65b700 -1 osd.8 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:48.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:48.514+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:48.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:48.514+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:48.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:48.514+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:48.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:48.514+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:48.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:48.514+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:49.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:49.251+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:49.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:49.251+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:49.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:49.251+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:49.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:49.251+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.961233+0000 front 2024-04-26T02:09:18.961251+0000 (oldest deadline 2024-04-26T02:09:44.261391+0000) 2024-04-26T02:09:49.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:49.158+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:49.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:49.158+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:49.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:49.158+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:49.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:49 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:49.158+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:49.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:49 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:49.476+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:49.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:49 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:49.476+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:49.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:49 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:49.476+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:49.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:49 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:49.476+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:49.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:49 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:49.476+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:49.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:49 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:49.509+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:49.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:49 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:49.509+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:49.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:49 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:49.509+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:49.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:49 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:49.509+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:49.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:49 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:49.509+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:50.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:50.291+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:50.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:50.291+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:50.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:50.291+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:50.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:50.291+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.961233+0000 front 2024-04-26T02:09:18.961251+0000 (oldest deadline 2024-04-26T02:09:44.261391+0000) 2024-04-26T02:09:50.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:50.203+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:50.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:50.203+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:50.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:50.203+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:50.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:50 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:50.203+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:50.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:50 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:50.507+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:50.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:50 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:50.507+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:50.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:50 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:50.507+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:50.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:50 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:50.507+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:50.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:50 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:50.507+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:50.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:50 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:50.468+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:50.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:50 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:50.468+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:50.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:50 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:50.468+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:50.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:50 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:50.468+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:50.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:50 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:50.468+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:50.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:50 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:50.468+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:09:51.641 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:51 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:51.167+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:51.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:51 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:51.167+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:51.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:51 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:51.167+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:51.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:51 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:51.167+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:51.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:51 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:51.255+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:51.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:51 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:51.255+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:51.643 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:51 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:51.255+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:51.643 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:51 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:51.255+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.961233+0000 front 2024-04-26T02:09:18.961251+0000 (oldest deadline 2024-04-26T02:09:44.261391+0000) 2024-04-26T02:09:51.643 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:51 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:51.255+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:27.762177+0000 front 2024-04-26T02:09:27.762199+0000 (oldest deadline 2024-04-26T02:09:51.162883+0000) 2024-04-26T02:09:51.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:51 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:51.509+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:51.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:51 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:51.509+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:51.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:51 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:51.509+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:51.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:51 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:51.509+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:51.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:51 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:51.509+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:51.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:51 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:51.463+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:51.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:51 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:51.463+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:51.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:51 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:51.463+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:51.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:51 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:51.463+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:51.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:51 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:51.463+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:51.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:51 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:51.463+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:09:52.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:52 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:52.212+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:52.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:52 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:52.212+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:52.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:52 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:52.212+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:52.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:52 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:52.212+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.961233+0000 front 2024-04-26T02:09:18.961251+0000 (oldest deadline 2024-04-26T02:09:44.261391+0000) 2024-04-26T02:09:52.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:52 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:52.212+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:27.762177+0000 front 2024-04-26T02:09:27.762199+0000 (oldest deadline 2024-04-26T02:09:51.162883+0000) 2024-04-26T02:09:52.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:52 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:52.206+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:52.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:52 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:52.206+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:52.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:52 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:52.206+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:52.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:52 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:52.206+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:52.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:52 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:52.206+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:09:52.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:52 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:52.555+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:52.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:52 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:52.555+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:52.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:52 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:52.555+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:52.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:52 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:52.555+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:52.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:52 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:52.555+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:52.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:52 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:52.555+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.788273+0000 front 2024-04-26T02:09:26.788501+0000 (oldest deadline 2024-04-26T02:09:52.088510+0000) 2024-04-26T02:09:52.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:52 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:52.466+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:52.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:52 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:52.466+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:52.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:52 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:52.466+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:52.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:52 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:52.466+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:52.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:52 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:52.466+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:52.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:52 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:52.466+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:09:53.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:53 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:53.217+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:51.654945+0000 front 2024-04-26T02:08:51.655047+0000 (oldest deadline 2024-04-26T02:09:15.155078+0000) 2024-04-26T02:09:53.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:53 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:53.217+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.957215+0000 front 2024-04-26T02:09:00.957236+0000 (oldest deadline 2024-04-26T02:09:22.657377+0000) 2024-04-26T02:09:53.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:53 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:53.217+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.959151+0000 front 2024-04-26T02:09:07.958953+0000 (oldest deadline 2024-04-26T02:09:33.858941+0000) 2024-04-26T02:09:53.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:53 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:53.217+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.961233+0000 front 2024-04-26T02:09:18.961251+0000 (oldest deadline 2024-04-26T02:09:44.261391+0000) 2024-04-26T02:09:53.642 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:53 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8[51428]: 2024-04-26T02:09:53.217+0000 7fd4a7c47700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:27.762177+0000 front 2024-04-26T02:09:27.762199+0000 (oldest deadline 2024-04-26T02:09:51.162883+0000) 2024-04-26T02:09:53.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:53 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:53.163+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:53.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:53 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:53.163+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:53.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:53 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:53.163+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:53.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:53 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:53.163+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:53.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:53 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:53.163+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:09:53.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:53.536+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:53.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:53.536+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:53.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:53.536+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:53.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:53.536+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:53.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:53.536+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:53.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:53 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:53.536+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.788273+0000 front 2024-04-26T02:09:26.788501+0000 (oldest deadline 2024-04-26T02:09:52.088510+0000) 2024-04-26T02:09:53.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:53 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:53.477+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:53.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:53 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:53.477+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:53.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:53 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:53.477+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:53.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:53 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:53.477+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:53.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:53 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:53.477+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:53.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:53 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:53.477+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:09:54.315 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:53 smithi188 podman[74362]: 2024-04-26 02:09:53.996790997 +0000 UTC m=+5.608373920 container died ae578cb99da6bcea1f676509f3d53c794aabd19a4ad09eb604e0d0e9f48df0be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-04-26T02:09:54.316 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:54 smithi188 podman[74362]: 2024-04-26 02:09:54.316865697 +0000 UTC m=+5.928448609 container cleanup ae578cb99da6bcea1f676509f3d53c794aabd19a4ad09eb604e0d0e9f48df0be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True) 2024-04-26T02:09:54.316 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:54 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:54.131+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:54.316 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:54 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:54.131+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:54.316 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:54 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:54.131+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:54.316 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:54 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:54.131+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:54.316 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:54 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:54.131+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:09:54.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:54 smithi188 bash[74362]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8 2024-04-26T02:09:54.641 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:54 smithi188 podman[74379]: 2024-04-26 02:09:54.502343535 +0000 UTC m=+0.503955991 container remove ae578cb99da6bcea1f676509f3d53c794aabd19a4ad09eb604e0d0e9f48df0be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-26T02:09:54.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:54 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:54.515+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:54.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:54 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:54.515+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:54.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:54 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:54.515+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:54.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:54 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:54.515+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:54.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:54 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:54.515+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:54.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:54 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:54.515+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.788273+0000 front 2024-04-26T02:09:26.788501+0000 (oldest deadline 2024-04-26T02:09:52.088510+0000) 2024-04-26T02:09:54.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:54 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:54.458+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:54.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:54 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:54.458+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:54.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:54 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:54.458+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:54.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:54 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:54.458+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:54.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:54 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:54.458+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:54.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:54 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:54.458+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:09:55.180 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:54 smithi188 podman[74472]: 2024-04-26 02:09:54.834281599 +0000 UTC m=+0.017634797 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:55.180 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:54 smithi188 podman[74472]: 2024-04-26 02:09:54.953054742 +0000 UTC m=+0.136407921 container create 158deb1d74f47463cc897c03e361a007d6dc6b68fac615f7bab8921aba440c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-26T02:09:55.180 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:55 smithi188 podman[74472]: 2024-04-26 02:09:55.181387477 +0000 UTC m=+0.364740657 container init 158deb1d74f47463cc897c03e361a007d6dc6b68fac615f7bab8921aba440c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-26T02:09:55.180 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:55 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:55.092+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:55.181 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:55 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:55.092+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:55.181 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:55 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:55.092+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:55.181 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:55 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:55.092+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:55.181 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:55 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:55.092+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:09:55.476 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:55 smithi188 podman[74472]: 2024-04-26 02:09:55.184832156 +0000 UTC m=+0.368185337 container start 158deb1d74f47463cc897c03e361a007d6dc6b68fac615f7bab8921aba440c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-26T02:09:55.476 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:55 smithi188 podman[74472]: 2024-04-26 02:09:55.27307535 +0000 UTC m=+0.456428540 container attach 158deb1d74f47463cc897c03e361a007d6dc6b68fac615f7bab8921aba440c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T02:09:55.766 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:55 smithi188 podman[74472]: 2024-04-26 02:09:55.477611795 +0000 UTC m=+0.660964974 container died 158deb1d74f47463cc897c03e361a007d6dc6b68fac615f7bab8921aba440c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-26T02:09:55.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:55.550+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:55.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:55.550+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:55.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:55.550+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:55.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:55.550+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:55.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:55.550+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:55.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:55 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:55.550+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.788273+0000 front 2024-04-26T02:09:26.788501+0000 (oldest deadline 2024-04-26T02:09:52.088510+0000) 2024-04-26T02:09:55.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:55 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:55.462+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:55.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:55 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:55.462+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:55.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:55 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:55.462+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:55.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:55 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:55.462+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:55.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:55 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:55.462+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:55.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:55 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:55.462+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:09:56.024 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 26 02:09:56 smithi188 podman[74497]: 2024-04-26 02:09:56.00994614 +0000 UTC m=+0.527037480 container remove 158deb1d74f47463cc897c03e361a007d6dc6b68fac615f7bab8921aba440c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T02:09:56.029 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.8.service' 2024-04-26T02:09:56.068 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:56.068 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-26T02:09:56.068 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-26T02:09:56.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.9 2024-04-26T02:09:56.278 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:56 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:56.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:56.279 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:56 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:56.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:56.279 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:56 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:56.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:56.279 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:56 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:56.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:56.279 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:56 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:56.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:09:56.545 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:56 smithi006 systemd[1]: Stopping Ceph osd.9 for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:56.546 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:56 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:56.544+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:56.692 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:56 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:56.437+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:56.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:56 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:56.437+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:56.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:56 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:56.437+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:56.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:56 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:56.437+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:56.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:56 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:56.437+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:56.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:56 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:56.437+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:09:56.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:56 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:56.544+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:56.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:56 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:56.544+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:56.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:56 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:56.544+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:56.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:56 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:56.544+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:56.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:56 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:56.544+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.788273+0000 front 2024-04-26T02:09:26.788501+0000 (oldest deadline 2024-04-26T02:09:52.088510+0000) 2024-04-26T02:09:56.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:56 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:56.659+0000 7f3224800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:56.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:56 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:56.659+0000 7f3224800700 -1 osd.9 137 *** Got signal Terminated *** 2024-04-26T02:09:56.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:56 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:56.659+0000 7f3224800700 -1 osd.9 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:57.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:57 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:57.122+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:57.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:57 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:57.122+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:57.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:57 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:57.122+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:57.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:57 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:57.122+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:57.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:57 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:57.122+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:09:57.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:57 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:57.566+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:57.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:57 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:57.566+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:57.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:57 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:57.566+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:57.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:57 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:57.566+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:57.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:57 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:57.566+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:57.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:57 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:57.566+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.788273+0000 front 2024-04-26T02:09:26.788501+0000 (oldest deadline 2024-04-26T02:09:52.088510+0000) 2024-04-26T02:09:57.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:57 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:57.464+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:57.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:57 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:57.464+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:57.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:57 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:57.464+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:57.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:57 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:57.464+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:57.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:57 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:57.464+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:57.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:57 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:57.464+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:09:58.641 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:58 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:58.148+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:58.641 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:58 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:58.148+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:58.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:58 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:58.148+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:58.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:58 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:58.148+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:58.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:58 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:58.148+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:09:58.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:58 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:58.148+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:09:58.692 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:58 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:58.418+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:58.692 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:58 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:58.418+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:58.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:58 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:58.418+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:58.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:58 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:58.418+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:58.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:58 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:58.418+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:58.693 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:58 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:58.418+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:09:58.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:58 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:58.542+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:58.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:58 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:58.542+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:58.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:58 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:58.542+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:58.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:58 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:58.542+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:58.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:58 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:58.542+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:58.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:58 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:58.542+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.788273+0000 front 2024-04-26T02:09:26.788501+0000 (oldest deadline 2024-04-26T02:09:52.088510+0000) 2024-04-26T02:09:58.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:58 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:58.542+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.090398+0000 front 2024-04-26T02:09:32.090467+0000 (oldest deadline 2024-04-26T02:09:58.394186+0000) 2024-04-26T02:09:59.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:59 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:59.118+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:09:59.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:59 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:59.118+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:09:59.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:59 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:59.118+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:09:59.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:59 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:59.118+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:09:59.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:59 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:59.118+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:09:59.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:09:59 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:09:59.118+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:09:59.818 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:59 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:59.547+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:09:59.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:59 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:59.547+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:09:59.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:59 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:59.547+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:09:59.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:59 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:59.547+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:09:59.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:59 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:59.547+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:09:59.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:59 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:59.547+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.788273+0000 front 2024-04-26T02:09:26.788501+0000 (oldest deadline 2024-04-26T02:09:52.088510+0000) 2024-04-26T02:09:59.819 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:09:59 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:09:59.547+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.090398+0000 front 2024-04-26T02:09:32.090467+0000 (oldest deadline 2024-04-26T02:09:58.394186+0000) 2024-04-26T02:09:59.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:59 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:59.460+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:09:59.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:59 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:59.460+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:09:59.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:59 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:59.460+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:09:59.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:59 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:59.460+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:09:59.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:59 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:59.460+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:09:59.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:59 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:59.460+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:09:59.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:09:59 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:09:59.461+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:35.551442+0000 front 2024-04-26T02:09:35.551672+0000 (oldest deadline 2024-04-26T02:09:58.451175+0000) 2024-04-26T02:10:00.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:00 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:00.101+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:00.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:00 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:00.101+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:00.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:00 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:00.101+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:00.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:00 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:00.101+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:00.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:00 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:00.101+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:00.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:00 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:00.101+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:00.594 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:00 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:00.592+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:10:00.594 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:00 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:00.592+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:10:00.595 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:00 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:00.592+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:10:00.595 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:00 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:00.592+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:10:00.595 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:00 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:00.592+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:10:00.595 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:00 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:00.592+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.788273+0000 front 2024-04-26T02:09:26.788501+0000 (oldest deadline 2024-04-26T02:09:52.088510+0000) 2024-04-26T02:10:00.595 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:00 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:00.592+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.090398+0000 front 2024-04-26T02:09:32.090467+0000 (oldest deadline 2024-04-26T02:09:58.394186+0000) 2024-04-26T02:10:00.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:00 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:00.447+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:10:00.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:00 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:00.447+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:10:00.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:00 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:00.447+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:10:00.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:00 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:00.447+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:10:00.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:00 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:00.447+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:10:00.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:00 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:00.447+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:10:00.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:00 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:00.447+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:35.551442+0000 front 2024-04-26T02:09:35.551672+0000 (oldest deadline 2024-04-26T02:09:58.451175+0000) 2024-04-26T02:10:01.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:01 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:01.116+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:01.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:01 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:01.116+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:01.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:01 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:01.116+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:01.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:01 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:01.116+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:01.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:01 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:01.116+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:01.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:01 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:01.116+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:01.811 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:01 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:01.559+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:44.465654+0000 front 2024-04-26T02:08:44.465763+0000 (oldest deadline 2024-04-26T02:09:08.565864+0000) 2024-04-26T02:10:01.811 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:01 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:01.559+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:53.769764+0000 front 2024-04-26T02:08:53.769698+0000 (oldest deadline 2024-04-26T02:09:16.069979+0000) 2024-04-26T02:10:01.811 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:01 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:01.559+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.673866+0000 front 2024-04-26T02:09:00.673930+0000 (oldest deadline 2024-04-26T02:09:24.774022+0000) 2024-04-26T02:10:01.811 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:01 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:01.559+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:07.676488+0000 front 2024-04-26T02:09:07.676574+0000 (oldest deadline 2024-04-26T02:09:31.176862+0000) 2024-04-26T02:10:01.811 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:01 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:01.559+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:16.981839+0000 front 2024-04-26T02:09:16.981916+0000 (oldest deadline 2024-04-26T02:09:41.682098+0000) 2024-04-26T02:10:01.812 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:01 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:01.559+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.788273+0000 front 2024-04-26T02:09:26.788501+0000 (oldest deadline 2024-04-26T02:09:52.088510+0000) 2024-04-26T02:10:01.812 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:01 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9[63556]: 2024-04-26T02:10:01.559+0000 7f3220000700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.090398+0000 front 2024-04-26T02:09:32.090467+0000 (oldest deadline 2024-04-26T02:09:58.394186+0000) 2024-04-26T02:10:01.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:01 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:01.448+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:10:01.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:01 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:01.448+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:10:01.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:01 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:01.448+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:10:01.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:01 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:01.448+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:10:01.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:01 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:01.448+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:10:01.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:01 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:01.448+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:10:01.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:01 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:01.448+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:35.551442+0000 front 2024-04-26T02:09:35.551672+0000 (oldest deadline 2024-04-26T02:09:58.451175+0000) 2024-04-26T02:10:02.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:02 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:02.107+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:02.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:02 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:02.107+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:02.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:02 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:02.107+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:02.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:02 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:02.107+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:02.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:02 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:02.107+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:02.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:02 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:02.107+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:02.782 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:02 smithi006 podman[111456]: 2024-04-26 02:10:02.341301658 +0000 UTC m=+5.814579287 container died 6e4b86ec159885dda293bdbbec66c8ebb8673d78622dc97ab38a9f6b405fd471 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:10:02.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:02 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:02.470+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:10:02.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:02 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:02.470+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:10:02.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:02 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:02.470+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:10:02.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:02 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:02.470+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:10:02.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:02 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:02.470+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:10:02.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:02 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:02.470+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:10:02.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:02 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:02.471+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:35.551442+0000 front 2024-04-26T02:09:35.551672+0000 (oldest deadline 2024-04-26T02:09:58.451175+0000) 2024-04-26T02:10:03.068 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:02 smithi006 podman[111456]: 2024-04-26 02:10:02.986779563 +0000 UTC m=+6.460057186 container cleanup 6e4b86ec159885dda293bdbbec66c8ebb8673d78622dc97ab38a9f6b405fd471 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:10:03.068 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:02 smithi006 bash[111456]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9 2024-04-26T02:10:03.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:03 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:03.114+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:03.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:03 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:03.114+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:03.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:03 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:03.114+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:03.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:03 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:03.114+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:03.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:03 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:03.114+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:03.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:03 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:03.114+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:03.568 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:03 smithi006 podman[111478]: 2024-04-26 02:10:03.196451498 +0000 UTC m=+0.838000468 container remove 6e4b86ec159885dda293bdbbec66c8ebb8673d78622dc97ab38a9f6b405fd471 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:10:03.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:03 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:03.476+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:10:03.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:03 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:03.476+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:10:03.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:03 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:03.476+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:10:03.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:03 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:03.476+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:10:03.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:03 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:03.476+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:10:03.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:03 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:03.476+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:10:03.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:03 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:03.476+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:35.551442+0000 front 2024-04-26T02:09:35.551672+0000 (oldest deadline 2024-04-26T02:09:58.451175+0000) 2024-04-26T02:10:04.376 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:04 smithi006 podman[111567]: 2024-04-26 02:10:04.007049624 +0000 UTC m=+0.037840444 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:10:04.376 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:04 smithi006 podman[111567]: 2024-04-26 02:10:04.113976101 +0000 UTC m=+0.144766952 container create 0ddc029ac46cf0f276feece627e6f515b2b82cdbe4ded2db3bfcfcac894081df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:10:04.376 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:04 smithi006 podman[111567]: 2024-04-26 02:10:04.376161559 +0000 UTC m=+0.406952417 container init 0ddc029ac46cf0f276feece627e6f515b2b82cdbe4ded2db3bfcfcac894081df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-04-26T02:10:04.382 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:04.088+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:04.382 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:04.088+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:04.382 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:04.088+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:04.383 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:04.088+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:04.383 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:04.088+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:04.383 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:04.088+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:04.776 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:04 smithi006 podman[111567]: 2024-04-26 02:10:04.388260432 +0000 UTC m=+0.419051290 container start 0ddc029ac46cf0f276feece627e6f515b2b82cdbe4ded2db3bfcfcac894081df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:10:04.777 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:04 smithi006 podman[111567]: 2024-04-26 02:10:04.516576795 +0000 UTC m=+0.547367625 container attach 0ddc029ac46cf0f276feece627e6f515b2b82cdbe4ded2db3bfcfcac894081df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-26T02:10:04.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:04 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:04.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:10:04.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:04 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:04.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:10:04.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:04 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:04.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:10:04.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:04 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:04.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:10:04.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:04 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:04.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:10:04.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:04 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:04.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:10:04.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:04 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:04.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:35.551442+0000 front 2024-04-26T02:09:35.551672+0000 (oldest deadline 2024-04-26T02:09:58.451175+0000) 2024-04-26T02:10:05.068 INFO:journalctl@ceph.osd.9.smithi006.stdout:Apr 26 02:10:04 smithi006 podman[111567]: 2024-04-26 02:10:04.85687924 +0000 UTC m=+0.887670054 container died 0ddc029ac46cf0f276feece627e6f515b2b82cdbe4ded2db3bfcfcac894081df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-26T02:10:05.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:05 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:05.040+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:05.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:05 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:05.040+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:05.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:05 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:05.040+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:05.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:05 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:05.040+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:05.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:05 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:05.040+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:05.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:05 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:05.040+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:05.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.9.service' 2024-04-26T02:10:05.646 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:05.646 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-26T02:10:05.646 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-26T02:10:05.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.10 2024-04-26T02:10:05.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:05 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:05.535+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:10:05.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:05 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:05.535+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:10:05.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:05 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:05.535+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:10:05.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:05 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:05.535+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:10:05.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:05 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:05.535+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:10:05.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:05 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:05.535+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:10:05.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:05 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:05.535+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:35.551442+0000 front 2024-04-26T02:09:35.551672+0000 (oldest deadline 2024-04-26T02:09:58.451175+0000) 2024-04-26T02:10:05.944 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:05 smithi029 systemd[1]: Stopping Ceph osd.10 for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:06.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.009+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:06.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.009+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:06.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.009+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:06.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.009+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:06.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.009+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:06.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.009+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:06.442 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:05 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:05.977+0000 7f733b363700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:06.442 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:05 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:05.977+0000 7f733b363700 -1 osd.10 137 *** Got signal Terminated *** 2024-04-26T02:10:06.442 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:05 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:05.977+0000 7f733b363700 -1 osd.10 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:10:06.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:06 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:06.488+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:10:06.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:06 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:06.488+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:10:06.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:06 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:06.488+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:10:06.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:06 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:06.488+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:10:06.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:06 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:06.488+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:10:06.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:06 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:06.488+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:10:06.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:06 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:06.488+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:35.551442+0000 front 2024-04-26T02:09:35.551672+0000 (oldest deadline 2024-04-26T02:09:58.451175+0000) 2024-04-26T02:10:07.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.997+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:07.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.997+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:07.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.997+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:07.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.997+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:07.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.997+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:07.393 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.997+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:07.393 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:06 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:06.997+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6820 osd.7 since back 2024-04-26T02:09:41.899598+0000 front 2024-04-26T02:09:41.899451+0000 (oldest deadline 2024-04-26T02:10:06.599348+0000) 2024-04-26T02:10:07.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:07 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:07.521+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:10:07.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:07 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:07.521+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:10:07.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:07 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:07.521+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:10:07.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:07 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:07.521+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:10:07.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:07 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:07.521+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:10:07.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:07 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:07.521+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:10:07.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:07 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:07.521+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:35.551442+0000 front 2024-04-26T02:09:35.551672+0000 (oldest deadline 2024-04-26T02:09:58.451175+0000) 2024-04-26T02:10:08.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.020+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:08.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.020+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:08.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.020+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:08.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.020+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:08.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.020+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:08.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.020+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:08.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.020+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6820 osd.7 since back 2024-04-26T02:09:41.899598+0000 front 2024-04-26T02:09:41.899451+0000 (oldest deadline 2024-04-26T02:10:06.599348+0000) 2024-04-26T02:10:08.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:08 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:08.493+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:10:08.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:08 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:08.493+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:10:08.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:08 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:08.493+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:10:08.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:08 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:08.493+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:10:08.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:08 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:08.493+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:10:08.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:08 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:08.493+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:10:08.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:08 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:08.493+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:35.551442+0000 front 2024-04-26T02:09:35.551672+0000 (oldest deadline 2024-04-26T02:09:58.451175+0000) 2024-04-26T02:10:09.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.994+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:09.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.994+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:09.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.994+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:09.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.994+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:09.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.994+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:09.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.994+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:09.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:08.994+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6820 osd.7 since back 2024-04-26T02:09:41.899598+0000 front 2024-04-26T02:09:41.899451+0000 (oldest deadline 2024-04-26T02:10:06.599348+0000) 2024-04-26T02:10:09.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:09 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:09.503+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:10:09.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:09 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:09.503+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:10:09.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:09 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:09.503+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:10:09.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:09 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:09.503+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:10:09.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:09 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:09.503+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:10:09.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:09 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:09.503+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:10:09.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:09 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:09.503+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:35.551442+0000 front 2024-04-26T02:09:35.551672+0000 (oldest deadline 2024-04-26T02:09:58.451175+0000) 2024-04-26T02:10:10.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:10 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:10.005+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:10.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:10 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:10.005+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:10.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:10 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:10.005+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:10.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:10 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:10.005+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:10.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:10 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:10.005+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:10.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:10 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:10.005+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:10.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:10 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:10.005+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6820 osd.7 since back 2024-04-26T02:09:41.899598+0000 front 2024-04-26T02:09:41.899451+0000 (oldest deadline 2024-04-26T02:10:06.599348+0000) 2024-04-26T02:10:10.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:10 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:10.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:42.939890+0000 front 2024-04-26T02:08:42.939993+0000 (oldest deadline 2024-04-26T02:09:06.340430+0000) 2024-04-26T02:10:10.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:10 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:10.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:50.842609+0000 front 2024-04-26T02:08:50.842632+0000 (oldest deadline 2024-04-26T02:09:15.542720+0000) 2024-04-26T02:10:10.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:10 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:10.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.144413+0000 front 2024-04-26T02:09:00.144359+0000 (oldest deadline 2024-04-26T02:09:23.044528+0000) 2024-04-26T02:10:10.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:10 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:10.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6814 osd.3 since back 2024-04-26T02:09:09.947089+0000 front 2024-04-26T02:09:09.947363+0000 (oldest deadline 2024-04-26T02:09:34.646970+0000) 2024-04-26T02:10:10.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:10 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:10.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:15.747884+0000 front 2024-04-26T02:09:15.747921+0000 (oldest deadline 2024-04-26T02:09:41.648076+0000) 2024-04-26T02:10:10.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:10 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:10.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.349358+0000 front 2024-04-26T02:09:26.349258+0000 (oldest deadline 2024-04-26T02:09:50.449172+0000) 2024-04-26T02:10:10.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:10 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10[58363]: 2024-04-26T02:10:10.504+0000 7f733394f700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:35.551442+0000 front 2024-04-26T02:09:35.551672+0000 (oldest deadline 2024-04-26T02:09:58.451175+0000) 2024-04-26T02:10:11.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:11.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:11.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:11.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:11.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:11.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:11.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:11.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:11.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:11.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:11.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:11.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:11.393 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:11 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:11.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6820 osd.7 since back 2024-04-26T02:09:41.899598+0000 front 2024-04-26T02:09:41.899451+0000 (oldest deadline 2024-04-26T02:10:06.599348+0000) 2024-04-26T02:10:11.639 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:11 smithi029 podman[73374]: 2024-04-26 02:10:11.374623462 +0000 UTC m=+5.526550549 container died 2fcaace8a9768b033966183676081517244ea2c5b9c2cc552bd0b9eb4e0d7cf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T02:10:11.942 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:11 smithi029 podman[73374]: 2024-04-26 02:10:11.739907783 +0000 UTC m=+5.891834869 container cleanup 2fcaace8a9768b033966183676081517244ea2c5b9c2cc552bd0b9eb4e0d7cf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-26T02:10:11.943 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:11 smithi029 bash[73374]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10 2024-04-26T02:10:12.368 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:12 smithi029 podman[73391]: 2024-04-26 02:10:12.000616821 +0000 UTC m=+0.627922689 container remove 2fcaace8a9768b033966183676081517244ea2c5b9c2cc552bd0b9eb4e0d7cf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:10:12.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:12 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:12.059+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:12.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:12 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:12.059+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:12.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:12 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:12.059+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:12.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:12 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:12.059+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:12.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:12 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:12.059+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:12.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:12 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:12.059+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:12.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:12 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:12.059+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6820 osd.7 since back 2024-04-26T02:09:41.899598+0000 front 2024-04-26T02:09:41.899451+0000 (oldest deadline 2024-04-26T02:10:06.599348+0000) 2024-04-26T02:10:12.618 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:12 smithi029 podman[73482]: 2024-04-26 02:10:12.272612111 +0000 UTC m=+0.018723634 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:10:12.618 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:12 smithi029 podman[73482]: 2024-04-26 02:10:12.370486194 +0000 UTC m=+0.116597717 container create 35e5d2120f997502f3d7d2b9f7e5eb832be191ad37253d915e31abdee73032c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:10:12.619 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:12 smithi029 podman[73482]: 2024-04-26 02:10:12.616516684 +0000 UTC m=+0.362628207 container init 35e5d2120f997502f3d7d2b9f7e5eb832be191ad37253d915e31abdee73032c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:10:12.619 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:12 smithi029 podman[73482]: 2024-04-26 02:10:12.619676183 +0000 UTC m=+0.365787711 container start 35e5d2120f997502f3d7d2b9f7e5eb832be191ad37253d915e31abdee73032c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-26T02:10:12.924 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:12 smithi029 podman[73482]: 2024-04-26 02:10:12.725549569 +0000 UTC m=+0.471661104 container attach 35e5d2120f997502f3d7d2b9f7e5eb832be191ad37253d915e31abdee73032c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-26T02:10:13.192 INFO:journalctl@ceph.osd.10.smithi029.stdout:Apr 26 02:10:12 smithi029 podman[73482]: 2024-04-26 02:10:12.925561272 +0000 UTC m=+0.671672802 container died 35e5d2120f997502f3d7d2b9f7e5eb832be191ad37253d915e31abdee73032c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-26T02:10:13.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:13.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:13.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:13.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:13.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:13.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:13.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:13.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:13.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:13.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:13.393 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:13.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:13.393 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:13.061+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6820 osd.7 since back 2024-04-26T02:09:41.899598+0000 front 2024-04-26T02:09:41.899451+0000 (oldest deadline 2024-04-26T02:10:06.599348+0000) 2024-04-26T02:10:13.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.10.service' 2024-04-26T02:10:13.566 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:13.567 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-26T02:10:13.567 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-26T02:10:13.567 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.11 2024-04-26T02:10:13.892 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:13 smithi188 systemd[1]: Stopping Ceph osd.11 for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:13.892 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:13.843+0000 7f836ce9f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:13.892 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:13.843+0000 7f836ce9f700 -1 osd.11 137 *** Got signal Terminated *** 2024-04-26T02:10:13.892 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:13 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:13.843+0000 7f836ce9f700 -1 osd.11 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:10:14.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:14.099+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:14.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:14.099+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:14.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:14.099+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:14.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:14.099+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:14.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:14.099+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:14.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:14.099+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:14.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:14 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:14.099+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6820 osd.7 since back 2024-04-26T02:09:41.899598+0000 front 2024-04-26T02:09:41.899451+0000 (oldest deadline 2024-04-26T02:10:06.599348+0000) 2024-04-26T02:10:15.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:15.083+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:15.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:15.083+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:15.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:15.083+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:15.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:15.083+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:15.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:15.083+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:15.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:15.083+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:15.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:15.083+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6820 osd.7 since back 2024-04-26T02:09:41.899598+0000 front 2024-04-26T02:09:41.899451+0000 (oldest deadline 2024-04-26T02:10:06.599348+0000) 2024-04-26T02:10:15.393 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:15 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:15.083+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2024-04-26T02:09:51.800804+0000 front 2024-04-26T02:09:51.800786+0000 (oldest deadline 2024-04-26T02:10:14.700757+0000) 2024-04-26T02:10:16.391 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:16.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:16.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:16.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:16.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:16.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:16.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:16.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:16.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:16.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:16.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:16.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:16.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:16.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6820 osd.7 since back 2024-04-26T02:09:41.899598+0000 front 2024-04-26T02:09:41.899451+0000 (oldest deadline 2024-04-26T02:10:06.599348+0000) 2024-04-26T02:10:16.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:16 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:16.054+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2024-04-26T02:09:51.800804+0000 front 2024-04-26T02:09:51.800786+0000 (oldest deadline 2024-04-26T02:10:14.700757+0000) 2024-04-26T02:10:17.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:17.076+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:17.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:17.076+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:17.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:17.076+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:17.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:17.076+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:17.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:17.076+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:17.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:17.076+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:17.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:17.076+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6820 osd.7 since back 2024-04-26T02:09:41.899598+0000 front 2024-04-26T02:09:41.899451+0000 (oldest deadline 2024-04-26T02:10:06.599348+0000) 2024-04-26T02:10:17.393 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:17 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:17.076+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2024-04-26T02:09:51.800804+0000 front 2024-04-26T02:09:51.800786+0000 (oldest deadline 2024-04-26T02:10:14.700757+0000) 2024-04-26T02:10:18.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:18.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-04-26T02:08:39.588845+0000 front 2024-04-26T02:08:39.588649+0000 (oldest deadline 2024-04-26T02:09:05.488613+0000) 2024-04-26T02:10:18.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:18.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6804 osd.1 since back 2024-04-26T02:08:48.890747+0000 front 2024-04-26T02:08:48.890616+0000 (oldest deadline 2024-04-26T02:09:14.691456+0000) 2024-04-26T02:10:18.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:18.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-26T02:09:00.592556+0000 front 2024-04-26T02:09:00.592394+0000 (oldest deadline 2024-04-26T02:09:24.092489+0000) 2024-04-26T02:10:18.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:18.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6812 osd.4 since back 2024-04-26T02:09:18.695492+0000 front 2024-04-26T02:09:18.695447+0000 (oldest deadline 2024-04-26T02:09:44.595235+0000) 2024-04-26T02:10:18.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:18.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-26T02:09:26.196789+0000 front 2024-04-26T02:09:26.196761+0000 (oldest deadline 2024-04-26T02:09:52.096838+0000) 2024-04-26T02:10:18.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:18.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.6:6822 osd.6 since back 2024-04-26T02:09:32.097667+0000 front 2024-04-26T02:09:32.097844+0000 (oldest deadline 2024-04-26T02:09:57.198516+0000) 2024-04-26T02:10:18.392 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:18.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.29:6820 osd.7 since back 2024-04-26T02:09:41.899598+0000 front 2024-04-26T02:09:41.899451+0000 (oldest deadline 2024-04-26T02:10:06.599348+0000) 2024-04-26T02:10:18.393 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:18 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11[58480]: 2024-04-26T02:10:18.091+0000 7f836548b700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2024-04-26T02:09:51.800804+0000 front 2024-04-26T02:09:51.800786+0000 (oldest deadline 2024-04-26T02:10:14.700757+0000) 2024-04-26T02:10:19.677 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:19 smithi188 podman[74591]: 2024-04-26 02:10:19.427350207 +0000 UTC m=+5.666588489 container died d8a773005a8d96b392b7c6ccd2356198a0c0878bda21fda4000355f9354982f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-26T02:10:19.952 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:19 smithi188 podman[74591]: 2024-04-26 02:10:19.768457932 +0000 UTC m=+6.007696213 container cleanup d8a773005a8d96b392b7c6ccd2356198a0c0878bda21fda4000355f9354982f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T02:10:19.952 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:19 smithi188 bash[74591]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11 2024-04-26T02:10:20.360 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:19 smithi188 podman[74608]: 2024-04-26 02:10:19.952877185 +0000 UTC m=+0.524112509 container remove d8a773005a8d96b392b7c6ccd2356198a0c0878bda21fda4000355f9354982f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:10:20.641 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:20 smithi188 podman[74699]: 2024-04-26 02:10:20.36221293 +0000 UTC m=+0.100441158 container create 064445aa393423708eb2ef61316faeb6c2f25ddd503ced54a1eb7832e52e0858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-26T02:10:20.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:20 smithi188 podman[74699]: 2024-04-26 02:10:20.279752108 +0000 UTC m=+0.017980303 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:10:20.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:20 smithi188 podman[74699]: 2024-04-26 02:10:20.57310332 +0000 UTC m=+0.311331517 container init 064445aa393423708eb2ef61316faeb6c2f25ddd503ced54a1eb7832e52e0858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-26T02:10:20.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:20 smithi188 podman[74699]: 2024-04-26 02:10:20.576388529 +0000 UTC m=+0.314616727 container start 064445aa393423708eb2ef61316faeb6c2f25ddd503ced54a1eb7832e52e0858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T02:10:21.141 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:20 smithi188 podman[74699]: 2024-04-26 02:10:20.647582497 +0000 UTC m=+0.385810685 container attach 064445aa393423708eb2ef61316faeb6c2f25ddd503ced54a1eb7832e52e0858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:10:21.142 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 26 02:10:20 smithi188 podman[74699]: 2024-04-26 02:10:20.876919303 +0000 UTC m=+0.615147489 container died 064445aa393423708eb2ef61316faeb6c2f25ddd503ced54a1eb7832e52e0858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-04-26T02:10:21.445 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@osd.11.service' 2024-04-26T02:10:21.481 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:21.482 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-26T02:10:21.482 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-26T02:10:21.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.a 2024-04-26T02:10:21.819 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 02:10:21 smithi006 systemd[1]: Stopping Ceph mds.a for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:22.568 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 02:10:22 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-a[70893]: 2024-04-26T02:10:22.068+0000 7f5d7a800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:22.569 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 02:10:22 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-a[70893]: 2024-04-26T02:10:22.068+0000 7f5d7a800700 -1 mds.a *** got signal Terminated *** 2024-04-26T02:10:26.127 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 02:10:25 smithi006 podman[111684]: 2024-04-26 02:10:25.857880201 +0000 UTC m=+3.907845073 container died 359947f8e3430bbe05223c419c29c8370d84876503fdb688c71dc4051997879d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T02:10:26.430 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 02:10:26 smithi006 podman[111684]: 2024-04-26 02:10:26.237601625 +0000 UTC m=+4.287566522 container cleanup 359947f8e3430bbe05223c419c29c8370d84876503fdb688c71dc4051997879d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:10:26.430 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 02:10:26 smithi006 bash[111684]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-a 2024-04-26T02:10:26.818 INFO:journalctl@ceph.mds.a.smithi006.stdout:Apr 26 02:10:26 smithi006 podman[111698]: 2024-04-26 02:10:26.430481683 +0000 UTC m=+0.554412999 container remove 359947f8e3430bbe05223c419c29c8370d84876503fdb688c71dc4051997879d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-26T02:10:26.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.a.service' 2024-04-26T02:10:27.005 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:27.006 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-26T02:10:27.006 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-26T02:10:27.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.d 2024-04-26T02:10:27.568 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 02:10:27 smithi006 systemd[1]: Stopping Ceph mds.d for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:28.068 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 02:10:27 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-d[69610]: 2024-04-26T02:10:27.673+0000 7f6ba7a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:28.069 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 02:10:27 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-d[69610]: 2024-04-26T02:10:27.674+0000 7f6ba7a00700 -1 mds.d *** got signal Terminated *** 2024-04-26T02:10:31.954 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 02:10:31 smithi006 podman[111804]: 2024-04-26 02:10:31.694365517 +0000 UTC m=+4.159591244 container died f7c3779515b9b0434484859310f831f2ea40ae5731c372d6dce2c4a09cdae79a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-26T02:10:32.318 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 02:10:32 smithi006 podman[111804]: 2024-04-26 02:10:32.074494536 +0000 UTC m=+4.539720256 container cleanup f7c3779515b9b0434484859310f831f2ea40ae5731c372d6dce2c4a09cdae79a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:10:32.319 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 02:10:32 smithi006 bash[111804]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-d 2024-04-26T02:10:32.818 INFO:journalctl@ceph.mds.d.smithi006.stdout:Apr 26 02:10:32 smithi006 podman[111817]: 2024-04-26 02:10:32.450642055 +0000 UTC m=+0.740503428 container remove f7c3779515b9b0434484859310f831f2ea40ae5731c372d6dce2c4a09cdae79a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=) 2024-04-26T02:10:32.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.d.service' 2024-04-26T02:10:33.102 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:33.103 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-26T02:10:33.103 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-26T02:10:33.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.g 2024-04-26T02:10:33.568 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:10:33 smithi006 systemd[1]: Stopping Ceph mds.g for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:34.068 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:10:33 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g[70436]: 2024-04-26T02:10:33.701+0000 7f795e000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:34.069 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:10:33 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g[70436]: 2024-04-26T02:10:33.701+0000 7f795e000700 -1 mds.g *** got signal Terminated *** 2024-04-26T02:10:36.068 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:10:35 smithi006 podman[111924]: 2024-04-26 02:10:35.772493438 +0000 UTC m=+2.184534524 container died c3d95a031c7f9d83e5b3280455c843a94cdcbf26e2798a3c1c43cc336f251de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:10:36.509 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:10:36 smithi006 podman[111924]: 2024-04-26 02:10:36.311317352 +0000 UTC m=+2.723358466 container cleanup c3d95a031c7f9d83e5b3280455c843a94cdcbf26e2798a3c1c43cc336f251de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-26T02:10:36.510 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:10:36 smithi006 bash[111924]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g 2024-04-26T02:10:36.818 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:10:36 smithi006 podman[111938]: 2024-04-26 02:10:36.604737139 +0000 UTC m=+0.812463946 container remove c3d95a031c7f9d83e5b3280455c843a94cdcbf26e2798a3c1c43cc336f251de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-26T02:10:37.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.g.service' 2024-04-26T02:10:37.119 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:10:37 smithi006 systemd[1]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.g.service: Deactivated successfully. 2024-04-26T02:10:37.119 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:10:37 smithi006 systemd[1]: Stopped Ceph mds.g for a5b72bee-036c-11ef-bc93-c7b262605968. 2024-04-26T02:10:37.119 INFO:journalctl@ceph.mds.g.smithi006.stdout:Apr 26 02:10:37 smithi006 systemd[1]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.g.service: Consumed 42.066s CPU time. 2024-04-26T02:10:37.153 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:37.153 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-26T02:10:37.154 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-26T02:10:37.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.j 2024-04-26T02:10:37.568 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 02:10:37 smithi006 systemd[1]: Stopping Ceph mds.j for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:38.068 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 02:10:37 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-j[69199]: 2024-04-26T02:10:37.706+0000 7f3aec200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:38.069 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 02:10:37 smithi006 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-j[69199]: 2024-04-26T02:10:37.706+0000 7f3aec200700 -1 mds.j *** got signal Terminated *** 2024-04-26T02:10:40.064 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 02:10:39 smithi006 podman[112052]: 2024-04-26 02:10:39.744275902 +0000 UTC m=+2.165047104 container stop 1eef28c3762a032050a3979688988c00e670bce79ca381e6760f54333486e581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0) 2024-04-26T02:10:40.065 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 02:10:39 smithi006 podman[112052]: 2024-04-26 02:10:39.770426202 +0000 UTC m=+2.191197446 container died 1eef28c3762a032050a3979688988c00e670bce79ca381e6760f54333486e581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True) 2024-04-26T02:10:40.318 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 02:10:40 smithi006 podman[112052]: 2024-04-26 02:10:40.290718303 +0000 UTC m=+2.711489550 container cleanup 1eef28c3762a032050a3979688988c00e670bce79ca381e6760f54333486e581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-26T02:10:40.318 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 02:10:40 smithi006 bash[112052]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-j 2024-04-26T02:10:40.819 INFO:journalctl@ceph.mds.j.smithi006.stdout:Apr 26 02:10:40 smithi006 podman[112067]: 2024-04-26 02:10:40.483128837 +0000 UTC m=+0.717486745 container remove 1eef28c3762a032050a3979688988c00e670bce79ca381e6760f54333486e581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:10:40.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.j.service' 2024-04-26T02:10:41.106 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:41.106 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-26T02:10:41.106 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-26T02:10:41.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.b 2024-04-26T02:10:41.419 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 02:10:41 smithi029 systemd[1]: Stopping Ceph mds.b for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:41.692 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 02:10:41 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-b[63628]: 2024-04-26T02:10:41.419+0000 7f5925a7a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:41.692 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 02:10:41 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-b[63628]: 2024-04-26T02:10:41.419+0000 7f5925a7a700 -1 mds.b *** got signal Terminated *** 2024-04-26T02:10:44.040 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 02:10:43 smithi029 podman[73586]: 2024-04-26 02:10:43.726018565 +0000 UTC m=+2.418191449 container died e0537abc3b1fbdfab894b944c36e2765e4c233e89c6268b6fb6347833c736b0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:10:44.408 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 02:10:44 smithi029 podman[73586]: 2024-04-26 02:10:44.115590709 +0000 UTC m=+2.807763598 container cleanup e0537abc3b1fbdfab894b944c36e2765e4c233e89c6268b6fb6347833c736b0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:10:44.408 INFO:journalctl@ceph.mds.b.smithi029.stdout:Apr 26 02:10:44 smithi029 bash[73586]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-b 2024-04-26T02:10:44.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.b.service' 2024-04-26T02:10:44.558 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:44.558 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-26T02:10:44.558 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-26T02:10:44.559 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.e 2024-04-26T02:10:44.692 INFO:journalctl@ceph.mds.e.smithi029.stdout:Apr 26 02:10:44 smithi029 systemd[1]: Stopping Ceph mds.e for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:45.192 INFO:journalctl@ceph.mds.e.smithi029.stdout:Apr 26 02:10:44 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-e[63002]: 2024-04-26T02:10:44.895+0000 7f6d9456a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:45.192 INFO:journalctl@ceph.mds.e.smithi029.stdout:Apr 26 02:10:44 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-e[63002]: 2024-04-26T02:10:44.895+0000 7f6d9456a700 -1 mds.e *** got signal Terminated *** 2024-04-26T02:10:47.690 INFO:journalctl@ceph.mds.e.smithi029.stdout:Apr 26 02:10:47 smithi029 podman[73719]: 2024-04-26 02:10:47.419496339 +0000 UTC m=+2.634204682 container died 2a76fddeb2efb2a237d94edc80404d4ff94cd74edd785ca26bb91db1d61e4dd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T02:10:47.942 INFO:journalctl@ceph.mds.e.smithi029.stdout:Apr 26 02:10:47 smithi029 podman[73719]: 2024-04-26 02:10:47.8088315 +0000 UTC m=+3.023539839 container cleanup 2a76fddeb2efb2a237d94edc80404d4ff94cd74edd785ca26bb91db1d61e4dd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-26T02:10:47.943 INFO:journalctl@ceph.mds.e.smithi029.stdout:Apr 26 02:10:47 smithi029 bash[73719]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-e 2024-04-26T02:10:48.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.e.service' 2024-04-26T02:10:48.223 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:48.224 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-26T02:10:48.224 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-26T02:10:48.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.h 2024-04-26T02:10:48.442 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 02:10:48 smithi029 systemd[1]: Stopping Ceph mds.h for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:48.942 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 02:10:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-h[63311]: 2024-04-26T02:10:48.554+0000 7f38616ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:48.942 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 02:10:48 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-h[63311]: 2024-04-26T02:10:48.554+0000 7f38616ca700 -1 mds.h *** got signal Terminated *** 2024-04-26T02:10:52.008 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 02:10:51 smithi029 podman[73841]: 2024-04-26 02:10:51.734669623 +0000 UTC m=+3.288196323 container died cbe1eaab51c05ef85dea0cbbeefd99211211dfc8c7f78dfa6aa573480c13c3e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T02:10:52.385 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 02:10:52 smithi029 podman[73841]: 2024-04-26 02:10:52.116963254 +0000 UTC m=+3.670489958 container cleanup cbe1eaab51c05ef85dea0cbbeefd99211211dfc8c7f78dfa6aa573480c13c3e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:10:52.385 INFO:journalctl@ceph.mds.h.smithi029.stdout:Apr 26 02:10:52 smithi029 bash[73841]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-h 2024-04-26T02:10:52.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.h.service' 2024-04-26T02:10:52.544 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:52.545 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-26T02:10:52.545 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-26T02:10:52.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.k 2024-04-26T02:10:52.692 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 02:10:52 smithi029 systemd[1]: Stopping Ceph mds.k for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:53.192 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 02:10:52 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-k[62679]: 2024-04-26T02:10:52.866+0000 7ff58ed36700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:53.192 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 02:10:52 smithi029 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-k[62679]: 2024-04-26T02:10:52.866+0000 7ff58ed36700 -1 mds.k *** got signal Terminated *** 2024-04-26T02:10:54.942 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 02:10:54 smithi029 podman[73965]: 2024-04-26 02:10:54.62183487 +0000 UTC m=+1.856294236 container died a4fa7301454115d8f20386df0122966991b395c0df9075179282478358c7c4d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-26T02:10:54.942 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 02:10:54 smithi029 podman[73965]: 2024-04-26 02:10:54.94324657 +0000 UTC m=+2.177705924 container cleanup a4fa7301454115d8f20386df0122966991b395c0df9075179282478358c7c4d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-26T02:10:55.192 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 02:10:54 smithi029 bash[73965]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-k 2024-04-26T02:10:55.192 INFO:journalctl@ceph.mds.k.smithi029.stdout:Apr 26 02:10:55 smithi029 podman[73979]: 2024-04-26 02:10:55.137037149 +0000 UTC m=+0.517174362 container remove a4fa7301454115d8f20386df0122966991b395c0df9075179282478358c7c4d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-k, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-26T02:10:55.373 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.k.service' 2024-04-26T02:10:55.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:55.405 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-26T02:10:55.405 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-26T02:10:55.405 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.c 2024-04-26T02:10:55.697 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 02:10:55 smithi188 systemd[1]: Stopping Ceph mds.c for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:55.957 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 02:10:55 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-c[63920]: 2024-04-26T02:10:55.696+0000 7f6b8e00c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:55.957 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 02:10:55 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-c[63920]: 2024-04-26T02:10:55.696+0000 7f6b8e00c700 -1 mds.c *** got signal Terminated *** 2024-04-26T02:11:00.089 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 02:10:59 smithi188 podman[74804]: 2024-04-26 02:10:59.710141205 +0000 UTC m=+4.111687867 container died 9a2bf011e344b905b07fd3daaff01bbafafcf168360d0ee3365410cc8dd22c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T02:11:00.391 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 02:11:00 smithi188 podman[74804]: 2024-04-26 02:11:00.171840397 +0000 UTC m=+4.573387061 container cleanup 9a2bf011e344b905b07fd3daaff01bbafafcf168360d0ee3365410cc8dd22c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:11:00.392 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 26 02:11:00 smithi188 bash[74804]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-c 2024-04-26T02:11:00.599 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.c.service' 2024-04-26T02:11:00.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:11:00.634 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-26T02:11:00.634 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-26T02:11:00.634 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.f 2024-04-26T02:11:01.076 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:11:00 smithi188 systemd[1]: Stopping Ceph mds.f for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:11:01.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:11:01 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: 2024-04-26T02:11:01.076+0000 7f8bae86b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:11:01.392 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:11:01 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f[68149]: 2024-04-26T02:11:01.076+0000 7f8bae86b700 -1 mds.f *** got signal Terminated *** 2024-04-26T02:11:04.010 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:11:03 smithi188 podman[74937]: 2024-04-26 02:11:03.740823654 +0000 UTC m=+2.884108128 container died 2aa418024757e4413413f89dcf27cae30090281135eb22f3021c5524156bc5c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-26T02:11:04.368 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:11:04 smithi188 podman[74937]: 2024-04-26 02:11:04.101316269 +0000 UTC m=+3.244600751 container cleanup 2aa418024757e4413413f89dcf27cae30090281135eb22f3021c5524156bc5c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-26T02:11:04.368 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 26 02:11:04 smithi188 bash[74937]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-f 2024-04-26T02:11:04.533 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.f.service' 2024-04-26T02:11:04.565 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:11:04.565 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-26T02:11:04.566 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-26T02:11:04.566 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.i 2024-04-26T02:11:04.892 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 02:11:04 smithi188 systemd[1]: Stopping Ceph mds.i for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:11:04.892 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 02:11:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i[64227]: 2024-04-26T02:11:04.870+0000 7fee57fad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:11:04.892 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 02:11:04 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i[64227]: 2024-04-26T02:11:04.870+0000 7fee57fad700 -1 mds.i *** got signal Terminated *** 2024-04-26T02:11:07.995 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 02:11:07 smithi188 podman[75057]: 2024-04-26 02:11:07.739650386 +0000 UTC m=+2.960947161 container died 70a79c79ae13c666d4753ba2f1fa776d60d941edb58a01589972b6496d113c44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-26T02:11:07.995 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 02:11:07 smithi188 podman[75057]: 2024-04-26 02:11:07.99602544 +0000 UTC m=+3.217322213 container cleanup 70a79c79ae13c666d4753ba2f1fa776d60d941edb58a01589972b6496d113c44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T02:11:08.350 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 02:11:07 smithi188 bash[75057]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i 2024-04-26T02:11:08.350 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 26 02:11:08 smithi188 podman[75071]: 2024-04-26 02:11:08.230720941 +0000 UTC m=+0.490109204 container remove 70a79c79ae13c666d4753ba2f1fa776d60d941edb58a01589972b6496d113c44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:11:08.366 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.i.service' 2024-04-26T02:11:08.396 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:11:08.397 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-26T02:11:08.397 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-26T02:11:08.397 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.l 2024-04-26T02:11:08.641 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 26 02:11:08 smithi188 systemd[1]: Stopping Ceph mds.l for a5b72bee-036c-11ef-bc93-c7b262605968... 2024-04-26T02:11:09.141 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 26 02:11:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-l[63295]: 2024-04-26T02:11:08.740+0000 7fd50842f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:11:09.142 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 26 02:11:08 smithi188 ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-l[63295]: 2024-04-26T02:11:08.740+0000 7fd50842f700 -1 mds.l *** got signal Terminated *** 2024-04-26T02:11:11.772 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 26 02:11:11 smithi188 podman[75178]: 2024-04-26 02:11:11.496112332 +0000 UTC m=+2.887726607 container died 109c8f8d7b6c79a016e44feeae75fa4e22fffa7d9ff2c125d404373d1bcdc60a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T02:11:12.100 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 26 02:11:11 smithi188 podman[75178]: 2024-04-26 02:11:11.866535293 +0000 UTC m=+3.258149555 container cleanup 109c8f8d7b6c79a016e44feeae75fa4e22fffa7d9ff2c125d404373d1bcdc60a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-26T02:11:12.100 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 26 02:11:11 smithi188 bash[75178]: ceph-a5b72bee-036c-11ef-bc93-c7b262605968-mds-l 2024-04-26T02:11:12.241 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5b72bee-036c-11ef-bc93-c7b262605968@mds.l.service' 2024-04-26T02:11:12.271 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:11:12.271 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-26T02:11:12.271 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a5b72bee-036c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T02:11:12.605 INFO:teuthology.orchestra.run.smithi006.stdout:Deleting cluster with fsid: a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T02:11:17.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a5b72bee-036c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T02:11:17.955 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T02:11:20.425 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a5b72bee-036c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T02:11:20.615 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T02:11:22.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:11:23.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:11:23.118 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:11:23.144 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-26T02:11:23.146 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/remote/smithi006/crash 2024-04-26T02:11:23.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/crash -- . 2024-04-26T02:11:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:tar: /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T02:11:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:tar: Error is not recoverable: exiting now 2024-04-26T02:11:23.280 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/remote/smithi029/crash 2024-04-26T02:11:23.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/crash -- . 2024-04-26T02:11:23.359 INFO:teuthology.orchestra.run.smithi029.stderr:tar: /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T02:11:23.359 INFO:teuthology.orchestra.run.smithi029.stderr:tar: Error is not recoverable: exiting now 2024-04-26T02:11:23.361 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/remote/smithi188/crash 2024-04-26T02:11:23.362 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/crash -- . 2024-04-26T02:11:23.396 INFO:teuthology.orchestra.run.smithi188.stderr:tar: /var/lib/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T02:11:23.396 INFO:teuthology.orchestra.run.smithi188.stderr:tar: Error is not recoverable: exiting now 2024-04-26T02:11:23.397 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-26T02:11:23.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-26T02:11:23.507 INFO:tasks.cephadm:Compressing logs... 2024-04-26T02:11:23.507 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T02:11:23.511 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T02:11:23.514 DEBUG:teuthology.orchestra.run.smithi188:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T02:11:23.541 INFO:teuthology.orchestra.run.smithi188.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.log 2024-04-26T02:11:23.541 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-26T02:11:23.542 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-26T02:11:23.542 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-26T02:11:23.542 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T02:11:23.542 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-26T02:11:23.542 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-26T02:11:23.542 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-26T02:11:23.542 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-26T02:11:23.543 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-26T02:11:23.543 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T02:11:23.543 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-26T02:11:23.543 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-26T02:11:23.543 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T02:11:23.543 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-26T02:11:23.543 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T02:11:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.log 2024-04-26T02:11:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:: No such file or directory 2024-04-26T02:11:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T02:11:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-26T02:11:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-26T02:11:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-26T02:11:23.549 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-26T02:11:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-26T02:11:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-26T02:11:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-26T02:11:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.h.log: 90.4% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T02:11:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T02:11:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.10.log: 87.4% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T02:11:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-26T02:11:23.551 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T02:11:23.551 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T02:11:23.556 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.e.log: 94.3% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-26T02:11:23.557 INFO:teuthology.orchestra.run.smithi029.stderr: 92.5% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mgr.y.log.gzgzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-26T02:11:23.557 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-26T02:11:23.557 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-26T02:11:23.565 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T02:11:23.568 INFO:teuthology.orchestra.run.smithi006.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-26T02:11:23.568 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.log 2024-04-26T02:11:23.569 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T02:11:23.570 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-26T02:11:23.571 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-26T02:11:23.572 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.k.log: 91.0% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T02:11:23.573 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-26T02:11:23.574 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.j.log: 87.4% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T02:11:23.574 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-26T02:11:23.574 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cephadm.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T02:11:23.576 INFO:teuthology.orchestra.run.smithi006.stderr: 90.3% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T02:11:23.576 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-26T02:11:23.576 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-26T02:11:23.577 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-26T02:11:23.579 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T02:11:23.582 INFO:teuthology.orchestra.run.smithi029.stderr: 94.3% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-26T02:11:23.583 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.6.log: /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.cephadm.log: 84.1% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T02:11:23.584 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-26T02:11:23.584 INFO:teuthology.orchestra.run.smithi006.stderr: 94.4% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-26T02:11:23.587 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T02:11:23.589 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-26T02:11:23.590 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-26T02:11:23.599 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T02:11:23.605 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cephadm.log: 90.9% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T02:11:23.609 INFO:teuthology.orchestra.run.smithi006.stderr: 94.3% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-26T02:11:23.618 INFO:teuthology.orchestra.run.smithi006.stderr: 93.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T02:11:23.618 INFO:teuthology.orchestra.run.smithi006.stderr: 94.2% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-26T02:11:23.665 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T02:11:23.671 INFO:teuthology.orchestra.run.smithi188.stderr: 83.7% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T02:11:23.686 INFO:teuthology.orchestra.run.smithi188.stderr: 87.4% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T02:11:23.750 INFO:teuthology.orchestra.run.smithi188.stderr: 90.4% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T02:11:23.762 INFO:teuthology.orchestra.run.smithi188.stderr: 94.5% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-26T02:11:23.786 INFO:teuthology.orchestra.run.smithi188.stderr: 92.6% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-26T02:11:23.878 INFO:teuthology.orchestra.run.smithi188.stderr: 91.0% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T02:11:24.193 INFO:teuthology.orchestra.run.smithi006.stderr: 90.1% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-26T02:11:25.504 INFO:teuthology.orchestra.run.smithi029.stderr: 92.9% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-26T02:11:25.616 INFO:teuthology.orchestra.run.smithi006.stderr: 90.2% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-26T02:11:31.720 INFO:teuthology.orchestra.run.smithi029.stderr: 94.9% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-26T02:11:32.702 INFO:teuthology.orchestra.run.smithi006.stderr: 94.9% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-26T02:11:34.706 INFO:teuthology.orchestra.run.smithi006.stderr: 93.6% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-26T02:11:35.143 INFO:teuthology.orchestra.run.smithi029.stderr: 94.0% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-26T02:11:35.255 INFO:teuthology.orchestra.run.smithi029.stderr: 93.9% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-26T02:11:35.773 INFO:teuthology.orchestra.run.smithi006.stderr: 93.8% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-26T02:11:36.758 INFO:teuthology.orchestra.run.smithi006.stderr: 93.8% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-26T02:11:37.487 INFO:teuthology.orchestra.run.smithi029.stderr: 93.8% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-26T02:11:37.932 INFO:teuthology.orchestra.run.smithi006.stderr: 93.7% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-26T02:11:37.937 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-26T02:11:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m14.393s 2024-04-26T02:11:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m57.988s 2024-04-26T02:11:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m4.763s 2024-04-26T02:11:38.141 INFO:teuthology.orchestra.run.smithi188.stderr: 91.5% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-26T02:11:39.976 INFO:teuthology.orchestra.run.smithi029.stderr: 93.9% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-26T02:12:08.353 INFO:teuthology.orchestra.run.smithi188.stderr: 94.8% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-26T02:13:40.413 INFO:teuthology.orchestra.run.smithi029.stderr: 93.1% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-26T02:13:40.414 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-04-26T02:13:40.415 INFO:teuthology.orchestra.run.smithi029.stderr:real 2m16.884s 2024-04-26T02:13:40.415 INFO:teuthology.orchestra.run.smithi029.stderr:user 3m14.497s 2024-04-26T02:13:40.415 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m5.749s 2024-04-26T02:14:16.228 INFO:teuthology.orchestra.run.smithi188.stderr: 93.8% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-26T02:14:27.317 INFO:teuthology.orchestra.run.smithi188.stderr: 93.9% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-26T02:14:45.073 INFO:teuthology.orchestra.run.smithi188.stderr: 94.0% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-26T02:15:02.784 INFO:teuthology.orchestra.run.smithi188.stderr: 93.9% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-26T02:24:34.162 INFO:teuthology.orchestra.run.smithi188.stderr: 93.1% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-26T02:25:08.090 INFO:teuthology.orchestra.run.smithi188.stderr: 93.1% -- replaced with /var/log/ceph/a5b72bee-036c-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-26T02:25:08.092 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-26T02:25:08.093 INFO:teuthology.orchestra.run.smithi188.stderr:real 13m44.561s 2024-04-26T02:25:08.093 INFO:teuthology.orchestra.run.smithi188.stderr:user 5m54.086s 2024-04-26T02:25:08.093 INFO:teuthology.orchestra.run.smithi188.stderr:sys 0m16.584s 2024-04-26T02:25:08.093 INFO:tasks.cephadm:Archiving logs... 2024-04-26T02:25:08.094 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/remote/smithi006/log 2024-04-26T02:25:08.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T02:25:15.463 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/remote/smithi029/log 2024-04-26T02:25:15.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T02:25:42.537 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/remote/smithi188/log 2024-04-26T02:25:42.538 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T02:26:37.297 INFO:tasks.cephadm:Removing cluster... 2024-04-26T02:26:37.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a5b72bee-036c-11ef-bc93-c7b262605968 --force 2024-04-26T02:26:37.629 INFO:teuthology.orchestra.run.smithi006.stdout:Deleting cluster with fsid: a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T02:26:38.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a5b72bee-036c-11ef-bc93-c7b262605968 --force 2024-04-26T02:26:38.581 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T02:26:39.075 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a5b72bee-036c-11ef-bc93-c7b262605968 --force 2024-04-26T02:26:39.477 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: a5b72bee-036c-11ef-bc93-c7b262605968 2024-04-26T02:26:40.102 INFO:tasks.cephadm:Removing cephadm ... 2024-04-26T02:26:40.102 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T02:26:40.163 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T02:26:40.179 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T02:26:40.194 INFO:tasks.cephadm:Teardown complete 2024-04-26T02:26:40.195 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T02:26:40.209 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T02:26:40.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T02:26:40.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T02:26:40.221 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T02:26:40.371 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T02:26:40.381 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T02:26:40.381 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:26:40.389 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:26:40.392 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:26:40.404 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-04-26T02:26:40.405 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-04-26T02:26:40.407 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T02:26:40.408 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-04-26T02:26:40.408 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv01.front.sepia.ceph.com 3 6 377 35 +1843us[+1809us] +/- 49ms 2024-04-26T02:26:40.408 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -1987us[-1987us] +/- 52ms 2024-04-26T02:26:40.408 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 -199us[ -233us] +/- 96ms 2024-04-26T02:26:40.408 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T02:26:40.436 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T02:26:40.436 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-26T02:26:40.437 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv01.front.sepia.ceph.com 3 6 377 39 +1748us[+1736us] +/- 49ms 2024-04-26T02:26:40.437 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 38 -2073us[-2073us] +/- 52ms 2024-04-26T02:26:40.437 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -280us[ -280us] +/- 96ms 2024-04-26T02:26:40.437 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T02:26:40.440 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-04-26T02:26:40.450 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T02:26:40.451 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-04-26T02:26:40.451 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv01.front.sepia.ceph.com 3 6 377 37 +1759us[+1755us] +/- 49ms 2024-04-26T02:26:40.451 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 38 -2051us[-2054us] +/- 52ms 2024-04-26T02:26:40.451 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -266us[ -266us] +/- 96ms 2024-04-26T02:26:40.451 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T02:26:40.452 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T02:26:40.463 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T02:26:40.464 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T02:26:40.475 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T02:26:40.704 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T02:26:40.774 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T02:26:40.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T02:26:41.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:26:41.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T02:26:41.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:26:41.033 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T02:26:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:26:41.062 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T02:26:41.073 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T02:26:41.084 INFO:teuthology.task.internal:Duration was 3784.425982 seconds 2024-04-26T02:26:41.084 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T02:26:41.096 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T02:26:41.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T02:26:41.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T02:26:41.106 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T02:26:41.144 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T02:26:41.177 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T02:26:41.243 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T02:26:41.491 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T02:26:41.491 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-04-26T02:26:41.491 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T02:26:41.569 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-04-26T02:26:41.569 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T02:26:41.597 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-04-26T02:26:41.598 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T02:26:41.626 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T02:26:41.626 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T02:26:41.634 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T02:26:41.640 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T02:26:41.714 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T02:26:41.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T02:26:41.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T02:26:41.722 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T02:26:42.697 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T02:26:42.711 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T02:26:42.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T02:26:42.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T02:26:42.826 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T02:26:42.864 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T02:26:42.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:26:42.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:26:42.886 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:26:42.911 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-04-26T02:26:42.936 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-04-26T02:26:42.973 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-04-26T02:26:43.015 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:26:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:26:43.062 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:26:43.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:26:43.080 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:26:43.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:26:43.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T02:26:43.109 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T02:26:43.110 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/remote/smithi006 2024-04-26T02:26:43.111 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T02:26:43.235 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/remote/smithi029 2024-04-26T02:26:43.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T02:26:43.277 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244/remote/smithi188 2024-04-26T02:26:43.277 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T02:26:43.318 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T02:26:43.318 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T02:26:43.326 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T02:26:43.329 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T02:26:43.379 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T02:26:43.394 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T02:26:43.394 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T02:26:43.406 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T02:26:43.406 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T02:26:43.414 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T02:26:43.417 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T02:26:43.432 INFO:teuthology.orchestra.run.smithi029.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 02:26 /home/ubuntu/cephtest 2024-04-26T02:26:43.434 INFO:teuthology.orchestra.run.smithi188.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 02:26 /home/ubuntu/cephtest 2024-04-26T02:26:43.467 INFO:teuthology.orchestra.run.smithi006.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 02:26 /home/ubuntu/cephtest 2024-04-26T02:26:43.471 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T02:26:43.483 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T02:26:43.512 INFO:teuthology.nuke:Checking targets against current locks 2024-04-26T02:26:43.537 DEBUG:teuthology.nuke:shortname: smithi006 2024-04-26T02:26:43.537 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T02:26:43.559 DEBUG:teuthology.nuke:shortname: smithi029 2024-04-26T02:26:43.560 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T02:26:43.565 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:14:39.186321', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBBNzsTXM9Wpr+WED+Yxz5T4VXMwHSAByAiUm8DupeamFZuCPKxNWYGvMMwZXxDMtu4ojQGVqHkls7nekgy6MeA='} 2024-04-26T02:26:43.584 DEBUG:teuthology.nuke:shortname: smithi188 2024-04-26T02:26:43.584 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T02:26:43.593 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:14:39.190078', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdzfaroBoqBqlk5NNToTIGJEX96lvgz/J2jXAz5KcOK3XxIWy0dvKJBN6qekKg7QOcuQlQmD4OxdR16cS+7ruY='} 2024-04-26T02:26:43.607 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674244', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:14:39.188288', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCww6rPjgGqw4giVR74eCtSZPt/X0gpwYQNij325FnYYCFpBIDgC0tlZQjfWzHiH3jPbnvvffyPhsPDj+jmPVlc='} 2024-04-26T02:26:43.616 INFO:teuthology.orchestra.console.smithi006:Power off 2024-04-26T02:26:43.617 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T02:26:43.641 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2024-04-26T02:26:43.642 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:26:43.663 INFO:teuthology.orchestra.console.smithi029:Power off 2024-04-26T02:26:43.664 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T02:26:43.678 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-04-26T02:26:43.690 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2024-04-26T02:26:43.690 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:26:43.707 INFO:teuthology.orchestra.console.smithi188:Power off 2024-04-26T02:26:43.708 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T02:26:43.717 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-04-26T02:26:43.732 DEBUG:teuthology.orchestra.console.smithi188:power off output: Chassis Power Control: Down/Off 2024-04-26T02:26:43.732 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:26:43.759 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-26T02:26:47.680 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:26:47.718 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:26:47.760 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:26:47.791 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-04-26T02:26:47.829 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-04-26T02:26:47.871 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-26T02:26:51.794 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:26:51.829 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:26:51.873 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:26:51.906 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-04-26T02:26:51.945 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-04-26T02:26:51.985 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-26T02:26:52.047 INFO:teuthology.orchestra.console.smithi029:Power off completed 2024-04-26T02:26:55.910 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:26:55.986 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:26:56.020 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2024-04-26T02:26:56.097 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2024-04-26T02:26:56.122 INFO:teuthology.orchestra.console.smithi006:Power off completed 2024-04-26T02:26:56.199 INFO:teuthology.orchestra.console.smithi188:Power off completed 2024-04-26T02:26:56.305 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 3784.4259815216064 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi006 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afa1933b0fdbb6c99c947d1eda34d661d23cd327 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c1ec8d2ba0174cc1916ce0d0ce23f3fd status: fail success: false 2024-04-26T02:26:56.306 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T02:26:56.385 INFO:teuthology.run:FAIL